REST API prototype [using all GCD data from the datadump]

[CT Garry]

The demo is ready with several dozen methods.  This is all GCD data. Nothing else.

http://comiccover.org/api/default.php

I could use some discussion with someone who is interested in this.

APIs can be used by programmers to get information from the GCD without using the legacy web UI or the data dump. What I have done so far demonstrates how the API might function. (see link above). My landing page lists functioning API call examples on my own website (left column) and links to the GCD in the middle column. The right column cues me on which data table is needed. This only utilizes information found in the data dump. It does not propose to present anything that is exclusive to the GCD website, such as the calendar or cover scans. There is more to come on this but people can try it now.

There are millions of records already accessible in the demo. Try it and change the method to another known record and get new results.

Thanks

Christopher

[CT Garry]

SSL version:  https://www.comiccover.org/api/default.php  

On Thu, Aug 6, 2020 at 9:27 PM CT Garry <ctg...@gmail.com> wrote:

The demo is ready with several dozen methods.  This is all GCD data. Nothing else.

I could use some discussion with someone who is interested in this.

APIs can be used by programmers to get information from the GCD without using the legacy web UI or the data dump. What I have done so far demonstrates how the API might function. (see link above). My landing page lists functioning API call examples on my own website (left column) and links to the GCD in the middle column. The right column cues me on which data table is needed. This only utilizes information found in the data dump. It does not propose to present anything that is exclusive to the GCD website, such as the calendar or cover scans. There is more to come on this but people can try it now.

There are millions of records already accessible in the demo. Try it and change the method to another known record and get new results.

Thanks

Christopher

--
You received this message because you are subscribed to the Google Groups "gcd-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gcd-tech+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/323f3ea0-8873-4898-9af9-0d33bfbf26b3o%40googlegroups.com.

[Juan F. Gonzalez]

In stories, it doesn't get the credits.

Juan

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CAO_1r4NjoZX%2Bex%3DODyipFiDQXxWhUwRRJtZw4jVvjoD8Eq8Arg%40mail.gmail.com.

[Juan F. Gonzalez]

Ok, I see it's a not yet implemented method.

Juan

[CT Garry]

If the credits are inline old-style, they'll be there. Object credits are going to be the second call, which as you say, is NIY.

In the example I supplied, shows the cover for Ascender #7, where the inline credits are blank:

{
  "id": 2782805,
  "title": "",
  "title_inferred": 0,
  "feature": "",
  "sequence_number": 0,
  "page_count": "1.000",
  "issue_id": 2041913,
  "script": "",
  "pencils": "",
  "inks": "",
  "colors": "",
  "letters": "",
  "editing": "",
  "genre": "",
  "characters": "",
  "synopsis": "",
  "reprint_notes": "",
  "created": "2019-12-08 10:08:16",
  "modified": "2019-12-08 10:08:16",
  "notes": "",
  "no_script": 1,
  "no_pencils": 0,
  "no_inks": 0,
  "no_colors": 0,
  "no_letters": 1,
  "no_editing": 1,
  "page_count_uncertain": 0,
  "type_id": 6,
  "job_number": "",
  "deleted": 0,
  "first_line": ""
}

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CAL5F5eq4oBTscz90rHX6bS%2BtMQS0XBFv18keY9GdjBD6ckY%3DoQ%40mail.gmail.com.

[CT Garry]

I've added 12 methods, one to each of the twelve "type" tables allowing a fetch of all records. This is in addition to the existing method for the "by id" retrievals. For example:

Existing (by id):

GET /v1/story_type/1

New (get all):

GET /v1/story_types

Thanks

Christopher

[Juan F. Gonzalez]

The methods you list, implemented or not, are a complete API?

Juan

--

You received this message because you are subscribed to the Google Groups "gcd-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gcd-tech+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/51bea0d8-1f31-4c9b-89a3-3eb8a494f88fo%40googlegroups.com.

[CT Garry]

Interesting question. There is more to add, I think. The demo focuses on what can methods be done with the data dump. And it's all read-only. REST API methodology allows for much more. The scope of this demo is thought to be limited to a "phase one". Meaning more advanced function can come in later phases.

A phase two set of methods might include more read-only APIs for site-only functions like "who are the users who have modifications to the series over its lifetime?" or "what's on the creator calendar today?" or "what are the new covers today?"

Another subsequence phase might make create/update/delete functionality too, but that is more sophisticated and should be done on a proper gcd beta. I can't do it from my demo platform.

Also, the landing page looks like a functional spec that just "lists methods." What's missing is all the decisions that need to be made about access permissions, throttling and authorization. In other words, this leads to the discussion on how to expose the API to developers - do we require credentials? Do we limit the number of transactions? How does reporting work? In a proper developer program, all API users (developers) are identified to the system including a organizationid, userid, applicationid, and application permissions or "scopes". Phase one could be "open to all" where a later phase can add more.

What I am hoping will happen in the short term is that we come to a phase one API that can be deployed on your beta.  That would probably to a technical port of what is in the final version of the demo.

And most of all it's important to understand what any API is for. This is a developer's tool for creating custom websites, phone apps, etc where the user interface and features largely don't have anything to do with GCD--just the data. Developers use it, not everyday consumers. This is not an alternate way for an every day reader to use GCD. Everyday readers will have no interest in the API.

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CAL5F5eqzSg69sp_s%2BgfjkiQ%3DAUdgsHyZwx%2BodXbk8r-pX7xFsg%40mail.gmail.com.

[Juan F. Gonzalez]

Yes, the question was not well expressed. 

What I'm seeing is that you are using an address similar to those in the website to generate a JSON file with the same data as the corresponding file, without the presentation.This addresses will be the language for the programmer writing applications, instead of the SQL or the Python library the site uses. This simplifies and limits the access. 

It's like a guide for scrapping. Or maybe not. If this is built on a dump you are doing you own pages, you just choose to mimic existing pages. Is that correct?

Juan

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CAO_1r4OpB%3D70gQ2dPZEypdy2DppX3JHj8kHYa1nxoNE6SJYLOw%40mail.gmail.com.

[CT Garry]

I'm happy to get inquiries.

You can think of it as a guide for scraping but the potential for a full REST API is so much more.  As you indicate, the truth is in the data. https://www.comics.org is one interface to that data--a tailored, mature application for web readers. The proposal is to begin to build  https://api.comics.org which would be a new interface to that data that is completely devoid of presentation and is featured well enough for developers to come up with their own solutions either webs or apps or other thin clients.

We can keep a good eye on the goals of the GCD and make sure that the API supports those goals. It's not a foregone conclusion that we will ever have an API, but this exercise should be informative.

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CAL5F5eqjxce5iLN5YxcEY_ZnOAz_vS9UXsQ8J6Sdd_d-p6K%2BFA%40mail.gmail.com.

[CT Garry]

All GET methods in an API are read-only, but they answer a question for the developer. There are infinite questions and some questions are going to work in phase one and some won't. For example, I don't recommend an open-ended search API. But I do recommend "byname" methods for specific tables. In other words, all the methods proposed right now are either "byid" or "showall" (if the table is short enough, like credit_types).  

To-do list for the Demo:

1. finish implementing the methods outlined.
2. work with GCD to get the feature_logo table into the data dump
3. solve interface issues with GoDaddy for the demo host that are limiting the size of the API responses.
4. begin adding 'byname" methods for looking into a table for records matching a string.
5. push the demo codebase to a public GIT repository
6. incorporate feedback and feature requests 
7. update the API demo landing page

Thanks

Christopher

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CAL5F5eqjxce5iLN5YxcEY_ZnOAz_vS9UXsQ8J6Sdd_d-p6K%2BFA%40mail.gmail.com.

[Juan F. Gonzalez]

It's like a LAMP application with an additional layer and without the top layer.

That top layer is the only thing the app programmer needs to do.

Juan

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CAO_1r4NexYz0BBxgNhNXRxPBrPRSK13X20P0gA-dB87XBiYB3w%40mail.gmail.com.

[CT Garry]

This particular REST API is written using LAMP, yes. A developer is not required to use any of those four components in order to work with the API.  Any user-agent that can generate an https request to the API is good to go for now -- this is what makes REST so magical.

[CT Garry]

3 updates:

• Source control has been moved to https://github.com/ctgarry/api.comics.org

• Deployed a 'name' search method for each of creator, series and publisher. Results are limited to 25 at time but can be paged, like this:

/v1/publisher/?name=dc&page=1

/v1/publisher/?name=dc&page=2

/v1/publisher/?name=dc&page=3

• The empty JSON set bug was RCAd (non-UTF8 characters were breaking JSON_encode). Fix resolved numerous bugs.

thanks

Christopher

[CT Garry]

Data updated to dump from 2020-07-29 to new version 2020-08-15

No feature changes.

Bug fix:

   /api/v1/feature_logo/93/ now returns data since the table has been added to the dump - thanks Miki

Thanks

Christopher "ctg"

[CT Garry]

A dozen new methods added 2020-08-16 -- includes complete handling of printer tables and issue reprints. Repo updated 2020-08-16. Based on data dump 2020-08-15. https://www.comiccover.org/api/default.php   

Thanks

Christopher "ctg"

[CT Garry]

More updates made and new methods published to v1. The goal for the foreseeable future is to get all the listed greyed out methods finished. Then gather feedback on what's working and what's missing. I am migrating the implementation as I go, so the source code is changing drastically as I go without disturbing any current function. For example, the staged JSON functions that had depended on mySql 5.7 JSON all broke on production because GoDaddy uses 5.6. That's solved and the nested results are working.  It would be worth discussing a style-guide at some point. field-naming, what database fields are better left out, etc. And I could use someone to write a test app leveraging the API. If wishes were fishes.

Repo updated 2020-08-19. Based on data dump 2020-08-15. https://www.comiccover.org/api/default.php    

 

Thanks for reading.

Christopher "ctg"

[CT Garry]

First draft of the complete functional spec is here: https://app.swaggerhub.com/search?type=API&owner=ctgarry. The two underlying YAML documents are together 4000 lines long, but the site UI makes it easy to read and walk through it--and to run the API calls from a form.  

Swagger is a tool for Opensource API specification. We use it at AT&T so that Product Development Engineers can speak to the implementation team. And it suits this project. "Swagger files" encapsulate everything that the spec-writer wants the developer to do. This is interesting because GCD has the option of going to a developer with the functional spec and saying 'this is what we want on our platform.'

This functional spec parallels the implementation on the demo page. Demo is 3/4 done compared to the functional spec.  I have been contacted by one developer looking at writing a device app. I've given him my phone number for text support on my existing prototype API for his app. 

Anyone else can ask for my number as well, either for a chat or development.

Repo updated 2020-08-21. Based on data dump 2020-08-15. https://www.comiccover.org/api/default.php 

Thanks

Christopher "ctg"

[Tony Rose]

I don't want to make light of what you've done by making a stupid interpretation of it, but you've done a lot of work!

tony

--

You received this message because you are subscribed to the Google Groups "gcd-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gcd-tech+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/ecd3ce83-8cf9-42e9-ae03-509881c27cden%40googlegroups.com.

[CT Garry]

Thank you sir.  I could say a lot, but I will say this in particular: I'm devoted to GCD because of its durability over the decades, and at the core of all GCD vision and charter, data is king. In my fifty years of hobbies, none has been more important to me than this one. If the API helps, then I will be happy.

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CALAa99o8poEzyNYPhGdzm9QL1E8swmrOHqm%2BKX223S58mGGmVw%40mail.gmail.com.

[Tony Rose]

Well said.

tony

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-tech/CAO_1r4NJ94Yjr9SxcKKmPDsguuR%3D9%2BZG%3DQPx7T51RVQjHWkw7w%40mail.gmail.com.

[CT Garry]

Data re-imported from https://www.comics.org/download - version 2020-08-29

[CT Garry]

Data re-imported from https://www.comics.org/download - version 2020-09-15

Good feedback so far. Keep it coming

[CT Garry]

Data re-imported from https://www.comics.org/download - version 2020-10-29  

[CT Garry]

2 updates:

Per request of jaymarvels, barcode searches are implemented now. Here is an example of an exact match:
https://www.comiccover.org/api/v1/issue/?barcode=76194120001990111
... and partial matches will return multiple comics. 

Data is re-imported twice a month from https://www.comics.org/download - current version 2020-11-29    

[CT Garry]

Data is imported again from https://www.comics.org/download - current version 2021-01-29

[CT Garry]

Data is imported again from https://www.comics.org/download - current version 2021-03-01 03:48:39

Apparently, the character data is not in the data dumps yet. As soon as that arrives I write APIs for that.

Thanks

Christopher

[Jochen G.]

The character data won't be in the dump before we are reasonably certain
the table design is stabilised.

Am 02.03.21 um 01:51 schrieb CT Garry:

>
>
> Data is imported again from https://www.comics.org/download - current
> version 2021-03-01 03:48:39
> Apparently, the character data is not in the data dumps yet. As soon as
> that arrives I write APIs for that.
>
> Thanks
> Christopher
>  
>
>
> On Thursday, August 6, 2020 at 9:27:46 PM
> UTC-7, CT Garry wrote:
>
>
> The demo is ready with several dozen
> methods.  This is all GCD data. Nothing
> else.
>
> http://comiccover.org/api/default.php

> *
> *
> *I could use some discussion with
> someone who is interested in this.*

>
> APIs can be used by programmers to get
> information from the GCD without using
> the legacy web UI or the data dump. What
> I have done so far demonstrates how the
> API might function. (see link above). My
> landing page lists functioning API call
> examples on my own website (left column)
> and links to the GCD in the middle
> column. The right column cues me on
> which data table is needed. This only
> utilizes information found in the data
> dump. It does not propose to present
> anything that is exclusive to the GCD
> website, such as the calendar or cover
> scans. There is more to come on this but
> people can try it now.
>

> *There are millions of records already

> accessible in the demo. Try it and
> change the method to another known

> record and get new results.*
>
> Thanks
> Christopher

>
> --
> You received this message because you are subscribed to the Google
> Groups "gcd-tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gcd-tech+u...@googlegroups.com

> <mailto:gcd-tech+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/gcd-tech/3b22d55c-e714-4297-a1ed-c036a35fcccan%40googlegroups.com
> <https://groups.google.com/d/msgid/gcd-tech/3b22d55c-e714-4297-a1ed-c036a35fcccan%40googlegroups.com?utm_medium=email&utm_source=footer>.