API

[Brian Pepple]

Hi,

I'm the developer over at Metron(1) and saw the thread on API Development (sadly, Google Groups doesn't let me reply to the existing thread), and thought I'd share my experience on how our users are consuming our API since you mentioned you didn't know how users could potentially use it.

By far, the most common usage appears to be folks retrieving metadata for tagging digital comics or using with a Plex-like systems, but some of the other uses I've seen are:

• Creating a pull list of upcoming comics to order from their local comic book store.
• Querying the API with a Comics UPC to get data to use for printing info labels to affix to the front of their bagged comics.
• The guys over at Omnibus were looking at using it, but I'm not sure what ever came of that.

For our users, the API is read-only, but we do provide post/patch API for our admins which make a lot of tasks (adding future comics, updating issues) **much** easier since I've written some python tools to help with those tasks.

To maintain QOS for the site, we require that users have an account to use the API (basic authentication), and use a throttle rate of around 30 requests/minute.

Anyway, that's a real basic explanation of our setup and hopefully that will be of some help to you. If you've got questions, don't hesitate to ask.

 

1) https://metron.cloud/

Take care,
/B

-- 
Brian Pepple <bpe...@metron.cloud>

https://about.me/brian.pepple

[Jochen G.]

Hi,

thanks for chiming in.

I guess my question was not so much about for what users it would use
it, but what data/access points they would like to have to be able to
use it.

Metadata for some of these tasks are the issue data, partly story data.
I assume dereferenced, so no foreign keys for stories or creators, but
the text.

But how would a user get there, i.e. would they need also search APIs
for series names and issue numbers ?

For barcodes that is a bit more simple, since only few resolve. But we
have hundreds of Batman #1 ?

Jochen

Am 12.05.24 um 21:59 schrieb Brian Pepple:

> Hi,
>
> I'm the developer over at Metron(1) and saw the thread on API
> Development (sadly, Google Groups doesn't let me reply to the existing
> thread), and thought I'd share my experience on how our users are
> consuming our API since you mentioned you didn't know how users could
> potentially use it.
>
> By far, the most common usage appears to be folks retrieving metadata
> for tagging digital comics or using with a Plex-like systems, but some
> of the other uses I've seen are:
>

> * Creating a pull list of upcoming comics to order from their local
> comic book store.
> * Querying the API with a Comics UPC to get data to use for printing

> info labels to affix to the front of their bagged comics.

> * The guys over at Omnibus were looking at using it, but I'm not sure

> what ever came of that.
>
>
> For our users, the API is read-only, but we do provide post/patch API
> for our admins which make a lot of tasks (adding future comics, updating
> issues) **much** easier since I've written some python tools to help
> with those tasks.
>
> To maintain QOS for the site, we require that users have an account to
> use the API (basic authentication), and use a throttle rate of around 30
> requests/minute.
>
> Anyway, that's a real basic explanation of our setup and hopefully that
> will be of some help to you. If you've got questions, don't hesitate to ask.
>
>
>
> 1) https://metron.cloud/
>
>
> Take care,
> /B
>
> --
> Brian Pepple<bpe...@metron.cloud>
>
> https://about.me/brian.pepple
>

> --
> You received this message because you are subscribed to the Google
> Groups "gcd-tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gcd-tech+u...@googlegroups.com
> <mailto:gcd-tech+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gcd-tech/778ae7b5-77da-400c-8df9-98bdbc621e98%40metron.cloud <https://groups.google.com/d/msgid/gcd-tech/778ae7b5-77da-400c-8df9-98bdbc621e98%40metron.cloud?utm_medium=email&utm_source=footer>.

[Brian Pepple]

On 5/13/24 16:50, Jochen G. wrote:

thanks for chiming in.

No problem. Figure it makes sense to share knowledge between projects since we most likely run into alot of the same problems.

I guess my question was not so much about for what users it would use it, but what data/access points they would like to have to be able to use it.

Metadata for some of these tasks are the issue data, partly story data. I assume dereferenced, so no foreign keys for stories or creators, but the text.

But how would a user get there, i.e. would they need also search APIs for series names and issue numbers ?

For barcodes that is a bit more simple, since only few resolve. But we have hundreds of Batman #1 ?
 

I've setup our API is to endpoints for our various models as follows:

• Arc
• Character
• Creator
• Issue
• Publisher
• Series
• Team
• Universe

Then we got serializers for ModelViewSet list actions which provides only the fields necessary to identify the item being searched for, and then another serializer with all the fields needed for the detail actions.

For example, here's our Arc Serializer (1), and the ArcViewSet (2).

So your example of searching for 'Batman #1' would be done by our users by make the following api call:

'https://metron.cloud/api/issue/?number=1&series_name=batman'

Which would return the attached 'batman_list_1.json' response.

Then user can then request the detail information for the  issue they want like this:

'https://metron.cloud/api/issue/34067/'

which would return the attached 'batman_detail_1.json' response.

We also add the foreign keys for our data, since our users could use that to prevent duplicates items in a Plex-like system when for example a name changes.

Anyway, I'm not 100% this is what you were asking, so if I was reading you question wrong just tell me.

1. https://github.com/bpepple/metron/blob/master/comicsdb/serializers/arc.py

2. https://github.com/bpepple/metron/blob/f1d50927f5ad233a2fe914a5ca821d04aa66bc76/comicsdb/views/viewsets.py#L55

[Richard]

In article <570a7b6f-b408-47eb...@garcke.de> you write:
>I guess my question was not so much about for what users it would use
>it, but what data/access points they would like to have to be able to
>use it.

Speaking for myself, I maintain my 'want list' as folders of images in
my phone. I wrote a nodejs program to scrape the entries from a (public)
collection on my.comics.org and update my local folder of photos to match
what the collection on my.comics.org contains. This means deleting
local images for issues not in the collection and downloading cover
images for issues in the collection that don't have photos locally.

The nodejs script is here:
<https://github.com/LegalizeAdulthood/ComicCovers>

If I had a REST API that would let me enumerate the issues in a
collection and obtain cover images for an issue, I could drop the
homebrew web scraping that I'm doing instead. Per-issue metadata
would be useful, but my metadata needs are pretty minimal (just need
to figure out the local name of the cover image).

-- Richard

--
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
The Terminals Wiki <http://terminals-wiki.org>
The Computer Graphics Museum <http://computergraphicsmuseum.org>
Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

[CT Garry]

Scraping for images is not supported in any way, and discouraged. I don't think we want to position comics.org as an image repository.

On Wednesday, May 15, 2024 at 11:57:45 AM UTC-7 

Speaking for myself, I maintain my 'want list' as folders of images in
my phone. I wrote a nodejs program to scrape the entries from a (public)
collection on my.comics.org and update my local folder of photos to match
what the collection on my.comics.org contains. This means deleting
local images for issues not in the collection and downloading cover
images for issues in the collection that don't have photos locally.

[Jochen G.]

If we provide the image URLs via an API and have the images available
via a CDN this wouldn't hamper us much in view of performance. But there
is a question if we want that.

Many are doing it anyway, while for individual purposes it is some
implementation work, I suspect the recent plugin to a comic reader makes
it easy for others.

Jochen

Am 16.05.24 um 20:38 schrieb CT Garry:

> Scraping for images is not supported in any way, and discouraged. I
> don't think we want to position comics.org as an image repository.
>
> On Wednesday, May 15, 2024 at 11:57:45 AM UTC-7
>
> Speaking for myself, I maintain my 'want list' as folders of images in
> my phone. I wrote a nodejs program to scrape the entries from a
> (public)

> collection on my.comics.org <http://my.comics.org> and update my

> local folder of photos to match

> what the collection on my.comics.org <http://my.comics.org>

> contains. This means deleting
> local images for issues not in the collection and downloading cover
> images for issues in the collection that don't have photos locally.
>
> If I had a REST API that would let me enumerate the issues in a
> collection and obtain cover images for an issue, I could drop the
> homebrew web scraping that I'm doing instead. Per-issue metadata
> would be useful, but my metadata needs are pretty minimal (just need
> to figure out the local name of the cover image).
>
> -- Richard
>

> --
> You received this message because you are subscribed to the Google
> Groups "gcd-tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gcd-tech+u...@googlegroups.com
> <mailto:gcd-tech+u...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/gcd-tech/6f8a3032-e38d-440b-9bc1-db6933334782n%40googlegroups.com <https://groups.google.com/d/msgid/gcd-tech/6f8a3032-e38d-440b-9bc1-db6933334782n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Richard]

In article <6f8a3032-e38d-440b...@googlegroups.com> you write:
>Scraping for images is not supported in any way, and discouraged. I don't
>think we want to position comics.org as an image repository.

The guidelines point out acceptable use cases, which boils down to:

- no mass indiscriminate scraping
- no repeat downloading; use a local image cache to avoid repeated d/l

...and I follow those guideline in my scraper. I don't think anyone
is using this code but me, so it's not any more burden than someone
browsing the site.

[Richard]

In article <6f8a3032-e38d-440b...@googlegroups.com> you write:

>Scraping for images is not supported in any way, and discouraged. I don't
>think we want to position comics.org as an image repository.

Also, if you're going through an API, then you have rate limiting
control over authenticated downloading of images based on access
token.

[CT Garry]

That's true, were we to implement an API, the access would be authenticated and rate limited. Furthermore, the API would supply the information about the image but not the image itself. The comics.org server for the image would remain the control point for proper use of the image URI. Your use case of taking that image payload and saving it to your own hosting (whether public or private) is still automation that you will have to implement, not us.

While a single user's impact is likely minimal, we're not in the habit of writing features for just one person. Whatever we do has to be extensible to the entire user base, with a Scotty fact of "times 10" to deal with unwieldy growth, however unlikely that is.

But in the end these images are not ours. I don't think we want the attention of being a source for distributing (willy nilly) all the copyrighted content without permission of the copyright owners. At the moment we have the right balance of "being sort of a place to get low grade images" and "not making it easy to violate copyrights en masse"

On Thu, May 16, 2024 at 3:37 PM Richard wrote: