Covers?

[Matthew Cox]

If you're running a local installation, or if you're (Like me) trying to run a homemade django app using the GCD SQL dump as the cononical data, how do you handle covers? Hotlinking isn't a good idea, especially given the severity of how Cloudflare treats outside links to the cover file server.

Thanks,

Matt

[Kyle Jerviss]

There currently isn't a solution for this, other than scanning all of
your books.

Before Cloudflare, it was somewhat easy to download covers, and it
seemed like a good trade to scan and upload any covers that GCD was
missing in exchange for being able to download GCD covers for use in a
private database. Now, not so much.

> --
> You received this message because you are subscribed to the Google
> Groups "gcd-tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gcd-tech+u...@googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/gcd-tech/018f0d7f-a4f1-475e-a37d-55dc770ab196n%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/gcd-tech/018f0d7f-a4f1-475e-a37d-55dc770ab196n%40googlegroups.com?utm_medium=email&utm_source=footer

[Matthew Cox]

Yes- agreed. Cloudflare settings have made hotlinking or downloading impossible. I don't know if there's a middle ground to protect versus bots or not, but it would be, as you say, a nice tradeoff.

To view this discussion visit https://groups.google.com/d/msgid/gcd-tech/132c920d6a2c28385cb17e378d301f3a%40jerviss.org.

[Jochen G.]

If anyone has a way to keep the site alive in the current bot-age while
allowing cover access we can investigate that.

It is not Cloudflare that is the problem, the bots are.

We cannot pass logged-in users through Cloudflare, which would be the
most obvious option.

Am 15.07.26 um 12:41 schrieb Kyle Jerviss:

[Kyle Jerviss]

The API service isn't behind Cloudflare, is it?

Or, I'd be willing to host a private, members-only, cover archive/backup
for free. There would be some technical issues to solve, but probably
not serious ones.

[Adam Hernandez]

All,

Can someone explain what the problem is like I'm 5 please. I'm currently running a web service api for my pullbox application that serves up the weekly releases and their metadata which includes cover links. I'm wondering if we can use that? 

Thanks, 

Adam

[Katie Delia]

I'm not sure I understand the problem, but what I THINK it is:

We get a lot of people/sites/bots that like to run through the site and pull all the cover images. This causes problems for the site, as there are over 1.35 million covers. The site gets huge lags, goes down completely, etc. So we put in place safeguards to stop that from happening. Those are the safeguards you're coming across.  

Anyone, feel free to correct me.

Katie----:>

[Kyle Jerviss]

The problem, from the other point of view, is that many people who
contribute to this site have personal databases for their own private
collections. Before the CloudFlare-ification of the site, we could use
covers that other people had contributed. Now, we can't.


I can't speak for anyone else, but about half of my cover scanning
workflow involves things that I don't need to do for my own private
database. This is work that I only do so that I can contribute those
cover scans here. (For my purposes, uncropped scans that show the edges
and corners are actually better.) Additionally, about 90% of the
"hassle" in my overall workflow involves waiting for submissions to be
approved, for the data dump to update, etc.

As I said below, doubling my scanning work was a good trade off because
it meant that I didn't need to scan every single comic book that I own.
There is a similar calculation that goes into the other series and issue
data - I fill in a lot more blanks here than what my private system
cares about. Doing this on a fraction of my comics in exchange for
being able to use community data on the others seems like a good trade.

I have several thousand more comics that I need to index. Part of the
reason (not the only reason) why I haven't been eager to resume working
is because I haven't decided at what level I'm going to be contributing
here going forward. Part of my workflow involved storing comics
unsealed and in a temporary box until my system could verify that my
database matched the GCD data and that the cover had been uploaded
correctly. That way I didn't seal and permanently file anything away
that might need a second look or a rescan or whatever. The way things
are now, the only way I can move forward is to turn that off.


Since the API is intended to be used by scripts, it is not behind
CloudFlare, so letting authenticated users fetch covers using the API
seems like a good solution all around. Set a rate limit based on the
account's IMPs and it can't be abused by new account spam.

And my offer to host a covers archive for members was serious, if the
API thing is unworkable for some reason. I've got the capacity. We'd
just need a way for the main site to provide a signed JWT (or something
similar) so that my servers can verify membership without anyone needing
to trust me with account info.


Just a quick bit about JWT: The main site server has everyone's account
info. It could generate a URL for logged-in members that includes an
encoded token that is cryptographically signed that simply states that
the person who has the token is a member in good standing, expires in X
hours. Another server can verify the signature and be confident that
whoever is presenting the token is authorized. The other server doesn't
need to know anything about the account - no password, not even the
username. There are some tricks that you can do with cookies and
redirects to make it even more convenient. This also makes each request
stateless, so you can scale it up very easily behind a load balancer.

>>> [2]

>>>>>
>
> --
> You received this message because you are subscribed to the Google
> Groups "gcd-tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gcd-tech+u...@googlegroups.com.
> To view this discussion visit

> https://groups.google.com/d/msgid/gcd-tech/118b0179-0a66-4466-ac89-7fa4f9e29ad1n%40googlegroups.com
> [3].
>
>
> Links:
> ------
> [1] http://40googlegroups.com
> [2] http://40googlegroups.com?utm_medium=email&utm_source=footer
> [3]
> https://groups.google.com/d/msgid/gcd-tech/118b0179-0a66-4466-ac89-7fa4f9e29ad1n%40googlegroups.com?utm_medium=email&utm_source=footer

[Jochen G.]

We could see to provide cover access also via an API-endpoint. API-Urls
go through Cloudflare, but (will) need authorization.

Jochen

Am 23.07.26 um 03:43 schrieb Kyle Jerviss:

[Adam Hernandez]

Jochen, should I update the PRD for APIv2 to include and endpoint for covers? That way its rate limited and required a key? I can do that easy enough.

Adam

[Jochen G.]

Yes, that would be good.

Thanks.
Jochen

Am 27.07.26 um 19:04 schrieb Adam Hernandez:

> a4f1-475e- <https://groups.google.com/d/msgid/gcd-tech/018f0d7f-
> a4f1-475e->
> >>>>
> >>>>>>> a37d-55dc770ab196n%40googlegroups.com
> <http://40googlegroups.com> [1]

> >>>>>>> [1].
> >>>>>>>
> >>>>>>>
> >>>>>>> Links:
> >>>>>>> ------
> >>>>>>> [1]
> >>>> https://groups.google.com/d/msgid/gcd-tech/018f0d7f-a4f1-475e-

> <https://groups.google.com/d/msgid/gcd-tech/018f0d7f-a4f1-475e->
> >>>>>>>
> >>>>
> >>>
> >> a37d-55dc770ab196n%40googlegroups.com?
> utm_medium=email&utm_source=footer <http://40googlegroups.com?
> utm_medium=email&utm_source=footer>

> >>>> [2]
> >>>>>>
> >>
> >>  --
> >> You received this message because you are subscribed to the Google
> >> Groups "gcd-tech" group.
> >> To unsubscribe from this group and stop receiving emails from
> it, send
> >> an email to gcd-tech+u...@googlegroups.com.
> >> To view this discussion visit
> >> https://groups.google.com/d/msgid/gcd-tech/118b0179-0a66-4466-

> <https://groups.google.com/d/msgid/gcd-tech/118b0179-0a66-4466->
> >> ac89-7fa4f9e29ad1n%40googlegroups.com <http://40googlegroups.com>
> >> [3].
> >>
> >>
> >> Links:
> >> ------
> >> [1] http://40googlegroups.com <http://40googlegroups.com>
> >> [2] http://40googlegroups.com?utm_medium=email&utm_source=footer
> <http://40googlegroups.com?utm_medium=email&utm_source=footer>
> >> [3] https://groups.google.com/d/msgid/gcd-
> tech/118b0179-0a66-4466- <https://groups.google.com/d/msgid/gcd-
> tech/118b0179-0a66-4466->
> >> ac89-7fa4f9e29ad1n%40googlegroups.com?
> utm_medium=email&utm_source=footer <http://40googlegroups.com?
> utm_medium=email&utm_source=footer>

> >
>
> --
> You received this message because you are subscribed to the Google
> Groups "gcd-tech" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to gcd-tech+u...@googlegroups.com <mailto:gcd-
> tech+uns...@googlegroups.com>.

> To view this discussion visit https://groups.google.com/d/msgid/gcd-

> tech/2fd6c407-2a3e-4b84-9e9b-dbdfa956d615n%40googlegroups.com <https://
> groups.google.com/d/msgid/gcd-tech/2fd6c407-2a3e-4b84-9e9b-
> dbdfa956d615n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Adam Hernandez]

I updated the PRD with the following changes to implement a covers end point. Let me know if anyone has any feedback. 

Adam

Tier 4 - Cover Scans

GET /api/v2/covers/                    List active cover scans

GET /api/v2/covers/{id}/              Cover scan detail

Resource semantics: Each API record maps one-to-one to an active Cover row and therefore to one owning Issue. A series does not have a separate cover entity; series covers are retrieved with the series filter. Variant covers are ordinary Cover rows owned by variant Issue rows.

Fields: id, issue (id, series id + name, descriptor, variant_of, variant_name, variant_cover_status), images (thumbnail_url, medium_url, large_url), is_wraparound, display_restricted, front_crop (detail only), created, modified, last_upload.

Filters: issue, series, publisher, variant_of, is_variant, variant_cover_status, is_wraparound, modified (range), created (range), last_upload (range).

Variant behavior: issue=<id> returns scans directly owned by that exact issue. It does not silently substitute the base issue's scan for a No Difference variant. The Issue endpoint remains the source for the effective display cover and may inherit the base cover when variant_cover_status is No Difference.

Image policy: List responses expose the identification thumbnail only. Detail responses expose all permitted generated sizes. When limit_display is true, thumbnail_url remains available while medium_url and large_url are null. Original uploads and replacement-history files are never exposed. The existing Issue cover_url behavior must be brought into alignment with this restriction.

Reprint boundary: Cover scans are not directly related to Story or Reprint rows. Cover-art reuse remains represented by cover stories and Reprint relationships. Phase 4A adds origin_story_type and target_story_type filters to /api/v2/reprints/ so clients can discover cover-to-cover and cover-to-interior reprints without assigning an inferred reprint relationship to a particular scan.

Performance and validation: Cover list/detail endpoints receive serializer, filter, view, soft-delete, conditional-request, and fixed-query-count tests. Query plans must be validated against a cover-populated beta or production-like dataset before adding any performance index; the current local production copy omits Cover rows. Any migration must be justified by measured EXPLAIN evidence rather than added speculatively.

[Kyle Jerviss]

I'm sure that being able to pull metadata about covers will be very
useful to some people, but that isn't the problem that this thread is
about.

Also, where did the policy section come from?

>>>>>>>>>> a37d-55dc770ab196n%40googlegroups.com [1]

>>> <http://40googlegroups.com> [1]
>>>>>>>>>> [1].
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Links:
>>>>>>>>>> ------
>>>>>>>>>> [1]
>>>>>>>
>> https://groups.google.com/d/msgid/gcd-tech/018f0d7f-a4f1-475e-
>>> <https://groups.google.com/d/msgid/gcd-tech/018f0d7f-a4f1-475e->
>>>>>>>>>>
>>>>>>>
>>>>>>

>>>>> a37d-55dc770ab196n%40googlegroups.com [1]?

>>> utm_medium=email&utm_source=footer <http://40googlegroups.com?
>>> utm_medium=email&utm_source=footer>
>>>>>>> [2]
>>>>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the
>> Google
>>>>> Groups "gcd-tech" group.
>>>>> To unsubscribe from this group and stop receiving emails from
>>> it, send
>>>>> an email to gcd-tech+u...@googlegroups.com.
>>>>> To view this discussion visit
>>>>> https://groups.google.com/d/msgid/gcd-tech/118b0179-0a66-4466-
>>> <https://groups.google.com/d/msgid/gcd-tech/118b0179-0a66-4466->

>>>>> ac89-7fa4f9e29ad1n%40googlegroups.com [1]

>> <http://40googlegroups.com>
>>>>> [3].
>>>>>
>>>>>
>>>>> Links:
>>>>> ------
>>>>> [1] http://40googlegroups.com <http://40googlegroups.com>
>>>>> [2]

>> http://40googlegroups.com?utm_medium=email&utm_source=footer [2]
>>> <http://40googlegroups.com?utm_medium=email&utm_source=footer [2]>

>>
>>>>> [3] https://groups.google.com/d/msgid/gcd-
>>> tech/118b0179-0a66-4466- <https://groups.google.com/d/msgid/gcd-
>>> tech/118b0179-0a66-4466->

>>>>> ac89-7fa4f9e29ad1n%40googlegroups.com [1]?

>>> utm_medium=email&utm_source=footer <http://40googlegroups.com?
>>> utm_medium=email&utm_source=footer>
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>
>>> Groups "gcd-tech" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>> send
>>> an email to gcd-tech+u...@googlegroups.com <mailto:gcd-
>>> tech+uns...@googlegroups.com>.
>>> To view this discussion visit
>> https://groups.google.com/d/msgid/gcd-

>>> tech/2fd6c407-2a3e-4b84-9e9b-dbdfa956d615n%40googlegroups.com [1]
>> <https://
>>> groups.google.com/d/msgid/gcd-tech/2fd6c407-2a3e-4b84-9e9b- [3]
>>>
>> dbdfa956d615n%40googlegroups.com?utm_medium=email&utm_source=footer
>> [2]>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "gcd-tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gcd-tech+u...@googlegroups.com.
> To view this discussion visit

> https://groups.google.com/d/msgid/gcd-tech/92d4ef5d-c875-4e3d-9e22-c6790c2b2b19n%40googlegroups.com
> [4].

>
>
> Links:
> ------
> [1] http://40googlegroups.com

> [2] http://40googlegroups.com?utm_medium=email&utm_source=footer
> [3] http://groups.google.com/d/msgid/gcd-tech/2fd6c407-2a3e-4b84-9e9b-
> [4]
> https://groups.google.com/d/msgid/gcd-tech/92d4ef5d-c875-4e3d-9e22-c6790c2b2b19n%40googlegroups.com?utm_medium=email&utm_source=footer

[Jochen G.]

Am 28.07.26 um 00:15 schrieb Adam Hernandez:

> Tier 4 - Cover Scans
>
> GET /api/v2/covers/                    List active cover scans

This one would allow a form of bulk access to our cover scans. This we
exclude (so far), due to concerns about legality. An alternative might
be strict limits on the number of covers downloaded in a week or month.
Not sure if this would work though.

The access path to covers is via issues, there we have the cover URL
already, to be replaced by the API access to the actual cover and scan.

> GET /api/v2/covers/{id}/              Cover scan detail
> Resource semantics:Each API record maps one-to-one to an active Cover
> row and therefore to one owning Issue. A series does not have a separate
> cover entity; series covers are retrieved with the series filter.
> Variant covers are ordinary Cover rows owned by variant Issue rows.

Some issues have two covers, e.g., flip covers or books with dust
jacket. How would these be handled ?

> Fields:id, issue (id, series id + name, descriptor, variant_of,
> variant_name, variant_cover_status), images (thumbnail_url, medium_url,
> large_url), is_wraparound, display_restricted, front_crop (detail only),
> created, modified, last_upload.
>
>
> Filters:issue, series, publisher, variant_of, is_variant,
> variant_cover_status, is_wraparound, modified (range), created (range),
> last_upload (range).

As written above, these filter would only make sense if we have the full
list, which would be against our no bulk access decision.

> Variant behavior: issue=<id> returns scans directly owned by that exact
> issue. It does not silently substitute the base issue's scan for a No
> Difference variant. The Issue endpoint remains the source for the
> effective display cover and may inherit the base cover when
> variant_cover_status is No Difference.
>
>
> Image policy:List responses expose the identification thumbnail only.
> Detail responses expose all permitted generated sizes. When
> limit_display is true, thumbnail_url remains available while medium_url
> and large_url are null. Original uploads and replacement-history files
> are never exposed. The existing Issue cover_url behavior must be brought
> into alignment with this restriction.

We have currently zero covers with limit_display. We had one occurrence
years ago (which is why this was implemented), where the cover files
were changed manually.
For consistency we could implement this, but not really worth much effort.

Jochen

[Jochen G.]

The cover files would also need to be accessible via an API-endpoint.

Am 28.07.26 um 03:27 schrieb Kyle Jerviss:

[Adam Hernandez]

Just FYI...The final PR (coming today) makes us feature complete on the original scope for APIv2. Based on this I'm going to wait on the covers end point until after we release the APIv2 and it settles in a bit. 

[Richard]

In article <2fd6c407-2a3e-4b84...@googlegroups.com> you write:
>
>Jochen, should I update the PRD for APIv2 to include and endpoint for
>covers? That way its rate limited and required a key? I can do that easy
>enough.

Speaking for myself, my "want list" application wants at least *links*
to covers. It already follows the advice in the wiki:

- download covers only once
- use a local cache to avoid downloading covers again

--
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
The Terminals Wiki <http://terminals-wiki.org>
The Computer Graphics Museum <http://computergraphicsmuseum.org>
Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

[Richard]

In article <2d54e413-a987-4efd...@googlegroups.com> you write:
>
>Just FYI...The final PR (coming today) makes us feature complete on the
>original scope for APIv2. Based on this I'm going to wait on the covers end
>point until after we release the APIv2 and it settles in a bit.

To clarify my use case:

- I enumerate items in a (public) collection
- For each item in the collection, I download the cover for the item
if I don't already have it
- I then reconcile the local download cache to folders on my phone

This lets me maintain my "want list" as essentially folders of cover
images on my phone.

I'm OK with everything being gated by API keys and rate limits. This
is a personal setup for me, not something that I'm selling or
marketing. The code is on github, however, so potentially anyone
could use this same mechanism for maintaining their want list.

[Jochen G.]

How do you pass Cloudflare when downloading covers ?

Am 03.08.26 um 22:57 schrieb Richard:

[Brian Pepple]

On 7/15/26 4:10 PM, 'Jochen G.' via gcd-tech wrote:
> If anyone has a way to keep the site alive in the current bot-age
> while allowing cover access we can investigate that.
>
> It is not Cloudflare that is the problem, the bots are.
>
> We cannot pass logged-in users through Cloudflare, which would be the
> most obvious option.

Have you looked into using Anubis [1] for dealing with the bots? I was
looking for a solution that wasn't Cloudflare, and end up going with
Anubis based on the experiences of the some of folks (Fedora , Linux
Kernel) I know that have deployed it in production. Took a little bit of
work tweaking the policy for it, but it's effectively handled the bot
problem. If you are interested, I'll include a link to my DEPLOYMENT.md
which shows my architecture, since for simplicity sake you'd most likely
want to deploy it with Docker/Podman.


1) https://anubis.techaro.lol/

2) https://github.com/Metron-Project/metron/blob/master/DEPLOYMENT.md


Take care,

/B

--
Brian Pepple <bpe...@metron.cloud>

https://about.me/brian.pepple

[Richard]

In article <4f042d7f-a1da-445f...@garcke.de> you write:
>How do you pass Cloudflare when downloading covers ?

Run a browser instance and manually pass the captcha.