Downloading the whole library at once

[can.bo...@gmail.com]

Hello, I have time-limited internet connection and would like to download all the books and their respective covers at once. Is it possible to do so? Thanks.

[Alex Cabal]

Hi there, there is no official way to do that, but perhaps you could put
a script together that parses our OPDS feed:
https://standardebooks.org/opds/all

[can.bo...@gmail.com]

Unfortunately, I have zero experience regarding coding/scripting. Guess I'll have to wait, then :c

[Jared Updike]

Can you be more specific about what you need? Which format would work for you? If epub, then the covers are included.

Are you interested in a ZIP archive of everything? My estimate for the size of that archive (for over 240 books, in epub format) is over 700 MB.

  Jared.

[can.bo...@gmail.com]

Hello Jared,
I'll be using a Paperwhite to read the books; so I would be happy if there was an archive of .azw3's of every book in the website. I'm fine with a zip, it'd be easier to download...

Size isn't an issue to be honest. Its just that I only have limited access to wifi and would like to use that time to download the whole collection at once.
Thanks!

[Jared Updike]

OK I see about the Kindle format. Instead of having me creating a massive, one-time ZIP (and then having to upload it), it is easier for me to post a list of the URLs on the web somewhere, as a text file (that way I can keep it up to date with new releases of books).

This list contains links to all the .azw3 files + the cover .jpg files. Unfortunately there is no connection with the cover JPG filenames and the books themselves. :-(

Then if you have a device that can run Chrome (desktop? not sure about Android Chrome or iOS browsers with regard to extensions) you can install the Chrono Download Manager extension (not vouching for its security or creepiness, just an example downloader). With this extension (or any software that can accept a new-line separate list of URLs), you can drag-and-drop the list of links (select the text in one window and drag to Tasks view of Chrono Download Manager) and it will create approx. 500 download tasks, and then you just wait it out and you will have everything downloaded.

- Chrono Download Manager : https://chrome.google.com/webstore/detail/chrono-download-manager/mciiogijehkdemklbdcbfkefimifhecn

(just an example bulk downloader, not vouching for its security implications)

Text file with URLs here:

- https://updike-org.s3-us-west-2.amazonaws.com/Publish/Ebookshelf/all-azw3.txt

(FYI I have a script to keep this list up to date as new books are released.)

  Jared.

[Alex Cabal]

Please be polite when using a bulk downloader. Our server is small and
privately funded. Bandwidth is not unlimited. Do not spam the server
with hundreds of connections at once.

Also please see our OPDS feed which lists all ebooks along with download
links.

> --
> You received this message because you are subscribed to the Google
> Groups "Standard Ebooks" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to standardebook...@googlegroups.com
> <mailto:standardebook...@googlegroups.com>.
> To post to this group, send email to standar...@googlegroups.com
> <mailto:standar...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/standardebooks.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/standardebooks/5416d0fe-7b5b-4472-a3c2-05f5757cabf1%40googlegroups.com
> <https://groups.google.com/d/msgid/standardebooks/5416d0fe-7b5b-4472-a3c2-05f5757cabf1%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Jared Updike]

The OPDS feed has links to the thumbnails but the files are all named cover.jpg which compared to the nicely named author_booktitle.epub is less useful. (However the webpage for each book has a link to a thumbnail_*.jpg cover file, but this is not listed in the OPDS.)

In the spirit of saving SE hosting costs, in my private use I have been as careful as I can to avoid redownloading books and data, or preferring to make Microsoft (Github) pay for it by using github when possible.

I can take the bulk download list down if that is helpful to prevent any excess general bulk downloading, I just thought: "a reader asked for help, but was told 'you need to be a programmer,' here is XML (twice now), which didn't seem like a super helpful or friendly answer". However I don't want to drain your resources, Alex.

  Jared.

[Jacob Press]

What about a torrent updated every 6-12 months? That would solve the bandwidth issue. 700 MB is nothing for a home connection. People have been seeding 10-100 GB torrents from home for over a decade.

[Alex Cabal]

Well updating it just every 6-12 months is probably not useful as books
are not just released fairly frequently, but also updated very
frequently. And someone will still have to maintain the torrent. As I
said earlier somewhere, scraping isn't a huge deal as long as the
scraper is polite about it. I was looking to implement some kind of
bandwidth throttling on the server just to deal with rude scrapers, not
*any* scrapers.

> --
> You received this message because you are subscribed to the Google
> Groups "Standard Ebooks" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to standardebook...@googlegroups.com
> <mailto:standardebook...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/standardebooks/0d4632aa-cf48-433a-a29e-e5f4960e3d52n%40googlegroups.com
> <https://groups.google.com/d/msgid/standardebooks/0d4632aa-cf48-433a-a29e-e5f4960e3d52n%40googlegroups.com?utm_medium=email&utm_source=footer>.