fanfic downloader qeustions
Ben Smith
bash, to turn them into fanfiction.net urls, and
thanks SO MUCH for the bash for the automation of your downloader.
I'm trying to archive ffnet, along with several dozen other people and
this is a great start! :)
your team, and you, rock
-Ben Smith
Roman Kirillov
Another option is to modify downloader.pay main method - you make up story URL and download them in a loop.
R
Ben Smith
so, any suggestions on what i would change?
On Dec 15, 12:05 am, Roman Kirillov <m...@romankirillov.info> wrote:
> Another option is to modify downloader.pay main method - you make up story
> URL and download them in a loop.
>
> R
Roman Kirillov
Not at my computer right now... will try to send you an example tomorrow (I'm in west coast u.s timezone now). If you remind me, that is :)
Ben Smith
- authors name - story title" but inside folders for each genre, like
Books, Moviesm, Anime, etc and the category option in the config file,
is giving me trouble.
mainly, it keeps saying i have the syntax wrong but im following the
examples in the comments to the letter.
On Dec 15, 12:11 am, Roman Kirillov <m...@romankirillov.info> wrote:
> Not at my computer right now... will try to send you an example tomorrow
> (I'm in west coast u.s timezone now). If you remind me, that is :)
Jim Miller
As the saying goes, there's more than one way to do it. One possibility is:
> #!/bin/bash
>
> cat $@ | awk '{ printf("http://www.fanfiction.net/s/%s/1/\n", $0); }' | xargs -n1 python downloader.py -f html
You're trying to *archive* ffnet? What, all of it? We've seen them
block people in past for downloading too much too fast...
> im also trying to get the file name format to be for example "Category
> - authors name - story title" but inside folders for each genre, like
> Books, Moviesm, Anime, etc and the category option in the config file,
> is giving me trouble.
> mainly, it keeps saying i have the syntax wrong but im following the
> examples in the comments to the letter.
One problem with that is that on some sites, a story can have multiple
categories, so the program keeps it internally as a list. I'm sure I
never tested trying to use category(or genre) in the file name, so I've
no idea what it will do. Nothing useful, I expect.
Have you considered using calibre for storage/organization? I highly
recommend it. If you use epubs, it will automatically pick up the
categories/genres from the metadata and tag accordingly.
Jim
On 12/14/2011 11:05 PM, Roman Kirillov wrote:
> Another option is to modify downloader.pay main method - you make up
> story URL and download them in a loop.
>
> R
>
> On Dec 14, 2011 9:03 PM, "Ben Smith" <brain...@gmail.com
> <mailto:brain...@gmail.com>> wrote:
>
> hey stupid question, but i have a massive file of story ids, whats the
> bash, to turn them into fanfiction.net <http://fanfiction.net> urls, and
> thanks SO MUCH for the bash for the automation of your downloader.
> I'm trying to archive ffnet, along with several dozen other people and
> this is a great start! :)
> your team, and you, rock
> -Ben Smith
>
--
Jim Miller
Retie...@gmail.com
William Suetholz
They actually frown on software like ours.. One reason, is that they
allow authors to pull all their stories. And, they seem to frequently
blacklist authors and pull all their stories. For the first case, they
are giving the authors rights that you/we are taking away by archiving
the stories in any fashion at all. It actually wouldn't take much for
them to block this software, so please don't annoy them.
Bill
William Suetholz
One of the worst of a multitude of bad things M$ has forced on us has
been the promoting spaces in file names. Unix allowed spaces, but kind
of discouraged it by having you have to put quotations around the file
names if you wanted to use them. M$ made some of it's core directory
structures contain spaces, what a crock! (Yes I'm a command line old
fogie)
That being said, we (Jim/Roman and I) tried to pick an output file
name that would avoid namespace collision (Site/Author(ID)/Story(ID))
customized for each different fanfiction site supported. If you wish
the extra information, add a tag that contains what you want, then use a
program like Calibre to manage the stories for you. We already have
added tags to the metadata for the Category and Genre when the site
provides that information. We scrape and process as much information as
we can about each story.
I actually have some shell scripts to grab story urls from FF.net's
pages and then download them with fanfic-downloader. Next time around I
was planning to build it into the downloader. Won't get to that for at
least a year. Anyways a version of those shell scripts are attached.
They should be a good starting point.
Sorry for the rant.. I just hate spaces in file names.
Bill
Ben Smith
by someone at some point.
Thinking otherwise is just naive. That said, I agree,
and I'll try not to hit them harder than normal.
there are only 3.6 million stories on ffnet, they should be saved,
hundred of thousands of hours of creativity went into them