Easiest Way to Import Stories from Fanfiction.net & General Advice

[Anagh Kumar Baranwal]

Hi,

Sorry if this is slightly off-topic but I haven't found a better forum for these questions.

---

With the recent ban of unofficial apps, my regular reader app Fanfiction Reader has stopped working so I am planning to migrate all the series to Calibre with the FFF plugin synced with BookFusion Cloud and read via the BookFusion app on the Android platform.

I have a script to retrieve the story IDs from the app database but I couldn't find anything to populate the Calibre database with those stories. Is my best option to create a dummy FFN.net account and favourite those stories and then provide that as a list to FFF?

---

Some general questions:

1. Any preferred/recommended format for Fanfiction display and download? Specifically regarding the styling etc.

2. Any advice regarding managing Fanfiction collections in Calibre or Calibre collections in general? I am planning to go through the following in the next few days:

* FFF Wiki
* Calibre Wiki
* Calibre Custom Columns
* Calibre Metadata Plugboards
Let me know if there is something else that is recommended reading.

3. What is the best/recommended way to bulk-update. There seem to be 2 options for this from what I can find:
* Add SMTP Email info directly into the FFF plugin in Calibre. This carries the risk of exposing the settings to all the installed plugins
* Use AutomatedFanfic (https://github.com/MrTyton/AutomatedFanfic) and add the SMTP Email info in its config so it can update the Calibre DB

4. Any other recommended reading or suggestions from the experienced folks?

Looking forward to any replies and help.

Thanks.

[Anagh Kumar Baranwal]

Let me know if there is a more appropriate forum for this post instead too.

[Pablo M]

If you already have the IDs all you need to do to add them to fanficfare is add the rest of the URL at the front and paste them... But anyway, if you already have an account I personally recommend to keep using that to manage your favorites, and simply set a forwarding email filter to send anything from their bot to a dummy email which you can expose to all calibre plugins without any worry.

I'm not the best person to help with calibre as I primarily use the CLI version . But the mobile read forum is where the discussion regarding calibre and the plugins usually happens. https://www.mobileread.com/forums/showthread.php?t=259221

[Jasmin]

Hi! I'm sorry I couldn't be of much help, but I was wondering if I could have a copy of the script you have to extract the story IDs from Fanfiction Reader? I'm having the same issue as you have with the app not updating anymore. Thank you!

[Anagh Kumar Baranwal]

> If you already have the IDs all you need to do to add them to fanficfare is add the rest of the URL at the front and paste them

Is there no simpler way than this? I would like to avoid copy-pasting hundreds of URLs manually.

> But the mobile read forum is where the discussion regarding calibre and the plugins usually happens. https://www.mobileread.com/forums/showthread.php?t=259221

Thanks, will take a look there.

> I could have a copy of the script you have to extract the story IDs from Fanfiction Reader?

Sure, I will clean it up and post it as a Gist on Github tomorrow. However, you will require root on your device since this script uses the database of the app directly.

[Pablo M]

You don't have to do that, you can add them all at once, just one per line. I'm guessing you have all the id's inside a file, as long as you have one per line all you need is to add "https://fanfiction.net/s/" before each ID, and then paste the entire thing to fanficfare.

Personally I would use Notepad++ and record a simple macro (press home, paste, go down) tell it to repeat it until the end of the file and watch everything getting done in a single second. If you don't have them one per line then it gets a bit more complicated, but honestly if you managed to make a script to get all the id's then it shouldn't be a problem.

[Jasmin]

Thank you! Can you please update with the link after you posted it? With the copy pasting of URLs, I think you could use the & function in Excel to merge the story id with the URL. Like copy paste your story ids im column A then in column B or B1 cell you can use this function;

="https://m.fanfiction.net/s/"&A1

and drag it down until the last row 

[Anagh Kumar Baranwal]

Generating it isn't the problem. It's adding them one by one or in batches manually to Calibre that's the issue. I like to automate this stuff as much as possible.

[Edward Clayton Andrews]

If you have an email fetch setup as I do, you can do what I do. Send story links as emails as you find them to that address. Next time you fetch updates, your 'added' stories will download too.

I find this method also stops me from getting site banned or throttled; too much automation here can lead you to trying to download a batch and realizing, wait, I just tried to download 50+ fics, no wonder ffn is trying to throttle me.

If you're DL an author's entire ficography (real word), there's extensions that will copy all URLs on a page or in a highlighted text for both firefox and chrome/ chromium. You can feed that output to calibre direct or email it to your calibre feed account.

For every step of automation,  just remember that's another potential break point and another spot you lose direct control. There are trade offs.

--
Project Home: https://github.com/JimmXinu/FanFicFare/wiki
---
You received this message because you are subscribed to the Google Groups "FanFicFare -- Downloader for FanFiction" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fanfic-downloa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/c5ede116-cf47-48ad-8f13-d68002b42a57n%40googlegroups.com.

[bugstomper]

I'm not sure how much manual work is too much for you, but as a one time thing this isn't very much. Since you do have a script to get all of story IDs, I assume you have no problem making that a list of story URLS, one per line. At least on my Mac (don't have a Windows version to check), if I run Calibre, then copy a list of URLs from some other editor or spreadsheet program, one URL per line, when I click on the FanFicFare Download from URLs menu item in Calibre, the URLs from my clipboard are already pasted in from the clipboard. So the operation is Copy, not even Copy and then Paste.

When I want to update stories that I already have put in Calibre, I use the Virtual Library column and select Publisher, and then W and then select www.fanfiction.net to see just the stories in my Calibre database that have been imported from that site. When you do that you will see that Calibre displays the search string publisher:"=www.fanfiction.net". If you type at the end of that and tag:"=In-Progress" you will get just the fanfiction net stories that are tagged as still in progress. You can then multi-select just those stories that you know are in need of updating and the FanFicFare menu item Update Existing FanFiction Books will be populated with those books and can be run to update all of them.

Since the only step that requires dealing with many books at a time is the initial import, it is not a big deal to update a book when you get an email alert of a new chapter being available.

[Anagh Kumar Baranwal]

Thanks for both of your suggestions.

> FanFicFare Download from URLs menu item in Calibre

I just noticed this menu item and yes, this solves my issue very nicely.

I was planning to anyway set up a dummy email account with SMTP/IMAP access for updates so that shouldn't be a problem.

---

My next step is to figure out the ideal settings Calibre then and that may be better posted/asked on the MobileRead forums instead as suggested earlier in this thread unless you would be willing to share those too per my questions on the first post?

Thanks again.

[Anagh Kumar Baranwal]

> Can you please update with the link after you posted it?

https://gist.github.com/darthShadow/fc868367b6cb949016ee7af7a3abf689

It needs to have the library.db from FanfictionReader in the same folder as the script and a reasonably newer python version (3.6+).

The story links will be written to fanfic_links.txt in the same folder.

[Jasmin]

Thank you very much! 

[jamie coady]

Hi I take it ff.net is still not playing ball and we can’t download them?

Off topic if anyone can message me off list with a few more sites to get hp fanfics from please.

 

 

Sent from Mail for Windows 10

--

Project Home: https://github.com/JimmXinu/FanFicFare/wiki
---
You received this message because you are subscribed to the Google Groups "FanFicFare -- Downloader for FanFiction" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fanfic-downloa...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/ca3bfce6-c945-435c-a905-939367e60879n%40googlegroups.com.

 

[Pablo M]

It's been working fine for a couple of days now, make sure you're using the latest version of the plugin.

[jamie coady]

I downloaded it a few days ago is there a later one? Will give it a try.

 

 

Sent from Mail for Windows 10

 

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/d5e2d016-7683-4440-876d-3e08bfdf504an%40googlegroups.com.

 

[jamie coady]

I put the url in like the following one.

https://www.fanfiction.net/s/11869129/1/The-Announcement

click ok and nothing happens. I must not have the latest update or I am doing something rong again.

 

 

Sent from Mail for Windows 10

 

From: Pablo M

Sent: 27 December 2020 09:48

To unsubscribe from this group and stop receiving emails from it, send an email to fanfic-downloa..@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/ca3bfce6-c945-435c-a905-939367e60879n%40googlegroups.com.

 

--
Project Home: https://github.com/JimmXinu/FanFicFare/wiki
---
You received this message because you are subscribed to the Google Groups "FanFicFare -- Downloader for FanFiction" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fanfic-downloa...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/d5e2d016-7683-4440-876d-3e08bfdf504an%40googlegroups.com.

 

[jamie coady]

Ok updated it again and it seams to be working. Thanks

 

 

Sent from Mail for Windows 10

 

From: Pablo M

Sent: 27 December 2020 09:48

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/d5e2d016-7683-4440-876d-3e08bfdf504an%40googlegroups.com.

 

[jamie coady]

Is there a limit on how many fics you can download at once? For example can you  do 6 links or just 5?

 

 

Sent from Mail for Windows 10

 

From: Pablo M

Sent: 27 December 2020 09:48

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/d5e2d016-7683-4440-876d-3e08bfdf504an%40googlegroups.com.

 

[Jim Miller]

You can do as many links as want, but ffnet is known to start blocking
you if you do too many downloads too fast. We don't know what 'too
many' and 'too fast' are defined as. I recommend only a dozen or so at
a time.