New adapter ideas... adult-fanfiction.org

[Gwydion Comyn]

I'm thinking about writing a set of adapters for the adult-fanfiction.org sites...

it would be one main one, then a set of smaller ones for each site, since they all have different site urls...

the old FanFictionDownloader used to be able to download from there, but, since the site has been changed, it no longer works, and since it is not being updated anymore, there is only manual downloading available at this time...

just thought I'd practice my coding with that... any interest out there?

GComyn

[Tara Li]

This is by far one of the most requested.  The issue has long been the TOS stuff.  However, my thinking is that if your required the user to have an account, you would be covered.

I used to use FLAGG, but it's been "recoming soon" for quite some time now.

I say go for it, and I suspect all you would really need is one adapter, as everything feeds from one database.  Just some minor conditionals here and there.

--
Project Home: https://github.com/JimmXinu/FanFicFare/wiki
---
You received this message because you are subscribed to the Google Groups "FanFicFare -- Downloader for FanFiction" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fanfic-downloa...@googlegroups.com.
Visit this group at https://groups.google.com/group/fanfic-downloader.
To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/b096a55f-cfad-4508-8d95-e111a1d4d2ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jim Miller]

The other project named FanFictionDownloader may have worked with
adult-fanfiction.org, but this one never has.

In past, it was always because:

> adultfanfiction.net / adult-fanfiction.org
>
> None of the current developers are interested in writing code to support it because of the ominous legal warnings and requirements to access the site
(
https://github.com/JimmXinu/FanFicFare/wiki/SupportedSites#adultfanfictionnet--adult-fanfictionorg
)

If the 'ominous legal warnings and requirements' have been toned down
and you'd like to work on it, feel free to do so.

You should probably figure out the github process, though, for that
level of involvement.

[Gwydion Comyn]

I was just looking at the TOS for adult-fanfiction.org, and saw this paragraph:

You agree as a user of AFF that you will not download or copy any of the content within without the express permission of the original user to post the content.

so... I probably should think about this more before I decide to continue...

GComyn

[Jason Straw]

So, the tos is broken by viewing the text of a story on any device.  You had to download the content to view it.  Not a lawyer, but i bet they'll rip that tos to shreds.

[Jim Miller]

That isn't the 'ominous legal warnings and requirements' I referred to.

As I recall, they used to require your real name, birth date and
included a bunch of text about your 'legal digital signature' with
references and quotes from different US laws.

[Tara Li]

Ok, here's the thing.  We're not trying to play Rules Lawyer here - we're trying to do the right thing  *while* making things more convenient to everyone.

Yes, the TOS is technically broken by simply viewing the page - the Supremes have not really weighed in on the full copyright implications of the manner in which the Internet works, and so technically copyright law is broken a few dozen times per page load - except that placing something on a website is probably implicit permission for copying for the purposes of viewing.

And many websites have similar language to the language on AFF - and many authors include similar language in their disclaimers.  The purpose is, of course, to prevent someone taking something, and reposting it as their own work - or just so that it doesn't get posted somewhere where the author can't make corrections when it's needed.

However, downloading and storing a copy locally is probably similar enough to the Supremes ruling that taping radio airplay for private enjoyment, and recording video on VHS for timeshifting, that we can safely consider the downloading for local enjoyment as a non-issue.

On Tue, Aug 2, 2016 at 7:56 AM, Jason Straw <jason...@gmail.com> wrote:

So, the tos is broken by viewing the text of a story on any device.  You had to download the content to view it.  Not a lawyer, but i bet they'll rip that tos to shreds.

--

Project Home: https://github.com/JimmXinu/FanFicFare/wiki
---
You received this message because you are subscribed to the Google Groups "FanFicFare -- Downloader for FanFiction" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fanfic-downloa...@googlegroups.com.
Visit this group at https://groups.google.com/group/fanfic-downloader.

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/CAHFOpneTFog%3DbTL6A%2BfeX8ODemxq5u4FWSi5PvENo5F-GjLwkg%40mail.gmail.com.

[Tara Li]

On Tue, Aug 2, 2016 at 9:16 AM, Jim Miller <retie...@gmail.com> wrote:

That isn't the 'ominous legal warnings and requirements' I referred to.

 

As I recall, they used to require your real name, birth date and included a bunch of text about your 'legal digital signature' with references and quotes from different US laws.

Ah, yes.  I haven't seen that in a while.  However, requiring that the user login should side-step such things.

[Gwydion Comyn]

Ok... It's been a while since I set up my account, so I don't remember what all I had to do... so I'll to with Tara Li on this, and begin working on it...

GComyn

[Tara Li]

YAY!  Speaking as a pre-OO programmer, thank you!  I've wanted this for some time myself, but not having the skills to do it, haven't been pushing the point.  However, I'll be quite willing to test this out across a broad range of the sub-sections (Books, Buffy, Cartoons, Movies, TV, Harry Potter [there's still a little bit in there...], Anime, Naruto...)

On Tue, Aug 2, 2016 at 12:56 PM, Gwydion Comyn <gwydio...@centurylink.net> wrote:

Ok... It's been a while since I set up my account, so I don't remember what all I had to do... so I'll to with Tara Li on this, and begin working on it...

GComyn

--

Project Home: https://github.com/JimmXinu/FanFicFare/wiki
---
You received this message because you are subscribed to the Google Groups "FanFicFare -- Downloader for FanFiction" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fanfic-downloa...@googlegroups.com.
Visit this group at https://groups.google.com/group/fanfic-downloader.

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/56fdc76e-6c66-41c1-b2b5-a4a0a1021844%40googlegroups.com.

[Gwydion Comyn]

Ok... I was going through some of the other adapters, and I found several ways to add the metadata...

One was :  self.story.setMetadata

the other was : self.story.addToList

which is better to use?

GComyn

[Gwydion Comyn]

Ok... I've figured out the setMetadata vs addTolist issue...

The way I've worked it is that there is a base_adapter_adultfanfictionor.py, with a different adapter for each "site"... since each adapter can only have one 'getSiteDomain' and 'getSiteAbbrev'...

The only issue I have right now is that I can't get it to login... *sigh*... you don't really NEED to login to get to the pages... so it works... just doesn't login correctly.

so... if I continue with this, I'll have 23 'adapters' to add... one for each of the sites, with the base that does the work...

GComym.

[Gwydion Comyn]

also... this means that there would need to be 22 additional sections in the Default.ini... *sigh*...

GComyn

[Jim Miller]

On 8/4/2016 10:51 PM, Gwydion Comyn wrote:
> One was : self.story.setMetadata
>
> the other was : self.story.addToList

setMetadata is for single values, like rating or numWords.

addToList is for list values, like genres or characters.

> The way I've worked it is that there is a base_adapter_adultfanfictionor.py, with a different adapter for each "site"... since each adapter can only have one 'getSiteDomain' and 'getSiteAbbrev'...
>
> The only issue I have right now is that I can't get it to login... *sigh*... you don't really NEED to login to get to the pages... so it works... just doesn't login correctly.
>
> so... if I continue with this, I'll have 23 'adapters' to add... one for each of the sites, with the base that does the work...

What is the scheme of the domain names? It is possible--and I suspect
desirable--to put them all in one adapter unless they truly are
different sites.

Look at getAcceptDomains in adapter_fimfictionnet.py or
adapter_twcslibrarynet.py for examples of using several domains in one
adapter. adapter_literotica.py is a more extreme example; but it's also
structured a bit differently.

Jim

[Tara Li]

The domains are just buffy.adult-fanfiction.org, books.adult-fanfiction.org, hp.adult-fanfiction.org, anime.adult-fanfiction.org, etc.  Probably fairly easy to fold them into one adapter, I expect - though I think it would be a scary bunch of CASE statements, perhaps.

--
Project Home: https://github.com/JimmXinu/FanFicFare/wiki
--- You received this message because you are subscribed to the Google Groups "FanFicFare -- Downloader for FanFiction" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fanfic-downloader+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/fanfic-downloader.

To view this discussion on the web visit https://groups.google.com/d/msgid/fanfic-downloader/bf71f34e-352f-903d-a095-32b87a332342%40gmail.com.

[Gwydion Comyn]

The issue I'm having with the multiple sitenames, is that they are not redirects to the same site... so one specific SiteDomain is not working... and the getSiteDomain proc does not send any variablies, so there is no way to dynamically set it at runtime... it is always whatever is set there... and it has to be one, since it is used in the rest of the code to determine the url for the story...

so hp.adult-fanfiction.org is different that buffy.adult-fanfiction.org, and that is differnt to the other 21 sites....

I'm trying to get it to wirh with the -s option, but it's not working...*sigh*..

Using 1 adapter per site, with a base adapter for the actual work, does work... though...

GComyn

[Jim Miller]

On 8/5/2016 5:36 PM, Gwydion Comyn wrote:
> The issue I'm having with the multiple sitenames, is that they are not
> redirects to the same site... so one specific SiteDomain is not
> working... and the getSiteDomain proc does not send any variablies, so
> there is no way to dynamically set it at runtime... it is always
> whatever is set there... and it has to be one, since it is used in the
> rest of the code to determine the url for the story...

I may have picked poor examples. The files
adapter_ponyfictionarchivenet.py and adapter_literotica.py both allow
canonical story URLs with different domain names under the same site domain.

Jim

[Gwydion Comyn]

Ok... I'm looking at that right now, and the adapter_ponyfictionarchivenet.py seems to have what I need... I'm working on it right now..

GComyn

[Gwydion Comyn]

Ok.... I've got it done... except that it still doesn't login... *sigh*...

the login is not on it's own page, and the action in the form is ""... which means that it uses the current page (I'm using the index.php page).

so I've got to figure out how to send/receive cookies *sigh*... and how, if it does, fff already does it...

every page I've seen on the web from my searching has them using the mechanize toolkit, but, since it's not already in the requirements, I'm trying to stay away from it...

So... it's done, and in 1 file as well... except for the login, so I'm not going to send it up just yet...

GComyn

[Gwydion Comyn]

Well...

Like I said above, I've got it done... it saves stories from all 22 different adult-fanfiction sites...

BUT, it still doesn't login... *sigh*...

Since the site doesn't REQUIRE a login to see the page, I'm not sure how to fix this... it seems it works off of cookies, and I can't see what is actually going on.. I've been using Firebug to see what is actually happening when you login/logout, but can't see it.. *sigh*...

Should I post this to the github site to see if other programmers, who don't read this list, have an idea?
and if so, where?  it is not an Issue, so it shouldn't go there....

GComyn

[Jim Miller]

If login isn't required to do the download, and the onerous legal text
is gone, there's no need to do it.

See adapter_literotica.py:131 for an example of requiring is_adult
without login.

[Gwydion Comyn]

Ok.. it's done, still not loging in, but it does require is_adult to be set true...

Now I have to figure out how to push it up to github... *sigh*... along with the default.ini and the caliber ini file... (need to fix that *sigh*)...

GComyn

[Gwydion Comyn]

Ok.... I've created a pull request... we'll see if it works or not...

GComyn

[Gwydion Comyn]

Well..

In the course of downloading stories, I've happened upon some exceptions to the programming that I did with the adapter... so I'm going to have to do another pull request to get it up to you...

I'll work on that in the next few days.

GComyn

[Jim Miller]

Okay. I'll watch for it.

Jim

[Gwydion Comyn]

Ok...

which do you think is better... hardcoding a few switches, or defaulting to another variable...

On the aff site, there are some stories that are dated 000-00-00... some in the updated date, others in the published date...

and some others have a published date of 0084, 0082, 0083...

right now, I've got it defaulting to the member date that is listed at the top of the page... but should I hardcode to change the 008 to 200?

GComyn

[Jim Miller]

That's nasty.

FanFicFare expects there to be dateUpdated. So much so that in
base_adapter.py(439,452), it sets dateUpdated to be datePublished or
dateCreated if necessary. (dateCreated is set automatically while
datePublished is less key.)

I haven't looked at aff, but on many sites we read the metadata from the
author page, if you haven't already looked at that.

Are the stories with bad dates all old? Or is it a re-occurring problem
on aff? It would be nice to be able to assume active stories with
recent updates have correct updated dates at least.

All 0's dates I would think should be discarded. 0084 etc dates could
be massaged to real dates, but how certain are you that they are all off
by a consistent amount? I don't track how 008 would become 200.

Jim

[Gwydion Comyn]

everyone that I've seen so far is from 200?...2002, 2003, 2004... so it looks like there was a problem early on... it hasn't come up in any other stories that I have processed.

And it only appears to be less than 10 stories total... so I'm going to hardcode it to change the 008 to 200...

and I already have the Updated date = Published date if the update date starts with 0000... so that is ok.

GComyn