AniDB banning Griffith users

[asterisk20xx]

AniDB is currently banning Griffith users who try to import data from their website. I spoke with some of the AniDB team in their IRC channel about it.The reason is that the operators do not approve of scraping the website, because there are dedicated APIs for retrieving data from AniDB in more efficient manners. Those APIs are detailed at:

http://wiki.anidb.net/w/API

AniDB would greatly appreciate proper access of their data by one of the APIs, and I'm sure Griffith users would also like to be able to use AniDB as a resource again. Please consider updating this plugin to use their APIs as soon as is reasonable.

[Michael Jahn]

Since September 2011 the plugin uses the HTTP API:
http://wiki.anidb.net/w/HTTP_API_Definition
That version of the plugin is released with version 0.13 of griffith.

> --
> You received this message because you are subscribed to the Google
> Groups "Griffith" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to griffith-use...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[asteri...@gmail.com]

This does not seem to be the case. Or perhaps this is only limited to Griffith under Windows?
The steps I took were as follows:

I Installed Griffith 0.13 for Windows on February 16, 2013, from www.griffith.cc
I searched for a title on AniDB with griffith. No results were found.
I opened www.anidb.info in my web browser with the intent of manually copy and pasting the information and received a notice saying I was banned and to join the IRC channel to request being unbanned.

I have just re-installed Griffith again from the website and the following image shows the banning message as well as the debug log.

http://i.imgur.com/kJ3WCgH.png

If the plugin was updated it appears that it was never included in the Windows installer.

Full debug is below:
---
2013-02-18T14:48:51: DEBUG: Griffith(gconsole:82): Starting Griffith 0.13
2013-02-18T14:48:51: DEBUG: Griffith(gconsole:83): Platform: Windows-7-6.1.7601-SP1 (nt)
2013-02-18T14:48:51: DEBUG: Griffith(gconsole:84): Dependencies:
gtk :: 2.22.0
gtk.glade :: 2.22.0
sqlalchemy :: 0.7.1
sqlite3 :: 2.6.0 (sqlite-lib 3.6.21)
reportlab :: 2.5
PIL :: True
psycopg2 :: 2.3.1 (dt dec pq3 ext)
MySQLdb :: 1.2.3.final.0
chardet :: 2.0.1
sqlite :: False
lxml :: True
2013-02-18T14:48:52: DEBUG: Griffith(griffith:1469): config file used: C:\Users\Noam\AppData\Roaming\griffith\griffith.cfg
2013-02-18T14:48:52: INFO: Griffith(initialize:101): Using Griffith directory: C:\Users\Noam\AppData\Roaming\griffith
C:\Program Files (x86)\Griffith\lib\initialize.py:140: GtkWarning: GtkSpinButton: setting an adjustment with non-zero page size is deprecated
2013-02-18T14:48:52: DEBUG: Griffith(sql:279): SELECT movies.number, movies.o_title, movies.title, movies.director, movies.poster_md5, movies.genre, movies.seen, movies.year, movies.runtime, movies.rating, movies.created, movies.updated
FROM movies ORDER BY movies.number ASC
2013-02-18T14:48:53: INFO: Griffith(initialize:873): Spellchecker is not available
2013-02-18T14:49:08: DEBUG: Griffith(add:284): reloading PluginMovieAniDB
2013-02-18T14:49:09: INFO: Griffith(PluginMovieAniDB:195): downloading title list from http://anidb.net/api/animetitles.xml.gz
---

[asteri...@gmail.com]

I spoke with Worf again just now in their IRC channel ( #an...@irc.synirc.org ). This is what he said:

+++ Griffith_User set to mode +ix

!lemming !help !rules | http://anidb.net -=- AniDB -=- 7548 anime, 133984 eps, 1017870 files (267894 GB), 52173 characters, 35453 creators, 57761 songs, 5611 collections, 10817 groups, 574683 users, 7003 reviews | AVDUMP2: http://static.anidb.net/client/avdump2/avdump2_6525.rar

Topic set by DerIdiot on Fri Feb 15 2013 14:50:47 GMT-0600 (Central Standard Time)

<Griffith_User>: Is worf in atm?

<worf>: you got yourself banned again?

<Griffith_User>: as an example yes

<Griffith_User>: https://groups.google.com/forum/?fromgroups=#!topic/griffith-user/_IA43S2Wyg0

<Griffith_User>: I brought up a topic on the API matter and the reply was "we fixed that in 2011".

<Griffith_User>: Obviously not. Or at least not to AniDB's satisfaction anyways.

<worf>: i told you last time that something tried accessing the main page using python

<worf>: telling them that would probably help them a lot with finding the issue

<Griffith_User>: And they say they've been using the HTTP API since Sept. 2011. I installed griffith on a new machine 2 days ago

<worf>: my best guess without having looked at any of the code is that they are "pinging" anidb by checking if the main page is there

<worf>: which is just plain stupid and retarded if thats really the case

<Griffith_User>: 2013-02-18T14:49:09: INFO: Griffith(PluginMovieAniDB:195): downloading title list from http://anidb.net/api/animetitles.xml.gz

<Griffith_User>: that seems to be the first (and only) thing it does.

<worf>: its lying

<Griffith_User>: but anyways, mainly I just wanted to let you know where the thread was if you wanted to let them know anything

<worf>: as simple as that

<worf>: i'm not going to tell them anything. if they want something from us they can come here

<worf>: or you can deliver them what i/we said

Does the information Worf provided help any?

[Michael Jahn]

Worf is right. I had overlooked a small piece of code.
I changed it with revision 1642.
You can fetch the fixed version of the file add.py from the source code repository and
replace the original version of that file of your installation.

--

[Noam Kaze]

Just tested it and it's working properly now!
Thank you so much for the prompt fix!

[Harro von Lavergne]

This has to do only indirectly with AniDB, instead it is about imdb (I could
not find out how to post directly).
Probably because imdb changed the layout of their pages again, since 1st
February the import of data from their side works only approximately, the year
especially is not imported at all.
Can this be fixed?
And then there is another question:
I installed the last ad-hoc-windows-version. It works but there is at least one
visual problem in it; when you open a film's description then on the film
information tab the background of the Rating slider is black. I know of course
that technically this isn't important but ... nevertheless ... a fix or an
update shall be appreciated.
Thanks in advance
hvl

[Michael Jahn]

Please try the current nightly build:
https://dl.dropbox.com/u/2838803/Griffith/griffith-0.13.1-rev1642-svn-win32.exe

If you have ongoing problems with imdb please give a sample movie which
doesn't work.

[Harro von Lavergne]

Michael Jahn <mikej06@...> writes:

>
> Please try the current nightly build:
> https://dl.dropbox.com/u/2838803/Griffith/griffith-0.13.1-rev1642-svn-
win32.exe
>
> If you have ongoing problems with imdb please give a sample movie which
> doesn't work.
>
> Am 19.02.2013 10:11, schrieb Harro von Lavergne:

...

> > Probably because imdb changed the layout of their pages again, since 1st
> > February the import of data from their side works only approximately, the
year
> > especially is not imported at all.
> > Can this be fixed?
> > And then there is another question:
> > I installed the last ad-hoc-windows-version. It works but there is at least
one
> > visual problem in it; when you open a film's description then on the film
> > information tab the background of the Rating slider is black. I know of
course
> > that technically this isn't important but ... nevertheless ... a fix or an
> > update shall be appreciated.
> > Thanks in advance
> > hvl
> >

Hello Mike,
Thanks a lot for the prompt reaction. I installed the "nightly build" and it
worked fine, at least almost. The black background of the "Rating" slider has
disappeared, that's fine.
But now the buts:
I downloaded from imdb.com the data for the film "The Importance Of Being
Earnest (2002)".
On the "Details" Tab
1st: The "production year" unfortunately still is the dummy "1899", obviously
the actual year 2002 is not imported.
2nd: The field "Country" is somewhat convoluted, it looks like this:
UK | USA
There are a lot of superfluous blanks.
And the entry in the field "Genre" results in this:
Comedy |
Drama |
Romance
that means there are invisible LF characters immediately behind the separator
bars |
I really hope the import script can be fixed, can't it?
Thanks a lot for your work, it is still highly appreciated.
hvl

[Michael Jahn]

a new try :)
https://dl.dropbox.com/u/2838803/Griffith/griffith-0.13.1-rev1644-svn-win32.exe

[Harro von Lavergne]

Michael Jahn <mikej06@...> writes:

>
> a new try :)
> https://dl.dropbox.com/u/2838803/Griffith/griffith-0.13.1-rev1644-svn-
win32.exe
>

Hi Mike,

That's it. I downloaded the 1644 version, it works as "suspected".
Actually I persanally applied the fixes I found in the "corrupted format" file
'PluginMovieIMDB.py'.
They worked as they were supposed to but then I could not find out how to fix
the rest of the problems - well, you did and everything is fine... Until the
next change by IMDB.
OK, keep on going strong.
hvl