Safebrowsing-Python

[Thejaswi Puthraya]

This is to inform you of the launch of safebrowsing-python[1]. It
provides python source files that are used to fetch the url hashes and
also perform query lookups. Also included are some django[2] stuff
that integrate safebrowsing-python and django.

Feel free to check out the wiki page on safebrowsing-python and mail
your comments on this mailing-list. I am also open to people joining
and contributing to the project.

Together (with the support of Google) we can make the web a safer
place!!!

Thanks to the Google Folks for such a wonderful service.

Cheers
Thejaswi Puthraya
http://thejaswi.info/

[Thejaswi Puthraya]

Sorry, to have forgotten the links.
Safebrowsing-python on http://code.google.com/p/safebrowsing-python
Django: http://www.djangoproject.com

On Jan 17, 8:51 pm, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

[markybob]

great, thanks :)

On Jan 17, 9:51 am, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

[markybob]

When you get a chance, could you give an example of how it's supposed
to be used? i keep getting this error:
File "/home/markybob/svn/safebrowsing-python/query_lookup/
query_lookup.py", line 43, in lookup_by_url
md5_hash_list.append(md5(url_comp).hexdigest())
TypeError: 'module' object is not callable

Thanks again,
Marcos

On Jan 17, 9:51 am, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

[Thejaswi Puthraya]

On Jan 20, 12:37 am, markybob <marky...@gmail.com> wrote:
> When you get a chance, could you give an example of how it's supposed
> to be used?  i keep getting this error:
>   File "/home/markybob/svn/safebrowsing-python/query_lookup/
> query_lookup.py", line 43, in lookup_by_url
>     md5_hash_list.append(md5(url_comp).hexdigest())
> TypeError: 'module' object is not callable

If you plan to make use of the query lookup file, then copy the
query_lookup.py into the application folder where you will use it and
then use something like below:

>>> from query_lookup import Lookup
>>> l = Lookup("dbpath")
>>> l.lookup_by_url("someurl.com")

Note: The folders that I have created under trunk have been done to
keep them logically different.

--
Cheers
Thejaswi Puthraya
http://thejaswi.info

[markybob]

That's actually exactly how I had it, but I'm still getting that
error.

Traceback (most recent call last):
File "test.py", line 3, in ?
l.lookup_by_url("http://google.com")

File "/home/markybob/svn/safebrowsing-python/query_lookup/
query_lookup.py", line 43, in lookup_by_url
md5_hash_list.append(md5(url_comp).hexdigest())
TypeError: 'module' object is not callable

On Jan 19, 9:09 pm, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

[Thejaswi Puthraya]

On Jan 21, 5:25 am, markybob <marky...@gmail.com> wrote:
> That's actually exactly how I had it, but I'm still getting that
> error.
>
> Traceback (most recent call last):
>   File "test.py", line 3, in ?
>     l.lookup_by_url("http://google.com")
>   File "/home/markybob/svn/safebrowsing-python/query_lookup/
> query_lookup.py", line 43, in lookup_by_url
>     md5_hash_list.append(md5(url_comp).hexdigest())
> TypeError: 'module' object is not callable

Can you let me know of the python version you are using and the
platform on which you are testing the program?

I can't reproduce the error that you are getting. I am using python
2.5 on linux. Check if you have the hashlib module and sqlite3 module.

[markybob]

I was using Python 2.4 on Debian. I switched to python 2.5, which
gets further along, at least :) Now I'm getting this error:

Traceback (most recent call last):

File "test.py", line 3, in <module>
l.lookup_by_url("http://www.google.com")
File "/home/markybob/svn/safebrowsing-python/query_lookup/
query_lookup.py", line 72, in lookup_by_url
cur.execute("select * from url_hashes_table where url_hash='%s';" %
(md5_hash))
sqlite3.DatabaseError: file is encrypted or is not a database

On Jan 20, 8:05 pm, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

[Thejaswi Puthraya]

On Jan 24, 5:08 am, markybob <marky...@gmail.com> wrote:
> I was using Python 2.4 on Debian.  I switched to python 2.5, which
> gets further along, at least :)  Now I'm getting this error:
>
> Traceback (most recent call last):
>   File "test.py", line 3, in <module>
>     l.lookup_by_url("http://www.google.com")
>   File "/home/markybob/svn/safebrowsing-python/query_lookup/
> query_lookup.py", line 72, in lookup_by_url
>     cur.execute("select * from url_hashes_table where url_hash='%s';" %
> (md5_hash))
> sqlite3.DatabaseError: file is encrypted or is not a database

How are you fetching the database? I use the script that is provided
here http://safebrowsing-python.googlecode.com/svn/trunk/hash_fetch/

Hopefully it helps. I am planning to make a screencast that might be
of some help.

[markybob]

I'm not being able to figure out hash_fetch, either. :p

from prepare_db import Google_Blacklist
l = Google_Blacklist("KEYHERE", "http://sb.google.com/safebrowsing/
update?client=api", "mal.db", "malware")
l.fetch_data()

gives me...

Traceback (most recent call last):
File "test.py", line 3, in <module>

l.fetch_data()
File "/home/markybob/svn/safebrowsing-python/hash_fetch/
prepare_db.py", line 35, in fetch_data
cur.execute("select * from %s_version;" %(self.badware_type))
sqlite3.OperationalError: no such table: malware_version

A screencast would be greatly appreciated! Thanks :)

On Jan 23, 8:18 pm, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

> How are you fetching the database? I use the script that is provided

> herehttp://safebrowsing-python.googlecode.com/svn/trunk/hash_fetch/

[Thejaswi Puthraya]

On Jan 24, 7:39 am, markybob <marky...@gmail.com> wrote:
> I'm not being able to figure out hash_fetch, either.  :p
>
> from prepare_db import Google_Blacklist
> l = Google_Blacklist("KEYHERE", "http://sb.google.com/safebrowsing/
> update?client=api", "mal.db", "malware")
> l.fetch_data()

First create a SQLite Db according to the schema given
http://code.google.com/p/safebrowsing-python/wiki/SQLSchema
and the url for the Google_Blacklist class is
"http://sb.google.com/safebrowsing/update?client=api&apikey=$
{key}&version=goog-${badware_type}-hash:${version}"

Enter it exactly as is.

Python string templates are used to substitute the key, badware_type
and version.

So this is how code would be

>>> from prepare_db import Google_Blacklist
>>>
>>> data_url = "http://sb.google.com/safebrowsing/update?client=api&apikey=${key}&version=goog-${badware_type}-hash:${version}"
# "malware" and "black" are valid options
>>> g = Google_Blacklist("api_key",data_url,db_path,"malware")
>>> g.fetch_data()

> gives me...
> Traceback (most recent call last):
>   File "test.py", line 3, in <module>
>     l.fetch_data()
>   File "/home/markybob/svn/safebrowsing-python/hash_fetch/
> prepare_db.py", line 35, in fetch_data
>     cur.execute("select * from %s_version;" %(self.badware_type))
> sqlite3.OperationalError: no such table: malware_version

This is because you don't have the sqlite db tables ready.

My code was only supposed to give pointers on how to use the api than
make it ready for say production-level code. But if you are willing to
contribute to make it more accessible I would be glad.

> A screencast would be greatly appreciated!  Thanks :)

Will work on it.

[markybob]

Thanks, those helped. I know this is a weird request, but do you have
the URL of a black/malware site, so I can test it? Every phishing
link I've found in my spam folder seems to pass and not get flagged,
and I want to be sure that I have things working properly.
Thanks,
Marcos

On Jan 24, 7:50 am, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

> On Jan 24, 7:39 am, markybob <marky...@gmail.com> wrote:
>
> > I'm not being able to figure out hash_fetch, either. :p
>
> > from prepare_db import Google_Blacklist
> > l = Google_Blacklist("KEYHERE", "http://sb.google.com/safebrowsing/
> > update?client=api", "mal.db", "malware")
> > l.fetch_data()
>

> First create a SQLite Db according to the schema givenhttp://code.google.com/p/safebrowsing-python/wiki/SQLSchema

[Thejaswi Puthraya]

On Jan 25, 2:30 am, markybob <marky...@gmail.com> wrote:
> Thanks, those helped.  I know this is a weird request, but do you have
> the URL of a black/malware site, so I can test it?  

Sure. Here are some that I have found.
1) hxxp://www.fxmp3.com/
2) http://astrasong.ru/mp3/.spre.php (something that was found in this
groups' archives ) ;-)

> Every phishing link I've found in my spam folder seems to pass and not
> get flagged, and I want to be sure that I have things working properly.

Currently safebrowsing-python does not support lookups by IP
Addresses. I am working on that feature.

[Thejaswi Puthraya]

On Jan 25, 6:54 am, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

> On Jan 25, 2:30 am, markybob <marky...@gmail.com> wrote:
>
> > Thanks, those helped.  I know this is a weird request, but do you have
> > the URL of a black/malware site, so I can test it?  
>
> Sure. Here are some that I have found.
> 1) hxxp://www.fxmp3.com/

It should be http://www.fxmp3.com
Some others that I have found are
0755mp3.com/
12345mp3.com/
apnaymp3.com/mp3songs/
av.qqbtmp3.com/
bbs.fxmp3.com/
dillonmp3.com/2.0/
www.52tingmp3.com/
www.apnaymp3.com/mp3songs/

Just go to stopbadware.org and check in request for review form. It
will ask you a portion of the url. I just put mp3.com and it returned
100s of results. But remember that only sites added by google will get
blacklisted by the api.

> 2)http://astrasong.ru/mp3/.spre.php(something that was found in this

[markybob]

figured out that my problem is with prepare_db. i created the db by
following the schema. however, prepare_db isnt inserting any data
into it. no errors, just nothing happens. maybe the url looking
inside the db with sqlitebrowser shows the table structures but no
data. maybe the url format in the usage wiki is wrong?

markybob@peg:~/svn/safebrowsing-python/hash_fetch$ md5sum google.db
f3bb17b55ad6f864259ab5dd8c9ecb01 google.db
markybob@peg:~/svn/safebrowsing-python/hash_fetch$ python2.5 test.py
markybob@peg:~/svn/safebrowsing-python/hash_fetch$ md5sum google.db
f3bb17b55ad6f864259ab5dd8c9ecb01 google.db


On Jan 24, 8:12 pm, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>

wrote:
> On Jan 25, 6:54 am, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
> wrote:
>
> > On Jan 25, 2:30 am, markybob <marky...@gmail.com> wrote:
>
> > > Thanks, those helped. I know this is a weird request, but do you have
> > > the URL of a black/malware site, so I can test it?
>
> > Sure. Here are some that I have found.
> > 1) hxxp://www.fxmp3.com/
>

> It should behttp://www.fxmp3.com

> Some others that I have found are
> 0755mp3.com/
> 12345mp3.com/
> apnaymp3.com/mp3songs/
> av.qqbtmp3.com/
> bbs.fxmp3.com/
> dillonmp3.com/2.0/www.52tingmp3.com/www.apnaymp3.com/mp3songs/
>
> Just go to stopbadware.org and check in request for review form. It
> will ask you a portion of the url. I just put mp3.com and it returned
> 100s of results. But remember that only sites added by google will get
> blacklisted by the api.
>

> > 2)http://astrasong.ru/mp3/.spre.php(somethingthat was found in this

[Thejaswi Puthraya]

On Jan 26, 4:02 am, markybob <marky...@gmail.com> wrote:
> figured out that my problem is with prepare_db.  i created the db by
> following the schema.  however, prepare_db isnt inserting any data
> into it.  no errors, just nothing happens. maybe the url looking
> inside the db with sqlitebrowser shows the table structures but no
> data.  maybe the url format in the usage wiki is wrong?
>
> markybob@peg:~/svn/safebrowsing-python/hash_fetch$ md5sum google.db
> f3bb17b55ad6f864259ab5dd8c9ecb01  google.db
> markybob@peg:~/svn/safebrowsing-python/hash_fetch$ python2.5 test.py
> markybob@peg:~/svn/safebrowsing-python/hash_fetch$ md5sum google.db
> f3bb17b55ad6f864259ab5dd8c9ecb01  google.db

I am unable to replicate your problem. I guess you are facing a
throttling problem. Google throttles requests based on the load.

Here is the output of my log at the webserver which is running the
same code.

Fri, 25 Jan 2008 07:28:14 The total number of sites added are 83
Fri, 25 Jan 2008 07:56:16 The total number of sites added are 32
Fri, 25 Jan 2008 08:28:51 The total number of sites added are 26
Fri, 25 Jan 2008 08:56:55 The total number of sites added are 6
Fri, 25 Jan 2008 09:28:04 The total number of sites added are 0
Fri, 25 Jan 2008 09:56:09 The total number of sites added are 39
Fri, 25 Jan 2008 10:28:39 The total number of sites added are 30
Fri, 25 Jan 2008 10:56:43 The total number of sites added are 33
Fri, 25 Jan 2008 11:28:23 The total number of sites added are 49
Fri, 25 Jan 2008 11:56:30 The total number of sites added are 28

[markybob]

You were right, I was being throttled. All good now. Thanks again :)

On Jan 25, 8:49 pm, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

[markybob]

even with a full database now, i cant get a possitive hit with any of
the sites you named or any other site. are you getting any possitve
hits ever?

On Jan 24, 8:12 pm, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>

wrote:
> On Jan 25, 6:54 am, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
> wrote:
>
> > On Jan 25, 2:30 am, markybob <marky...@gmail.com> wrote:
>
> > > Thanks, those helped. I know this is a weird request, but do you have
> > > the URL of a black/malware site, so I can test it?
>
> > Sure. Here are some that I have found.
> > 1) hxxp://www.fxmp3.com/
>

> It should behttp://www.fxmp3.com

> Some others that I have found are
> 0755mp3.com/
> 12345mp3.com/
> apnaymp3.com/mp3songs/
> av.qqbtmp3.com/
> bbs.fxmp3.com/
> dillonmp3.com/2.0/www.52tingmp3.com/www.apnaymp3.com/mp3songs/
>
> Just go to stopbadware.org and check in request for review form. It
> will ask you a portion of the url. I just put mp3.com and it returned
> 100s of results. But remember that only sites added by google will get
> blacklisted by the api.
>

> > 2)http://astrasong.ru/mp3/.spre.php(somethingthat was found in this

[Thejaswi Puthraya]

On Mar 4, 10:52 am, markybob <marky...@gmail.com> wrote:
> even with a full database now, i cant get a possitive hit with any of
> the sites you named or any other site.  are you getting any possitve
> hits ever?

Yes. With all those sites. If you have a doubt you can also check up
those sites and these too (http://www.keygen.ru/ and http://mp3top.cn).
To cross check whether it works or not you can look it up on my site
at http://thejaswi.info/projects/safe_browsing/ The same code that is
available in the repository is being used at my site too.

[Dan]

Heya - love ur demo site but I'm having problems getting it two run
getting

no module named sqlite3

but I have installed sqlite :(




On Mar 4, 11:49 am, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

> On Mar 4, 10:52 am, markybob <marky...@gmail.com> wrote:
>
> > even with a full database now, i cant get a possitive hit with any of
> > the sites you named or any other site. are you getting any possitve
> > hits ever?
>
> Yes. With all those sites. If you have a doubt you can also check up

> those sites and these too (http://www.keygen.ru/andhttp://mp3top.cn).

> To cross check whether it works or not you can look it up on my site

> athttp://thejaswi.info/projects/safe_browsing/The same code that is

[Dan]

Making head-way but

root@lvps212-241-214-252:/usr/local/bin# update-gsb-db.py

Traceback (most recent call last):

File "/usr/local/bin/update-gsb-db.py", line 3, in <module>
from pygsb.hash_fetch import prepare_db
ImportError: No module named pygsb.hash_fetch


this hash module - where is it?

On Mar 10, 3:02 pm, Dan <dancu...@gmail.com> wrote:
> Heya - love ur demo site but I'm having problems getting it two run
> getting
>
> no module named sqlite3
>
> but I have installed sqlite :(
>
> On Mar 4, 11:49 am, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
> wrote:
>
> > On Mar 4, 10:52 am, markybob <marky...@gmail.com> wrote:
>
> > > even with a full database now, i cant get a possitive hit with any of
> > > the sites you named or any other site. are you getting any possitve
> > > hits ever?
>
> > Yes. With all those sites. If you have a doubt you can also check up
> > those sites and these too (http://www.keygen.ru/andhttp://mp3top.cn).
> > To cross check whether it works or not you can look it up on my site

> > athttp://thejaswi.info/projects/safe_browsing/Thesame code that is

[Thejaswi Puthraya]

On Mar 10, 8:02 pm, Dan <dancu...@gmail.com> wrote:
> Heya - love ur demo site but I'm having problems getting it two run
> getting
>
> no module named sqlite3
>
> but I have installed sqlite :(

I underestimated the use of Python2.4. The code that I had written was
python2.5 compatible (python2.5 includes the sqlite3 module by
default). I have made changes that make them easier to work with
python2.4

Check http://oss.itsystementwicklung.de/trac/pysqlite/ for the
pysqlite module for python2.4 if you haven't downloaded it.

--

[Dan]

Thanks for the input.

I upgraded my Python to 2.5 because of this very error. The db
preparing works well now but.


update-gsb-db.py

Traceback (most recent call last):

File "/usr/local/bin/update-gsb-db.py", line 3, in <module>
from pygsb.hash_fetch import prepare_db
ImportError: No module named pygsb.hash_fetch

On Mar 11, 1:03 pm, Thejaswi Puthraya <thejaswi.puthr...@gmail.com>
wrote:

> On Mar 10, 8:02 pm, Dan <dancu...@gmail.com> wrote:
>
> > Heya - love ur demo site but I'm having problems getting it two run
> > getting
>
> > no module named sqlite3
>
> > but I have installed sqlite :(
>
> I underestimated the use of Python2.4. The code that I had written was
> python2.5 compatible (python2.5 includes the sqlite3 module by
> default). I have made changes that make them easier to work with
> python2.4
>

> Checkhttp://oss.itsystementwicklung.de/trac/pysqlite/for the

[Dan]

whoops wrong thread! Still - found out that prepare_db was sitting
with 2.4! doh. Ran set-up again with 2.5 works ok.

[markybob]

Things work somewhat wrong. It flags http://www.apnaymp3.com/mp3songs/
as a malware site, but not http://www.apnaymp3.com/mp3songs ...just
removing the trailing forward slash. Are there ways around these
limitations?

Thanks

[Thejaswi Puthraya]

On Mar 19, 3:20 pm, markybob <marky...@gmail.com> wrote:
> Things work somewhat wrong.  It flagshttp://www.apnaymp3.com/mp3songs/
> as a malware site, but nothttp://www.apnaymp3.com/mp3songs ...just

> removing the trailing forward slash.  Are there ways around these
> limitations?

1) MD5 of http://www.apnaymp3.com/mp3songs/ is
ace36a1334dc66714bbdc7d60e138c7d
& that of http://www.apnaymp3.com/mp3songs is
a80daec878ef21b11f80857b6d4f0945
Notice the difference.

2) If you have actually read the Developer's Guide of Safebrowsing,
you would have read about URL canonicalization. The url must be
canonicalized before it is input to the system. Read
http://code.google.com/apis/safebrowsing/developers_guide.html for
more details. I think you are not doing the lookup in the right way.

3) Read RFC2396. It clearly mentions that a canonicalized URL need not
point to the same resource as that pointed by a URl without
canonicalization. So Google has done the right thing by hashing the
canonicalized URLs.

[Mother]

I can't get this to work on python 2.4.

>>> l = Lookup(db_path)
>>> l.lookup_by_url("http://www.mozilla.com/firefox/its-a-trap.html")
>>> l.lookup_by_url("http://www.google.com/tools/firefox/safebrowsing/phish-o-rama.html")

I've got hashlib for 2.4. I've got pysqlite2.

Creating and downloading the safebrowsing works, but it won't report
anything as M.

Maybe I should just get 2.5...

Thoughts?

Thanks,
MO.THER>

[Thejaswi Puthraya]

On Apr 4, 2:01 am, Mother <xenmot...@gmail.com> wrote:
> I can't get this to work on python 2.4.
>
> >>> l = Lookup(db_path)
> >>> l.lookup_by_url("http://www.mozilla.com/firefox/its-a-trap.html")
> >>> l.lookup_by_url("http://www.google.com/tools/firefox/safebrowsing/phish-o-rama.html")

Both the URLs will not be available in the database. Check
http://code.google.com/apis/safebrowsing/developers_guide.html#PerformingLookups
for more details.

> I've got hashlib for 2.4. I've got pysqlite2.

Sufficient to work with safebrowsing python.

> Creating and downloading the safebrowsing works, but it won't report
> anything as M.

Check some sites available in this thread that i have mentioned.

> Maybe I should just get 2.5...

This decision is totally up to you. I don't think it will make a lot
of difference.

[mo ther]

On Fri, Apr 4, 2008 at 10:29 AM, Thejaswi Puthraya <thejaswi...@gmail.com> wrote:

On Apr 4, 2:01 am, Mother <xenmot...@gmail.com> wrote:
> I can't get this to work on python 2.4.
>
> >>> l = Lookup(db_path)
> >>> l.lookup_by_url("http://www.mozilla.com/firefox/its-a-trap.html")
> >>> l.lookup_by_url("http://www.google.com/tools/firefox/safebrowsing/phish-o-rama.html")

Both the URLs will not be available in the database. Check
http://code.google.com/apis/safebrowsing/developers_guide.html#PerformingLookups
for more details.

> I've got hashlib for 2.4. I've got pysqlite2.

Sufficient to work with safebrowsing python.

> Creating and downloading the safebrowsing works, but it won't report
> anything as M.

Check some sites available in this thread that i have mentioned.

You don't think the above URLs that I tried are appropriate?

 

> Maybe I should just get 2.5...

This decision is totally up to you. I don't think it will make a lot
of difference.

It doesn't.

Does anyone have the code from the svn repo working?

Thanks,

MOTHER>

[Mother]

On Apr 4, 2:06 pm, "mo ther" <xenmot...@gmail.com> wrote:
> On Fri, Apr 4, 2008 at 10:29 AM, Thejaswi Puthraya <
>
>
>
> thejaswi.puthr...@gmail.com> wrote:
>
> > On Apr 4, 2:01 am, Mother <xenmot...@gmail.com> wrote:
> > > I can't get this to work on python 2.4.
>
> > > >>> l = Lookup(db_path)
> > > >>> l.lookup_by_url("http://www.mozilla.com/firefox/its-a-trap.html")
> > > >>> l.lookup_by_url("
> >http://www.google.com/tools/firefox/safebrowsing/phish-o-rama.html")

I think what was confusing me is the wiki example:

>>> from query_lookup import Lookup
>>>
>>> l = Lookup("path_to_sqlite_db")
>>> l.lookup_by_url("http://thejaswi.info/")
>>>
>>> l.lookup_by_url("www.fxmp3.com")
'M'
>>>

which doesn't work:

>>> l.lookup_by_url("www.fxmp3.com")

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
File "/home/curtis/working/safebrowser-python/query_lookup/
query_lookup.py", line 66, in lookup_by_url
raise AttributeError("Invalid URL.")
AttributeError: Invalid URL.

But yeah, another example in this thread does:

>>> l.lookup_by_url("http://www.apnaymp3.com/mp3songs/")
'M'

Thanks,
MOTHER>

>
> > Both the URLs will not be available in the database. Check
>

> >http://code.google.com/apis/safebrowsing/developers_guide.html#Perfor...

[Sean Murphy]

On Apr 3, 2008, at 5:01 PM, Mother wrote:

> I can't get this to work on python 2.4.
>
>>>> l = Lookup(db_path)
>>>> l.lookup_by_url("http://www.mozilla.com/firefox/its-a-trap.html")
>>>> l.lookup_by_url("http://www.google.com/tools/firefox/safebrowsing/phish-o-rama.html
>>>> ")
>

> Creating and downloading the safebrowsing works, but it won't report
> anything as M.

Those test URLs do not actually exist in Google's database. They are
only useful for verifying the status of phishing protection in
Firefox, and are manually inserted in the client-side local database
by Firefox's url-classifier component:

See <http://mxr.mozilla.org/firefox/source/browser/components/safebrowsing/content/malware-warden.js#59
>.

The latter URL existed to test phishing protection in Firefox when it
was formerly provided by a Google supplied extension. Now that the
code has been integrated directly into the browser, the first URL is
what they currently use.

Having said that, you'll have better luck testing lookup effectiveness
by using some verified URLs from http://phishtank.com.

Hope that helps,
Sean.

[Thejaswi Puthraya]

On Apr 5, 1:18 am, Mother <xenmot...@gmail.com> wrote:
> On Apr 4, 2:06 pm, "mo ther" <xenmot...@gmail.com> wrote:
>
> > On Fri, Apr 4, 2008 at 10:29 AM, Thejaswi Puthraya <
>
> > thejaswi.puthr...@gmail.com> wrote:
>
> > > On Apr 4, 2:01 am, Mother <xenmot...@gmail.com> wrote:
> > > > I can't get this to work on python 2.4.
>
> > > > >>> l = Lookup(db_path)
> > > > >>> l.lookup_by_url("http://www.mozilla.com/firefox/its-a-trap.html")
> > > > >>> l.lookup_by_url("
> > >http://www.google.com/tools/firefox/safebrowsing/phish-o-rama.html")
>
> I think what was confusing me is the wiki example:
>
>
>
> >>> from query_lookup import Lookup
>
> >>> l = Lookup("path_to_sqlite_db")
> >>> l.lookup_by_url("http://thejaswi.info/")
>
> >>> l.lookup_by_url("www.fxmp3.com")
> 'M'
>
> which doesn't work:
>
> >>> l.lookup_by_url("www.fxmp3.com")
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/home/curtis/working/safebrowser-python/query_lookup/
> query_lookup.py", line 66, in lookup_by_url
> raise AttributeError("Invalid URL.")
> AttributeError: Invalid URL.
>

That site doesn't exist anymore. One more point that I would want to
get to the notice is that once a site has been flagged, it
might be reviewed and removed from the list. I have changed the
example in the wiki.

A small info about how the site works is:
First it tries to verify whether the site is up. Next it queries the
database.

> But yeah, another example in this thread does:
>
> >>> l.lookup_by_url("http://www.apnaymp3.com/mp3songs/")
>
> 'M'