Malware URL is detected in the Google safebrowsing check site but not with the API v4

[Javier Podavini]

Hello.

I'm having an issue with certain URLs.

go.trackmyclicks202.com

Is detected as malware in https://www.google.com/transparencyreport/safebrowsing/diagnostic

But I tried to detect it with the API using every single list available in the Lookup API and it's safe..

The site shows the red screen in chrome and is causing us problems. What can be the problem? is it a bug in the API or a missing list?

Thanks.

[Claudia Artenta]

Hello, 

I'm also facing the same troubles, for example with these URLs: 

http://109.236.83.247/zrd.php?1=5&2=21689437959&3=&4=999&rand=20936&5=2

https://the2pappsoftsubgroup.info/23po4ntogs/cba/index.php

The API is returning that this URL is safe, but with this online tool https://www.google.com/transparencyreport/safebrowsing/diagnostic and the browser directly, it's detected as malicious.

Any help is appreciated.

Thanks

[Azilet Beishenaliev]

Among three mentioned URLs above, I got one as unsafe and two others as safe.
I suspect it is related to region constraint we are sending in update requests. For example, I specify "US" region. What regions do you use... Javier, Claudia?

Would be very useful if Alex Wozniack the moderator dropped a comment here ;)

[Miclain Keffeler]

On Thursday, August 31, 2017 at 1:14:24 PM UTC-5, Miclain Keffeler wrote:

I am having the same problem with a social engineering url that I will not post for obvious reasons (sorry about that haha)

In lookup API v4 it shows its clean but from the website it shows it is social engineering of some kind. Are they using different databases or is there a region based problem?

I am using all possible lists. 

Let us know!

Miclain Keffeler

[Miclain Keffeler]

Vicente,

I have not yet had any luck with this. Also have not received any support outside of this group since posting.

Can we get a moderator or somebody else in here who understands the innerworkings, or can explain this phenomenon with something reasonable? 

On Thursday, September 14, 2017 at 10:47:48 AM UTC-5, Vicente Gil wrote:

Is there any support assistant here?

I'm having this issue for several days and nobody has solved it yet.

[Fanny Dwargee]

The moderator(s) never answered questions about that issue although being asked for so many times. :(

I'm afraid Google doesn't want you for replacing Chrome. :(

Take a look to my previous post at https://groups.google.com/d/topic/google-safe-browsing-api/9ILxRp5hY4Y/discussion and tell me what do you think

Regards

[Alex Wozniak]

Hi all,

Apologies for the lack of response!

There were some recent discrepancies discovered between our Transparency Report and our public Safe Browsing API. These should be resolved in the near future. Please do let us know if you see any unexpected behavior and we'll take a look in a more timely fashion.

Thanks,

Alex

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.
To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

[Alex Wozniak]

Hi David,

Could you please share some specific examples?

It's worth noting that the Transparency Report and API serve different purposes. There was some recent discussion on the Github repo related to this topic if you're interested: https://github.com/google/safebrowsing/issues/30#issuecomment-341249434

Thanks,

Alex

On Sat, Nov 4, 2017 at 4:03 AM <dru...@homebridge.com> wrote:

Are there any updates on this.  I finally spent so long getting the resources together for this and now it seems that the API doesn't catch everything that the Transparency Report does.  It doesn't appear that they use the same set of data.  Should we make HTTP calls to the Transparency Report until this is sorted out? 

Cheers,

David

[dru...@homebridge.com]

sry, typed a large reply and tried to make the URLs unclickable but it got deleted anyway.  Here are two examples of searches that are safe in the API and are blocked in the TR with a red unsafe (as opposed to yellow warning).

http : gmg . com . pk / cig-bin/cig-bin/3/index.php

http : gmg . com . pk

I'll try to get more but wanted to get two out now..

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Alex Wozniak]

Thanks, David. We are looking into this and will follow up shortly.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

[Fanny Dwargee]

Can more than 2 months be expected as "short" in Google terms?

What's the point of the API if it's unable to get in par with the TR? It's near to unusable IMHO

Regards

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Alex Wozniak]

Hi Fanny,

Apologies for not updating this thread in a timely fashion. The subsequent investigation did not turn up any issues. As mentioned in a related discussion on GitHub, some API clients receive different lists due to data sharing restrictions, which in some occasions may lead to discrepancies.

[Fanny Dwargee]

Thank you Alex

[Rajesh Developer]

hey guys.. i had some problem. i don't know what is wrong. i received same hash from the Google Safe Browsing update API for multiple region. can anyone tell then what is the need for region or i'm doing something wrong???

[Eriq VanBibber]

Not sure if it was missed in previous posts and other topics, but the hashes in the GSB api are VERY SPECIFIC and do NOT provide a "ranking" of a URL or resource.

The transparency report take a forward-looking approach to a url or resource, meaning that it will look at the specific URL and "extensions" to it to determine a "level of safety".

The GSB api is a very binary response of GOOD or BAD and nothing in-between.

So, a URL that shows as BAD in the GSB api, should also show bad in the transparency report, however the reverse cannot be expected to be true.

I agree with the overall topics around this, and i think Google *should* implement an API to "rank" a resource for safety, but for now GSB does not do this.

The other problematic part is that the GSB is only a set of hashes, so we (as developers) cannot do any work ourselves to 'explore' stuff.  what i mean is no ability to take a url and check to see if there are any similar URLs in the hash-set.  The hashing precludes any sort of relationship checking between 2 or more URLs :(.

I hope this helps the community on this topic.

@GSBTeam:  chime in here if i've misquoted anything, but this has been my experience up to now.

[Rajesh Developer]

thank you Eriq VanBibber. can you share any java code to hash the url for GSB?. i hashed the urls.. some of the hash are not found the rawhash obtained from GSB update api. but those urls were marked as threated.

[Eriq VanBibber]

Rajesh,

Unfortunately, all my code is in .NET and is private to the company I represent.

what i can say is that i was able to hash the URLs properly by following the API documentation and using their "example set" to validate my algorithm.

the difficulty is not getting the hash code itself, but properly formatting the URL to the requirements of the API.

i suggest looking at this page, and writing your algorithm to test the examples they provide as well: https://developers.google.com/safe-browsing/v4/urls-hashing

[Ben Sanders]

I agree with Eriq, using the provided test examples is the best way to ensure you will produce hashes compatible with Safe Browsing. After canonicalization, there are also the suffix/prefix expressions to allow a single url to try a variety of 'related' url hashes.

Other tips:

* Ensure you are hashing the ascii bytes of the url, and not in utf-16 or other format. And make sure there aren't any trailing characters (null byte, whitespace)

* The returned bytes are prefixes of the full hash, typically 4 bytes. So you'd just compare the first four bytes of your generated hash to the prefix to determine if there is a match.

[Rajesh Developer]

no problem Eriq. thank you for the quick response. i'll check.

[Eriq VanBibber]

Ben's comments are valid and also appreciated.

However, just for clarity, when Ben mentions "related" url hashes, he only means related formats of the same URL.  He doesn't mean any case of being able to relate two distinctly different urls.  You can get hashes for "flavors" of the same url:  like http://this.is.bad/folder/folder/resource and http://this.is.bad/folder/folder, the latter form simply being one that has removed the last 'folder' element of the path.

however, trying to compare http://this.is.bad/folder to http://this.is.bad/folder-two is impossible because the hash of both of those would be wildly different.

@ben.  i still did not get a response to my concern about the canonicalization.  can you look at my post about URL Hashing Rules?

-Eriq

[وزير عزرائيل‎]

[Eriq VanBibber]

Javier,

please read thru the previous replies to this post, specifically the post from Aug 27.  i have mentioned the differences between Google's Transparency report and the SafeBrowse lists.  They are not the same and can only be correlated in one direction:  resources found as bad in SafeBrowse should also show in the Transparency Report, but the other direction should not be expected.

-Eriq

[Rajesh Developer]

hi Eriq,

i'm confused a bit on Google Safe Browsing update api.

Step 1: Google suggested to create a list of expressions from the given URL.

Step 2: then they asked to check the prefix hash of the each expression against the localdatabase(where the RawHash was stored).

           if any of the hash was not found on the localdatabase means it will safe.

           else if any  one prefix is found means it will have chance to marked as threat.

for verification they suggest to use FullHashAPI.

if the full hash of the expression(which prefix is found on local database) was matched with any one of the hash on the response means definitely it will be threated. otherwise it is safe.

now i'm confused on, is Google trying to say if any one of the expression is found to be threated means the "Given url" is must be marked as threated?

[Eriq VanBibber]

Here's how this works...

imagine these resources in GSB are marked as BAD:

http://this.is.a.bad.site/dont_go_here

http://scary-place.com/gimme_your_identity

http://1.2.3.4/your_cc_will_be_stolen/if-you-browse-here

in order to protect the privacy of the data that google has on so many, and for efficient lookups and reduced storage, hashes of those 3 resources are stored.

(these are fake hashes only for illustration)

0000000000000000000000001

22222222222221111111110000

5858585858585858585858585

When you pull the hash table from GSB, you are only getting the first 4 bytes of the above hashes:

0000

2222

5858

Now, consider that you have a resource identifier like this:

http://not.really.bad/this-is-a-good-place

Further, imagine that the hash of this resource was:

5858121212121212121212121

You grab the first four bytes of your hash - 5858 - and compare to the hash set you received from GSB.

Well, there's a match, but you cannot be sure that you actually have a bad resource, only possibly.

So, you take the full hash and send it up to to GSB.

GSB compares the entire hash to it's table of full hashes, which in this example, doesn't match and the resource is safe.

Further, this DOES NOT MEAN that the resource is "suspicious" or "could be dangerous".  There is NO relationship between the hashes that can be used to "guess" the possible safety of a resource.

Let me know if this makes sense.

-Eriq

[Elizabeth C. Ferrari, REALTOR, MFI, LC CNT TC]

Recently paid GoDaddy to join GetFound for my site: https://elizabethcferrarirealtor.com/

I keep getting this error message:It appears this website contains malware. Please enter a different URL.

I've requested that Safe Browsing remove the error, to no avail.

Any help is appreciated.

[Alex Wozniak]

Hi Elizabeth,

Safe Browsing has not flagged that website as malware: https://transparencyreport.google.com/safe-browsing/search?url=elizabethcferrarirealtor.com

It is likely that GoDaddy are using a different service for this malware check. I would recommend reaching out to GoDaddy directly to diagnose the issue further.

Alex

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.
To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--

Alex Wozniak | SWE, Safe Browsing | aw...@google.com | 734-748-3306

[Elizabeth C. Ferrari]

Thanks.

So here's the situation... I originally used WordPress to build the site and had a lot of trouble.

I switched to GoDaddy GoCentral website builder.

GoDaddy is stating that Google 'blacklisted' my site.

How can I get it removed???

THANKS!

[Alex Wozniak]

Hi Elizabeth,

Sorry for the trouble you're experiencing. As mentioned previously, there's no indication on our side that Safe Browsing has ever flagged your website as a malware risk. I recommend you point them to the Transparency Report link I sent in the previous response.

Good luck!

Alex