API v4 not flagging URLs that both Chrome and Firefox flag

[Sean J]

Hello,

I am using v4 of the safe-browsing API, and am running into some trouble. The API does not flag some malicious websites that both Chrome and Firefox browsers flag as unsafe. This is strange to me as it is my understanding that both this API and those browsers use the same data to check against.

Here is the body of my request:

{'client': {'clientId': 'opslog', 'clientVersion': '1.0.0'},
 'threatInfo': {'platformTypes': ['ANY_PLATFORM'],
                'threatEntries': [{'url': 'tripadvisor-co-uk.com'}],
                'threatEntryTypes': ['THREAT_ENTRY_TYPE_UNSPECIFIED',
                                     'URL',
                                     'EXECUTABLE',
                                     'IP_RANGE'],
                'threatTypes': ['THREAT_TYPE_UNSPECIFIED',
                                'MALWARE',
                                'SOCIAL_ENGINEERING',
                                'UNWANTED_SOFTWARE',
                                'POTENTIALLY_HARMFUL_APPLICATION']}}

and the response:

{}

I know my API is configured correctly, though, because the API does respond to some URLs... For example:

request:

{'client': {'clientId': 'opslog', 'clientVersion': '1.0.0'},
 'threatInfo': {'platformTypes': ['ANY_PLATFORM'],
                'threatEntries': [{'url': 'lupytehoq.com'}],
                'threatEntryTypes': ['THREAT_ENTRY_TYPE_UNSPECIFIED',
                                     'URL',
                                     'EXECUTABLE',
                                     'IP_RANGE'],
                'threatTypes': ['THREAT_TYPE_UNSPECIFIED',
                                'MALWARE',
                                'SOCIAL_ENGINEERING',
                                'UNWANTED_SOFTWARE',
                                'POTENTIALLY_HARMFUL_APPLICATION']}}

response:

{u'matches': [{u'cacheDuration': u'300.000s',
               u'platformType': u'ANY_PLATFORM',
               u'threat': {u'url': u'lupytehoq.com'},
               u'threatEntryType': u'URL',
               u'threatType': u'MALWARE'}]}

(Note, the single quotes and unicode identifiers are there because I am using the python requests library and printing information about the response and this is how python is formatting it).

Thank you

[Sean Cox]

Theoretically, I believe the URLs are supposed to be canonicalized before the request is made.

[Moussa Oumar]

I am having the exact same issue using the V4 Lookup API.

After some more research, it seems like this is actually pretty common.

I wonder if the browser is actually using a different implementation / API or querying from a different database.

My Post: https://groups.google.com/forum/#!topic/google-safe-browsing-api/HSmnQ6bVNk4

[Emmanuel Sellier]

A different Database is the actual answer.

Check in the Firefox source code. You'll get the answer.

Emmanuel

[Fanny Dwargee]

Is the API_ABUSE (threat type 6) the database you're referring to?

As you can see on https://dxr.mozilla.org/mozilla-central/source/toolkit/components/url-classifier/chromium/safebrowsing.proto...

// Types of threats.

enum ThreatType {

  // Unknown.

  THREAT_TYPE_UNSPECIFIED = 0;

  // Malware threat type.

  MALWARE_THREAT = 1;

  // Social engineering threat type.

  SOCIAL_ENGINEERING_PUBLIC = 2;

  // Unwanted software threat type.

  UNWANTED_SOFTWARE = 3;

  // Potentially harmful application threat type.

  POTENTIALLY_HARMFUL_APPLICATION = 4;

  // Social engineering threat type for internal use.

  SOCIAL_ENGINEERING = 5;

  // API abuse threat type.

  API_ABUSE = 6;

}

Fanny