Update API

[Igor]

1)  I use https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch

I have response with rawHashes like ""AD+C4wBRfD4Apy6kAKjZQACyoBgAwKR3AMvDPQDRY4UA36D4AOCD+gDtt1EA9EE+ASy9IwFukRUBehE1AXyTKwGbEVYBnYA8AaKfxwIE4QICQLlHAm4bpwKG/wkCoZgOAq1IvQKuQRkCub/6Aw11CgMZkwUDME88A0n/SwNP15cDXKsLA2TjOwOhfvQD1zMVBBDGJQQaG4cEMa4rBFL2lARzzsMEdetqBHY2eASgLwUExICoBM26tQTR/H8E++MjBRhDLAUm+qsFcbWYBYNeCgWDrA8FwmulBg0s+AZpsxQGkXoPBp3ROQaeJtwGnufDBtt6Ggb1ejgHKajeB1iyaweF+RQHpuGuB6r4UwfixhYH6HmjB/QyeAgB0hsICOvdCB/kFAgjgX4IMym/CF5bewhngTIIe+oKCQFKmAkOIqsJEbQhCSQEHwkxqTEJc1VACXzErgmERpoJj+MnCaUFZQmnYi0JrnUWCa9QRAnCsfMJxKM6Cim/XwozLsQKdFaZCryQowshAgILKkDuC5pOAguz3x0L/fOcDAEc8gwQrQIMMj1YDDz5LwyawKQMn3YYDKe7WQzF9UQNHxiQDUJDBg1SSB4NWTtQDXYpGg2KOjQNzL3YDedH2A38gbsOEi4wDiwCEQ5K0DMOkeHWDpZ0nA64nzYO4/fODwnyVQ9ZxvYQA1YmECstfxAtI+AQYhEOEGJZIhBoU9cQcUEWEHhgxRCGDYIQsY4iENH54RELzIMRE1rPETGorhFB9/ARm9EYEc1iPRHu9SkSADmKEhrvBxI4Jq8SRblXEk8U/BJizZ4SlvmREtgnOhLdV3AS66vDEvLgShMLV3ITLq8jEzFQzxNuw5wToGgUE7UhjhO+ynYT0zRVE+eawhQDGc4UG3wQFCbXEhSPlNUUyjM6FN5jEBTgD5gU4JwtFSlZwRU8ijEVU8T5FV/VQxV6ERUVoAkvFc5jzBYtnS0WMZouFl4m2hZtt4kWj5boFpygYxac2iwW6ZplF027+xdOQv0XXocpF2qDYhewkn8XuIk/F"

What method can I use for decoding URLs, for saving in my own database? 

Is it even posible?

2)  Api https://safebrowsing.googleapis.com/v4/threatMatches

  I want use it for filtering harmful urls.

  I got all empty json from all my requests.

Please give me example url for response with harmful link

Maybe I don't understand correctly how to use api?(

[Ben Sanders]

The API is optimized around both bandwidth usage and preserving privacy for end users, which means that there is some work to be done encoding/decoding.  For a concrete example, we have an API client implemented in Go here: https://github.com/google/safebrowsing

1) With compressionType raw, and by using JSON, you have a list of RawHashes (https://developers.google.com/safe-browsing/v4/reference/rest/v4/threatListUpdates/fetch#rawhashes). The prefix tells you how long each hash prefix is, and the long rawHashes string is a base64'ed string of the binary data. The typical case is that the prefix size is 4, so after decoding the long rawHashes string from base64, every 4 bytes is a separate hash prefix.

Example:

{

  prefixSize: 4

  rawHashes: QUFBQUJCQkI=

}

So we first decode from base64 and get "AAAABBBB".

Then we split after every 4th byte, and get two hash prefixes, ["AAAA", "BBBB"].

To check if a URL matches a hash prefix, then we do some other work documented here: (https://developers.google.com/safe-browsing/v4/urls-hashing).

Essentially, you go through a canonicalization step, then attempt a match for several URL fragments, from the full URL down to just the bare domain. For each of those suffix/prefix expressions, hash it with SHA256, and then compare it with the hash-prefixes before. For our example, we'd take the first 4 bytes of each SHA256 we create, and compare them to our set of 4 byte prefixes (If there were longer prefixes, we would check against those too).

If there is a match, it _may_ be in the blacklist, though 4 bytes isn't quite enough to rule out false positives. After getting a partial hash match, we need to check with the server to see if the URL is blacklisted. The fullhashes.find API lets clients do that in a privacy preserving way.

Instead of sending the URL, we just send that matching prefix from before. The server will then send all matching SHA256s that are on the blacklist (usually just one. Sometimes more. Sometimes less (if an item was removed from the blacklist recently)). Then you can match locally if any of those full SHA256s match what you have locally.

Again, when dealing with JSON and the API, all binary fields (including the prefixes you send, and the returned sha256) need to be base64 encoded/decoded.

We have test URLs here: http://testsafebrowsing.appspot.com/

http://testsafebrowsing.appspot[.]com/s/phishing.html  (Have to escape the URL so that the Google Group doesn't delete is as a phishing link itself ;))

After canonicalization and prefix/suffix expression splitting, we will test a bunch of variants.

testsafebrowsing.appspot[.]com/s/phishing.html

testsafebrowsing.appspot[.]com/s/

testsafebrowsing.appspot[.]com/

appspot[.]com/s/phishing.html

appspot[.]com/s/

appspot[.]com/

Each of these will be hashed with SHA256. The first one (Which will match the local DB) is efbd4c3ab44f327eb13ca942ad7c7f0ab47ec260a4d0b8051684a01b2ef35220 in hex encoded form (it's not readable in binary).

So we'd send the first 4 bytes (efbd4c3a in binary), encoded as base64, which is 771MOg==. Then you'd get the full hash back as the base64'ed binary hash: 771MOrRPMn6xPKlCrXx/CrR+wmCk0LgFFoSgGy7zUiA=.

Example command line (linux): echo '{client: {clientId: "test", clientVersion: "manual"}, threatInfo: {threatTypes: ["SOCIAL_ENGINEERING", "MALWARE"], platformTypes: ["ANY_PLATFORM"], threatEntryTypes: ["URL"], threatEntries: [{hash: "771MOg=="}]}}' | curl -X POST -H "Content-Type: application/json" -d @- https://safebrowsing.googleapis.com/v4/fullHashes:find?key=<YourAPIKey>

(Make sure to put your API key in)

2) If you want to use the threatMatches API instead, you can do so, though it isn't privacy preserving.

The command is very similar, except you include the url to check instead of a hash prefix. You also don't have to check the DB ahead of time, or perform all the prefix/suffix expression work. The server will essentially do all that for you.

Example command line (linux): echo '{client: {clientId: "test", clientVersion: "manual"}, threatInfo: {threatTypes: ["SOCIAL_ENGINEERING", "MALWARE"], platformTypes: ["ANY_PLATFORM"], threatEntryTypes: ["URL"], threatEntries: [{url: "http://testsafebrowsing.appspot[.]com/s/phishing.html"}]}}'| curl -X POST -H "Content-Type: application/json" -d @- https://safebrowsing.googleapis.com/v4/threatMatches:find?key=<YourAPIKey>

Note that I escaped the "phishing" URL again with brackets, so you'll have to remove them for the URL to work.

There's a number of other details related to caching, exponential backoff for retries, request frequency, and more in the documentation: https://developers.google.com/safe-browsing/v4/

[Rajesh Developer]

hi Ben,

i clearly understand these steps. but i had some doubts.

let assume i had a url: sub.domain.maindomain.com/abc/xyz/123/index.html

i hash this url by suggested in the official doc. after hashing i got.

sub.domain.maindomain.com/abc/xyz/123/index.html/

sub.domain.maindomain.com/abc/xyz/123/

sub.domain.maindomain.com/abc/xyz/

sub.domain.maindomain.com/abc/

domain.maindomain.com/abc/xyz/123/index.html

maindomain.com/abc/xyz/123/index.html

...

..

..

...

now assume that the prefix hash of domain.maindomain.com/abc/xyz/ is matched with the rawhash[get from FetchThreatList api].

then i go for verification by FullHash API. the full hash of the domain.maindomain.com/abc/xyz/ perfectly matched with FullHash got from API response.

here is my doubt.

is it ok to say my Main URL: sub.domain.maindomain.com/abc/xyz/123/index.html was marked as Threated by Google if any one of the expression was found to be threated???

i mean one of the expression domain.maindomain.com/abc/xyz/ was found to be threated means it indicates the Main URL: sub.domain.maindomain.com/abc/xyz/123/index.html was threated???

why i'm asking this while hashing the url Google Suggested to create a 5 different string for the Host.

sub.domain.maindomain.com point to some location. it may have a website for support or something else.

domain.maindomain.com point to anothre location and it may also have a webapp like admin portal to control the website.

maindomain.com may also have a website for shopping or something else.

domain.maindomain.com/abc/xyz doesn't needs to points the same location that sub.domain.maindomain.com/abc/xyz/123/index.html points. The paths can be vary.

is it correct to say one of the expression was threated means we can say the main url will also be threated.

but when i go for the Lookup API. The Main URL: sub.domain.maindomain.com/abc/xyz/123/index.html was responsed with threated and under which type it was.

is Lookup Api working as the local database concept[ hash the url match with hash. etc ]. if it so means then how they say the url was threated in Lookup API???

[Ben Sanders]

Note that we try to block the most specific URL possible that provides coverage of the malicious URLs. So we wouldn't block a 'broad' url (covering an entire domain) unless it was necessary (in particular, where every site on a domain is infected, or where the paths are unique per request and can't be blocked individually).

On Fri, Sep 14, 2018 at 7:44 AM 'Alex Wozniak' via Google Safe Browsing API <google-safe-...@googlegroups.com> wrote:

Safe Browsing operates on host-suffix/path-prefix expressions. We will render a verdict on an expression if we believe that any URL "covered" by that expression presents a threat to the user, including additional host prefix components and path suffix components. So if we render a verdict on the expression "foo.com/", we are saying that any hosts and paths on that domain also pose a threat, e.g. http://foo.com/, http://bar.foo.com/baz/index.html. However, if we issue a verdict on the expression "bar.foo.com/baz/index.html", this means that the URL http://foo.com/ is actually safe.

Regarding the Lookup API, we return your input back to you in the response. If we have a host-suffix/path-prefix expression that covers your input, it will be annotated with the corresponding threat.

Hopefully that makes sense. Happy to clarify further if necessary.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.
To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-safe-browsi...@googlegroups.com.
To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

[Rajesh Developer]

Thanks for the quick response Alex and Ben... now i cleared... :)

[Victor Valle]

Update API on Gmail and Google Chrome

[Pulsar9]

How do I do that?

On Sun, Oct 7, 2018 at 2:28 AM Victor Valle <raid...@calebvalle.net.in> wrote:

Update API on Gmail and Google Chrome

[Rajesh Developer]

hi Ben,

I had some doubts about GSB V4. 

how can I use Google Safe Browsing with Threat Entry Type as Executable?

what is it for?

can you suggest any example for me, please?

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Ben Sanders]

You would use the executable threat entry type if you were checking a file (like a Windows executable/program), instead of a URL. In that case, you would set the ThreatEntry digest to a sha256 of the program in question, and the ThreatEntryType to 'EXECUTABLE'

On a linux computer, you can just run 'sha256sum suspiciousFile' to get the sha256 of the file.

Ideally, we would provide blacklist information for any URL associated with malware downloads (and prevent the download that way), but this is another layer of checking/protection.

P.S. As described in the docs, the digest is base64 encoded before being sent to the server, if you are calling our API with JSON (https://developers.google.com/safe-browsing/v4/reference/rest/v4/ThreatEntry)

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsi...@googlegroups.com.

[soltani djallel]

I do not understand nor know what you mean, who you are, and why you send me

The esteemed gentleman:

Garanti sans virus. www.avast.com

‫في الخميس، 15 نوفمبر 2018 في 6:13 م تمت كتابة ما يلي بواسطة ‪'Ben Sanders' via Google Safe Browsing API‬‏ <‪google-safe-...@googlegroups.com‬‏>:‬

Garanti sans virus. www.avast.com

[Rajesh Developer]

thanks ben.. :)

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[นส. ธนวุฒิสุทธิภัทร์]

ออน

ในวันที่ ศ. 16 พ.ย. 2018 12:16 Rajesh Developer <rrraj...@gmail.com เขียนว่า:

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

[Rajesh Developer]

hi Ben,

may I know what will be the min and max cache duration for a Google Safe Browsing API?

sometimes I got 300s sometimes I got 1700s appx. 

the cache duration ranges will very helpful for my application.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Rajeshwaran R]

hi Alex,

may I know what will be the min and max cache duration for a Google Safe Browsing API?

sometimes I got 300s sometimes I got 1700s appx. 

the cache duration ranges will very helpful for my application.

[Rajeshwaran R]

thanks adam.

I had some problem while generating the hash prefix for URL.

some of the generated hash prefixes was not match with the local hash of Google Safe Browsing API V4 but it was suspected to be a threat in Lookup API.

I had added my code below. can you find what I'm doing wrong?

        MessageDigest digest = MessageDigest.getInstance("SHA-256");

        byte[] urlHash = digest.digest(url.getBytes(Charset.forName("ascii")));

        byte[] encodedhash = Arrays.copyOfRange(urlHash, 0, prefixSize);

        String prefixHash = Hex.encodeHexString(encodedhash);

        String fullUrlHash = Hex.encodeHexString(urlHash);

        System.out.println("Prfix hash: "+prefixHash);

        System.out.println("Full Url hash: "+fullUrlHash);

        int count=1;

        for (int i = 0; i < base64Decoded.length; i = i + prefixSize) {

        byte[] array = Arrays.copyOfRange(base64Decoded, i, i + prefixSize);

        String updateHashVal = Hex.encodeHexString(array);

                        if (prefixHash.equals(updateHashVal)) {

System.out.println("***>>>>matched<<<<****");

break;

}

        }

On Thursday, November 29, 2018 at 11:50:07 PM UTC+5:30, Alex Wozniak wrote:

Hi,

We do not have a documented range for the cache duration. In practice you should most commonly see around 5 minutes.

Alex Wozniak | SWE, Safe Browsing | aw...@google.com | 734-748-3306

[นส. ธนวุฒิสุทธิภัทร์]

โอเคคับป๋ม

Aw

ในวันที่ ศ. 21 ธ.ค. 2018 3:21 AM Rajeshwaran R <rrraj...@gmail.com เขียนว่า:

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/xDd3FJYKKOQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--

Alex Wozniak | SWE, Safe Browsing | aw...@google.com | 734-748-3306

--

You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

[Ben Sanders]

Does one of the hash prefixes match? For a given 'threat', usually only one of its prefixes will match something in the prefix lists.

Example: If SafeBrowsing has determined that somesite.com/bad/ is phishing, then if you look up somesite.com/bad/page.html, you will have 3 prefixes to check:

somesite.com/ (no match)

somesite.com/bad/ (match)

somesite.com/bad/page.html (no match)

Then, with the full hash check for somesite.com/bad/ comes back and is a match, you can say that somesite.com/bad/page.html is also bad. The system is designed to minimize the number of hash prefixes that need to be distributed, so it 'blocks' at a coarser granularity if it is determined that all the subpages are infected/malicious.