Modifying google's safe browsing go package to interact with the Update API

[Dylan Warnock]

Are there instructions for configuring the this package to run against the Update API with a persistent database? It would appear doable after looking at safebrowsing.go file but as is the current endpoints seem to only interact with Lookup API. 

[Alex Wozniak]

Hi Dylan,

The package on Github actually implements its own Lookup API on top of an implementation of the Update API. It persists the database to a path defined in the config in safebrowser.go: https://github.com/google/safebrowsing/blob/master/safebrowser.go#L198

Let me know if that makes sense!

Thanks,

Alex

On Tue, Nov 1, 2016 at 2:54 PM Dylan Warnock <warnoc...@gmail.com> wrote:

Are there instructions for configuring the this package to run against the Update API with a persistent database? It would appear doable after looking at safebrowsing.go file but as is the current endpoints seem to only interact with Lookup API. 

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.
To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

[Dylan Warnock]

Hey Alex,

Does that mean it saves the results to a database as they are gathered from the LookUp api? Or does it create a db of hash prefixes as specified in the Update API docs? Would the file referenced at https://github.com/google/safebrowsing/blob/master/safebrowser.go#L198 be similar to rails database.yml file? I'm not super familiar go.

Thanks,

Dylan

On Tuesday, November 1, 2016 at 4:58:06 PM UTC-5, Alex Wozniak wrote:

Hi Dylan,

The package on Github actually implements its own Lookup API on top of an implementation of the Update API. It persists the database to a path defined in the config in safebrowser.go: https://github.com/google/safebrowsing/blob/master/safebrowser.go#L198

Let me know if that makes sense!

Thanks,

Alex

On Tue, Nov 1, 2016 at 2:54 PM Dylan Warnock <warnoc...@gmail.com> wrote:

Are there instructions for configuring the this package to run against the Update API with a persistent database? It would appear doable after looking at safebrowsing.go file but as is the current endpoints seem to only interact with Lookup API. 

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Dylan Warnock]

Is it just going to write the blacklist to a file on the host machine? 

[Alex Wozniak]

Hi Dylan,

It creates a database of hash prefixes using the Update API. The database is stored in a file as shown here: https://github.com/google/safebrowsing/blob/c2df0f33af31470035c6adf857e60670d582374c/database.go#L296. It's basically the gob encoded hash prefixes in gzipped format.

Hopefully that helps!

Alex

On Tue, Nov 1, 2016 at 6:08 PM Dylan Warnock <warnoc...@gmail.com> wrote:

Is it just going to write the blacklist to a file on the host machine? 

On Tuesday, November 1, 2016 at 7:37:30 PM UTC-5, Dylan Warnock wrote:

Hey Alex,

Does that mean it saves the results to a database as they are gathered from the LookUp api? Or does it create a db of hash prefixes as specified in the Update API docs? Would the file referenced at https://github.com/google/safebrowsing/blob/master/safebrowser.go#L198 be similar to rails database.yml file? I'm not super familiar go.

Thanks,

Dylan

On Tuesday, November 1, 2016 at 4:58:06 PM UTC-5, Alex Wozniak wrote:

Hi Dylan,

The package on Github actually implements its own Lookup API on top of an implementation of the Update API. It persists the database to a path defined in the config in safebrowser.go: https://github.com/google/safebrowsing/blob/master/safebrowser.go#L198

Let me know if that makes sense!

Thanks,

Alex

On Tue, Nov 1, 2016 at 2:54 PM Dylan Warnock <warnoc...@gmail.com> wrote:

Are there instructions for configuring the this package to run against the Update API with a persistent database? It would appear doable after looking at safebrowsing.go file but as is the current endpoints seem to only interact with Lookup API. 

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

[Dylan Warnock]

Hey Alex!

Can the file just be set to some text file here https://github.com/google/safebrowsing/blob/c2df0f33af31470035c6adf857e60670d582374c/safebrowser.go#L224 by adding the line c.DBPath = "prefixes.txt"?

On Wednesday, November 2, 2016 at 10:52:38 AM UTC-5, Alex Wozniak wrote:

Hi Dylan,

It creates a database of hash prefixes using the Update API. The database is stored in a file as shown here: https://github.com/google/safebrowsing/blob/c2df0f33af31470035c6adf857e60670d582374c/database.go#L296. It's basically the gob encoded hash prefixes in gzipped format.

Hopefully that helps!

Alex

On Tue, Nov 1, 2016 at 6:08 PM Dylan Warnock <warnoc...@gmail.com> wrote:

Is it just going to write the blacklist to a file on the host machine? 

On Tuesday, November 1, 2016 at 7:37:30 PM UTC-5, Dylan Warnock wrote:

Hey Alex,

Does that mean it saves the results to a database as they are gathered from the LookUp api? Or does it create a db of hash prefixes as specified in the Update API docs? Would the file referenced at https://github.com/google/safebrowsing/blob/master/safebrowser.go#L198 be similar to rails database.yml file? I'm not super familiar go.

Thanks,

Dylan

On Tuesday, November 1, 2016 at 4:58:06 PM UTC-5, Alex Wozniak wrote:

Hi Dylan,

The package on Github actually implements its own Lookup API on top of an implementation of the Update API. It persists the database to a path defined in the config in safebrowser.go: https://github.com/google/safebrowsing/blob/master/safebrowser.go#L198

Let me know if that makes sense!

Thanks,

Alex

On Tue, Nov 1, 2016 at 2:54 PM Dylan Warnock <warnoc...@gmail.com> wrote:

Are there instructions for configuring the this package to run against the Update API with a persistent database? It would appear doable after looking at safebrowsing.go file but as is the current endpoints seem to only interact with Lookup API. 

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Alex Wozniak]

Hi Dylan,

The path is set via flag: https://github.com/google/safebrowsing/blob/c2df0f33af31470035c6adf857e60670d582374c/cmd/sbserver/main.go#L225

Phishing is collected by default; it's also known as "social engineering": https://github.com/google/safebrowsing/blob/c2df0f33af31470035c6adf857e60670d582374c/cmd/sbserver/main.go#L137

-Alex

On Fri, Nov 4, 2016 at 11:07 AM Dylan Warnock <warnoc...@gmail.com> wrote:

So I believe this is where you set the file path: https://github.com/google/safebrowsing/blob/c2df0f33af31470035c6adf857e60670d582374c/cmd/sbserver/main.go#L225 ?

How do I go about setting the ThreatLists values? It appears by default phishing pages are not returned

Dylan

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

[Dylan Warnock]

Now I'm just trying to understand the code. Where exactly does the db file get read? Also how efficient is the reading of this file? Does a gob stream allow for indexing?

Thanks for all the help!

Dylan

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Alex Wozniak]

Hi Dylan,

The db is loaded from file in Init: https://github.com/google/safebrowsing/blob/master/database.go#L104

It might be helpful to understand your use case a bit more. I'd like to understand why you're interested in indexing.

-Alex

Dylan

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

[Dylan Warnock]

The goal is to prevent users from posting URLs google deems as malicious to keep our pages from being docked in google search results. The number of users is high so the number of urls to validate could be high as well. This client will be a micro service receiving requests from a number of different applications. I'm trying to understand how the pieces of this go client work together (caching, file read/write) in order to see if it should be tweak to speed of checking of prefixes. 

Dylan

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Alex Wozniak]

Thanks Dylan. You can use cmd/sbserver/main.go as your microservice, which exposes an HTTP JSON API. Here's example usage: https://github.com/google/safebrowsing/blob/master/cmd/sbserver/main.go#L82

Since this client is built on top of the Update API, it's built for high frequency checks. Do note that each instance of the server consults its own cache. This should be fine, though, since prefix collisions are rare; worst case your microservice will make a hash request on each server. This should fit your needs out of the box :)

Dylan

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

[Dylan Warnock]

One last question. Have you had any luck setting up this package inside of a docker container? 

[Alex Wozniak]

Hi Dylan, unfortunately I don't have much experience with docker containers.

--

[Dylan Warnock]

Why keep the prefixes in a file instead of memory? I think this is my last question.

On Tuesday, November 8, 2016 at 11:37:25 AM UTC-6, Alex Wozniak wrote:

Hi Dylan, unfortunately I don't have much experience with docker containers.

On Tue, Nov 8, 2016 at 7:08 AM Dylan Warnock <warnoc...@gmail.com> wrote:

One last question. Have you had any luck setting up this package inside of a docker container? 

On Tuesday, November 1, 2016 at 4:54:39 PM UTC-5, Dylan Warnock wrote:

Are there instructions for configuring the this package to run against the Update API with a persistent database? It would appear doable after looking at safebrowsing.go file but as is the current endpoints seem to only interact with Lookup API. 

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[Alex Wozniak]

Hey Dylan,

An in-memory store is initialized via file, if available (https://github.com/google/safebrowsing/blob/c2df0f33af31470035c6adf857e60670d582374c/database.go#L104). The hashes are also persisted to disk so the client doesn't have to do a full update on every start up.

Hopefully that helps!

On Tue, Nov 8, 2016 at 12:38 PM Dylan Warnock <warnoc...@gmail.com> wrote:

Why keep the prefixes in a file instead of memory? I think this is my last question.

On Tuesday, November 8, 2016 at 11:37:25 AM UTC-6, Alex Wozniak wrote:

Hi Dylan, unfortunately I don't have much experience with docker containers.

On Tue, Nov 8, 2016 at 7:08 AM Dylan Warnock <warnoc...@gmail.com> wrote:

One last question. Have you had any luck setting up this package inside of a docker container? 

On Tuesday, November 1, 2016 at 4:54:39 PM UTC-5, Dylan Warnock wrote:

Are there instructions for configuring the this package to run against the Update API with a persistent database? It would appear doable after looking at safebrowsing.go file but as is the current endpoints seem to only interact with Lookup API. 

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

[Dylan Warnock]

Hey Alex,

On start up I'm running into the following 'database.go:106: load failure: EOF' and my db file isn't being populated.

My file lives here /src/github.com/google/safebrowsing/cmd/prefixes.txt. Any idea what would cause this?

Thanks again,

Dylan

On Tuesday, November 8, 2016 at 3:00:19 PM UTC-6, Alex Wozniak wrote:

Hey Dylan,

An in-memory store is initialized via file, if available (https://github.com/google/safebrowsing/blob/c2df0f33af31470035c6adf857e60670d582374c/database.go#L104). The hashes are also persisted to disk so the client doesn't have to do a full update on every start up.

Hopefully that helps!

On Tue, Nov 8, 2016 at 12:38 PM Dylan Warnock <warnoc...@gmail.com> wrote:

Why keep the prefixes in a file instead of memory? I think this is my last question.

On Tuesday, November 8, 2016 at 11:37:25 AM UTC-6, Alex Wozniak wrote:

Hi Dylan, unfortunately I don't have much experience with docker containers.

On Tue, Nov 8, 2016 at 7:08 AM Dylan Warnock <warnoc...@gmail.com> wrote:

One last question. Have you had any luck setting up this package inside of a docker container? 

On Tuesday, November 1, 2016 at 4:54:39 PM UTC-5, Dylan Warnock wrote:

Are there instructions for configuring the this package to run against the Update API with a persistent database? It would appear doable after looking at safebrowsing.go file but as is the current endpoints seem to only interact with Lookup API. 

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsing-api+unsub...@googlegroups.com.

[joe...@google.com]

Hey Dylan,

Did you manually create the prefixes.txt file? Or was it generated by the safebrowsing package? Looking at the code, it seems that the most probable way io.EOF gets returned is if the prefixes.txt is an empty file.

JT

[Dylan Warnock]

I manually created the file. The file is empty to begin with but after an update the file should be populated right?

[Priyanka Tanvani]

Were you able to populate the txt file with the hashes information ?

[Dylan Warnock]

I was! Is it possible to use this API with some kind of service account rather than individual developer credentials? 

--

You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/qBGtM6wlilw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsub...@googlegroups.com.
To post to this group, send email to google-safe-browsing-api@googlegroups.com.

Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

--

Dylan Warnock

University of Texas at Austin, 15

BBA, Management Information Systems

BSA, Computer Science