GSBv2 lookups returning 403 errors

[Deepak]

Hi,

I have been observing a lot of GSBv2 lookups failing today with 403
errors. I have a updated local cache and am using a single API key.
The percentage of lookups that failed today with 403 errors is ~10%

Any ideas why this may be happening ?

-Deepak

[Patrick Kelley]

I had this problem as well.

When I was checking hash prefixes against google's full hashes, google was returning a 204 (no content). If I asked the same request too many times, I started getting 403's.

Make sure your code is storing some local stub after receiving a 204 to ensure you do not spam google's servers. Keep track of all outbound connections to google.

Sent from my iPhone

> --
> You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.
> To post to this group, send email to google-safe-...@googlegroups.com.
> To unsubscribe from this group, send email to google-safe-browsi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-safe-browsing-api?hl=en.
>

[Deepak]

Thanks for the reply Patrick!

Do you remember how much time you cached this information ? Also, is
there any reference to this in the GSBv2 API Doc because I couldn't
find any information on this.

-Deepak

On Dec 8, 6:20 pm, Patrick Kelley <peanutbutterkrac...@gmail.com>
wrote:

> I had this problem as well.
>
> When I was checking hash prefixes against google's full hashes, google was returning a 204 (no content). If I asked the same request too many times, I started getting 403's.
>
> Make sure your code is storing some local stub after receiving a 204 to ensure you do not spam google's servers.  Keep track of all outbound connections to google.
>
> Sent from my iPhone
>

[Patrick Kelley]

I picked an arbitrary timeout of 5 minutes. I don't believe this scenario is covered in the API documentation.

How many URLs are you scanning per hour?

Sent from my iPhone

[Garrett Casto]

You should only be getting 403's if you are sending more than ~10qps. Are you sending more requests than that? We ask that you inform us if you are going to send us more than 10,000 qps for this reason, so we exempt you if necessary. 

As for caching, I think that the reason we don't have this covered in the normal documentation is that different clients have done different things, and we haven't decided what we think should be the standard. The easiest thing to do is what Chrome does, which is cache until the next update. If you are getting throttled just because of multiple requests, this will probably fix the issue.

Garrett

[ameshkov]

Hello Garret,

I have informed you about our API-key and users count in april.
There was no answer, so I don't know if our API-key is provisioned for
additioanl users or not.
Some days ago we start getting 403 errors.

What should I do know?

On 9 дек, 22:58, Garrett Casto <gca...@google.com> wrote:
> You should only be getting 403's if you are sending more than ~10qps. Are
> you sending more requests than that? We ask that you inform us if you are
> going to send us more than 10,000 qps for this reason, so we exempt you if
> necessary.
>
> As for caching, I think that the reason we don't have this covered in the
> normal documentation is that different clients have done different things,
> and we haven't decided what we think should be the standard. The easiest
> thing to do is what Chrome does, which is cache until the next update. If
> you are getting throttled just because of multiple requests, this will
> probably fix the issue.
>
> Garrett
>
> On Thu, Dec 8, 2011 at 8:29 PM, Patrick Kelley <
>
>
>
>
>
>
>
> peanutbutterkrac...@gmail.com> wrote:
> > I picked an arbitrary timeout of 5 minutes. I don't believe this scenario
> > is  covered in the API documentation.
>
> > How many URLs are you scanning per hour?
>
> > Sent from my iPhone
>

[ameshkov]

By informing you I mean that I've send a message to "antiphish-malware-
cap...@google.com".

[Garrett Casto]

I'll look into this and respond to you off list.

[EarlyAdopter]

Sort of a minor point here, but regarding the HTTP 403 errors I'm
getting when testing with a bad API key... The docs say it should be
401 (no mention of 403).

Obviously, no biggie. 401 and 403 are pretty similar and it's of
course easy to test for both.

-EA

[EarlyAdopter]

Just a small FYI here... I've noticed that the bad-API-key HTTP 403
error is documented in the Safe Browsing Lookup API Developer's Guide
as 401 instead of 403. Not a big deal of course, as they're pretty
similar and can be handled identically.

-EA

[FirefighterBlu3]

Garrett, over the last several days i've noticed all of my MX servers
have transitioned from working to 404, and now to 403 with "We're
sorry... ... but your computer or network may be sending automated
queries. To protect our users, we can't process your request right
now." for my 30 minute updates for the GSB spamassassin plugin. They
all have different API keys as needed save one which i'm cloning right
at the moment.

As for QPS, it's -possible- that the main spam checking server surges
to just over 13 (when google decides to unload 200 emails at once to
some users of mine), but the average inbound including whitelisted
emails is .06 to .8 emails per second and most of them don't have URLs
in them. But, that should be irrelevant, as the plugin downloads the
cache once per 30 minutes.

Did something change recently?

-d

On Dec 9 2011, 1:58 pm, Garrett Casto <gca...@google.com> wrote:
> You should only be getting 403's if you are sending more than ~10qps. Are
> you sending more requests than that? We ask that you inform us if you are
> going to send us more than 10,000 qps for this reason, so we exempt you if
> necessary.
>
> As for caching, I think that the reason we don't have this covered in the
> normal documentation is that different clients have done different things,
> and we haven't decided what we think should be the standard. The easiest
> thing to do is what Chrome does, which is cache until the next update. If
> you are getting throttled just because of multiple requests, this will
> probably fix the issue.
>
> Garrett
>
> On Thu, Dec 8, 2011 at 8:29 PM, Patrick Kelley <
>
>
>
>
>
>
>
> peanutbutterkrac...@gmail.com> wrote:
> > I picked an arbitrary timeout of 5 minutes. I don't believe this scenario
> > is  covered in the API documentation.
>
> > How many URLs are you scanning per hour?
>
> > Sent from my iPhone
>

[Garrett Casto]

Not that I know of. I'll see if I can take a look later, but it might
be a day or two before I can get to it.

[FirefighterBlu3]

would it help if i post one of my server's api key and IP etc?

-david

On Jan 5, 5:14 pm, Garrett Casto <gca...@google.com> wrote:
> Not that I know of. I'll see if I can take a look later, but it might
> be a day or two before I can get to it.
>
> On Thu, Jan 5, 2012 at 10:11 AM, FirefighterBlu3
>