DEADLINE_EXCEEDED exception

[Priyanka Tanvani]

Hi, 

I'm getting the following error : 

com.google.api.client.googleapis.json.GoogleJsonResponseException: 504 Gateway Time-out
{
  "code" : 504,
  "errors" : [ {
    "domain" : "global",
    "message" : "Deadline expired before operation could complete.",
    "reason" : "backendError"
  } ],
  "message" : "Deadline expired before operation could complete.",
  "status" : "DEADLINE_EXCEEDED"
}

But, I'm sure that my daily quota has not exceeded.

By looking at the console, i can see that the number of requests per second is not more than the default quota (3000 req/100sec).

What other reason can be there for the above error ?

[Alex Wozniak]

Thanks for your message. This means that the backend for whatever reason was unable to service your request in a timely fashion. These errors should be extremely rare. Are you seeing them consistently?

[Priyanka Tanvani]

Hi Alex, 

Yes, i'm seeing these errors consistently. 
When my concurrency level is <= 5 then this errors is not coming but on increasing the level of concurrency above 5 i'm seeing it consistently.
Also, i cannot keep the concurrency level to 5 since that is taking a lot of time.

Thanks.

-Priyanka.

[Alex Wozniak]

Could you send me an example request (with API key redacted?)

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.
To post to this group, send email to google-safe-...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

[Priyanka Tanvani]

Hi Alex, 

I'm using the google  java client for google safe browsing : https://developers.google.com/resources/api-libraries/documentation/safebrowsing/v4/java/latest/ 
So, i'm not handling the http requests and responses at my end. 
In each request to find fullHashes, i'm sending 5000 prefixes.

Also, I've registered a request for increasing my usage quota and i've also sent a mail to 

antiphish-ma...@google.com for the same, but i've not got a reply yet (it has been more than 10 days now).

-Priyanka

[Priyanka Tanvani]

Hi Alex, 

I've attached the file which contains the sample data for one find fullHashes request.

-Priyanka

[Alex Wozniak]

Hi Priyanka,

May I ask why you're sending 5000 hash prefixes? You should only send a hash request when there requested prefix(es) are present in the set of hashes served by the threatListUpdates:fetch method.

If you would like to use the Lookup API, you can send a maximum of 500 URLs per request.

Let me know if that helps!

Thanks,

Alex

--

[Priyanka Tanvani]

Hi Alex, 

To reduce the latency of my application, i have to cache the fullHashes of urls present in the threat lists. That's why i'm periodically sending all prefixes present in the threatlists by batching 5k prefixes in a single request.

Thanks.

-Priyanka

[Alex Wozniak]

Hi Priyanka,

Some additional details about what you're trying to accomplish would be helpful. From what you've described, it sounds like you're trying to capture the entire list of full hashes from our server. If so, our API is not intended to be used in this manner.

Thanks,

Alex

--

[Priyanka Tanvani]

Hi Alex, 

Thanks for the quick reply. Yes, i was trying to do that only. 
Actually, i want to be able to check for each url (that i'm redirecting my user to) whether it is present in the threat list or not. So, i was thinking if i can cache the threat url fullhashes, it would be helpful.

Also, i wanted to ask what is the maximum number of prefixes that i can query for fullHashes in a single findFullHashes request ?

-Priyanka.

[Alex Wozniak]

Hi Priyanka,

You should implement caching according to the documentation here: https://developers.google.com/safe-browsing/v4/caching

I'd encourage you to use the Github repo or Chromium as references for implementing the cache.

Based on your use case, I don't quite follow why you're sending large batches of hash prefixes. I would just send a hash request for each URL that a user is attempting to visit that matches a hash prefix from the local database that's periodically updated using the threatListUpdates:fetch endpoint.

Hopefully that makes sense!

Thanks,

Alex

--