Unreliable fetching from BigQuery

148 views
Skip to first unread message

Jeff Schnitzer

unread,
Aug 20, 2015, 12:11:51 PM8/20/15
to Google App Engine
I'm getting a lot of "Could not fetch URL" errors from BigQuery. I'm not driving significant amounts of traffic to it (yet), but I get one of these maybe every 50 or so queries:


I'm using the standard google-published bigquery client library (Java). Any idea what is wrong? I don't have problems fetching from any other service.

I'm ramping up traffic next week and this is going to start happening a lot.... 

Thanks,
Jeff

Patrice (Cloud Platform Support)

unread,
Aug 20, 2015, 4:43:56 PM8/20/15
to Google App Engine, je...@infohazard.org
Hi Jeff,

Just a quick check to be sure : Are you hammering the server with those requests? Just to make sure there's not a burst limit holding you back. 

In general, whenever you make those requests, are you sure the underlying query finished running? 

Normally I would suggest you give me an example of a jobID that returned this with a timestamp and relevant data, and then send this to the BQ team. But the BQ team is highly active on StackOverflow, and do these checks if all the relevant information is provided. 

Cheers!

Jeff Schnitzer

unread,
Aug 25, 2015, 1:45:22 AM8/25/15
to Google App Engine
Sorry about the late response... it's been an exciting week :)

I'm not hammering BQ yet. Probably a few dozen requests a minute at peak.

The problem does not appear to be in BQ itself; I'm not even getting a jobid back. It's also not just queries; I also get errors periodically from insertAll and creating tables. It's the same opaque "Could not fetch URL" - not timeout, nothing that explains the underlying cause. Here's a representative stacktrace:


at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException (URLFetchServiceImpl.java:140)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch (URLFetchServiceImpl.java:45)
at com.google.api.client.extensions.appengine.http.UrlFetchRequest.execute (UrlFetchRequest.java:74)
at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:972)




"Could not fetch URL" sounds like a DNS failure or something like that. But there's no way that could happen for www.googleapis.com, right?

Thanks,
Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/19ac2e15-72b3-4189-ac26-a09674bcf8fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ryan (Cloud Platform Support)

unread,
Aug 25, 2015, 9:32:30 AM8/25/15
to Google App Engine, je...@infohazard.org
Salutations Jeff,

Can you send me the code you are using and full stack traces with time stamps (more recent ones the better and multiple ones for comparison).

Thanks


On Thursday, August 20, 2015 at 12:11:51 PM UTC-4, Jeff Schnitzer wrote:

Jeff Schnitzer

unread,
Aug 25, 2015, 1:51:43 PM8/25/15
to Google App Engine
Thanks for looking into this. Here's a gist with some full stacktraces and timestamps, plus the relevant code:


One is a deferred task, one is a cron, but I also have failures with normal user-facing query requests. I can dig some of those up too if it helps.

Thanks,
Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a15cec2d-a66d-496b-9f99-91b5b3829322%40googlegroups.com.

Ryan (Cloud Platform Support)

unread,
Aug 25, 2015, 2:11:36 PM8/25/15
to Google App Engine, je...@infohazard.org
More the better. What is your app id?

On Tuesday, August 25, 2015 at 1:51:43 PM UTC-4, Jeff Schnitzer wrote:
Thanks for looking into this. Here's a gist with some full stacktraces and timestamps, plus the relevant code:


One is a deferred task, one is a cron, but I also have failures with normal user-facing query requests. I can dig some of those up too if it helps.

Thanks,
Jeff
On Tue, Aug 25, 2015 at 6:32 AM, Ryan (Cloud Platform Support) <rbru...@google.com> wrote:
Salutations Jeff,

Can you send me the code you are using and full stack traces with time stamps (more recent ones the better and multiple ones for comparison).

Thanks

On Thursday, August 20, 2015 at 12:11:51 PM UTC-4, Jeff Schnitzer wrote:
I'm getting a lot of "Could not fetch URL" errors from BigQuery. I'm not driving significant amounts of traffic to it (yet), but I get one of these maybe every 50 or so queries:


I'm using the standard google-published bigquery client library (Java). Any idea what is wrong? I don't have problems fetching from any other service.

I'm ramping up traffic next week and this is going to start happening a lot.... 

Thanks,
Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Jeff Schnitzer

unread,
Aug 25, 2015, 3:22:15 PM8/25/15
to Google App Engine
gearlaunch-hub

I'll add some more to the gist.

Thanks,
Jeff

On Tue, Aug 25, 2015 at 11:11 AM, Ryan (Cloud Platform Support) <rbru...@google.com> wrote:
More the better. What is your app id?

On Tuesday, August 25, 2015 at 1:51:43 PM UTC-4, Jeff Schnitzer wrote:
Thanks for looking into this. Here's a gist with some full stacktraces and timestamps, plus the relevant code:


One is a deferred task, one is a cron, but I also have failures with normal user-facing query requests. I can dig some of those up too if it helps.

Thanks,
Jeff
On Tue, Aug 25, 2015 at 6:32 AM, Ryan (Cloud Platform Support) <rbru...@google.com> wrote:
Salutations Jeff,

Can you send me the code you are using and full stack traces with time stamps (more recent ones the better and multiple ones for comparison).

Thanks

On Thursday, August 20, 2015 at 12:11:51 PM UTC-4, Jeff Schnitzer wrote:
I'm getting a lot of "Could not fetch URL" errors from BigQuery. I'm not driving significant amounts of traffic to it (yet), but I get one of these maybe every 50 or so queries:


I'm using the standard google-published bigquery client library (Java). Any idea what is wrong? I don't have problems fetching from any other service.

I'm ramping up traffic next week and this is going to start happening a lot.... 

Thanks,
Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7ff2bc22-d8bf-4795-9a08-44b7e83a0c56%40googlegroups.com.

Jeff Schnitzer

unread,
Aug 25, 2015, 3:52:28 PM8/25/15
to Google App Engine
I put the query stacktrace and code in there, plus a few more timestamps of occurrences that happened today/yesterday.

Thanks,
Jeff

Patrice (Cloud Platform Support)

unread,
Aug 26, 2015, 11:25:59 AM8/26/15
to Google App Engine, je...@infohazard.org
Hey Jeff,

Thank you for that gist, it definitely contains everything we can possibly be needing :).

Just quickly looking at your "Insert All", I think there might be a way to completely get out of the "URL not find" bind. I can't help but notice you try the insert once, with no fallback. Is there some kind of exponential backoff in another layer of your application? Because using an exponential backoff to that insertAll (and probably to all the other calls that sometimes throw similar errors) should take you out of this.

I will still investigate to try and see what I can gather as to the root cause of this. As soon as I find anything, I'll update the thread here.

Cheers!
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Jeff Schnitzer

unread,
Aug 26, 2015, 6:17:27 PM8/26/15
to Google App Engine
The inserts are all running from a task queue with the default backoff behavior, so I should be ok there. All the inserts seem to work on retry - I'm not losing data - I just get errors in my logs and a nastygram from Rollbar.

Failing queries are a bigger issue because users see that.

Thanks,
Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/403de718-0190-425a-8d5c-cca90325e4bc%40googlegroups.com.

Patrice (Cloud Platform Support)

unread,
Aug 27, 2015, 10:29:53 AM8/27/15
to Google App Engine, je...@infohazard.org
Hey Jeff!

At least you're not losing data because of this. 

Normally you can do the same for your queries though. We always suggest to code defensively, just in case there is an intermittent issue, or a network issue, or maintenance on your instance, or any number of things. 

I'm still trying to understand if what you're seeing is expected or higher than the norm. As soon as I get anything I will let you know.

Cheers
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Patrice (Cloud Platform Support)

unread,
Sep 1, 2015, 9:52:46 AM9/1/15
to Google App Engine, je...@infohazard.org
Hi again Jeff,

So, looking back into the issue and your code, the only thing I can think of right now is that this may not be a BQ issue, but a URLfetch one. Are you sure your URLFetch has a proper timeout setup? This may not be simply because your timeout is too short, and sometimes the server takes a little bit longer than expected to respond, and that throws everything out.

I don't see in your samples how you setup your timeout for URLFetch. Has it been set elsewhere, or do you use the default one? The default being only 5 seconds, its very possible this is your issue here.

Let me know if setting your deadline OR your default deadline helps :).

Cheers!

Jeff Schnitzer

unread,
Sep 2, 2015, 3:59:25 AM9/2/15
to Patrice (Cloud Platform Support), Google App Engine
I also expect that it is a URLFetch issue and not a BQ issue. If it is a timeout, it is not a typical timeout - the error is not SocketTimeoutException. I'm using the standard Google BQ library with the UrlFetchTransport. There does not appear to be a general way of setting a timeout in this library, although I do set a timeout for queries of 30,000 milliseconds and still see this error. I have not yet tried to debug into the Google BQ client library to verify whether it actually sets the timeout value.

I don't expect that it's a timeout issue - I've seen those, and they are more obvious than "Could not fetch URL".

Jeff

Cheers!
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Patrice (Cloud Platform Support)

unread,
Sep 2, 2015, 5:00:47 PM9/2/15
to Google App Engine, pvout...@google.com, je...@infohazard.org
Hey Jeff,

Timeout for queries is not the same as the timeout for the URLFetch. Since your call is made from within the BQ client lib, it's possible the normal error is mangled up there.

Now it may be a bit cumbersome to write, but maybe using URLFetch directly to call the BQ urls and setting a longer timeout there might solve the issue? At the very least, since the call will be a purely URLFetch one, the error message might contain more information than it currently does.

Cheers
Cheers!
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages