Socket Timeout errors on URLFetch calls

[Hugo Visser]

My app makes heavy use of URLFetch to get content and I'm seeing socket time out exceptions on App Engine standard (Java). Am I the only one experiencing this? Would ditching URLFetch for normal url connections fix this?

Thanks,

Hugo

[Joshua Smith]

URLFetch is a little flaky in app engine. Always has been, kind of like mail. I suspect rolling your own fetch using URL connections wouldn’t help at all.

Over the many years I’ve been using GAE, I’ve noticed that both mail and url fetch errors tend to cluster. Like storms that pass through, causing lots of errors, and then move on and everything is good for a few weeks.

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7c5a6697-9a73-4a83-8ce6-8b5953a5b69a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rokesh]

Does this help? https://cloud.google.com/appengine/docs/standard/java/issue-requests

[George (Cloud Platform Support)]

Hi Hugo, 

What type of time out exceptions do you refer to? Which errors are the most frequently encountered? You are surely not the only one facing this situation. How would you like to see URLFetch replaced? Do you use Java 7 or 8? Which features described on the "Issuing HTTP(S) Requests" documentation page do you actually use? 

[Hugo Visser]

I'm literally getting java.net.SocketTimeoutException, meaning that the connection to the host times out. I have billing enabled so that can't be the cause, but maybe it's related.

I thought the error was because that particular host was having troubles, but I'm seeing it on other hosts too now. I'm did a quick change to replace my URLFetch calls with normal socket based http calls through OkHttp, and that seems to be better, though it's to soon to really tell.

I'm using Java 8, but since this is an older app it's still using the URLFetchService as that was the only way to do a request in the past. I don't really need it to be replaced, I just need it to work like it used to and not fail at random :-)

[Hugo Visser]

Not using URLFetchService seems to have worked, I'm not seeing any SocketTimeoutExceptions anymore since I've deployed a version of my app that uses OkHttp to replace the URLFetchService. Of course that might be a coincidence but the errors that have been occurring throughout the day are gone now. I actually implemented the URLFetchService interface for my OkHttp implementation, which I'm happy to share if someone needs it too.

[George (Cloud Platform Support)]

In the "Issuing an HTTP request" sub-chapter of the "Issuing HTTP(S) Requests" document, this possibility is indicated: "You can also use the third-party requests library as long as you configure it to use URLFetch." There is no doubt that other people will be interested in details about your successful implementation. 

[Hugo Visser]

I think you are misunderstanding what I'm saying. On the Java 8 runtime you can do without URLFetch because the limitations that were in effect on the older runtimes are lifted. Because of that I can work around it by using any other http library, which is what the documentation is referring to. I'm glad that this is possible but it's still a workaround.

From the errors I was encountering, I still believe the actual URLFetch service has an issue and has flaky behaviour (as Joshua also mentions), which should be resolved by Google as it may affect other apps using that service, especially older App Engine apps migrating from the previous runtime(s). 

[George (Cloud Platform Support)]

Are the URLFetch errors associated with a certain situation, or at least a certain period in time? To report an issue, this has to be reproducible or, at least, associated with certain parameters, such as time. There is little chance to address an issue one can't reproduce reliably.