Constraints of the Socket API

[Jeff Schnitzer]

I'm calling a third-party HTTP service from a task that sometimes takes more than 60s to respond. Since the URLFetchService has a hard limit of 60s, I thought I might be able to work around this with the Socket API - there do not appear to be any documented time limits other than the 2m idle.

I wired in the Apache HttpClient... and I'm still getting an exception at 60s, even though I explicitly set the timeout to 118s:

Caused by: java.net.SocketException: Socket operation timed out: The API call remote_socket.Receive() took too long to respond and was cancelled. at com.google.appengine.api.socket.SocketApiHelper.makeSyncCall(SocketApiHelper.java:87) at com.google.appengine.api.socket.AppEngineSocketImpl.receive(AppEngineSocketImpl.java:750) at com.google.appengine.api.socket.AppEngineSocketInputStream.read(AppEngineSocketInputStream.java:36) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:944) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:901) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)

Question #1: Is there a limit missing from the documentation?

Question #2: Will I have any success with the NIO version of httpclient? I'm guessing not.

Question #3: Any suggestions for ways to work around this? It's part of an already-too-complicated multistep cascade of transactional task operations. This will be very hard to split up in any reasonable way.

Thanks,

Jeff

[Miguel Vitorino]

Hi,

Have you thought about calling it from a push task queue?

Task queue initiated requests have a deadline of 10 mins

Miguel

[Miguel Vitorino]

Hi sorry, hasty response. You obviously tried that and doesn't work. We do use socket api for periods longer than 2 mins but in Go. Can't help in java.

[Nick (Cloud Platform Support)]

Question #1: Is there a limit missing from the documentation?

It appears as though this is a possibility. As you point out the only limit mentioned in the docs is 2 minutes, and you're below that amount of time when you get the exception. It's possible the limit you're reaching is unrelated to setSoTimeout(), or that setSoTimeout() is ineffective.

 

Overall, I think this issue should be forwarded to the public issue tracker with a reproducing example snippet or entire app archive. This way, if the behaviour is not correct, it can be quickly corrected. 

Question #2: Will I have any success with the NIO version of httpclient? I'm guessing not.

The docs state that "There is currently no support for sockets via java.nio.SocketChannel or otherjava.nio classes."

Question #3: Any suggestions for ways to work around this? It's part of an already-too-complicated multistep cascade of transactional task operations. This will be very hard to split up in any reasonable way.

From my view, your options are to either A) figure out a fix (or wait for a fix) for this socket issue, B) make a feature request for longer deadlines on UrlFetch, or C) use a Managed VM instance which will allow network connections not mediated through the UrlFetch service, allowing longer timeouts. This assumes that "fix the HTTP endpoint which is taking more than 60 seconds to respond" is off the table.