When executed rarely, every other request results in IOException

[crosser]

I am writing an app for Android FroYo, consequently it uses Apache HttpClient for the API requests (I call Latitude and Calendar, using OAuth2.0). The library version is 1.6.0-beta.

When subsequent calls are done once a minute, it works. When the idle interval between the calls is 20 min, every second call results in

java.io.IOException: SSL shutdown failed: I/O error during system call, Broken pipe

The exception happens when the new call is initiated, but it is in close():

W/System.err(  265): java.io.IOException: SSL shutdown failed: I/O error during system call, Broken pipe
W/System.err(  265): java.io.IOException: SSL shutdown failed: I/O error during system call, Broken pipe
W/System.err(  265):    at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl.nativeclose(Native Method)
W/System.err(  265):    at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl.close(OpenSSLSocketImpl.java:958)
W/System.err(  265):    at org.apache.http.impl.SocketHttpClientConnection.close(SocketHttpClientConnection.java:205)
W/System.err(  265):    at org.apache.http.impl.conn.DefaultClientConnection.close(DefaultClientConnection.java:161)
W/System.err(  265):    at org.apache.http.impl.conn.AbstractPooledConnAdapter.close(AbstractPooledConnAdapter.java:158)

I am looking into this document: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html and there is this paragraph:

CoreConnectionPNames.STALE_CONNECTION_CHECK='http.connection.stalecheck':  determines whether stale connection check is to be used. Disabling stale connection check may result in a noticeable performance improvement (the check can cause up to 30 millisecond overhead per request) at the risk of getting an I/O error when executing a request over a connection that has been closed at the server side. This parameter expects a value of type java.lang.Boolean. For performance critical operations the check should be disabled. If this parameter is not set, the stale connection check will be performed before each request execution.

It looks like my case: the server presumably closes the connection after timeout, and on the next attempt to use it, the client tries to reuse it, sees that it is dead, tries to close it and reopen a new one, but on close, the SSL exception is uncaught (which probably it should be ignored).

Does my explanation make sense? How should I deal with this problem?

Thanks,

Eugene

[crosser]

After some investigation, this is my current theory:

Presumably when it tries to execute a new request, the Apache HttpClient connection manager notices that connection(s) are stale and tries to close them before re-opening to execute the current request.

SocketHttpClientConnection.close looks like this:

   public void close() throws IOException {
       if (!this.open) {
           return;
       }
       this.open = false;
       Socket sock = this.socket;
       try {
           doFlush();
           try {
               try {
                   sock.shutdownOutput();
               } catch (IOException ignore) {
               }
               try {
                   sock.shutdownInput();
               } catch (IOException ignore) {
               }
           } catch (UnsupportedOperationException ignore) {
               // if one isn't supported, the other one isn't either
           }
       } finally {
           sock.close();
       }
   }

It does catch and ignore IOException from shutdown() calls. Having done shutdown for both read and write, the assumption is that sock.close() will never throw a spurious exception, even if the other side is long gone.

But in our case the socket is not a TCP socket, but an SSL one. Presumably, socket close() is executed by

org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl.close(OpenSSLSocketImpl.java:958)

I cannot find the source code of this module anywhere (there is no such file in the Apache Harmony tree). Anyway, my hypothesis is that while regular TCP socket close() succeeds when the peer is already gone, the implementation of close() in this class tries to close the SSL session prior to closing the TCP socket, and that involves sending a message over the underlying TCP connection. Possibly the IOException that happens if the peer has already closed the TCP connection should have been ignored in this case. And possibly it is not. Therefore it is propagated upstream, and nobody upstream expects that close would throw an exception in such a trivial case.

Any thoughts if this is right, and what would be the best way around it?

Thanks,

Eugene

[Alex Cohn]

Have you tried the new 1.7 beta?

[Yaniv Inbar (יניב ענבר)‎]

Thanks for reporting the problem.  We have not had a chance to investigate this issue yet, so it is unlikely to behave differently with the new version 1.7.  Just for clarification, the issue you encountered actually happens on an Android device with FroYo SDK or with a later version?

Yaniv Inbar
Senior Software Engineer
Google Inc.

[crosser]

I have not tried 1.7 and I only built my app for FroYo because that is what my target device runs. The behaviour is the same on the device and in the emulator.

My primary suspect would be to check if the following piece of code is actually present in OpenSSLSocketImpl.java in the FroYo version of the library:

                    // Shut down the SSL connection, per se.
                    try {
                        if (handshakeStarted) {
                            BlockGuard.getThreadPolicy().onNetwork();
                            NativeCrypto.SSL_shutdown(sslNativePointer, fd, this);
                        }
                    } catch (IOException ignored) {
                        /*
                         * Note that although close() can throw
                         * IOException, the RI does not throw if there
                         * is problem sending a "close notify" which
                         * can happen if the underlying socket is closed.
                         */
                    }

But I did not get hold of the source repository yet. The behaviour feels as if the catch...ignore is missing.

Thanks,

Eugene

On Tuesday, March 13, 2012 8:09:26 PM UTC+4, יניב ענבר Yaniv Inbar wrote:

Thanks for reporting the problem.  We have not had a chance to investigate this issue yet, so it is unlikely to behave differently with the new version 1.7.  Just for clarification, the issue you encountered actually happens on an Android device with FroYo SDK or with a later version?

Yaniv Inbar
Senior Software Engineer
Google Inc.

[Alex Cohn]

Attached find the files from Android 2.2_r1 source tree and, for
comparison, from 4.0_r1.

tl;nr:

the Froyo version performs

try { if (handshakeStarted) nativeclose();} catch (IOException ex) {
pendingException=ex; }

, while in ICS the IOException is ignored. Froyo does the same cleanup
and after that re-throws the pendingException.

BR,
Alex

[crosser]

So I would think that this re-throw is the culprit!
Would you explain how to replace just one source fine from the library? Then I could check this theory. (Sorry, I am a complete novice in java, this is my first java program.)

On Wednesday, March 14, 2012 12:34:40 PM UTC+4, Alex Cohn wrote:

Attached find the files from Android 2.2_r1 source tree and, for
comparison, from 4.0_r1.

tl;nr:

the Froyo version performs

  try { if (handshakeStarted) nativeclose();} catch (IOException ex) {
pendingException=ex; }

, while in ICS the IOException is ignored. Froyo does the same cleanup
and after that re-throws the pendingException.

BR,
Alex

[Alex Cohn]

No you can't replace a single file. You can compile and use an
alternative version of harmony library, but then you must supply both
Java code and native code.

Enjoy