"java.io.IOException: Stream closed" in background thread

[Stuart Sierra]

Hello,

I have a long-polling web app with Aleph 0.2.1-beta2. I have a default uncaught exception handler set to catch exceptions on background threads. Occasionally, it catches this error:

java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(Unknown Source)
        at java.io.BufferedInputStream.read(Unknown Source)
        at aleph.formats$input_stream__GT_channel_.invoke(formats.clj:116)
        at aleph.formats$input_stream__GT_channel$fn__7745.invoke(formats.clj:137)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Unknown Source)

Any ideas as to the cause or solution?

Thanks,
-S

[Zach Tellman]

Am I correct in assuming that your HTTP response has an InputStream as
a body? If so, the assumption made is that it will be exhausted via
.read, and then closed by Aleph. What appears to be happening is that
the InputStream is being closed elsewhere, unexpectedly.

Without knowing more about your application, that's as specific as I
can be. Are you seeing truncated output anywhere?

Zach

[Stuart Sierra]

> Am I correct in assuming that your HTTP response has an InputStream as a body?

That's the thing: it doesn't. There's nothing that streams data from an InputStream back to the client. Mostly it's a synchronous Ring app, serving up static files and some generated HTML.

On one route only, the server accepts long polls from a web browser. Those polls always contain a complete POST request, i.e., the client is not streaming data to the server.

The server holds on to the channel associated with the long-poll request. The server writes a 1-character string to the channel every minute to prevent timeouts. When there is new information available for that client, it writes the data and closes the channel.

Since this error happens infrequently, I was wondering if a client could start making a request, then shut down and close the socket before it's finished, causing an error when Aleph/Netty tries to read the request body.

Thanks for such a quick response.
-S

[Zach Tellman]

The function aleph.formats/input-stream->channel is only used when an
HTTP response has an InputStream for a body. It spawns off a thread
so that synchronously reading from the InputStream doesn't block
anything else (Netty is parsimonious with its threads, using one for
too long can break things), as a result InputStreams aren't used
internally anywhere, just supported in this case since the Ring spec
requires it.

When you use wrap-ring-handler the body is represented as an
InputStream, but that's the inverse transform and at any rate the
InputStream is only created when the entire streamed body has been
received (this is a bit of a hack, but again is to prevent people from
breaking Netty).

Is it possible there's a transformation to an InputStream in your
middleware? That's the only explanation I can think of.

Zach

[Stuart Sierra]

Is it possible there's a transformation to an InputStream in your
middleware?  That's the only explanation I can think of.

Quite possible. We will investigate. Thanks for your help.
-S

[Stuart Sierra]

We may have found it: we're using Ring's wrap-resource middleware, which creates an input stream.

https://github.com/mmcgrana/ring/blob/efeb62c103c7e2a6774778cd2fb616cf5d6aa83d/ring-core/src/ring/util/response.clj#L104

I have no idea how the input stream is getting closed prematurely, but that could be the source of the problem.

-S

[Zach Tellman]

I think I know what's happening.

The closing of the InputStream is hooked into the closing of the
channel, which typically happens once input-stream->channel has
exhausted the InputStream. However, the channel is also closed if the
client disconnects, which can happen mid-read. In this case, the
channel would be closed, which would close the InputStream, which
would then cause .read to throw an exception.

That would mean these exceptions aren't a symptom of a problem, since
this is just a reaction to the client unexpected disconnecting. This
would make sense, given how infrequent you've said these errors are.
Spurious exceptions are a problem, so I'll definitely be fixing this,
but I don't think you need to worry about them in the meantime.

Thanks for the report. Let me know if I can help you with anything else.

Zach

[Stuart Sierra]

Thanks for the detailed and rapid reply, Zach!

This caused a problem for us initially because we were calling (System/exit 1) in response to any uncaught exception. (I started doing this as a general policy after seeing apps stop functioning with no visible indication that something was wrong.) We've disabled that for now, and we can also catch this particular exception as a special case now that we know what it is.

Thanks,
-S