[Completely OT] Detecting socket closes

[Maarten Koopmans]

Hi,

This post is completely off-topic, but the root of a problem I'm
running into lies close to that of a sophisticated web framework
written in Scala (as it runs on request driven HTTP - where my problem
is).

I am wrapping up (and soon will be releasing on github) a pluggable
cloud backed storage project that runs the web side of things with
Lift, and a WebDAV daemon that is custom written in Scala. The daemon
is a custom written small HTTP server (as WebDAV is HTTP with some
extra verbs). Custom because it needed extra verbs, pluggable storage
back ends (S3, local,...), metering, encryption, compression etc.

So far, so good.

The above is CPU intensive due to the encryption, so every request
runs in its own "share-nothing" thread, in more or less constant
memory. However.... I have one problem that people doing Java/Scala
development may have seen, especially on Ubuntu 10.04 LTS Server with
both OpenJDK6 and Sun JDK 6. When a client starts downloading a file
but closes the connection brutally, say at 2%, the server thread does
not get an IOException on write (and flush), but simply keeps writing.

I've tried quite a few things:
- check for input and output shutdown
- send TCP OOB data
- flush after every write (i.e. no buffering above 16KB)

The reason that it's annoying is that a thread writing to a closed
socket uses decryption + decompression CPU cycles at the expense of
other, functioning clients. Especially clients that cache heavily,
such as OS X Finder, and are disconnected cause ~8 parallel useless
"downloads" that need to finish.

As said above, it is completely OT, and I feel a bit brutal asking it
here, but the Lift community is generally kind, experienced and smart
, and I figured there is a fair chance that people can tell me that
it's hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP
request/response and premature closing sockets.

I'd appreciate any input - it's the last thing standing between me
documenting and releasing this beast to the world (which I hope will
be a better place with it).

--Maarten

[David Pollak]

If you get an IOException at any point, can't you use that to denote the OutputStream as dead?

Also, I'd look around to see if there's a SocketChannel floating around someplace in the output hierarchy and then do an isOpen() on the SocketChannel.

--Maarten

--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

--

Visi.Pro, Cloud Computing for the Rest of Us http://visi.pro

Lift, the simply functional web framework http://liftweb.net

Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im

[Maarten Koopmans]

That's the weird thing: I don't get *any* IOException. As I am using "classic I/O" due to the CPU intensiveness I don't have A SocketChannel at all, but I may consider a medium rewrite if that's tge only route.

I've had some off-list suggestions like fronting with haproxy and tweaking TCP.

I'll surely write diwn my findings for the world... This is too obscure a problem in 2012.

Thanks and I'll report back!

--Maarten

[David Pollak]

How about using Netty to host the app.  I'm betting you can learn a lot more about the connection from Netty.

[Maarten Koopmans]

Good one, but will require some rewriting. I'll look how much and estimate if/when I can do that.

[Maarten Koopmans]

Argh. I set the socket timeout now on the client socket to 5 seconds,
and *removed an accidental catch-all*. That one prevented me from
having the thread "crash" on closing the socket.... (now I do see the
exception - you have to set SO_TIMEOUT low though).

Thanks for your advice and patience!

Best, Maarten

On Mon, Apr 16, 2012 at 11:00 PM, Maarten Koopmans

[Lukasz Kuczera]

Good that you solved it. Oftentimes you just need to say loud about the problem to get it solved :)