[Very OT] Detecting socket closes
Maarten Koopmans
This post is completely off-topic, but the root of a problem I'm
running into lies close to that of a sophisticated NoSQL store
written in Java (it uses Voldemort as NoSQL metadata store).
I am wrapping up (and soon will be releasing on github) a pluggable
cloud backed storage project that runs the web side of things with
Lift, and a WebDAV daemon that is custom written in Scala. The daemon
is a custom written small HTTP server (as WebDAV is HTTP with some
extra verbs). Custom because it needed extra verbs, pluggable storage
back ends (S3, local,...), metering, encryption, compression etc.
So far, so good, with Voldemort as metadata store to map between
filenames and e.g. blobs in S3, providing locks, encryption keys etc.
The above is CPU intensive due to the encryption, so every request
runs in its own "share-nothing" thread, in more or less constant
memory. Old-school, no NIO, but as said, cpu-intensive.
However.... I have one problem that people doing Java/Scala
development may have seen, especially on Ubuntu 10.04 LTS Server with
both OpenJDK6 and Sun JDK 6. When a client starts downloading a file
but closes the connection brutally, say at 2%, the server thread does
not get an IOException on write (and flush), but simply keeps writing.
I've tried quite a few things:
- check for input and output shutdown
- send TCP OOB data
- flush after every write (i.e. no buffering above 16KB)
The reason that it's annoying is that a thread writing to a closed
socket uses decryption + decompression CPU cycles at the expense of
other, functioning clients. Especially clients that cache heavily,
such as OS X Finder, and are disconnected cause ~8 parallel useless
"downloads" that need to finish.
As said above, it is completely OT, and I feel a bit brutal asking it
here, but the Voldemort community is generally experienced and smart,
and I figured there is a fair chance that people can tell me that
it's either hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP
request/response and premature closing sockets.
I'd appreciate any input - it's the last thing standing between me
documenting and releasing this beast to the world (which I hope will
be a better place with it). And a nice showcase for Voldemort :-)
--Maarten