This post is completely off-topic, but the root of a problem I'm running into lies close to that of a sophisticated web framework written in Scala (as it runs on request driven HTTP - where my problem is).
I am wrapping up (and soon will be releasing on github) a pluggable cloud backed storage project that runs the web side of things with Lift, and a WebDAV daemon that is custom written in Scala. The daemon is a custom written small HTTP server (as WebDAV is HTTP with some extra verbs). Custom because it needed extra verbs, pluggable storage back ends (S3, local,...), metering, encryption, compression etc.
So far, so good.
The above is CPU intensive due to the encryption, so every request runs in its own "share-nothing" thread, in more or less constant memory. However.... I have one problem that people doing Java/Scala development may have seen, especially on Ubuntu 10.04 LTS Server with both OpenJDK6 and Sun JDK 6. When a client starts downloading a file but closes the connection brutally, say at 2%, the server thread does not get an IOException on write (and flush), but simply keeps writing.
I've tried quite a few things: - check for input and output shutdown - send TCP OOB data - flush after every write (i.e. no buffering above 16KB)
The reason that it's annoying is that a thread writing to a closed socket uses decryption + decompression CPU cycles at the expense of other, functioning clients. Especially clients that cache heavily, such as OS X Finder, and are disconnected cause ~8 parallel useless "downloads" that need to finish.
As said above, it is completely OT, and I feel a bit brutal asking it here, but the Lift community is generally kind, experienced and smart , and I figured there is a fair chance that people can tell me that it's hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP request/response and premature closing sockets.
I'd appreciate any input - it's the last thing standing between me documenting and releasing this beast to the world (which I hope will be a better place with it).
If you get an IOException at any point, can't you use that to denote the OutputStream as dead?
Also, I'd look around to see if there's a SocketChannel floating around someplace in the output hierarchy and then do an isOpen() on the SocketChannel.
On Mon, Apr 16, 2012 at 11:27 AM, Maarten Koopmans <
> This post is completely off-topic, but the root of a problem I'm > running into lies close to that of a sophisticated web framework > written in Scala (as it runs on request driven HTTP - where my problem > is).
> I am wrapping up (and soon will be releasing on github) a pluggable > cloud backed storage project that runs the web side of things with > Lift, and a WebDAV daemon that is custom written in Scala. The daemon > is a custom written small HTTP server (as WebDAV is HTTP with some > extra verbs). Custom because it needed extra verbs, pluggable storage > back ends (S3, local,...), metering, encryption, compression etc.
> So far, so good.
> The above is CPU intensive due to the encryption, so every request > runs in its own "share-nothing" thread, in more or less constant > memory. However.... I have one problem that people doing Java/Scala > development may have seen, especially on Ubuntu 10.04 LTS Server with > both OpenJDK6 and Sun JDK 6. When a client starts downloading a file > but closes the connection brutally, say at 2%, the server thread does > not get an IOException on write (and flush), but simply keeps writing.
> I've tried quite a few things: > - check for input and output shutdown > - send TCP OOB data > - flush after every write (i.e. no buffering above 16KB)
> The reason that it's annoying is that a thread writing to a closed > socket uses decryption + decompression CPU cycles at the expense of > other, functioning clients. Especially clients that cache heavily, > such as OS X Finder, and are disconnected cause ~8 parallel useless > "downloads" that need to finish.
> As said above, it is completely OT, and I feel a bit brutal asking it > here, but the Lift community is generally kind, experienced and smart > , and I figured there is a fair chance that people can tell me that > it's hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP > request/response and premature closing sockets.
> I'd appreciate any input - it's the last thing standing between me > documenting and releasing this beast to the world (which I hope will > be a better place with it).
That's the weird thing: I don't get *any* IOException. As I am using "classic I/O" due to the CPU intensiveness I don't have A SocketChannel at all, but I may consider a medium rewrite if that's tge only route.
I've had some off-list suggestions like fronting with haproxy and tweaking TCP.
I'll surely write diwn my findings for the world... This is too obscure a problem in 2012.
On Monday, April 16, 2012, David Pollak wrote: > If you get an IOException at any point, can't you use that to denote the > OutputStream as dead?
> Also, I'd look around to see if there's a SocketChannel floating around > someplace in the output hierarchy and then do an isOpen() on the > SocketChannel.
> On Mon, Apr 16, 2012 at 11:27 AM, Maarten Koopmans < > maarten.koopm...@gmail.com <javascript:_e({}, 'cvml', > 'maarten.koopm...@gmail.com');>> wrote:
>> Hi,
>> This post is completely off-topic, but the root of a problem I'm >> running into lies close to that of a sophisticated web framework >> written in Scala (as it runs on request driven HTTP - where my problem >> is).
>> I am wrapping up (and soon will be releasing on github) a pluggable >> cloud backed storage project that runs the web side of things with >> Lift, and a WebDAV daemon that is custom written in Scala. The daemon >> is a custom written small HTTP server (as WebDAV is HTTP with some >> extra verbs). Custom because it needed extra verbs, pluggable storage >> back ends (S3, local,...), metering, encryption, compression etc.
>> So far, so good.
>> The above is CPU intensive due to the encryption, so every request >> runs in its own "share-nothing" thread, in more or less constant >> memory. However.... I have one problem that people doing Java/Scala >> development may have seen, especially on Ubuntu 10.04 LTS Server with >> both OpenJDK6 and Sun JDK 6. When a client starts downloading a file >> but closes the connection brutally, say at 2%, the server thread does >> not get an IOException on write (and flush), but simply keeps writing.
>> I've tried quite a few things: >> - check for input and output shutdown >> - send TCP OOB data >> - flush after every write (i.e. no buffering above 16KB)
>> The reason that it's annoying is that a thread writing to a closed >> socket uses decryption + decompression CPU cycles at the expense of >> other, functioning clients. Especially clients that cache heavily, >> such as OS X Finder, and are disconnected cause ~8 parallel useless >> "downloads" that need to finish.
>> As said above, it is completely OT, and I feel a bit brutal asking it >> here, but the Lift community is generally kind, experienced and smart >> , and I figured there is a fair chance that people can tell me that >> it's hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP >> request/response and premature closing sockets.
>> I'd appreciate any input - it's the last thing standing between me >> documenting and releasing this beast to the world (which I hope will >> be a better place with it).
maarten.koopm...@gmail.com> wrote: > That's the weird thing: I don't get *any* IOException. As I am using > "classic I/O" due to the CPU intensiveness I don't have A SocketChannel at > all, but I may consider a medium rewrite if that's tge only route.
> I've had some off-list suggestions like fronting with haproxy and tweaking > TCP.
> I'll surely write diwn my findings for the world... This is too obscure a > problem in 2012.
> Thanks and I'll report back!
> --Maarten
> On Monday, April 16, 2012, David Pollak wrote:
>> If you get an IOException at any point, can't you use that to denote the >> OutputStream as dead?
>> Also, I'd look around to see if there's a SocketChannel floating around >> someplace in the output hierarchy and then do an isOpen() on the >> SocketChannel.
>> On Mon, Apr 16, 2012 at 11:27 AM, Maarten Koopmans < >> maarten.koopm...@gmail.com> wrote:
>>> Hi,
>>> This post is completely off-topic, but the root of a problem I'm >>> running into lies close to that of a sophisticated web framework >>> written in Scala (as it runs on request driven HTTP - where my problem >>> is).
>>> I am wrapping up (and soon will be releasing on github) a pluggable >>> cloud backed storage project that runs the web side of things with >>> Lift, and a WebDAV daemon that is custom written in Scala. The daemon >>> is a custom written small HTTP server (as WebDAV is HTTP with some >>> extra verbs). Custom because it needed extra verbs, pluggable storage >>> back ends (S3, local,...), metering, encryption, compression etc.
>>> So far, so good.
>>> The above is CPU intensive due to the encryption, so every request >>> runs in its own "share-nothing" thread, in more or less constant >>> memory. However.... I have one problem that people doing Java/Scala >>> development may have seen, especially on Ubuntu 10.04 LTS Server with >>> both OpenJDK6 and Sun JDK 6. When a client starts downloading a file >>> but closes the connection brutally, say at 2%, the server thread does >>> not get an IOException on write (and flush), but simply keeps writing.
>>> I've tried quite a few things: >>> - check for input and output shutdown >>> - send TCP OOB data >>> - flush after every write (i.e. no buffering above 16KB)
>>> The reason that it's annoying is that a thread writing to a closed >>> socket uses decryption + decompression CPU cycles at the expense of >>> other, functioning clients. Especially clients that cache heavily, >>> such as OS X Finder, and are disconnected cause ~8 parallel useless >>> "downloads" that need to finish.
>>> As said above, it is completely OT, and I feel a bit brutal asking it >>> here, but the Lift community is generally kind, experienced and smart >>> , and I figured there is a fair chance that people can tell me that >>> it's hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP >>> request/response and premature closing sockets.
>>> I'd appreciate any input - it's the last thing standing between me >>> documenting and releasing this beast to the world (which I hope will >>> be a better place with it).
On Monday, April 16, 2012, David Pollak wrote: > How about using Netty to host the app. I'm betting you can learn a lot > more about the connection from Netty.
> On Mon, Apr 16, 2012 at 12:08 PM, Maarten Koopmans < > maarten.koopm...@gmail.com> wrote:
> That's the weird thing: I don't get *any* IOException. As I am using > "classic I/O" due to the CPU intensiveness I don't have A SocketChannel at > all, but I may consider a medium rewrite if that's tge only route.
> I've had some off-list suggestions like fronting with haproxy and tweaking > TCP.
> I'll surely write diwn my findings for the world... This is too obscure a > problem in 2012.
> Thanks and I'll report back!
> --Maarten
> On Monday, April 16, 2012, David Pollak wrote:
> If you get an IOException at any point, can't you use that to denote the > OutputStream as dead?
> Also, I'd look around to see if there's a SocketChannel floating around > someplace in the output hierarchy and then do an isOpen() on the > SocketChannel.
> On Mon, Apr 16, 2012 at 11:27 AM, Maarten Koopmans < > maarten.koopm...@gmail.com> wrote:
> Hi,
> This post is completely off-topic, but the root of a problem I'm > running into lies close to that of a sophisticated web framework > written in Scala (as it runs on request driven HTTP - where my problem > is).
> I am wrapping up (and soon will be releasing on github) a pluggable > cloud backed storage project that runs the web side of things with > Lift, and a WebDAV daemon that is custom written in Scala. The daemon > is a custom written small HTTP server (as WebDAV is HTTP with some > extra verbs). Custom because it needed extra verbs, pluggable storage > back ends (S3, local,...), metering, encryption, compression etc.
> So far, so good.
> The above is CPU intensive due to the encryption, so every request > runs in its own "share-nothing" thread, in more or less constant > memory. However.... I have one problem that people doing Java/Scala > development may have seen, especially on Ubuntu 10.04 LTS Server with > both OpenJDK6 and Sun JDK 6. When a client starts downloading a file > but closes the connection brutally, say at 2%, the server thread does > not get an IOException on write (and flush), but simply keeps writing.
> I've tried quite a few things: > - check for input and output shutdown > - send TCP OOB data > - flush after every write (i.e. no buffering above 16KB)
> The reason that it's annoying is that a thread writing to a closed > socket uses decryption + decompression CPU cycles at the expense of > other, functioning clients. Especially clients that cache heavily, > such as OS X Finder, and are disconnected cause ~8 parallel useless > "downloads" that need to finish.
> As said above, it is completely OT, and I feel a bit brutal asking it > here, but the Lift community is generally kind, experienced and smart > , and I figured there is a fair chance that people can tell me that > it's hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP > request/response and premature closing sockets.
> I'd appreciate any input - it's the last thing standing between me > documenting and releasing this beast to the world (which I hope will > be a better place with it).
Argh. I set the socket timeout now on the client socket to 5 seconds, and *removed an accidental catch-all*. That one prevented me from having the thread "crash" on closing the socket.... (now I do see the exception - you have to set SO_TIMEOUT low though).
Thanks for your advice and patience!
Best, Maarten
On Mon, Apr 16, 2012 at 11:00 PM, Maarten Koopmans
<maarten.koopm...@gmail.com> wrote: > Good one, but will require some rewriting. I'll look how much and estimate > if/when I can do that.
> On Monday, April 16, 2012, David Pollak wrote:
>> How about using Netty to host the app. I'm betting you can learn a lot >> more about the connection from Netty.
>> On Mon, Apr 16, 2012 at 12:08 PM, Maarten Koopmans >> <maarten.koopm...@gmail.com> wrote:
>> That's the weird thing: I don't get *any* IOException. As I am using >> "classic I/O" due to the CPU intensiveness I don't have A SocketChannel at >> all, but I may consider a medium rewrite if that's tge only route.
>> I've had some off-list suggestions like fronting with haproxy and tweaking >> TCP.
>> I'll surely write diwn my findings for the world... This is too obscure a >> problem in 2012.
>> Thanks and I'll report back!
>> --Maarten
>> On Monday, April 16, 2012, David Pollak wrote:
>> If you get an IOException at any point, can't you use that to denote the >> OutputStream as dead?
>> Also, I'd look around to see if there's a SocketChannel floating around >> someplace in the output hierarchy and then do an isOpen() on the >> SocketChannel.
>> On Mon, Apr 16, 2012 at 11:27 AM, Maarten Koopmans >> <maarten.koopm...@gmail.com> wrote:
>> Hi,
>> This post is completely off-topic, but the root of a problem I'm >> running into lies close to that of a sophisticated web framework >> written in Scala (as it runs on request driven HTTP - where my problem >> is).
>> I am wrapping up (and soon will be releasing on github) a pluggable >> cloud backed storage project that runs the web side of things with >> Lift, and a WebDAV daemon that is custom written in Scala. The daemon >> is a custom written small HTTP server (as WebDAV is HTTP with some >> extra verbs). Custom because it needed extra verbs, pluggable storage >> back ends (S3, local,...), metering, encryption, compression etc.
>> So far, so good.
>> The above is CPU intensive due to the encryption, so every request >> runs in its own "share-nothing" thread, in more or less constant >> memory. However.... I have one problem that people doing Java/Scala >> development may have seen, especially on Ubuntu 10.04 LTS Server with >> both OpenJDK6 and Sun JDK 6. When a client starts downloading a file >> but closes the connection brutally, say at 2%, the server thread does >> not get an IOException on write (and flush), but simply keeps writing.
>> I've tried quite a few things: >> - check for input and output shutdown >> - send TCP OOB data >> - flush after every write (i.e. no buffering above 16KB)
>> The reason that it's annoying is that a thread writing to a closed >> socket uses decryption + decompression CPU cycles at the expense of >> other, functioning clients. Especially clients that cache heavily, >> such as OS X Finder, and are disconnected cause ~8 parallel useless >> "downloads" that need to finish.
>> As said above, it is completely OT, and I feel a bit brutal asking it >> here, but the Lift community is generally kind, experienced and smart >> , and I figured there is a fair chance that people can tell me that >> it's hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP >> request/response and premature closing sockets.
>> I'd appreciate any input - it's the last thing standing between me >> documenting and releasing this beast to the world (which I hope will >> be a better place with it).
On Tuesday, April 17, 2012 11:28:37 AM UTC+2, maarten wrote:
> Argh. I set the socket timeout now on the client socket to 5 seconds, > and *removed an accidental catch-all*. That one prevented me from > having the thread "crash" on closing the socket.... (now I do see the > exception - you have to set SO_TIMEOUT low though).
> Thanks for your advice and patience!
> Best, Maarten
> On Mon, Apr 16, 2012 at 11:00 PM, Maarten Koopmans > <maarten.koopm...@gmail.com> wrote: > > Good one, but will require some rewriting. I'll look how much and > estimate > > if/when I can do that.
> > On Monday, April 16, 2012, David Pollak wrote:
> >> How about using Netty to host the app. I'm betting you can learn a lot > >> more about the connection from Netty.
> >> On Mon, Apr 16, 2012 at 12:08 PM, Maarten Koopmans > >> <maarten.koopm...@gmail.com> wrote:
> >> That's the weird thing: I don't get *any* IOException. As I am using > >> "classic I/O" due to the CPU intensiveness I don't have A SocketChannel > at > >> all, but I may consider a medium rewrite if that's tge only route.
> >> I've had some off-list suggestions like fronting with haproxy and > tweaking > >> TCP.
> >> I'll surely write diwn my findings for the world... This is too obscure > a > >> problem in 2012.
> >> Thanks and I'll report back!
> >> --Maarten
> >> On Monday, April 16, 2012, David Pollak wrote:
> >> If you get an IOException at any point, can't you use that to denote the > >> OutputStream as dead?
> >> Also, I'd look around to see if there's a SocketChannel floating around > >> someplace in the output hierarchy and then do an isOpen() on the > >> SocketChannel.
> >> On Mon, Apr 16, 2012 at 11:27 AM, Maarten Koopmans > >> <maarten.koopm...@gmail.com> wrote:
> >> Hi,
> >> This post is completely off-topic, but the root of a problem I'm > >> running into lies close to that of a sophisticated web framework > >> written in Scala (as it runs on request driven HTTP - where my problem > >> is).
> >> I am wrapping up (and soon will be releasing on github) a pluggable > >> cloud backed storage project that runs the web side of things with > >> Lift, and a WebDAV daemon that is custom written in Scala. The daemon > >> is a custom written small HTTP server (as WebDAV is HTTP with some > >> extra verbs). Custom because it needed extra verbs, pluggable storage > >> back ends (S3, local,...), metering, encryption, compression etc.
> >> So far, so good.
> >> The above is CPU intensive due to the encryption, so every request > >> runs in its own "share-nothing" thread, in more or less constant > >> memory. However.... I have one problem that people doing Java/Scala > >> development may have seen, especially on Ubuntu 10.04 LTS Server with > >> both OpenJDK6 and Sun JDK 6. When a client starts downloading a file > >> but closes the connection brutally, say at 2%, the server thread does > >> not get an IOException on write (and flush), but simply keeps writing.
> >> I've tried quite a few things: > >> - check for input and output shutdown > >> - send TCP OOB data > >> - flush after every write (i.e. no buffering above 16KB)
> >> The reason that it's annoying is that a thread writing to a closed > >> socket uses decryption + decompression CPU cycles at the expense of > >> other, functioning clients. Especially clients that cache heavily, > >> such as OS X Finder, and are disconnected cause ~8 parallel useless > >> "downloads" that need to finish.
> >> As said above, it is completely OT, and I feel a bit brutal asking it > >> here, but the Lift community is generally kind, experienced and smart > >> , and I figured there is a fair chance that people can tell me that > >> it's hopeless or there is a trick with TCP, JVM, Scala, Linux and HTTP > >> request/response and premature closing sockets.
> >> I'd appreciate any input - it's the last thing standing between me > >> documenting and releasing this beast to the world (which I hope will > >> be a better place with it).
