Performance problems on heroku

[Mark Engelberg]

Every once in a while, I'm seeing response times spike, even when overall throughput, requests per second, and memory usage remains low.  Looking at heroku's monitoring, the problem often coincides with an "H27" error, which is their code for a client terminating the connection prematurely, perhaps by closing their browser in the middle of a request.

My app is very simple: I'm serving up static files (including a clojurescript-compiled js file) and process some post requests to log info to a heroku provisioned mongo database.  I'm using a basic luminus clojurescript template with the default immutant server.

A note to Heroku's customer support received the following response:

"Depending on how your application is setup, it could be that your application is being blocked by long running connections, such as the clients that causing the H27s, and is causing a knock on effect that is slowing down subsequent requests."

My takeaway from all this is that the most likely explanation is that the behavior of immutant is such that if a client terminates their connection in the middle of the request, it hangs up the server and prevents it from serving up other requests until after the timeout.  That sounds... bad.  Is there a way to change that behavior?  If not, what is the easiest way to switch to one of the other luminus-supported servers?  Which server would solve this problem and give me the best performance on heroku?

[Mark Engelberg]

I'm reading through the luminus documentation on immutant server tuning: http://www.luminusweb.net/docs/servers.md

According to immutant document, the default for number of io-threads is number of available processors.  Does that mean that if I'm running on one dyno on Heroku, that i/o is a completely blocking operation?  In other words, does it mean that if a client downloads the static file at a slow rate, it is blocking all other requests?  What would be a more sensible setting here?  Does something like aleph handle this differently?

[Dmitri]

Note that there's a separate threadpool for IO and worker threads. So, if you only have a single IO thread assigned, that will only block IO requests. You'd have to play around with the thread pool to see what's optimal in your situation. Aleph allows using async IO, but you'd have to setup routes to handle that explicitly as I recall, I haven't looked at it recently.

[Mark Engelberg]

What's a typical number of io threads to use, as a starting point?

[Dmitri]

It depends on how large the files being served are. Thread switching has its own overhead, so in cases where files are small switching might actually add overhead. I would try 2 or 4 and see if the performance looks better. Normally, you'd be able to profile that. I'm not familiar with the heroku diagnostic tools, but likely there are some metrics available to help you see resource utilization.

[Mark Engelberg]

I'm lacking a clear mental model of how the i/o thread performs work.  Does one thread mean it can literally handle only one request at a time?  Does it mean that a slow client or a slow response from the db will hang the server?  If so, then 2-4 threads doesn't seem like nearly enough, because that would still mean a maximum of 4 simultaneous requests.

[Juraj Martinka]

Disclaimer: I'm not an immutant/undertow expert; I use luminus-based apps on a daily basis but I've never tuned threads config for underlying http server.

Here are some resources to start with:

* http://immutant.org/documentation/current/apidoc/immutant.web.html

* http://undertow.io/undertow-docs/undertow-docs-1.4.0/index.html#management-of-io-and-worker-threads

>  Does one thread mean it can literally handle only one request at a time? 

I don't think so. Requests are handled by worker threads by default.

> Does it mean that a slow client or a slow response from the db will hang the server? 

No, at least not for the "slow response from the db" part (see the previous point)

Notice that even if you set `:dispatch false` some ring body types will be handled by worker threads anyway: http://immutant.org/documentation/current/apidoc/immutant.web.html

If your handlers are compute-bound, you may be able to gain some performance by setting :dispatch? to false. This causes the handlers to run on Undertow’s I/O threads, avoiding the context switch of dispatching them to the worker thread pool, at the risk of refusing client requests under load. Note that when :dispatch? is false, returning a seq, File, or InputStream as a ring body will cause that request to be dispatched to a worker thread at write time to prevent blocking an I/O thread.

See here for more info about IO threads worker threads: http://undertow.io/undertow-docs/undertow-docs-1.4.0/listeners.html#xnio-workers

WORKER_IO_THREADS
The number of IO threads to create. IO threads perform non blocking tasks, and should never perform blocking operations because they are responsible for multiple connections, so while the operation is blocking other connections will essentially hang. Two IO threads per CPU core is a reasonable default.
WORKER_TASK_CORE_THREADS
The number of threads in the workers blocking task thread pool. When performing blocking operations such as Servlet requests threads from this pool will be used. In general it is hard to give a reasonable default for this, as it depends on the server workload. Generally this should be reasonably high, around 10 per CPU core.

[Mark Engelberg]

Thanks.  This is interesting and useful information.

I find it rather curious that the docs recommend a default of 2 io threads per processor, but then actually implement 1 io thread per processor (according to the following link):
http://immutant.org/documentation/2.0.0-beta2/apidoc/immutant.web.undertow.html

Perhaps if immutant isn't following its own recommendations, luminus should override the default in the template to be 2 io threads per processor.

[Dmitri]

That seems reasonable, I'll update the template to use 2 io threads per cpu as the default.