Configured registration timeout of 1 second expired, stopping

[Stig Brautaset]

I am running a Scala Spray api on Heroku and I keep getting the following warnings in my log:

2013-09-03T15:27:01.414845+00:00 app[web.4]: 15:27:01 WARN  [akka://s3cret/user/IO-HTTP/listener-0/51053] - Configured registration timeout of 1 second expired, stopping

We use spray version 1.1-M8.

Increasing from 2 to 4 web dynos reduced the occurence of this from 12% to 7% in my tests, but I do not get these when benchmarking locally. Is this more likely to be caused by the heroku app being halfway across the planet, or some weird interaction with heroku's routing infrastructure? 

Stig

[Johannes Rudolph]

Hi Stig,

On Tue, Sep 3, 2013 at 7:01 PM, Stig Brautaset <sbrau...@gmail.com> wrote:

2013-09-03T15:27:01.414845+00:00 app[web.4]: 15:27:01 WARN  [akka://s3cret/user/IO-HTTP/listener-0/51053] - Configured registration timeout of 1 second expired, stopping

We use spray version 1.1-M8.

Are the servers loaded? The warning indicates that after a connection attempt was made your HttpService didn't respond in 1 second. This is not necessarily a spray issue but in many cases this is a sign that you may suffer from thread starvation. Are you doing blocking calls in your HttpService or in any other actor on the same akka dispatcher as your HttpService?

You can try setting the `spray.can.server.registration-timeout` setting higher, but that is probably more a cure for the symptom than a real solution.

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Stig Brautaset]

Hi Johannes,

On Wednesday, 4 September 2013 08:27:31 UTC+1, Johannes Rudolph wrote:

On Tue, Sep 3, 2013 at 7:01 PM, Stig Brautaset <sbrau...@gmail.com> wrote:

2013-09-03T15:27:01.414845+00:00 app[web.4]: 15:27:01 WARN  [akka://s3cret/user/IO-HTTP/listener-0/51053] - Configured registration timeout of 1 second expired, stopping

We use spray version 1.1-M8.

Are the servers loaded? The warning indicates that after a connection attempt was made your HttpService didn't respond in 1 second. This is not necessarily a spray issue but in many cases this is a sign that you may suffer from thread starvation. Are you doing blocking calls in your HttpService or in any other actor on the same akka dispatcher as your HttpService?

I suspect you hit the nail on the head. I do blocking calls on other actors, and I haven't configured any custom dispatchers. When running the tests locally my db was on the same machine so the latency would be much less than when running on Heroku. I will configure a custom dispatcher for my db workers and test again.

You can try setting the `spray.can.server.registration-timeout` setting higher, but that is probably more a cure for the symptom than a real solution.

Yes. I tried that, but I just ran into timeouts elsewhere in akka.io.

Stig 

[Johannes Rudolph]

On Wed, Sep 4, 2013 at 12:04 PM, Stig Brautaset <sbrau...@gmail.com> wrote:

You can try setting the `spray.can.server.registration-timeout` setting higher, but that is probably more a cure for the symptom than a real solution.

Yes. I tried that, but I just ran into timeouts elsewhere in akka.io.

Yeah, that makes sense. It seems the registration timeout is then just an "accidental canary" showing that something is wrong.

Btw. you are not the first one having this issue:

https://groups.google.com/d/topic/spray-user/oF37WUPPtbk/discussion