[Play 2.5.3] Exception on com.typesafe.netty.HandlerSubscriber

[Thibault Meyer]

Hi,

We have this exception many time per days.

An exception was thrown by com.typesafe.netty.HandlerSubscriber$3.operationComplete()

java.util.NoSuchElementException: http-handler-body-subscriber
	at io.netty.channel.DefaultChannelPipeline.getContextOrDie(DefaultChannelPipeline.java:1093)
	at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:432)
	at com.typesafe.netty.http.HttpStreamsHandler.completeBody(HttpStreamsHandler.java:294)
	at com.typesafe.netty.http.HttpStreamsHandler.access$400(HttpStreamsHandler.java:14)
	at com.typesafe.netty.http.HttpStreamsHandler$3$1.run(HttpStreamsHandler.java:276)
	at com.typesafe.netty.http.HttpStreamsHandler.executeInEventLoop(HttpStreamsHandler.java:331)
	at com.typesafe.netty.http.HttpStreamsHandler.access$300(HttpStreamsHandler.java:14)
	at com.typesafe.netty.http.HttpStreamsHandler$3.complete(HttpStreamsHandler.java:273)
	at com.typesafe.netty.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:244)
	at com.typesafe.netty.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:241)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:683)
	at io.netty.util.concurrent.DefaultPromise$LateListeners.run(DefaultPromise.java:854)
	at io.netty.util.concurrent.DefaultPromise$LateListenerNotifier.run(DefaultPromise.java:882)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:280)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
	at java.lang.Thread.run(Thread.java:745)

[Greg Methvin]

This looks like https://github.com/playframework/playframework/issues/5866. The root cause was an akka streams bug (https://github.com/akka/akka/issues/20031), fixed in Akka 2.4.3. Play 2.5.3 depends on Akka 2.4.4.

Can you provide more context on the code that's causing this error?

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/76c231a7-79dd-4a99-a09b-7024f418aaa4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Greg Methvin

Senior Software Engineer

Lightbend, Inc.

[alex s]

понедельник, 2 мая 2016 г., 9:18:03 UTC+3 пользователь Greg Methvin написал:

This looks like https://github.com/playframework/playframework/issues/5866. The root cause was an akka streams bug (https://github.com/akka/akka/issues/20031), fixed in Akka 2.4.3. Play 2.5.3 depends on Akka 2.4.4.

Can you provide more context on the code that's causing this error?

I'm not the op (obviously), but at least in my case this error does involve nginx (v1.4.6, v1.8.1) and asset files.

Reproduction steps:

1. Set up nginx:

...

upstream play {
  server 127.0.0.1:9000;
}

server {
  proxy_buffering  off;
  ...
  proxy_http_version 1.1;

  listen 127.0.0.1:80;
  server_name play.devel;

  location / {
    proxy_pass http://play;
  }
}

2. Run default play-scala template:

% activator
Choose from these featured templates or enter a template name:
> play-scala
Enter a name for your application
> streams-exception
% cd streams-exception/
% sbt run

3. Run wrk:

% wrk http://1270.0.1:80/assets/javascripts/hello.js

[Thibault Meyer]

It seem to be the same issue for us.

[alex s]

понедельник, 2 мая 2016 г., 12:30:06 UTC+3 пользователь alex s написал:

I'm not the op (obviously), but at least in my case this error does involve nginx (v1.4.6, v1.8.1) and asset files.

FYI, with keepalive enabled I don't get any exceptions:

...

upstream play {
  server 127.0.0.1:9000;

  keepalive 32; # <--
}

server {
  proxy_buffering  off;
  ...
  proxy_http_version 1.1; # <--
  proxy_set_header Connection ""; # <--

[alex s]

понедельник, 2 мая 2016 г., 12:30:06 UTC+3 пользователь alex s написал:

Reproduction steps:

Well, let's skip the first two steps:

wrk -H "Connection: close" <any-url-served-by-the-assets-controller>

I'd like to stress that this exception is reproducible with Play 2.5.3 and Akka 2.4.4.

[Justin Lei]

I'm seeing this in my log files quite a lot too ever since I upgraded to Play 2.5.3.

The only thing I have to do to reproduce this is to go to a page that serves JS/CSS/image files and ctrl+F5.

Adding a keepalive to my nginx conf as Alex S suggests does seem to prevent the messages from coming up, but I suspect it's masking the real issue instead of really addressing it.

[Justin Lei]

Never mind, setting keepalive in nginx just reduces the rate at which my log file gets these messages; it doesn't completely eliminate it.

[Thibault Meyer]

I agree, it seem to have a missing check on the Play code. Or the logger message severity is not correct (maybe switch from warning to debug)

[Will Sargent]

For keepalive, you should also have SO_KEEPALIVE set:

play.server.netty.options.child.SO_KEEPALIVE=true 

Also consider setting your transport to "native" and then you can set TCP_FASTOPTION as well.

See https://www.playframework.com/documentation/2.5.x/SettingsNetty for more details.

--

You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/61330e72-0679-4ff3-9c30-de93c8cc97c1%40googlegroups.com.

[Will Sargent]

Whoops, that should be option singular, not plural:

play.server.netty.option.child.SO_KEEPALIVE

[Thibault Meyer]

Hi Will,

adding SO_KEEPALIVE on Play 2.5.3 and Connection "" on Nginx dont fix the issue.

[Thibault Meyer]

Hi Will,

Any news about this issue ? It will be fixed on Play 2.5.4 ? or do you have any work around to don't have theses messages ?

Le jeudi 12 mai 2016 16:26:44 UTC+2, Will Sargent a écrit :

[Christian Schmitt]

You don't need play for the fix. just add that to your build:

"com.typesafe.netty" % "netty-reactive-streams-http" % "1.0.6"

And it will be fixed

[Thibault Meyer]

Thanks Christian.