Cannot use Play 2.5 with ElasticSearch 5 because of Netty 4 vs 4.1. Play 2.6?

[Kaj Magnus Lindberg]

Hi,

Play 2.5 uses Netty 4.0, but the ElasticSearch (ES) client uses Netty 4.1. The result is that 4.0 gets evicted and 4.1 used instead...

...And this results in the below null pointer exception when attempting to start Play:

java.lang.NullPointerException

    at io.netty.channel.group.DefaultChannelGroup.add(DefaultChannelGroup.java:146)

    at play.core.server.NettyServer.bind(NettyServer.scala:140)

    at play.core.server.NettyServer.play$core$server$NettyServer$$bindChannel(NettyServer.scala:224)

    at play.core.server.NettyServer$$anonfun$1.apply(NettyServer.scala:216)

    at play.core.server.NettyServer$$anonfun$1.apply(NettyServer.scala:216)

    at scala.Option.map(Option.scala:146)

    at play.core.server.NettyServer.<init>(NettyServer.scala:216)

    at play.core.server.NettyServerProvider.createServer(NettyServer.scala:279)

    at play.core.server.NettyServerProvider.createServer(NettyServer.scala:278)

    at play.core.server.DevServerStart$$anonfun$mainDev$1.apply(DevServerStart.scala:235)

    at play.core.server.DevServerStart$$anonfun$mainDev$1.apply(DevServerStart.scala:65)

    at play.utils.Threads$.withContextClassLoader(Threads.scala:21)

    at play.core.server.DevServerStart$.mainDev(DevServerStart.scala:64)

    at play.core.server.DevServerStart$.mainDevHttpMode(DevServerStart.scala:54)

    at play.core.server.DevServerStart.mainDevHttpMode(DevServerStart.scala)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:498)

    at play.runsupport.Reloader$.startDevMode(Reloader.scala:234)

    at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.devModeServer$lzycompute$1(PlayRun.scala:74)

    at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.play$sbt$run$PlayRun$$anonfun$$anonfun$$anonfun$$devModeServer$1(PlayRun.scala:74)

    at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.apply(PlayRun.scala:100)

    at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.apply(PlayRun.scala:53)

    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)

Currently I'm using ElasticSearch (ES) alpha-5, because that one still uses Netty 4.0 and works with Play. In my case, using an alpha version of ES felt (feels) okay, since my project was (is) in alpha/beta status itself anyway.

But "never" being able to upgrade from alpha-5 could become problematic.

Does someone have some thoughts about what makes sense for me to do? (in order to solve the [cannot upgr from ES-5-alpha to ES-5-final] problem)

What I'm thinking I could do:  (and I suppose this can be of interest to anyone who wants to add ES to a Play 2.5 app? — they likely want to use the most recent version of ES (i.e. ES 5 with Netty 4.1))

- Try to make play 2.5 use Netty 4.1, as described in this thread: 

    https://groups.google.com/d/topic/play-framework/UUDZZRpAv3w/discussion

    ("Will Play 2.5.x upgrade to netty 4.1?")

    But to me that seems like complicated, and … feels like off-topic work for me.

- Try to backport the ES client to use Netty 4.0 (I can ask in the ES forum about this. But I guess this'd be even more complicated)

- Wait for Play 2.6 to be released? When might that happen? Roughly which month would anyone guess?

- Since my own project is in alpha-beta status itself, a Play 2.6 alpha or beta version would probably be OK in my case.

   Does anyone know when a 2.6 alpha or beta might be available?

Thanks for building Play b.t.w., & best regards,

KajMagnus

[Christian Schmitt]

you can use the akka server: https://www.playframework.com/documentation/2.5.x/AkkaHttpServer#Akka-HTTP-server-backend-(experimental) 

[Kaj Magnus Lindberg]

Hi Christian,

Thanks, that looks like one thing I can do.

I also recently found out that ES 5 will include another HTTP client with fever dependencies (in addition to the "old" TransportClient).

And that they're going to add a developer friendly API to it — instead of just JSON which is what it wants right now.

( See: https://www.elastic.co/blog/elasticsearch-5-0-0-released, the section "Java REST client":

"... and has minimal dependencies, resulting in fewer dependency conflicts than the Transport client ... we will be adding an API..."

And apparently it doensn't depend on Netty, because when I add the "Java REST client" instead of the "Transpors client", then Play Fmw starts fine,

+ running `dependency-tree` in SBT shows that Netty 4.0 no longer gets evicted. )

Ok so later on, I'll either give Akka HTTP Server a try, or (more likely) wait for the ES people to add a developer friendly API to the REST client.

Regards,

KajMagnus

On Thursday, November 3, 2016 at 3:51:27 PM UTC+1, Christian Schmitt wrote:

you can use the akka server: https://www.playframework.com/documentation/2.5.x/AkkaHttpServer#Akka-HTTP-server-backend-(experimental) 

Am Donnerstag, 3. November 2016 08:10:55 UTC+1 schrieb Kaj Magnus Lindberg:

Hi,

Play 2.5 uses Netty 4.0, but the ElasticSearch (ES) client uses Netty 4.1. The result is that 4.0 gets evicted and 4.1 used instead...

...And this results in the below null pointer exception when attempting to start Play:

[...] 

[Christian Schmitt]

Actually the Akka Server is also pretty good, despites the experimental flag. The akka team also made akka-http really really fast.
So don't be too scared. I actually use elasticsearch 5 + the akka http in one project. We use: "org.elasticsearch.client" % "transport" % "5.0.0".

However this client doesn't use Netty, I guess they use the apache httpclient. However we wanted to be netty free, since maybe we need to use play-ws which will than depend on netty.

P.S.: to get the best performance out of it you should use promises:

val promise = Promise[SearchResponse]()
client.search(request, new ActionListener[SearchResponse] {
  def onFailure(e: Exception): Unit = promise.tryFailure(e)
  def onResponse(response: SearchResponse): Unit = promise.trySuccess(response)
})
promise.future

[Kaj Magnus Lindberg]

Hi Christian,

Ok, I did get a bit scared :- )  Good to know that they made akka-http fast, looking forwards to using it in Play 2.6 or 3.0.

I already use  play-ws, and also an open auth lib, Silhouette, which apparently depends on play-netty-server (with Netty 4.0).

So, when I switched to akka-http-server, I ran into some weird problems, because of Netty 4.0 evicted.

Looks to me I'll need to wait for the ES REST client then. Anyway ES 5-alpha4 seems stable enough for me right now :- )

(  Not sure if you were referring to the ES 5 TransportClient when you wrote "this client doesn't use netty"? Or the ES 5 REST client.

As far as I can tell the TransportClient does use Netty:

[info]   +-org.elasticsearch.client:transport:5.0.0

[info]   | ...

[info]   | +-org.elasticsearch.plugin:transport-netty3-client:5.0.0

[info]   | | +-io.netty:netty:3.10.6.Final   <— weird. Why both Netty 3.10.6 and 4.1.5 (below)

[info]   | | 

[info]   | +-org.elasticsearch.plugin:transport-netty4-client:5.0.0

[info]   | | +-io.netty:netty-buffer:4.1.5.Final

[info]   | | | ...

[info]   | | | 

[info]   | | +-io.netty:netty-codec-http:4.1.5.Final

[info]   | | | ...

[info]   | | |       

[info]   | | +-io.netty:netty-codec:4.1.5.Final

[info]   | | | ...

[info]   | | |     

[info]   | | +-io.netty:netty-common:4.1.5.Final

[info]   | | +-io.netty:netty-handler:4.1.5.Final

[info]   | | | ...

[info]   | | |     

[info]   | | +-io.netty:netty-resolver:4.1.5.Final

[info]   | | | ...

[info]   | | | 

[info]   | | +-io.netty:netty-transport:4.1.5.Final

[info]   | |   ...

[info]   | |     

[info]   | +-org.elasticsearch:elasticsearch:5.0.0

[info]   |   +...  

 )

Thanks for the code snippet with a promise & ActionListener[SearchResponse]. That's how my code looks like already :- )

Regards,
KajMagnus

[Christian Schmitt]

well you are right, it has a dependency on net :(

however for sillhoute you can exclude the transitive dependency via sbt exclude http://www.scala-sbt.org/0.13/docs/Library-Management.html#Exclude+Transitive+Dependencies it shouldn’t make any problems since sillhoute doesn’t use any netty stuff.

for async-http-client which play uses you can exclude the dependency on async-http-client and manually add org.asynchttpclient % async-http-client % 2.1.0-alpha1, for development that is just fine, however for production it misses several patches.

-- 
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/41212027-ef85-4113-a79d-efa29ccfc880%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joao Antunes]

For all of you interested in this issue.

There's a 'quickfix', that can be found here:

https://github.com/sksamuel/elastic4s/issues/666