Is this netty problem a show stopper?

[Ananth kumar]

Found Netty's native epoll transport in the classpath, but epoll is not available. Using NIO instead

java.lang.UnsatisfiedLinkError: no netty-transport-native-epoll in java.library.path

        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)

        at java.lang.Runtime.loadLibrary0(Runtime.java:849)

        at java.lang.System.loadLibrary(System.java:1088)

        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:168)

        at io.netty.channel.epoll.Native.<clinit>(Native.java:49)

        at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:30)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:190)

        at com.datastax.driver.core.NettyUtil.<clinit>(NettyUtil.java:68)

        at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:98)

        at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:690)

        at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1210)

        at com.datastax.driver.core.Cluster.init(Cluster.java:158)

        at com.datastax.driver.core.Cluster.connect(Cluster.java:248)


Kindly clarify.

[Ananth kumar]

Am having the following jars in the classpath already

netty-buffer-4.0.27.Final.jar                  netty-common-4.0.27.Final.jar                  netty-transport-4.0.27.Final.jar

netty-codec-4.0.27.Final.jar                   netty-handler-4.0.27.Final.jar                 netty-transport-native-epoll-4.0.27.Final.jar

[Olivier Michallat]

Hi,

Netty 4.0.x provides a new native epoll transport. This works only on Linux and requires that you declare netty-transport-native-epoll.jar explicitly as a dependency (it's optional in the driver's POM).

Which OS do you get the error on? Is it in your IDE (this would explain why the optional dependency was pulled)?

In any case, this is not an error, if the epoll transport cannot be loaded, the driver will default to the NIO transport.

--

Olivier Michallat

Driver & tools engineer, DataStax

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Ananth kumar]

I got this error on my local machine that is running Ubuntu 14.0.4. LTS version. 

My production system are all running on CentOS 6.4(Final) version. Is this error expected over there?

Do we need to set the library path to invoke this setting ? I guess both are recent Linux installations. As you told , it picks up the NIO transport mechanism but just want to make sure on this. Which one provides the optimal performance ?

[Andrew Tolbert]

Hi Ananth,

I think the issue might be that you are not using the platform specific native-epoll jar.   netty-transport-native-epoll-4.0.27.Final.jar does not contain the native library for epoll, you need to use netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar (if on 64-bit linux) instead.

How are you grabbing your dependencies?  Are you using maven or providing the jars manually?  If manually you can download the jar directly from here.

Thanks,

Andy

[Ananth kumar]

HI Andrew,

 All dependencies were available from the datastax driver download link. 

http://downloads.datastax.com/java-driver/

So does this carry a wrong dependency?

[Olivier Michallat]

Hi,

Yes that was an oversight, we should probably not include this JAR in the binary distribution. I've opened JAVA-766 to fix this.

--

Olivier Michallat

Driver & tools engineer, DataStax

[Peter Luttrell]

Today attempted to migrate another library that we use: Redisson (https://github.com/mrniko/redisson) from version 1.2.1 to 1.3.0 and that caused the 2.1.6 version of the DataStax Java driver to write a WARN + stack trace instead of an INFO to the log.

With 'org.redisson:redisson:1.2.1', I see:

INFO  [com.datastax.driver.core.NettyUtil] (MSC service thread 1-2) Did not find Netty's native epoll transport in the classpath, defaulting to NIO.

With 'org.redisson:redisson:1.3.0', I see:

WARN  [com.datastax.driver.core.NettyUtil] (MSC service thread 1-1) Found Netty's native epoll transport in the classpath, but epoll is not available. Using NIO instead.: java.lang.UnsatisfiedLinkError: no netty-transport-native-epoll in java.library.path

at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1857) [rt.jar:1.8.0_25]

at java.lang.Runtime.loadLibrary0(Runtime.java:870) [rt.jar:1.8.0_25]

at java.lang.System.loadLibrary(System.java:1119) [rt.jar:1.8.0_25]

at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:168) [netty-common-4.0.29.Final.jar:4.0.29.Final]

at io.netty.channel.epoll.Native.<clinit>(Native.java:50) [netty-transport-native-epoll-4.0.29.Final.jar:4.0.29.Final]

at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:30) [netty-transport-native-epoll-4.0.29.Final.jar:4.0.29.Final]

at java.lang.Class.forName0(Native Method) [rt.jar:1.8.0_25]

at java.lang.Class.forName(Class.java:260) [rt.jar:1.8.0_25]

at com.datastax.driver.core.NettyUtil.<clinit>(NettyUtil.java:68) [cassandra-driver-core-2.2.0-rc1.jar:]

at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:101) [cassandra-driver-core-2.2.0-rc1.jar:]

at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:698) [cassandra-driver-core-2.2.0-rc1.jar:]

at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1244) [cassandra-driver-core-2.2.0-rc1.jar:]

at com.datastax.driver.core.Cluster.init(Cluster.java:159) [cassandra-driver-core-2.2.0-rc1.jar:]

at com.datastax.driver.core.Cluster.connect(Cluster.java:249) [cassandra-driver-core-2.2.0-rc1.jar:]

at com.datastax.driver.core.Cluster.connect(Cluster.java:282) [cassandra-driver-core-2.2.0-rc1.jar:]

...

That JIRA ticket mentioned above looks to be resolved and the fix versions are listed as 2.0.11 and 2.1.7, but I could not find 2.1.7 in MavenCentral.

I also tried version 2.2.0-rc1 of the DataStax Java driver and see the same thing.

I also tried adding a dependency on 'io.netty:netty-transport-native-epoll:4.0.29.Final', which I believe contains the linux-x86_64 jars but still see the same thing.

For reference, I'm deploying to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-44-generic x86_64).

Please advise.

Thanks,

Peter Luttrell

[Andrew Tolbert]

Hi Peter,

It looks like the problem is occurring because redisson is listing netty-transport-native-epoll as a dependency, but it is not marking it as optional when it really should (in my opinion) like the java-driver does.   You can fix this problem by adding the architecture specific classifier to your dependencies:

        <dependency>

            <groupId>io.netty</groupId>

            <artifactId>netty-transport-native-epoll</artifactId>

            <version>${netty.version}</version>

            <!-- Explicitly bring in the linux classifier, test may fail on 32-bit linux -->

            <classifier>linux-x86_64</classifier>

            <scope>test</scope>

        </dependency>

It looks like redisson supports enabling epoll by providing a system property (Configuration.useLinuxNativeEpoll), but the jar is always on the classpath if you pull them in as a dependency.  The way the java-driver works is if it detects the epoll jar is present it will attempt to load epoll and if it cannot it will fail fast, instead of backing off.  The line of reasoning here is that if the user includes epoll in their classpath they intend to use it.

Andy

[Andrew Tolbert]

Link to Github issue on Redisson project: https://github.com/mrniko/redisson/issues/184

[Andrew Tolbert]

Just realized in my comment about adding the dependency that I had an element 'scope' listed at test.  You'll want to omit that line and also replace ${netty.version} with the version of netty you'd like to use.

[Peter Luttrell]

Andrew,

Many thanks for the reply. Adding a dependency on 'io.netty:netty-transport-native-epoll:4.0.29.Final:linux-x86_64' did indeed solve the problem. Thanks again for pointing me at the solution.

On a related topic, what can you advise about if and when to use epoll? I can't seem to find much published performance data on the web other then your comments on the JIRA ticket (JAVA-622).

Thanks,

Peter Luttrell

[Andrew Tolbert]

Hi Peter,

In limited testing I haven't found too much of a difference throughput wise.   That being said, if you are on a linux system, it is definitely worth considering and we've had no issues with it.  I suspect that if you use it you may not notice a boost in normal use, but you may notice more effective memory utilization/less GC activity.   It would be a fun exercise to get some benchmarks (if none such exist) comparing the NIO and epoll transport implementations.

Norman Maurer wrote a blog post 'JNI Performance - Welcome to the Dark Side' which gives some insight into the motivation of netty-transport-native-epoll module.   To summarize:

• 'The idea was to write a transport implementation that outperforms what ships with java.nio by making optimal use of the Thread-Model that powers Netty and is optimized for linux.'
  
•  'Uses edge-triggered epoll' (compared to level-triggered, read this for a good explanation of the difference with level-triggered)
  
• '...it outperforms the other netty transport which uses java.nio'
• '...produces less garbage' (From: http://netty.io/wiki/native-transports.html)'
• Observed a noticeable performance improvement with it ('a boost of about 63k req/sec for a total of about 593k req/sec')

The idea in the java-driver is that if the netty-transport-native-epoll module is present and EpollEventLoopGroup can be loaded that it is used.   This behavior can be explicitly disabled by passing in the system property '-Dcom.datastax.driver.FORCE_NIO=true'

Thanks,

Andy

[Paul Wais]

Was there a resolution here? I'm still seeing this warning.

My ivy.xml (using ant):

<dependencies>
  <dependency org="com.datastax.cassandra" name="cassandra-driver-core" rev="3.0.0"/>
  <dependency org="org.slf4j" name="slf4j-log4j12" rev="1.7.2"/>
  <dependency org="io.netty" name="netty-transport-native-epoll" rev="4.0.33.Final-linux-x86_64"/>
  <dependency org="ant-contrib" name="ant-contrib" rev="1.0b3"/>
</dependencies>

I see that I have netty-transport-native-epoll-4.0.33.Final.jar on my classpath.

In the cassandra log, I see:

INFO [main] 2016-04-20 23:42:50,644 NativeTransportService.java:70 - Netty using native Epoll event loop

[Andrew Tolbert]

Hi Paul,

I'd expect the jar name to contain the classifier (i.e. netty-transport-native-epoll-4.0.33.Final-linux-x86_64.jar) in the name.  I think for ivy you can specify the classifier by calling out the maven namespace in your root element and then using the classifier attribute, i.e.:

<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">

  ...

  <dependencies>

    ...

    <dependency org="io.netty" name="netty-transport-native-epoll" rev="4.0.33.Final" m:classifier="linux-x86_64"/> 

    ...

  </dependencies>

</ivy>

Can you give that a try and let me know how it goes?

Thanks!

Andy

[zhengli]

hi Andrew ，

I add the architecture specific classifier in my dependencies, but still see  the same thing as follows, any thoughts? 

thx

2016-10-10 11:23:19 - Found Netty's native epoll transport in the classpath, but epoll is not available. Using NIO instead.

java.lang.UnsatisfiedLinkError: could not load a native library: netty-transport-native-epoll

        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:217)

        at io.netty.channel.epoll.Native.<clinit>(Native.java:46)

        at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:30)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at com.datastax.driver.core.NettyUtil.<clinit>(NettyUtil.java:68)

        at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:101)

        at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:709)

        at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1386)

        at com.datastax.driver.core.Cluster.init(Cluster.java:162)

        at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:341)

        at com.datastax.driver.core.Cluster.connect(Cluster.java:286)

        ......

Caused by: java.lang.RuntimeException: Unable to obtain transfered field for DefaultFileRegion

        at java.lang.ClassLoader$NativeLibrary.load(Native Method)

        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1938)

        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1821)

        at java.lang.Runtime.load0(Runtime.java:809)

        at java.lang.System.load(System.java:1086)

        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:214)

在 2015年7月10日星期五 UTC+8上午6:20:14，Andrew Tolbert写道：