java.lang.NoClassDefFoundError: com/zaxxer/hikari/pool/HikariPool$$Lambda$13

[Julie Sur]

Hi, 

I am using java 1.8u45 and I am seeing below error in my log. Any ideas on what I am missing in my config ?

TomcatLog Servlet.service() for servlet [default] in context with path [/] threw exception [Filter execution threw an exception] with root cause

java.lang.NoClassDefFoundError: com/zaxxer/hikari/pool/HikariPool$$Lambda$13

     at com.zaxxer.hikari.pool.HikariPool$$Lambda$13/1401577192.get$Lambda(Unknown Source)

     at com.zaxxer.hikari.pool.HikariPool.closeConnection(HikariPool.java:382)

     at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:183)

     at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:91)

 ......................

     at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)

     at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)

     at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)

     at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)

     at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)

     at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:558)

     at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:377)

     at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263)

     at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)

     ……………..

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)

     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)

     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)

     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)

     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)

     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537)

     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1085)

     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)

     at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)

     at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1556)

     at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1513)

     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

     at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

     at java.lang.Thread.run(Thread.java:745) 

[Mirco Dotta]

Hi Julie,

From my experience, NoClassDefFoundError occurs when you have a dependency that may have been compiled with HikariCP version X, and at linkage you are using HikariCP version Y (where X != Y). From the stacktrace, the dependency I’d look at is Hibernate. Check what version of HikariCP is used by the Hibernate version you depend on, and look in your build if you have an explicit dependency on HikariCP. If that’s the case, an easy to fix is to remove the explicit dependency from your build and just use the HikariCP version that is transitively added by Hibernate.

Hope this helps

Cheers,

Mirco

P.S. For my own curiosity, could you let us know what version of HikariCP you are using in your build, and what is the version used by Hibernate?

----------------

Mirco Dotta - @mircodotta

Typesafe – Build reactive apps!

--
You received this message because you are subscribed to the Google Groups "HikariCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hikari-cp+...@googlegroups.com.
Visit this group at http://groups.google.com/group/hikari-cp.
For more options, visit https://groups.google.com/d/optout.

[Brett Wooldridge]

It looks like HikariCP 2.3.x, because the class that cannot be found is a Lambda class. But it is internal, and not referencable from external classes like Hibernate. Which to me indicates a likelihood of two HikariCP jars in the classpath.

One thing to double-check is that an older version of the jar is not cached in Tomat's deployment directory.

[Julie Sur]

We are using HikariCP 2.2.5. 

[Julie Sur]

I realize that although my application uses java 1.8u45 , the default OS Java is 6u31 on the server.  Should that make any difference ? 

[Brett Wooldridge]

If your application uses 1.8u45, via your own installed JRE/JDK, the OS default should not make a difference.  If you are concerned, then depend on HikariCP-java6 version 2.3.12.

-Brett

[Aaron Holland]

This issue is a real head scratcher for me.  We are still on Hibernate 3.6.8, so there are no bundled Hikari classes there. 

We are definitely using Java 8 since our code is compiled into Java 8 classes, so it wouldn't run with any other Java version. 

Interestingly, none of the other Lambdas in HikariPool are throwing an error, and on my own development machine I can stick a break point on the problematic line and everything runs fine.  This is the only client that is seeing this error, so I figure it's environmental.

From the Googling I've done, it appears that you can get similar errors through serialization and reflection, so I'm wondering if it's some sort of proxy causing the error. I've recommended the client temporarily disable AppDynamics (which they are using, it's not standard with our product) to see if the error still occurs, but they are quite reluctant to do so.

Interestingly, on the client's other installations, increasing the pool size has made the issue "go away" (the error no longer occurs in the logs) but obviously it's not a permanent solution.

I get the feeling that the issue is not specifically Hikari related, but would be interested in any opinions. Cheers.

Aaron Holland

[Brett Wooldridge]

If the customer insists on running AppDynamics, they should ensure that it is at least version 3.9, as version 3.8 did not support Java 8.  Additionally, they have explicit documentation on how to instrument lambdas.  I suspect a race condition or other issue with their Java agent's bytecode instrumentation.

Their doc says:

In AppDynamics 3.6 and later, all Runnables, Callables and Threads are instrumented by default except those that are excluded by the agent configuration in app-agent-config.xml. 

From this agent documentation on the app-agent-config.xml file, I suggest the following:

<fork-config>
  <!-- exclude HikariCP internal classes -->
  <excludes filter-type="STARTSWITH" filter-value="com.zaxxer.hikari.pool/,com.zaxxer.hikari.proxy/,com.zaxxer.hikari.util/"/>

   ...

There is little to no business value in AppDynamics instrumenting these classes, rather it just adds overhead (and possibly instability).

My suggestion of running HikariCP-java6 (the Java 6/7) version was to eliminate lambdas entirely from the equation.  Even if you are running Java 8, there is no compatibility issue with using the HikariCP-java6 artifact.

Hope some of this helps.

-Brett

p.s. increasing the pool size may be altering the timing characteristics of the application, and therefore side-stepping whatever race condition may be occurring with their instrumentation.

[Aaron Holland]

Thanks for your feedback Brett, very much appreciated.

They say they are App Dynamics 4.0.7.0 build 40.  I'll recommend the class exclusion and see how we go.

Thanks again!

[Aaron Holland]

Turns out they *were* using the 3.7 App Dynamics Agent!  Thanks for your comments Brett. They finally disabled it until they can put an upgraded version in and everything is looking good so far.

I would just like to say that we switched to Hikari early this year for it's ability to recover much better than Bone CP from DB disconnects. I'm very happy with the performance and stability of Hikari!

Cheers,

Aaron Holland