Strange java.lang.VerifyError with ActorSystem initialization

[Matlik]

I'm encountering this issue which I've never seen before, and was wondering if someone could point me in the right direction.  I have some Akka streams 2.4.10 code (with Scala 2.11.8) that is embedded into a GlassFish 4.0 server running Oracle Java 1.8.0_131-b11.  All this is wrapped into a Docker container for deployment that uses the Ubuntu as the base image. Things were all working great for a while, but something appears to have changed that is causing us some grief.  

The docker image is built with the latest Oracle Java 8 JDK and base OS (Ubuntu 14.04).  When we needed to rebuild the image with a minor configuration change, we are now consistently getting the following error:

 [#|2017-06-19T19:15:09.770+0000|WARNING|glassfish 4.0|javax.enterprise.web|_ThreadID=45;_ThreadName=http-listener-1(1);_TimeMillis=1497899709770;_LevelValue=900;|

   StandardWrapperValve[TimeSeriesServlet]: Servlet.service() for servlet TimeSeriesServlet threw exception

 java.lang.VerifyError: Stack map does not match the one at exception handler 98

 Exception Details:

   Location:

     akka/dispatch/Envelope.<init>(Ljava/lang/Object;Lakka/actor/ActorRef;)V @98: athrow

   Reason:

     Current frame's flags are not assignable to stack map frame's.

   Current Frame:

     bci: @0

     flags: { flagThisUninit }

     locals: { uninitializedThis, 'java/lang/Object', 'akka/actor/ActorRef' }

     stack: { 'java/lang/Throwable' }

   Stackmap Frame:

     bci: @98

     flags: { }

     locals: { top, 'java/lang/Object', 'akka/actor/ActorRef' }

     stack: { 'java/lang/Throwable' }

   Bytecode:

     0x0000000: 2a2b b500 322a 2cb5 0037 2ab7 0085 014e

     0x0000010: b200 8703 3201 0105 bd00 0859 032a 5359

     0x0000020: 0405 bd00 0859 032b 5359 042c 5353 b900

     0x0000030: 8d04 004e a700 053a 052a b800 932d c600

     0x0000040: 232d 1295 0105 bd00 0859 0311 00b1 bb00

     0x0000050: 975a 5fb7 009a 5359 0401 53b9 008d 0400

     0x0000060: 57b1 bf                                

   Exception Handler Table:

     bci [16, 52] => handler: 55

     bci [0, 98] => handler: 98

   Stackmap Table:

     full_frame(@55,{Object[#6],Object[#8],Object[#120],Object[#8]},{Object[#130]})

     same_frame(@57)

     same_frame(@97)

     full_frame(@98,{Top,Object[#8],Object[#120]},{Object[#130]})

 

        at akka.dispatch.Envelope$.apply(AbstractDispatcher.scala:27)

        at akka.actor.Cell$class.sendMessage(ActorCell.scala:295)

        at akka.actor.ActorCell.sendMessage(ActorCell.scala:374)

        at akka.actor.RepointableActorRef.$bang(RepointableActorRef.scala:171)

        at akka.actor.ActorRef.tell(ActorRef.scala:128)

        at akka.pattern.AskableActorRef$.internalAsk$extension(AskSupport.scala:295)

        at akka.pattern.AskableActorRef$.$qmark$extension1(AskSupport.scala:281)

        at akka.event.LoggingBus$class.akka$event$LoggingBus$$addLogger(Logging.scala:181)

        at akka.event.LoggingBus$$anonfun$4$$anonfun$apply$4.apply(Logging.scala:116)

        at akka.event.LoggingBus$$anonfun$4$$anonfun$apply$4.apply(Logging.scala:115)

        at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)

        at scala.util.Try$.apply(Try.scala:192)

        at scala.util.Success.map(Try.scala:237)

        at akka.event.LoggingBus$$anonfun$4.apply(Logging.scala:115)

        at akka.event.LoggingBus$$anonfun$4.apply(Logging.scala:112)

        at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683)

        at scala.collection.Iterator$class.foreach(Iterator.scala:893)

        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)

        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

        at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682)

        at akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:112)

        at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683)

        at scala.collection.Iterator$class.foreach(Iterator.scala:893)

        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)

        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

        at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682)

        at akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:112)

        at akka.event.EventStream.startDefaultLoggers(EventStream.scala:28)

        at akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:667)

        at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:648)

        at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:645)

        at akka.actor.ActorSystemImpl._start(ActorSystem.scala:645)

        at akka.actor.ActorSystemImpl.start(ActorSystem.scala:661)

        at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)

        at akka.actor.ActorSystem$.apply(ActorSystem.scala:109)

        at com.vnomicscorp.timeseries.tsfs.TsfsApplicationContext.<init>(TsfsApplicationContext.scala:42)

And this is all kicked off just by calling this line of code:

  implicit val actorSystem = ActorSystem("tsfs")

This far, I tried various versions of the base OS and tried updating Akka with zero effect. Any suggestions would be greatly appreciated!  

Thank you, 

James

[Konrad “ktoso” Malawski]

It’s rather unlikely that Akka itself is at fault here.

You could try using the latest version of the lib etc, however I don’t think that’s it…

Do other java apps work in your env (are you sure it’s not weird / broken)?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Matlik]

I find it strange that it would be Akka, but this does appear to only impact the Akka/Scala code in my project.  It is effectively a Java application that wraps the Akka/Scala functionality for part of it.  The plain Java functionality appears to be unaffected by this bug.

I was hoping someone who understands this stack trace output might be able to shed some light on what exactly it means.  I've been working with the JVM for over 15 years and have never before seen this output.  My inclination is to think I've somehow got a different and incompatible version of a lib being used at runtime than what is used at compile time, but I've been unsuccessful at identifying anything like that with Akka's JARs in my build.  Is there enough information to poinpoint exactly what the disconnect is?  

[Konrad “ktoso” Malawski]

These crazy things are best debugged locally to be honest.

Try cleaning your ivy caches, maybe the jar file got corrupted somehow etc?

[Roland Kuhn]

Just a thought: are you using the Scala distribution’s runner to run the application? There were versions that added an old version of Akka to the bootclasspath.

Regards,

Roland

[Ionuț G. Stan]

What's the actual akka-actor version being used there? I've investigated
a bit and it doesn't seem to be akka-actor_2.11:2.4.10. The stackmap
frames are different I'm seeing with 2.4.10 are different than what's
show in your stack trace.

Once a colleague bumped into a similar issue with Scala and it solved it
by upgrading the JVM. Not sure what's the problem in your case.

For a bit of background. The JVM is performing some bytecode security
checks when it loads .class files — to ensure there's no possibility for
stack underflows or overflows in the generated bytecode, or that the
arguments on stack have the types expected by the method receiving them,
etc.

Before JVM 1.7 this was a thing done entirely by the JVM. After 1.7,
compilers are sometimes required to emit "proofs" that the code they
generated is behaving correctly. These proofs are the StackMapTables in
the .class files.

Apparently, your JVM verified whether the emitted proofs are valid and
figured out they're not. This can happen when there's a mismatch between
the JVM algorithm that performs the verification and the compiler
algorithm that generates the proofs. Which one is in your case I have no
idea.

> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google
> Groups "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to akka-user+...@googlegroups.com

> <mailto:akka-user+...@googlegroups.com>.

> To post to this group, send email to akka...@googlegroups.com

> <mailto:akka...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.


--

Ionuț G. Stan | http://igstan.ro | http://bucharestfp.ro

[Ionuț G. Stan]

Do you use any tool that performs bytecode instrumentation? From your
error message it appears there's try/catch block in there, but there's
no such thing in the source code for akka-actor_2.11:2.4.10:

https://github.com/akka/akka/blob/v2.4.10/akka-actor/src/main/scala/akka/dispatch/AbstractDispatcher.scala#L27

If yes, that tool might be the culprit.

[Matlik]

The bytecode manipulation appears to be a promising lead.  We have had NewRelic and Jolokia in place for a long time, both of which use Java agents to perform bytecode instrumentation.  I can't explain why this problem suddenly appeared (I suspect but have not confirmed it is somehow related to the newer JVM version).  The error looks to have cleared up when we remove these.

Thank you!

[Chris Marsh]

I just experienced the same issue this week. Turns out I had an older version of the newrelic java agent installed which caused the issue. I updated the newrelic java agent to the latest version (as of this writing ver 3.41.0) and that fixed the issue. 

Thanks to this thread I could stop pounding my head against the wall. 

Thanks!