really strange MatchError - scala.MatchError: None (of class scala.None$)

[SWachter]

Hi all,

after some restructuring of an Akka stream application I get a really
strange match error:

scala.MatchError: None (of class scala.None$)
at
scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:192)
~[scala-library-2.11.8.jar:?]

The called method "getOrElseUpdate" clearly handles the "None" case:

def getOrElseUpdate(key: A, op: => B): B =
get(key) match {
case Some(v) => v
case None => val d = op; this(key) = d; d
}

Used versions: Scala 2.11.8; Akka 2.4.4

Can anyone hint what the problem might be?

TIA

Stefan

[SWachter]

I investigated the situation further and found that the MatchError
only happens if some newly introduced graph construction code is used.
In particular, I have a graph that is created with the Scala GraphDSL
and that has a custom shape.

If that graph is imported into another graph (in order to "decorate"
its behaviour) and the other graph is used instead of the original
graph then the MatchError occurs.

Turning auto-fusing off changes the location where the match error occurs:

scala.MatchError: None (of class scala.None$)

at akka.stream.impl.fusing.Recover.onPull(Ops.scala:181)
~[akka-stream_2.11-2.4.4.jar:?]

Again, looking at the code the "None" case is clearly handled there:

override def onPull(ctx: Context[T]): SyncDirective =
recovered match {
case Some(value) ⇒ ctx.pushAndFinish(value)
case None ⇒ ctx.pull()
}

SWachter – Mi, 11. Mai 2016 9:08

> --
> >>>>>>>>>> Read the docs: akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: groups.google.com/group/akka-user
> --- You received this message because you are subscribed to the
> Google Groups "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to akka-user+...@googlegroups.com.
> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at groups.google.com/group/akka-user.
> For more options, visit groups.google.com/d/optout.
>

[Akka Team]

Hi Stefan,

Do you have some small sample that reproduces this issue? This looks almost like a bug, but I am not sure how this can happen at all, so I would like to see a code snippet that somehow exercises this issue.

-Endre

--

     Read the docs: http://akka.io/docs/
     Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
     Search the archives: https://groups.google.com/group/akka-user

---You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.

Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - Reactive apps on the JVM

Blog: letitcrash.com
Twitter: @akkateam

[SWachter]

Hi Endre,

unfortunately I could not extract a small example. It seems that the
bug appears if various graph components are wired in certain ways.

(Do you use macros for the graph processing? When googling for such a
MatchError sometime macros seemed to be involved?)

The component where the bug appeared is a flow that first persists
something (using the reactive stream Mongo driver) and then the
outcome of that operation has to be combined with the original input.
Nearly all of our flows work with tuples. The first member represents
the usual data whereas the second member is used to transport
arbitrary context.

The chained components are:

- map (save request in context)
- map (transform request to Mongo document)
- flatMapConcat (persist Mongo document; with a source that is created
from the Mongo publisher, mapped, and recovered)
- map (retrieve orignal request from context and combine it with the
output of the Mongo operation)

When I replaced the stream such that the Mongo operation is "forked"
then no exception occurs:

- alsoTo ( map (request to Mongo document; as above) -> flatMapConcat
(with source from Mongo publisher; as above) -> Sink.ignore)
- map (request, assuming that the Mongo operation succeded)

However, that flow has a race condition. We need to wait until the
request is persisted. Therefore I rewrote the flow using the GraphDSL:

request -> bc(2)
bc -> storeInMongo (map & flatMapConcat as above) -> zipMongoResultWithRequest
bc -> zipMongoResultWithInput

That version worked. It seems that the "Broadcast/Zip" does somehow
prevent the graph from getting screwed.

I also tried to intersperse "async" and "buffer" boundaries to no avail.

--Stefan

Akka Team – Do, 12. Mai 2016 15:23

[Stefan Wachter]

Hi Endre,

thanks for looking into the issue. A "mixed Scala versions" issue was my first thought too. The output below is the result of "mvn dependency:tree". I see no clue that parts of different Scala versions are in the classpath. Could it be a classloader issue. Maybe the "None$" class gets loaded by two different classloaders? I am running out of ideas...

The used java version is:

java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

--Stefan

[INFO] com.bosch.inst.bikesensor:backend:jar:1.0-SNAPSHOT
[INFO] +- com.bosch.inst.bikesensor:amqp:jar:1.0-SNAPSHOT:compile
[INFO] |  +- com.typesafe.akka:akka-stream_2.11:jar:2.4.4:compile
[INFO] |  |  \- com.typesafe:ssl-config-akka_2.11:jar:0.2.1:compile
[INFO] |  |     \- com.typesafe:ssl-config-core_2.11:jar:0.2.1:compile
[INFO] |  |        \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] |  \- eu.rekawek.toxiproxy:toxiproxy-java:jar:1.0:compile
[INFO] +- com.typesafe.akka:akka-actor_2.11:jar:2.4.4:compile
[INFO] |  \- com.typesafe:config:jar:1.3.0:compile
[INFO] +- com.typesafe.akka:akka-slf4j_2.11:jar:2.4.4:compile
[INFO] +- com.typesafe.akka:akka-http-experimental_2.11:jar:2.4.4:compile
[INFO] |  \- com.typesafe.akka:akka-http-core_2.11:jar:2.4.4:compile
[INFO] |     \- com.typesafe.akka:akka-parsing_2.11:jar:2.4.4:compile
[INFO] +- org.scala-lang:scala-library:jar:2.11.8:compile
[INFO] +- org.scala-lang.modules:scala-java8-compat_2.11:jar:0.5.0:compile
[INFO] +- org.scala-lang.modules:scala-xml_2.11:jar:1.0.4:compile
[INFO] +- org.mongodb:mongodb-driver-async:jar:3.2.2:compile
[INFO] |  +- org.mongodb:mongodb-driver-core:jar:3.2.2:compile
[INFO] |  \- org.mongodb:bson:jar:3.2.2:compile
[INFO] +- org.mongodb:mongodb-driver-reactivestreams:jar:1.2.0:compile
[INFO] |  \- org.reactivestreams:reactive-streams:jar:1.0.0:compile
[INFO] +- com.google.code.gson:gson:jar:2.6.1:compile
[INFO] +- io.gsonfire:gson-fire:jar:1.5.1:compile
[INFO] +- com.cedarsoftware:json-io:jar:4.4.0:compile
[INFO] +- org.apache.logging.log4j:log4j-core:jar:2.3:compile
[INFO] |  \- org.apache.logging.log4j:log4j-api:jar:2.3:compile
[INFO] +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.3:compile
[INFO] +- io.kamon:kamon-core_2.11:jar:0.6.0:compile
[INFO] |  \- org.hdrhistogram:HdrHistogram:jar:2.1.7:compile
[INFO] +- io.kamon:kamon-system-metrics_2.11:jar:0.6.0:compile
[INFO] |  \- io.kamon:sigar-loader:jar:1.6.5-rev002:compile
[INFO] +- io.kamon:kamon-akka_2.11:jar:0.6.0:compile
[INFO] |  \- io.kamon:kamon-scala_2.11:jar:0.6.0:compile
[INFO] +- io.kamon:kamon-statsd_2.11:jar:0.6.0:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.5.4:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.5.4:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.5.4:compile
[INFO] +- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] +- org.hibernate:hibernate-validator:jar:5.1.3.Final:compile
[INFO] |  +- org.jboss.logging:jboss-logging:jar:3.1.3.GA:compile
[INFO] |  \- com.fasterxml:classmate:jar:1.0.0:compile
[INFO] +- javax.el:el-api:jar:2.2:compile
[INFO] +- org.glassfish:javax.el:jar:3.0.0:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- com.jayway.restassured:rest-assured:jar:2.9.0:test
[INFO] |  +- org.codehaus.groovy:groovy:jar:2.4.4:test
[INFO] |  +- org.codehaus.groovy:groovy-xml:jar:2.4.4:test
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.1:test
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.3:test
[INFO] |  |  +- commons-logging:commons-logging:jar:1.2:test
[INFO] |  |  \- commons-codec:commons-codec:jar:1.10:test
[INFO] |  +- org.apache.httpcomponents:httpmime:jar:4.5.1:test
[INFO] |  +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] |  +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:test
[INFO] |  +- com.jayway.restassured:json-path:jar:2.9.0:test
[INFO] |  |  +- org.codehaus.groovy:groovy-json:jar:2.4.4:test
[INFO] |  |  \- com.jayway.restassured:rest-assured-common:jar:2.9.0:test
[INFO] |  \- com.jayway.restassured:xml-path:jar:2.9.0:test
[INFO] |     \- org.apache.commons:commons-lang3:jar:3.3.2:test
[INFO] +- com.jayway.restassured:scala-support:jar:2.9.0:test
[INFO] +- org.mockito:mockito-core:jar:1.10.19:test
[INFO] |  \- org.objenesis:objenesis:jar:2.1:test
[INFO] +- org.easytesting:fest-assert:jar:1.4:test
[INFO] |  \- org.easytesting:fest-util:jar:1.1.6:test
[INFO] +- org.scalatest:scalatest_2.11:jar:2.2.5:test
[INFO] |  \- org.scala-lang:scala-reflect:jar:2.11.8:test
[INFO] +- com.typesafe.akka:akka-testkit_2.11:jar:2.4.4:test
[INFO] +- com.typesafe.akka:akka-stream-testkit_2.11:jar:2.4.4:test
[INFO] +- com.github.julien-truffaut:monocle-core_2.11:jar:1.2.0:test
[INFO] |  \- org.scalaz:scalaz-core_2.11:jar:7.2.0:test
[INFO] +- org.slf4j:slf4j-api:jar:1.7.12:compile
[INFO] +- com.github.tomakehurst:wiremock:jar:1.57:test
[INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26:test
[INFO] |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:test
[INFO] |  |  \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.2.3:test
[INFO] |  |  \- org.json:json:jar:20090211:test
[INFO] |  +- net.sf.jopt-simple:jopt-simple:jar:4.9:test
[INFO] |  +- xmlunit:xmlunit:jar:1.6:test
[INFO] |  \- com.jayway.jsonpath:json-path:jar:0.8.1:test
[INFO] |     +- net.minidev:json-smart:jar:1.1.1:test
[INFO] |     \- commons-lang:commons-lang:jar:2.6:test
[INFO] +- commons-io:commons-io:jar:2.4:compile
[INFO] +- com.rabbitmq:amqp-client:jar:3.6.1:compile
[INFO] \- com.google.guava:guava:jar:19.0:compile



Akka Team – Do, 12. Mai 2016 20:50
> Hi Stefan,
>
>
> Looking at the stacktraces, this is not an Akka issue, something is wrong with your setup (this error is thrown from places which are not called directly by Akka). Are you sure you are not mixing incompatible Scala versions or artifacts released for different versions of Scala?
>
>
> What JVM version are you using?
>
>
> -Endre

[Akka Team]

Hi Stefan,

Unfortunately all I can tell you that I have no clue :( Maybe try to extract parts of your application in a clean project (stubbing out stuff if necessary) and see if the issue comes up there.

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Stefan Wachter]

The MatchError bites me again. There are two different None$ instance involved (see screenshot). The first None$@8702 and the second None$7275. Yet, both None$ instance are loaded by the same ClassLoader and have the same Class (see screenshot):

[Akka Team]

Hi Stefan,

That looks really weird. What it seems like that None is instantiated somehow more than once, making the match to fail on expecting a certain instance (I guess).

Unfortunately I don't really know what to say here, the code where the error happens is simple Scala code that expects a proper Option[] to be returned by Scala. This is either a Scala bug or something weird is going on in your system as there shouldn't be more than one instances of None.

Can you cross-post your question on the Scala mailing list? They might be able to provide more help as my knowledge ends here unfortunately.

-Endre

[Stefan Wachter]

Hi Endre,

problem solved! Json Zaug Zaugg gave the right hint. It had to do with serialization. I use com.cedarsoftware.util.io.JsonReader to deserialize Json. JsonReader happily instantiates new None$ instances. A custom ClassFactory resolves the issue

JsonReader.assignInstantiator(None.getClass, new ClassFactory

{ override def newInstance(c: Class[_]): AnyRef = None }

)

Thanks for your help

--Stefan

[Akka Team]

That was one of my tips actually, but I rejected it since the None arrived from a part of code where I did not expect serialization to happen.

-Endre