jvm crash when serializing with kryo

[Tim Pigden]

Hi

I get a jvm crash when calling something that ulitmately calls kryo - which is where the problem is happening.

Unfortunately these dumps are completely meaningless to me but I'd be really grateful if someone could suggest what might be going on

This is the stack section of the dump, the whole is attached as a file

It's pretty consistent - crashes every time - but there's a lot of code in the stack

Stack: [0x00007f1ea9e49000,0x00007f1ea9f4a000], sp=0x00007f1ea9f46740, free space=1013k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 2491 C1 com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;Z)Z (205 bytes) @ 0x00007f1eb52e9990 [0x00007f1eb52e9760+0x230]
j  com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;Lcom/esotericsoftware/kryo/Serializer;)V+43
J 2495 C1 com.esotericsoftware.kryo.serializers.ObjectField.write(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V (420 bytes) @ 0x00007f1eb52ee9ac [0x00007f1eb52ed120+0x188c]
j  com.esotericsoftware.kryo.serializers.FieldSerializer.write(Lcom/esotericsoftware/kryo/Kryo;Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V+101
j  com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;Lcom/esotericsoftware/kryo/Serializer;)V+184
J 2495 C1 com.esotericsoftware.kryo.serializers.ObjectField.write(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V (420 bytes) @ 0x00007f1eb52ee9ac [0x00007f1eb52ed120+0x188c]
j  com.esotericsoftware.kryo.serializers.FieldSerializer.write(Lcom/esotericsoftware/kryo/Kryo;Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V+101
J 2513 C1 com.esotericsoftware.kryo.Kryo.writeObject(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;Lcom/esotericsoftware/kryo/Serializer;)V (187 bytes) @ 0x00007f1eb530c46c [0x00007f1eb530c020+0x44c]
J 2495 C1 com.esotericsoftware.kryo.serializers.ObjectField.write(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V (420 bytes) @ 0x00007f1eb52ee5cc [0x00007f1eb52ed120+0x14ac]
j  com.esotericsoftware.kryo.serializers.FieldSerializer.write(Lcom/esotericsoftware/kryo/Kryo;Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V+101
j  com.esotericsoftware.kryo.Kryo.writeObject(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;Lcom/esotericsoftware/kryo/Serializer;)V+125
j  com.esotericsoftware.kryo.serializers.ObjectField.write(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V+143
j  com.esotericsoftware.kryo.serializers.FieldSerializer.write(Lcom/esotericsoftware/kryo/Kryo;Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V+101
j  com.esotericsoftware.kryo.Kryo.writeClassAndObject(Lcom/esotericsoftware/kryo/io/Output;Ljava/lang/Object;)V+150
j  com.romix.akka.serialization.kryo.KryoBasedSerializer.toBinary(Ljava/lang/Object;)[B+30
j  com.romix.akka.serialization.kryo.KryoSerializer.toBinary(Ljava/lang/Object;)[B+7
j  akka.persistence.serialization.MessageSerializer.akka$persistence$serialization$MessageSerializer$$payloadBuilder$1(Ljava/lang/Object;)Lakka/persistence/serialization/MessageFormats$PersistentPayload$Builder;+127
j  akka.persistence.serialization.MessageSerializer.akka$persistence$serialization$MessageSerializer$$persistentPayloadBuilder(Ljava/lang/Object;)Lakka/persistence/serialization/MessageFormats$PersistentPayload$Builder;+66
j  akka.persistence.serialization.MessageSerializer.akka$persistence$serialization$MessageSerializer$$persistentMessageBuilder(Lakka/persistence/PersistentRepr;)Lakka/persistence/serialization/MessageFormats$PersistentMessage$Builder;+171
j  akka.persistence.serialization.MessageSerializer.toBinary(Ljava/lang/Object;)[B+16
j  akka.serialization.Serialization$$anonfun$serialize$1.apply()[B+15
j  akka.serialization.Serialization$$anonfun$serialize$1.apply()Ljava/lang/Object;+1
j  scala.util.Try$.apply(Lscala/Function0;)Lscala/util/Try;+5
j  akka.serialization.Serialization.serialize(Ljava/lang/Object;)Lscala/util/Try;+12
j  akka.persistence.inmemory.serialization.AkkaSerializationProxy.serialize(Ljava/lang/Object;)Lscala/util/Try;+32
j  akka.persistence.inmemory.serialization.SerializationFacade.serializeARepr$1(Lakka/persistence/PersistentRepr;Lscala/collection/immutable/Set;)Lscala/util/Try;+5
j  akka.persistence.inmemory.serialization.SerializationFacade.akka$persistence$inmemory$serialization$SerializationFacade$$serializeTaggedOrRepr$1(Lakka/persistence/PersistentRepr;)Lscala/util/Try;+56
j  akka.persistence.inmemory.serialization.SerializationFacade$$anonfun$1.apply(Lakka/persistence/PersistentRepr;)Lscala/util/Try;+5
j  akka.persistence.inmemory.serialization.SerializationFacade$$anonfun$1.apply(Ljava/lang/Object;)Ljava/lang/Object;+5
J 1631 C1 scala.collection.TraversableLike$$anonfun$map$1.apply(Ljava/lang/Object;)Ljava/lang/Object; (6 bytes) @ 0x00007f1eb512dafc [0x00007f1eb512d920+0x1dc]
J 1561 C1 scala.collection.immutable.List.foreach(Lscala/Function1;)V (32 bytes) @ 0x00007f1eb5100644 [0x00007f1eb5100480+0x1c4]
J 1618 C1 scala.collection.immutable.List.map(Lscala/Function1;Lscala/collection/generic/CanBuildFrom;)Ljava/lang/Object; (122 bytes) @ 0x00007f1eb511b4fc [0x00007f1eb511a4a0+0x105c]
j  akka.persistence.inmemory.serialization.SerializationFacade.akka$persistence$inmemory$serialization$SerializationFacade$$serializeAtomicWrite(Lakka/persistence/AtomicWrite;)Lscala/util/Try;+18
j  akka.persistence.inmemory.serialization.SerializationFacade$$anonfun$serialize$1.apply(Lakka/persistence/AtomicWrite;)Lscala/util/Try;+5
j  akka.persistence.inmemory.serialization.SerializationFacade$$anonfun$serialize$1.apply(Ljava/lang/Object;)Ljava/lang/Object;+5
j  akka.stream.impl.fusing.Map.onPush(Ljava/lang/Object;Lakka/stream/stage/Context;)Lakka/stream/stage/SyncDirective;+6
j  akka.stream.impl.fusing.Map.onPush(Ljava/lang/Object;Lakka/stream/stage/Context;)Lakka/stream/stage/Directive;+3
j  akka.stream.stage.AbstractStage$PushPullGraphLogic$$anon$1.onPush()V+31
j  akka.stream.impl.fusing.GraphInterpreter.processElement$1(I)V+37
j  akka.stream.impl.fusing.GraphInterpreter.processEvent(I)V+22
j  akka.stream.impl.fusing.GraphInterpreter.execute(I)V+166
j  akka.stream.impl.fusing.GraphInterpreterShell.runBatch()V+44
j  akka.stream.impl.fusing.GraphInterpreterShell.init(Lakka/actor/ActorRef;Lakka/stream/impl/SubFusingActorMaterializerImpl;)V+133
j  akka.stream.impl.fusing.ActorGraphInterpreter.tryInit(Lakka/stream/impl/fusing/GraphInterpreterShell;)Z+9
j  akka.stream.impl.fusing.ActorGraphInterpreter.preStart()V+5
j  akka.actor.Actor$class.aroundPreStart(Lakka/actor/Actor;)V+1
j  akka.stream.impl.fusing.ActorGraphInterpreter.aroundPreStart()V+1
j  akka.actor.ActorCell.create(Lscala/Option;)V+26
j  akka.actor.ActorCell.invokeAll$1(Lakka/dispatch/sysmsg/SystemMessage;I)V+279
j  akka.actor.ActorCell.systemInvoke(Lakka/dispatch/sysmsg/SystemMessage;)V+6
j  akka.dispatch.Mailbox.processAllSystemMessages()V+51
j  akka.dispatch.Mailbox.run()V+8
j  akka.dispatch.Mailbox.exec()Z+1
j  scala.concurrent.forkjoin.ForkJoinTask.doExec()I+10

[Martin Grotzke]

Hi,

have you checked if there are similar issues already tracked
(https://github.com/EsotericSoftware/kryo/issues?utf8=%E2%9C%93&q=is%3Aissue+crash)?

The first in the list is
https://github.com/EsotericSoftware/kryo/issues/379 which leads to
https://github.com/romix/akka-kryo-serialization/pull/66

Maybe this one or other issues are helpful?

Cheers,
Martin

> --
> You received this message because you are subscribed to the "kryo-users"
> group.
> http://groups.google.com/group/kryo-users
> ---
> You received this message because you are subscribed to the Google
> Groups "kryo-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kryo-users+...@googlegroups.com
> <mailto:kryo-users+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Tim Pigden]

I've identified the cause of my problem which may be of value to anyone else. Pardon the akka-specific nature of the answer in the "wrong" forum  ..

The problem is occurring because one of my Akka messages - which I'm attempting to persist does not serialize.

below is a fragment of my akka config

akka {

actor {
  serialize-messages = on

  serializers {
    java = "akka.serialization.JavaSerializer"
    # Define kryo serializer
    kryo = "com.romix.akka.serialization.kryo.KryoSerializer"
  }

  serialization-bindings {
    "java.io.Serializable" = java
  }

In the above setup I get an exception thrown by akka that the message is not serializable.

java.io.NotSerializableException

If I change the serialization bindings to kryo, then I get 

com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
tail (com.optrak.opkakka.ddd.CommandResultBuses$UpdateSucceeded$)
b (scalaz.$bslash$div$minus)
result (com.optrak.opkakka.ddd.CommandResultBuses$CommandResultEnvelope)
   at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) ~[kryo-3.0.3.jar:na]

If I turn the serialize messages off then I get the jvm fault in some cases and in others it simply doesn't do the serialization.

With java serialization on and messages off, it simply doesn't do the serialization.

So in summary:

always put 

  serialize-messages = on

in your unit tests and make sure the unit tests coverage will serialize everything that you might want to serialize