Serializing with twitter chill

[omid]

Hi,

I am using twitter chill to serialize my objects, after adding Scala-Graph to my case class and try to serialize I got this :

Caused by: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: scalax.collection.GraphBase$Edge$ 

Any suggestion ?

[Peter Empen]

Hi Omid,
I'm personally not using Kryo/Twitter Chill. You should ask the Kryo experts directly under what conditions that exception is thrown.
Probably they have some preconditions not met by Graph for Scala instances so maybe you need to write a custom Kryo serializer...?
Peter

[omid]

1- I used chill before and it was working fine, but I will post an issue as well.

2- pickling also not working well with a little big graphs, what is the serialization that you recommend ?

[Peter Empen]

Tested are Java standard Serialization and JSON covered by the graph-json module.
It is known that Scala pickler has an issue as well.
Otherwise, you could serialize all outer nodes and edges by hand and use the graph builder to reconstruct the graph.

[M.Ahsen Taqi Kazmi]

Hi Omid

 I am having problems while useing chill for serialization
if you don't mind can you please guide me to get going with chill?

I am trying to run the sample code for Chill-Scala from https://github.com/xitrum-framework/chill-scala


I have two dummy case classes

one is User

    @SerialVersionUID(1)
    case class User(name :String, age : Int) extends Serializable

 
and the other is Student

    @SerialVersionUID(2)
    case class Student(sub : String , id : Int , user : User) extends Serializable

here is my code for serialization  that i have modified  from above mentioned github sample

    object SeriDeseri {
      private val kryo = {
        val a = KryoSerializer.registerAll
        val k = new Kryo
        a(k)
        k
      }
   
      def toBytes(student : Student): Array[Byte] = {
        println("********** serializing")
        val b = new ByteArrayOutputStream
        val o = new Output(b)
        kryo.writeObject(o, student)
        o.close()
        b.toByteArray
      }
   
      def fromBytes[Student](bytes: Array[Byte])(implicit m: Manifest[Student]): Option[Student] = {
        println("********** Deserializing")
        val i = new Input(bytes)
        try {
          val t = kryo.readObject(i, m.runtimeClass.asInstanceOf[Class[Student]])
          Option(t)
        } catch {
          case NonFatal(e) => None
        } finally {
          i.close()
        }
      }
    }

  

here is the code for my Main class

    val user = new User( "Ahsen", 14)
    val stu = new Student("oop", 12, user)
     
      val serial : Array[Byte] = SeriDeseri.toBytes(stu)
      val deserial :Option[Student] = SeriDeseri.fromBytes(serial)
      val obj  = deserial match{
        case Some(objec) => println(objec)
        case None => println("----------- Nothing was deserialized")
      }

Now the problem is when i run this code it gives me `java.lang.InstantiationError: scala.runtime.Nothing$` exception

here are complete stack traces

> [info] Running kryotest.Main

> ********** serializing

> ********** Deserializing
[error] (run-main-0) java.lang.InstantiationError: scala.runtime.Nothing$
> java.lang.InstantiationError: scala.runtime.Nothing$     at
> scala.runtime.Nothing$ConstructorAccess.newInstance(Unknown Source)
>     at
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy$1.newInstance(Kryo.java:1193)
>     at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1061)     at
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>     at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>     at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:657)     at
> com.test.hcast_serializer.SeriDeseri$.fromBytes(SeriDeseri.scala:32)
>     at kryotest.Main$.delayedEndpoint$kryotest$Main$1(Main.scala:31)     at
> kryotest.Main$delayedInit$body.apply(Main.scala:9)     at
> scala.Function0$class.apply$mcV$sp(Function0.scala:40)     at
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>     at scala.App$$anonfun$main$1.apply(App.scala:76)     at
> scala.App$$anonfun$main$1.apply(App.scala:76)     at
> scala.collection.immutable.List.foreach(List.scala:383)     at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
>     at scala.App$class.main(App.scala:76)     at
> kryotest.Main$.main(Main.scala:9)     at kryotest.Main.main(Main.scala)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:497) [trace] Stack
> trace suppressed: run last compile:run for the full output.
> java.lang.RuntimeException: Nonzero exit code: 1     at
> scala.sys.package$.error(package.scala:27) [trace] Stack trace
> suppressed: run last compile:run for the full output. [error]
> (compile:run) Nonzero exit code: 1 [error] Total time: 4 s, completed
> Aug 31, 2015 3:15:54 PM

Please tell me what am i doing wrong?

[thibaudch]

Hi,

I am facing the same problem with Spark (using Kryo serialization) when trying to broadcast my Graph object.

Does anyone have a solution?

[Peter Empen]

Concerning the error message "nable to create serializer... for class: scalax.collection.GraphBase$Edge$" you should ask the library provider, why.
What you can still do is to let `edges.toOuter` serialize and compose the graph by Graph.from at the other side.

[thibaudch]

Hi Peter,

Thanks, I am able to re-compose my Graph on a Spark worker machine.

I am now facing a strange issue: 

I would like to get all the reachable nodes with a max depth from a "currentNode": 

graph.get(currentNode).outerNodeTraverser.withMaxDepth(maxDepth).toList // => List[Int]

But I got this error: 

scala.MatchError: null
 at scalax.collection.TraverserImpl$Impl$class.$init$(TraverserImpl.scala:42)
 at scalax.collection.GraphTraversalImpl$OuterNodeTraverser.<init>(GraphTraversalImpl.scala:280)
 at scalax.collection.GraphTraversalImpl$class.outerNodeTraverser(GraphTraversalImpl.scala:299)
 at scalax.collection.immutable.DefaultGraphImpl.outerNodeTraverser(Graph.scala:41)
 at scalax.collection.immutable.DefaultGraphImpl.outerNodeTraverser(Graph.scala:41)
 at scalax.collection.GraphTraversal$TraverserInnerNode$class.outerNodeTraverser(GraphTraversal.scala:416)
 at scalax.collection.GraphLike$NodeBase.outerNodeTraverser(Graph.scala:143)
        ....

Unfortunately I am not able to reproduce this error locally, but facing it only when I am deploying my job on a Spark cluster.

Any ideas?

[Peter Empen]

It seems you still have a deserialization issue. How did you recompose the graph?
Peter

[thibaudch]

Hi,

I don't think it's a deserialization issue. I tried 2 approaches:

- Using Kryo serialization: export the graph to json, broadcast the json string recompose the graph from the json string

- Using java serialization and exporting directly the Graph

In both cases I faced the same issue.

The problem is coming from this matching in TraverserImpl.scala :

ordering match {
	case nO: NodeOrdering => (true, nO, nO.reverse,
	false, null, null)
	case eO: EdgeOrdering => (false, null, null,
	true, eO, eO.reverse)
	case _ : NoOrdering => (false, null, null,
	false, null, null)
	}

I have the feeling that "ordering" is not correctly instantiated.

I tried to return only : (false, null, null, false, null, null), recompiled and it worked as expected

Thanks