Re: [kryo] How should I use Kryo for serializing abstract classes and their concrete implementations

[Nate]

On Wed, Jun 20, 2012 at 2:46 AM, Richards <hbkri...@gmail.com> wrote:

Hi,

I would like to use Kryo Serialization for my project. I am new to kryo and I have gone through the documentation. But I am quite confused about its usage as we relied on java serialization till now.

The use case is like this:
I have an abstract class Constant which has different concrete implementations like NumberConstant, DateConstant, NumberRangeConstant, StringConstant, etc.

I would like to serialize these implementations before they are exchanged between JVM.

What I did was:

/***************************************************************************************************************************************************************/
abstract class Constant implements Serializable, KryoSerializable{
//some methods are there
}

/***************************************************************************************************************************************************************/
class NumberConstant extends Constant implements Serializable, KryoSerializable{

double value;

public NumberConstant(){
value=0.0;
}

//setters, getters, hashCode(), equals() also present

public void read(Kryo kryo, Input input) {
    value = new Double(input.readDouble());
    input.close();

Don't close the input during deserialization.
 

}

public void write(Kryo kryo, Output output) {
    output.writeDouble(value.doubleValue());
    output.flush();
    output.close();

Don't close the output during serialization.
 

}

}

/***************************************************************************************************************************************************************/
class NumberRangeConstant extends Constant implements Serializable, KryoSerializable{

private NumberConstant min;
private NumberConstant max;

public NumberRangeConstant(NumberConstant min, private NumberConstant max){
this.min=min;
this.max = max;
}

//some helper methods to compare ranges are also present here.

public void read(Kryo kryo, Input input) {
//1. Is registration required at this point?
    kryo.register(NumericConstant.class);

Register all classes you want to register before any serialization or deserialization. Registering is optional.
 

// 2. I think I should read only these object without classes. Am I correct?
    start = kryo.readObject(input, NumericConstant.class);

The rules are here:
https://code.google.com/p/kryo/#Reading_and_writing
What you have is fine if you know it won't be null.
 

    end = kryo.readObject(input, NumericConstant.class);
    input.close();

You get the idea by now.
 

}

@Override
public void write(Kryo kryo, Output output) {
//3. Is registration required at this point?
    kryo.register(NumericConstant.class);
//4. I think I have to write the objects only and not their classes. Am I correct?
    kryo.writeObject(output, start);
    kryo.writeObject(output, end);
    output.close();
    output.flush();
}
}

/***************************************************************************************************************************************************************/
Similarly I have other constant classes also.
5. How do I create a Serilizer for these constants? Is the following code correct?

public class ConstantSerializer extends com.esotericsoftware.kryo.Serializer<Constant> {

    @Override
    public void write(Kryo kryo, Output output, Constant constant) {
        kryo.writeClassAndObject(output, constant);

You need to write the data for the constant here. If you call Kryo to write it, it will call your serializer to write it: infinite loop.

The real problem is you should only register serializers for concrete types, not interfaces or abstract classes.
 

    }

    @Override
    public Constant create(Kryo kryo, Input input, Class<Constant> constantClass) {
        /* 6. What should I do if there are no default constructors? Moreover there different implementations for Constant class*/
        return (Constant) kryo.readClassAndObject(input);
    }

    //7. Should I override the read() also? If yes how should I do it?

The latest Kryo does not have create and read methods. You should update to the latest.
 

}

/***************************************************************************************************************************************************************/

8. I would like to use a Map<Integer,Constant> which should be shared between JVMs. How should I use a Serializer for that? There can be different kinds of Constants in the map. I think I should use both MapSerializer and ConstantSerializer. Can you show me a sample code for that?

This isn't clear. You have the same map in both JVMs and you only want to set an int key instead of a whole Constant? Or you want to send the map?

You seem to be pretty much all over the place here. Maybe try a simpler example? Create a new simple class that only has one field and serialize it. Maybe breakpoint and step through what Kryo is doing, it isn't much code at all.

Also, Kryo can serialize most classes (yours that I've seen are very simple) without you doing anything. You don't have to register any classes, your classes don't need to implement anything, and you don't have to write any serializers. Use the quickstart code here:
https://code.google.com/p/kryo/#Quickstart
Use writeClassAndObject/readClassAndObject if you have a Constant, since you may not be sure what the concrete type is. You can also pass it a Map<Integer, Constant> and it will just work.

-Nate

[Richards Peter]

Hi Nate,

Thanks for the quick reply. Actually I am trying to use the twitter project called Storm for one of our use cases. It is said in the project that kryo serialization is faster than java serialization. We wanted to send a Map<Integer,Constant> from one bolt(which is a storm concept) to another. These bolts may run on different JVMs. This message passing was taking lot of time. But when we tried to pass primitive data types it was working very fast. Storm internally uses kryo for serialization as documented in kryo website.

Related to question 7:
Storm is currently using kryo 2.04 in its latest release. So I am not sure if I can migrate to the newer version. I have seen in one of your mails that kryo 2.16 is lightning fast. I will cross check this with Storm's architect.

Related to my question 8 and your answer:
I wanted to send the Map<Integer,Constant> from one bolt to another. Is it fine if I register only the concrete implementations of Constant class(which are Kryo Serializable) and use it without worrying about the Serializer? Will the receiving process be able to deserialize the map correctly?

Thanks,
Richards Peter.

[Nate]

On Wed, Jun 20, 2012 at 4:12 AM, Richards Peter <hbkri...@gmail.com> wrote:

Hi Nate,

Thanks for the quick reply. Actually I am trying to use the twitter project called Storm for one of our use cases. It is said in the project that kryo serialization is faster than java serialization. We wanted to send a Map<Integer,Constant> from one bolt(which is a storm concept) to another. These bolts may run on different JVMs. This message passing was taking lot of time. But when we tried to pass primitive data types it was working very fast. Storm internally uses kryo for serialization as documented in kryo website.

Related to question 7:
Storm is currently using kryo 2.04 in its latest release. So I am not sure if I can migrate to the newer version. I have seen in one of your mails that kryo 2.16 is lightning fast. I will cross check this with Storm's architect.

Kryo 2 is fast, but yeah some things done lately make it even faster.
 

Related to my question 8 and your answer:
I wanted to send the Map<Integer,Constant> from one bolt to another. Is it fine if I register only the concrete implementations of Constant class(which are Kryo Serializable) and use it without worrying about the Serializer? Will the receiving process be able to deserialize the map correctly?

If you want to register classes, register all concrete classes that are not primitives, primitive wrappers, or String. So register HashMap and your concrete implementations of Constant. The Constant implementations do not need to implement KryoSerializable. They can for maximum speed, but maybe try without it first.

-Nate

[Richards Peter]

Thank you Nate,

I will try both the approaches.

Richards Peter

[Richards Peter]

Hi Nate,

In case I am not making the classes KryoSerializable, is it mandatory to have a default constructor? If it is mandatory, can you suggest methods by which I can call the parametrized constructors?

Thanks,
Richards Peter.

[Nate]

https://code.google.com/p/kryo/#Object_creation

-Nate

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users

[Richards Peter]

Thank you so much Nate,

Richards Peter.

[Richards Peter]

Hi Nate,

I would like to ask one more doubt related to this post which I made sometime back.

Yesterday I was trying to serialize List<Constant[]> due to some requirement in my project. Unfortunately I was not able to do this in the twitter project called storm. I have not made any custom serializer for Constant class. But I have registered Constant class for kryo serialization. I get the following exception during serialization:
java.lang.RuntimeException: java.lang.IllegalArgumentException: Class is not registered: com.ama.Constant[]

I tried to serialize Constant[][] also. It also threw a similar exception. But when I tried to serialize List<List<Constant>> everything worked fine. I would like to know why serializing List<Constant[]> is failing.

Thanks,
Richards.

[Nate]

Register Constant[].class.

-Nate

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users

---
You received this message because you are subscribed to the Google Groups "kryo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Richards Peter]

Hi Nate,

Thanks for your quick reply. I tried to register the array -> Constant[]. But that throws a class not found exception for the array.

java.lang.RuntimeException: java.lang.ClassNotFoundException: com/ama/Constant[]
at backtype.storm.serialization.SerializationFactory.getKryo(SerializationFactory.java:70)

The jar containing Constant class is present in class path. Otherwise serialization of List<List<Constant>> should have failed.

I am not sure about the registration of this array for kryo serialization. My class declaration is having the following structure:

public class Constant implements someInterfaces{

//some public methods

}

I have not declared any constructors for this class.

Thanks,
Richards.

[Richards Peter]

Hi Nate,

I have posted a query on the same issue in storm-user group. I am not sure whether it is a problem in storm.

Please let me know if you would like me to try something for this serialization issue.

Thanks,
Richards Peter.

[Nate]

I guess storm uses some configuration file to configure Kryo. You likely need to specify the class in a way Class.forName can handle. Eg:

System.out.println(com.example.Example[].class.getName());
// prints: [Lcom.example.Example;

-Nate

Thanks,
Richards.

[Richards Peter]

Nate,

Wow! That was the exact reason. Now everything is working fine.

Thanks a lot,
Richards.