How to serialize Map<String, Object> -- getting buffer underrun on read

[Steve Storck]

If I have a HashMap<String, Object>, what is the proper way to serialize it?  I am able to serialize my whole object except for that map, and when I am deserializing it, I am getting a buffer underrun.  I am setting it up so that I am using a MapSerializer, I'm setting the key type to String, and setting allowNullKeys to false.  I am not setting the value type or the value serializer.  I have this set up in a custom serializer and it is set up the same in the write method as it is in the read method.  Unfortunately, I cannot post the code because it is at work and I am at home right now.  But any guidance in serializing this map would be greatly appreciated.  Thanks in advance.

[Steve Storck]

By the way, if I change the map to String, String, and I do everything else the same (but set value type to String), then the serialization/deserialization works perfectly.  Unfortunately, that won't work in my application.

[Dewful]

Not an expert, but my advice is that it's not great to keep it as an 'object'. What type of object(s) does this field store (i.e. what classes)? Usually you need to know the concrete class at some point. Posting the code might help someone chime in. You'll need a custom serializer probably.

On Mon, Dec 3, 2018 at 5:25 PM Steve Storck <stev...@gmail.com> wrote:

By the way, if I change the map to String, String, and I do everything else the same (but set value type to String), then the serialization/deserialization works perfectly.  Unfortunately, that won't work in my application.

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
---
You received this message because you are subscribed to the Google Groups "kryo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nate]

If the MapSerializer doesn't know the value type, it will write the type before the data. MapSerializerTest works so likely there is a bug in your serializer. Otherwise, you can create an issue with an SSCCE.

[Steve973]

Here is what I am doing...

public void write(Kryo kryo, Output output, Entity entity) {

    ... other field serialization here ...

    MapSerializer<HashMap> mapSerializer = (MapSerializer<HashMap>) kryo.getSerializer(HashMap.class);

    mapSerializer.setKeyClass(String.class);

    mapSerializer.setKeysCanBeNull(false);

    mapSerializer.setValuesCanBeNull(true);

    kryo.writeObject(output, entity.getPropertiesMap(), mapSerializer);

    ... other field serialization here ...

}

The above completes without error (with minlog.TRACE enabled)

public Entry read(Kryo kryo, Input input, Entity entity) {

    ... other field de-serialization here ...

    MapSerializer<HashMap> mapSerializer = (MapSerializer<HashMap>) kryo.getSerializer(HashMap.class);

    mapSerializer.setKeyClass(String.class);

    mapSerializer.setKeysCanBeNull(false);

    mapSerializer.setValuesCanBeNull(true);

    HashMap<String, Object) propertiesMap = kryo.readObject(input, HashMap.class, mapSerializer);

    ... other field de-serialization here ...

}

[Nate]

That isn't an SSCCE. One thing stands out: you use kryo.getSerializer(HashMap.class) to get the serializer Kryo will use for HashMap instances, then you modify the settings and use it to write/read a map. You should do all your serializer configuration up front. Consider if some of your code serializes a HashMap, then your code runs which changes the config for all HashMaps.

[Steve S]

My apologies -- I did not actually intend for that to be an SSCCE.  At work, I develop on an enclave, and my internet machine is a thin client.  So it is not feasible for me to write a simple example that demonstrates my issue.  I just included a couple of snippets to see if I am using the right approach.

However, I have actually resolved all of my issues by scrapping the custom serializers and using bind annotations on all of the fields of my classes.  That worked immediately!  I figured that i would give Kryo the opportunity to handle everything itself and it looks like that was the best approach.  Then I registered all of the classes that are tied into the main class and a few others that Kryo wanted.

I am not sure if this is the way to get the best (or nearly best) performance, but it definitely works and I have not found any cases where there were errors.  I even used deflater streams and they worked on the first try.

I can leave it as-is, and I'm getting better performance than Jackson.  For something that takes Jackson 10 seconds, it takes Kryo approximately three seconds which is a nice boost.  Any thoughts on what else I might do to get even better performance?  Or am I getting the best bang-for-buck this way with the annotations?

Thanks again for all of your help!