serialization and deserialization of []byte

[rakesh...@gmail.com]

Hi,

I have a doubt where if I am writing proto encoded data which will be []byte in golang, will the hazelcast serialize this again? 

[Neil Stevenson]

By default it won't.

When your Go routine does "Put(key1, value1)" both the key and value have to be serialized somehow to send across the network.

How this is done will depend on the type of the struct, but if you're using proto then Protobuf will do the serialization to turn this into an array of bytes for sending.

By default, Hazelcast will store your data in the format transmitted, an array of bytes.

It has no need at this point to deserialize. (1)

When you go "Get(key1)" the key has to be serialized to send to Hazelcast, using the same method as before.

At this point Hazelcast just needs to match "key1" against "key2" to find your data, by comparing bytes.

It has no need at this point to deserialize. (2).

If a match is found, "value1" would be returned if it's a match, but "value1" is already serialized as an array of bytes so is ready to transmit.

It has no need at this point to serialize. (3).

Basic operations don't require Hazelcast to have any knowledge of your structs, at points (1), (2) and (3) above. 

If these are the kind you use, then Hazelcast has no need for the Protobuf definitions for your types.

If you use them, more advance operations will require Hazelcast to have these definitions.

For example, you may want to find the maximum value of something object, which requires Hazelcast to understand the internal structure.

This is the design trade-off. 

It is more efficient to scan the objects on Hazelcast when they're stored than to retrieve them all to your Go client just to find one match, but this requires your type definitions to exist on the server-side.

Hazelcast on the server-side is Java, and Protobuf can generate Java from your "*.proto" definitions, so this really shouldn't be difficult.

It requires a little Java knowledge but not much. If you get stuck, just ask.

N

[Yuce Tekol]

Hi,

You can check out this sample which serializes/deserializes Avro encoded data: https://gist.github.com/yuce/c904166de1eb9240f19a342b34efb32a and adapt it to work with Protobuf.

Note that Hazelcast Go Client supports only Hazelcast IMDG v3 at the moment, but we are working on v4 support.

Let me know if you have any questions,

Yuce

[rakesh...@gmail.com]

Hi,

thanks for the clarification. 

Most of the time i will be doing get, put or predicate look up on Keys(like wildcard pattern matching on keys)

I will not be doing any predicate look up on values so hazelcast need not be aware of protobuf definition in this case.

[rakesh...@gmail.com]

Hi Yuce,

Thanks for the pointers. Any ETA for v4 support in golang?

[Yuce Tekol]

There is no public ETA, but I'll reply again once we have the time for an alpha version.