Recommendation for Java serialization libraries for Redis

[Ritesh Tijoriwala]

Hi,

Are there any java serialization libraries around which help me convert complex Java objects to binary/json format so I can store them in Redis as either byte[] or String? Some pointers/insight into their performance will also be helpful.

Regards,

Ritesh

[Marc Gravell]

protobuf (protocol buffers) would be my choice, although there are **PLENTY** of alternatives. We use protobuf for storing .NET objects in redis. To squeeze an extra ounce or two out, if the message is non-trivial in size, we also do a speculative compress (text data tends to compress nicely). I haven't done any java tests, but for .NET protobuf is basically the fastest and smallest-output serializer/deserializer that I'm aware of. Plus because it is not language/platform-specific, it would be trivial to store from .NET, and load from java / php / c++ / whatever you want.

Human readable (json etc) is all well and good, but for my purposes: I'd rather have smaller.

Marc

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/C0yTUa7ovhQJ.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
Regards,

Marc

[Salvatore Sanfilippo]

Question: are you sure you want to store things serialized into Redis
instead of *mapping* your data into Redis?

Example, if you want to store an User inside Redis, maybe mapping it
into a Redis hash with the right fields is a better approach compared
to opaque serialization.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/C0yTUa7ovhQJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.



--

Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Marc Gravell]

I guess it depends on the scenario. Hashes will work great for flat objects, but in a complex structure (sub-objects, lists, etc) it could take a lot of hits to redis to reconstruct even a moderate graph. Which is not a criticism of hashes - they have many great uses: but there are other cases where you just want to store and retrieve a graph as simply and quickly as possible, without necessarily trying to model the structure completely in the back-end-store's terms - serialization is great for that. Yes, it is opaque, but it is also insanely fast and simple.

Regards,

Marc

[Salvatore Sanfilippo]

Serializing is great indeed sometimes, I perfectly understand your
reasoning, but I tend to use it only when I just need to cache.
The reason is that I'm a bit worried, for real data, to store those
data in any language / implementation dependent way.
I guess there is no right/wrong way to do it but is a tradeoff... :-)

Salvatore

[Ritesh Tijoriwala]

Thanks Salvatore and Marc.

Question: are you sure you want to store things serialized into Redis
instead of *mapping* your data into Redis?

Example, if you want to store an User inside Redis, maybe mapping it
into a Redis hash with the right fields is a better approach compared
to opaque serialization.

Are there any examples of this that I can look into? Sorry I am a bit new to Redis and getting up to speed.

Thanks,

Ritesh 

[Marc Gravell]

Perhaps at the high level:

# write a record with multiple hash fields

hmset users/123 name Fred id 123 ssn 012345 addr "12 foo lane"

# write a single hash field

hset users/123 name Freddy

# get all fields of that user

hgetall users/123

# get specified fields of that user

hmget users/123 name addr

see: http://redis.io/commands#hash

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/pwN9GmB290YJ.

To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
Regards,

Marc

[Colin Vipurs]

This is something we do for short term storage (in the order of a few
minutes) and just use Jackson to serialize to Json. One thing to bear
in mind with this approach is that if you can't just dump all your
data you'll want to put in place some kind of versioning so that if
your data structure changes you can roll this out live.

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/redis-db/-/C0yTUa7ovhQJ.

> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.



--

Maybe she awoke to see the roommate's boyfriend swinging from the
chandelier wearing a boar's head.

Something which you, I, and everyone else would call "Tuesday", of course.

[Javier Guerra Giraldez]

On Thu, Nov 22, 2012 at 4:36 AM, Marc Gravell <marc.g...@gmail.com> wrote:
> protobuf (protocol buffers) would be my choice, although there are
> **PLENTY** of alternatives.

message pack has the advantage that it's readable by server-side Lua scripts.

But I think many Java people would like the static-typing nature of protobuf.

Of course, like Salvatore already said, first consider seriously not
to serialize the object, but instead using Redis structures. Usually
that means HASH for most of the object fields, ZSETs for indexes, SETs
for tags, etc.

--
Javier

[v...@torkit.ru]

Protobuf for binary, quick serialization
Gson for JSON
JAXB for XML

[Dvir Volk]

Thrift is also a good option for fast binary serialization, but I don't usually serialize objects, I create them on redis.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--

Dvir Volk

Chief Architect, Everything.me

http://everything.me 

[Vasily Vasilkov]

Thrift is also a good option for fast binary serialization, but I don't usually serialize objects, I create them on redis.

О©╫

Agree, Thrift is a good choice too, but I've found the protobuf documentation a much more better.

[Bret A. Barker]

+1 for Jackson as the best Java -> JSON serializer around.

On Sat, Dec 01, 2012 at 10:32:23AM -0800, Manish Malik wrote:
>
> We have found Jackson (https://github.com/FasterXML/jackson-databind) to be
> the most apt, and FlexJSON (http://flexjson.sourceforge.net/) a close
> second. We chose Jackson as we store objects interchangeably in Redis and
> MongoDB, and there are usable projects for that as well
> (jackson-mongo-mapper), hence things stay flexible.
>
> Hope that helps.
>
> Best,
> Manish
>

[Vincent Nonnenmacher]

Have you take a look at Avro ?

its used in HBase and is now a separated project in Apache tree : http://avro.apache.org/docs/current/

there is comparison there between Avro, Thrift and Proto Buf

and a very good discussion article on it by the marvelous Ilya Grigorik here : http://www.igvita.com/2011/08/01/protocol-buffers-avro-thrift-messagepack/

along with some benchmark links.

[Vincent Nonnenmacher]

Sound like MessagePack is already integrated with redis through scripting as there is a C extension for Lua.

[Pawel Veselov]

On Sat, Dec 1, 2012 at 4:41 PM, Bret A. Barker <br...@abitrandom.net> wrote:

+1 for Jackson as the best Java -> JSON serializer around.

I've been using google-gson, and am quite happy with it.