Storing clojure lists and maps in Redis

[Shoeb Bhinderwala]

I am trying to use Redis as a data structure cache for my clojure
application. Does anybody have experience/code/ideas that can write/
read a clojure complex data structure to the Redis cache.

For example I have a list of maps as shown below:

(def m1
[{
"total" {:end_mv_base 721470021.02M, :ret_base -2.1510871798903652,
:val_added_base -15748655.52M, :adj_mv_base 758084903.7M,
:beg_mv_base 740767309.10M, :pct_contrib_fx
0.6974363304530005,
:ccy_spot 0.49639198233447535, :pct_contrib_base
-2.1510871798904274,
:sec_count 90, :ret_period_base -2.1523676719086393}

"55" {:end_mv_base 5513170.48M, :ret_base 7.038528325335092,
:val_added_base 362529.33M, :adj_mv_base 5541883.033M,
:beg_mv_base 5150641.15M, :pct_contrib_fx
0.004896503822395695,
:ccy_spot 0.004572016877500618, :pct_contrib_base
0.04959174749327096,
:sec_count 1, :ret_period_base 7.0385282034552965}

"45" {:end_mv_base 4265060.12M, :ret_base -1.1595615568738782,
:val_added_base -50036.20M, :adj_mv_base 4550627.682M,
:beg_mv_base 4315096.32M, :pct_contrib_fx
0.0045309069298927205,
:ccy_spot 0.003163500114766722, :pct_contrib_base
-0.00614820276001893,
:sec_count 1, :ret_period_base -1.1595616030211309}
}])

How can I store the above to Redis and read it back to recreate it?

Thanks
Shoeb

[Daniel Pittman]

On Tue, Jan 3, 2012 at 22:30, Shoeb Bhinderwala
<shoeb.bh...@gmail.com> wrote:

> I am trying to use Redis as a data structure cache for my clojure
> application. Does anybody have experience/code/ideas that can write/
> read a clojure complex data structure to the Redis cache.

[…]

> How can I store the above to Redis and read it back to recreate it?

Personally, I would use JSON, MesssagePack, Protocol Buffers, or
something else language independent to encode the data. This has two
advantages: one, someone else already worked out how to do it, and
ported it to Java or Clojure, and two, you can also access it from
some other language. (When, eventually, you find you need that.)

Daniel
--
♲ Made with 100 percent post-consumer electrons

[Baishampayan Ghose]

Shoeb,

What about storing it as a string? You can either use pr-str or
data.json/generate-string.

You can then read it back using read-string or the equivalent json fn.

Regards,
BG

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

--
Baishampayan Ghose
b.ghose at gmail.com

[Softaddicts]

We use Clojure's reader representation to serialize data structures as strings here and we
still have some Java and JRuby code around.

Previously we were using YAML (json was in it's infancy when we opted for YAML).
A real pain in the ass... Now we use a couple of protocols callable from
every where to decipher messages in an unified way crossing languages
and representations (from/to low level representations, strings or persistent structures).

It's faster, shorter, much more transparent, easier to debug and elegant.

Luc

> Shoeb,
>
> What about storing it as a string? You can either use pr-str or
> data.json/generate-string.
>
> You can then read it back using read-string or the equivalent json fn.
>
> Regards,
> BG
>
> On Wed, Jan 4, 2012 at 12:00 PM, Shoeb Bhinderwala
> <shoeb.bh...@gmail.com> wrote:
> > I am trying to use Redis as a data structure cache for my clojure
> > application. Does anybody have experience/code/ideas that can write/
> > read a clojure complex data structure to the Redis cache.
> >
> > For example I have a list of maps as shown below:

>sing >

Softaddicts<lprefo...@softaddicts.ca> sent by ibisMail!

[Shoeb Bhinderwala]

Daniel, Baishampayan -

Thanks for the suggestion.

I was able to do this extremely easily using the JSON library. I used
the json-str and read-json functions.

Shoeb

[Peter Taoussanis]

read/pr-str works well, but it's painfully slow relative to something
like Redis. JSON libraries and the like would be faster, but might
require more contortions if you're using lots of Clojure data types.
My 2c: the best overall compromise atm is the Deep-Freeze
serialization library (https://github.com/halgari/deep-freeze). It
gets you very decent performance and great support for Clojure data
types.

You don't mention what Redis client you're using, but be aware that if
you're going to be going the binary serialization route, you'll want
to communicate with Redis via byte[]s rather than JVM strings. If
you're using Jedis, take a look at BinaryJedis.

Hope that helps!

--
Peter

[Sam Ritchie]

Here's an example of a Ring session store that writes clojure maps to and from Redis:

https://github.com/wuzhe/clj-redis-session/blob/master/src/clj_redis_session/core.clj

In addition to Deep Freeze, you might check out Kryo, accessible via Alex Miller's Carbonite library. I use Kryo in both Cascalog and ElephantDB for Clojure data serialization, and it's been working out wonderfully. Here are some kryo benchmarks if you're interested.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

--

Sam Ritchie, Twitter Inc

703.662.1337

@sritchie09

(Too brief? Here's why! http://emailcharter.org)

[Shoeb Bhinderwala]

Hi Peter -
I looked at deep-freeze but did not quite understand how to use it.
I used the following to freeze my Clojure complex data structure -
results (map of list of maps) and persist to redis:
      (redis/hmset k k (deep-freeze/freeze-to-array results))
Then I tried to retrieve and thaw it as follows:
      (deep-freeze/thaw-from-array (redis/hget k k))
The thaw gives me the following exception:
   java.lang.String cannot be cast to [B - (class
java.lang.ClassCastException)
What am I doing wrong?
ThanksShoeb

On Jan 4, 10:55 am, Peter Taoussanis <ptaoussa...@gmail.com> wrote:
> read/pr-str works well, but it's painfully slow relative to something

> likeRedis. JSON libraries and the like would be faster, but might

> require more contortions if you're using lots of Clojure data types.
> My 2c: the best overall compromise atm is the Deep-Freeze
> serialization library (https://github.com/halgari/deep-freeze). It
> gets you very decent performance and great support for Clojure data
> types.
>

> You don't mention whatRedisclient you're using, but be aware that if

> you're going to be going the binary serialization route, you'll want

> to communicate withRedisvia byte[]s rather than JVM strings. If

[Timothy Baldridge]

Can we get a complete code listing? Also what client are you using? It
looks as if your redis client is returning a string, and we're
expecting a byte array

Timothy

[Shoeb Bhinderwala]

Hi Tim

I am using redis-clojure client: https://github.com/tavisrudd/redis-clojure

Below is the complete code listing. The thaw invocation gives me the
error:

java.lang.String cannot be cast to [B - (class
java.lang.ClassCastException)

------ code ------------
(ns my-app
(:require [redis.core :as redis] [deep-freeze.core :as df])
(:use [clojure.tools.logging :only (debug info)]))

(defn thaw [k]
(redis/with-server {:host "dev-1" :port 6379 :db 0}
(let [j (redis/hget k k)]
(if (nil? j)
(do
(info (str "Not found in redis cache " k))
nil)

(do
(info (str "Thawed from redis cache " k))
(df/thaw-from-array j))))))

(defn freeze [k results]
(redis/with-server {:host "dev-1" :port 6379 :db 0}
(redis/hmset k k (df/freeze-to-array results))))

On Jan 5, 3:14 pm, Timothy Baldridge <tbaldri...@gmail.com> wrote:
> Can we get a complete code listing? Also what client are you using? It

> looks as if yourredisclient is returning a string, and we're

[Peter Taoussanis]

Yup, Timothy is correct.

Basically Redis's native datatype is a bytestring: http://redis.io/topics/internals-sds.
This is actually more like a JVM ByteArray than String, so libraries
like Jedis (which take Strings), do some coercions for you.

clj-redis uses Jedis underneath, so it expects Strings. But deep-
freeze uses byte[]s, so these coercions are what's causing you
trouble.

You should be able to test this by trying (redis/hmset k k (String.
(deep-freeze/freeze-to-array results)) to freeze and (deep-freeze/thaw-
from-array (.getBytes (redis/hget k k)) to thaw. That should work, but
it's inefficient because you're coercing from byte[]s to String, then
Jedis is coercing from String back to byte[]s.

A better solution would be to communicate with Redis directly with
byte[]s in the first place. This is possible using BinaryJedis, but
there's no support in clj-redis for it yet. I've forked clj-redis and
am slowly doing some experiments here: https://github.com/ptaoussanis/clj-redis
(see binset/binget), but that's FAR from being useable now.


The real problem here is clj-redis not being a good fit for binary
serialization at this point. That will probably improve in time, so if
you'd prefer to stay with clj-redis and you're not in a hurry for
performance I'd probably stick with read/pr-str for now to keeps
things simple.

If you really need the extra performance right now, you'll need to
interface with BinaryJedis directly for values you want to freeze/
thaw.

Hope that helps a bit at least,

--
Peter

[Peter Taoussanis]

Oh wow, sorry- I didn't see your reply in time and for some reason
figured you were using clj-redis.

This is actually easier since (if I recall correctly) redis-clojure is
able to write byte[]s and has an as-bytes macro for reading.

So you'd want something like this (untested):

(defn thaw [k]
(redis/with-server {:host "dev-1" :port 6379 :db 0}

(let [j (redis/as-bytes (redis/hget k k))]

(if (nil? j)
(do
(info (str "Not found in redis cache " k))
nil)
(do
(info (str "Thawed from redis cache " k))
(df/thaw-from-array j))))))

(defn freeze [k results]
(redis/with-server {:host "dev-1" :port 6379 :db 0}
(redis/hmset k k (df/freeze-to-array results))))

--
Peter

[Shoeb Bhinderwala]

Thanks a lot Peter. Worked great! I did some rudimentary bench marking
for large data sets and found deep-freeze to be 10 times faster on
average compared to JSON serialization. That is really a huge
performance difference.

[Shoeb Bhinderwala]

Peter/Tim -

Also want to commend you on this amazing high performant library (deep-
freeze) that you have written.

Shoeb

On Jan 6, 3:05 am, Shoeb Bhinderwala <shoeb.bhinderw...@gmail.com>
wrote:

[Peter Taoussanis]

Tim's the one to thank: I didn't do much :)

> I did some rudimentary bench marking for large data sets and found deep-freeze to be 10 times faster on
> average compared to JSON serialization. That is really a huge
> performance difference.

Some final comments if performance really is a major factor for you:

1. You should compare deep-freeze with and without compression in your
real environment with real data. Compression hits writing speeds more
than reading speeds but may or may not still get you a net win
depending on your particular environment since compression decreases
the amount of data going around- especially for large payloads.

2. In my tests, redis-clojure was significantly slower than clj-redis
due to its protocol implementation. Being built on Jedis, I expect clj-
redis to maintain better performance characteristics going into the
future (particularly if you end up one day wanting bleeding-edge stuff
like clustering, etc.).

3. I hadn't heard of Kryo before this thread so I haven't tested it-
but I wouldn't be surprised if a library wrapping something like Kryo
could still significantly beat deep-freeze performance. If time
allows, I'll try look into this in future and see if deep-freeze
couldn't itself benefit from using something similar.

Cheers,

--
Peter

[Sam Ritchie]

Peter,

Check out Carbonite for a great wrapper library around Kryo. Here are the API tests, with round trip examples: https://github.com/revelytix/carbonite/blob/master/test/carbonite/test_api.clj.

Cheers,

Sam

--
Peter

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Alex Miller]

BTW, I have not done an actual release of Carbonite to put it out in
the repos but I hope to get around to that soon. I have been a bit
busy with Clojure/West. :) I plan to set it up in our Revelytix CI
server so it can be regularly tested as well.

On Jan 6, 12:27 pm, Sam Ritchie <sritchi...@gmail.com> wrote:
> Peter,
>
> Check out Carbonite for a great wrapper library around Kryo. Here are the

> API tests, with round trip examples:https://github.com/revelytix/carbonite/blob/master/test/carbonite/tes...
> .
>
> Cheers,
> Sam

[Murphy McMahon]

It sounds like maybe you've already solved your problem, Shoeb, but
just in case:

The book _Clojure in Action_ has a section on data mapping and data
persistence using Redis. You can download the source code here:
http://www.manning.com/rathore/source-code.zip

(The relevant files are chapter12_redis_datamapper.clj and
chapter12_redis_persistence.clj)

M