Clojure + Redis

[Gabi]

On first look, Redis and Clojure seems to be a perfect match. They
both handle sets and maps efficiently. If one could find an easy way
to store and retrieve Clojure data structures to Redis (even a small
subset- just a list or a set), a distributed clojure app could be very
easy (and effective?) thing to do - The stateless Clojure nodes would
share and operate on the same central data structure which is stored
in Redis). What do you thing ? Is it worth investigating further?

[Robert Campbell]

I think anything which lowers the impedance mismatch between Clojure
data structures and a persistent store is worth investigating. I'd
love to find an ACID, transactional store which accepts native
structures. Right now I'm using CouchDB, and while JSON is close
enough, it still requires a mapping between the Clojure and Couch
worlds. Not supporting set, and not allowing fields with dashes
(JavaScript thinks it's a minus) are some of the annoyances.

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

[Vagif Verdi]

There are other NoSQL datastores written in java, like Voldemort.
Perhaps if you investigate them, you will find one that will be much
easier to integrate with clojure.

[Steve Purcell]

Not sure if it's any help, but here's a variant of memoize I wrote, which stores arbitrary readable/printable objects to redis:

http://gist.github.com/266689

(If there's any interest, I'll wrap it up in a github project and push it to clojars.)

Redis isn't a hierarchical store, so its array/set operations would only benefit the most shallow of data structures.

-Steve

[Constantine Vetoshev]

On Dec 30, 10:40 am, Robert Campbell <rrc...@gmail.com> wrote:
> I think anything which lowers the impedance mismatch between Clojure
> data structures and a persistent store is worth investigating. I'd
> love to find an ACID, transactional store which accepts native
> structures.

Have you looked at Cupboard (http://github.com/gcv/cupboard)? It tries
to make it easy to store native Clojure data structures, and it is as
ACID and transactional as the underlying Berkeley DB JE
implementation. If you have looked at it, I'd appreciate some feedback
as to where it fell short of your expectations. :)

I plan to start adding distributed storage support in the next few
months. The latest JE release has master-slave replication, which I
think will work well for Cupboard.

[Gabi]

Yes. I think it is of much interest. What if I stored a shared data
structure in redis (only because its the fastest), using your memoize
variant, and process (maybe even updated it) it in parallel from
different Clojure nodes. Some kind of primitive map/reduce mechanism I
think.

[jem]

Something else to look at might be the Apache Jackrabbit project at
http://jackrabbit.apache.org/.

I've been looking at tools along these lines as well, and recently
looked at Redis for the same reasons. Right now, though, I'm focusing
my attention on Jackrabbit which is an implementation of the JSR170
repository specifications. It supports hierarchical data and lists.
I was looking for some way to save dynamic objects (basically runtime
defined maps) and this looks like it might work.

JSR170 defines a repository as:

"A content repository consists of one or more workspaces, each of
which contains a tree of items. An item is either a node or a
property. Each node may have zero or more child nodes and zero or more
child properties. There is a single root node per workspace, which has
no parent. All other nodes have one parent. Properties have one parent
(a node) and cannot have children; they are the leaves of the tree.
All of the actual content in the repository is stored within the
values of the properties."

The available property types cover what appear to be all the bases,
including references to other nodes to prevent cycles when
serializing. It can support annotated Java objects or use the Node
building capability directly and allows querying similar to XPath.
Since it is a Java project, using it from Clojure should be trivial to
use.

[Steve Purcell]

There you go: http://github.com/purcell/redis-memo

I doubt the memoize functions provided therein will be directly useful to you, but you may find a few lines of helpful code there. Best of luck with your experiment.

-Steve

[Gabi]

Jackrabbit is heavy. It might be powerful but I am sure it is much
slower than Redis or MongoDB.

On Dec 31 2009, 6:59 pm, jem <jere.mcdev...@gmail.com> wrote:
> Something else to look at might be the Apache Jackrabbit project athttp://jackrabbit.apache.org/.

[Julian Morrison]

I've just recently been poking around these NoSQLs investigating their
features, so...

Redis has limited data structures - flat un-nested lists and sets, and
plain strings. It doesn't have sets exactly - just keys and values.
Nothing nested at all, unless you serialize to strings. No indexes,
although you can hack up your own.

To be honest, Redis isn't that impressive versus what's in Clojure
already. It's an in-memory DB (so it's not much different from ref
+dosync) and it intermittently spits a snapshot to disk. If you can
live with an in-process DB, you could copy (and exceed) its features
including snapshot saving in a page of pure Clojure code, and beat it
on speed too.

Contrast MongoDB: slower because it bothers to save things, but still
around twice as fast as MySQL and much faster than CouchDB (cite: the
benchmarks page). Arbitrarily nested collections, indexes, atomic
updates (in place operations like inc and append, or atomic compare-
and-set), JSON syntax, typed data, replication (built in) and sharding
(via a broker process).

(MongoDB downsides: it grows files in a very greedy way to try and
minimize data fragmentation, and it needs a 64bit machine to store
more than about 2Gb.)

[Julian Morrison]

Oops, typo - I meant, doesn't have hashes.

[Gabi]

I am interested in the idea: Completely stateless set of Clojure nodes
(on many machines), operating on a central state stored in some
datastore.
If transactions could be managed somehow, I think it would be very
compelling model for many applications.

[Tom Hicks]

Have you looked at Neo4J? I have no experience with it but
someone in the forum just announced a Clojure wrapper for it:

http://groups.google.com/group/clojure/browse_thread/thread/9628c622784ff45a#
cheers,
-t

[Gabi]

How about congomongo (http://github.com/somnium/congomongo) ?
Have anybody used it ? Seems good choice for storing state in central
location..

On Jan 4, 2:40 am, Tom Hicks <hickstoh...@gmail.com> wrote:
> Have you looked at Neo4J? I have no experience with it but
> someone in the forum just announced a Clojure wrapper for it:
>

> http://groups.google.com/group/clojure/browse_thread/thread/9628c6227...

[Shantanu Kumar]

On Jan 2, 5:12 am, Gabi <bugspy...@gmail.com> wrote:
> I am interested in the idea: Completely stateless set of Clojure nodes
> (on many machines), operating on a central state stored in some
> datastore.
> If transactions could be managed somehow, I think it would be very
> compelling model for many applications.

Do you mean distributed transactions?

[Gabi]

Maybe, though I would avoid distributed transactions as much as
possible. They are complex and slow creatures.

[Gabi]

What if I wanted to use Redis just persist binary (serialized) clojure
objects ?
What's the easiest (and fastest) way to serialize/de-serialize vectors
or lists in Clojure ? (so the can stored as blobs in Redis)

[Steve Purcell]

Read the code I posted in this thread and put up on github after you expressed interest.

That's part of what it does, using the reader/printer representation.

Alternatives would include standard Java binary serialisation or 3rd party libraries (Hessian/Burlap?).

-Steve

[anders nawroth]

There's actually been some activity from different people regarding
Clojure + Neo4j, see:
http://wiki.neo4j.org/content/Clojure

/anders

On Jan 4, 1:40 am, Tom Hicks <hickstoh...@gmail.com> wrote:
> Have you looked at Neo4J? I have no experience with it but
> someone in the forum just announced a Clojure wrapper for it:
>

> http://groups.google.com/group/clojure/browse_thread/thread/9628c6227...

[Gabi]

I think you should do "(binding [*print-dup* true] (pr-str value).."
instead of just (pr-str value) in the encode-value function. (line 20
in redis_memo.clj)

[Steve Purcell]

Indeed, thanks - I realized that earlier today myself!

-Steve

[Steve Purcell]

Or rather, more importantly, I realised I should bind *print-level* to nil...

As for *print-dup*, I guess there's a danger that stored values might become unreadable if any implementation structure classes were to get renamed in a future clojure release.

-Steve

On 5 Jan 2010, at 09:24, Gabi wrote: