Redis as a graph database

[Mason Jones]

Hello all,

We're starting to do a little looking into graph databases, as we have
relationships between some data objects that could probably be nicely
modeled using something like neo4j. However, it also got me thinking
about using Redis as a graph database, with sets describing "edges"
between nodes/elements. I wondered if anyone here has done any work
like this, and what you think about it...

Thanks for any thoughts.

[Pieter Noordhuis]

Hi Mason,

Not sure if users did something like this before. You're right that this can be modeled pretty well using Redis, but is also has some problems. Lets say you store all the outedges for all nodes in the graph. To get the inedges for a specific node, you would have to either traverse all sets, or duplicate your data. This is one problem that comes to mind, but there are probably more ;-).

Cheers,

Pieter

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Keenan Brock]

Hi Mason.

You may want to check out http://github.com/twitter/flockdb (twitter
graph solution) and Neo4J to get some ideas.

--Keenan

> > redis-db+u...@googlegroups.com<redis-db%2Bunsubscribe@googlegroups.c om>
> > .

[Mason Jones]

On Tue, Jun 15, 2010 at 11:57 PM, Pieter Noordhuis
<pcnoo...@gmail.com> wrote:
> Hi Mason,
> Not sure if users did something like this before. You're right that this can
> be modeled pretty well using Redis, but is also has some problems. Lets say
> you store all the outedges for all nodes in the graph. To get the inedges
> for a specific node, you would have to either traverse all sets, or
> duplicate your data. This is one problem that comes to mind, but there are
> probably more ;-).

Yes, that's an interesting problem. As you note, I suppose that it's
possible using more sets could make it work, but the duplication of
data might make the dataset grow too large too quickly. I've been
contemplating whether there are any tricks to make use of the set
operators (intersection, etc) to work the edges bidirectionally
without duplicating everything... We'll see!

[Mason Jones]

On Wed, Jun 16, 2010 at 2:07 PM, Keenan Brock <kee...@thebrocks.net> wrote:
> Hi Mason.
>
> You may want to check out http://github.com/twitter/flockdb (twitter
> graph solution) and Neo4J to get some ideas.

Indeed, Neo4J seems to be the best solution out there (and
neo4jr-social provides some nice features out of the box for those of
us using Ruby), but since we're already using Redis in our world, it
would be great to use it. I like to try to minimize the moving parts
when I can. Plus it's simply an interesting question!

> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

[Rolly]

Hi Mason,

I've been playing around with graph structures in Redis, it's working
well. I'm using hashes for node / edge metadata, and sets for edge
ids. I'm still looking into what the best methods of doing graph
traversal, search, and filtering are. Maybe indexing the data using
the Sphinx Search engine (or similar) might be a good approach for
searching and filtering?

[Mason Jones]

Hello, thanks for the reply about this. You say that you're using sets
for the edge ids, but I'm not quite sure what you mean by that: could
you explain further? Are you duplicating data to handle the
bidirectional nature of the graphs (as Pieter mentioned earlier)?

For searching, I would expect that using something designed for that
purpose like Sphinx, Solr, etc would be the best bet. For filtering,
it's hard to say -- depends on your precise use cases, I would think.

Thanks again -- I'd love to see any further details you can share.

[Josiah Carlson]

On Wed, Jun 16, 2010 at 5:08 PM, Mason Jones <maso...@gmail.com> wrote:
> On Tue, Jun 15, 2010 at 11:57 PM, Pieter Noordhuis
> <pcnoo...@gmail.com> wrote:
>> Hi Mason,
>> Not sure if users did something like this before. You're right that this can
>> be modeled pretty well using Redis, but is also has some problems. Lets say
>> you store all the outedges for all nodes in the graph. To get the inedges
>> for a specific node, you would have to either traverse all sets, or
>> duplicate your data. This is one problem that comes to mind, but there are
>> probably more ;-).
>
> Yes, that's an interesting problem. As you note, I suppose that it's
> possible using more sets could make it work, but the duplication of
> data might make the dataset grow too large too quickly. I've been
> contemplating whether there are any tricks to make use of the set
> operators (intersection, etc) to work the edges bidirectionally
> without duplicating everything... We'll see!

It's only 2x as large on the graph side of things. If you're running
into issues where a factor of 2x will kill your application, you need
to work on how to scale your application (outgoing edges on one
machine, incoming edges on another is a very basic first step,
sharding is the next obvious step; be careful about writing to
multiple shards at the same time). Incidentally, if you only care
about twitter-style "following" relationships, you could have up to 1
entry in each of 4 sets for a bi-directional follow relationship: x
follows y, y is being followed by x, y follows x, x is being followed
by y... which will grow pretty fast for any nontrivial graph.

Regards,
- Josiah

[Roi Lipman]

You can try redis graph module, it is a Redis module as the name implies, which means it’s runs within Redis (so hopefully decent performance).

It also exposes a simple query language which is based on Neo4J’s Cypher, so those who are familiar with it can get started quickly.