redis as backend for mappa

11 views
Skip to first unread message

chris

unread,
Dec 28, 2010, 9:50:33 PM12/28/10
to Mappa - Topic Maps
Hi there,

I am very interested in getting a working storage backend for mappa,
but I have no idea where to start or to look. Would something like
redis (see http://redis.io; key/value store) or CouchDB (JSON based
map/reduce interface) be a good match?

Best wishes,

Christian

Lars Heuer

unread,
Jan 6, 2011, 6:35:35 AM1/6/11
to chris
Hi Christian

> I am very interested in getting a working storage backend for mappa,
> but I have no idea where to start or to look.

Sorry for the delay.

Yes, a working, persistent backend for Mappa would be cool. I have the
need for a backend which runs under Google App Engine since I have a
project which should use GAE and Mappa.

A generic persistent backend which runs outside of GAE would be
useful addition.

> Would something like redis (see http://redis.io; key/value store) or
> CouchDB (JSON based map/reduce interface) be a good match?

I've never done something with Redis and my CouchDB experience is
limited, but I'd favor CouchDB. A Topic Maps engine which uses CouchDB
exists in theory already at [1] (paper at [2]).

To answer your question "where to look at": The memory backend [2]
defines the neccessary methods. Each backend must specify a

def create_connection(**kw):
# [...]
return MyConnection()

method. Mappa should discover the backend automatically by using entry
points.

The memory backend uses:

entry_points = """
[mappa.store]
mem = maplin.store.mem
"""

If the user uses

import mappa

conn = mappa.connect(backend='couchdb')

Mappa would try to discover a storage plugin which is named "couchdb".

Maybe you want to wait for the GAE backend which should make things
clearer?


[1] <https://code.google.com/p/couchtm/>
[2] <http://tmra.de/2009/talks/CouchDB_and_Topic_Maps>
[3]
<https://code.google.com/p/mappa/source/browse/#svn%2Fmappa-store-mem%2Ftrunk%2Fsrc%2Fmaplin%2Fstore%2Fmem%253Fstate%253Dclosed>

Best regards,
Lars
--
Semagia
<http://www.semagia.com>

Christian Wittern

unread,
Jan 6, 2011, 6:07:09 PM1/6/11
to ma...@googlegroups.com
Hi Lars,

On 6 January 2011 20:35, Lars Heuer <he...@semagia.com> wrote:
>
> Sorry for the delay.

No problem. With open source you get what you pay for and I am
already using of it whatever I can:-)


> Yes, a working, persistent backend for Mappa would be cool. I have the
> need for a backend which runs under Google App Engine since I have a
> project which should use GAE and Mappa.

Cool. This might actually be useful as is as some kind of
tm-in-the-cloud service.

>
> A generic persistent backend which runs outside of GAE would be
> useful addition.

Quite true.

>
>> Would something like redis (see http://redis.io; key/value store) or
>> CouchDB (JSON based map/reduce interface) be a good match?
>
> I've never done something with Redis and my CouchDB experience is
> limited, but I'd favor CouchDB. A Topic Maps engine which uses CouchDB
> exists in theory already at [1] (paper at [2]).

Yeah, I came across that, but that is only hot air or so it seems.
Thinking of it a bit more, I think actually Redis is a better fit,
since topic map data are in fact little information slices and not so
much documents.
[useful explanation skipped]


>
> Maybe you want to wait for the GAE backend which should make things
> clearer?

Well, for the moment I will wait for what you come up with. If
anything, I have the suspicion that it will be easier to go from
there.

ALl the best,

Christian

--
Christian Wittern, Kyoto

Lars Heuer

unread,
Jan 6, 2011, 6:45:45 PM1/6/11
to Christian Wittern
Hi Christian,

[...]


> No problem. With open source you get what you pay for and I am
> already using of it whatever I can:-)

Yeah, unfortunately, you're interested in Open Source *and* Topic
Maps. So, you're lost ;)


[...]


> Yeah, I came across that, but that is only hot air or so it seems.
> Thinking of it a bit more, I think actually Redis is a better fit,
> since topic map data are in fact little information slices and not so
> much documents.
> [useful explanation skipped]

[...]

Since I favor CouchDB I'd be interested in your thin air and high
hopes explanations. :) Seriously: I'd be interested in your thoughts
since I favor CouchDB because it's available at all
platforms/programming languages while Redis isn't that popular (and
supported) acc. to my understanding. If Mappa supports CouchDB it
won't make the same mistakes of CouchTM, of course. ;)

Christian Wittern

unread,
Jan 6, 2011, 7:26:08 PM1/6/11
to ma...@googlegroups.com
On 2011-01-07 08:45, Lars Heuer wrote:
> Hi Christian,
>
> [...]
>> No problem. With open source you get what you pay for and I am
>> already using of it whatever I can:-)
> Yeah, unfortunately, you're interested in Open Source *and* Topic
> Maps. So, you're lost ;)
That might be true, but I am not giving up so soon:-)

> Since I favor CouchDB I'd be interested in your thin air and high
> hopes explanations. :) Seriously: I'd be interested in your thoughts
> since I favor CouchDB because it's available at all
> platforms/programming languages while Redis isn't that popular (and
> supported) acc. to my understanding. If Mappa supports CouchDB it
> won't make the same mistakes of CouchTM, of course. ;)
>

Many years ago, I made the mistake to assume a XML database would be a good
fit for Topicmaps since they are expressed in XML (that was at the XTM 1.0
stage). The same type of thinking might favour Couchdb, since there is a
JSON format for TM and it would be straightforward to map it to CouchDB.

But, as I do not need to tell you, the problems are on quite a different
level, so here is a random list of things that I see in favor of redis:

* very fast key/value store (with value datatypes being strings, hashes,
lists, sets and sorted sets)
* C (or maybe C++) with very few dependencies, trivial to compile on almost
any platform
* custom protocol that is easy to implement
* no need for server (db) side programming (in CouchDB, you would need to
program the views etc. for the map/reduce on the server, as far as I know)
* excellent Python bindings (but that is also true fo CouchDB).

So in as far as a Topicmap can be expressed as a series of key/value
statements, to me it looks like it will be easier and more useful to have a
backend based on redis, whereas I see the value for CouchDB in the way you
can program a complete web application there directly on the server. In
that sense, not as a backend for mappa, but as a topicmap server in its own
right, it might be very useful, but that would be a different project.

All the best,

Christian

Lars Heuer

unread,
Jan 8, 2011, 1:42:16 PM1/8/11
to Christian Wittern
Hi Christian,

[...]


> Many years ago, I made the mistake to assume a XML database would be a good
> fit for Topicmaps since they are expressed in XML (that was at the XTM 1.0

We've all been there. :)


> stage). The same type of thinking might favour Couchdb, since there is a
> JSON format for TM and it would be straightforward to map it to CouchDB.

Well, the mapping to CouchDB wouldn't necessarily look like JSON Topic
Maps [1]. Well, I am saying that without having any clue how the
mapping would look alike.


> But, as I do not need to tell you, the problems are on quite a different
> level, so here is a random list of things that I see in favor of redis:

[...]

Thanks for sharing your thoughts. I take a look at Redis after I
finished/started(!) my GAE backend. Native support for datatypes like
sets and lists sounds interesting. Maybe it would be possible to
create a generic key/value store layer that could support other
key/value stores as well.

Lars Heuer

unread,
Feb 18, 2011, 3:24:46 PM2/18/11
to chris
[...]

> but I have no idea where to start or to look. Would something like
> redis (see http://redis.io; key/value store) or CouchDB (JSON based
> map/reduce interface) be a good match?

FYI: MaJorToM's Redis schema:
<http://docs.topicmapslab.de/majortom/redis.html>

Christian Wittern

unread,
Feb 23, 2011, 7:57:21 PM2/23/11
to ma...@googlegroups.com
On 2011-02-19 5:24, Lars Heuer wrote:
> [...]
>
>> but I have no idea where to start or to look. Would something like
>> redis (see http://redis.io; key/value store) or CouchDB (JSON based
>> map/reduce interface) be a good match?
>>
> FYI: MaJorToM's Redis schema:
> <http://docs.topicmapslab.de/majortom/redis.html>
>

This sounds interesting and well thought through. Now we only need to
port this to mappa!

Christian

Reply all
Reply to author
Forward
0 new messages