redis as backend for mappa
chris
I am very interested in getting a working storage backend for mappa,
but I have no idea where to start or to look. Would something like
redis (see http://redis.io; key/value store) or CouchDB (JSON based
map/reduce interface) be a good match?
Best wishes,
Christian
Lars Heuer
> I am very interested in getting a working storage backend for mappa,
> but I have no idea where to start or to look.
Sorry for the delay.
Yes, a working, persistent backend for Mappa would be cool. I have the
need for a backend which runs under Google App Engine since I have a
project which should use GAE and Mappa.
A generic persistent backend which runs outside of GAE would be
useful addition.
> Would something like redis (see http://redis.io; key/value store) or
> CouchDB (JSON based map/reduce interface) be a good match?
I've never done something with Redis and my CouchDB experience is
limited, but I'd favor CouchDB. A Topic Maps engine which uses CouchDB
exists in theory already at [1] (paper at [2]).
To answer your question "where to look at": The memory backend [2]
defines the neccessary methods. Each backend must specify a
def create_connection(**kw):
# [...]
return MyConnection()
method. Mappa should discover the backend automatically by using entry
points.
The memory backend uses:
entry_points = """
[mappa.store]
mem = maplin.store.mem
"""
If the user uses
import mappa
conn = mappa.connect(backend='couchdb')
Mappa would try to discover a storage plugin which is named "couchdb".
Maybe you want to wait for the GAE backend which should make things
clearer?
[1] <https://code.google.com/p/couchtm/>
[2] <http://tmra.de/2009/talks/CouchDB_and_Topic_Maps>
[3]
<https://code.google.com/p/mappa/source/browse/#svn%2Fmappa-store-mem%2Ftrunk%2Fsrc%2Fmaplin%2Fstore%2Fmem%253Fstate%253Dclosed>
Best regards,
Lars
--
Semagia
<http://www.semagia.com>
Christian Wittern
On 6 January 2011 20:35, Lars Heuer <he...@semagia.com> wrote:
>
> Sorry for the delay.
No problem. With open source you get what you pay for and I am
already using of it whatever I can:-)
> Yes, a working, persistent backend for Mappa would be cool. I have the
> need for a backend which runs under Google App Engine since I have a
> project which should use GAE and Mappa.
Cool. This might actually be useful as is as some kind of
tm-in-the-cloud service.
>
> A generic persistent backend which runs outside of GAE would be
> useful addition.
Quite true.
>
>> Would something like redis (see http://redis.io; key/value store) or
>> CouchDB (JSON based map/reduce interface) be a good match?
>
> I've never done something with Redis and my CouchDB experience is
> limited, but I'd favor CouchDB. A Topic Maps engine which uses CouchDB
> exists in theory already at [1] (paper at [2]).
Yeah, I came across that, but that is only hot air or so it seems.
Thinking of it a bit more, I think actually Redis is a better fit,
since topic map data are in fact little information slices and not so
much documents.
[useful explanation skipped]
>
> Maybe you want to wait for the GAE backend which should make things
> clearer?
Well, for the moment I will wait for what you come up with. If
anything, I have the suspicion that it will be easier to go from
there.
ALl the best,
Christian
--
Christian Wittern, Kyoto
Lars Heuer
[...]
> No problem. With open source you get what you pay for and I am
> already using of it whatever I can:-)
Yeah, unfortunately, you're interested in Open Source *and* Topic
Maps. So, you're lost ;)
[...]
> Yeah, I came across that, but that is only hot air or so it seems.
> Thinking of it a bit more, I think actually Redis is a better fit,
> since topic map data are in fact little information slices and not so
> much documents.
> [useful explanation skipped]
[...]
Since I favor CouchDB I'd be interested in your thin air and high
hopes explanations. :) Seriously: I'd be interested in your thoughts
since I favor CouchDB because it's available at all
platforms/programming languages while Redis isn't that popular (and
supported) acc. to my understanding. If Mappa supports CouchDB it
won't make the same mistakes of CouchTM, of course. ;)
Christian Wittern
> Hi Christian,
>
> [...]
>> No problem. With open source you get what you pay for and I am
>> already using of it whatever I can:-)
> Yeah, unfortunately, you're interested in Open Source *and* Topic
> Maps. So, you're lost ;)
> Since I favor CouchDB I'd be interested in your thin air and high
> hopes explanations. :) Seriously: I'd be interested in your thoughts
> since I favor CouchDB because it's available at all
> platforms/programming languages while Redis isn't that popular (and
> supported) acc. to my understanding. If Mappa supports CouchDB it
> won't make the same mistakes of CouchTM, of course. ;)
>
Many years ago, I made the mistake to assume a XML database would be a good
fit for Topicmaps since they are expressed in XML (that was at the XTM 1.0
stage). The same type of thinking might favour Couchdb, since there is a
JSON format for TM and it would be straightforward to map it to CouchDB.
But, as I do not need to tell you, the problems are on quite a different
level, so here is a random list of things that I see in favor of redis:
* very fast key/value store (with value datatypes being strings, hashes,
lists, sets and sorted sets)
* C (or maybe C++) with very few dependencies, trivial to compile on almost
any platform
* custom protocol that is easy to implement
* no need for server (db) side programming (in CouchDB, you would need to
program the views etc. for the map/reduce on the server, as far as I know)
* excellent Python bindings (but that is also true fo CouchDB).
So in as far as a Topicmap can be expressed as a series of key/value
statements, to me it looks like it will be easier and more useful to have a
backend based on redis, whereas I see the value for CouchDB in the way you
can program a complete web application there directly on the server. In
that sense, not as a backend for mappa, but as a topicmap server in its own
right, it might be very useful, but that would be a different project.
All the best,
Christian
Lars Heuer
[...]
> Many years ago, I made the mistake to assume a XML database would be a good
> fit for Topicmaps since they are expressed in XML (that was at the XTM 1.0
We've all been there. :)
> stage). The same type of thinking might favour Couchdb, since there is a
> JSON format for TM and it would be straightforward to map it to CouchDB.
Well, the mapping to CouchDB wouldn't necessarily look like JSON Topic
Maps [1]. Well, I am saying that without having any clue how the
mapping would look alike.
> But, as I do not need to tell you, the problems are on quite a different
> level, so here is a random list of things that I see in favor of redis:
[...]
Thanks for sharing your thoughts. I take a look at Redis after I
finished/started(!) my GAE backend. Native support for datatypes like
sets and lists sounds interesting. Maybe it would be possible to
create a generic key/value store layer that could support other
key/value stores as well.
Lars Heuer
> but I have no idea where to start or to look. Would something like
> redis (see http://redis.io; key/value store) or CouchDB (JSON based
> map/reduce interface) be a good match?
FYI: MaJorToM's Redis schema:
<http://docs.topicmapslab.de/majortom/redis.html>
Christian Wittern
> [...]
>
>> but I have no idea where to start or to look. Would something like
>> redis (see http://redis.io; key/value store) or CouchDB (JSON based
>> map/reduce interface) be a good match?
>>
> FYI: MaJorToM's Redis schema:
> <http://docs.topicmapslab.de/majortom/redis.html>
>
This sounds interesting and well thought through. Now we only need to
port this to mappa!
Christian