HTTP Interface to memcache

[Vijay Raghunathan]

Has there been any discussion on exposing a REST style interface to
memcached? It seems like a really natural way to access the cache
since it already supports similar notion (GET/POST/PUT/DELETE would
map very well to get/set/add/delete). Additionally clients are
already written in every language and you can use normal http
proxies/load-balances in front of them seamlessly. After seeing how
easy it is to interact with CouchDB (not that memcache doesn't have an
easy API), I immediately wished that memcached supported a similar
interface. Immediate use-cases for such an interface would be
a) asynchronous access becomes a lot easier (since HTTP clients exists
for many event-based systems)
b) One can imagine a set of proxy servers that know the topography of
the cache and are able to proxy requests to the correct node.

I've attached a quick and dirty patch that adds support for GET &
POSTs (libevents builtin HTTP client doesn't support PUTs or DELETEs
yet) that illustrates the idea. It leaks, and seems to truncate data
in odd places but I think it gets the point across.

vijay

[Ray Krueger]

> Has there been any discussion on exposing a REST style interface to
> memcached?

At one point I wrote a web app that had a REST interface to run out in
front of memcached. The idea being that the REST front-end could be
load balanced and configured such that clients only need to know one
address for the cache. Then operations folks and what not could manage
the number of caches and their arrangements behind that. The interface
was something like this...

Actually, when you look at PUT and POST in their pure forms the
mapping is actually a little odd...

GET /{cache_name}/{key} = get
DELETE /{cache_name}/{key} = delete
PUT /{cache_name}/{key} = set
POST /{cache_name}/{key} = ?

I map PUT to "set" because in this case the user knows the destination
of the resource, the user knows the full URI they want the resource to
have. I think this is the purest mapping of PUT, as it ties well to
how set works. The question then is how does one support the memcache
"add" and "replace"? POST is meant for sending a resource into a
factory location where the user does NOT know the final URI. Following
this constraint, there isn't a clear mapping of POST to the memcache
protocol.

A slightly more overloaded mapping might look like...
PUT /{cache_name}/{key} = add
POST /{cache_name}/{key} = set

Here, POST is being used incorrectly and PUT should arguably be used
as "replace" not "add".

Maybe the best thing to do is PUT=set and POST is overloaded to
support the more exotic use cases like "add", "replace", and maybe
even "incr"/"decr". That's the industry standard REST style I think.
Try and use GET/PUT/DELETE, give up and overload the shit out of POST
:P

Oh, and "cache_name" above was meant to be separate named memcache
clusters for different applications...

[John Allspaw]

I remember being the "crazy old coot" and asking this same question a bit ago:

http://lists.danga.com/pipermail/memcached/2004-November/000838.html

BrianA managed to hack something out that is along the same lines, but I haven't used it:
http://tangent.org/608/mod_memcached.html

-j

--
John Allspaw
http://flickr.com/photos/allspaw

[Josh Snyder]

This is a cool idea. However, one downside is that it will increase
the payload required to get/set a key, sometimes significantly.

Memcached is usually used for performance and scaling, and one of the
issues one encounters down that road is network load. We are extremely
latency sensitive, and we ended up rewriting the structure of both our
values and our keys to reduce their size (keys get sent back on a
multiget), which ended up yielding significant performance benefits
for us.

For folks that don't have this problem, and/or typically have very
large values stored, the http front end might be a nice thing to have.
It's just worth bearing in mind that it has a few negative
consequences as well.

-josh

[Vijay Raghunathan]

Hmm ... according to wikipedia(1), POST means Create (so I guess add
in memcache terms) and PUT means Update or Create (so set). It is
weird that POST knows the resource name. The way that CouchDB handles
this is pretty clean(2) (imo). POST means create a document with a
server generated key (not that useful to me, but who knows) and PUT
means create/update a resource with a given name.

It's unfortunate that libevent doesn't support PUT/DELETE (yet!!)
because those 4 verbs would cover most common uses cases. I think
it'd be pretty easy to add that in, however.

vijay

1 http://en.wikipedia.org/wiki/Representational_State_Transfer
2 http://wiki.apache.org/couchdb/HTTP_Document_API

[bokmann]

You could literally write this in about 16 lines of code using
Sinatra, a ruby dsl for creating web apps. I'm a bit busy for the
next few weeks, but when I find some downtime, I'd take that on. I
think it would be better done as something separate from memcache
ayway - I don't need it trying to take on new responsibilities,
listening on other ports, etc.

-db

>  memcached_http.diff
> 7KViewDownload