Webconsole debugging tool built into memcached

[Clint Webb]

I have been using memcached for a number of years now, and been amused by the many times that people new to memcached ask "How do I view the list of keys stored in memcached".

There is a perfectly valid reason why you should not want to view the list of keys stored in memcached.  If your process requires functionality like that, then likely you are using memcached in a way that it wasn't designed to be used, or have some flaws in the design of your application.

That being said, there are some advantages, when developing, to be able to view what is in the cache.  It would be nice to start a fresh cache, do some operation, and then look at the contents of the cache to see if everything went in correctly.  I've even built unit tests that check the contents of a cached entry to ensure that the code itself has succeeded.

I also thought that it would be nice to have something where I can view cached entries, and even do commands on those cached entries (or even add new entries), to check certain functionality in applications that get data from the cache.

To take it a step further, for ease of debugging, I thought it would be wonderful to have a watch-list of cache items, so that I can test a piece of code, and then at a glance at the contents of a number of keys.

To that end, I have built onto memcached a web-based debugging console, which I call the webconsole.  It uses the libevent queue that is already central to memcached, but provides a non-intrusive layer that provides some interaction with the running memcached instance.

I did most of the coding last weekend, and twiddled with it a bit during the week on the train, but will be finishing it up this weekend.

I have a half-completed version running at http://static.rhokz.com:8080/
When you hit that URL, you are actually hitting a mini-webserver running inside memcached (of sorts). 
The command line I used to start it is: ./memcached -d -u memcached -w 8080
That server is running in the UK, and so keep in mind that there is a little bit of latency that you wouldn't normally have when hitting it over a LAN, or something a bit closer (I'm actually in Australia, so its noticeable for me).

Have a look at what I have up there and see if you feel it might be useful.  I haven't finished the 'watch-list' part, but basically it will use cookies to store different sets of keys that you want to watch, and you can open and close those different sets, it would then show those keys, and their values, and change their colors when they are changed, gradually fading back to white after time (the longer it has been since it was changed), and so on.   All information is stored in the browser itself, and no session information is kept in memcached.  This means that the browser itself is responsible for maintaining the watch-list.

This is not very finished, but I thought I would make it available now, to hear any feedback about it.
The Lookup tab works, but not very integrated with other parts.
The Watch tab is incomplete and not included in the running demo.
The List tab works, but is very basic, and currently you have to reload the page in your browser to refresh it.
The Commands tab, set is the only command I have really implemented so far.
The Stats tab does nothing at the moment, but I expect it will have all the usual stats info.

Later today, I will be forking Dustin's latest branch on github and putting the code up there.

Even though this is for debugging, the code is very benign and would not impact a production instance if the '-w' option is not used (therefore not activating the webconsole).   Even if webconsole is active it should not affect a production environment much if it is not being used.

I have not yet made sure it is safe for mutli-threads, since I was using it single-threaded, I will add that in this weekend also.

On a technical note, when you hit that URL, the entire front-end code (javascript, html and css) is provided in a single page, and all interaction from that point is done through ajax. This has a side benefit in that when you have webconsole running (by adding -w <port> to the command line) you end up with an http interface that returns data in XML.  Including a list of all current keys, the ability to do a set using a http request.   Effectively this has probably become another memcached protocol (when complete).

For example, you could get a list of all keys in this test instance by hitting http://static.rhokz.com:8080/list
or set a key with:
http://static.rhokz.com:8080/set?key=something&value=value

Feel free to play around with the test s
Looking forward to hearing your ideas and comments. 


--
"Be excellent to each other"

[mike]

This is neat. I can see many useful reasons for this.

However, wouldn't it be simple enough to make a UI using PHP or
something that was totally portable and you could list multiple
servers and such? This says "built in" to memcached. Maybe that's what
scares me... is this actually bolted into memcached itself or am I
just getting stuck on the words :)

I suppose it would have to connect using sockets as some of this stuff
is not part of the memcached PECL packages (or maybe it is) ... but it
would be like a mini phpMemcachedAdmin ;)

[Jay Paroline]

Hi Clint,

This looks like a potentially incredibly useful tool for debugging. I
would be hesitant to commit to having that on our production servers,
but I would certainly run it on our dev servers.

Anyway, being able to see all keys by name is actually not all that
helpful for me because we md5 most of our keys, otherwise they end up
being too long. If it were possible to see all keys in order of when
they were added/modified, that would be the most useful memcache tool
I've ever used. Actually it would be the only memcache tool. :)

[Dustin]

On Mar 13, 3:14 am, Clint Webb <webb.cl...@gmail.com> wrote:

> I also thought that it would be nice to have something where I can view
> cached entries, and even do commands on those cached entries (or even add
> new entries), to check certain functionality in applications that get data
> from the cache.

We've talked about an HTTP interface before, and it makes sense (and
could certainly have a chunk of HTML to make it easier for browsers to
interact).

> The List tab works, but is very basic, and currently you have to reload the
> page in your browser to refresh it.

My only concern is that this might be a rather painful piece of
functionality to maintain. It's using some internal stuff to walk
slabs, but a lot of development has gone into engine abstraction which
would be incompatible with the list code. :/

[Marc Bollinger]

However, wouldn't it be simple enough to make a UI using PHP or
something that was totally portable and you could list multiple
servers and such? This says "built in" to memcached. Maybe that's what
scares me... is this actually bolted into memcached itself or am I
just getting stuck on the words :)

Just by briefly reading his email, it sounds like he's adding his own callbacks to libevent and intercepting events, then storing them for display via the in-process webserver. I'm guessing this was a performance consideration, but PHP really doesn't offer the sort of capability to hook in that way--you'd wind up at the same point we're currently at, where people complain about not being able to dump out the contents of memcached. I'm not saying don't be scared (though this shouldn't be going into production, anyway), though ;-)

[Clint Webb]

Thanks.

It is definitely meant for debugging applications that use memcached.  And it shouldn't be turned on for production systems.  However, the same built binary could be used for testing and production because the webconsole code is not used or even initialized if the -w option isn't set.

I've been trying to integrate the code that was working on an older branch, into the current master branch of dustin's repo, but the 'conn' object is now more integrated, and when making those function calls through webconsole, we dont have a conn object, so a few bits are broken.

I'll push the code back to my github fork so you can view the code if your interested in how it works.  Even though it doesn't compile there at the moment.
http://github.com/hyper/memcached/tree/webconsole

It works by using the evhttp stuff that is part of libevent that memcached is already using.  It basically listens on a different port and processes the http requests and makes callbacks for particular urls.   It has callbacks for operations like 'list', 'get', 'set', 'incr', etc... and a generic callback (for all other urls) that returns a webpage with all the html, css and javascript that runs what you see.    So if -w option is not set it does nothing. 

I was able to do everything without making any code changes to other code in memcached, but now that the 'conn' object is passed around a lot more for thread stats, it makes it a bit more difficult.

[NICK VERBECK]

I like some of your ideas and may have to borrow them for
Memcached-Manager. Mainly the watching keys idea.

However along the lines of seeing what keys and slabs are in your
servers. There is a PHP (memcache.php) script that comes with the Pecl
package as well as download-able from his website that allows you to
not only see basic stats but as well the slabs/keys for each server.
As always its only recommend for Dev environments.

--
Nick Verbeck - NerdyNick
----------------------------------------------------
NerdyNick.com
SkeletalDesign.com
VivaLaOpenSource.com
Coloco.ubuntu-rocks.org