memcached debugging/profiling

[Maksym Melnychok]

Hi,

are there any good tools to debug/profile memcached production setup without interrupting it?

i'm specifically interested in top key misses and slow queries

thanks in advance!

[Trond Norbye]

You may use dtrace if you're running on Solaris

Cheers,

Trond

[Maksym Melnychok]

unfortunately no, it's on Debian

[a.]

telnet <host> 11211

stats

should give a basic overview

[Maksym Melnychok]

@ a. yes i did that and numbers are bad :) 

that's why i need a bit more sophisticated tool

[Paul Gale]

Maksym,

>that's why i need a bit more sophisticated tool

In that case take a look at this:

You can use the open-source Maatkit toolkit (http://www.maatkit.org/) to analyze your memcached traffic. It's not just for MySql usage you know! :)

This article explains how: http://www.mysqlperformanceblog.com/2009/07/24/maatkit-now-supports-memcached/
(this article is referenced from here: http://www.xaprb.com/blog/2009/07/25/analyze-and-optimize-memcached-usage-with-maatkit/)

Iif you're on Linux use the mk-query-digest utility from the Maatkit toolkit to analyze network traffic captured using tcpdump thus (assuming you're running memcached on its default port of 11211):

   sudo tcpdump -s 65535 -x -n -q -tttt -i eth0 port 11211 > memc_tcpdump.txt
   mk-query-digest --type memcached memc_tcpdump.txt

The link to the article above is a couple of years old now. The functionality of mk-queryt-digest with regards to memcached has improved somewhat since the article was written.

HTH

Paul

[Maksym Melnychok]

Paul, thanks a lot!

i'll try it out

[Peter Phaal]

You may be interested in experimenting with sFlow:
http://blog.sflow.com/search/label/Memcache

Installing sFlow instrumentation does require a patch to the Memcached
code, so probably doesn't help your current situation.

FYI sFlow can also be used to instrument the web tier, there are
currently implementations for Apache and NGINX
http://code.google.com/p/mod-sflow/
http://code.google.com/p/nginx-sflow-module/

Cheers,
Peter