instrumenting memcached with sFlow

[neilmckee]

Hello All,

Here is a patch against the central repo that adds sFlow monitoring.
Please comment!
http://www.inmon.com/technology/sfmc_20100923.zip

For background on this, please read:
http://blog.sflow.com/2010/08/sflow-host-structures.html

If you compile with this patch, then you can run with "memcached -u
nobody -o sflow=on", and it will pick up configuration from /etc/
hsflowd.auto, such as:

sampling=400
polling=20
agentIP=10.0.0.112
collector=10.0.0.111 6343

"agentIP" should be the IP of the server, and "collector" should be
the IP of an sflow collector such as "sflowtool":
http://www.inmon.com/bin/sflowtool-3.17.tar.gz

(This config file is generated automatically if you install hsflowd,
the host sFlow daemon from host-sflow.sourceforge.net which
contributes an sFlow feed of server performance stats)

Neil


NOTES:

(1). Provided the sampling-rate is set appropriately, the overhead
should be roughly equivalent to adding one extra stats counter. The
critical path is just a decrement-and-test on a per-thread counter (no
locking).

(2). The changes are all within "#ifdef SFLOW" except for (3) below.

(3). I changed memcached.h:struct conn->request_addr to be of type
struct sockaddr_storage so it would work for IPv6 too.

[neilmckee]

I posted this patch as an enhancement to the tracking system:
http://code.google.com/p/memcached/issues/detail?id=157

My original post wasn't very clear about the problem space being
addressed: it is targeted at the challenge of identifying "hot keys"
in an operational cluster. For background, see:
http://blog.sflow.com/2010/09/memcached.html

Neil

On Sep 23, 5:19 pm, neilmckee <neil.mckee...@gmail.com> wrote:
> Hello All,
>
> Here is a patch against the central repo that adds sFlow monitoring.

> Please comment!http://www.inmon.com/technology/sfmc_20100923.zip

[neilmckee]

The following git fork includes these sFlow changes:

http://github.com/sflow/memcached

To run with sFlow enabled:

./autogen.sh
./configure --enable-sflow
make
./memcached -u nobody -o sflow=on

Neil

[Peter Phaal]

The following article shows how to report on hot keys using sFlow data
collected from the Memcached cluster:
http://blog.sflow.com/2010/10/memcached-hot-keys.html

[benjaminDC]

What I have done so far:

Clean deb 6 64bit install:

apt-get install git
apt-get install autoconf
apt-get install automake
apt-get install autotools-dev
apt-get install libevent-dev

git clone https://github.com/sflow/memcached.git

./autogen.sh

 ./configure --enable-sflow

make
make check
make install

# memcached -u nobody -o sflow=on
Illegal suboption "sflow=on"

:-/ Am I doing something (very) wrong?

Op dinsdag 28 september 2010 21:30:04 UTC+2 schreef neilmckee het volgende:

[Peter Phaal]

Those instructions are out of date, the sflow option is no longer
required. Just build and run the sFlow instrumented version of
Memcached the way you normally would - there are no sFlow specific
options.

You should also install a Host sFlow agent (hsflowd) on each server in
your Memcached cluster.
http://host-sflow.sourceforge.net/

Configuration of the sFlow settings is handled by hsflowd - which also
sends hosts CPU, memory, disk and network performance metrics. The
following article describes how sFlow agents interact to monitor a
server:
http://blog.sflow.com/2012/01/host-sflow-distributed-agent.html

What software are you planning to use for performance reporting? The
latest version of Ganglia will trend host and memcached performance
metrics from your cluster:
http://blog.sflow.com/2011/12/using-ganglia-to-monitor-memcache.html

There are also some scripts you can modify to look at top keys, top
missed keys, top clients, value sizes etc
http://blog.sflow.com/2010/10/memcached-hot-keys.html
http://blog.sflow.com/2010/10/memcached-missed-keys.html
http://blog.sflow.com/2012/01/forwarding-using-sflowtool.html

If you are going to Velocity, you might be interested in the talk on
optimizing HTTP/Memcached performance using sFlow (the talk
description includes some interesting examples):
http://velocityconf.com/velocity2012/public/schedule/detail/23487

Peter

[benjaminDC]

Thank you for your response!

I was planning to use PRTG Monitor (because is reads all other snmp traffic from our network),
but unfortunately it isn't capable of reading countersample packets :-/

Unless I can configure hsflowd to send flow sample types instead of countersample

Tnx again for your response!

Benjamin



Op vrijdag 24 september 2010 02:19:03 UTC+2 schreef neilmckee het volgende:

Op vrijdag 24 september 2010 02:19:03 UTC+2 schreef neilmckee het volgende: