What's New in Memcached? (part one)

[Dustin]

We've done a lot of work since the latest 1.4.x release, but most of
it has been invisible to most users. To catch up those who haven't
been actively watching development, we're going to ease you into how
it affects different segments of users, starting with general cache
users.

We don't expect much to have changed from the perspective of the
typical memcached user. The purpose of most of the work is to turn
memcached into an extensible platform on top of which users can
experiment with new storage mechanisms, extending with new commands,
changing logging, adding instrumentation, etc... This is mostly a
reaction to mailing lists posts where someone wants to do something
that someone else doesn't want done. We've gone out of our way to
ensure that both can have what they want (and nobody has to fork).

Table of Contents
=================
1 Major Points
2 New stats
2.1 Stats returned by the default stats command
2.1.1 libevent
2.1.2 rejected_conns
2.2 topkeys
2.3 aggregate
2.4 settings
2.4.1 extension
2.4.2 topkeys
3 How to Get It
3.1 Via Git
3.2 Download Source
3.3 Building


1 Major Points
~~~~~~~~~~~~~~~

The biggest change is that the code is no longer one blob, but instead
a set of modules the user may enable based upon their preference. This
enables others to create their own backing store, and still use the
same command parser and core protocol handling as everyone else.

Another change users may like is that the logging mechanism is another
module, so that you can get all of the output to syslog (or windows
event log if you like).

We now have official support for Microsoft Windows! We're compiling
the source on Microsoft Windows every day, so we find and fix bugs
there at the same time as we do on the UNIX versions. The automatic
testing isn't as good as what we've got on UNIX, but believe me: we're
working on it!

Part of the point of going to modular components means that they are
easier to test in a test program without having to "fork()" and test
everything "over the wire". We have tests that cover specific
internals that can run very quickly.

2 New stats
~~~~~~~~~~~~

There are a number of new stats introduced in the next release.
The key supplied in the status command is passed to the storage
engine to allow the storage engine to add extra information
to the existing stats commands, and to create their own stat
commands.

2.1 Stats returned by the default stats command
================================================

2.1.1 libevent
---------------

Over the time we've seen a lot of bugs around people using an old
version of libevent. That's part of the reason why we bundle a well
known version of libevent in the release distribution. Memcached
checks the libevent version during startup, and will refuse to start
if the one used is too old. Since most operating systems use shared
libraries these days, you might be using another version than the one
you originally used when you first built memcached. In order for us to
see which library people are using we decided to put it into the stats
as well.

2.1.2 rejected_conns
---------------------

The number of times a connection attempt was refused (due when we're
hitting the maximum number of connections.

2.2 topkeys
============

You may get information about the most popular keys in memcached by
exporting the environment variable MEMCACHED_TOP_KEYS to the number
of keys you would want memcached to keep track of. There is no
such thing as a free lunch, so enabling this will have an impact on
memory and speed. We've decided to _disable_ this by default for now,
so you need to export this variable to enable the feature. Ex:

me@localhost:> MEMCACHED_TOP_KEYS=10 ./memcached

Running "stats topkeys" would return something like (I've added
the line breaks and indentations to make it more readable in
this document):

STAT my-key2 get_hits=0,get_misses=1,cmd_set=0,incr_hits=0,
incr_misses=0,decr_hits=0,decr_misses=0,delete_hits=0,
delete_misses=0,evictions=0,cas_hits=0,cas_badval=0,
cas_misses=0,ctime=2,atime=2
STAT my-key1 get_hits=1,get_misses=0,cmd_set=1,incr_hits=0,
incr_misses=0,decr_hits=0,decr_misses=0,delete_hits=0,
delete_misses=0,evictions=0,cas_hits=0,cas_badval=0,
cas_misses=0,ctime=12,atime=12

We hope to characterize the impact of this new stat before the
final release and would like to include your feedback.

2.3 aggregate
==============

The combination of the storage engine interface and the SASL auth
allows for the combination of a connection-based stats. The aggregate
subcommand is used to aggregate the stats from all of the connections
on the server. The stats returned from the aggregate subcommand is the
same as the normal stats command.

2.4 settings
=============

2.4.1 extension
----------------

Displays one of the extensions loaded (may appear multiple times).

ex:

STAT logger syslog
STAT ascii_extension scrub
STAT ascii_extension noop
STAT ascii_extension echo

2.4.2 topkeys
--------------

The number of keys we are monitoring.

3 How to Get It
~~~~~~~~~~~~~~~~

We haven't started shipping tarballs yet (though we've done a lot of
work to get those working everywhere), so currently you can get the
code via github.

3.1 Via Git
============

We've been keeping the =engine-pu= branch up to date on github with
the latest changes. You can get it and build it like this:

git clone git://github.com/memcached/memcached
cd memcached
git checkout -b engine-pu -t origin/engine-pu

3.2 Download Source
====================

If you don't want to use git, you can download the latest snapshot of
that branch using the following URL:

[https://github.com/memcached/memcached/tarball/engine-pu]

Do note that this is not a dist build, so you will need automake,
autoconf, libtool and libevent in addition to your compiler. The dist
build greatly simplifies the build.

3.3 Building
=============

In either of the above cases, you'll do a developer source build using
the following incantation:

./config/autorun.sh
./configure
make

[Henrik Schröder]

Awesome! Quick question: How are you compiling on windows? Mingw? Or some other solution?


/Henrik

[Trond Norbye]

On 11. apr. 2011, at 17.03, Henrik Schröder wrote:

> Awesome! Quick question: How are you compiling on windows? Mingw? Or some other solution?
>
>

We're using mingw right now... The problem is that we don't have another compiler with C99 support on windows for the moment (or at least none that I'm aware of ;-))

Trond

[Dustin]

Well, there is icc, but we haven't put any significant effort into
that. Intel makes it hard to get. I've got an older version for
Linux, so I know it *will* build under icc, but that's about it.

But yeah, we declare this done with mingw support.

[Henrik Schröder]

On Mon, Apr 11, 2011 at 17:26, Trond Norbye <trond....@gmail.com> wrote:

We're using mingw right now... The problem is that we don't have another compiler with C99 support on windows for the moment (or at least none that I'm aware of ;-))

How was this version built?

http://blog.elijaa.org/index.php?post/2010/08/25/Memcached-1.4.5-for-Windows

That doesn't look mingw-built to me?


/Henrik

[Trond Norbye]

How does a mingw built version look to you? (because it is built by using mingw...)

Trond

[dumpster]

REMOVE

[dumpster]

REMOVE ME

[Henrik Schröder]

On Tue, Apr 12, 2011 at 11:58, Trond Norbye <trond....@gmail.com> wrote:

How does a mingw built version look to you? (because it is built by using mingw...)

Compare:

http://s3.amazonaws.com/downloads.northscale.com/memcached-win32-1.4.4-14.zip
http://blog.elijaa.org/public/memcached-1.4.4-win32.rar

Both are 1.4.4, but the top one includes a libgcc dll and a mingw dll, and the bottom one doesn't. And since the bottom one doesn't, I assumed it wasn't built using mingw, but some other method that still links in the pthread dll. But if it was, what's the difference?

Also related to this, how hard is it to compile the latest memcached for windows in mingw now? Install mingw, configure, and make? Or are there any gotchas on the way?


/Henrik

[Trond Norbye]

On 12. apr. 2011, at 14.34, Henrik Schröder wrote:

> Both are 1.4.4, but the top one includes a libgcc dll and a mingw dll, and the bottom one doesn't. And since the bottom one doesn't, I assumed it wasn't built using mingw, but some other method that still links in the pthread dll. But if it was, what's the difference?
>

ah, don't know about that :-) I guess it probably depends on the version of the compiler and how much it links into the binary...

> Also related to this, how hard is it to compile the latest memcached for windows in mingw now? Install mingw, configure, and make? Or are there any gotchas on the way?
>
>

It's been a while since last time I tried it, but it was pretty straight forward: http://goo.gl/nxf5O

Cheers,

Trond

[Henrik Schröder]

On Tue, Apr 12, 2011 at 14:57, Trond Norbye <trond....@gmail.com> wrote:

It's been a while since last time I tried it, but it was pretty straight forward: http://goo.gl/nxf5O

Thanks, that actually worked pretty ok, except the branch is now called engine-pu instead of engine, and you also need the libevent dll if you want to make a binary package of it, but otherwise it worked ok. Good to know, and good that it's a lot easier to get it to compile for windows now.


/Henrik

[Dustin]

On Apr 12, 10:03 am, Henrik Schröder <skro...@gmail.com> wrote:

> On Tue, Apr 12, 2011 at 14:57, Trond Norbye <trond.nor...@gmail.com> wrote:
>
> > It's been a while since last time I tried it, but it was pretty straight
> > forward:http://goo.gl/nxf5O
>
> Thanks, that actually worked pretty ok, except the branch is now called
> engine-pu instead of engine, and you also need the libevent dll if you want
> to make a binary package of it, but otherwise it worked ok. Good to know,
> and good that it's a lot easier to get it to compile for windows now.

engine-pu can probably become engine now.