Memcached Beta 1.3.2 Released

[Dustin]

We've just released memcached beta 1.3.2. It's available for
download immediately at the google code site:

http://code.google.com/p/memcached/downloads

There is one known bug (UDP support is somewhat lacking), but it was
recently discovered by client authors attempting to implement the
protocol, so I think we're good.

We have quite a bit of confidence in this release as we've probably
spent more effort testing it than implementing new features, but I
wouldn't recommend you immediately switch all of your production sites
over. I would, however, hope that you'd ignore my recommendations and
do it anyway (or at least, test it really, really hard).

Release notes are below. They're also in beta, so if something's
wrong (you're not credited properly, or an important new feature was
overlooked), be sure to let us know so we can get it right before the
next release.

Memcached 1.3 Beta 2 Release Notes
==================================

Date: 2009-03-11 Wed


Table of Contents
=================
1 New Features
1.1 Binary Protocol
1.1.1 Client Availability
1.2 Performance
1.3 Stats
1.3.1 New Stats
1.3.2 More Granular Stats
1.3.3 Removed stats
2 Bug Fixes
3 Development Info
4 Feedback
5 Contributors


1 New Features
~~~~~~~~~~~~~~~

1.1 Binary Protocol
====================

A new feature that brings new features. We now have goodness like
CAS-everywhere (e.g. delete), silent, but verifiable mutation
commands, and many other wonders.

Note that the original protocol is *not* deprecated. It will be
supported indefinitely, although some new features may only be
available in the binary protocol.

1.1.1 Client Availability
--------------------------

Many clients for the binary protocol are available.

* C

libmemcached supports just about anything you can do with a
memcached
protocol and is the foundation for many clients in many different
languages (which you can find linked from the project page).

Project page: [http://tangent.org/552/libmemcached.html]

* Java

spymemcached has very good text and binary protocol support over
IPv4
and IPv6 with a quite comprehensive test suite.

Project page: [http://code.google.com/p/spymemcached/]

* Protocol Spec

NIH problem? Go write your own client. :)

[http://cloud.github.com/downloads/dustin/memcached/protocol-
binary.txt]


1.2 Performance
================

Lots of effort has gone into increasing performance.

Facebook-inspired contention reduction with per-thread stat collection
and the Facebook connection dispatch and thread starvation prevention
contributions helped our scalability.

Lock analysis also showed us that we had quite a bit of contention on
hash table expansion which has been moved into its own thread, greatly
improving the scalability on multicore hardware.

A variety of smaller things also shook out of performance testing and
analysis.

1.3 Stats
==========

There are several new stats and some new ways to look at older stats.

1.3.1 New Stats
----------------

* Delete

The global stats now contain statistics on deletion.

delete_hits refers to the number of times a deletion command was
issued which resulted in a modification of the cache while
delete_misses refers to the number of times a deletion command was
issued which had no effect due to a key mismatch.

* Incr/Decr

Incr and decr each have a pair of stats showing when a
successful/unsuccessful incr occurred. incr_hits, incr_misses,
decr_hits, and decr_misses show where such mutations worked and
where
they failed to find an existing object to mutate.

* CAS

CAS stats are tracked in three different ways:

+ cas_hits

Number of attempts to CAS in a new value that worked.

+ cas_misses

Number of attempts to CAS in a value where the key was not found.

+ cas_badval

Number of attempts to CAS in a value where the CAS failed due to
the
object changing between the gets and the update.

* slab class evicted time

Per slab class, you can now see how recently accessed the most
recent
evicted data was. This is a useful gauge to determine eviction
velocity on a slab so you can know whether evictions are healthy or
if
you've got a problem.


1.3.2 More Granular Stats
--------------------------

Where possible, stats are now tracked individually by slab class. The
following stats are available on a per-slab-class basis (via "stats
slabs"):

* get_hits
* cmd_set
* delete_hits
* incr_hits
* decr_hits
* cas_hits
* cas_badval

(misses are obviously not available as they refer to a non-existent
item)

1.3.3 Removed stats
--------------------

"stats malloc" and "stats maps" have been removed.

If you depended on these commands for anything, please let us know so
we can bring them back in a more maintainable way.

2 Bug Fixes
~~~~~~~~~~~~

* Build fixes on ubuntu (gcc and icc) and FreeBSD
* bad interaction with cas + incr (bug 15)
* setuid failures are reported properly at daemonization time
* decr overflow causing unnecessary truncation to 0 (bug 21)
* failure to bind on Linux with no network (i.e. laptop dev)
* some memcached-tool cleanup

3 Development Info
~~~~~~~~~~~~~~~~~~~

We've added a bunch of tests and new code coverage reports.

All included code in this release has been tested against the
following platforms (using the in-tree test suite):

* ubuntu 8.10 (64-bit, both gcc and icc)
* ubuntu 8.04 (32-bit)
* OS X 10.5 (ppc and intel)
* OpenSolaris 5.11 x86 (with and without dtrace)
* FreeBSD 7 x86

4 Feedback
~~~~~~~~~~~

Please try this version. Make it suffer. Report feedback to the list
or file bugs as you find them.

* Mailing List: [http://groups.google.com/group/memcached]
* Issue Tracker: [http://code.google.com/p/memcached/issues/list]
* IRC: #memcached on freenode

5 Contributors
~~~~~~~~~~~~~~~

The following people contributed to this release since 1.2.6.

Note that this is based on who contributed changes, not how they were
done. In many cases, a code snippet on the mailing list or a bug
report ended up as a commit with your name on it.

Note that this is just a summary of how many changes each person made
which doesn't necessarily reflect how significant each change was.
For details on what led up into a branch, either grab the git repo and
look at the output of `git log 1.2.6..1.3.2` or use a web view.

* Repo list: [http://code.google.com/p/memcached/wiki/
DevelopmentRepos]
* Web View: [http://github.com/dustin/memcached/commits/1.3.2]

104 Dustin Sallings
49 Trond Norbye
32 Toru Maesaka
31 dormando
8 Steve Yen
7 hachi
6 Aaron Stone
6 Brian Aker
4 Victor Kirkebo
2 Ricky Zhou
1 Jonathan Bastien-Filiatrault
1 Evan Klitzke
1 Eric Lambert

[Toru Maesaka]

Awesome! thanks big time for getting this out.

One thing I'd like to add to Dustin's message is the space
efficiency option ("-C") for those that don't take advantage of
CAS. You can read more about this at:

- http://torum.net/2008/12/more-memcached-space/
- http://blogs.sun.com/trond/entry/not_using_cas_disable_them

For those that are too busy to visit the above URLs, in brief you
can save 8 bytes per item if you start memcached with the -C
option. Do note though, that you're disabling CAS so you won't
be able to take benefit of CAS operations on that instance anymore.

For those that don't care, well... happy caching! you can boost
the space efficiency of memcached and cache more data :)

Cheers,
Toru

[Clint Webb]

I'm coding up something that I think might be useful for memcached.  It isn't very intrusive, but what is the correct branch I should be basing it on?  I should probably get on IRC and chat with you guys more.

--
"Be excellent to each other"

[Dustin]

On Mar 11, 11:11 pm, Clint Webb <webb.cl...@gmail.com> wrote:
> I'm coding up something that I think might be useful for memcached.  It
> isn't very intrusive, but what is the correct branch I should be basing it
> on?  I should probably get on IRC and chat with you guys more.

This release was peeled off of my master branch at github:

http://github.com/dustin/memcached

You'll find the tags in there as well.

[Dustin]

On Mar 11, 11:09 pm, Toru Maesaka <tmaes...@gmail.com> wrote:

> One thing I'd like to add to Dustin's message is the space
> efficiency option ("-C") for those that don't take advantage of
> CAS. You can read more about this at:
>
> -http://torum.net/2008/12/more-memcached-space/
> -http://blogs.sun.com/trond/entry/not_using_cas_disable_them

Thanks a lot for pointing that out. 8 doesn't sound like much to
many people, but some FairlyBig users were unhappy about the item
expansion.

Now if we could only get *really* fancy and dynamically enable it
per-item, that'd be awesome. (It's actually possible and might be
quite beneficial for other sparsely used item properties).

[Yigit Boyar]

hi all,

I'm developing a software where memcached is also used as locking server.

The problem is that, i wanna make this fail safe; which basically means
if the memcached server goes down, another one starts and the locks are
transformed. This basically handled by keeping two instances up, one as
master and if the master dies, slave becomes master and an new slave
comes live.

This is my guess, but I'm looking for some good best practices not to
invent wheel from scratch.

So is there any link you can provide where a best practice for memcached
based locking is present ?

btw. i'm using php in a distributed environment.

thnks.

[Dustin]

Please don't use the reply button to start a thread.

[Toru Maesaka]

Here's another thing I just remembered about the 1.3 series.

In the current 1.2 series, you have the option of building
and running the memcached daemon in multi threaded mode.

This has changed in 1.3, in a way that the server is always built
in multi threaded mode. IIRC, the daemon will spool four threads
by default but you can change this with the -t option.

Cheers!
Toru

[Henrik Schröder]

The best practice is to not use memached at all for this. It is a cache, if you store something in it, you are in no way guaranteed that it will be there the next time you ask for it. It is temporary, not permanent, and as such an extremely bad match for a distributed lock where it is absolutely critical that you do not lose data.


/Henrik

[Yigit Boyar]

ok, let me more clarify this topic since I got one more email about not
using memcache for locking.

First of all, I know memcached is not reliable by design but if you
visit http://en.wikipedia.org/wiki/Memcached you will see that I'm not
the only/first person to try this.( there is a link to
IPC::Lock::Memcached - fast locking via memcached).

For more information about my software, the locking mechanism is also
backed up by a MVCC implementation in db layer. In other words, I
already have a backup mechanism if memcached fails not to loose data
integrity, but the cost of rolling back a bunch of transactions is not
small, so I dont want to see MVCC exceptions hanging around.

In addition, I need to guarantee that at least one memcached server is
alive not to freeze my system.(e.g. if nobody can obtain lock, nobody
can take an action).

Now comes the case of saving locks when a server fails. In that case, we
may loose ~ 10K locks which will cause ~ 7K operation rollbacks. What I
think is, with a memcache master slave implementation, this number can
be decreased to a ommittable number.
Of course there is another solution to this problem, basically depending
on db layer MVCC implementation if memcache fails, but this will be a
highly coupling between these two, which wont be preferable by design.

Anyway, to clarify, any master-slave like replication of memcached may
be helpful for my case.

[Dustin]

On Mar 14, 5:27 am, Yigit Boyar <ybo...@gmail.com> wrote:
> ok, let me more clarify this topic since I got one more email about not
> using memcache for locking.

I do wish you could post your topics and clarifications in a thread
that isn't a release announcement since you've removed that topic from
the list. Too late for that, I suppose.

> For more information about my software, the locking mechanism is also
> backed up by a MVCC implementation in db layer. In other words, I
> already have a backup mechanism if memcached fails not to loose data
> integrity, but the cost of rolling back a bunch of transactions is not
> small, so I dont want to see MVCC exceptions hanging around.

You're just thinking about your application incorrectly. You have a
DB-based lock that you want to try to have hopeful fast access to via
memcached. Memcached should just be caching the lock information that
you're persisting in your database.

> In addition, I need to guarantee that at least one memcached server is
> alive not to freeze my system.(e.g. if nobody can obtain lock, nobody
> can take an action).

Failure to access a cache is not failure to obtain a lock. Were I
to implement this using my java client, for example, I might wait up
to a half second for a cached lock before obtaining the lock the more
expensive way, and would always asynchronously cache the lock after
synchronously storing it in the reliable layer.

> Now comes the case of saving locks when a server fails. In that case, we
> may loose ~ 10K locks which will cause ~ 7K operation rollbacks. What I
> think is, with a memcache master slave implementation, this number can
> be decreased to a ommittable number.

No information is lost just because a memcached server fails, so I
don't see why you'd feel that you'd lose locks.

> Of course there is another solution to this problem, basically depending
> on db layer MVCC implementation if memcache fails, but this will be a
> highly coupling between these two, which wont be preferable by design.

It's not a coupling issue, it's an optimization issue. You design
the application first, then you add optimizations. In this case, like
all others for which it was designed, memcached is just an
optimization tool.

You *can* use memcached as a ``hopeful'' kind of lock -- that is,
where the lock itself is just an optimization. I've used it quite
successfully for job deduplication where any memcached failure (server
down, restarted, LRU evicted, object expired, net split, etc...)
results in unnecessary, but harmless work being performed.

I have a similar case where I do best-effort message deduplication
where delivery of a duplicate message would just be a minor annoyance
and the cost of being sure is too high for the app.

Any place where I needed *real* mutual exclusion locks, I used a
proper lock server. I wrote one that had the semantics I needed last
time I needed one, but several such things exist on their own in the
wild.

> Anyway, to clarify, any master-slave like replication of memcached may
> be helpful for my case.

Replication would only make things worse. That would make the fast
and volatile software *significantly* slower and more complicated (I
assume you'd also want to keep hash table usage patterns in sync so
the LRUs are consistent as well as replicating evictions on insert
since the same object wouldn't necessarily get evicted from two
different servers due to the set of item locks held at transfer time
being different on each server as they are attempting to free memory
for a new item store operation).


To your first point: yes, it's been done before. People will
probably do it again. It may have even been successful almost all of
the time (I doubt anyone's publishing numbers on how many false-
positives they get on locks).

If that's good enough for you, then great. Don't worry about
failures.

If you want optimization over a slower, but safer underlying lock
mechanism, just write the code as if you assume it will always fail
and you should be happy with the additional performance a cache brings
you.

However, if you want a good core lock server, don't look to a cache
designed to be lossy and volatile.

[dormando]

Excellent work on the development release, folks ;)

I'm a bit behind on all of the changes in the development tree (uh ohs),
but I know they've been thorough on performance and correctness testing.
Please give us feedback on what you're doing to test this release if you
are, so we may feel more confident about hammering out a stable release.

-Dormando

[JC]

Hi,

sorry, I haven't put it in production :-P but I played a little bit
with it and I have one strange behaviour when using the binary
protocol.
With small items (10 bytes key / <100 bytes content), there are some
40ms blackout periods during fetch calls. The duration is always very
stable, it looks like an explicitely defined value but I can't find
which one !

So, in ASCII protocol, this work fine (no more than at most 1ms gap
between answers from the server): environment seems then ok. I made my
last test with server and client locally to be sure network can be
ruled out.

On client side, I trace up to the system call:
# 1:30:12 423177
error= poll(fds, 1, ptr->root->poll_timeout); (40 ms timeout is not in
this variable ;-)
# 1:30:12 461606

On server side, (94 is the item on which it blocked)
1:30:12 422266
<32 GET item94 (add_bin_header function)
1:30:12 422386

There is definitly something systematic, it happens always when
fetching almost the same item (at most 1-2 items before or after),
always with the same blocking duration.
I haven't been yet down the lowest layer on server side. Is there some
kind of buffering that could cause that ?
I am really doubting this is some os/network latency/buffering, it is
really too reproducible.

Has anyone an idea what this could be ?

---
Jean-Charles

[JC]

I am answering to myself ;-)

Well, first, this only occurs on old suse boxes (9, 10.1), and above
all this seems to be linked to the client rather than the server
(targeting the same server from an ubuntu is enough to make the
behaviour desappear) so I guess I have nothing to complain about the
1.3.2 :-)

Anyway, if anyone is aware of such ~40 ms poll latency in old suse
configuration or with kernels<=2.6.16.46-0.12-smp , I am interested.

---
Jean-Charles

[Dustin]

On Mar 24, 4:33 am, JC <jc.redou...@gmail.com> wrote:
> I am answering to myself ;-)
>
> Well, first, this only occurs on old suse boxes (9, 10.1), and above
> all this seems to be linked to the client rather than the server
> (targeting the same server from an ubuntu is enough to make the
> behaviour desappear) so I guess I have nothing to complain about the
> 1.3.2 :-)
>
> Anyway, if anyone is aware of such ~40 ms poll latency in old suse
> configuration or with kernels<=2.6.16.46-0.12-smp , I am interested.

What version of libevent are you using? Is it possible that this is
a factor?

I'd tend to want to blame the kernel, but it'd be good to collect as
much info as possible.

Are you saying that you don't see this happening on the server, but
only the client? (That is, a client on a newer OS works better)?

[JC]

On Mar 25, 12:08 am, Dustin <dsalli...@gmail.com> wrote:

>   What version of libevent are you using?  Is it possible that this is
> a factor?

1-4-9, also tested with 1-4-8, the behaviour is the same. I used the
exact same libevent in all the tests (exhibiting latency or not) so I
tend to rule out libevent.

>   I'd tend to want to blame the kernel, but it'd be good to collect as
> much info as possible.
>
>   Are you saying that you don't see this happening on the server, but
> only the client?  (That is, a client on a newer OS works better)?

Yes:
server on suse 9 , client on suse 9: there is latency
server on suse 9, client on latest ubuntu: no latency
server on ubunutu, client on suse 9: latency
all on ubuntu: no latency

The very strange thing is that is only occurs when using the binary
protocol, ascii one is working perfectly in every configuration.
Is there really that much of a difference in network exchange patterns
between ascii and binary ?

---
Jean-Charles