What's new in memcached (part 2)
Trond Norbye
=======================
(part two - new feature proposals)
Table of Contents
=================
1 Protocol
1.1 Virtual buckets!
1.2 TAP
1.3 New commands
1.3.1 VERBOSITY
1.3.2 TOUCH, GAT and GATQ
1.3.3 SET_VBUCKET, GET_VBUCKET, DEL_VBUCKET
1.3.4 TAP_CONNECT
1.3.5 TAP_MUTATION, TAP_DELETE, TAP_FLUSH
1.3.6 TAP_OPAQUE
1.3.7 TAP_VBUCKET_SET
1.3.8 TAP_CHECKPOINT_START and TAP_CHECKPOINT_END
2 Modularity
2.1 Engines
2.2 Extensions
2.2.1 Logger
2.2.2 Daemon
2.2.3 ASCII commands
3 New stats
3.1 Stats returned by the default stats command
3.1.1 libevent
3.1.2 rejected_conns
3.1.3 stats related to TAP
3.2 topkeys
3.3 aggregate
3.4 settings
3.4.1 extension
3.4.2 topkeys
1 Protocol
~~~~~~~~~~~
Intentionally, there is no significant difference in protocol over
1.4.x. There is one minor change, but it should be transparent to
most users.
1.1 Virtual buckets!
=====================
We don't know who originally came up with the idea, but we've heard
rumors that it might be Anatoly Vorobey or Brad Fitzpatrick. In lieu
of a full explanation on this, the concept is that instead of mapping
each key to a server we map it to a virtual bucket. These virtual
buckets are then distributed across all of the servers. To ease the
introduction of this we've assigned the two reserved bytes in the
binary protocol for specifying the vbucket id, which allowed us to
avoid protocol extensions.
Note that this change should allow for complete compatibility if the
clients and the server are not aware of vbuckets. These should have
been set to 0 according to the original binary protocol specification,
which means that they will always use vbucket 0.
The idea is that we can move these vbuckets between servers such that
you can "grow" or "shrink" your cluster without losing data in your
cache. The classic memcached caching engine does _not_ implement
support for multiple vbuckets right now, but it is on the roadmap to
create a version of the engine in memcached to support this (it is a
question of memory efficiency, and there are currently not many
clients that support them).
Defining this now will allow us to start moving down the path to
vbuckets in the default_engine and allow other engine implementors to
consider vbuckets in their design.
You can read more about the mechanics of it here:
[http://dustin.github.com/2010/06/29/memcached-vbuckets.html]
However, you _cannot_ use a mix of clients that are vbucket aware and
clients who don't use vbuckets, but then again it doesn't make sense
to use a vbucket aware backend if your clients don't know how to
access them. This is why we believe a protocol change isn't
warranted.
Defining this now will allow us to start moving down the path to
vbuckets in the default_engine and allow other engine implementors to
consider vbuckets in their design.
1.2 TAP
========
In order to facilitate vbucket transfers, among other use cases where
people want to see what's inside the server, we added to the binary
protocol a set of commands collectively called TAP. The intention is
to allow "clients" to receive a stream of notifications whenever data
change in the server. It is solely up to the backing store to
implement this, so it can make decisions about what resources are used
to implement TAP. This functionality is commonly needed enough though
that the core is aware of it, leaving specific implementation to
engines.
1.3 New commands
=================
There are a few new commands available. The following sections
provides a brief description of them. Please check protocol_binary.h
for the implementation details.
1.3.1 VERBOSITY
----------------
We did not have an equivalent of the verbosity command in the textual
protocol. This command allows the user to change the verbosity level
on your running server by using the binary protocol. Why do we need
this? There is a command line option you may use to disable the ascii
protocol, so we need this command in order to change the logging level
in those configurations.
1.3.2 TOUCH, GAT and GATQ
--------------------------
One of the problems with the existing commands in memcached is that
you couldn't tell the memcached server that the object is still valid
and we just want a longer expiration. Normally you want to put an
expiry time on the objects, so that you can get an indication if your
cache is big enough (by watching the eviction stats.. if your
memcached server has a high eviction rate your cache isn't big enough
for what you want to have in there). The normal idea is that the items
you're normally using would be bumped to the front of your LRU (and
hence not be kicked out immediately).
The touch command lets you set the expiry time for an object without
retrieving the object. In most cases, you will not want to do this
unless you provide a CAS value to ensure that you're touching the
correct version of the object.
GAT means "get and touch" and returns the object in addition to
setting a new expiration time. This allows you to have a rolling
window of expiry that has a TTL in addition to the access time. For
example, you can instruct memcached to allow an object to live no
later than five minutes after the last time it was access (but as
always, it may expire sooner).
1.3.3 SET_VBUCKET, GET_VBUCKET, DEL_VBUCKET
--------------------------------------------
These commands are used to set, get or delete a vbucket on the server.
1.3.4 TAP_CONNECT
------------------
Connect and request that the server initialize a TAP stream.
The point of this command is to allow clients to connect and specify a
few things about the data they wish to receive. Specifically, the
client will typically specify a date either in the past or in the
future along with specifying a vbucket. The server will then stream
data mutated since that given date or if a future date is specified,
only stream new mutations as they arrive. The specific details about
which mutations to send may vary on implementation.
1.3.5 TAP_MUTATION, TAP_DELETE, TAP_FLUSH
------------------------------------------
TAP_MUTATION is a notification that an item changed value in the
server.
The mutation typically comes with the new value.
TAP_DELETE is a notification that a key was deleted on the server.
Finally, to avoid having to send a complete list of all the keys in
the server when the user issues a flush, we can send a single message
(TAP_FLUSH) representing the flush. Please note that the FLUSH
message means _ALL_ vbuckets, and not just a single vbucket.
1.3.6 TAP_OPAQUE
-----------------
To allow storage engines to send their own messages over the tap
stream between each other, a tap opaque message is defined. It is
completely up to the storage engine to specify the internal layout of
the package.
1.3.7 TAP_VBUCKET_SET
----------------------
This is a message requesting a vbucket set. It is similar to the
set_vbucket command, with the difference that this message comes over
a tap connection (with the extra info a tap message contains)
1.3.8 TAP_CHECKPOINT_START and TAP_CHECKPOINT_END
--------------------------------------------------
The checkpoint start and end messages may be used by engine who wants
to use checkpoints. Checkpoints are an optional feature that may be
used by some engines to allow clients to start at a checkpoint
position. By doing so, the client need not do a full "backfill" even
if it is revisiting a server after having been gone for a while. The
TAP_CHECKPOINT_START tells a client that it's the start of a new
checkpoint, and the TAP_CHECKPOINT_END tells the client when it's
received everything for that given checkpoint.
2 Modularity
~~~~~~~~~~~~~
As we mentioned in the first email on changes, one big difference with
this new work is that we've tried to refactor memcached into being a
modular application instead of being monolithic. In the future, we'd
like to make the command parser as a separate module, so that we may
load the parsers separately.
2.1 Engines
============
We've done a lot of work trying to refactor the code in memcached to
avoid the tight coupling between the command protocol parser and the
actual item storage.
The idea with the engine interface is that the memcached process loads
a dynamically loadable object and calls a well known function to get a
set of function pointers. All communication between the memcached
process and the engine is performed through these function
pointers. The memcached process provides a set of services to the
engine as well through another set of function pointers.
The beauty of this is that the user may choose between a set of
different storage engines that suites their runtime
environment. People have different requirements for their server. Some
people need ACID, others may prefer ecstacy ;-) The storage interface
may let them design their app by using the memcached protocol, and
they can just swap in the backend that suites their needs (may it be
persistence, replication (sync or lazily) etc..)
2.2 Extensions
===============
The item storage isn't the only place we've tried to create a level of
modularity. People run memcached in different environments with
different requirements. You specify the extensions you want to use by
adding the -X command line argument.
2.2.1 Logger
-------------
We've seen a lot of different requests when it comes to logging. Some
want it to a file, some to syslog (or Windows event log) and some want
it to standard out. By default memcached will print to stderr, but
you may specify a different logger by loading the appropriate module
with the -X command line argument
2.2.2 Daemon
-------------
You might want to have some daemons providing extra services inside
your memcached server. Examples would be things like a doors server
to provide additional access to your server (Trond's favorite), or
perhaps a "dispatcher" offering a threadpool for your engines to
use?).
2.2.3 ASCII commands
---------------------
If you really need to extend the ASCII protocol, you may now load
additional ASCII commands as loadable modules. We don't need a
separate module for binary commands, because those are already handled
inside memcached due to the fixed semantics on the protocol. This
isn't necessarily encouraged, but sometimes it is required to get
something done quick.
3 New stats
~~~~~~~~~~~~
There are a number of new stats introduced. The key supplied in the
status command is passed to the storage engine to allow the storage
engine to add extra information to the existing stats commands, and to
create their own stat commands.
3.1 Stats returned by the default stats command
================================================
3.1.1 libevent
---------------
Over the time we've seen a lot of bugs around people using an old
version of libevent. That's part of the reason why we bundle a well
known version of libevent in the release distribution. Memcached
checks the libevent version during startup, and will refuse to start
if the one used is too old. Since most operating systems use shared
libraries these days, you might be using another version than the one
you originally used when you first built memcached. In order for us to
see which library people are using we decided to put it into the stats
as well.
3.1.2 rejected_conns
---------------------
The number of times a connection attempt was refused (due when we're
hitting the maximum number of connections.
3.1.3 stats related to TAP
---------------------------
There are a number of stats related to the packages used in the TAP
protocol. These stats will only appear if they are non-zero:
tap_checkpoint_start_received tap_checkpoint_start_sent
tap_checkpoint_end_received tap_checkpoint_end_sent
tap_connect_received tap_delete_received tap_delete_sent
tap_flush_received tap_flush_sent tap_mutation_received
tap_mutation_sent tap_opaque_received tap_opaque_sent
tap_vbucket_set_received tap_vbucket_set_sent
3.2 topkeys
============
You may get information about the most popular keys in memcached by
exporting the environment variable MEMCACHED_TOP_KEYS to the number of
keys you would want memcached to keep track of. There is no such thing
as a free lunch, so enabling this can have a small memory and speed
impact. We've decided to _disable_ this by default, so you need to
export this variable to enable the feature. Ex:
me@localhost:> MEMCACHED_TOP_KEYS=10 ./memcached
Running "stats topkeys" would return something like
STAT my-key2 get_hits=0,get_misses=1,cmd_set=0,incr_hits=0,
incr_misses=0,decr_hits=0,decr_misses=0,delete_hits=0,
delete_misses=0,evictions=0,cas_hits=0,cas_badval=0,
cas_misses=0,ctime=2,atime=2
STAT my-key1
get_hits=1,get_misses=0,cmd_set=1,incr_hits=0,
incr_misses=0,decr_hits=0,decr_misses=0,delete_hits=0,
delete_misses=0,evictions=0,cas_hits=0,cas_badval=0,
cas_misses=0,ctime=12,atime=12
(Line breaks and indentations added to make it more readable in this
document):
3.3 aggregate
==============
The combination of the storage engine interface and the SASL auth
allows for the combination of a connection-based stats. The aggregate
subcommand is used to aggregate the stats from all of the connections
on the server. The stats returned from the aggregate subcommand is the
same as the normal stats command.
3.4 settings
=============
There are times an engine may want to share details about it's
configuration through stats. This argument to stats will get you
there.
Just to show a couple of examples...
3.4.1 extension
----------------
Displays one of the extensions loaded (may appear multiple times).
ex:
STAT logger syslog
STAT ascii_extension scrub
STAT ascii_extension noop
STAT ascii_extension echo
3.4.2 topkeys
--------------
The number of keys we are monitoring.
There may be many other settings exposed, depending on the engine's
configuration.
Adam Lee
is there somewhere i can copy edit this document?
a bit nitpicky, i know, but i found a few mistakes just while browsing it... section 2.1 both "suites" should be "suits," section 3.4 "it's" should be "its," etc.
awl
Marc Bollinger
Dustin
On Apr 11, 3:48 pm, Marc Bollinger <mbollin...@gmail.com> wrote:
> I was actually going to ask if this was a draft of a changelog going on
> github/memcache.org/etc, because it already seems pretty well-formatted to
> be a one-off email, and would be useful to point others to, without
> forwarding.
communicate the work to list readers and see what kind of discussion
ensues. It's slightly surprising that the discussion is about the
form of the communication, but it's reasonable.
As we are prepping a new release, having a release page with release
notes is consistent with our previous releases, so it makes sense to
me.
In the meantime, this is the canonical URL:
http://groups.google.com/group/memcached/browse_thread/thread/4041bead80c8f14c
:)
dormando
everything? Does anything suck? Need more clarification? Whatever?
http://code.google.com/p/memcached/downloads/detail?name=memcached-1.6.0_beta1.tar.gz
^ easy-bake oven form beta release. Passes tests on a bunch of platforms,
but possibly not OpenBSD.
Make evaluating! Give major feedback.
-Dormando
dormando
On Mon, 11 Apr 2011, Adam Lee wrote:
>
> is there somewhere i can copy edit this document?
>
> a bit nitpicky, i know, but i found a few mistakes just while browsing it... section 2.1 both "suites" should be "suits," section 3.4 "it's" should
> be "its," etc.
>
> awl
>
Does anything besides the english not sit well with you???
Trond Norbye
On 12. apr. 2011, at 05.55, dormando wrote:
> ps. folks please look this over and evaluate. Do you understand
> everything? Does anything suck? Need more clarification? Whatever?
>
> http://code.google.com/p/memcached/downloads/detail?name=memcached-1.6.0_beta1.tar.gz
> ^ easy-bake oven form beta release. Passes tests on a bunch of platforms,
> but possibly not OpenBSD.
>
I've fixed that bug. See: https://github.com/memcached/memcached/commit/cc3941084188195fc8b43fcdc05cec3dab5a4bd4
Cheers,
Trond
Nelz
- Nelz
Trond Norbye
On 12. apr. 2011, at 18.56, Nelz wrote:
> What is the difference between GAT and GATQ?
>
GAT will return with "KEY NOT FOUND" if the item isn't in the cache, whereas GATQ only returns information about the items found.
Cheers,
Trond
Brian Aker
On Apr 11, 2011, at 12:01 PM, Trond Norbye wrote:
> Intentionally, there is no significant difference in protocol over
> 1.4.x. There is one minor change, but it should be transparent to
> most users.
What is the change?
When doing a diff I could one change in an ENUM, and also the addition of "touch" which has no description.
> Defining this now will allow us to start moving down the path to
> vbuckets in the default_engine and allow other engine implementors to
> consider vbuckets in their design.
Looking through the notes I don't see any discussion of what the format for configuration is. When looking through what Dustin has it seems to be a reference to some JSON which isn't documented. Also, is everyone expected now to install CouchDB in order to use Memcached? That seems like a hefty requirement.
> change in the server. It is solely up to the backing store to
> implement this, so it can make decisions about what resources are used
> to implement TAP. This functionality is commonly needed enough though
> that the core is aware of it, leaving specific implementation to
> engines.
Can this stream be documented? How about picking a name other then TAP so that there is no confusion between this and "System Tap" which the Linux kernel folks have.
> There are a few new commands available. The following sections
> provides a brief description of them. Please check protocol_binary.h
> for the implementation details.
There is just an ENUM in the file, that isn't enough for anyone who is perusing the document. The original document Brad did is a good example of what should be done.
> The touch command lets you set the expiry time for an object without
> retrieving the object. In most cases, you will not want to do this
> unless you provide a CAS value to ensure that you're touching the
> correct version of the object.
What happens if the object is not available? Is it possible that the server will send an error back? What errors?
> 1.3.3 SET_VBUCKET, GET_VBUCKET, DEL_VBUCKET
> --------------------------------------------
>
> These commands are used to set, get or delete a vbucket on the server.
How should these commands be structured? What possible errors exist? Is there a reference implementation?
> 2.1 Engines
> ============
> environment. People have different requirements for their server. Some
> people need ACID, others may prefer ecstacy ;-) The storage interface
How can you perform XA with this? If not, does this mean that each operation in transactional? How is a conflict reported? If this is an ACID interface, what is the isolation for events?
> 3.3 aggregate
> ==============
>
> The combination of the storage engine interface and the SASL auth
> allows for the combination of a connection-based stats. The aggregate
> subcommand is used to aggregate the stats from all of the connections
> on the server. The stats returned from the aggregate subcommand is the
> same as the normal stats command.
Are all of the stats being updated so that users can see it? How is this being performed in a backwards compatible way?
Its great to see work being done, but we need a lot more information to support this.
Alan has commented, and from a quick run through it looks like top keys cuts performance in half. Is this going to be solved before a GA release?
Cheers,
-Brian
Dustin
> > Intentionally, there is no significant difference in protocol over
> > 1.4.x. There is one minor change, but it should be transparent to
> > most users.
>
> What is the change?
means that retroactively, all operations have been specifying vbucket
0.
> When doing a diff I could one change in an ENUM, and also the addition of "touch" which has no description.
include/memcached/protocol_binary.h" It will show you things like
this:
commit eeaeeede5ddaaed8179389876866756ce1c05158
Author: Trond Norbye <trond....@gmail.com>
Date: Tue May 18 21:55:02 2010 +0200
Use the reserved bytes in the protocol header for vbucket
diff --git a/include/memcached/protocol_binary.h b/include/
memcached/protocol_bi
index 1c9a2a4..1e10ac0 100644
--- a/include/memcached/protocol_binary.h
+++ b/include/memcached/protocol_binary.h
@@ -73,6 +73,7 @@ extern "C"
PROTOCOL_BINARY_RESPONSE_EINVAL = 0x04,
PROTOCOL_BINARY_RESPONSE_NOT_STORED = 0x05,
PROTOCOL_BINARY_RESPONSE_DELTA_BADVAL = 0x06,
+ PROTOCOL_BINARY_RESPONSE_NOT_MY_VBUCKET = 0x07,
PROTOCOL_BINARY_RESPONSE_AUTH_ERROR = 0x20,
PROTOCOL_BINARY_RESPONSE_AUTH_CONTINUE = 0x21,
PROTOCOL_BINARY_RESPONSE_UNKNOWN_COMMAND = 0x81,
@@ -168,7 +169,7 @@ extern "C"
uint16_t keylen;
uint8_t extlen;
uint8_t datatype;
- uint16_t reserved;
+ uint16_t vbucket;
uint32_t bodylen;
uint32_t opaque;
uint64_t cas;
> Looking through the notes I don't see any discussion of what the format for configuration is. When looking through what Dustin has it seems to be a reference to some JSON which isn't documented. Also, is everyone expected now to install CouchDB in order to use Memcached? That seems like a hefty requirement.
related to CouchDB or even JSON in the memcached tree anywhere.
vbuckets are numbers with states corresponding to numbers in the
binary protocol. You turn them on or you turn them off using vbucket
commands.
It's only there for plumbing. It's best thought of as a hashing mode
with a tiny bit of help from the server. The client hashes the key to
a vbucket, and then has a mapping of the vbucket to the server. The
client tells the server that it wants something stored under vbucket x
and the server can refuse it.
The memcached server cares neither why the client sent vbucket x or
why it was told it should or shouldn't service requests to vbucket x.
The only difference between this and any other hashing strategy is
that the server can participate just ever so slightly in the hashing
to verify you're not sending things to the wrong place when two
clients have conflicting server lists.
This commit added the actual commits to the tree:
https://github.com/memcached/memcached/commit/8f449322cd31f9af28cd7849b1eb0a09030cf2bc
It has stuff like this in it:
+ * Definition of the packet used by set vbucket
+ */
+ typedef union {
+ struct {
+ protocol_binary_request_header header;
+ struct {
+ vbucket_state_t state;
+ } body;
+ } message;
+ uint8_t bytes[sizeof(protocol_binary_request_header) +
sizeof(vbucket_s
+ } protocol_binary_request_set_vbucket;
+ /**
+ * Definition of the packet returned from set vbucket
+ */
+ typedef protocol_binary_response_no_extras
protocol_binary_response_set_vbu
+ /**
+ * Definition of the packet used by del vbucket
+ */
+ typedef protocol_binary_request_no_extras
protocol_binary_request_del_vbuck
+ /**
+ * Definition of the packet returned from del vbucket
+ */
+ typedef protocol_binary_response_no_extras
protocol_binary_response_del_vbu
+
+ /**
+ * Definition of the packet used by get vbucket
+ */
+ typedef protocol_binary_request_no_extras
protocol_binary_request_get_vbuck
+
+ /**
+ * Definition of the packet returned from get vbucket
+ */
+ typedef union {
+ struct {
+ protocol_binary_response_header header;
+ struct {
+ vbucket_state_t state;
+ } body;
+ } message;
+ uint8_t bytes[sizeof(protocol_binary_response_header) +
sizeof(vbucket_
+ } protocol_binary_response_get_vbucket;
> Can this stream be documented? How about picking a name other then TAP so that there is no confusion between this and "System Tap" which the Linux kernel folks have.
floating around. I think enough people have implemented it now in
enough languages for enough use cases that we've hit most of the major
snags so there's not a lot of justification for not widely
distributing documentation that was collected as a result of doing
that work.
It's basically no different from the binary protocol in general, it
just runs the other way. I've got a doc I wrote a while back I'll put
somewhere if we can't find anything better. I'll send that out today
or so.
I don't know that I'm that concerned about the name being that bad.
It seems kind of obvious for a way to tap into the operations going on
in something and use that info to make decisions or build stuff. Just
because the Linux guys also had the same idea doesn't make it a bad
one. :)
> > There are a few new commands available. The following sections
> > provides a brief description of them. Please check protocol_binary.h
> > for the implementation details.
>
> There is just an ENUM in the file, that isn't enough for anyone who is perusing the document. The original document Brad did is a good example of what should be done.
enums. Most of those have at least some documentation. Where it's
insufficient we can make it better, of course.
Extension documents and/or protocol_binary core doc can be updated
when things are completely finalized.
> > The touch command lets you set the expiry time for an object without
> > retrieving the object. In most cases, you will not want to do this
> > unless you provide a CAS value to ensure that you're touching the
> > correct version of the object.
>
> What happens if the object is not available? Is it possible that the server will send an error back? What errors?
> > 1.3.3 SET_VBUCKET, GET_VBUCKET, DEL_VBUCKET
> > --------------------------------------------
>
> > These commands are used to set, get or delete a vbucket on the server.
>
> How should these commands be structured? What possible errors exist? Is there a reference implementation?
implements
them.
> > 2.1 Engines
> > ============
> > environment. People have different requirements for their server. Some
> > people need ACID, others may prefer ecstacy ;-) The storage interface
>
> How can you perform XA with this? If not, does this mean that each operation in transactional? How is a conflict reported? If this is an ACID interface, what is the isolation for events?
> Are all of the stats being updated so that users can see it? How is this being performed in a backwards compatible way?
add new stats without affecting most clients (I can't speak for all).
> Its great to see work being done, but we need a lot more information to support this.
>
> Alan has commented, and from a quick run through it looks like top keys cuts performance in half. Is this going to be solved before a GA release?
decided how to resolve it. This is exactly the kind of discussion
we'd like to have at this point.
On one hand, if someone can't deal with only 200k ops/s on a server,
they may not want to enable that feature in testing at this phase. On
the other, we may not want to be adding features that we can't support
at full speed.