New optimized engine for Memecached 1.6

[rajiv]

Hi,

 

For anyone interested, I just posted a new 1.6 branch called "bagLRU" at https://github.com/rajiv-kapoor/memcached/tree/bagLRU that includes an optimized engine called “bag_lru_engine”.  The Bag LRU engine implements striped locks for parallel hash table accesses and a scheme for handling requests that give us lock-free GETs.  Additionally I have added a command line option to allow setting CPU affinity for Memcached worker threads.  I have noticed that CPU affinity along with NIC queue (IRQ) affinity can help improve performance significantly.

 

To use the Bag LRU engine use the “-E path_to_memcached_root/engines/.libs/bag_lru_engine.so” option to Memcached

To affinitize Memcached worker threads to “CPUs” (cores or logical threads) use the command line option “-T n”.  Where "n" is the increment used for identifying the next CPU to bind the Memcached thread to.  For example, "-T 1" will bind CPUs sequentially to the Memcached threads starting with CPU 0.  “-T 2” will skip every other CPU starting with CPU 0.  Default is “-T 0” - meaning no thread affinity.

 

Using mcblaster to generate a GETs only load, I have tested the Bag LRU engine on a 16 core Sandybridge system from 1 to 16 threads and it shows almost linear scaling with number of cores.  For this test thread pinning option used was “-T 1”.

 

If you do try it out, I would love to get feedback on it.

 

Thanks,

\rajiv

Intel, Corp.

[p4 guru]

Hi rajiv

Could you elaborate on how to use Bag LRU, not sure what you mean by use -E path to memcached root wherever bag_lru_engine.so is ?

thanks

On Saturday, June 30, 2012 4:00:00 AM UTC+10, rajiv wrote:

Hi,

 

To use the Bag LRU engine use the “-E path_to_memcached_root/engines/.libs/bag_lru_engine.so” option to Memcached

 

Thanks,

\rajiv

Intel, Corp.

[rajiv]

I just realised that there was a typo in the -E option in my original post.  It should read :

 

“-E path_to_memcached_root/.libs/bag_lru_engine.so”

 

Say you installed the memcached source tree at say /home/memcached.  After you complete the build, a bunch of memcached related shared objects will get placed in /home/memcached.libs

 

Among those shared objects will be the ones specific to the "engines".  By default the standard 1.6 memcahed tree includes one engine called "default_engine" whose correponding shared object is /home/memcached/.libs/default_engine.so.  The bagLRU branch has an additional eninge called "bag_lru_engine" whose implementation is in /home/memcached/.libs/bag_lru_engine.so .  If you do not use the "-E" option to memcached it uses the default_engine.  In order to use the bag LRU optimizations you need to use “-E /home/memcached/.libs/bag_lru_engine.so" option when launching memcached.

 

Hope that clears it up.

 

thanks,

\rajiv

[Trond Norbye]

Hi,

I just looked at your branch and:

1) the first patch there that adds the -T option contains just part of the functionality. Most of it is (incorrectly) placed in the next patch that adds the new engine

- cpu.h use err.h which I don't think is available on all systems (at least some of my old builders didn't have it)

- cpu.h use code that doesn't work on one of my builders (solaris):

daemon/cpu.h: In function ‘bind_thread_to_cpu’:

daemon/cpu.h:109:5: error: unknown type name ‘cpu_set_t’

daemon/cpu.h:110:5: error: implicit declaration of function ‘CPU_ZERO’ [-Werror=implicit-function-declaration]

daemon/cpu.h:111:5: error: implicit declaration of function ‘CPU_SET’ [-Werror=implicit-function-declaration]

daemon/cpu.h:112:5: error: implicit declaration of function ‘sched_setaffinity’ [-Werror=implicit-function-declaration]

daemon/cpu.h:112:37: error: ‘cpu_set_t’ undeclared (first use in this function)

I think your configure script should check for the methods you support and just print a warning if you don't know how it's implemented (instead of falling back to a linux way). For solaris you should look at pset_create/pset_bind etc.

2) The next patch contains a number of other problems:

- The code is pinned to linux with:

"engines/bag_lru_engine/items_bl.c", line 19: cannot find include file: <linux/unistd.h>

(If I remove the linux/ part it builds on my boxes, how does linux/unistd.h differ from unistd.h?)

- The code assumes you're using the gcc compiler (given the 

"engines/bag_lru_engine/items_bl.c", line 81: error: implicit function declaration: __sync_bool_compare_and_swap

"engines/bag_lru_engine/items_bl.c", line 750: error: implicit function declaration: __sync_add_and_fetch

"engines/bag_lru_engine/items_bl.c", line 2078: error: implicit function declaration: __sync_sub_and_fetch

"engines/bag_lru_engine/items_bl.c", line 2125: error: statement not reached

I didn't look too closely at the code, but it seems to be "a lot" of code that is commented out.. that should be cleaned up.

Cheers,

Trond

--
Trond Norbye

[maysam lavasani]

Hi Rajiv,

I was reading the paper "Enhancing the Scalability of Memcached" and found it interesting. nice work.

Have you guys tested get on top of UDP? (I assume the published performance numbers are based on get over TCP).

Regards,

Maysam

UT Austin 

PhD student

[Aris Ioannou]

and how can I install this branch? There is no configure file in the source code...

Regards

Aris