Consistent hashing compatible with php/perl

[Vicente Aguilar]

Hi

We're configuring a cluster with a 1st line of nginx-based web servers acting as proxy/cache/memcached/balancer in front of a 2nd line of Tomcat/php-fpm servers. We've developed a Tomcat filter that stores all our pages on the memcached servers, and then the nginx servers first check if the page is already available on memcache, else they proxy the request to the Tomcats.

Everything works OK when we have just one memcache server, but when we have more than one we can't make the nginx memcache hashing algorithm and the spymemcached one match. 

On the nginx side we're using this backend to access the memached servers:

http://wiki.nginx.org/NginxHttpUpstreamConsistentHash

http://github.com/replay/ngx_http_consistent_hash

And for PHP we use:

http://pecl.php.net/package/memcache

And they both match, 100% of the times. But we can't get spymemcached to match that same hashing too, we got roughly a 50% coincidence with 2 memcached servers so the algorithms don't match.

We're initializing spymemcached like this:

            DefaultConnectionFactory dfaultConnFctry = new DefaultConnectionFactory( DefaultConnectionFactory.DEFAULT_OP_QUEUE_LEN,
            DefaultConnectionFactory.DEFAULT_READ_BUFFER_SIZE,
            HashAlgorithm.CRC32_HASH);
            c = new MemcachedClient(dfaultConnFctry, AddrUtil.getAddresses(f.getProperty("presentation.staticweb.memcache")));

And of course that getProperty returns the same array of memcached servers and in the same order as we've them on both nginx and PHP.

How could we reconfigure spymemcached to match the others? Or alternatively, anybody knows a memcached consistent hashing algorithm that works 100% on these three implementations, nginx, php and Java/spymemcached?

Thanks in advance.

Regads,

-- 

 Vicente Aguilar <bis...@bisente.com> | http://www.bisente.com

[Dustin]

On Jul 27, 1:05 am, Vicente Aguilar <bise...@bisente.com> wrote:

> And they both match, 100% of the times. But we can't get spymemcached to match that same hashing too, we got roughly a 50% coincidence with 2 memcached servers so the algorithms don't match.
>
> We're initializing spymemcached like this:
>
>             DefaultConnectionFactory dfaultConnFctry = new DefaultConnectionFactory( DefaultConnectionFactory.DEFAULT_OP_QUEUE_LEN,
>             DefaultConnectionFactory.DEFAULT_READ_BUFFER_SIZE,
>             HashAlgorithm.CRC32_HASH);
>             c = new MemcachedClient(dfaultConnFctry, AddrUtil.getAddresses(f.getProperty("presentation.staticweb.memcache")));

You're not using a consistent hash here. You're using a modulus
hash with crc32 (which does little more than slow it down for your
uses).

Their "consistent hash" is probably ketama. You can configure
spymemcached to use that. If they're doing ketama correctly, it'll be
compatible.

[Vicente Aguilar]

El 04/08/2010, a las 18:43, Dustin escribió:

            DefaultConnectionFactory dfaultConnFctry = new DefaultConnectionFactory( DefaultConnectionFactory.DEFAULT_OP_QUEUE_LEN,

            DefaultConnectionFactory.DEFAULT_READ_BUFFER_SIZE,

            HashAlgorithm.CRC32_HASH);

            c = new MemcachedClient(dfaultConnFctry, AddrUtil.getAddresses(f.getProperty("presentation.staticweb.memcache")));

 You're not using a consistent hash here.  You're using a modulus
hash with crc32 (which does little more than slow it down for your
uses).

 Their "consistent hash" is probably ketama.  You can configure
spymemcached to use that.  If they're doing ketama correctly, it'll be
compatible.

Some time ago we also tried with:

new DefaultConnectionFactory( DefaultConnectionFactory.DEFAULT_OP_QUEUE_LEN,
            DefaultConnectionFactory.DEFAULT_READ_BUFFER_SIZE,
            HashAlgorithm.KETAMA_HASH);

But it also didn't match. :-( Or maybe we're doing something wrong, our developers don't seem to completely get how to configure spymemcached, we're basically on a trial-error method here.

Nobody else here is working with memcache from both PHP and Java? I don't mind switching to other library on either platform as long as we can get them to work together.

Thanks

[Chirag]

>And for PHP we use: http://pecl.php.net/package/memcache

PECL::memcache is implemented using libmemcache (see http://php.net/memcache)
PECL::memcached is implemented using libmemcached (see http://php.net/memcached
and http://github.com/andreiz/php-memcached).
libmemcached is a newer library and seems to be updated much more
frequently.

Inside libmemcached, there is a setting called
'MEMCACHED_BEHAVIOR_KETAMA_COMPAT' which can be set to
MEMCACHED_KETAMA_COMPAT_LIBMEMCACHED or MEMCACHED_KETAMA_COMPAT_SPY

Unfortunately, it doesn't look like PECL::memcached exposes this level
of granularity. We can set
$cache = new Memcached(self::SK_MEMCACHE_PREFIX);
$cache->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true)

This option enables a weighted consistent hashing distribution, but it
doesn't look like net.spy.memcached.KetamaNodeLocator supports
weighted nodes.

I hope this helps!

Thanks,
Chirag

>  Vicente Aguilar <bise...@bisente.com> |http://www.bisente.com

[Vicente Aguilar]

Hi

Inside libmemcached, there is a setting called
'MEMCACHED_BEHAVIOR_KETAMA_COMPAT' which can be set to
MEMCACHED_KETAMA_COMPAT_LIBMEMCACHED or MEMCACHED_KETAMA_COMPAT_SPY

Unfortunately, it doesn't look like PECL::memcached exposes this level
of granularity. We can set
       $cache = new Memcached(self::SK_MEMCACHE_PREFIX);
       $cache->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true)

Interesting. Never actually looked into libmemcached itself, just the Java and PHP wrappers. Will look into it.

Thanks Chirag