How to scale net.rubyeye.xmemcached.transcoders.SerializingTranscoder.decode ? ( for response time )

[Andras Szerdahelyi]

Hello list,

i might be facing a small scalability bump in my application.

I'm doing about 7k GETs and 1k CAS operations a second to a memcached cluster of 2 nodes and noticed the high quantiles for the GETs declide rather badly ( bad is up for definition, but anyway ). My application is extremely latency sensitive ( real time bidding ) and this is hurting me badly and is not at all present with just a little less load ( non-linear performance )

these are the high quantiles for a GET operation via xmemcached

99th percentile - 20ms

98th percentile - 12ms

95th percentile - 8ms

with about half the requests/sec ( but same concurrency ), i get a fraction of the latency, the 99th percentile is at around 3ms.

Now, what i noticed is that this single call is taking up 20% of CPU time

 net.rubyeye.xmemcached.transcoders.SerializingTranscoder.decode:86

( relevant part of profiler output attached )

- During the peak of 7k read 1k CAS, my load is around 10 on a 24 CPU system. 

- on avg this GET/CAS traffic is produced by 25 concurrently running threads

- i have 2 connections configured

- i have 2 Memcache Client instances with random distribution of requests

my questions are:

- are these numbers realistic?

- should i use a different serializer?

- could something else be happening that i'm not aware of? ( although: no errors/warnings from net.rubyeye* )

i appreciate any assistance!

thanks,

Andras 

[dennis zhuang]

Hi, sorry to late to reply you.

I think you can try to reduce the merge factor in xmemcached by:

 memcachedClient.setMergeFactor(50);

Xmemcached client merges get requests as an multi-get command to reduce requests count.But it increase latency.If you application is latency sensitive,i think you can try to reduce the factor(the default value is 150,it means it will try to merge at most 150 get requests as an multi-get request).

Also,you can disable merging buffers by:

memcachedClient.setOptimizeMergeBuffer(false);

Xmemcached also try to merge commands as an packet that is socket buffer size to send to memcached.All these solutions are trying to maximum the throughput,but increase latency.

At last,you can try some TCP options to reduce latency:

MemcachedClientBuilder builder = new XMemcachedClientBuilder(
                    AddrUtil.getAddresses(servers));
         builder.setSocketOption(StandardSocketOption.SO_RCVBUF, 32 * 1024); 
            builder.setSocketOption(StandardSocketOption.SO_SNDBUF, 16 * 1024); 
            builder.setSocketOption(StandardSocketOption.TCP_NODELAY, true);

 

2013/3/21 Andras Szerdahelyi <andras.sz...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "xmemcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xmemcached+...@googlegroups.com.
To post to this group, send email to xmemc...@googlegroups.com.
Visit this group at http://groups.google.com/group/xmemcached?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
庄晓丹
Email:        killm...@gmail.com xzh...@avos.com
Site:           http://fnil.net
Twitter:      @killme2008

[dennis zhuang]

Yep, using a fast serializer is better,but you must be carefully when the cache items are accessed by other systems that aren't written in java.

You can find more serialization at here:

https://github.com/eishay/jvm-serializers/wiki

I think JSON is a good choice,you can write your own json transcoder using a json lib.Just implements the Transcoder interface:

http://fnil.net/docs/xmemcached/net/rubyeye/xmemcached/transcoders/Transcoder.html

And set it to your memached client builder.

2013/3/22 dennis zhuang <killm...@gmail.com>

[Andras Szerdahelyi]

Thanks!

I was not aware of the mergeFactor / optimizeMergeBuffer client options. Those with a different serializer ( i'm trying Kryo now ) should help reduce my latency.

regards,

Andras