Recommended performance settings

[Bryant]

Hi,
I've completed my integration with xmemcached and have some
questions about performance. After testing and validating my site I
began to let production traffic hit the server. I start immediately
getting several

Timed out(1000) waiting for operation

Which I handle and do not impact my site externally, but they
usually mean that a cached value wasn't retrieved or stored in
memcached.

I think this timeout is set directly on the MemcachedClient and not
via the XMemcachedClientFactory. But more broadly, what are good
baseline settings for a highly concurrent environment using
XMemcached? I was sort of surprised that operations where hitting a
1s limit. I'm trying to use memcached to avoid long delays retrieving
data and 1s is a pretty long delay.

I saw there was a page for performancing tuning, but I can't
understand most of it (English?). The one paramater that jumps out is
the number of read threads, The source says on Linux this should be
some number other than zero (the comment is unclear what's a good
number though) and the default is zero (presumably for windows).

What configuration was used for your benchmark tests?

As it stands now I think I'm faster hitting the origin data than
accessing it via memcached especially given the timeouts.

Bryant

[Bryant]

Oh and to be clear. I have a single MemcachedClient instance and I
have 100s of threads using it. If I need to pool the MemcachedClients
or something similar that would be good to know too, thanks!

[dennis zhuang]

hi,byrant

Yes,xmemcached and spymemcached will throw a TimeoutException if concurrent threads is too many.

I have some questions:

First,how many memcached servers do you have?

On linux platform,recommend you to set Configuration.setReadThreadCount(num),where num is the memcached server's number.

builder.getConfiguration().setReadThreadCount(2);

Second,how many threads access xmemcached when xmemcached start to throw TimeoutException?In fact,you have to set more long operation timeout in a high concurrent environment.Change the default operaton timeout through

 MemcachedClient.setOpTimeout(timeout).

If your data stored in memcached is small(less than 1k),you may disable nagle alogithm by 

builder.getConfiguration().setTcpNoDelay(true);

and change tcp receive buffer size to fit data's size 

builder.getConfiguration().setTcpRecvBufferSize(4*1024);

At last,i think performance turnning must have more test.It is associated with your application and enviroment.In the benchmark, xmemcached and spymemcached's operation timeout are both set to three seconds.

If your data stored in memcached is independent of each other,you can pool memcached clients to handle high concurrent requests.

2009/8/8 Bryant <publi...@myrete.com>

Oh and to be clear.  I have a single MemcachedClient instance and I
have 100s of threads using it.  If I need to pool the MemcachedClients
or something similar that would be good to know too, thanks!

--
庄晓丹

[Bryant]

Thanks for the help... so I went ahead an increased the read count,
and created a pool of MemcachedClient objects. This has gotten rid of
the timeouts I was seeing before. However, now that those errors are
gone I can see errors saying that no available session exists..

There is no avriable session at this moment

What causes this type of exception? In other words, why would no
session be available?

Thanks,

Bryant

On Aug 9, 7:52 am, dennis zhuang <killme2...@gmail.com> wrote:
> hi,byrantYes,xmemcached and spymemcached will throw a TimeoutException if

> concurrent threads is too many.
> I have some questions:
> First,how many memcached servers do you have?
> On linux platform,recommend you to set
> Configuration.setReadThreadCount(num),where num is the memcached server's
> number.
>
> builder.getConfiguration().setReadThreadCount(2);
>
> Second,how many threads access xmemcached when xmemcached start to throw
> TimeoutException?In fact,you have to set more long operation timeout in a
> high concurrent environment.Change the default operaton timeout through
>
>  MemcachedClient.setOpTimeout(timeout).
>
> If your data stored in memcached is small(less than 1k),you may disable
> nagle alogithm by
>
> builder.getConfiguration().setTcpNoDelay(true);
>
> and change tcp receive buffer size to fit data's size
>
> builder.getConfiguration().setTcpRecvBufferSize(4*1024);
>
> At last,i think performance turnning must have more test.It is associated
> with your application and enviroment.In the benchmark, xmemcached and
> spymemcached's operation timeout are both set to three seconds.
>
> If your data stored in memcached is independent of each other,you can pool
> memcached clients to handle high concurrent requests.
>

> 2009/8/8 Bryant <publish...@myrete.com>

[dennis zhuang]

Talk with xmemcached when xmemcached client was not connected to memcached server yet,it will throw a MemcachedException with message

There is no avriable session at this moment

I think you program maybe invoke xmemcached's methods before it has been connected.

builder.build();

This method would blocked until memcached servers has been connected successfully or fail.If it fail,xmemcached will try to heal the connection.

2009/8/12 Bryant <publi...@myrete.com>

--
庄晓丹

[Bryant]

Hi Dennis,
Okay, I was using builder.build(). It seems like it occasionally
loses connections (as I've mentioned outside of Eclipse the healing
doesn't work for me on my development mac or production linux box).

So I'm going to detail a little bit of what I'm seeing. Before I
dig in to that, I want to say I have a ton of respect for the time it
takes to put out an open source project. I have two myself. Also
your support has been fantastic. It's very helpful to get feedback
like this. So as I'm asking questions, I hope they come across as 'I
want to learn' and not me flaming or being a jerk. I totally
understand that this could be poor setup or usage on my part and not a
reflection of the project in general.

Okay so observations.

I've created a connection pool that has 25 MemcachedClient's (to
support 300 threads). I'm running against a localhost memcached
instance. I now get fewer connection timeouts, but in general
accessing data from memcached is an order of magnitude or two slower
than just fetching the data from it's origin source. That is I'm not
deriving any benefit from attempting to cache the data I'm caching.
Data cached is mostly less than 20k in size. Some data is closer to
300k, but that is in the minority.

When I profile my production system, I find that 80% of the time of my
server is locked up trying to access memcached...

40.942% 1517.000s show hide sun.misc.Unsafe.park()
at sun.misc.Unsafe.park()
at java.util.concurrent.locks.LockSupport.park()
at java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject.await()
at java.util.concurrent.ArrayBlockingQueue.take()
at java.util.concurrent.ThreadPoolExecutor.getTask()
at java.util.concurrent.ThreadPoolExecutor$Worker.run()
at java.lang.Thread.run()

20.485% 759.000s show hide sun.misc.Unsafe.park()
at sun.misc.Unsafe.park()
at java.util.concurrent.locks.LockSupport.park()
at java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject.await()
at java.util.concurrent.LinkedBlockingQueue.take()
at net.rubyeye.xmemcached.impl.MemcachedConnector$SessionMonitor.run
()

20.460% 758.100s show hide sun.nio.ch.EPollArrayWrapper.epollWait()
at sun.nio.ch.EPollArrayWrapper.epollWait()
at sun.nio.ch.EPollArrayWrapper.poll()
at sun.nio.ch.EPollSelectorImpl.doSelect()
at sun.nio.ch.SelectorImpl.lockAndDoSelect()
at sun.nio.ch.SelectorImpl.select()
at com.google.code.yanf4j.nio.impl.Reactor.run()

Even if I disable memcached caching (I've written my site so that I
can toggle memcached usage on and off in production). I still see a
huge percentage of time being consumed by the idle XMemcacheClients

In fact, it's roughly the same as above...

45.463% 972.400s show hide sun.misc.Unsafe.park()
at sun.misc.Unsafe.park()
at java.util.concurrent.locks.LockSupport.park()
at java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject.await()
at java.util.concurrent.ArrayBlockingQueue.take()
at java.util.concurrent.ThreadPoolExecutor.getTask()
at java.util.concurrent.ThreadPoolExecutor$Worker.run()
at java.lang.Thread.run()

26.228% 561.000s show hide sun.nio.ch.EPollArrayWrapper.epollWait()
at sun.nio.ch.EPollArrayWrapper.epollWait()
at sun.nio.ch.EPollArrayWrapper.poll()
at sun.nio.ch.EPollSelectorImpl.doSelect()
at sun.nio.ch.SelectorImpl.lockAndDoSelect()
at sun.nio.ch.SelectorImpl.select()
at com.google.code.yanf4j.nio.impl.Reactor.run()

26.228% 561.000s show hide sun.misc.Unsafe.park()
at sun.misc.Unsafe.park()
at java.util.concurrent.locks.LockSupport.park()
at java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject.await()
at java.util.concurrent.LinkedBlockingQueue.take()
at net.rubyeye.xmemcached.impl.MemcachedConnector$SessionMonitor.run
()

So it appears to me that the clients function in a 'busy-wait-loop'
and are not awoken when work exists but are constantly polling. Is
this the case? I would have assumed idle connections such as these
would be in a wait state.

Also, since these percentages are nearly the same regardless of
whether or not the clients are actually used, I suspect that most of
the delay is unrelated to accessing memcached and is more around the
cpu cycles being consumed by the clients themselves.

Are you aware of anyone using your client in a high scale multi
threaded environment? Again, please don't take this as me being mean,
I just need to understand what I should expect interms of access
times. To me anything more than a 3/10 or 4/10 and I'm better off
returning to the origin data and not caching it. I'd prefer it be
milliseconds.

In your performance benchmark. Were you pooling MemcachedClients? If
so, how big was the pool? Were the multiple threads allowed to share
clients or did each thread have to acquire/release a connection?

A few other thoughts.

Have you thought about logging events in Xemcached at info level for
things like connecting, adding servers, etc? Everything seems to be
logged at warning level. I can of course filter warning log events
out, but in cases where an unexpected error occurs it would be nice
for this to be handled separately than the information events that are
also logged as warnings.

Also, I get a fair amount of these exceptions, I'm not sure what they
mean but they seem to accompany removing sessions...

[2009-08-12 18:21:56.913] {Thread-516} WARNING
com.google.code.yanf4j.nio.impl.AbstractController remove session xxxx:
1624
[2009-08-12 18:21:56.914] {Thread-516} SEVERE
com.google.code.yanf4j.nio.impl.Reactor
java.lang.UnsupportedOperationException
[2009-08-12 18:21:56.914] {Thread-516} at
net.rubyeye.xmemcached.command.Command.setWriteBuffer(Command.java)
[2009-08-12 18:21:56.914] {Thread-516} at
com.google.code.yanf4j.nio.impl.DefaultTCPSession.writeToChannel
(DefaultTCPSession.java)
[2009-08-12 18:21:56.914] {Thread-516} at
com.google.code.yanf4j.nio.impl.AbstractSession.onWrite
(AbstractSession.java)
[2009-08-12 18:21:56.914] {Thread-516} at
com.google.code.yanf4j.nio.impl.AbstractSession.onEvent
(AbstractSession.java)
[2009-08-12 18:21:56.914] {Thread-516} at
com.google.code.yanf4j.nio.impl.SocketChannelController.dispatchWriteEvent
(SocketChannelController.java)
[2009-08-12 18:21:56.914] {Thread-516} at
com.google.code.yanf4j.nio.impl.AbstractController.onWrite
(AbstractController.java)
[2009-08-12 18:21:56.914] {Thread-516} at
com.google.code.yanf4j.nio.impl.Reactor.dispatchEvent(Reactor.java)
[2009-08-12 18:21:56.914] {Thread-516} at
com.google.code.yanf4j.nio.impl.Reactor.run(Reactor.java)
[2009-08-12 18:21:56.914] {Thread-516}
[2009-08-12 18:21:56.914] {Thread-515} WARNING
com.google.code.yanf4j.nio.impl.AbstractController Try to reconnect to
xxx.com:1624 for 1 times
[2009-08-12 18:21:56.915] {Thread-516} WARNING
com.google.code.yanf4j.nio.impl.AbstractController add session xxx.com:
1624

So the questions are again..

1. Do the clients function in a busy wait loop, consuming cycles even
when not being used?
2. Do you know of a production site using your client in a
multithreaded environment? I'd like to ask them about their setup and
what sort of times they see.
3. Does your bench mark share concurrent clients across threads or
does it use a pool with exclusive access to a client per thread.

Thanks,

Bryant

On Aug 11, 6:48 pm, dennis zhuang <killme2...@gmail.com> wrote:
> Talk with xmemcached when xmemcached client was not connected to memcached
> server yet,it will throw a MemcachedException with message
> There is no avriable session at this moment
>
> I think you program maybe invoke xmemcached's methods before it has been
> connected.
>
> builder.build();
>
> This method would blocked until memcached servers has been connected
> successfully or fail.If it fail,xmemcached will try to heal the connection.
>

> 2009/8/12 Bryant <publish...@myrete.com>

[dennis zhuang]

If your data size is 20k,and have 300 threads,i think you have to distribute your application to reduce the pressure of one single node.When data size is larger,the tps of memcached client would decrease very much.Maybe you want to pool more clients to handle these requests,but you have to known that each memcached client has opend a Selector and start some threads,if your pool is too large,then the cost of memcached clients is so great that break your application down.I think a 30-clients pool is too large.  

Thanks for your report,your application is a typical enviroment of high concurrent,it is not only about memcached client ,but also a architecture problem.

2009/8/13 Bryant <publi...@myrete.com>

--
庄晓丹

[dennis zhuang]

If you want to pool xmemcached client,in fact you don't need to implement a client pool by yourself,you can just increase server's weight to creat more connections insteadof creating more clients.For example,if you set a server's weight to 3,then xmemcached will creat three connections to this memcached server.

2009/8/13 dennis zhuang <killm...@gmail.com>

--
庄晓丹

[dennis]

Forgive me,i make a mistake here,increase the weight doesn't creat new
connections but use the same connection as many references.I think
xmemcached have to support client pool(in fact, it is a connection
pool),i will try to implement this feature.

On Aug 25, 9:15 am, dennis zhuang <killme2...@gmail.com> wrote:
> If you want to pool xmemcached client,in fact you don't need to implement a
> client pool by yourself,you can just increase server's weight to creat more
> connections insteadof creating more clients.For example,if you set a
> server's weight to 3,then xmemcached will creat three connections to this
> memcached server.
>

> 2009/8/13 dennis zhuang <killme2...@gmail.com>

>
> > If your data size is 20k,and have 300 threads,i think you have to
> > distribute your application to reduce the pressure of one single node.When
> > data size is larger,the tps of memcached client would decrease very
> > much.Maybe you want to pool more clients to handle these requests,but you
> > have to known that each memcached client has opend a Selector and start some
> > threads,if your pool is too large,then the cost of memcached clients is so
> > great that break your application down.I think a 30-clients pool is too
> > large.
> > Thanks for your report,your application is a typical enviroment of high
> > concurrent,it is not only about memcached client ,but also a architecture
> > problem.
>

> > 2009/8/13 Bryant <publish...@myrete.com>

> ...
>
> read more »