Conn_yields, what does it mean?

[Lampkin]

Hi,

We're having a problem on our site, our site is C# based and using the
Enyim client library to connect to memcached version 1.4.1. We're
seeing an issue whereby objects are moving between memcached instances
and the distribution of objects between instances is uneven, I've
placed the stats from all 3 instances at the end.

I've been using a key to track certain configuration options, I'm not
sure when it started or if it's always been the case, but this key
starts in machine 1, after a minute or so also ends up in machine 2
and finally after a little while longer in machine 3. The code which
sets this key tries to get the key, if it can't find it, reverts to
defaults then sets it again.

We are using multi-get operations to retrieve large blocks of objects
and I understand this is a factor. Looking through the stats I've
noticed the conn_yields figure is very high, especially high on the
machine which everything seems weighted towards. I've looked at the
140 release notes but I'm still unclear what this means and what the
implications for the client would be? The client code is written in
such a way that if it has connection problems with a machine, it will
essentially drop that instance from it's list for 2 minutes then move
onto another instance.

Running the site locally, the conn_yields figure stays are zero, I
haven't yet managed to get it to increment.

If there's any further information on what conn_yields is I'd
appreciate being pointed in it's direction.

Machine 1:
pid 13307
uptime 3655193
time 1295524708
version 1.4.1
pointer_size 64
rusage_user 17003.106627
rusage_system 50838.445204
curr_connections 2406
total_connections 217961
connection_structures 16524
cmd_get 2415743505
cmd_set 72281582
cmd_flush 2
get_hits 2356349403
get_misses 59394102
delete_misses 0
delete_hits 15
incr_misses 0
incr_hits 0
decr_misses 0
decr_hits 0
cas_misses 0
cas_hits 0
cas_badval 0
bytes_read 880604464747
bytes_written 37640769535050
limit_maxbytes 7340032000
accepting_conns 1
listen_disabled_num 0
threads 5
conn_yields 289005
bytes 460185939
curr_items 231739
total_items 72281582
evictions 0

Machine 2:
pid 9982
uptime 3648026
time 1295524820
version 1.4.1
pointer_size 64
rusage_user 4823.213432
rusage_system 13898.220584
curr_connections 2375
total_connections 212736
connection_structures 21352
cmd_get 894481047
cmd_set 33463399
cmd_flush 2
get_hits 864040668
get_misses 30440379
delete_misses 2
delete_hits 5
incr_misses 0
incr_hits 0
decr_misses 0
decr_hits 0
cas_misses 0
cas_hits 0
cas_badval 0
bytes_read 473656621966
bytes_written 18799490063006
limit_maxbytes 7340032000
accepting_conns 1
listen_disabled_num 0
threads 5
conn_yields 1781
bytes 242260103
curr_items 123278
total_items 33463399
evictions 0

Machine 3:
pid 13283
uptime 3648041
time 1295524820
version 1.4.1
pointer_size 64
rusage_user 3207.792474
rusage_system 9052.337735
curr_connections 2375
total_connections 203761
connection_structures 14343
cmd_get 606330100
cmd_set 22662713
cmd_flush 2
get_hits 578153779
get_misses 28176321
delete_misses 0
delete_hits 21
incr_misses 0
incr_hits 0
decr_misses 0
decr_hits 0
cas_misses 0
cas_hits 0
cas_badval 0
bytes_read 293621360835
bytes_written 12359029477102
limit_maxbytes 7340032000
accepting_conns 1
listen_disabled_num 0
threads 5
conn_yields 11075
bytes 160258546
curr_items 80693
total_items 22662713
evictions 0

[Paul Gale]

The definition of conn_yields as taken from the latest protocol.txt document for memcached is:

"Number of times any connection yielded to another due to hitting the -R limit."

Note that the default for -R is 20. This prevents a single connection from hogging the memcached instance and starving other connections of their requests.

 

[Sebastian Llabres]

Thanks, we kind of understood that already. I'm just wondering, what behaviour would I expect to see from the client?  What is the R limit exactly? For instance, if I send a multiget with more than the R limit it doesn't result in a connection yield, even if I send a large quantity concurrently.  

I literally had to open up thousands of threads and applications to reproduce some connection yields locally, which given the circumstances isn't easy to debug :D

Thanks!

[Paul Gale]

For local verification purposes you could always set the value of -R on the command line to be lower than its default of 20 (unless it latches at that value) as well as set the value of -t to be set to 1 or 2 perhaps (default is 4).

>What is the R limit exactly?

Please clarify: are you asking what it's value is or what behavior it governs or something else?

Thanks,
Paul

[dormando]

It's more about one connection getting a huge sack of commands in one
read. multigets should be an easy way to get it to yield though, if you
use thousands. Though if you're using binprot it'll be a lot harder to
cause a yield, as it still tends to split commands into different packets.

[Sebastian Llabres]

On Tue, Jan 25, 2011 at 4:42 PM, Paul Gale <paul....@gmail.com> wrote:

For local verification purposes you could always set the value of -R on the command line to be lower than its default of 20 (unless it latches at that value) as well as set the value of -t to be set to 1 or 2 perhaps (default is 4).

>What is the R limit exactly?

Please clarify: are you asking what it's value is or what behavior it governs or something else?

Thanks,
Paul

What the value means? I, wrongly it seems, assumed that if I set the R value to 1, then sending a multiget with more than 1 get would cause a connection yield.  This turned out not to be the case.  So I guess I'm asking what behaviour it governs.

Thanks for the replies!

[Sebastian Llabres]

By connection, does that apply to everything coming from one machine? Or does it apply only to clients running on that machine? Or even the individual request made by the client?

[Paul Gale]

I'm assuming (although I don't know for sure) that unless there is another client to yield to then it won't as there's no benefit in doing so. Perhaps if you launched a number of clients that all made requests at the same time then that might be a more realistic test. Also, see how the behavior varies, if at all, when doing a multiget of ten keys vs. ten sequential gets on the same keys for example.  Just a thought.

Paul