high load average

[Andre]

Hi Everybody,

we are new to membase, at the moment the cluster is working just fine,
but we have pretty high load average...

our environment:

2 webserver behind a load balancer
the membase is installed directly on the webservers, version 1.6.5.3
each webserver has 16 GB ram and a cpu with 2x AMD Opteron(tm)
Processor 250

the cluster is defined to use 12 GB ram of each node
there is a membase bucket: RAM Quota: 16GB, Replication: disabled
RAM Used: 3.5%, Item Count: 283602, Memory Bytes Used: 603M
so the items are about 2K each in average

at the moment the web load balancer gets 10 requests per second in
average, every web request makes a "get" and a "set" to the mambase
cluster, so there are 20 operations per second on the cluster in
average

actually not very much load... we expect up to a couple of thousend
request per second in the future...

what we are very worried about is the high load average at the moment
on the nodes, without being requested frequently...
the load average is pending between 1.5 and 3.5, at the same time the
CPU is used just at about 10%

what are we doing wrong? any suggestions?

best regards
andre

[Matt Ingenthron]

Hi Andre,

Even on an idle cluster, there's a lot of paying attention to statistics
of what's happening. This will remain constant as you add load (the
statistics captured remain the same, whether idle or under load). I can
say from experience, you'll get your "couple of thousands of requests
per second" without any trouble, and probably without much more CPU time
than you currently see.

If you find it to not be the case, please let us know and we can help out.

Regards,

Matt

--
Matt Ingenthron
Couchbase, Inc.

[Andre]

Hi Matt,

thanks a lot for the reply!
indeed, the response times are pretty good, everything is working as
supposed to

so there is no problem to have a very high load average for long
time?... i could observe the value above 5 during frequent visits,
while our cpu doas have only two cores!
usually i was worrying about high load average, so may be i should
reconsider my understanding of the load average...

regards
andre

[Perry Krug]

Load average can come from a variety of factors, it's more of a high level value that can lead you to look in other places.

One of the most common reasons (especially for Membase) has to do with access to disk.  Can you look deeper into your CPU stats and see if there is a significant amount of IO wait?

Perry Krug Solutions Architect direct: 831-824-4123 email: pe...@couchbase.com