Cluster CPU usage disproportionate

[Guy]

Hi,

I have a cluster of 3 nodes that are identical in every way. I have 2
membase buckets set up across the 3 nodes. We have 2 moxi clients
configured on each client web server, each moxi client points to one
of the 2 buckets. I manage the moxi cluster config file on each web
server using puppet so the order of the membase servers in the cluster
config file is the same on each web client. So far I haven't seen any
issues with these servers and everything appears to be ok.

However, today I was looking at the membase admin console and for some
reason the CPU usage on one of the nodes is under much higher load
than the other 2. The cpu usage on 2 of the nodes is between 0 to 5%
whereas on the other node it consistenly hangs around 50%. I watched
it on and off for about half an hour and I didn't see any fluctuation
from this pattern.

Any idea why this is? Is this expected behaviour?

Thanks,
Guy

[Guy]

[Guy]

My apologies for posting twice.

[Matt Ingenthron]

It is expected behavior to a degree.

It is true that every node is like every other node, but there are
actually some internal, non-critical processes that run in one and only
one place. This probably accounts for the additional CPU utilization.

The other thing that may account for it is that all of your moxis may be
pointed to that one system. It is actually a best practice to distribute
the moxi load and there is (unfortunately) a known bug with memory
utilization from client connections that should be addressed soon. We've
only seen it actually affect things in very large deployments.

Hope that helps,

Matt

--
Matt Ingenthron
Couchbase, Inc.

[Aliaksey Kandratsenka]

50% cpu usage is too large and looks completely abnormal. Can you ssh to this node and check which process eats CPU.

Also filing bug here http://www.couchbase.com/issues/secure/Dashboard.jspa would be nice.

[Guy]

Thanks for the feedback. I checked out the problem node and see in top
that the %CPU fluctuates wildly between lowish values (~30%) and high
values (108%?). However the load average is quite low. On the other
(non-problematic) nodes however, the %CPU is only sitting around 2 -
5% which corresponds with what I see on the admin page.

As for the moxi clients being pointed to the one system, I have the
moxi-cluster config set up for each bucket with the pipe-separated
string of all 3 server nodes. However all cluster config files on all
machines have exactly the same server order (the problem server is
last). I'm not sure if the order is a factor.

Regards,
Guy

On Mar 8, 6:11 pm, Aliaksey Kandratsenka <alkondrate...@gmail.com>
wrote:

[Chad Kouse]

Which process was fluctuating on CPU utilization?

--chad

[Aliaksey Kandratsenka]

On Fri, Mar 9, 2012 at 08:10, Guy <g...@eastsidegamestudio.com> wrote:

Thanks for the feedback. I checked out the problem node and see in top
that the %CPU fluctuates wildly between lowish values (~30%) and high
values (108%?). However the load average is quite low. On the other
(non-problematic) nodes however, the %CPU is only sitting around 2 -
5% which corresponds with what I see on the admin page.

As for the moxi clients being pointed to the one system, I have the
moxi-cluster config set up for  each bucket with the pipe-separated
string of all 3 server nodes. However all cluster config files on all
machines have exactly the same server order (the problem server is
last). I'm not sure if the order is a factor.

It could be, but I don't think it could be that high.

Moxi tries to contact nodes in order. And first successfully reached node is used for streaming cluster configuration. So all moxis will typically use same node and that places some higher load on this first node.

But we've seen this causing noticeable issues only on far larger deployments.

So what process is eating CPU?

[Guy]

Actually, it appears to be alternating between memcached and beam.smp.
They really are both fluctuating fairly rapidly across a wide range of
values.

On Mar 9, 8:44 am, Aliaksey Kandratsenka <alkondrate...@gmail.com>
wrote:

[Guy]

I should mention, this is a small cluster - 3 cloud instances with
Softlayer. Each server has 2 cores and 1GB RAM.

[Aliaksey Kandratsenka]

On Fri, Mar 9, 2012 at 11:31, Guy <g...@eastsidegamestudio.com> wrote:

Actually, it appears to be alternating between memcached and beam.smp.
They really are both fluctuating fairly rapidly across a wide range of
values.

Please, file bug with diagnostics from this "broken" machine.

[Guy]

Ok, will do.

On Mar 9, 11:34 am, Aliaksey Kandratsenka <alkondrate...@gmail.com>
wrote:

[xerxes]

I am having similar issues with 3 nodes in a cluster and cpu of one of
them spiking to max while others are very low. Load is very low and also
low traffic to the server.
I am not familiar what moxi does and how that can help with load distribution between the nodes.
The dominant processes are beam and memcache.

Thanks for your help.
Farid