memcached hangs suddenly and slows down

[rch]

I am using memcached in my production environment.
It works fine 90% of the time.
But suddenly, it slows down. when I try to connect via telnet the
connect part is slow.
How can I debug this?

Also one more thing that is not clear is, stats shows the current
connections more than 1024. I am starting memcached with default
settings. so, I thought max connections that should be allowed are
1024. But I see below.. which is strange.

STAT curr_connections 1063
STAT total_connections 79538
STAT connection_structures 1092

At this time the server is responding fine. Why am I seeing more than
1024 in current connections?

[Ravi Chittari]

Any one who can help on this?

[Edward M. Goldberg]

Well I see that you are not getting lots of replies so I will step in.

What I do is run a small CRON job that does a few Memcache actions
like:

Set key
Get key
Get status....

And watch the output for issues on each server. I use plain commands
like:

echo "stats" | nc localhost 11211

This way I see all of the interaction with no client software in the
way.

When an error shows up in these simple commands I send an Alert to the
RightScale Monitor System I use.

You may have a Munin or other monitor system.

Now I see the exact moment when the Memcached goes sour. This
provides a big hint as to why it is happening.

Issues I see are:

Out Of Memory errors.
Time out on connect.
No Connect.

After a few week of looking at the issues, I can get all of these
issues fixed and the code works great.

It does take a few hits at first to find issues like DNS falures, Set
up values set too low.

Please ask for help, Memcache is a great Cache....

Edward M. Goldberg
http://myCloudWatcher.com/

[Ravi Chittari]

Thanks Edward.
I do get an alarm when things are bad on the memcached. But the issue I want to figure out is, why and how it got itself into a bad state.
Do you suggest turning on the debug logs?

>>>>Out Of Memory errors.
>>>>Time out on connect.
>>>>No Connect.

>>>>After a few week of looking at the issues,  I can get all of these
>>>>issues fixed and the code works great.

Where are your fixes going? on the client side or on the server side?

Thanks,
Ravi.

[dormando]

Hey,

What version are you on?

Is the machine it's on swapping? how much memory is free? cpu free? what's
the general load on the box?

What's the commandline you're using to start memcached?

-Dormando

[Ravi Chittari]

Version is 1.2.5

memcached -d -m 256
it is not going into swap or anything like that.  It has an 8GB RAM,  the total RAM used up is 5GB for other apps. 

But when it hangs, I see that cpu is 100% and memcached is using up the cpu.

Thanks,
Ravi.

[dormando]

Please upgrade; version 1.2.5 has a number of spin/crash bugs that have
since been fixed.

We highly recommend 1.4.1, but 1.2.8 is still available.

-Dormando

[Ravi Chittari]

Thanks Edward..  One last question.
Actually the version we are using is 1.2.2, I believe 1.2.2 has similar issues as 1.2.5.
You think 1.2.2 has similar issues as 1.2.5?

is 1.2.8 stable version but less features than 1.4.1?

Thanks,
Ravi.

[dormando]

1.2.2 is even worse than 1.2.5. Please upgrade to 1.4.1 if possible.