event_queue_remove not on queue 8
Beier
but last Friday and this morning, during peak traffic time memcached
server went down without much error message. the only message logged
is:
[err] event_queue_remove: 0x62fad8(fd 27) not on queue 8
my guess is something wrong with libevent? but no idea what made this
happening. any help will be really appreciated
Aaron Stone
libraries (just for completeness).
What do you mean by "went down"? Did it become unresponsive? Did the
process die? Memory use spin out of control? Anything else catch on
fire?
Beier
libevent 1.4.13
memcached 1.4.3
client library is libmemcached-0.35
what I mean by 'went down' is, the service is dead, I had to start
memcached again
and here is the log file:
[err] event_queue_remove: 0x62fad8(fd 27) not on queue 8
[err] event_queue_remove: 0x62fad8(fd 27) not on queue 8
the service was dead twice, that's why you see two of the same
messages:
these are the options when memcached is started
-d
-m 2048
-c 3000
Thanks!
Aaron Stone
I am not so deeply familiar with libevent.
On Mon, Feb 22, 2010 at 10:49 AM, Beier <beie...@gmail.com> wrote:
> Hi Aaron,
>
> libevent 1.4.13
> memcached 1.4.3
>
> client library is libmemcached-0.35
>
> what I mean by 'went down' is, the service is dead, I had to start
> memcached again
Ok, but did the process die, or did the process stay alive but become
unresponsive to the network?
Beier
nothing.
On Feb 22, 11:07 am, Aaron Stone <sodab...@gmail.com> wrote:
> Ok, should be enough information for a local expert to begin to help.
> I am not so deeply familiar with libevent.
>
dormando
On Mon, 22 Feb 2010, Beier wrote:
> The process was totally dead, "ps aux | grep memcache" returns
> nothing.
Is your host on a 32-bit or 64-bit OS? What distribution of linux are you
running?
Was your memcached installed from source or a package? I've not seen or
heard of mc causing that error before, but I've seen a lot of bizarre
errors happen when folks run mc on a 32-bit kernel and set -m 2048, as the
actual memory usage will go above the 2G barrier.
Beier
Amazon EC2, it was installed from source
max_connection was the default 1024. I suspect that the connection
might went over 1024 (at peak time the server deals with about 500
hits / sec), so now I set it to 5000 and it has been working well so
far. But I'm expecting 'too many connections' error instead of
event_queue error, could it be libevent's problem?
dormando
On Sat, 27 Feb 2010, Beier wrote:
> the memcached sever is hosted on a 64 bit Ubuntu8.0.4 distribution on
> Amazon EC2, it was installed from source
>
> max_connection was the default 1024. I suspect that the connection
> might went over 1024 (at peak time the server deals with about 500
> hits / sec), so now I set it to 5000 and it has been working well so
> far. But I'm expecting 'too many connections' error instead of
> event_queue error, could it be libevent's problem?
Smells like a libevent bug... or perhaps a new bug from us. Not sure; will
research this more.
You can tell if you're hitting mas conns by looking at the
listen_disabled_num counter. If you hit max conns and it's crashing every
time instead of limiting new conns, you've got a bug we're interested in
fixing.
Any chance you could test that out real quick?
Beier
starts, and then close the database and memcached connection right
before php script finishes execution.
Based on the logging info I have, what happened was that there was
something wrong with mysql. queries were taking very long, so both
mysql and memcached connection stayed open for seconds. After I
changed memcached connection to 5000, the mysql issued occurred again
the next morning and I found out there were about 3500 memcached
'current connections', which is much higher than peak time around 500.
Imagine 3500 connections needed when I had it 1024 in config. So I
believe it crashed because of too many connections.
dormando
> starts, and then close the database and memcached connection right
> before php script finishes execution.
>
> Based on the logging info I have, what happened was that there was
> something wrong with mysql. queries were taking very long, so both
> mysql and memcached connection stayed open for seconds. After I
> changed memcached connection to 5000, the mysql issued occurred again
> the next morning and I found out there were about 3500 memcached
> 'current connections', which is much higher than peak time around 500.
> Imagine 3500 connections needed when I had it 1024 in config. So I
> believe it crashed because of too many connections.
Like I said before, it's not supposed to crash on too many connections.
Can you reproduce this in isolation? IE; start another instance on that
box, then have a script open up 1000+ connections to it and see if it
crashes?
If you can you could get a gdb backtrace which would be useful.
Beier
this time I'm getting too many connections error instead of event
queue error. It's hard to reproduce the exact environment when the
error occurred last week as our dev team has deployed new codes
including the fix for mysql query issue that caused the whole thing.
but I'll definitely keep an eye on it and you'll be the first to know
as soon as this happens again, I'll try to include gdb dump as well.
huan ling
在 2010年2月23日星期二 UTC+8上午2:16:58，Beier写道：
dormando
not have that problem.
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>