redis cpu usage comes odd/high
爵溪
I've use redis in production env 2 or 3 monthes . but recently the cpu
usage really become odd . very time restart it , at the beginning
it's just work fine , low cpu ,low server load ,maybe 30(or more)
minitues later , redis-server will cost about 20~40% cpu ,server load
rise too . except that everything is just fine , no error log ,read
and write work fine too.
here is the server info :
###################
redis_version:1.2.0
arch_bits:32
multiplexing_api:epoll
uptime_in_seconds:14147
uptime_in_days:0
connected_clients:3
connected_slaves:0
used_memory:438743977
changes_since_last_save:70
bgsave_in_progress:0
last_save_time:1263796621
bgrewriteaof_in_progress:0
total_connections_received:77103
total_commands_processed:129246302
role:master
db0:keys=505299,expires=0
#############
redis.conf:
daemonize yes
pidfile /var/run/redis.pid
port 6379
timeout 90
save 900 1
save 600 10
dbfilename dump.rdb
dir ./
loglevel notice
logfile /var/log/redis.log
databases 16
maxclients 20
glueoutputbuf no
shareobjects no
shareobjectspoolsize 1024
I store some user notifications and some backend queues . no matter it
have a slave or not , it always perform like this .
redis is really cool , and any body have idea what the problem?
Salvatore Sanfilippo
> hi,all
> I've use redis in production env 2 or 3 monthes . but recently the cpu
> usage really become odd . very time restart it , at the beginning
> it's just work fine , low cpu ,low server load ,maybe 30(or more)
> minitues later , redis-server will cost about 20~40% cpu ,server load
Hello,
is it possible that Redis is actually swapping because you are low in memory?
This seems like the most probable cause.
Don't trust the memory reported by Redis, use PS instead to see if
it's using too much memory as Redis is only able to return the amount
of memory it requested, but the allocator can use more memory because
of memory fragmentation, malloc() overhead, and so forth.
If it's not a memory related issue, but Redis is actually doing too
much work for some strange reason, like a bug in the event loop so
that it returns again and again in the same file descriptor to read
something and there is nothing to read instead, try the following:
1) start redis using gdb: "gdb ./redis-server", then at the gdb prompt
use: "run redis.conf" (with daemonize off)
2) once you start seeing the 30% CPU issue, break it with ctrl+C, gdb
will tell you what Redis is doing
3) issue a "bt" command to get the backtrace.
4) issue a "continue" command in gdb to make it continue
5) Repeat steps 2,3,4 a few times. Annotating every time the back trace.
6) Finally report here your findings :)
Thanks!
Cheers,
Salvatore
--
Salvatore 'antirez' Sanfilippo
http://invece.org
"Once you have something that grows faster than education grows,
you’re always going to get a pop culture.", Alan Kay
爵溪
loglevel to debug and find that :
...
18 Jan 20:04:14 . 0 clients connected (0 slaves), 442181987 bytes in
use, 0 shared objects
18 Jan 20:04:19 . 0 clients connected (0 slaves), 442182342 bytes in
use, 0 shared objects
18 Jan 20:04:24 . 0 clients connected (0 slaves), 442182342 bytes in
use, 0 shared objects
18 Jan 20:04:29 . 0 clients connected (0 slaves), 442182351 bytes in
use, 0 shared objects
18 Jan 20:04:34 . 0 clients connected (0 slaves), 442182861 bytes in
use, 0 shared objects
18 Jan 20:04:39 . 0 clients connected (0 slaves), 442182861 bytes in
use, 0 shared objects
18 Jan 20:04:44 . 0 clients connected (0 slaves), 442183355 bytes in
use, 0 shared objects
18 Jan 20:04:49 . 0 clients connected (0 slaves), 442183659 bytes in
use, 0 shared objects
18 Jan 20:04:54 . 0 clients connected (0 slaves), 442183659 bytes in
use, 0 shared objects
18 Jan 20:04:59 . 0 clients connected (0 slaves), 442184026 bytes in
use, 0 shared objects
18 Jan 20:05:04 . 2 clients connected (0 slaves), 442180403 bytes in
use, 0 shared objects
18 Jan 20:05:09 . 2 clients connected (0 slaves), 442180789 bytes in
use, 0 shared objects
18 Jan 20:05:14 . 2 clients connected (0 slaves), 442181049 bytes in
use, 0 shared objects
18 Jan 20:05:19 . 2 clients connected (0 slaves), 442181855 bytes in
use, 0 shared objects
18 Jan 20:05:24 . 2 clients connected (0 slaves), 442182185 bytes in
use, 0 shared objects
18 Jan 20:05:29 . 3 clients connected (0 slaves), 442182359 bytes in
use, 0 shared objects
18 Jan 20:05:34 . 2 clients connected (0 slaves), 442182214 bytes in
use, 0 shared objects
18 Jan 20:05:39 . 2 clients connected (0 slaves), 442182619 bytes in
use, 0 shared objects
18 Jan 20:05:44 . 2 clients connected (0 slaves), 442182563 bytes in
use, 0 shared objects
18 Jan 20:05:49 . 2 clients connected (0 slaves), 442182563 bytes in
use, 0 shared objects
18 Jan 20:05:54 . 2 clients connected (0 slaves), 442183173 bytes in
use, 0 shared objects
18 Jan 20:05:59 . 2 clients connected (0 slaves), 442183154 bytes in
use, 0 shared objects
18 Jan 20:06:04 . 2 clients connected (0 slaves), 442180763 bytes in
use, 0 shared objects
........
i restart the server last night round 19:00 , this morning log says
there '4 clients connected ' and not release all the time . is the
'timeout' config not working? or sth wrong with my client code ? i'm
using the php lib 'Redisent' .
On Jan 18, 5:13 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> Salvatore 'antirez' Sanfilippohttp://invece.org
Salvatore Sanfilippo
> 18 Jan 20:06:04 . 2 clients connected (0 slaves), 442180763 bytes in
> use, 0 shared objects
> ........
> i restart the server last night round 19:00 , this morning log says
> there '4 clients connected ' and not release all the time . is the
> 'timeout' config not working? or sth wrong with my client code ? i'm
> using the php lib 'Redisent' .
Hello,
are you sure this clients are actually not connected?
If this is true the event loop is actually running against this
clients without luck, but read should return an error and the clients
should be closed ASAP.
Can you tell us a bit more? Like operations this clients are
performing, version of the Linux kernel, and so forth.
I don't know the Redisent PHP lib, my guess is that the very first
thing to understand is if the client connections are or are not
"true". You can see it with "netstat -an" for instance.
If you are not sure how to use netstat to check this, please feel free
to send the output of netstat here while Redis claims there are N open
connections.
Another thing you should try is to disable "epoll" and check if this
still happens. This is how to do it:
1) make clean
2) edit config.h
3) remove the following lines:
/* test for polling API */
#ifdef __linux__
#define HAVE_EPOLL 1
#endif
4) recompile
and check if this still happens.
爵溪
process the backend queue , the web server process the cron job maybe
cause all of these , the php processing become 'zombie'
everytime ,this is the reason why connection not release . I found it
by stop the web server ,the connection released immediatly and cpu
work fine too . It should be the web server's(or my code's ) problem
but redis' .
after all, maybe the redis should have some feature like ' this
connections really live in a long time ,let's stop it!' :)
thank you very much. redis is really awesome . sorry for poor
english. :)
On Jan 19, 5:51 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> Salvatore 'antirez' Sanfilippohttp://invece.org
Salvatore Sanfilippo
> everytime ,this is the reason why connection not release . I found it
> by stop the web server ,the connection released immediatly and cpu
> work fine too . It should be the web server's(or my code's ) problem
> but redis' .
> after all, maybe the redis should have some feature like ' this
> connections really live in a long time ,let's stop it!' :)
Hello!
Happy to know you tracked the problem. Btw Redis already has such a
feature, if the clients are remaining connected I suspect this
connections are not just idle, but there is activity inside. Maybe the
clients are stuck into a loop calling Redis commands or alike.
> thank you very much. redis is really awesome . sorry for poor
> english. :)
No problem I'm also not native english and many people on this list
showed a lot of patience with my barely understandable sentences many
times ;) So this is a list where different level of english are
allowed!
Cheers,
Salvatore
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
>
>
--
Salvatore 'antirez' Sanfilippo