redis 100% cpu usage

[Birdy]

Hi,

We are seeing 100% cpu usage by redis-server process. It was up for atleast 60-90 mins. Then we restarted it. During this time output of info command returned in 20-40 seconds.

We are not using slow commands like keys etc. Also only 20 connections were present all the time.

The poor man's profiler returned following:

[root@CSS-FU-2 bin]# gdb -ex "set pagination 0" -ex "thread apply all bt" \

>   --batch -p 15742

[Thread debugging using libthread_db enabled]

[New Thread 0x42d91940 (LWP 15745)]

[New Thread 0x42590940 (LWP 15744)]

0x0000003d2080d96b in write () from /lib64/libpthread.so.0

Thread 3 (Thread 0x42590940 (LWP 15744)):

#0  0x0000003d2080af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000000043142b in bioProcessBackgroundJobs (arg=<value optimized out>) at bio.c:119

#2  0x0000003d2080677d in start_thread () from /lib64/libpthread.so.0

#3  0x0000003d1f8d30dd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x42d91940 (LWP 15745)):

#0  0x0000003d2080af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000000043142b in bioProcessBackgroundJobs (arg=<value optimized out>) at bio.c:119

#2  0x0000003d2080677d in start_thread () from /lib64/libpthread.so.0

#3  0x0000003d1f8d30dd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f26e9ad6700 (LWP 15742)):

#0  0x0000003d2080d96b in write () from /lib64/libpthread.so.0

#1  0x00000000004180ed in sendReplyToClient (el=<value optimized out>, fd=45, privdata=<value optimized out>, mask=-1) at networking.c:589

#2  0x000000000040bdea in aeProcessEvents (eventLoop=0x7f26e8c47000, flags=3) at ae.c:346

#3  0x000000000040c04e in aeMain (eventLoop=0x7f26e8c47000) at ae.c:387

#4  0x0000000000410d8d in main (argc=<value optimized out>, argv=0x7fff66068238) at redis.c:1680

Can you please check it out?

Thanks,

Anurag Berdia

[Marty Weiner]

Can you post the contents of:
redis-cli info

Often 100% usage points to out of available fd's.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/1kkUf-eAwisJ.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Birdy]

Unfortunately we do not have info logs of that time.

But the memory usage was around 3.8 GB. Our maxmemory limit is 4GB.

Also the kernel cpu was more than the user cpu(Almost double).

Hope this helps.

Thanks,

Anurag Berdia

[Birdy]

Also another of our process running, used file descriptors heavily (10-12 K). Can this impact redis?

At the same time the system wide setting for file descriptors was default(1024).

Is it a know thing that if redis is short of fds, it will go to 100% cpu usage. Can you describe this point?

Thanks in Advance.

Anurag Berdia

[CharSyam]

Excuse me.

At that time, Did you use rdb with heavy inserts?

one of my friends experienced something like that.

in his case. He inserted big amount of items.

and he used rdb. in such case. Redis forks to create child process to save RDB.

that time, redis can use memory twice and OS will start swapping.

Thank you.

2012/8/9 Birdy <anurag...@gmail.com>

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/mtMwCFIhmcEJ.

[Birdy]

Hi CharSyam,

We are not using any dumping (rdb or async etc) as such. But we are using replication.

As i have seen it, it creates the rdb file when it first time sync up with slave. 

So at that time what you are suggesting might be possible i guess.

PS: We are also inserting big amount of data in redis. (Around 1.4 GB in 1 min)

Thanks,

Anurag Berdia

[CharSyam]

How big size of memory do you use?

I just suggest adding extra memory.

If you use 4G memory, I suggest more than 8G mem.

Good Luck.

2012/8/14 Birdy <anurag...@gmail.com>

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/YAhcwBBPL6oJ.