latency caused by memcpy?

[Justfly Ho]

A few days ago, I found a latency issue caused by the accept system call.:

https://github.com/antirez/redis/issues/4007

Because we don't know what actually happened, we stop all conns which are not keepalived to solve that problem. 

With the tool we created to solve above problem, we found another latency issue caused by memcpy. Weird again.

The latency is larger than 2 seconds. And redis server's stack shows:

Thread 1 (Thread 0x7f30c350b720 (LWP 18548)):

#0  0x00000034e4c899b7 in memcpy () from /lib64/libc.so.6

#1  0x00000000004940db in je_arena_ralloc ()

#2  0x00000000004867d1 in je_realloc ()

#3  0x0000000000423685 in zrealloc ()

#4  0x0000000000421e07 in sdsMakeRoomFor ()

#5  0x000000000042a0d1 in readQueryFromClient ()

#6  0x0000000000418c9c in aeProcessEvents ()

#7  0x0000000000418f5b in aeMain ()

#8  0x0000000000421990 in main ()

And it seems that only a few redis instance on a same host have this problem, not all of them. 

We hope someone can help us! Thanks a lot!

[Salvatore Sanfilippo]

Hello, please could you enable the slowlog and check what is the entry
causing this? It's possible that the problem is due to setting very
very large values into Redis. A few days ago I spotted an instance
where users set 180MB values in a string, causing latency issues in
O(1) commands since when the allocations / memcpy operations are no
longer constant time factors (considering them bound to a given max),
those kind of problems start to arise. However... it could also be a
different thing, like memory pages swapped on disk, but it seems
unlikely since those are the query buffer buffers.

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.



--
Salvatore 'antirez' Sanfilippo
open source developer - Redis Labs https://redislabs.com

"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.

[Justfly Ho]

Hi Thanks for your reply!

I've tried enable the slowlog  but nothing was found  :(

The latency issue occurs before processing a command, so slowlog can not capture it.

Also I enabled the latency monitor, nothing was found .

在 2017年5月24日星期三 UTC+8下午6:52:19，Salvatore Sanfilippo写道：