Lua memory keeps growing in steps

[Mark Martinec]

Using redis 2.8 from git, having four Lua routines with 200 lines of Lua code
altogether, fairly straightforward code with some but not much dynamic
memory usage (no cyclic structures as far as I can tell), taking about
a millisecond for each EVALSHA call, single redis instance with no slaves,
used for processing mail, one call to each routine about every few seconds.

Noticed today that the INFO command reports almost 4 MB of memory
used for Lua (and about 180 MB for everything else) after running the
application for a few days - which seemed like excessive amount of Lua
memory, considering that near the application start the Lua memory
usage reported is about 50 kB.

The real surprise came when I collected and plotted Lua memory usage
with an independently running shell command, repeated about every
second:

while true; do
redis-cli eval 'return collectgarbage("count") * 1024' 0;
sleep 1
done

Will attach the graph, hope it makes it to the mailing list, otherwise
I'll post a link. The diagram shows Lua memory usage on the y axis,
with samples taken approx every second of elapsed time (the x axis).

The plot (blue dots) is just incredibly unbelievable: the Lua memory
(above some initial offset) keeps doubling in discrete steps, each step
takes about twice the elapsed time of the previous one - see the diagram
and it will become obvious what I'm saying.

The random excursions above each baseline segment show that
the Lua automatic garbage collector works just fine, trimming the
memory usage back to about its current baseline every now an then,
frequently enough. It is the large steps in a baseline that are most weird.


To rule out any potential artifacts of a Lua automatic garbage
collector, I repeated the experiment (red dots), restarting redis-server
and the application, but this time the application calls collectgarbage()
after every mail message being processed. As expected the random
memory excursions are now trimmed very frequently, so the weird
baseline steps are now even more obvious.

The funny thing is that steps take place near seconds ?, ?, 250, 500,
1000, 2000, 4000, 8000. (I attribute the small skew to the shell
while ... sleep loop, although I may be wrong). There is nothing
time-related in the application that could explain this, as far as I
can tell. Interestingly the red dots seem to have slightly longer
time period.

Any ideas for an explanation???
(btw, I haven't tried it with 2.6)


Mark

[Josiah Carlson]

It looks to me like a leak and/or memory fragmentation, though a leak seems more likely unless you have instructed Redis to not use jemalloc.

Can you provide us with a couple more or less complete INFO outputs separated by a couple hours?

 - Josiah

  Mark

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Salvatore Sanfilippo]

Hello, this was fixed yesterday, it was a Lua memory leak introduced
in 2.8 by commits about error reporting.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Mark Martinec]

Salvatore Sanfilippo wrote:
> Hello, this was fixed yesterday, it was a Lua memory leak introduced
> in 2.8 by commits about error reporting.

That's what I get for running an old version :)
Great, thanks, looks good now!

Mark

[Salvatore Sanfilippo]

Thank you for confirming bug & fix :-)

Salvatore