Redis 2.6.2 with jemalloc mem fragmentation between 2 and 3

[sb56637]

Hi everyone,

I'm using Redis 2.6.2 with Drupal 6 for caching. I'm NOT an expert on caching and I am totally ignorant about exactly how the Redis module for Drupal actually stores and retrieves cached data from Redis, so please excuse my ignorance. I'm using Drupal on a Webfaction.com CentOS 6 server (not dedicated), so I needed to reduce the number of mySQL queries, and the Redis cache has saved my site. So thank you very much for Redis.

I'm concerned about the memory fragmentation ratio between 2 and 3 after a few days of uptime. Since it's not a dedicated server, I don't have loads of RAM, so I have maxmemory set to 50M, and in reality I don't need more than that because my used memory peak is only about 20M. So the fragmentation is not exactly devastating for me given the small dataset in question. But at any rate I thought it would be helpful for the Redis devs to know that even with jemalloc, there are apparently certain usage cases that produce rather high memory fragmentation. And if I can reduce the memory usage a bit more, I can probably configure Drupal to cache a few additional big tables on Redis.

Here's my INFO ALL:

http://pastebin.com/PgHegfup

And my redis.conf, it's all pretty standard stuff. Probably the only non-standard option is "maxmemory-policy allkeys-random".

http://pastebin.com/Y08AmdU7

Is there anything I can do to reduce fragmentation, either with config options or possibly compile options?

Thanks very much!
Have a great day.

[Didier Spezia]

Hi,

there is a good chance the bad fragmentation ratio is not caused by actual fragmentation,

but rather by the fact some keys are deleted while the allocator cannot give back the

memory to the system.

used_memory:6423832

used_memory_rss:15855616

used_memory_peak:21095608

Peak memory is higher than rss , so it means keys have been deleted, and at a point, more memory

was used. Actually more than 50K del operations have been done:

# Commandstats

cmdstat_del:calls=50595,usec=5774517,usec_per_call=114.13

The fragmentation ratio is just calculated as rss/used, so when rss memory is still high

because the allocator has not given unused memory back to the system, the fragmentation

ratio appears bad.

To understand the difference between internal and external fragmentation, you can have

a look at: http://en.wikipedia.org/wiki/Fragmentation_(computing)

To evaluate the internal fragmentation of the allocator, just save, restart the Redis instance,

and have a look at the fragmentation ratio with INFO.

External fragmentation can only be evaluated if the global size of your data is stable or grows.

If the volume of data is decreasing, then the ratio given by INFO is not really meaningful IMO.

With jemalloc, there is no way to force the allocator to give memory back to the system.

If you really want to optimize this, you can try tcmalloc (Redis compile time option) with

your workload and check if you can improve this behavior by tweaking the tcmalloc configuration

(TCMALLOC_RELEASE_RATE)

Regards,

Didier.

[sb56637]

Hi Didier,

Thanks a lot for the informative response. You're right, I guess I should have phrased this issue as "Redis not giving memory back to OS after deletions".

The behavior isn't what I expected after a "shutdown save" and a restart and reload of the dump file. Now the RSS is still much higher than the amount of data stored. Here's my new "info all" about 1 minute after the restart (it was approximately the same immediately after the restart):

http://pastebin.com/psQXRTg0

If I were to try it with tcmalloc, do you have any suggestions for a TCMALLOC_RELEASE_RATE for starters? The website is a high-traffic forum, so that means frequent additions and modifications (which I imagine would translate into deletions for Redis).

Thanks again for the kind help,

Cheers.

Sb

[M. Edward (Ed) Borasky]

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/G18tYuVwDUMJ.
>
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

On Linux, it's possible for a process like Redis to see a more
detailed view of its memory usage at the OS level than just looking at
RSS values. The 'pmap' command formats the information in
human-readable form, but the raw data could be read by a Redis process
from the /proc/self directory.


--
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?

[Salvatore Sanfilippo]

Hi Edward! I'm not sure how this is interesting from the point of view
of fragmentation, as in this particular case the only issue is that
the instance contained more data than currently. But, what you point
out, is very interesting and I used this multiple times in debugging
with success.

For instance one thing that we could do with the /proc info you
propose, is to print the amount of memory that was subject to copy on
write after the child process exited.

Cheers,
Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[M. Edward (Ed) Borasky]

On Thu, Nov 8, 2012 at 12:05 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> Hi Edward! I'm not sure how this is interesting from the point of view
> of fragmentation, as in this particular case the only issue is that
> the instance contained more data than currently. But, what you point
> out, is very interesting and I used this multiple times in debugging
> with success.
>
> For instance one thing that we could do with the /proc info you
> propose, is to print the amount of memory that was subject to copy on
> write after the child process exited.
>
> Cheers,
> Salvatore

I was thinking of something like "slowlog" - snapshots of the writable
segment sizes at a config-file specificed interval with a
config-file-specified "circular buffer length" and a command to read
the values out and reset them. I'm doing it externally with "pmap -d"
but I haven't built a live application yet where I'd want to use it.
And I have no idea what the equivalent techniques are for BSD or MacOS
X or Solaris.

Speaking of BSD, I've been seriously considering putting up a BSD
partition on my workstation. Are there a lot of Redis-on-BSD users in
the real world I can lean on for mentoring?

[sb56637]

Interesting update:

After 12 days of uptime with relatively constant, fairly heavy traffic, used_memory_human is 8.91M, and fragmentation is hovering around 1. In fact it occasionally dips below 1. Not exactly sure how that can happen, apparently Redis compresses data or something? So I'm not sure why fragmentation has improved so much. Maybe my data set is more stable and jmalloc has been slowly working on returning memory to the OS?

[Salvatore Sanfilippo]

Once you fill your instance again with data, fragmentation starts to
be meaningfull again basically (it is not meaningful when the peak of
used memory is far from the currently used memory).

As for fragmentation < 1, the OS swapped out a few memory pages of
data you don't use apparently, or alternatively you have blank pages
in your dataset that the OS can avoid allocating.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/redis-db/-/2dDKRxVoSXAJ.