fragmentation observations

[Jason Priebe]

I was reading an earlier thread about fragmentation and the possible
need to stop and restart redis to defragment its memory space.

I've been running an experiment for a week or so where I've been doing
about 50 redis set() calls per second nonstop. I'm reading a fairly
large data structure from redis, updating it, and putting it back (I'm
using php serialize/gzip/base64 encoding for the objects, so I'm
pretty sure that the object sizes change almost every time they are
written back to redis, even with small changes to the data structure).

I would have expected a lot of fragmentation, since redis may have
allocated a slot to hold the object, but then the change in size would
require a new allocation for the larger string.

But that doesn't seem to be the case so far. Here's what I have in
redis right now:

290696 objects
average of 820 bytes/object
total bytes: 238625479
bytes used by redis: 290717303

When I stopped and started redis, the memory usage did not change. If
I'm understanding this correctly, this suggests that there was little
to no fragmentation at all (unless the fragmentation is preserved when
the database is loaded from disk, which would surprise me).

It looks like there's about 20% overhead for the data storage, which
seems to be consistent with previous posts in this group. Not too
bad.

Speed has been very good. While I'm doing about 100 commands/s, the
CPU is at about 99% idle (granted, this is a 16-core machine with 16GB
of RAM). When it is writing the database to disk, it drops to about
92% idle.

This is really great stuff, Salvatore!

[Salvatore Sanfilippo]

On Tue, Apr 21, 2009 at 11:16 PM, Jason Priebe <jason....@gmail.com> wrote:
>
> I was reading an earlier thread about fragmentation and the possible
> need to stop and restart redis to defragment its memory space.
>
> I've been running an experiment for a week or so where I've been doing
> about 50 redis set() calls per second nonstop.  I'm reading a fairly
> large data structure from redis, updating it, and putting it back (I'm
> using php serialize/gzip/base64 encoding for the objects, so I'm
> pretty sure that the object sizes change almost every time they are
> written back to redis, even with small changes to the data structure).

That's a very interesting experiment, with an access pattern very
similar to real world applications that can run using Redis.

> I would have expected a lot of fragmentation, since redis may have
> allocated a slot to hold the object, but then the change in size would
> require a new allocation for the larger string.
>
> But that doesn't seem to be the case so far.  Here's what I have in
> redis right now:
>
> 290696 objects
> average of 820 bytes/object
> total bytes: 238625479
> bytes used by redis: 290717303

That's great. The kind of fragmentation we can't see it is page
locality fragmentation. Still I'm pretty sure an optimized malloc can
do better than a trivial slab allocator about this issue.

> When I stopped and started redis, the memory usage did not change.  If
> I'm understanding this correctly, this suggests that there was little
> to no fragmentation at all (unless the fragmentation is preserved when
> the database is loaded from disk, which would surprise me).

When the DB is stopped and started the objects are created and there
are almost only malloc()-s, so it should really defragment everything

.

> It looks like there's about 20% overhead for the data storage, which
> seems to be consistent with previous posts in this group.  Not too
> bad.
>
> Speed has been very good.  While I'm doing about 100 commands/s, the
> CPU is at about 99% idle (granted, this is a 16-core machine with 16GB
> of RAM).  When it is writing the database to disk, it drops to about
> 92% idle.

Nice values. Redis 0.091 may take a bit more CPU on saving since it
tries to compress data btw.

> This is really great stuff, Salvatore!

Thanks and thank you very much for sharing this data with us.
I'm even more convinced that it's not wise to fix a problem we
currently don't have. Memory fragmentation basically can't be fixed,
it's a problem in "nature" basically: the more you know about the
allocation patterns and objects life cycle the more you can write a
vertical allocator to mitigate the problem. But then possibly you
fragment less but have less cache locality so it is important to write
a *good* allocator suited for the usage pattern you have, and the
current malloc() implementations are very smart compared to mallocs of
years ago. For example glibc() malloc is one of the most advanced
(takes different pools of memory for different threads to avoid
locking and so on).

After all apart Redis objects that Redis is readly caching on a free
list, all the other allocations are very hard to guess: most is about
the application using Redis. For now what we can do is to improve
caching of trivially reusable objects. For instance client structures.
Once we start to experiment memory fragmentation problems there will
be to check what the allocation pattern leading to this problems are
and if a change to the code is fixing or not the issue. Basically
there is the need to see the problem "working" to find the best fix,
if this will be needed at all :)

Regards,
Salvatore

> >
>



--
Salvatore 'antirez' Sanfilippo
http://invece.org