Some way to reduce Redis fragmentation ratio ? or replace Redis ?

[Tu Pham Phuong]

Hi all, 

  My main project use redis for quick storage log , i just provide 1 api to put log to redis list, hundred call per second, millions call per day, log message of same day store in same list, after one day, i have pop all element of last day to storage log in other persisten database. Because a lot of operation like pop, del, my Redis fragmentation ratio > 4.0, very high. I need log and release resource very fast to reuse but have limited ressource.

And i just think, not tested, some way to issue my problem is:

 1 - Write shell with this trick to reset redis  fragmentation ratio http://stackoverflow.com/questions/9535790/reset-redis-used-memory-peak-stat

 2 - Use memcached to storage list of logs, use key name like: a1,a2,a3, ...an and 2 key storage biggest and smallest key, it may replace redis?

 3 - Use queue management like RabbitMQ

 

Really need a good advise, thanks you.

[Josiah Carlson]

If you have the memory to store all of your data in Redis for a day, why do you care that your fragmentation ratio is high? The fragmentation ratio just says "you used to use a lot of memory, you aren't using as much now, but freeing it won't help". The resetting of fragmentation ratio from stackoverflow doesn't actually do anything productive, except for make you feel better (it doesn't change the actual memory use of the process), so isn't useful.

As for using memcached, memcached also doesn't release memory back to the OS (it is actually worse than Redis in this regard due to its arena allocation), so this won't help you.

RabbitMQ might reduce memory usage, as data it receives is written to disk, but you may have issues when you are reading that data later (it might not be as fast as you need it to be).

My advice: empty your queue as you are filling it rather than waiting until the end of the day to empty it. That will keep your peak memory use low, it will keep your fragmentation ratio low, and it will help reduce the peak load on your persistent database (whatever it is). You can run a simple daemon or cron job that clears it out every hour, writing to your persistent database. That should reduce your peak memory use significantly and avoid high fragmentation ratios.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Matt Palmer]

On Sat, Oct 05, 2013 at 10:13:57AM -0700, Tu Pham Phuong wrote:
> My main project use redis for quick storage log , i just provide 1 api to
> put log to redis list, hundred call per second, millions call per day, log
> message of same day store in same list, after one day, i have pop all
> element of last day to storage log in other persisten database. Because a

Stop doing that -- process jobs as they arrive. Presumably you won't be
able to keep up with the rate of jobs as they come in, but you'll
significantly reduce the "peak" memory consumption in Redis, and thus reduce
your fragmentation ratio.

- Matt