Estimation of maxmemory to determine RAM usage by redis

[Dbz Fan]

Starting with basic question:Is maxmemory is the maximum amount of out system RAM we are allowing redis to use?  

What are the parameters need to be considered while determining the maxmemory in redis.

Initially I just assumed that the size of my dataset should be sufficient to determine the maxmemory but it turns out that the replication and client buffers need to be considered too.Is that right.

If yes then it would mean I have to determine no. of clients and then average output buffer size of the clients.I think it would be challenging  to determine the average output buffer size.
 When we say client what all entities are included here.As per my knowledge the client includes the users of redis(jedis in my case), the slaves and the sentinels.

Apart from the above stated please let me know if there are other parameters I need to consider for determining the maxmemory.

[Josiah Carlson]

Replies inline.

On Sun, Apr 5, 2015 at 12:18 AM, Dbz Fan <manass...@gmail.com> wrote:

Starting with basic question:Is maxmemory is the maximum amount of out system RAM we are allowing redis to use?  

Not quite. It is the amount of memory that the memory allocator has returned to us, rounded up to the blocksize that was actually allocated (your 63 byte allocation almost always is really 64 under the hood). But it's pretty much the same thing, yeah. Modulo some small overage for stackframes, temporary allocations, etc.

 

What are the parameters need to be considered while determining the maxmemory in redis.

Too many to list. Pick reasonable numbers for everything (buffers, AOF vs. RDB, etc.), get a reasonable machine/vm, then put like 5% load on it if that is possible. Then 10%, and figure out what you really need. Then buy/rent a large enough machine that you expect to grow, or be able to use failover strategies to upgrade as necessary. If your workload is amenable to sharding, consider testing Redis Cluster, Twemproxy/Nutcracker, or even the Redis Labs on-premesis solution. I've been meaning to test the latter for my organization, but I've been too busy.

Initially I just assumed that the size of my dataset should be sufficient to determine the maxmemory but it turns out that the replication and client buffers need to be considered too.Is that right.

Yep, that is right.

 

If yes then it would mean I have to determine no. of clients and then average output buffer size of the clients.I think it would be challenging  to determine the average output buffer size.
 When we say client what all entities are included here.As per my knowledge the client includes the users of redis(jedis in my case), the slaves and the sentinels.

Unless you are maxing out your network, buffers are only really about the size of a single request from a client. If you're pulling down 1M on 10k clients, well that's 10 gigs. Not tiny, but not that crazy - I know where you can rent a single 6TB box. But if your requests are closer to 100k, well then that's just 1G, which costs like $20/month to rent.

 

Apart from the above stated please let me know if there are other parameters I need to consider for determining the maxmemory.

Start testing small, make reasonable projections. That's what I do.

 - Josiah

-- 

You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Dbz Fan]

Also I was thinking of used_memory (from info command) to estimate RAM. For example : I could check the increase in used_memory when I insert a certain number of keys and determine RAM from that. So if used_memory includes memory related to the data size and all buffers we can straight a way go for it.

[Josiah Carlson]

Yes, used_memory does include buffer sizes, but if you want those buffer sizes to be representative of your actual load, then you'd have to be using connections to Redis in a way that is representative of your actual load. If you're only ever sending one or two commands at a time per client, and those responses are typically small, maybe you don't need to worry about that part. But if you've got slaves, then you may want to budget some space for replication backlog for short-duration disconnections, based on your write throughput.

 - Josiah

[Dbz Fan]

What about memory fragementation

[Josiah Carlson]

Watch mem_fragmentation_ratio output from INFO. Different workloads will have different expected fragmentation ratios, which are difficult to predict in advance. Unless Redis is swapped out to disk by the OS, many workloads tend to see fragmentation ratios in the 1.1-1.4 range. If you've got more than 10 megs of data in Redis, and you see the ratio fall below 1, you are swapped out. If you are seeing fragmentation ratios of over 1.5, it's commonly due to cyclic "add a bunch of data, work on it, delete all of it" operations, and may be fixable.

 - Josiah

[Dbz Fan]

Ok currently my fragmentation ratio is around 1.15 -1 .27. 
How much memory does one need to take into account for memory fragmentation, as that too would be added in estimating RAM right?

[Josiah Carlson]

Yes and no. mem_fragmentation_ratio is exactly used_memory_rss / used_memory . The used_memory_rss value is what the OS says is currently in RAM for the process (which is why swapped out Redis results in low fragmentation ratios). The used_memory value is what Redis has accounted for during its runtime. If you want to take fragmentation into your estimates, say 1.5 * used_memory for expected worst-case behavior, and maybe 1.75 * used_memory for insurance if you want to be paranoid.

 - Josiah

[Dbz Fan]

 So  while estimating maxmemory should we add up the replication backlog size too?

 If yes then why do we have the seperate "repl-backlog-size" property in redis.conf for replication backlog?

If no then estimating RAM would be addition of maxmemory and replication backlog size (maxmemory + repl-backlog-size) ?

[Josiah Carlson]

repl-backlog-size represents the maximum size of the replication backlog. There is just 1 backlog, which is shared among all slaves.

If your slaves are slow or temporarily disconnected, the backlog will grow, and (if I recall the source correctly) will be reflected in  used_memory, which is used to determine whether the system has hit the limit specified in the 'maxmemory' setting.

 - Josiah