Can redis scale well with 11 billion keys?
1,633 views
Skip to first unread message
darshan kamat
Oct 30, 2017, 12:54:03 PM10/30/17
to Redis DB
My requirement is to store around 11 billion keys with no expiration and data needs to be persistent.Data structure used would be only hash and sets.Is redis advisable to be used for such high key requirement.If yes what should my setup look like ?
hva...@gmail.com
Oct 30, 2017, 11:55:40 PM10/30/17
to Redis DB
You mention hashes for your data, is that because you plan to follow the model that Instagram described some years ago in this blog post?
The most significant question is whether you will have enough RAM to hold your data. In a non-sharded Redis configuration, all the keys must fit into a single Redis server process. In a sharded Redis configuration, the keys are distributed around your group of servers, so each server has only a portion of the total. Your clients can decide which keys are kept on which Redis servers, or with Redis Cluster, the Redis server processes can decide.
So the questions about this are:
Non-sharded or sharded?
If non-sharded:
If sharded:
For both types:
The answers to these questions will be the first determinants of the feasability of keeping your data in Redis.
The most significant question is whether you will have enough RAM to hold your data. In a non-sharded Redis configuration, all the keys must fit into a single Redis server process. In a sharded Redis configuration, the keys are distributed around your group of servers, so each server has only a portion of the total. Your clients can decide which keys are kept on which Redis servers, or with Redis Cluster, the Redis server processes can decide.
So the questions about this are:
Non-sharded or sharded?
If non-sharded:
What cpu and ram will you have available to the Redis server process(es)?
If sharded:
How many servers will you spread your data across?
What cpu and ram will each server have available to Redis server process(es)?
For both types:
How many bytes will the key names be?
How many bytes will the values be?
The answers to these questions will be the first determinants of the feasability of keeping your data in Redis.
AlexanderB
Oct 31, 2017, 1:30:42 PM10/31/17
to Redis DB
One other issue to consider with very large data sets in redis is that it can take a significant time to reload a server after a reload / failure of some sort. What really matters is the total dataset size, not just the number of keys, so I'd suggest loading a few thousand sample keys matching what your application will use into a redis instance, and using that to measure approximately what the data size would be if you extrapolated.
On Monday, October 30, 2017 at 9:54:03 AM UTC-7, darshan kamat wrote:
Rudy Jadeja
Apr 12, 2018, 9:25:38 AM4/12/18
to Redis DB
Did you end up using Redis for this use case? I have a similar load and am currently looking into Redis.
Joshua Scott
Apr 13, 2018, 10:18:15 AM4/13/18
to Redis DB
For 10 million keys, if the keys and values are each 8 bytes, the used memory on redis 4.0.1 is 845MB, so 1 billion would be about 84GB. Redis clusters can definitely handle this amount of data - we have clusters with around 10x that much data- but the cost of RAM becomes a factor in deciding whether you want to go that route.
Regards,
Josh Scott
Regards,
Josh Scott
Reply all
Reply to author
Forward
0 new messages