Store huge in memory data

[kranthi kumar]

Hi,

I am new to Redis DB. Please help me to understand better. Let me walk you through my use case, explain does this DB fit into it or not?
In a network device, would like to calculate statistics for each flow. My device supports 10 million flows and need to store 25 statistics for a single flow and each flow is identified by a key of 6 tuples. This device need to store these statistics for T (5 minutes) interval of time in memory and forward to third party DB storage. The in-memory for a flow get update for each packet received on that flow. Can I use Redis DB for this? If yes, What are the existed customers that are using this for the same purpose? 

[Dvir Volk]

Sounds like a very fitting use case for redis. Without too much calculations, I would assume this takes something in the order of magnitude of 10GBs of RAM, which very reasonable even for a single redis instance. 

If a single instance is too costly, you can use a redis cluster and distribute your keys over many instances with sharding. 

I'm not sure who is using redis *exactly* for this, but real time aggregative stats are a very common use case for redis. 

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[hva...@gmail.com]

Have you evaluated the rate at which data would be stored into Redis (and removed from Redis as the statistics are forwarded to the third-party storage)?  It's good if the amount of data can fit into a reasonable amount of memory, but you also need to evaluate the rate bytes are being written.  I.e., since each flow is updated per packet, what's the expected rate of updates?  How many bytes in each update?  If this rate of bytes written into Redis is not excessive, then it can probably work for you.

[kranthi kumar]

Suppose for 1 million flows, packet update receives for each 1 sec and need to update 15 stats each of size 4 bytes? Then, how much memory does Redis takes to store this data for 1 minute? (10^6 flows * 60 bytes * 60 sec = 3600 MB). Does Redis takes any additional memory?

[Simone Mosciatti]

Of course you have some overhead, but it is at most linear.

The best thing you can do is simply to try and test it.

If it does't works you may come back and ask for more help :)