Storage Limitations for rqlite

[Dan Sikes]

Hi, I came across rqlite and was curios what "real world" limitations there were on it in terms of a data storage limit?

Some context around my use case, 

I have around 5B events per month from a single customer that need to be processed, indexed and stored in a way that can be queried for reporting. Each event is roughly 2kb in size is roughly 10TB. This is from a single customer (my largest) but I do have others that I am processing data for. 

Could rqlite be configured to reasonably handle 30-40 TB worth of data (i would have 90 days worth of events that would be queryable in this system)? If so, how many nodes would you suggest be in use for optimal performance.

Thanks,

[Philip O'Toole]

rqlite stores all its data in SQLite database. So the question to ask is "can SQLite store 30-40TB of data". In its default mode the SQLite database exists only in memory. Perhaps you'll need to store it on-disk.

I suspect 30-40TB is going to be too large for rqlite, however. Under certain scenarios (when a node joins the cluster for example) the entire SQLite database is transmitted from node to node. Sending multiple TBs over the network probably isn't going to work.

Finally rqlite does not scale horizontally -- in the sense that adding more nodes does not allow you to spread your dataset over more machines. Instead you add nodes for reliability and fault tolerance. There will be a complete copy of your data under every node, so the question "how many nodes would you suggest" is not relevant to your data storage requirements.

Let me know if you need more information.

Philip

--
You received this message because you are subscribed to the Google Groups "rqlite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rqlite+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/fd05fee5-0c89-4f85-93cf-d77b442d6dcan%40googlegroups.com.

[Dan Sikes]

Ok, got it. 

Thanks for the quick reply. It definitely looks like a great project. Just needing something to handle large data.

Dan