rqlite latency across the internet

[Ben Standefer]

Has anybody tried running an rqlite cluster across the internet? I've read in here that because of raft and consistency guarantees (on writes?), performance degrades in high latency environment. In the past I've used Cassandra, which has high write throughput even in high latency environments.

If I have an rqlite node running on a user's laptop on US East Coast and it's doing a bunch of local writes and has a leader on the US West Coast, am I going to Have A Bad Time™? It seems like the advice it to batch large transactions, which makes sense... but is there a WAL that does compactions and allows local nodes to write at high volumes and "eventually" get the updates to the leader?

Thanks... awesome project, I love it.

–Ben Standefer

[Philip O'Toole]

Inline -- thanks.

On Sat, Nov 20, 2021 at 12:35 AM Ben Standefer <bensta...@gmail.com> wrote:

Has anybody tried running an rqlite cluster across the internet?

I have a couple of times, for testing purposes.

https://www.philipotoole.com/rqlite-v3-0-1-globally-replicating-sqlite/

https://docs.google.com/presentation/d/1Q8lQgCaODlecHa2hS-OeMGoij22oMZ_xhyeRTrQnn6o/edit#slide=id.gec8333eb5a_0_0

Performance will drop off as latency increases, the presentation above is from a talk I gave to CMU.

 

I've read in here that because of raft and consistency guarantees (on writes?), performance degrades in high latency environment. In the past I've used Cassandra, which has high write throughput even in high latency environments.

Cassandra is a very different type of database, and works very differently. That high-write throughput comes with less strong guarantees about data availability, consistency, and durability. You may lose data in certain failure modes with Cassandra, that you will not with rqlite.

 

If I have an rqlite node running on a user's laptop on US East Coast and it's doing a bunch of local writes and has a leader on the US West Coast, am I going to Have A Bad Time™?

Strictly speaking there is no such thing as "local writes". All write requests are sent to the leader for execution. Your requests to your East Coast node are transparently handled by rqlite, which forwards the write-request to the leader, waits for the Raft process to complete, and then responds to the client. Reads *can* be serviced locally however, if you choose "none" consistency.

https://github.com/rqlite/rqlite/blob/master/DOC/CONSISTENCY.md

 

It seems like the advice it to batch large transactions, which makes sense... but is there a WAL that does compactions and allows local nodes to write at high volumes and "eventually" get the updates to the leader?

No, no such WAL exists. The best thing you can do is put as many requests as possible in a single request to rqlite:

https://github.com/rqlite/rqlite/blob/master/DOC/BULK.md

There would be nothing stopping you putting some sort of WAL in front of rqlite if you wanted, gathering up a bunch of requests, and then sending them to the rqlite. This would help.

 

Thanks... awesome project, I love it.

Cheers,

Philip 

[Ben Standefer]

Thank you for the replies. I love this project, thanks for explaining.