ReadOnly / Non-Voters adding write latency according to my tests

[Gerrit]

Hi Philip,

I have done some research in order to check the performance of my software.
I did some performance Tests on different cluster-sizes with my software starting with just one node. I did this tests with at maximum 20 voters and repeated it with 20 nodes where there where at most 9 voters with the exess nodes becoming non-voters / read-only-nodes.

According to READ_ONLY_NODES.md you said "if you need a high volume of reads, without adding write latency to the cluster".

As you can see in the image above, there was no difference in time behavior using all nodes as voters and capped the voter size to 9 nodes.

Some information on how i did the test:
Before and after each write request the current system time is read using the highresolution clock in C++. This will be done a thousand times for each cluster-size.
Afterwards the avarage will be calculated displaying the times you see in the image above. For example one write request with a cluster of two nodes takes about 1.5ms.

Test environment:
- Running in docker on one host machine (therefore network delay wont be a thing)
- did this with rqlite 5.5.1 and the latest release 5.10.2
- double checked that non voters are actually started with -raft-non-voter=true

- times are always calculated on the leader node

I expected that the times for the restricted voter-size-cluster will no longer increase. Instead, the cluster with non-voters behaves the same as the cluster with voters.

Might this be a bug or did i missunderstood the sentence "without adding write latency"?

Best regards,

Gerrit

[Philip O'Toole]

Hi - thanks for your detailed report. You may be right as I have not run the same level of testing that you have done.

My statement about "reduced write latency" comes from the fact that with non-voting nodes, the leader will not wait for those nodes to vote after a change (obviously). I next *assume* that the Hashicorp code will reply to the client (in this case your test code) once the change has been voted on. From your testing it looks like this *is not actually the case*. 

I would next need to examine the Hashicorp code. What your findings imply is that my assumption is wrong and that while the Hashicorp code doesn't wait for a vote from non-voting nodes, it is waiting for an acknowledgement from those non-voting nodes.

A bug is always possible, of course, but I have end-to-end testing in place that ensures those votes truly are not voting. You can check this yourself, by stopping the non-voting nodes but seeing that changes can still be made to the cluster.

Thanks again -- it looks like a statement I make in the docs may be wrong. Non-voting nodes may increase read-scalability within incurring the cost of voting. But write-latencies may actually increase. I will need to examine the Hashicorp code to confirm.

Philip

--
You received this message because you are subscribed to the Google Groups "rqlite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rqlite+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/d3928987-05a0-4473-baef-ef4089ecc562n%40googlegroups.com.

[Gerrit]

Hi,

Thanks for your reply.

Indeed it seems like the leader is also waiting for an acknowledgement from read-only nodes.

I can confirm that those nodes are truly not voting.

Please let me know if you have an new insights or if I can help you by any matters.

Gerrit

[Philip O'Toole]

I suspect my assumption is wrong, but I'll look into the code. I have updated the docs:

https://github.com/rqlite/rqlite/commit/f4956b983aaf4be90f625f4f633c17fd6424f79e

Thanks again for flagging this. I hope it doesn't prevent you from using rqlite.

To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/a2b874a6-a893-47dd-9da7-b97056537204n%40googlegroups.com.

[Philip O'Toole]

>>Non-voting nodes may increase read-scalability within incurring the cost of voting.

Typo: *without* incurring the cost of voting.

[Gerrit]

I will repeat this test on monday but then with real hardware and normal network. I am still curious if this makes a difference, since there is no network delay in a docker-net on one host machine.

I will share the results as soon as possible.

Either way, we will continue to use rqlite. With our control tool, we can also simply leave those read-only nodes offline and add them to the cluster if other nodes fail. I guess our concept will still work this way.

Thanks again for your fast replys and this awesome software!

Gerrit