Best way to increase throughput in etcd/raft

[susanggg...@gmail.com]

I'm trying to determine if there is any tuning that can be done to increase throughput in etc/raft, possibly by running multiple instances of it or by tweaking knobs.

My understanding is that each Propose is buffered in the etcd/raft library until the user of the etcd/raft library reads from the node.Ready() channel. At that point the user has to do an fsync to disk, which will be the most expensive operation in processing the batch. Therefore, throughput will increase the larger the batch size is, because we will do fewer fsyncs per megabyte. The next batch size will be determined by how fast the user is able to process the data from Ready(). Are there any other knobs that can be used to tune the batch size to increase throughput?

Suppose one is trying to fully utilize a machine that only has one SSD. Would it ever make sense to run multiple instances of the raft library to try to parellize things? Or would that possibly reduce throughput, as the batch size would probably be smaller on each instance of the raft library, meaning that more fsync's to disk per megabyte, since both instances of the raft library will be using the same disk?

Thanks,

S

[richardgav...@gmail.com]

It's an interesting question, as I myself was trying to benchmark aspects of etcd. I actually disabled disk operations and, while latency decreased, throughput overall (as measured by number of Raft Proposals per second) did not, which was surprising to me. I tried to formulate a reason for this using Little's Law, but wasn't able to fully understand the throughput capacity of etcd's Raft. You might take a look here to get more ideas.

Rick