LogCabin benchmarks

[A. Jesse Jiryu Davis]

Hello, I'm prototyping an enhancement to Raft, by modifying LogCabin. I want to measure my enhancement's performance. Has anyone implemented some benchmark (e.g. YCSB) against LogCabin? Thanks!

[Diego Ongaro]

Hi Jesse,

Have you seen https://github.com/logcabin/logcabin/blob/master/Examples/Benchmark.cc ? It looks very basic but is probably a good starting point.

I benchmarked LogCabin a long time ago for my dissertation ( Chapter 10 of https://github.com/ongardie/dissertation ). I don't remember the details but might have notes on the setup - let me know if that matters to you and I can dig it up. I don't remember if I used Examples/Benchmark.cc for this.

There is a "nasty-thesis-wip" tag in the LogCabin repo that included some performance hacks, and I think I used that version or close to it in my dissertation benchmarking. Not all of those changes made it into the main branch, unfortunately. https://ongardie.net/blog/logcabin-2015-02-27/ points to https://github.com/logcabin/logcabin/issues/95 , which concluded that the pipelining of AppendEntries RPCs was the only thing left to clean up and merge, and that never happened.

BTW, if you're using LogCabin for more than a prototype, I'd love to hear about it.

Hope this helps,

Diego

On Fri, Dec 6, 2024 at 7:02 PM 'A. Jesse Jiryu Davis' via raft-dev <raft...@googlegroups.com> wrote:

Hello, I'm prototyping an enhancement to Raft, by modifying LogCabin. I want to measure my enhancement's performance. Has anyone implemented some benchmark (e.g. YCSB) against LogCabin? Thanks!

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/raft-dev/336c4044-9a0d-4eb4-b2c0-25d33de7d21bn%40googlegroups.com.

[A. Jesse Jiryu Davis]

Thanks very much, Diego, I'll check out those links. I'm using LogCabin only for prototyping. I'll share the result here once the paper's published. (First it has to be accepted somewhere....) By the way, the code's nice, not too hard to modify, thanks for the effort you made when you were in grad school.

[A. Jesse Jiryu Davis]

I want to integrate the LogCabin client with a C++ implementation of YCSB. Did you give any thought to using the LogCabin client outside of LogCabin's own source tree? It seems like "scons install" builds liblogcabin.a but doesn't install it anywhere, and it also doesn't install any header files. Just making sure I didn't miss something.

[Diego Ongaro]

Hi Jesse,

Thanks for your kind words and good luck with your paper. I don't remember much about the build system. It seems like logically there ought to be a server install target and a client-dev install target, but I don't see that there. There's a brief mention on my blog where I seemed motivated to do the server part and didn't talk about the client-dev part: https://ongardie.net/blog/logcabin-2015-02-11/#:~:text=scons%20install . So I don't think you're missing anything. Scale's build system probably just reached in and grabbed those client files.

-Diego

To view this discussion visit https://groups.google.com/d/msgid/raft-dev/34b97cf8-9653-4a86-a191-f85719cd7e47n%40googlegroups.com.

[Nate Hardt]

Hi Diego and Jesse,

> Scale's build system probably just reached in and grabbed those client files

Yes, can confirm, Scale's build system does package the LogCabin client files into a logcabin-devel rpm. Since the referenced blog post, we have moved away from scons' rpmbuild and just use a simple rpm spec file to pull what we need after the build. Our logcabin-devel rpm includes:

build/liblogcabin.a
include/LogCabin/Client.h
include/LogCabin/Util.h
include/LogCabin/Debug.h
build/Protocol/Client.pb.h

We have not implemented specific LogCabin performance benchmark tests but we run automated regression tests and a large number of functional tests across a lot of different hardware setups.

Looking forward to seeing your changes!

Nate

[A. Jesse Jiryu Davis]

Hi, I'm trying to reproduce the throughput claimed in Diego's thesis and I'm coming up short. Diego used three servers (details in "Consensus: Bridging Theory and Practice" chapter 10). He "disabled log compaction" (by setting snapshotMinLogSize to a huge number?). His custom insert-only benchmark program (Benchmark.cc) with 100 threads and 1k per write got 19,500 writes per second. I'm attempting the same setup, with much beefier hardware (m7g.xlarge on EC2). The client and servers are all in the same EC2 "placement group" for minimal latency. I only get 2350 writes per second.

Diego, I can't find your "nasty-thesis-wip" branch on GitHub, but perhaps it would be useful, do you still have access to it? And any other notes you still have about the benchmark. Thanks!

[A. Jesse Jiryu Davis]

Nevermind, I found the "nasty-thesis-wip" tag after re-reading your earlier message: https://github.com/logcabin/logcabin/commit/a7ce12da. I'll try this.

[A. Jesse Jiryu Davis]

I don't get the same performance numbers with nasty-thesis-wip that you recorded in your thesis, but I got closer. I also discovered that changing the build type from DEBUG (the default) to RELEASE makes a huge difference, much bigger than I'd have expected — the build type controls not just compiler optimizations but also some very expensive assertions. Thanks for your help, I think I have a good enough baseline to proceed with my leader-lease experiment.

[Diego Ongaro]

Ouch, benchmarking with debug builds is not rewarding.

I'm not sure that m7g.xlarge is much better hardware (for LogCabin) than what I had in my dissertation (Table 10.1). I had a local SSD, whereas m7g.xlarge uses EBS. LogCabin does a lot of fdatasync calls on the log segment files, and I don't know how fast EBS is at that.

You might try the m7gd family, which is similar but has a slice of a local NVMe disk. Note that the larger instance types get allocated more guaranteed disk bandwidth: https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#gp_instance-store . Higher up on that page, you can see that the larger instance types also get more guaranteed network bandwidth. LogCabin was never optimized for fast networks or today's faster disks, but you should be able to match my dissertation numbers.

And today I learned that LogCabin runs on ARM.

I'm curious, how does the master branch (RELEASE) compare to nasty-thesis-wip tag (RELEASE) in your environment? When I created that tag, I was really hoping that no one would be using it beyond the short term.

I probably do have more notes somewhere...

-Diego

To view this discussion visit https://groups.google.com/d/msgid/raft-dev/674dafd2-d9de-4ea6-8695-ec73811dfbf9n%40googlegroups.com.

[A. Jesse Jiryu Davis]

I had to make some changes for ARM. I also ported the SCons files from Python 2 to Python 3 and made some changes for compatibility with the latest g++. I did the same atop your nasty-thesis-wip tag. I put snapshotMinLogSize=9999999 in the servers' config files to disable snapshotting. Running Benchmark.cc on the main branch got 70,767 inserts in 30 seconds, nasty-thesis-wip got 71,063 — basically the same! I'll try instances with SSDs, that's a good idea.

To view this discussion visit https://groups.google.com/d/msgid/raft-dev/CABUcDvOoSEcaGzaaa-DWnY4eE5Z2W0kLx5Rr8ejM_0gogb7W-A%40mail.gmail.com.

[A. Jesse Jiryu Davis]

Turns out I'm using SSDs already, not EBS.