Remote compaction has a better performance than local

[Jiuzhi Yu]

Hi,

I am testing out the remote compaction and observed a weird performance issue. Details are:

Workload: The workload is to manually compact a dataset around 3.3GB by using db_bench. There are in total four levels of compaction, from 0 to 2, 2 to 3, 3 to 4 and 4 to 5. The maximum subcompaction number is set to 1, and maximum background compaction is set to 1 as well. We timed the function: CompactionJob::ProcessKeyValueCompaction inside the file: db/compaction/compaction.cc. We measured the I/O time and total time, and calculated CPU time by subtracting I/O time from the total time.

Settings: The remote worker is on a different process on the same machine. The compaction service communicates with the worker via shared memory. The CPU frequency is fixed for all cores (i.e. the same frequency for RocksDB and the worker).

Case 1: No compaction service registered. In this case, the compaction will be done by one background thread from RocksDB.

Case 2: We register a compaction service, which will transmit the compaction job to the remote worker. There is only one remote worker.

Result: the case 1 and case 2 have similar I/O time, but case 2 has a smaller CPU time compared to case 1 (around 10-15% smaller).

Some useful observations: When we monitored the CPU usage by top, we observed the cpu time like this:

Case 1: RocksDB: ~75 userspace, ~8 system, ~13 idle, ~4 wa (I/O)

Case 2: Worker: ~75 userspace, ~9 system, ~ 6 idle, ~ 10 wa (I/O)

We wonder what may be the case of the performance difference? According to the experiment settings, we should expect similar CPU performance for the two cases.

Thank you!

Best regards,

Jiuzhi