Chronicle Performance Drop

[Pascal]

Hi,

I am doing some research for a high throughput, low latency system, and I thought Chronicle could be a nice piece to integrate for the logging.
I have been running some performance tests for myself to assess how usable Chronicle would be.

The test is a simple write to Chronicle, the purpose is to see the throughput. Messages are 1kB big Strings (it's pretty fat, but it's a requirement of my overall system).
I would like to see how long it takes to write 1M, 3M, 5M of theses messages (so respectively 1GB, 3GB and 5GB worth of data). I have attached an extract of the code I ran.

The system I ran the test on has 48GB of RAM (20 of which are allocated to the JVM) and 24CPUs. It however does not have SSD, but this should not be a concern for my problem.

The problem is that according to the results, there is a drop in performance midway:

Msg: 1024 B		Throughput@Iteration
Nb Messages	Avg Throughput [MB/s]	1	2	3	4	5	6	7
1'000'000	273.2857143	245	279	274	278	281	279	277
3'000'000	168.7142857	252	281	280	85	79	96	108
5'000'000	150.7142857	257	250	88	82	115	133	130

At each iteration, [1, 3, 5]M messages are written into Chronicle. I do delete the file created by Chronicle (the .cq4) at the end of each iteration and proceed with Garbage Collection to make each iteration as independent as possible.

The performance drop usually occurs after 10M messages have been written (~10GB). To be convinced, I ran multiple 30 iterations tests, each iteration writing 1M messages (= 1GB of data). Here is the result of one of the run:

30 iterations run	Throughput [MB/s]
Iteration 1-5	243	265	260	258	259
Iteration 6-10	260	262	262	260	259
Iteration 11-15	185	103	90	92	93
Iteration 16-20	95	100	102	106	109
Iteration 21-25	114	124	125	128	133
Iteration 26-30	136	142	148	148	148

Similarly, after ~10M messages are written, the throughput starts going down.

I do suspect the Operating System to be the cause of it, due to swapping, but what I cannot understand is why the performance goes back to ~145MB/s rather than the ~250MB/s we have at the beginning.

Could anyone help me with some explanation?

Thanks in advance !

Best,

Pascal

[Peter Lawrey]

By default, your OS allows 10% of main memory to be uncommitted to disk. The OS will be writing to disk while you are writing so the point at which you hit a slow down depends on how much faster you are writing to the queue than your disk subsystem can handle.
There is an issue that even if you close and delete the files they will still be written to disk. This should be fixed in the next couple of weeks.
You can increase the dirty threshold however you are better off having a disk subsystem which is both faster enough and large enough for you sustained throughput.
My guess is your disks can sustain ~80 MB/s which is still over 6 TB per day. You should be able to achieve bursts of higher than 280 MB/s. I suspect the string encoding is hurting you. I have achieved write rates of 900 MB/s to an SSD

I can look at the code tomorrow.

Regards, Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pascal]

Hey Peter,

Thanks for you swift reply.

Unfortunately, I do not have any SSD yet... I doubt SSD would increase the throughput from ~200MB/s to ~900MB/s anyway.
I tried to write Bytes<> instead (I found some snippets from your own tests on GitHub) of String and it indeed gave a better throughput:

30 iterations run

Throughput [MB/s]

Iteration 1-5	301	300	318	309	314
Iteration 6-10	296	300	302	297	314
Iteration 11-15	214	245	189	172	158
Iteration 16-20	153	146	145	144	143
Iteration 21-25	145	146	146	146	147
Iteration 26-30	147	146	146	145	144

This is on average 40 MB/s faster, which is already a good start :). Still it is nothing near your 900MB/s. Would you mind sharing the code or pointing me a reference to such a test?


Thank you for your work and help, it is highly appreciated !

Best,

p

[Peter Lawrey]

That is a good question.  It has been a while since I ran the test, so let me dust it off and run it again. Note: this will just shorten the time to fill the disk cache and once you hit the performance of your disk subsystem, that will be your bottleneck.

If I was to buy a new one today, I would consider one of these http://www.intel.co.uk/content/www/uk/en/solid-state-drives/intel-ssd-dc-family-for-pcie.html they support up to 3,000 MB/s write speed, in which case the capacity will be the limiting factor.

Regards,

Peter.