Understanding VanillaChronicle performance

[Ross Black]

Hi,

We are a bit puzzled about some performance numbers we are seeing when using VanillaChronicle V3.6.2 on linux.

In a simple test we write 10M messages from 10 threads (each message is ~ 2k.  The writer can writer about 200k messages/sec).  On completion of writing we run a single-thread reader, which is able to read messages at approx 10k messages/sec.
If instead we run the writer and reader concurrently, the reader is only able to read at approx 20 messages/sec, and is obviously unable to keep up with the messages from the writer.  Periodically the performance of the reader will jump back to >1k messages/sec for a very short time, and then quickly back to 20 messages/sec.

The results are not consistent across different machines, with wildly varying throughput figures.  On some machines an SSD performs worse than spinning-rust hard drive.

To me it looks like there is contention or locking on something shared by the writer and reader (OS paging?)

I would appreciate any insights on how to diagnose the issue (particularly where we should start looking, and what tools might provide some relevant info).

Thanks,
Ross

[Ross Black]

I just discovered that the issue does not exist for Chronicle V3.6.0 and earlier versions.  I assume the VanillaChronicle index/data format changed after V3.6.0, since I cannot get a 3.6.2 reader to read from 3.6.0 data.

Using V3.6.0 I can get a sustained throughput of ~200k messages/sec for both reader and writer.

[Peter Lawrey]

When you are writing at 400 MB/s you need to take careful considation of your disk subsystem. I would try writing at a much lower rate of say 40 MB/s and see how that performs on your hard ware and OS. Increase the throughput until you start to see undesirable behaviour.
Note: an SSD can write faster but possibly allow you to overload some other resource such as the paging.

The number of messages per second starts to matter around 200 byte messages or smaller. Beyond that it is the total band width which is your main concern.

Note: 400 MB/s is over 30 TB/day.

Regards, Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ross Black]

Hi Peter,

Thanks for the quick response.

When running the concurrent writer+reader test I throttled the writer back to ~500 messages/sec  (< 2Mbyte/sec showing on iotop).
The reader was still only able to read at ~200 messages/sec.

This problem only occurs with Chronicle V3.6.2.
It does not occur with V3.6.0 or earlier.

I assume a code change after V3.6.0 has introduced a performance regression?

Thanks,
Ross

[Peter Lawrey]

That sounds like an issue.
Have you tried Queue 4.x which is the one we support for free?

Regards, Peter.

[Peter Lawrey]

It would be useful to post a test which reproduces this behaviour,,if only to ensure future versions don't do this.

Regards, Peter.