Chronicle as Write Ahead Log and rotation mecanism

[Mathieu ANCELIN]

Hi guys,

I use Java Chronicle in one of my projects (it's an awesome library BTW, keep up the good work !) but i'm facing some issues.

First let me introduce some code :

https://gist.github.com/mathieuancelin/0fcacb380513733e52fe

I use Java Chronicle as a Write Ahead Log, to log some events (API requests) and replay them later asynchronously.

To do that, I use a rotation algorithm as such :

- Write events in the WAL (in a first thread)

- Every n seconds (in a separate thread), rotate the current indexed chronicle (https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-writeaheadlog-java-L57)

  - close the current appender and indexed chronicle (https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-writeaheadlog-java-L62)

  - create a new indexed chronicle and its appender (https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-writeaheadlog-java-L29) so the first thread can continue to write in that new indexed chronicle

- then I replay all the events from the rotated chronicle (the old one) to process them elsewhere (https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-writeaheadlog-java-L84)

- and I finally delete the rotated chronicle since it's no more useful (using ChronicleTools.deleteOnExit)

The events I'm storing in the IndexedChronicle are pretty straightforward, 

https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-asynapirequest-java

and the most important part of this event is the payload (which can be pretty big, up to several millions of characters)

https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-asynapirequest-java-L11

The issues I'm facing are :

- The files of the rotated Chronicles are still referenced in the system as deleted (using 'lsof -p {pid}' => "java {pid} root mem DEL ---- ---- ---- /usr/local/soft/wal/soft.index" etc ...) even if they are not seen as file anymore. It expose my system to several ulimit issues and even disk space issues (on my Mac, with Mac OS 10.8.x and a SSD drive, the usage of my WAL algorithm eats all my disk space until there is no more left).

- When the event payload is big, if I use the RandomDataOutput.writeUTF, then the content of the payload is truncated no matter what I do when I write the event in the Chronicle (https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-asynapirequest-java-L64). So I used another technique (https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-asynapirequest-java-L60) but I'm not sure about the efficiency with big blob of data. I wrote a test case for that, https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-walfailwithutftest-java

- It seems that the previous issue is linked with size of the excerpt i'm trying to write in the chronicle (https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-writeaheadlog-java-L43), but I have difficulties to understand how to compute accuratly the size of the excerpt in an efficient manner. Can someone help me with that ?

I hope I made myself clear.

If you need some clarification, don't hesitate.

Best regards

--

Mathieu

[Peter Lawrey]

A couple of notes:

- Chronicle is designed to be read as it is written so you don't need to be opening and closing files so often just to pass the data to another thread. You can read a entry in about 0.1 micro-seconds after it is written.

- You can rotate the Chronicle logs to save space, but generally disk space is cheap and I suggest doing this once a day or once a week on a production system.  I appreciate a development system might only have a few GB free space.

- Chronicle 2.1 will support automatic file rotation for every N excerpts (where N is a power of 2)

- the size of the excerpt doesn't have to be exact.  You can't exceed the capacity but might not lose much if it is more than you need e.g. 1% if you make it too big as it will shrink-wrap the size.

The practical limit to the size of an excerpt is about 32 MB without tuning and about 256 MB with tuning.  To give you some context, Chronicle 2.x has particularly optimised messages smaller than 64 bytes and 16 byte messages are 3x faster.

I can add a test for very large messages, what is the largest you are trying to send?

Regards,

    Peter.

--
You received this message because you are subscribed to the Google Groups "Java Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Peter Lawrey]

Can you create an issue for the large text for me to investigate?  BTW Chronicle 2.x attempts to cache read text fields which would be very bad if you have very large one.

On 8 October 2013 14:46, Mathieu ANCELIN <mathieu...@gmail.com> wrote:

--

[Mathieu ANCELIN]

The messages sent can weight few Kbytes to 2 or 3 Mbytes.

I've written a test here : https://gist.github.com/mathieuancelin/0fcacb380513733e52fe#file-walfailwithutftest-java

Each test case using writeUTF fail with a big message.

I will open an issue on the github project

[Peter Lawrey]

3 MB shouldn't be an issue. Will hold off the next release until I have investigated it.

[Josh Suereth]

On Tuesday, October 8, 2013 10:35:21 AM UTC-4, Peter Lawrey wrote:

A couple of notes:

- Chronicle is designed to be read as it is written so you don't need to be opening and closing files so often just to pass the data to another thread. You can read a entry in about 0.1 micro-seconds after it is written.

- You can rotate the Chronicle logs to save space, but generally disk space is cheap and I suggest doing this once a day or once a week on a production system.  I appreciate a development system might only have a few GB free space.

- Chronicle 2.1 will support automatic file rotation for every N excerpts (where N is a power of 2)

 Where might a find a 2.1 (or even 2.0) release artifact of Chronicle?  I've been looking for a good high-performance logging framework for a dev tool I work on, and Chronicle caught my eye as possible the best one out there currently.

Thanks!

- Josh

 

[Peter Lawrey]

You can find Chronicle 3.0.1 in maven central http://search.maven.org/#artifactdetails%7Cnet.openhft%7Cchronicle%7C3.0.1%7Cbundle  The source is on github.

You will want VanillaChronicle if you want rolling files. It also support concurrent writes from different processes.

It is also lock less and can be used in a GC-less way, and supports replication between servers.

--

You received this message because you are subscribed to the Google Groups "Java Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ryan Lea]

Are concurrent writes from multiple threads within a single process now supported also?

[Peter Lawrey]

IndexedChronicle is still single threaded.
VanillaChronicle supports concurrent writers across threads and processes.