Is Chronicle Queue for me?

[victo...@gmail.com]

Hi,

I just discovered Chronicle and I almost feel like it is what I am looking for, but I have some open questions.

What I am really interested in Chronicle Queue is not so much the low latency (i.e. I could accept using IndexedChronicle even with multiple reader/writers using a lock, it's not a big problem for me, and I will use serialization most certainly) and interprocess communication (for now I'm interested in exchange between writer/reader in the same JVM), but it is more for the persistence aspect of it.

What I am not clear about is the following:
 1) Are the thing stored in the queue always written to disk or is there some kind of "first keep in memory and when there is no more (configurable) space, store to disk"?
 2) If the second case, does it mean I can configure my queue to always write to disk if desired?
 3) If I understand well, the memory used is not managed by the JVM itself but by the OS (through the use of memory mapped file)?
 4) If yes, does it mean if the JVM crash, what was not yet written to disk is not lost? And will be written to disk eventually?
 5) If yes, If I start the application again, will it be able to resume reading and writing to the same queue without loss of message?

Thank you for your help!

Victor

[Peter Lawrey]

Hello Victor,

1) By default, Queue writes synchronously to the OS and lets the OS write the data when it can.  You can change this so that it will not return from finish() unless the OS says it has written the data to disk.

2) Yes.

3) The memory is managed by the OS so there is nothing for the JVM to do as such.  This memory doesn't use any heap (except a few objects to manage the native memory)

4) after finish() returns, and the JVM crashes, the data will still be written to disk.  If the JVM crashes in the middle of writing a message, the message is truncated.  If the OS crashes before data has been written, it can be lost.  If this is a concern you can use replication to mitigate this.

5) If the OS hasn't died, you will be able to continue reading and writing from where you were up date.

Regards,

   Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[victo...@gmail.com]

Thanks for the quick answer.

I have more questions to clarify what I understood:
 1) If I understood well, data stored in the queue is not removed after it has been read, right? So I should take care of it if I want to remove things… is there some configurable way of doing that automatically (like keeping N messages, or keeping for a given time…) or should it be done outside of the JVM?
 2) To be able to continue reading after a crash, I must know the index where I was, right?

And a bonus question: am I trying to twist Chronicle too much with the usage I intend to do?

For the record, it is to transfer messages inbetween services in an opensource ESB (http://petals.ow2.org): memory is often a problem when there is a huge quantity of messages that can't all be processed at a given moment.
For now we drop messages (some other ESB would just crash with an OOM exception) but that's not acceptable, so persistence to disk is the solution we are going to use, but history is not really what we are interested in (maybe that could be an interesting feature to exploit). If we start using Chronicle, maybe next step would be to use it to exchange messages between distributed nodes (Petals is distributed, for now we use some kind of simple socket).

Thank you for your help, Chroncle seems really great, it would be nice to use it :)

[Peter Lawrey]

The Queue is a persistent Queue where old entries are never removed.  The assumption is that they will be archived or deleted as whole files when you no longer need them. It has to be done outside the JVM at the moment.  The assumption is that disk space is cheap.  If you are not writing at a high rate you might find you use only a few GB per day and you can delete the files after they rotate at the end of the day. You can also do hourly rotation if you need (or faster)

To continue after a restart you need to know the index to read from.  This can mean storing the index in another file like Chronicle Map or marking the entries as you process them e.g. with a time stamp or a flag.

Chronicle Queue is pretty flexible if low level.  It is good where you have a Producer/Consumer situation.

As Queue is an open ended queue, you never need to worry about it filling up or a slow consumer impacting the producer.  You can have consumers which are behind the producer by more than main memory.

We support replication for queue however you can easily implement your own.

Regards,

   Peter.

[victo...@gmail.com]

Thank you for all the explanations! We will try to put Chronicle into use in Petals and see if it goes as we want :)

I will most certainly be back in the future for extra questions.