Chronicle Queue: am I using it wrong?

[Daniel López]

Hi there,

I just started doing some initial testing with Chronicle Queue and I'm not sure it's an appropriate use case, as I'm having some things that do not seem to fit the model.

I have a Java Agent that I use to profile some methods in a living application and I was testing if I could use a Chronicle Queue to communicate this agent with the application runnig in the same host that uses the profiling data to do some calculation and send it to the appropriate place, independently of our app.

Given that the agent can be called from various threads and I have no proper way of knowing when the application is closed from the agent, I'm creating a new chronicle/appender/excerpt on each metho being profiles, which doesn't seem to be ideal. I tried creating just one Chronicle and appender and synchronise the access, but then I get OutOfMemory errors after 20something really short messages, probably because of not closing the appender/chronicle.

On the other hand, even though creating all the objects on each call works, I get a new data file each time I store a message, so each one of my messages (timestamp+long+100max string) is consuming 64MB of disk... I could adjust that using the data block size setting... but is it "right" to have one entry per data block?

Maybe I have not found the proper way to do it, or it's not the right tool for this specific job?

Thanks,

D

[Luca Burgazzoli]

Unless you need to physically separate data from different methods (have their information stored in different files) you do not need to create multiple chronicles but you only need an instance of VanillaChronicle you can share among threads.

Inside each method you can safely create appenders on demand:

ExcerptAppender appender chronicle.createAppender();
appender.startExcerpt()
...
appender.finish();
appender.close();

This may have some overhead in term of resource utilization as Chronicle need to open/close memory mapped files multiple times so it may have some impacts on the profiling you are trying to implement. Best option is to have a limited number of threads and have an appender per thread which you close when the thread terminates.

[Peter Lawrey]

You should be able to create just one chronicle and reuse it. If you are using vanilla chronicle, it would create a file for each thread and I suggest you only use a limited number of threads if you want an efficient solution.
We recommend using a 64 bit jvm as you can rapidly run out of virtual memory on a 32 bit jvm.
Finally if you are going to create lots of threads, you really want linux as it supports sparse files. Ie you only use on disk the number of pages touched not the 64 MiB you see.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel López]

Hi,

Thanks for your help. Yes, I already moved my code so just one Chronicle is created. The appender issues is different though, as being a Java Agent external to the application means I have no control over the threads created in the application. One idea I might try is to use a background thread with a BlockingQueue that takes care of actually writing to the appender, unique, and the profile function simply adds records to that queue. 

I'll have a different process in a different JVM that reads the Chronicle and does something with the data. Once that something is done, I don't need the data anymore, does rolling the chronicle take care of that? I mean, I don't need to remove it explicitely but I don't want to have it there "forever". Does rolling "clear the queue" or do I have to explicitely remove it periodically so I don't run out of space?

Las but not least, what are the possible side-effects of not closing the Chronicle explicitely when the JVM is shutdown? Again, as the agent is external to the application I don't have a explicit notification of when the application goes down. I can add shutdown hooks or try to intercept a call to on the the methods that are called when the application is shutdown, but given that it means the JVM is going down, I'm just wondering if it's worth doing so, if some buffers might not have been flushed, for example.

Thanks again,

D.

[Peter Lawrey]

The next version of chronicle will be easier to use for those who can not control their use of threads. In the mean time you could use an ExecutorService with a small pool (possibly 1 thread)

You have to delete files externally. There isn't currently s means of deleting files for you. The assumption is you have plenty of space and a simple script to remove old files. This might not be a valid assumption and it might make sense to support this in the library.

If you shutdown in middle of writing a message, the whole message is lost. Once you finish() a message you don't need to do anything.
The close() is provided only to reduce resources for a running process eg one doing many unit tests. If your process dies it cleans them up anyway.
Note : if you use a BlockingQueue unwritten messages will be lost on a shutdown.

[Daniel López]

I see. I'm added a plain simple Runnable/Thread as I don't think I need an executor for this, for now. In my case, if the process goes down and some messages are lost, it's not really an issue as there's no need to profile a dead process :). I understand it might be an issue in some cases but not in this one, so as far as we don't tie other resources, I'm good.

And yeah, having some mechanism to "recycle" the queue would be nice. We are going to use the queue to communicate one JVM to another, so once the message is "delivered" I don't need to keep it forever, so having a max size and having the queue clean the old files when it rolls (the easiest fix I can think of) would make it simpler to use without external scripts. I see why it's not there as the original use case mandated persistence, but I think it would be nice to have it for others.

All in all, so far so good :). Thanks,

D.