VanillaChronicle fills up my disk

[Puiu Vlad]

Hi,

I am using Chronicle 3.6.0 on Windows and VanillaChronicle to be able to write to a chronicle from multiple threads.

However, in certain circumstances my disk fills up. The culprit seems to be the VanillaChronicle which creates a lot of files, each of size 65,536 KB, most of which contain only one record. For example, file data-4640-0 contains several records but is by no means full, and data-4640-1 through data-4640-1188 contain only one record each (I did not examine all files, but the ones I did did contain one record). 

Is this a known issue? Is there a fix available in 3.6.2?

Thanks,

Vladimir

[Peter Lawrey]

It is most likely a problem with how the code is being used. Are you sure the appender is only used by one thread?

In any case, I highly recommend using Queue v4 as it is the supported version of queue and only creates one file per day/cycle.

Regards, Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rob Austin]

Vladimir 

the latest version is 

<dependency>
    <groupId>net.openhft</groupId>
    <artifactId>chronicle-queue</artifactId>
    <version>4.5.2</version>
</dependency>

please retest with this.

Rob

[Peter Lawrey]

You should note; SingleChronicleQueue works differently to VanillaChronicleQueue. In particular; VanillaChronicleQueue made too many assumptions about how you use threads in your application which turned out to not suit many use cases. e.g. often developers had little control over the threads that are created.

SingleChronicleQueue still has a restriction that Appender(s) and Tailer(s) are single threaded. 

The problem with VanillaChronicleQueue is that it also assumed;

- The appender would only be used by one thread ever. Often people would create an appender and pass it between threads. Even with locking this didin't work with Vanilla CQ.

- The number of threads writing is controlled to a small number. Sometimes people would create a thread per message! Or would write from a third party library which created threads in an uncontrolled manner.

- The number of possible threads could be determined from the OS.  Determining the maximum number of threads wasn't as standard across favours of Linux and Windows as I though it would be.

SingleChronicleQueue lifts all these restrictions by using just one file per cycle. I suspect this doesn't help performance but it should improve usability. We are considering a new implementation of a multi-file storage for performance, but I am not sure when this will happen.

Regards,

   Peter.

[Puiu Vlad]

Hi Peter,

First, I have been working on my "hobby" project for some years now (started with Queue 2 and now I am at 3), Moving from version 2 to version 3 was relatively easy, but I am not sure how easy would be to move to version 4 now that I have considerable code written for version 3.

Second, the project I am working on is sort of a prototype and I am doing it in my spare time, that's why it takes so long. I just noticed the issue when I tried to stress test the system. It is a distributed system with many components (8 right now) running on two machines, one with 12 processors and the other one with 4. Therefore I need the remote tailer feature (1) which may no longer be supported or may be available only in the "enterprise" version.

The component that has the issue is indeed multi threaded. There are only 7 threads, not one per message, as I can see from the existing chronicle files - see (2) below: the thread id is the number in between dashes, and the second number is the max value for 'last data index' at the time the disk filled up.

My recollection of the details of version 3, backed by my code below (3), is that the Vanilla chronicle could be used to write concurrently to a chronicle from multiple threads, whereas an indexed chronicle could be used only when writing from a single thread. If that is not the case, I will need to change my code to deal with this restriction, which is probably the best option.

Any other suggestion?

Thanks,

Vladimir

(1)

_chronicle = ChronicleQueueBuilder.remoteTailer().connectAddress(host, port).readSpinCount(1).build();

_source = ChronicleQueueBuilder.source(_chronicle).bindAddress(Utils._localhost, port).build();

(2)

data-1112-399

data-1548-0

data-3640-467

data-4640-1188

data-5096-375

data-5324-0

data-6380-0

(3)

if (concurrentWrites) {

  _chronicle = ChronicleQueueBuilder.vanilla(fileName).build();

 else {

  _chronicle = ChronicleQueueBuilder.indexed(fileName).build();

[Peter Lawrey]

Hi Puiu,

   Each Appender and Tailer is single threaded.

  For Indexed Chronicle Queue, you can have multiple threads reading, however only one thread can be writing at any time. You can use locking to ensure this.

   For Vanilla Chronicle Queue, you can have multiple thread writing as well, however each thread needs it's own appender.

   As this is a pet project when can make replication in Queue v4 available for evaluation purposes.

Regards,

   Peter.

[Puiu Vlad]

Hi Peter,

That is wonderful news, twice.

First, what you said was that each VCQ needed a thread local appender. I tried that and it worked. Awesome!!!

Second, I will let you know when I am ready to upgrade. As I said before, I have limited time to look into differences between version 3 and 4 and how to re-write my code to work with 4. So, SingleChronicleQueue is more like IndexedChronicle rather than VanillaChronice, right? If I want to write to a SCQ from multiple threads I will have to manage concurrency myself, correct?

Thanks,

Vladimir

[Peter Lawrey]

SCQ should be simpler. You need to call acquireAppender () and it will give you a thread local appender. It has a number of methods for writing data such as objects, String or bytes. Ie one liners for each message.

Regards, Peter.