Chronicle Logs creates multiple files per thread.

[Deepesh Mehta]

Hi,

We have been using Vanilla Text CQ Logs for our application logging. Our application is multithreaded but CQLog instance is Singleton.  Overall performance of CQ Logs is extremely good but what we have observed is for each thread multiple data files are created.

We use default size for data (64MB) and Index (16MB) files. We run our application with 20 threads and I would have hoped only 20 data files and 1 index file but instead multiple data files are created per thread (each with 64MB). Most of the time each data file do not carry much logging details but on file system each file occupies 64 MB of data.

When ran in debug mode, I could see that creation of file is completely dependent on the appender used and ideally for 1 thread same appender should be used.

I think storing the VanillaAppender as a Weak Reference in appenderCache is causing multiple appenders to be created per thread which then ends up creating multiple data files per thread .

Does these appenders need to be stored as Weak References?

@NotNull

@Override

public VanillaAppender createAppender() throws IOException {

WeakReference<VanillaAppender> ref = appenderCache.get();

VanillaAppender appender = null;

if (ref != null) {

appender = ref.get();

if (appender != null && appender.unmapped()) {

appender = null;

}

}

if (appender == null) {

appender = createAppender0();

appenderCache.set(new WeakReference<>(appender));

}

return appender;

}

Thanks & Regards

Deepesh Mehta

[Luca Burgazzoli]

Which chronicle-logger implementation are you talking about? slf4j, log4j, logback

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
lb

[Deepesh Mehta]

Hi Luca,

We are not using any logger implementations. Previously we did use log4j as well as slf4j and noticed same behaviour.

Currently we directly use TextChronicleLogWriter to write logs in Chronicle data files.

Please see snippet of code below for logger initialisation.

ChronicleQueueBuilder queueBuilder = ChronicleQueueBuilder.vanilla(_path).indexBlockSize(INDEX_SIZE).dataBlockSize(DATA_SIZE);

_chronicle = queueBuilder.build();

ChronicleLogWriter _log =ChronicleLogWriters.text(_chronicle, ChronicleLog.DEFAULT_DATE_FORMAT, 3);

Thanks & Regards

Deepesh Mehta

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
lb

[Luca Burgazzoli]

And you shares the ChronicleLogWriter instance among threads ?

Can you also list the files you have in the data directory ?

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
lb

--

You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

[Deepesh Mehta]

Hi Luca,

Yes, instance of ChronicleLogWriter is a singleton which is shared among multiple threads.  Below are the data file names when ran with 10 threads.

As you can see 17 data files got created (some threads creating more than 1 file).

When debugged this through I found that at times appenderCache lost references to the old appender Object and hence for same thread new appender was created. Snippet of the code is present in the original post.

1) data-9164-1

2) data-9164-0

3) data-8824-1

4) data-8824-0

5) data-8816-0

6) data-8464-1

7) data-8464-0

8) data-6804-1

9) data-6804-0

10) data-5112-1

11) data-5112-0

12) data-5020-0

13) data-4476-1

14) data-4476-0

15) data-4396-1

16) data-4396-0

17) data-3772-0

Thanks & Regards

Deepesh Mehta

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
lb

[Luca Burgazzoli]

If the reference is lost, a gc should have triggered.

There might be a bug in how the appender is recreated as it should pick-up the file it was on before (if not full) however chronicle v3 is not actively developed so if you do not need to use a logging framework best would be to migrate to chronicle-queue v4. 

I will work to make chronicle logger compatible with queue v4 in the coming weeks.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
lb

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
lb

[Peter Lawrey]

It may be worth considering chronicle queue v4. This creates only 1 file per cycle (per day by default)

[Deepesh Mehta]

Thanks Luca and Peter for prompt responses.

I will await for you to upgrade Chronicle Logger in order to make it compatible with Chronicle Queue v4. 

Once new version of Chronicle Logger is released, please keep me posted here and we will then upgrade our application.

Thanks & Regards

Deepesh Mehta

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
lb

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Deepesh Mehta]

Hi Luca,Peter

Has Chronicle Queue v4 released? On your official website's Maven link I still see 3.6.2 as the latest version.

Thanks & Regards

Deepesh Mehta

[Peter Lawrey]

Hi Deepesh,

   The link on the WP site needs to be updated. In our documentation.

https://github.com/OpenHFT/Chronicle-Queue

It notes that Queue 4.x is the latest and how to migrate to it.

The link to find the latest is

http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22net.openhft%22%20AND%20a%3A%22chronicle-queue%22

Regards,

   Peter.