Using Chronicle Queue index

[Sam Rose]

Hi,

I am looking for some advice on using Chronicle Queue to solve one of my problems.

I need to have a persistent store of a very large number of Strings that represent ids. Basically there will be a single ID generator that fills the queue. There will be several consumers that then read from this queue. In order to ensure that an ID is not used twice, items should be removed from the queue when they have been read by a consumer.

I only have access to Chronicle version 3.4.4 right now. I see from the 'Reading-the-Chronicle-after-a-shutdown' that there are several solutions I can use to ensure that I don't read the same ID twice.

It seems I need to store the index of the last read ID and just keep increasing that index as I read elements.

Do you think storing the ID in a Chronicle Map would be the best way to go here?

Are there any examples of how this might be done best? Also, I would need to remove the old queue files that contained IDs that have already been read. Are there any examples of this too?

Thanks!

[Peter Lawrey]

Hello,
   I suggest using Chronicle Queue v4.x I have some examples of how to ensure a reader does reprocess messages it has already processed in my blog. See microservices part 2 & 3 in particular.
https://vanilla-java.github.io/

You can also store the index in Chronicle Map by saving the index and resetting the Tailer to play from that point. When the index rolls between cycles you could delete older files.

Regards, Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sam]

Hi Peter,

Unfortunately only version 3.4.4 is available to me right now. There are limitations to library access for the project I am working on.

I read your blog posts, is something similar available in v3? If not, do you recommend any other methods?

Can you please provide a small code sample of the logic for deleting the older files? 

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

[Peter Lawrey]

Hello Sam,
   Are you using Vanilla Chronicle?  If so, you can determine from the index when the cycle has changed. At this point you can delete the old directory. We haven't supported V3 for quite some time so I don't remember all the details.

Best regards, Peter.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Sam]

Thanks Peter, I'll try and implement what you advised and get back to you if there are any issues.

[Sam]

Hi again,

I've managed to get hold of Chronicle v4.5.15.

I am using a persistent map to keep the index and it is working well.

The only issue I have is with deleting old files.

Is there a simple way to get the file path given a cycle? Right now I am using a StoreFileListener to add the file path to a map with the cycle as a key. When I am retrieving elements and see the cycle had changed, I look up the map and delete that file. I am getting issues with the delete though since the file being deleted is still open it seems.

Any advice?

[Rob Austin]

ChronicleQueue q = ….

final File file = q.file();
q.close();
if (file.isDirectory())
    IOTools.shallowDeleteDirWithFiles((file));

IOTools is in chronicle core

[Sam]

Hi Rob,

This will delete all of the queue files meaning unprocessed elements will be lost.

I want to delete just the files that have been completely read hence why I keep track of the cycle numbers.

Any way of getting the path of the file given a cycle?

Thanks

[Rob Austin]

if you know the index of the message you have just read, you can get the cycle from the index

int cycle = ((SingleChronicleQueue) queue).rollCycle().toCycle(<index>)

  or you can find all the cycles between 2 index by calling

 ((SingleChronicleQueue) queue).listCyclesBetween(  lowerCycle,   upperCycle) 

from the cycle you can work out the filename, as an example, this codes shows you how you can create a tree map keyed on key=cycle, value=filename 

see net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.StoreSupplier#cycleTree

private NavigableMap<Long, File> cycleTree() {

    final File parentFile = path;

    if (!parentFile.exists())
        throw new AssertionError("parentFile=" + parentFile.getName() + " does not exist");

    final RollingResourcesCache dateCache = SingleChronicleQueue.this.dateCache;
    final NavigableMap<Long, File> tree = new TreeMap<>();

    final File[] files = parentFile.listFiles((File file) -> file.getName().endsWith(SUFFIX));

    for (File file : files) {
        tree.put(dateCache.toLong(file), file);
    }

    return tree;

}

Rob

[Sam]

I am able to get a map of cycle-->filename using the StoreFileListener.

When I called File.delete() it fails. It seems the file is still held by Chronicle.

When does Chronicle release the file so that it can be deleted? When I debug, I see that the listeners onRelease method isn't called even when the queue moved on to a new cycle.

[Rob Austin]

are you running on windows ?

[Sam]

Yes, I am developing on windows but will be deploying on Linux.

[Rob Austin]

I take your point regarding - "When does Chronicle release the file so that it can be deleted? When I debug, I see that the listeners onRelease method isn’t called even when the queue moved on to a new cycle. “

this is something that we will investigate, 

Could you retest on linux as ( I think ) the file handles behave differently. I’d be interested to know if you have the same problem on linux.

Rob

[Sam]

Hi Rob,

I have tested on Linux and the files are being deleted as expected.

I do see some messages saying "Rolled n times to find the next cycle file. This can occur if your appenders have not written anything for a while ... "

Currently I have a roll cycle of test seconds so I'm guessing the above error is due to no data being written for gaps longer than one second. Is this safe?

Thanks

[Rob Austin]

Yes, the code is working as expected, the reason for this warning is that you may not we writing a message every second, it’s warning you have some seconds without corresponding queue files.

I do not recommend that you use test secondly in production !

Rob

[Sam]

Ok cool.

One last thing, if I leave out '.rollCycle' when constructing the queue, what is the default behaviour?

I'd prefer to have the file roll on a weekly basis.

[Rob Austin]

the default is  RollCycles.DAILY;

see net.openhft.chronicle.queue.impl.AbstractChronicleQueueBuilder#AbstractChronicleQueueBuilder for defaults 

public AbstractChronicleQueueBuilder(File path) {
    this.rollCycle = RollCycles.DAILY;
    this.blockSize = 64L << 20;
    this.path = path;
    this.wireType = WireType.BINARY_LIGHT;
    this.epoch = 0;
    this.bufferCapacity = 2 << 20;
    this.indexSpacing = -1;
    this.indexCount = -1;
}

we don't currently provide a weekly roll cycle, this is whats currently available

TEST_SECONDLY("yyyyMMdd-HHmmss", 1000, 1 << 15, 4), // only good for testing
MINUTELY("yyyyMMdd-HHmm", 60 * 1000, 2 << 10, 16), // 64 million entries per minute
TEST_HOURLY("yyyyMMdd-HH", 60 * 60 * 1000, 16, 4), // 512 entries per hour.
HOURLY("yyyyMMdd-HH", 60 * 60 * 1000, 4 << 10, 16), // 256 million entries per hour.
TEST_DAILY("yyyyMMdd", 24 * 60 * 60 * 1000, 8, 1), // Only good for testing - 63 entries per day
TEST2_DAILY("yyyyMMdd", 24 * 60 * 60 * 1000, 16, 2), // Only good for testing
TEST4_DAILY("yyyyMMdd", 24 * 60 * 60 * 1000, 32, 4), // Only good for testing
SMALL_DAILY("yyyyMMdd", 24 * 60 * 60 * 1000, 8 << 10, 8), // 512 million entries per day
DAILY("yyyyMMdd", 24 * 60 * 60 * 1000, 16 << 10, 16), // 4 billion entries per day
LARGE_DAILY("yyyyMMdd", 24 * 60 * 60 * 1000, 32 << 10, 32), // 32 billion entries per day
XLARGE_DAILY("yyyyMMdd", 24 * 60 * 60 * 1000, 128 << 10, 256), // 2 trillion entries per day
HUGE_DAILY("yyyyMMdd", 24 * 60 * 60 * 1000, 512 << 10, 1024), // 256 trillion entries per day
;

[Sam]

Ok, I'll stick with daily.

Thanks a lot for all your help!

[Nicholas Whitehead]

For deleting rolled files when running in Windows (detected using os.name sysprop) I registered the file with a scheduled task that periodically attempts to delete registered files. Eventually, the file handle is released and deletion succeeds. Works ok.