Is there a proper way to cleanup processed files in chronicle queue?

[Sarath Chandran]

Hi,

I am trying to use the most recent version 4.4.4 of chronicle queue. My use case is to read queue's data from multiple consumers. Queue's source is written by a single producer. I need to ensure that each consumer processing at a different speed has successfully read the data from queue. Lets say I have 4 consumers, then of these 4, whichever has processed to least index number (i.e slowest) is taken and files before that has to be deleted. By this way I can ensure that the rest of the consumers are reading ahead and no data is lost in transfer.

File deletion is to avoid disk out of space issues. I have written a sample code and wanted to ensure if this is the preferred way to cleanup the resource files.

public static void cleanup(SingleChronicleQueue queue, Long lastReadIndex) {
    RollingResourcesCache cache = new RollingResourcesCache(queue.rollCycle(), queue.epoch(), (name) -> new File(queue.file().getAbsolutePath(), name + SingleChronicleQueue.SUFFIX));
    ExcerptTailer reader = queue.createTailer();
    if (reader.moveToIndex(lastReadIndex)) {
        Long lastCleanableTime = cache.resourceFor(reader.cycle()).millis;
        Long loadedResourceTime = cache.resourceFor(reader.toStart().cycle()).millis;
        while(lastCleanableTime > loadedResourceTime) {
            RollingResourcesCache.Resource currentResource = cache.resourceFor(reader.toStart().cycle());
            loadedResourceTime = currentResource.millis;
            log("deleting file => " + currentResource.path);
            currentResource.path.delete();
        }
    } else {
        log("Bug!! Didn't move to index: "+ lastReadIndex);
    }
}

Note that i am still losing the last cycle's file totally even if my reader is reading at some random index within the file. Is there APIs like previous cycle or next cycle? Do suggest if there is better alternative.

Also, I wanted to know 3 other parameters used for building the queue.
1. BlockSize 
2. IndexCount
3. IndexSpacing
This confuses me a bit, because I am testing with SECONDLY roll cycle. Therefore, the queue data files are rolled each second. The default file size is 80 MB (on a mac machine). Now, how are the parameters like block size (which i assume is the maximum file size for data before rolling to new file) or index count (number of indexes for the data file to be used before rolling to new file).

Basically, my question is are there 3 factors for a new data file to be created. (1. RollCycle Time, 2. Block Size of data 3. Index Count). And what is the purpose of index spacing? Sorry if the question is too noobie :)

[Peter Lawrey]

Note the latest version 4.4.4 you can add a StoreFileListener so you can be notified when a new file is created or no longer needed.

@FunctionalInterface

public interface StoreFileListener {

    default void onAcquired(int cycle, File file) {

    }

    void onReleased(int cycle, File file);

}

----------------

The RollCycle time determines the cycle count. e.g. if your epoch is 0 , the cycle is the number of cycles since 1970.

    default int current(TimeProvider time, long epoch) {

        return (int) ((time.currentTimeMillis() - epoch) / length());

    }

Note: you can play with the SetTimeProvider to test whether daily rolling does what you want in a controlled manner.

The indexCount is the number of entries in the index table, The indexSpacing determines the space between indexed entries.  The maxiumum number entries per cycle is indexCount ^2 * indexSpacing.

The blockSize determine the basic block size however an overlap between block of 25% is used to make accessed more efficient. The default blockSize is 64 MB and so the minimum file size is 80 MB.  On UNIX and MacOSX files are allocated lazily so you only the the pages on disk which you touch, this means no matter the blockSize the minimum file space used by default is around 0.5 MB (for the two levels of indexes)

A basic assumption is that disk space is cheap. An hour of your time is worth at least 11 GB even on minimum wage and if you spend more than a few minutes saving a GB it might not be worth your time.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mickael Marrache]

Note that you will not be able to remove your file in the StoreFileListener if you are using Windows. There is an open issue on this.