Regarding Chronicle Queue indexes

[Vachagan Balayan]

Could you please explain how indexes are calculated for new chronicle entries...

For example this is of the outputs i'll get if i print [ cycle  : index : value ], where chronicle queue is rolled up minutely... 

Value is just text output of current date time...

Here is a sample output...

24424837 : 104903876125130779 : 2016-06-09 16:37:59

As i understand this id contains datetime information? or at least minute information? 

I'm trying to calculate what the id will be if i want to stream all entries starting from some N point in time...
Is there any utility to calculate these indexes from time instants?

[Vachagan Balayan]

Here is more output

24424845 : 104903910484869178 : 2016-06-09 16:45:57
24424845 : 104903910484869179 : 2016-06-09 16:45:58
24424845 : 104903910484869180 : 2016-06-09 16:45:59
2016-06-10 00:46:09,002 DEBUG[main] net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts : moveToIndex: 174b18e 0
24424846 : 104903914779836417 : 2016-06-09 16:46:00
24424846 : 104903914779836418 : 2016-06-09 16:46:01
24424846 : 104903914779836419 : 2016-06-09 16:46:02

I can see that Index is incrementing within a cycle, but changes a lot depending on which cycle is that...
Is there any documentation on this, where i could understand how the index is being calculated? and how do i calculate one myself...

[Rob Austin]

It uses a 2d tree. Each file has one primary and many secondary indexes ( the secondary indexes are built up as required ). The entries and indexes are in the same file. 

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Lawrey]

The top bits of the index are in cycles since epoch by default. This is shifted by N bits, often 32 in most cases.

You can use RollCycle.toIndex (cycle, sequence) to get any individual index. Eg.
long minuteCycle = System.currentTimeMillis () / 60_000;

This assume you have minutely cycles. You can have other cycles like 5 minutely.

Regards, Peter.

--

[Vachagan Balayan]

Oh this is perfect. 
Just to make sure i got this, whenever i want to find the first index of some cycle i do something like

RollCycles.MINUTELY.toIndex(cycle, 1)

(seems that first sequence number is 1 not 0?...)

[Rob Austin]

I think it should be zero. 

Sent from my iPhone

[Vachagan Balayan]

Rob here is the code snipped i use to test this... (its kotlin)

    while (true) {
        // todo : find out how to calculate right id for given long timestamp
        val message = tl.readText() ?: break
        val cycle = tl.cycle()
        if (lastCycle != cycle) {
            println("cycle : " + RollCycles.MINUTELY.toIndex(tl.cycle(), 0))
            lastCycle = cycle
        }
        println("${tl.cycle()} : ${tl.index()} : $message")
    }

piece of output when cycle changes

24425694 : 104907556912103482 : 2016-06-10 06:54:57
24425694 : 104907556912103483 : 2016-06-10 06:54:58
24425694 : 104907556912103484 : 2016-06-10 06:54:59
2016-06-10 15:31:52,774 DEBUG[main] net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts : moveToIndex: 174b4df 0
cycle : 104907561207070720
24425695 : 104907561207070721 : 2016-06-10 06:55:00
24425695 : 104907561207070722 : 2016-06-10 06:55:01
24425695 : 104907561207070723 : 2016-06-10 06:55:02

as you see first entry of new cycle starts with 1 not zero...

correct me if i'm getting this wrong.

My goal is to be able to find first id that i need to read from having a timestamp... 

So essentially i only need to implement this...

fun RollCycles.firstIdOfCycleContaining(timestamp: Long): Int

I need to stream data based on timestamp (like stream everything staring from this instant...)
So this will take timestamp -> calculate the cycle > then use that method to compute the first index... (which seems to use 1 not 0)

[Rob Austin]

I'm sorry I am away at the moment and I don't have my laptop with me. I will have to take a look when I get back. 

Sent from my iPhone

--

[Peter Lawrey]

When you set the index() of a tailer it is the index you expect to read next.

After the cycle changes this will have a sequence number of 1, because it just read a message with a sequence number of 0.

The tailer doesn't know until you attempt to read the an entry whether it has to roll or not, you only know whether after reading an entry whether it had to roll.

--