Get the current sequences being processed by the EventHandlers

[nachiket chaudhary]

Hi all,
I am working on a system where we have multiple EventHandlers from a single output RingBuffer.

Observation is that, at some point we see that the EventHandling does not proceed further.
Suspicion is that one of the EventHandler (E1) gets stuck.

And hence the other EventHandlers (E2) which have the EventHandler (E1) as barrier can't process the messages as it's not released by the EventHandler(E1).

We think that due to slowness in E1, E2 also waits for E1 to release the Event which can be processed.

Unfortunately, due to absence of proper debugging information we are not able to root cause this.

Query:
=====
Is it possible to get the current sequence numbers being processed by the EventHandlers ?
I think if we see somewhere that this sequence number is same for some elongated time, might give us some more information to proceed further.

Can anyone please point me to any way to dump the sequence-numbers being processed at any instant by the Event Handlers ?

Thanks,
- Nachiket C.

[Michael Barker]

We generally handle exposing the sequence number ourselves by storing the sequence in a field based on the value passed into the event handler.  This is something that I'm going to look at, doing this should be easier.

Do you used the Disruptor class to construct your ring buffer instances?

Mike.

--
You received this message because you are subscribed to the Google Groups "Disruptor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmax-disrupto...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Barker]

There are a couple of options available.

If you are using the Disruptor class, then you can do the following:

Disruptor<..> d = ...

EventHander<...> handler = ...

long currentSequence = d.getSequenceValueFor(handler);.

If you are building the BatchEventProcessor manually, then you can call EventProcessor.getSequence() to get hold of the sequence.

Mike.

[nachiket chaudhary]

Hi Mike,
thanks for the response.

We create the RingBuffer using following API:

 RingBuffer.createMultiProducer(ModelTaskHolder::new, INPUT_QUEUE_CAPACITY, new PowerFriendlyWaitStrategy());

So I think as you had mentioned, we might have to add more debugging capabilities to store those SequenceNumbers and then expose over JMX may be.

Unfortunately, we don't have any such existing information stored in the product.

Is it possible to enable some kind of LMAX-Disruptor logging which can give us more information on this ?

Thanks,
- Nachiket C.

[Michael Barker]

How do you add EventHandlers to listen to the ring buffer?

[nachiket chaudhary]

Hi Mike,
Thanks for your response.
Please find the requested information below.

Each  EventHandler is associated with it's own separate SequenceNumber (in this context - call it sequence). 
We maintain a group of such first level of  EventHandlers in  'SequenceGroup' - call it  earlyGroup.    
 earlyGroup.addWhileRunning(ringBuffer, sequence); 
 
We add the first level of  EventHandlers using no barriers such as follows:
"ringBuffer.newBarrier()"

And for second level of  EventHandlers, we use the first level of event-handles as barriers using following API:
"ringBuffer.newBarrier(earlyGroup)""

And then later, we use the following APIs in EventProcessor thread to get messages based on the configured barriers:
"sequenceBarrier.waitFor(seqNr)"

Please let me know if you need any more information.

Thanks,
- Nachiket C.

[Michael Barker]

How are you running the EventHandlers?  From the looks of it you are not using the BatchEventProcessor, instead doing your own event processing?  As you are doing your own construction at what point are you calling RingBuffer.addGatingSequences?

Mike.

[nachiket chaudhary]

As mentioned in previous mail, there are two levels of EventHandlers - early and late.
After late EventHandlers, we have a gating sequence :

final Sequence sequence = new Sequence();
ringBuffer.addGatingSequences(sequence);

And yes, we don't use the  BatchEventProcessor.
In gist, the flow will somewhat look like this -

while (true) {
    final long availableSeqNr = sequenceBarrier.waitFor(seqNr);
    for (; seqNr <= availableSeqNr; seqNr++) {
        // Get event using following and process it in the EventHandler
        ringBuffer.get(seqNr);

    }
}

Thanks,
- Nachiket C.

[Michael Barker]

Unfortunately as you are not using many of the supplied classes (e.g. BatchEventProcessor, Disruptor) to handle the events or set up the handlers there are very few places where logging could be injected such that we can track what is going on.  I think you will need to add your own monitoring to the track the sequences.

[nachiket chaudhary]

Okay, in that case we will see if it's possible to add some extra custom logs and provide a debug build to customer.
Thanks a lot Michael for your help.

- Nachiket C.