State for event in LMAX Disruptor

[Sơn Phạm]

Hi guys. 

I'm new to LMAX Disruptor and the concept of sequence.

I'm trying to migrate my application from using multiple queues between threads to a single ring buffer.

The flow of my application is described as follows:

1. There is a server to receive requests from clients. After receiving a request (called Request1), claim 1 event slot from the ring (called Slot1), overwrite the event object fields with data from the request, and publish it. For example:

sequence = ring.next()

event = ring.get(sequence)

event.setData("someData")

ring.publish(sequence)

2.  When the consumer receives an event from the ring, it has to make an asynchronous call to an external service (via HTTP, GRPC,...) to fetch more data before continue processing the event. For example:

EventHandler{

    public void onEvent(Event event){

        processingData();

        sendRequest();

    }

}

3. Because the call was asynchronous, the onEvent function returns right after the call, and the event object (Slot1) is returned to the ring (assumed that there is only one consumer). While waiting for the external process to respond,  the event object Slot1  might be used again to process other requests (because the ring.next() and ring.get() are circular), so the data of the processing request (Request1) might be overwritten. This is unexpected because if Slot1 is overwritten then when the external service responds, I can not get the original data of Request1 by reading data of the event at Slot1 in the ring.

At this point, I think I have to cache the event's data before it is returned to the ring. But this might require many object creations and copying, which is quite expensive. I want to tell the disruptor that the slot of the current event is waiting for more data, so other producers can not use this slot to publish events for the new incoming requests. For example: When fetching a new sequence from the ring, I can override the ring.next() function to add a check if the event object in the slot is truly available for overwriting, which means it has received data from external service and finished processing before committing back to the ring.

Are there any solutions for this?