Re: Disruptor re-injection/re-entry & reservation (dead lock prevention)

[Michael Barker]

Hi Rob,

Firstly it is unlikely that we would implement this. Supporting
publishing back to the same ring buffer is just going to add
complexity.

My question would be what is step #3 doing. If this is just business
logic and does not require any sort of expensive blocking I/O, then I
would just inline step 2,3 and 4 into the same event handler.

Mike.

On Sat, May 18, 2013 at 3:47 AM, <crods...@gmail.com> wrote:
> Hello all,
>
> Wanted to ask if there was a better way to do something and suggest a
> feature if not.
>
> For simplicity, say you have a single RingBuffer, it has many
> EventProcessors. One of the EventProcessor's utilizes a resource that
> should only be accessed by a single thread because you want to avoid locks,
> synchronization, etc. For example, imagine it uses an off-heap NIO Buffer.
>
> Now let's also say you have to typical read-modify-write workflow, where you
> need to read data from the NIO Buffer, modify that data and then put the
> update back in the NIO Buffer.
>
> 1: do some operations
> 2: read from buffer (only a single thread should access buffer).
> 3: perform logic. create new value to store in buffer (possibly new entry
> or overwrite of old).
> 4: write to buffer (only a single thread should access buffer).
> 5: do some other operations.
>
> Now let's say the steps above map to individual EventProcessors in the
> RingBuffer. You can't have EP 2 and EP 4 access the same buffer because
> then you need locking and that would kill disruptor performance.
>
> One idea is to simply re-inject back into the ringbuffer so step 2 can
> perform reads or writes (depending on what stage the buffer entry is at).
> Step 4 would go away. Event processors like 1, 3, 5 would just ignore the
> entry (see there is no work and move to the next entry).
>
> I think this would work well, except there is one problem. What if the
> incoming rate of requests is high, the ring buffer may be full and now step
> 5 can't re-inject. Deadlock.
>
> To solve that problem, if the RingBuffer supported 'reserved slots', maybe
> things could be made to work. I.E. say there are 1000 slots in the
> RingBuffer, and we set it up with 2 reserved slots. Normal injection can
> only use 998 slots, the last 2 are reserved only for use of the
> 're-injector' at step 5. That way there are always slots available and
> deadlock isn't possible.
>
> Or is there a better approach?
>
> Thanks
> Rob
>
> --
> You received this message because you are subscribed to the Google Groups
> "Disruptor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lmax-disrupto...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[crods...@gmail.com]

Ideally the data is in the off-heap memory and 2,3 & 4 could be combined, but that may not always be the case.  The system has to deal with a "cache miss" (data not in off-heap memory) and perform IO to get the data.

I agree implementing 'reserved slots' in the ring buffer would lead to undue complexity in that code for just this one use case.  So it's not a good approach.  But what about another approach?

What if there were (for lack of a better name) a Priority Producer.  Usually it would just work as normal adding events to the RingBuffer in FIFO order.  Re-entrant events however would always be placed in the RingBuffer in preference to normal events.  (I'm not talking about changing the ordering of events already in the RingBuffer. Just always adding re-entrant events before new events if re-entrant events are waiting.)  The Priority Producer would of course always attempt to place re-entrant events directly in the ring buffer if possible, but if not it could internally buffer (in a queue or another smaller RingBuffer?) re-entrant events until there was room in the primary RingBuffer.  This way, events can be put back into the RingBuffer, deadlock is prevented, and in the normal situation of the ring being nearly empty there isn't a huge performance hit.  WDYT?

Alternately, think of it as a RingBuffer with multiple producers.  But events are always taken from one producer first (re-entrant gets priority), and only if it has no events are they taken from the other (new events).  If the first producer also provided minimal buffering (internal queue or another small RingBuffer) deadlock would be prevented.

Is either of these feasible and possible to keep performance?

Thanks

[Michael Barker]

Actually I've realised that you could get most of the way there with
the current API. If you made your main event publisher look something
like:

int reserve = ....
RingBuffer buffer = ....

boolean published = false;
do {
if (buffer.hasAvailableCapacity(reserve) &&
buffer.tryPublishEvent(.....)) {
break;
}
// Do some sort of back off, e.g. Thread.sleep(1)/LockSupport.parkNanos(1),
// drop the incoming message
// or buffer in the current thread and try to publish it with the next event
} while (true);

It doesn't guarantee dead lock prevention as there will be a race
between hasAvailableCapacity and tryPublishEvent, but unless you're
willing to drop events at some point any sort of circular relationship
between events can result in a dead lock.

Mike.