Is disruptor for situation where I have a slow consumer that must process events sequentially?

[Becky]

I have read about parallalizing a consumer with multiple threads, but what can I do if I have a slow consumer that needs to process events in sequence, in other words, they cannot be parallelized by multiple threads? My understanding is that a slow consumer will eventually impact the whole systems causing all other consumers to be affective. Is that correct? What can be done with the disruptor in that situation if anything?

[Michael Barker]

In a producer/consumer situation where the consumer does not perform particularly well, then eventually under sustained load eventually that consumer will cause back-pressure throughout the system as the ring buffer will fill up and the producer will have to make a decision about what to do with messages that can't be enqueued immediately.  Using solutions like queues and the Disruptor can help with handling bursts of traffic as long as your steady state load is below the the maximum processing rate of the consumer and the bursts are not too long or high.  If the slow consumer is still an issue then there are a couple of questions you can ask of your requirements.

1) Do you need to process every message?  If a message is skipped/lost, will it break the system.  If not, then when the ring buffer is full then the producer can drop all incoming requests.  It doesn't necessarily give the most desirable behaviour, but does prevent the slow consumer from impacting the rest of the system.  We do this for things like log messages.

2) Can you partition?  While you need to process messages in order, does every message have to be in order or can that be reduced to for all messages containing the same key need to be in order.  E.g. if I am processing messages for customers.  I care that customer A's messages are in the correct order and customer B's in the correct order, but I don't care about the total order of both customer A's and B's messages.  In that situation I can partition the processing of customer A and customer B onto separate threads.  I ensure that each incoming message has a key (e.g. customerId) that I can apply a consistent hash on so that messages for the same customer will be processed by the same consumer.

3) Can you batch?  I most cases slowness if often caused writing to some sort of slow interface, e.g. network, disk.  In both of those cases aggregating multiple events into a single request can be a useful way of achieving better performance.  E.g. with disk I/O there will be very little difference between doing a synchronous write of 1 byte and a synchronous write of 4096 bytes, so if you messages are 200 bytes then you could aggregate ~20 of them together into a single write.  Similarly with network I/O.  The Disruptor has a endOfBatch flag on the onEvent method of the EventHandler.  You can use this flag as a signal to tell you when you need to flush your batch of messages.  There was a long discussion about batching in this thread: (https://groups.google.com/forum/#!topic/lmax-disruptor/FSVPJ6tQdaY) which includes some code examples.  If you are in the situation mentioned in (1) then you can also use the endOfBatch flag to drop/coalesce message.  This is a useful approach when dealing with market data where you only care about the last event for any given instrument.

Beyond that, there is nothing much the Disruptor can do.  If you consumer is slow, then you need to fix the consumer.  We had a similar problem at one point with a piece of software and eventually did a ground-up re-write, took 9 months in this case, but it was worth it in the end, although we provided more than performance enhancements.

Mike.

On 17 June 2014 12:25, Becky <rahlv...@gmail.com> wrote:

I have read about parallalizing a consumer with multiple threads, but what can I do if I have a slow consumer that needs to process events in sequence, in other words, they cannot be parallelized by multiple threads? My understanding is that a slow consumer will eventually impact the whole systems causing all other consumers to be affective. Is that correct? What can be done with the disruptor in that situation if anything?

--
You received this message because you are subscribed to the Google Groups "Disruptor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmax-disrupto...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yang Derek]

Mike is right. This situation was in my way before but I managed to fix the slowness of the consumer.

在 2014年6月17日星期二UTC+8上午8时25分34秒，Becky写道：

[Jason Koch]

I think there's a number four?

Can you split the consumer into multiple stages? If so, you could pipeline the work to multiple stages, using .then() if data is not changed, or additional ringbuffers if the data is transformed between stages.

Good example of this pipeline approach is the way that Trish's early talks discuss the creating of a car in multiple stages.

Of course ymmv and testing is required to be sure overheads make the extra work worthwhile.

On 17 Jun 2014, at 12:01 pm, Michael Barker <mik...@gmail.com> wrote:

In a producer/consumer situation where the consumer does not perform particularly well, then eventually under sustained load eventually that consumer will cause back-pressure throughout the system as the ring buffer will fill up and the producer will have to make a decision about what to do with messages that can't be enqueued immediately.  Using solutions like queues and the Disruptor can help with handling bursts of traffic as long as your steady state load is below the the maximum processing rate of the consumer and the bursts are not too long or high.  If the slow consumer is still an issue then there are a couple of questions you can ask of your requirements

1) Do you need to process every message?  If a message is skipped/lost, will it break the system.  If not, then when the ring buffer is full then the producer can drop all incoming requests.  It doesn't necessarily give the most desirable behaviour, but does prevent the slow consumer from impacting the rest of the system.  We do this for things like log messages.

2) Can you partition?  While you need to process messages in order, does every message have to be in order or can that be reduced to for all messages containing the same key need to be in order.  E.g. if I am processing messages for customers.  I care that customer A's messages are in the correct order and customer B's in the correct order, but I don't care about the total order of both customer A's and B's messages.  In that situation I can partition the processing of customer A and customer B onto separate threads.  I ensure that each incoming message has a key (e.g. customerId) that I can apply a consistent hash on so that messages for the same customer will be processed by the same consumer.

3) Can you batch?  I most cases slowness if often caused writing to some sort of slow interface, e.g. network, disk.  In both of those cases aggregating multiple events into a single request can be a useful way of achieving better performance.  E.g. with disk I/O there will be very little difference between doing a synchronous write of 1 byte and a synchronous write of 4096 bytes, so if you messages are 200 bytes then you could aggregate ~20 of them together into a single write.  Similarly with network I/O.  The Disruptor has a endOfBatch flag on the onEvent method of the EventHandler.  You can use this flag as a signal to tell you when you need to flush your batch of messages.  There was a long discussion about batching in this thread: (https://groups.google.com/forum/#!topic/lmax-disruptor/FSVPJ6tQdaY) which includes some code examples.  If you are in the situation mentioned in (1) then you can also use the endOfBatch flag to drop/coalesce message.  This is a useful approach when dealing with market data where you only care about the last event for any given instrument.

Beyond that, there is nothing much the Disruptor can do.  If you consumer is slow, then you need to fix the consumer.  We had a similar problem at one point with a piece of software and eventually did a ground-up re-write, took 9 months in this case, but it was worth it in the end, although we provided more than performance enhancements.

Mike.