trouble getting disruptor to actually be fast

[Michael]

Hi, I've been trying to find a use for disruptor in my application.
I've tried to use it a couple times and it ends up being slower than
using ExecutorService, Callable & ArrayBlockingQueue. So I'm
obviously not hitting the sweet spot for its best performance.

In my latest attempt, I have a set of work tasks that I was performing
sequentially; the user's interface to this is an Iterator, so each
call to next I'd perform the next task and return the result. This
seemed like an obvious target for executing concurrently; the user's
iterator would pull off a queue that is populated by the worker
Callables executed via the ExecutorService.

Implementing this exact same design via disruptor where the workers
put the objects onto the ring buffer and a consumer pulls them off and
puts them in the queue is both slower than using java concurrent
primitives and suffers from large latency spikes.

I persisted with using disruptor because the resulting iterator has
some transformations applied to it and it seemed like these steps
could just be additional event handlers in disruptor so they too could
be performed concurrently rather than as the user iterates over the
final Iterator.

I created event handlers for these and inserted them in between the
original producer threads and the final consumer which puts the
results on the queue for the Iterator, which worked ok. The minimum
execution time is comparable with the non-disruptor solution, but it
still suffers from massive spikes, sometimes up to three orders of
magnitude.

I realize that the last step of putting the results onto a
BlockingQueue that's visible to the user is a bit of a bottleneck, but
the consumer event handler & the user are in different threads, so a
plain collection is not really appropriate. I'd like to pull directly
off the ring buffer and see if that helps, but I don't know if that's
possible.

Furthermore, I'm curious if there are any tips or tricks for using
disruptor to get the most out of it. The performance claims are
impressive, and I'd like to use if its even half that fast, but I have
yet to use it and have it be faster than just using the plain old java
concurrent primitives.

Thanks.

Michael

[Michael Barker]

Hi Michael,

A couple of questions:

- What wait and claim strategies have you configured for your disruptor instances.

- What is the platform that you are testing on?

- Can you produce the source code for a cut down performance test to illustrates your problem?

- Have you profiled the test to see where the time is being sent?

Mike.

[Mike Grove]

On Wed, Feb 29, 2012 at 2:50 PM, Michael Barker <mik...@gmail.com> wrote:
> Hi Michael,
>
> A couple of questions:
>
> - What wait and claim strategies have you configured for your disruptor
> instances.

Currently using the YieldingWaitStrategy & MultiThreadedClaimStrategy
w/ a buffer size of 8192. Setting up the disruptor using the DSL, ala
handleEventsWith(foo).then(bar).then(baz).

> - What is the platform that you are testing on?

OSX 10.6.6, Java 1.6.0_22

> - Can you produce the source code for a cut down performance test to
> illustrates your problem?

Not easily, this is embedded in a fairly large system at the moment
trying to replace a sequential pipeline with something that will take
more advantage of CPU resources. I started patterning the code off of
ThreePublisherToOneProcessorSequencedThroughputTest, though the number
of publishers varies in my current tests from 2 to 57. The single
processor is the one that just pulled off the ringBuffer and put it
into a BlockingQueue for further processing, 4 more stages performed
sequentially as each thing comes off the queue, after that, these are
then pushed to the user.

While that worked, it seemed that was not an optimal use of disruptor,
so I then tried moving those four stages into the workflow disruptor
is using so that events passed through them before getting put into
the BlockingQueue -- at this point, they're ready to send to the user.

There was very little difference in performance between these two
setups; small enough that it was with the expected JVM wiggle, which I
found quite surprising. My guess is that the bottleneck is either
getting stuff onto the ring buffer or getting stuff off.

> - Have you profiled the test to see where the time is being sent?
>

I am attempting to get the netbeans profiler going with my latest code
to try and discern what could be going on. If i can pin point
something that looks like its in disruptor, I can try and distill my
prototype down into a relevant sample. In the meantime, i was hoping
for some suggestions on setup in case I was doing something obviously
dumb.

Is there a way to take something off the end of the ringbuffer so I
can treat it like a queue, that way i could reduce one of the
bottlenecks of having to transfer results from the buffer to a
BlockingQueue.

Cheers,

Michael

[Michael Barker]

I must admit I'm struggling to picture your solution, therefore it's difficult for me to know where the potential bottleneck is.  However, there are couple of things that you could try and see the how they fair.  If you are using a version earlier than 2.8, then you should upgrade.  Earlier versions of the disruptor would struggle when the number of the publishers exceeded the number of available cores.  You may be filling up the ring buffer and the back off is causing the unusual jitter.  Either have a separate tool (e.g. JMX MBean) track the sequences of the event handler and the cursor or use the hasAvailableCapacity() method to determine if this the case.  Experiment with larger buffer sizes for both the ring buffer and the MultithreadedClaimStrategy.

OSX 10.6.6, Java 1.6.0_22

Actually I was more interested in the type and number of cores.

Mike.