Re: Disruptor and multiple JVMs

[Michael Barker]

Your main challenge with be how to have the consumer ensure the
correct ordering and visibility of the data that is being written to
the memory mapped file. The current implementation of the Disruptor
relies on the Sequence class providing the necessary happens before
edges. If you are backing onto a memory mapped file being read by
another process then the sequence objects won't be visible to another
process, also how will you make the consumer sequences available to
the Disruptor to ensure that you don't wrap the ring buffer?

There are solutions to these, but it's probably not possible with the
Disruptor as is. I think that it can be done, however you may want to
look at Java Chronicle
(https://github.com/peter-lawrey/Java-Chronicle) as it may be a closer
fit to what you're trying to achieve.

Mike.

On Mon, Feb 25, 2013 at 3:20 AM, adam.h. <adam.p...@gmail.com> wrote:
> Hi Disruptors,
>
> I'm working on transaction engine which is split for several components
> running on separate JVM's. Although Disruptor was originally designed to
> pass messages between threads on the same JVM, the overall concept seems to
> be very tempting also in my case. So my thinking would be along these lines:
>
> - producer creates the Disruptor and allows it to allocate memory via
> MappedByteBuffer
> - producer stores marshalled messages (Google Protobufs)
> - consumers (different JVMs) can link to the same memory mapped buffer
> - consumers will unmarshall messages before passing them to the business
> logic processor
>
> Do you see any drawbacks of that design? Am I loosing any of advantages
> using Disruptor that way?
>
> Thanks,
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Disruptor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lmax-disrupto...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[adam.h.]

Mike, thanks for the pointer

[adam.h.]

So it turned out that inside one JVM the Disruptor model is very good fit for my application, and between different JVM's I just need simple FIFO connection. I made few tests with MappedByteBuffer and got performance like 400-500k of 1kB transactions per second between 2 different JVMs. This is more than I expected, however I would like to confirm with you guys if I'm not doing something stupid.

In my FIFO implementation I'm assuming that there will be always 1 producer and 1 consumer, therefore I did not use any kind of locks or CAS commands. The FIFO is round robin table of 1kB memory chunks and values of <first> and <last> indexes are stored in first (not used) data block. I only located them in distance bigger than 64 bytes, to make sure that I will not have false share problem described by Martin. These variables are modified by using unsafe.getIntVolatile or unsafe.putIntVolatile.

The sequence (simplified) for producer is always:

- check if enough space (<last>+1 != <first>)

- put data

- <last> = <last> + 1

The sequence (simplified) for consumer is always:

- check if buffer not empty (<last> != <first>)

- get data

- <first> = <first> + 1

So it seems that everything works as expected, during several tests I did not miss any of these messages. However this "unlocked" FIFO model is somehow new way of thinking for me, so I just want to check if I'm not overlooking something here.

[Nitsan Wakart]

2 points I took into consideration in my implementation:

1. Make sure the offheap memory you use is aligned such that you know you are setting your counters in unaligned positions or crossing the cache line boundary.

2. If you are limiting usage to 1 producer/consumer you can use putOrdered rather than volatile to write new sequence numbers.

[Nitsan Wakart]

I wrote 2 relevant posts on this topic which you might find useful:

1. Aligned direct memory access in java - showing how to go about aligning your offheap buffer. Note that since JDK 7 direct buffers are no longer necessarily page aligned.

2. Concurrent memory access and alignment implications - illustrating what happens when you do unaligned memory access.

There's another post on lazySet/putOrdered compared with volatile access, this is also in use in the Sequence class.

[adam.h.]

Thanks Nitsan, this is great stuff.

[Nitsan Wakart]

Finally put it in a blog post: http://psy-lob-saw.blogspot.com/2013/04/lock-free-ipc-queue.html

Enjoy