Concurrency and Off-Heap access (m-mapped file) in Java for inter-process communication doesn't work

1,003 views
Skip to first unread message

bob.tw...@gmail.com

unread,
Apr 30, 2015, 12:14:28 AM4/30/15
to mechanica...@googlegroups.com


Concurrency over unsafe and aligned memory does NOT work in Java. According to my tests it works 90% of the time. I was close!

What I am saying is:

Two JVMs exchanging messages through a memory-mapped file, in other words, both are accessing the same unsafe memory address through the MappedByteBuffer:

- First JVM writes a long to an aligned memory address (addressPointer % 8 == 0)

- Second JVM never sees the update. It keeps trying to re-read the long from the address but keeps getting an old value. Forever.

Is there a workaround to make this work or is it simply impossible to do it in Java?

If it is impossible, how does someone go about synchronizing access to a memory mapped file in Java, so two processes can share it and use it appropriately as a transfer concurrent queue?

Wojciech Kudla

unread,
Apr 30, 2015, 2:09:28 AM4/30/15
to mechanica...@googlegroups.com

We're using Peter's chronicle in many of our deployments. Some other environments employ IPC over a more lightweight, bespoke solutions (using mmapped files too).

Not sure if I'm interpreting your statement correctly, but IPC over mmapped files regardless of whether you use bytebuffer or unsafe does work in Java.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Wojciech Kudla

unread,
Apr 30, 2015, 2:15:47 AM4/30/15
to mechanica...@googlegroups.com

Are you using any memory fences in your tests?

bob.tw...@gmail.com

unread,
Apr 30, 2015, 2:38:34 AM4/30/15
to mechanica...@googlegroups.com
Are you using any memory fences in your tests?

No. How do I do that? I tried putLongVolatile. It did not work either :( I suspect is a visibility problem. One JVM gets the value cached in its CPU and don't get the write coming from the other JVM. Not sure how that can happen if I am using putLongVolatile. Passing null to its first argument. Hope someone can shed a light.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

Wojciech Kudla

unread,
Apr 30, 2015, 2:42:50 AM4/30/15
to mechanica...@googlegroups.com

How are you obtaining the value on the reader side?

On 30 Apr 2015 07:38, <bob.tw...@gmail.com> wrote:
Are you using any memory fences in your tests?

No. How do I do that? I tried putLongVolatile. It did not work either :( I suspect is a visibility problem. One JVM gets the value cached in its CPU and don't get the write coming from the other JVM. Not sure how that can happen if I am using putLongVolatile. Passing null to its first argument. Hope someone can shed a light.

On Thursday, April 30, 2015 at 2:15:47 AM UTC-4, Wojciech Kudla wrote:

Are you using any memory fences in your tests?

On 30 Apr 2015 07:09, "Wojciech Kudla" <wojciec...@gmail.com> wrote:

We're using Peter's chronicle in many of our deployments. Some other environments employ IPC over a more lightweight, bespoke solutions (using mmapped files too).

Not sure if I'm interpreting your statement correctly, but IPC over mmapped files regardless of whether you use bytebuffer or unsafe does work in Java.

On 30 Apr 2015 05:14, <bob.tw...@gmail.com> wrote:


Concurrency over unsafe and aligned memory does NOT work in Java. According to my tests it works 90% of the time. I was close!

What I am saying is:

Two JVMs exchanging messages through a memory-mapped file, in other words, both are accessing the same unsafe memory address through the MappedByteBuffer:

- First JVM writes a long to an aligned memory address (addressPointer % 8 == 0)

- Second JVM never sees the update. It keeps trying to re-read the long from the address but keeps getting an old value. Forever.

Is there a workaround to make this work or is it simply impossible to do it in Java?

If it is impossible, how does someone go about synchronizing access to a memory mapped file in Java, so two processes can share it and use it appropriately as a transfer concurrent queue?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

bob.tw...@gmail.com

unread,
Apr 30, 2015, 2:48:08 AM4/30/15
to mechanica...@googlegroups.com
I tried everything:

unsafe.getLong
unsafe.getLongVolatile
mappedBuffer.getLong

No luck !!!


On Thursday, April 30, 2015 at 2:42:50 AM UTC-4, Wojciech Kudla wrote:

How are you obtaining the value on the reader side?

On 30 Apr 2015 07:38, <bob.tw...@gmail.com> wrote:
Are you using any memory fences in your tests?

No. How do I do that? I tried putLongVolatile. It did not work either :( I suspect is a visibility problem. One JVM gets the value cached in its CPU and don't get the write coming from the other JVM. Not sure how that can happen if I am using putLongVolatile. Passing null to its first argument. Hope someone can shed a light.

On Thursday, April 30, 2015 at 2:15:47 AM UTC-4, Wojciech Kudla wrote:

Are you using any memory fences in your tests?

On 30 Apr 2015 07:09, "Wojciech Kudla" <wojciec...@gmail.com> wrote:

We're using Peter's chronicle in many of our deployments. Some other environments employ IPC over a more lightweight, bespoke solutions (using mmapped files too).

Not sure if I'm interpreting your statement correctly, but IPC over mmapped files regardless of whether you use bytebuffer or unsafe does work in Java.

On 30 Apr 2015 05:14, <bob.tw...@gmail.com> wrote:


Concurrency over unsafe and aligned memory does NOT work in Java. According to my tests it works 90% of the time. I was close!

What I am saying is:

Two JVMs exchanging messages through a memory-mapped file, in other words, both are accessing the same unsafe memory address through the MappedByteBuffer:

- First JVM writes a long to an aligned memory address (addressPointer % 8 == 0)

- Second JVM never sees the update. It keeps trying to re-read the long from the address but keeps getting an old value. Forever.

Is there a workaround to make this work or is it simply impossible to do it in Java?

If it is impossible, how does someone go about synchronizing access to a memory mapped file in Java, so two processes can share it and use it appropriately as a transfer concurrent queue?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

rick.ow...@gmail.com

unread,
Apr 30, 2015, 2:56:10 AM4/30/15
to mechanica...@googlegroups.com
Just my 2 cents here: why go through the trouble of doing IPC, serious? Just use the network. Is cleaner, bug-free and totally predictable. Easy to manage. Yes, it takes a coupler of microseconds more but do you really care for that? I bet not...

Martin Thompson

unread,
Apr 30, 2015, 3:03:29 AM4/30/15
to mechanica...@googlegroups.com
Have you worked with a large market data feed in finance doing low-latency trading?

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

Wojciech Kudla

unread,
Apr 30, 2015, 3:04:21 AM4/30/15
to mechanica...@googlegroups.com

I'm pretty sure for many people on this list a few micros per handoff makes a difference.
Also, mmap-backed IPC gives you journalling for (almost) free.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

rick.ow...@gmail.com

unread,
Apr 30, 2015, 3:16:07 AM4/30/15
to mechanica...@googlegroups.com
> Have you worked with a large market data feed in finance doing low-latency trading?

You won't be able to run everything on a single machine with 512 cpu cores anyways :) Stuff  running on the same machine can be placed inside the same JVM and talk to each other through disruptor. Not saying is the only way, but it is how we used to trade with some positive PNL :)
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

Martin Thompson

unread,
Apr 30, 2015, 3:16:26 AM4/30/15
to mechanica...@googlegroups.com
The few microseconds is typically about 16 via loopback in a realistic TCP application with a huge variance due to scheduling jitter. So if you do the math and want anything remotely predictable then you cannot deal with any problem space going into the 10s of thousands of event/messages per second. Then you need to add your data marshalling and everything else you doing on those threads.

Always go with the simplest solution for your requirements. For many the requirements are greater.

bob.tw...@gmail.com

unread,
Apr 30, 2015, 3:18:52 AM4/30/15
to mechanica...@googlegroups.com
Can anyone shed a light on my visibility problem? :P How to proper handle writer and readers exchanging sequences through the shared memory-mapped file without having visibility problems?

rick.ow...@gmail.com

unread,
Apr 30, 2015, 3:33:11 AM4/30/15
to mechanica...@googlegroups.com
If micros count then let's move everything to the same JVM and spend nanos instead of micros through pipelining (disruptor). Everything else goes to a distributed system, spamming multiple machines, not a single one. Decoupling is paramount.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

Martin Thompson

unread,
Apr 30, 2015, 3:45:09 AM4/30/15
to mechanica...@googlegroups.com

I've found it works perfectly well. You can see an many examples in the Agrona and Aeron projects. Simplest being the AtomicCounter class or even something more complicated like the ManyToOneRingBuffer.

https://github.com/real-logic/Agrona

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

Martin Thompson

unread,
Apr 30, 2015, 3:50:26 AM4/30/15
to mechanica...@googlegroups.com

Numa regions on the same machine are effectively distributed nodes. IPC is the best way to communicate between them if decoupled. This is why people typically use IPC. One big address space that spans multiple nodes is often the wrong solution.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

bob.tw...@gmail.com

unread,
Apr 30, 2015, 3:57:51 AM4/30/15
to mechanica...@googlegroups.com

Thanks, Martin. I took a look but still not sure why my IPC queue is not working. Putting in words the problem is:

Tow JVMs sharing a sequence (read and write sequence) through a shared memory-mapped file. Pretty much like the disruptor. What do I have to do to guarantee the visibility of these sequences? I believe I am doing everything:

- aligning in memory
- padding to avoid false sharing
- using unsafe.putLongVolatile to write the sequence
- using unsafe.getLongVolatile to read the sequence

Do I have to do anything else to synchronize and guarantee visibility? I wish I had the AtomicLong here, but we are across JVMs doing IPC :)
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

Martin Thompson

unread,
Apr 30, 2015, 4:07:00 AM4/30/15
to mechanica...@googlegroups.com

It you take the simplest thing that can possibility work then build from there is usually the best way to start. How about map a file in two processes as read and write. Then wrap the mapped buffer with my UnsafeBuffer class. Then write with putLongVolatile in one process and read with getLongVolatile in the other. Use offset 0 in the file for simplicity. You need to spin on the read to wait for the publication.