Question on LMAX unmarshaler
Min Zhou
From Martin's post, it refer that
" The unmarshaler turns the event data from the wire into a java
object that can be used to invoke behavior on the Business Logic
Processor. Therefore, unlike the other consumers, it needs to modify
the data in the ring buffer so it can store this unmarshaled object.
The rule here is that consumers are permitted to write to the ring
buffer, but each writable field can only have one parallel consumer
that's allowed to write to it. This preserves the principle of only
having a single writer."
Unmarshaler is the other writer of that ring buffer except the
receiver(event publisher). In order to accord the "single-write
principle", the unmarshaler should write to another field of the
original Event. For example, if we define the event type like below
public class DisruptorMessageEvent {
private Object field1;
private Object field2
public void setField1(Object o) {
this.field1 = o;
}
public void setField2(Object o) {
this.field2 = o;
}
public final static EventFactory<DisruptorMessageEvent>
EVENT_FACTORY =
new EventFactory<DisruptorMessageEvent>() {
public DisruptorMessageEvent newInstance() {
return new DisruptorMessageEvent();
}
};
and field1 is set by an EventPubliser.
long sequence = ringBuffer.next();
ringBuffer.get(sequence).setField1(object1);
ringBuffer.publish(sequence);
How to modify that event from the unmarshaler? Is there any similar
example on the web or in the source code of disruptor?
Thanks,
Min
Dave Farley
In our (LMAX's) case the producer writes the incoming bytes that represent a raw event into one field in a "message" object stored in the ring buffer. The unmarshalling event handler reads these bytes and translates them into a more usable form for the business logic, which it writes back to another, different, field (or fields) in the same "message" object.
When the marshaller, and other event handlers are finished, the business-logic event handler will call the appropriate method with the arguments. So the single-writer principle is preserved.
Dave Farley
Min Zhou
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Michael Barker
The flippant answer: "Use a messaging bus". We use Ultra Messaging
from Informatica, I've also heard good things about ØMQ. We let the
messaging bus deal with ordering and fragmentation. We add our own
reliability and late join support using the disruptor. The basic
premise is to sequence all of the packets coming across the network
and re-request any that are missed. We use the ring buffer as a store
for messages on the publishing side. If we miss a message we can
rewind the event handler to the missed message and redeliver.
Mike.
Adnan Selimovic
Michael Barker
> using some kind of object pool or always creating new objects for every
> event?
We create a new object for each event. Object pooling is a technique
that I'd only apply with the utmost care. It can create more problems
that it fixes. Note that the information in Martin Fowler's article
is now a bit out of date. We un-marshal on the same thread as the
business logic. It's simpler and has very little performance impact.
It also gives the JVM a chance to apply escape analysis and
potentially allocate the business event on the stack. After a couple
of conversations with experts in the VM area it was suggested that
this is friendlier to the GC than allocating an object on one thread
passing it to another and nulling it out quickly afterward.
However, I don't consider our solution to be the most optimal. There
is a zero garbage approach that I demonstrated at Devoxx this year.
I'm going to put up some example code as soon as I get it packaged
cleanly.
Mike.
Adnan Selimovic
Michael Barker
> presentation?
No problem, I've posted my most recent slides from Devoxx and JAX on my blog:
http://bad-concurrency.blogspot.com/2012/02/slides-from-recent-presentations.html
Paul
How do you rewind the Event Processor?
Thanks, Paul
>
> >> Your example here is pretty much it.
>
> >> In our (LMAX's) case the producer writes the incoming bytes that represent
> >> a raw event into one field in a "message" object stored in the ring buffer.
> >> The unmarshalling event handler reads these bytes and translates them into a
> >> more usable form for the business logic, which it writes back to another,
> >> different, field (or fields) in the same "message" object.
>
> >> When the marshaller, and other event handlers are finished, the
> >> business-logic event handler will call the appropriate method with the
> >> arguments. So the single-writer principle is preserved.
>
> >> Dave Farley
>
Michael Barker
and set it's sequence to a lower value (within a specified threshold).
Mike.
Andrei
It's an interesting approach. I presume you're using UDP ?
Can you please give more details about the messaging layer and how the
services are communicating between each other (pub/sub request/
reply) ? For instance, rewind/redeliver is it a separate request to
upstream feed ?
Thanks,
Andrei.
>
> >> Your example here is pretty much it.
>
> >> In our (LMAX's) case the producer writes the incoming bytes that represent
> >> a raw event into one field in a "message" object stored in the ring buffer.
> >> The unmarshalling event handler reads these bytes and translates them into a
> >> more usable form for the business logic, which it writes back to another,
> >> different, field (or fields) in the same "message" object.
>
> >> When the marshaller, and other event handlers are finished, the
> >> business-logic event handler will call the appropriate method with the
> >> arguments. So the single-writer principle is preserved.
>
> >> Dave Farley
>
Michael Barker
> services are communicating between each other (pub/sub request/
> reply) ? For instance, rewind/redeliver is it a separate request to
> upstream feed ?
Our messaging is topic based, i.e. pub/sub. Topics that we have
designated as reliable, will have a second associated topic, suffixed
with the string "_nak". A separate listener to the nak topic will
perform the rewind activity as required.
Mike.
Daniel Cegiełka
Andrei
Do you keep (queue) messages in the same java process or delegate to a 3rd party (middleware) ?
Andrei
Thanks for informative article. Do you think zeroMQ is an overkill in the context of socket to socket connections when no reliable delivery is required ? I'm thinking how would it compare to netty/MINA frameworks (we don't have more than 50 subscribers per process).
Regards.
On Saturday, February 25, 2012 9:32:27 AM UTC-5, danice wrote:
Quite well described.best regards,Daniel
2012/2/25 Michael Barker
> Can you please give more details about the messaging layer and how the
Michael Barker
Thanks for reply., Mike.
Do you keep (queue) messages in the same java process or delegate to a 3rd party (middleware) ?
Dmitry Shilovich
Thank you.