How to deal with the GC situations after the Ringbuffer is full

[Yang Derek]

I have a producer with around 70k messages and 3-4 consumers in the Ringbuffer.

The size of value event if around 25kb, according to the Memory Analysis in Eclipse, which means the total size of messages is 25kb*70k=1.7Gb (around).

I try to set the buffer size to be 64*1024=65536. It runs very fast at the beginning but when it comes to deal with around the 67000th message. The GC happens and it takes nearly 30-40s to wait for the GC. So the whole time is 55s.

And then I change the buffer size to be 4*1024=4096. It runs a little bit slower than the first trial above at the beginning but it's not obviously wasting time for GC. So the total time is 29s.

Here's my question:

It's obvious that the larger buffer size doesn't mean the better performance. But one of the consumer may be slow and I want the message would be in the ring buffer as soon as possible, so small ring buffer is not a option because the slow consumer may block the producer. I want to know if there's any way to minimize the impact of GC when the ringbuffer is set to a large number.

Thx

[deepesh soni]

best way is to pre-initialize the ring buffer with message containers. Instead of creating new message objects,update the message containers with the message contents. Your message containers will live as long as you are running the disruptor and no GC will happen 

[Rajiv Kurian]

I'd also be interested in knowing what GC settings people use in production when using the Disruptor in a long running process. What do people do if their ring-buffer entries end up taking quite a bit of heap (say more than a GB or two)?

Why is a small ring-buffer not an option? Blocking the producer when the consumer cannot keep up is desirable, since it will apply back-pressure through the pipeline. Having a giant ring-buffer will not block the producer but your events wont get processed any faster. Ultimately if the consumer cannot keep up the ring-buffer will get full. Ideally with test-runs for your application, you can figure out the smallest ring-buffer size that helps you stick to your SLAs. Keeping a bigger buffer is not an automatic win.

Thx

[Michael Barker]

If you are constructing new instances of objects each time that you
place an event in the Disruptor then you should make sure that you
null it out after all of the consumers have finished with it. You can
do that either inline or as a separate EventHandler that gates on all
of the other EventHandlers.

Mike.

> --
> You received this message because you are subscribed to the Google Groups
> "Disruptor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lmax-disrupto...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Yang Derek]

Hi Deepesh,

Thx for your reply.

Let say I receive a message object named FixReport every time.

What I'm doing right now is just create another object named FixReportContainer, having the FixReport inside as a field.

And the disruptor is initialized with FixReportContainer.

Your suggestion is to pass every field in the FixReport in the FixReportContainer to avoid the GC recycle the FixReport object??

在 2014年2月18日星期二UTC+8下午3时45分36秒，deepesh soni写道：

[Yang Derek]

Hi Mike

Thank  you for your reply. My disruptor is initialized with event object named FixReportContainer let say. And each time new instance of FixReport Object is pass into FixReportContainer as a field. I think the problem is that FixReport would be collected by the GC. So I should null the FixReport once all consumers finish using it by adding a separate EventHandler, right?

Another question is :

is it possible to turn off the one consumer within the onEvent() method. I have one consumer to connect to db but I want to turn it off if its connection is lost.

is there any workaround to shut it down?

在 2014年2月24日星期一UTC+8上午4时19分12秒，mikeb01写道：

[Michael Barker]

See inline

> Thank you for your reply. My disruptor is initialized with event object
> named FixReportContainer let say. And each time new instance of FixReport
> Object is pass into FixReportContainer as a field. I think the problem is
> that FixReport would be collected by the GC. So I should null the FixReport
> once all consumers finish using it by adding a separate EventHandler, right?

Yes.

> Another question is :
>
> is it possible to turn off the one consumer within the onEvent() method. I
> have one consumer to connect to db but I want to turn it off if its
> connection is lost.
>
> is there any workaround to shut it down?

You would have to manually remove it's sequence from the RingBuffer.
RingBuffer.removeGatingSequence(DBConsumer.getSequence()). However,
in practise what we tend to do in these situations is leave the
consumer running and have it just drop messages on the floor if the
database connection is not available. Much simpler and easier to
implement the logic to handle the case where the connection comes back
again.

Mike.

[Yang Derek]

#1 Sorry I'm not fully understanding the meaning of inline here. Do you mean the FixReport class should be declared as final? Or somethihng else?

#2 I don't understand this sentence "have it just drop messages on the floor if the database connection is not available". Do you mean the database consumer just do nothing when the connection is lost? If the connection is lost, it will throw exceptions.

在 2014年2月26日星期三UTC+8上午10时54分59秒，mikeb01写道：

[黄成]

Hi Mike,

I have a question related to this one about GC.

As I can avoid GC by using primitive data in Events, but the Business Logic module may translate events to business objects which may cause GC. Since they are in the same JVM, RingBuffer performance is still affected by VM GC.

How do you handle such situations?

Jay

[Michael Barker]

1) Move the business logic data off-heap and go to zero allocation on the common hot paths.  Have an Eden big enough to hold all of the allocations required between maintenance windows.

2) Use Azul's Zing.

We are current going with #2, if I was starting from scratch I would go with #1.  If neither is possible, then you need to spend time tuning GC such that you get best performance for your system end to end.  At this point the RingBuffer is only a small component of a larger system and in this situation you should worry more about the system a whole.

Mike

[黄成]

Hi Mike,

Many thanks for sharing your great tips.

在 2014年3月6日星期四UTC+8上午4时09分06秒，mikeb01写道：

[Yang Derek]

Hi Mike

Moving the business logic data off-heap is really a great idea.

Any concern about this? Or anything needed to be careful of?

在 2014年3月6日星期四UTC+8上午4时09分06秒，mikeb01写道：

[Michael Barker]

There is lot to be aware of depending on how you do it.  There is good article on InfoQ about this type of thing (http://www.infoq.com/articles/Open-JDK-and-HashMap-Off-Heap).  However, at LMAX we don't have off-heap code for any of our business logic in production.  Most of the off-heap work we do is for infrastructure code.

Mike.

For more options, visit https://groups.google.com/d/optout.