Using Disruptor to convert DB calls from sync to async

[Deepak Gupta]

Hi Team,

I am a newbie to disruptor library, Our use case is in one of our api which is heavily used has 1 DB read and 1 DB write operation involved.

Since read operation is local so that is not a problem, but writes can be remote,so we are planning to convert that call as async using this library.

We don't have to chain any operations and don't care about the sequence of items processing.

Till now based on our understanding with documentation on lmax git handle and some blogs we come up with the this strategy:

1 Disruptor 

1 Producer 

4/5 Consumer (Basically WorkerPool with 4/5 Workers)

BusySpinWaitStrategy 

We need experts comments on this are we going in right direction or instead of WorkerPool we should go ahead with multiple consumers as event handlers which divides the work based on hashing/bucketing or any other mechanism.

Deepak Gupta

[Michael Barker]

Typically when doing writes to a DB, we used a single consumer as this allows the consumer to group inserts into the same table into batches, which can improve throughput.  It's difficult to know whether it is appropriate within your application.

--
You received this message because you are subscribed to the Google Groups "Disruptor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmax-disrupto...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Deepak Gupta]

Mike , 

In our application we can not live with batching, because in batch mechanism we have to wait till the buffer is full to process that batch, which might be a delay of 1 second or so .

since it is security related application we can not have a delay of 1 second to update the user's state in db.

[Michael Barker]

We don't do time based batching either (we have latency targets of around 100µs).  The disruptor support adaptive batching which allows for batching when the arrival rate start to exceed the transaction time.  A good description of how this works is here: https://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html.

[Jason Koch]

Deepak,

I have had good results with the adaptive / smart batching as mentioned by Michael to relieve load on a database for log writes.

To do this, have a buffer that fills, and flush / write it to the database in two scenarios:

1) if the buffer is full, write the buffer

2) if the endOfBatch flag is set, write the buffer

You will need to test to see if it meets your throughput needs. The challenge you will have with Disruptor in this scenario is that your DB must be able to keep up throughput with a single writer, but if you are reliant on writing to a cluster of different nodes or have high latency you might need a different strategy.

I would not recommend WorkerPool as there is no easy way to get batching to work correctly, all workers will need to write each transaction to the database independently.

An alternative I have successfully used is without disruptor, which allows for multiple writer threads is to use a simple BlockingQueue and queue elements to this. Then you can have n threads perform a take followed by drainTo to collect batches of data to send.

Thanks

Jason

To unsubscribe from this group and stop receiving emails from it, send an email to lmax-disruptor+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Disruptor" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lmax-disruptor+unsubscribe@googlegroups.com.

[Deepak Gupta]

Hello everyone,

A bit reply to this thread, but can anyone through some light for this use case should we consider Kafka ?

Will it be a wise call to evaluate both the systems ? If we have a highly concurrent system and can afford to lose some of the events what should be choice ? and if we can not afford to lose even one event what should we go ahead with ?

[Michael Barker]

A bit reply to this thread, but can anyone through some light for this use case should we consider Kafka ?

No idea.

 

Will it be a wise call to evaluate both the systems ? If we have a highly concurrent system and can afford to lose some of the events what should be choice ? and if we can not afford to lose even one event what should we go ahead with ?

I think this conversation is drifting well away from the topic of the Disruptor.  My only comment is build a simple solution that could meet your needs then build the tests to prove that it works (e.g. tests failure scenarios for message loss, and production like volumes).  Then if it doesn't work you have the tests that you can re-apply to a subsequent solution.

Mike.