Producers, consumers, and the event bus

[d465h...@gmail.com]

Hi

I have an application which uses a standard producer-consumer model. Producers put data on bounded queues, consumers read data from bounded queues. Producers block when the queue is full, consumers when it is empty.

I have been thinking about how to implement this in Vert.x, and have sort of hit a road-block on a number of issues. I will address the first one here.

I could do the whole thing using worker verticles, but if I do that I basically might as well use my current setup (which uses a thread pool). So I was thinking of using non-blocking verticles (as recommended in the docs), and have each producer put its data on the event bus, and have each consumer subscribe to the appropriate events on the bus.

If I do this I have a question. What happens if the event bus is full? Do the producers block, or are the events just thrown away? Or is the event bus unbounded? In the latter case what happens if the producers are much faster than the consumers, and the event bus just consumes all of memory?

thanks
graham

[Brian Maso]

I know other actor-ish systems have you set up the type of mailbox you want your consumers to use: in-memory only,
overflow to disk, disk-only, etc.

I assume the vertx bus, which is currently in-memory only, would eventually deliver you an OutOfMemoryError -- though I must admit I haven't actually tried to provoke that kind of behavior yet. In any event, it seems to me a you'd want to have a window on the event bus backlog, and add new consumer instances (or fewer producers) when the backlog starts getting large.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Best regards,
Brian Maso
(949) 395-8551
Follow me: @bmaso
br...@blumenfeld-maso.com

[d465h...@gmail.com]

Changing the number of producers isn't doable, but changing the number of consumers may be doable -- nice idea :-)

What I am sort of getting at with my questions is that a standard queuing approach to producers and consumers has a natural throttling mechanism built in (when the queues become full -- the mechanism of course has a cost), and I am trying to understand how to simulate that mechanism in an asynchronous framework like Vert.x.

[Daryl Teo]

I believe what you are trying to do runs contrary to the "never block" philosophy of vertx. If it cannot put something in , then it should be notified as such, and not sit and wait.

Daryl

[Daryl Teo]

Also have a look at https://github.com/vert-x/mod-work-queue if it helps.

Daryl

[d465h...@gmail.com]

So if a producer can't put something on the queue then it's only other choice is to sleep and try again -- so a spin loop with a delay?

[Daryl Teo]

Or it could be notified when the queue is not full, maybe.

Daryl