Re: [guava] Batching for blocking queues in Guava

[Louis Wasserman]

In some ways it sounds like you need a blocking queue with two Semaphores attached, one for the number of elements that can be taken, one for the number of elements that can be added.

On Tue, May 14, 2013 at 4:35 AM, <re...@exmachina.nl> wrote:

Hi all,

I have a requirement for a blocking queue that implements some sort of batching. In other words, I want a BlockingQueue implementation where the take() method blocks until either the queue reaches a specified batch size or (optionally) an expiration time. On the flipside put() will block until the amount of elements that were available at the time the queue reached the batch size or the expiration are drained.

The use case is that I have units of work that are exceptionally more efficient when done as larger batches. I would like producers of work to be able to add individual items to the queue above and consumers to ideally be able to drain larger batches if they are available. For example, I could wait for the queue to contain either 100 items or for 100ms to pass before my consumer is allowed to drain it.

My question is; can anyone think of any parts of the Guava library that does the above or would make implementing the above more straightforward than me creating a BlockingQueue implementation myself? I assume this would actually be a relatively common requirement for certain domains so perhaps the Guava team has already added something similar or some of the building blocks.

Thanks,

Remon.

--
--
guava-...@googlegroups.com
Project site: http://guava-libraries.googlecode.com
This group: http://groups.google.com/group/guava-discuss
 
This list is for general discussion.
To report an issue: http://code.google.com/p/guava-libraries/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "guava")
 
---
You received this message because you are subscribed to the Google Groups "guava-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guava-discus...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Louis Wasserman

[re...@exmachina.nl]

Hi,

I just discovered the Queues.drain/drainUninterruptably methods which roughly do what I needed. I assumed these would work the same as the JDK drainTo equivalents but after rereading the docs it seems the Guava versions actually block until the specified number of elements is available which is perfect.

My own implementation was indeed Semaphore based.

[Chris Povirk]

The closest thing I can think of is Queues.drain:

http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/collect/Queues.html#drain%28java.util.concurrent.BlockingQueue,%20java.util.Collection,%20int,%20long,%20java.util.concurrent.TimeUnit%29

I don't know how closely it fits your requirements. For example, if it
times out without seeing the requested number of elements, it still
removes the elements that were present. Perhaps more significantly,
while drain() provides a possible implementation of take(), it does
nothing for put(). (Well, "does nothing" isn't a very helpful
description. I'll put it this way: Because drain() removes elements as
they become available, it will immediately make that space available
for new elements to be put().) Both of these behaviors may or may not
be what you need.

[Dimitris Andreou]

Yup, this was designed exactly with batching use cases in mind, I'd be surprised if this was useful for anything else.

[re...@exmachina.nl]

It actually fits my needs perfectly. I ran into this method but failed to read the javadoc carefully. Thanks!