How to implement a queue

[kristian]

I am wondering what best way is to implement a queue in Voldemort? A
typical example would be a site where users put items into a shopping
cart which is then submitted for processing at some later time. The
items in the cart could simply be stored in a single value with e.g.
the session id as the key, but it would be nice to also be able to
store the 'completed' carts in Voldemort. One obvious way would be to
store the list of keys for the carts under a single key, however, this
would quickly run into scalability problems I assume.

Another option would be to have N lists and have the client pick one
at random to add its cart key to and to process them iterate over the
lists to get all the carts. I am still concerned that adding an item
to a list would be slow since it requires reading the entire list,
adding the item and then putting it back, but if N is large this might
not be much of an issue. A large N would require more time to iterate
over all the lists, however, but perhaps that is not an issue in
practice.

Have anyone run into this problem before and have any advice to give?
Is Voldemort appropriate for this use case or should I be looking at
something else?

Thanks!

[Rob Adams]

If you want a queue where you're guaranteeing that each item is processed by exactly one node, then you're not going to have a lot of success with voldemort.  Such a data structure requires synchronization.  Your best bet is to rethink your problem and see if you can solve it without requiring coordination among nodes, or to augment Voldemort with a queuing service such as ActiveMQ or something like Zookeeper which was used by digg to solve a similar issue with Cassandra.  Note that the distributed queues will not scale as well as Voldemort itself, but often this is OK since usually this is a small part of your overall system.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.

[kristian]

Having each item be processed only once is not a requirement, I should
have been more clear on this point. The 'completed' carts in my above
example will be copied into a relational db and this copy operation is
idempotent. However, the relational db is not guaranteed to be
available at all times, but I do not want that to effect the service.
The idea is thus to store a queue in Voldemort and have an
asynchronous process read this queue and copy the data over to the
relational db at some frequency, re-trying at some later time if the
relational db is not available.

I am also hoping that the problem can be solved without using
additional software to keep complexity down.

> > project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>

[Geir Magnusson Jr.]

Why not use... a queue?

geir

> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.

[kristian]

Can you elaborate?

[Geir Magnusson Jr.]

I'm not really trying to be flip - but I always have an allergic reaction to people trying to make things that aren't queues into queues :)

There are lots of options that work really well, from any of a number of solid JMS brokers (OpenMQ, ActiveMQ...) to RabbitMQ to that thing Twitter did (Kestral?) to ....

If you're feeling really lucky, you could use SQS...

geir

[kristian]

Thanks for the suggestions, all good. However, I was hoping to avoid
having to set up another complicated infrastructure since Voldemort is
already scalable and more importantly replicated and highly available
out of the box. Perhaps other people have also tried to do this and
have some advice / experience to share.

[Tatu Saloranta]

On Sat, Mar 13, 2010 at 10:32 PM, kristian <kre...@gmail.com> wrote:
> Thanks for the suggestions, all good. However, I was hoping to avoid
> having to set up another complicated infrastructure since Voldemort is
> already scalable and more importantly replicated and highly available
> out of the box. Perhaps other people have also tried to do this and
> have some advice / experience to share.

Sorry to be blunt, but I can not see how anyone would suggest building
your own queue implementation on a distributed key value store. Right
tool for the job and all that.
Even implementing queue using, say, Oracle is bit foolish; but that's
much simpler than nosql k/v stores.

If complexity bothers you, perhaps consider Amazon's Simple Queue
Service (SQS); it is very simple, highly scalable and reliable?

-+ Tatu +-

[tom gogii]

Hi Kristian,

I highly recommend that you look into activemq 5.3

I have been able to use it with almost out of the box configuration -
swapped to the performance configuration in my production environment
with about 1000+ msg/s flowing throughout various queues in my
system.

Writing a consumer, and producer are trivial if you follow the
examples.

Tom