Announcing HotQueue: Redis-backed message queue for Python

[richardhenry]

Just wanted to quickly post about a Python library I've written that
allows you to use Redis as a message queue within Python applications.

Here's the docs and introduction:
http://richardhenry.github.com/hotqueue/

I'm planning to add a worker decorator by v0.2.0, so that you can turn
any function into a worker, and a way to consume from multiple queues
by v0.3.0.

Anyone here interested in this? Looking for feedback, feature requests
and bug reports. :)

Thanks!

Richard

[Josiah Carlson]

Queues in the Python standard library (multiprocessing.Queue and
Queue.Queue) use put/get for enqueue/dequeue. Not only are they
shorter, they are easier to type ;)

The double-ended queue in Python (collections.deque) uses
append/popleft (or pop(0) ) so as to be a mostly drop-in replacement
for lists. It also doesn't do blocking gets/puts. Going with put/get
is still probably a better idea.

Two questions:
1. Why use what you have written vs. using Redis lists directly?
2. Why use what you have written vs. using celery + carrot + ghettoq:
http://ask.github.com/celery/tutorials/otherqueues.html ?

If your queue can offer optional message acking with timeouts, then
your queue library could go head-to-head with systems like Amazon's
Simple Message Queue, RabbitMQ, ActiveMQ, 0MQ, etc. (of course in such
a situation, the library itself would be providing all of the brains,
vs. the server in these other systems, but your library could pull 95%
of the brains into a daemon, with the 5% being implemented for
multiple languages/runtimes easily). If you are not looking to be a
replacement for these systems, then don't worry. :)

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[richardhenry]

Our busiest queue is enqueuing and dequeuing around a thousand
messages/sec, and has 5 workers. Although I'm yet to do any
benchmarks, HotQueue been a pretty good drop-in replacement for
RabbitMQ, and we're running one less server (we were already using
Redis).

If you don't have to share a queue between programs written in
languages other than Python, then this might be a viable option for
you.

> 1. Why use what you have written vs. using Redis lists directly?

Really, that's all you're doing with HotQueue, only that
Queue.consume() is much nicer than while True:
redis.blpop('mykey' ...). Think of HotQueue as a helper class, not as
a bulky library.

> 2. Why use what you have written vs. using celery + carrot + ghettoq

I like Celery, but we wouldn't be using many of it's features. It
seemed unnecessary for us to install three packages to achieve
something that we accomplished in a helper class.

It's not suitable for everything, but really suitable for some
things. :)

Richard

[Josiah Carlson]

Richard,

I'm a fan of concise libraries with only the features you need. We'd
also used RabbitMQ in the past, and actually ran into catastrophic
crash bugs with it (the same ones that Reddit ran into, incidentally).

If you're happy with what it does, then I would encourage you to
ignore my previous suggestions. :)

- Josiah

[richardhenry]

Thanks. I will make the API methods a little more concise though. ;)

Richard