Kombu Questions
Stephen J Day
2. Is Kombu ready for beta usage or is it still in alpha? Are there
any features that aren't even close to implementation?
3. What problems with Carrot does Kombu address?
4. Is there a list of unsolvable issues/user complaints with Carrot
that can used to help shape some of the questions above?
I know you have probably stated these somewhere else, but I didn't see
anything on the mailing list. This is partially inspired by the
conversation under http://github.com/ask/celery/issues/issue/141.
Thank you,
Stephen
Ask Solem
On Sep 10, 2010, at 10:44 PM, Stephen J Day wrote:
> 1. What is the current road map for Carrot/Kombu?
Carrot is being discontinued, replaced by Kombu.
I will still fix bugs in Carrot, but will probably not introduce any new features.
Carrot has some really nasty design flaws, the most important one being that some
settings are shared between the exchange and the queue::
c = Consumer(queue="foo", exchange="foo", auto_delete=True)
here the Consumer declares both the exchange and the queue, and the auto_delete
attribute applies to both of them. There is no easy way to define only the queue to be auto_delete for example. Kombu solves this and much more.
Kombu is still under planning. There are two major things that needs to be done:
* Look at how Kombu can be used with async frameworks/clients like txamqp, eventlet, etc.
How can the API best accommodate both sync and async use?
* Look at what needs to be done to pika for it to become the default backend.
* Decide wether Kombu should use AMQP 1.0 terminology. If it's possible to translate
1.0 semantics to 0.8 then we should probably go for it.
>
> 2. Is Kombu ready for beta usage or is it still in alpha? Are there
> any features that aren't even close to implementation?
Kombu does everything Carrot does and more. It does have quite a lot
of unittests already. the only
thing it doesn't have is users, so I can't say how stable it is.
I would definitely recommend people to start using/looking at it.
One thing that is not tested enough is the BrokerConnection.ensure + .ensure_connection
methods. We need more users to get this right.
.ensure_connection ensures that a connection is established. If it can't connect it will
retry a configurable amount of times, with an increasing retry delay.
.ensure is like .ensure_connection but for AMQP methods::
def errback(exc, interval):
logger.error("Couldn't send message: %s. Retrying in %ss." % (
exc, interval))
insured = connection.ensure(producer.publish, errback=errback, max_retries=5)
insured(message, routing_key="foo")
There are nothing big missing in Kombu, there's only the points listed above.
Also the support for immediate/mandatory is started, but not finished.
Kombu is more complete than carrot, just not as well tested.
>
> 3. What problems with Carrot does Kombu address?
>
I did list the shared settings problem above.
There's also the inclusion of explicit channels. In carrot you can
pass backends around to e.g. use the same channel for two consumers,
but it's not that easy (and not even documented anywhere).
Also in Kombu the queue/exchange declarations are separate objects, so they can be used
in settings etc, without depending on an active AMQP channel.
The backend interface (now called transports) is also much simpler, and there's now something
called virtual transports.
Virtual transports are taken from ghettoq, and they make it simple to emulate AMQP semantics
for backends that doesn't support it. Like redis, databases, mongodb, etc, etc.
> 4. Is there a list of unsolvable issues/user complaints with Carrot
> that can used to help shape some of the questions above?
>
Most complaints are solved by the above. Mandatory/Immediate support is coming.
The most imporant question right now is how to support an txamqp(twisted) transport.
> I know you have probably stated these somewhere else, but I didn't see
> anything on the mailing list. This is partially inspired by the
> conversation under http://github.com/ask/celery/issues/issue/141.
>
I've talked about it in other settings, and on IRC of course, but I also have to
remember that talking to myself in the shower doesn't really include others,
so thanks for asking. :)
--
{Ask Solem,
+47 98435213 | twitter.com/asksol }.
Stephen Day
>
> * Look at how Kombu can be used with async frameworks/clients like txamqp, eventlet, etc.
> How can the API best accommodate both sync and async use?
a reasonable, portable asynchronous backend would look like.
Basically, all method calls to the channel (basic_get, basic_publish,
queue_declare, etc.) would buffer messages inside of the channel. For
methods that have a return, there would be a deferred like object that
get populated when the response is ready. This would be a local,
non-blocking operation. From here, the magic would be done in an io
function. In code, the usage might look like this:
defer0 = channel.queue_declare('foo')
defer1 = channel.basic_publish(Message('foo'))
channel.perform() # or drain_events or what have you
if defer0.ready():
print defer0.value
The value here is that the socket association with the given
connection could be installed into an epoll descriptor and monitored
for events, with perform as a callback. "perform" would drain the
buffer and populate any deferred objects (note that these aren't
"Twisted Deferreds"). "perform" could even take a boolean for whether
it should block or not, and even a timeout. (Perhaps, this could even
be the kernel of a new project I should get started on.)
The main goal would be to port this pattern to the Kombu api.
>
> * Look at what needs to be done to pika for it to become the default backend.
Is this really necessary? Pika's goal of providing an asynchronous
client is noble but the implementation is overblown and really doesn't
provide much over the barryp amqplib. It might be better to get barryp
to patch amqplib to give the necessary features or fork it and expose
the socket for kqueue/epoll.
> * Decide wether Kombu should use AMQP 1.0 terminology. If it's possible to translate
> 1.0 semantics to 0.8 then we should probably go for it.
I have not looked at what this actually encompasses, but I would err
on the side of being conservative. If you follow the main server
implementation (RabbitMQ), you are going to be relatively safe. There
are some notes here:
http://www.amqp.org/confluence/display/AMQP/0.x+compatibility+for+1.0+brokers
However, it looks like they have made some wild decisions here (moving
to "Node" and "Link"). A cursory look at the spec makes me think 1-0
is a completely different protocol. Its not completely apparent if
brokers like RabbitMQ will be able to support 1-0 without a complete
rewrite. If anyone has some insight into what these guys are smoking,
please chime in.
>
>>
>> 2. Is Kombu ready for beta usage or is it still in alpha? Are there
>> any features that aren't even close to implementation?
>
> Kombu does everything Carrot does and more. It does have quite a lot
> of unittests already. the only
> thing it doesn't have is users, so I can't say how stable it is.
>
> I would definitely recommend people to start using/looking at it.
>
> One thing that is not tested enough is the BrokerConnection.ensure + .ensure_connection
> methods. We need more users to get this right.
>
> .ensure_connection ensures that a connection is established. If it can't connect it will
> retry a configurable amount of times, with an increasing retry delay.
>
> .ensure is like .ensure_connection but for AMQP methods::
>
> def errback(exc, interval):
> logger.error("Couldn't send message: %s. Retrying in %ss." % (
> exc, interval))
>
> insured = connection.ensure(producer.publish, errback=errback, max_retries=5)
> insured(message, routing_key="foo")
Excellent command over the english language for variable naming here,
btw. I'll make sure to give Kombu a try and checkout these kinds of
features.
>> 3. What problems with Carrot does Kombu address?
>>
>
> I did list the shared settings problem above.
> There's also the inclusion of explicit channels. In carrot you can
> pass backends around to e.g. use the same channel for two consumers,
> but it's not that easy (and not even documented anywhere).
As an aside, I always felt that the carrot api too closely coupled the
backend and the consumers. For instance, if the connection went for
some reason, one had to rebuild all of the producers/consumers (this
isn't entirely true, but almost). It was often easier to let the
process die and then restart it when there are connection problems.
Pika does have some code to manage reconnection. but I don't know that
its ready yet.
Hope this all helps,
Stephen