Important: the future of Bunny (a Ruby RabbitMQ client)

[Michael Klishin]

First, a little announcement: I released Bunny 0.8.0 to rubygems.org [1]. This release
is identical to 0.7.9 and will be used as a mark for the last release of Bunny that implements AMQP 0.8.

Now, to the fun part.

Bunny is a library with complicated history: it is old, very feature incomplete, not maintained, has odd API decisions, has
no documentation and is stuck at AMQP 0.8. In addition, it does not frame message bodies, which means with
future RabbitMQ versions it will not be possible to send messages larger than the size of one frame (typically 128K).

Over the years nobody took over Bunny's maintainership, resulting in a crazy number of forks each fixing an issue or
two in a very ad-hoc manner. Sometimes this involved copying some amqp gem code in a haste to support URI connections
for Heroku. Sometimes it involved reinventing half of what's already available in the amq-protocol gem just to get
array and map (attribute tables) serialization in message headers.

Needless to say, this is not a healthy situation for a library to be in.

At the same time, Bunny is popular and I can see why certain people, companies and users cannot move to Hot Bunnies (which is JRuby-only) or deal with EventMachine that amqp gem uses. There are popular PaaS providers that use
Bunny in their doc examples because of the simplicity. OpsCode Chef still seems to depend on Bunny. And the list goes on.

This cannot go on forever. At some point, projects that rely on Bunny with either get stuck with old RabbitMQ versions
or will simply break. Both options really suck.

So I decided to do to Bunny what we did to amqp gem in 2010 and 2011: rewrite it from scratch, make it feature complete
and keep it backwards compatible where possible, but within reason. It worked well for amqp gem and besides
Ruby 1.8.7-p249 users (amqp gem does not support that particular patch level because it has a serious bug that would
require rewriting half of the library to work around), all non-trivial codebases that use amqp gem eventually upgraded to 0.8.x and now 0.9.x with very little or no effort.

Some highlights of what I have in mind:

* Move to AMQP 0.9.1, use amq-protocol that is a standalone library with pretty good test coverage, no I/O parts (only pure functions) and which is based on rabbitmq-codegen, a tool that official clients suck as the Java one use.
* Rewrite Bunny's test suite which is, hm, not particularly complete
* Make Bunny as feature complete as amqp gem and Hot Bunnies
* Provide lower-level, non-OO (AMQP is not OO, as are most of network protocols) client
* Deprecate but not remove oddities current Bunny API has
* Avoid race conditions and other known fundamental problems Bunny has thanks to the way it implements network communication

I am not sure what to do about the docs yet. Porting rubyamqp.info guides for Bunny is definitely not as much work as it
took to write them but it almost certainly will take a month or more. I may or may not be motivated enough to do that.

The fun starts on July 1st when I go on a vacation.


A short FAQ for Bunny users:

** Who are you? Why should I trust you to make sweeping changes to Bunny? Our CTO wants to know!

I main amqp gem, Langohr (a Clojure client), co-maintain Hot Bunnies with Theo Hultberg and have been using and
digging in internals of the Java client on and off for a few years.

On top of that, I wrote all doc guides at http://rubyamqp.info/. Chris Duncan, the original author of Bunny, helped me with editing them.

In other words, I am pretty qualified to design and write an AMQP 0.9.1 client, in any language.
Possibly even more qualified than your CTO.


** Will my code break?

We will do our best to break as little as possible. It is doable, as amqp gem experience shows.


** Why do I need AMQP 0.9.1?

AMQP 0.8 may or may not be around after a few RabbitMQ releases. Regardless of the exact version, the
goal is to make Bunny feature complete and all reasonably well maintained RabbitMQ clients have moved on to
AMQP 0.9.1.

In addition, you will be able to use RabbitMQ extensions to AMQP 0.9.1. Some of them are very powerful.


** Do you even realize why Bunny users use Bunny?

Yes. Because of the simplicity. We will make sure to preserve this nice aspect of Bunny, don't worry.


** Will you use EventMachine?

No, absolutely not. I am as tired of jumping through the hoops to not block the event loop as you are.
We will use that old boring TCP sockets thing.




1. https://rubygems.org/gems/bunny/versions/0.8.0

MK

mic...@defprotocol.org

[Michael Klishin]

Michael Klishin:

* Move to AMQP 0.9.1, use amq-protocol that is a standalone library with pretty good test coverage, no I/O parts (only pure functions) and which is based on rabbitmq-codegen, a tool that official clients suck as the Java one use.

This should read "such as the Java one"

MK 

[Chris Duncan]

Hi Michael,

--

This is a very welcome announcement. I know that what you propose is not a trivial amount of work. Please let me know if there is anything that I can help you with.

[Michael Klishin]

Chris Duncan:

> Please let me know if there is anything that I can help you with.

Chris,

I am working on the I/O part (framing of incoming data) and trying to understand this method:

https://github.com/ruby-amqp/bunny/blob/master/lib/bunny/client.rb#L121-150

Client#channel is assigned to channel 0 (which is "special"). Then we have a case branch that checks if a frame is not on channel 0 but then pushes a frame to the channel 0's buffer and recursively calls itself.

I am a little puzzled by what's the intent here. Maybe there is a different code path for reading incoming data and
this method only handles initial connection negotiation stage?

My plan is to reuse most of amq-client's framing code, by the way. In amq-client, each connection keeps a separate buffer for each
channel and can detect whether we received a complete message, the clears the buffer and passes things on
to handlers.

MK

mic...@defprotocol.org

[Chris Duncan]

Hi Michael,

That case branch is left over from a naive attempt to handle
multi-channel message processing in the same Bunny client. I can't
recall why I thought that it might be useful. However, I'm not sure if
anybody ever tried to use the same Bunny client with multiple channels,
but I don't remember any particular interest in it. I'm pretty sure that
you can get rid of that code unless anybody else out there has any
objections.

The amq-client framing code would be a great enhancement.

Cheers,

Chris

[Michael Klishin]

2012/6/23 Chris Duncan <chr...@frugalit.co.uk>

That case branch is left over from a naive attempt to handle
multi-channel message processing in the same Bunny client. I can't
recall why I thought that it might be useful

That's more or less what I figured. amqp gem 0.6 had the same problem. We solved it by addign a default channel for each connection and implementing a way to open and work with any number of channels.
I am doing the same for Bunny, Bunny seems to rely on the single implicit channel less.

So far connection start/tuning/open, connection.close, channel.open, channel.close, queue.declare, queue.delete work. They are not the hard part but still require most of the networking part to be in place.

--
MK

[Chris Duncan]

Michael,

I think that it's worth saying that although Bunny is not feature complete with respect to AMQP and RabbitMQ, it has been able to communicate with brokers that implement the AMQP v0.9.1 spec for some time now. In Bunny versions prior to 0.8.0, the AMQP v0.8 spec is used by default, however, there is a constructor argument that can be used if the 0.9.1 spec is required. It looks like this -

Bunny.new(:spec => "0.9.1")

In fact, you can use any string value for the :spec argument that is not "0.8" to switch to v0.9.1.

Please note that the current Bunny v0.8.0 will ONLY communicate with AMQP v0.9.1 brokers and no longer provides the :spec constructor argument. If this causes anyone a problem because they are still using an AMQP v0.8 broker, then do not move to Bunny v0.8.0.

Regards,

Chris

[Michael Klishin]

A progress update, late Nov 2012:

https://groups.google.com/forum/?fromgroups=#!topic/ruby-amqp/7gxdN9xxeOE

MK

[Mark]

Glad to hear. Do you have have some sort of project management tool you are using to help organize the features and bugs? If so, perhaps we as a group could help out

--
Documentation guides: http://bit.ly/rubyamqp
Code examples: http://bit.ly/amq-gem-examples
API reference: http://bit.ly/mDm1JE
 
Drop by #rabbitmq on irc.freenode.net
Bug tracker: https://github.com/ruby-amqp/amqp/issues
 
Post to the group: ruby...@googlegroups.com | unsubscribe: ruby-amqp+...@googlegroups.com

[Michael Klishin]

2012/11/21 Mark <static....@gmail.com>

Glad to hear. Do you have have some sort of project management tool you are using to help organize the features and bugs? If so, perhaps we as a group could help out

It is too early for that right now but we will use GitHub issues.

--
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin