[rabbitmq-discuss] 2.0: The Future of Pika

[Gavin M. Roy]

I've been listening to a lot of the conversations at PyCon about asynchronous development. The basic sentiment I've picked up on is that Callback Passing Style (CPS) is not currently in favor in the Python community.  This in addition to the popular use of the BlockingConnection in Pika has lead me to think about how to plan Pika's future enhancements. After some conversation with Tony, I believe I have an outline that should appeal to Python developers while keeping Pika asynchronous at its core and retaining CPS. I think that CPS is very powerful and believe it's still very important to Pika's future.

After I release 0.9.5, I will start development on Pika 2.0 which will be an effort to create a more pythonic approach to using Pika while retaining the ability to use CSP and keeping it asynchronous at its core.

The roadmap for changes to Pika 2.0:

- Backwards incompatible change that drops Python 2.4, 2.5 support

- Adding Python 3 support

- Remove existing connection adapter system

- Implement new pattern for use, behavior based use focused on both Asynchronous callback passing style and "Pythonic" development.

  - Both behaviors available from the same API calling same classes and methods

  - Async:

    - Merge existing connections into one connection system with IOLoop override

      - Supporting internal IOLoop, tornado, twisted

  - Pythonic:

    - high-level blocking on synchronous AMQP commands

    - Generator for receiving messages from Basic.Publish

- API notation closer to AMQP spec for implementation.

- *.*Ok frames will only be passed back in CPS use.

  - Calling methods like queue.declare will return a success indicator and attributes returned in the Ok frame will be assigned to attributes of the class.

- basic.consume and basic.get will return a single object with a materialized view of the Method, Header and Body frames.

- Build in support for specific broker types and pure AMQP 0-9-1.

Here's an example of what I expect Pika 2.0 to look like for non-CSP use. Note this is more of an idea of how it will work for someone using Pika than a spec or actual code. 

    from pika.rabbitmq import Connection

    from pika import Basic

    from pika import Channel

    from pika import Exchange

    from pika import Queue

    

    from sys import exit

    

    # All the attributes can be passed in via constructor or assigned

    connection = Connection()

    connection.host = 'localhost'

    connection.port = 5762

    connection.user = 'guest'

    connection.pass = 'guest'

    connection.vhost = '/'

    

    # Not much new here

    try:

        connection.open()

    except pika.ConnectException as e:

        print "Could not connect: %s" % e

        sys.exit(0)

    

    # Channel construction outside of connection context, instead pass

    # the Connection in

    channel = Channel()

    try:

        channel.open(connection)

    except pika.TimeoutException as e:

        print "Could not open a channel: %s" % e

    except pika.ConnectionClosedException as e:

        print "Could not open a channel, the connection is closed" 

    

    # All the attributes can be passed in via constructor or assigned

    exchange = Exchange(channel)

    exchange.name = 'test'

    exchange.type = 'fanout'

    exchange.durable = True

    exchange.declare()

    

    # All the attributes can be passed in via constructor or assigned

    queue = Queue(channel)

    queue.name = 'my_queue'

    queue.auto_delete = False

    queue.durable = True

    queue.passive = False

    

    # Declare the queue and expect a bool

    if not queue.declare():

        raise Exception("Could not declare my queue")

    

    # Print info about the queue that was mapped automatically when

    # Queue.DeclareOk was received 

    print 'Queue "%s"' % queue.name

    print ' Depth     : ' % queue.message_count

    print ' Consumers : %i' % queue.consumer_count  

    

    # Bind the queue

    queue.bind(exchange=exchange, routing_key='test.my_queue')

    

    # A generator returns one object for a message

    for message in Basic.consume(my_channel, routing_key="test.my_queue"):

        print 'Delivery Tag   : %s' % message.delivery_tag

        print 'Channel        : %i' % message.channel

        print 'Body Size      : %i' % len(message.body)

        print 'Properties'

        print '  Content-Type : %s' % message.properties.content_type

        print '  Timestamp    : %s' % message.properties.timestamp

        print '  User Id      : %s' % message.properties.user_id

        print '  App Id       : %s' % message.properties.app_id

        print 'Body           : %s' % message.body

        

I am looking for feedback on this direction. Do these changes and the example make sense to existing Pika and RabbitMQ uses?  Would you change anything about this direction? What would you improve?

Regards,

Gavin

[Jason J. W. Williams]

I like the cleanness of the new style, and think the blocking adapter is the core of what Pika brings to the party. Whether you're using Eventlet, Tornado or Twisted, all of the async frameworks have their own idiosyncrasies that it seems to me make it difficult to have one unified adapter for async.

Also, the 0.9 series already broke backwards compatibility once. Doing it again with 2.0 I think would make folks already dependent on it circumspect about doing development with the 0.9 series. If the API is going to change again I would recommend that the old API be turned into a wrapper around the new approach, that way anyone developing with the 0.9 series has a clear upgrade path without their projects breaking. It'll allow folks to develop new projects on Pika today knowing they'll still work tomorrow, and that they can ease into the new style as it's available.

Just my thoughts. :)

-J

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Matthias Radestock]

Gavin M. Roy wrote:
> - Calling methods like queue.declare will return a success indicator
> and attributes returned in the Ok frame will be assigned to attributes
> of the class.

What do instances of classes represent? Looks like they'd have to
contain a mixture of attributes with completely different purposes. In
particular there is a conflation of attributes for AMQP entities and
AMQP commands.

For example, 'passive' is not an attribute of an AMQP queue; it's a flag
on the queue.declare command. And 'message_count' is only something you
get as a return of queue.declare, so accessing that attribute in any
other context is meaningless.

Matthias.

[Gavin M. Roy]

Thanks for the feedback, this is the type of conversation I need to have as I'm fleshing things out.

On Sun, Mar 13, 2011 at 6:30 AM, Matthias Radestock <matt...@rabbitmq.com> wrote:

What do instances of classes represent? Looks like they'd have to contain a mixture of attributes with completely different purposes.

Quite possibly, again the goal being simplification and making it more pythonic. If you have a Queue (within Rabbit and in a general sense), message-count and consumer-count are of course transient, but they are still an attribute of a queue.

The problem with this notion is the transient nature of the values. They are potentially invalid as soon as they are returned. Perhaps, while keeping a more pythonic idiom in mind, it might be better for synchronous command response syntax to be like:

     message_count, consumer_count = queue.declare(channel)

 

In particular there is a conflation of attributes for AMQP entities and AMQP commands.

That was the initial thought, where it adds value. Where it confuses, it should not be done.

For example, 'passive' is not an attribute of an AMQP queue; it's a flag on the queue.declare command.

Right, in this case, it's probably better to pass that as an argument in the method call.

 

And 'message_count' is only something you get as a return of queue.declare, so accessing that attribute in any other context is meaningless.

Per my point on this above, I absolutely agree here. I'm trying to walk a line of simplification while staying true to the protocol. I there there is a win in simplification, for example, in the aggregation of content from the multiple frames of Basic.Deliver and Basic.GetOk. I don't think a developer will be shorted in having one object containing all of the values of the three frames.

The major change for the driver, from my perspective, is the removal of auto-generated driver mixin rpc command methods to having a object model closer to what is seen in spec.py, with the AMQP methods accessed through their class directly.  Think Basic.ack(channel, delivery_tag) instead of channel.basic_ack(delivery_tag).

[Matthias Radestock]

Gavin,

Gavin M. Roy wrote:
> The major change for the driver, from my perspective, is the removal of
> auto-generated driver mixin rpc command methods to having a object model
> closer to what is seen in spec.py, with the AMQP methods accessed
> through their class directly. Think Basic.ack(channel, delivery_tag)
> instead of channel.basic_ack(delivery_tag).

It's a common misconception about AMQP that the spec's class/method
terminology implies some sort of sensible mapping to the corresponding
OO terms. It does not.

What state is associated with instances of the Basic class?

Or with instances of the Queue class?

When I have an instance of Queue in your proposed API, what does it
represent?

1) All queues of any name, in any vhost, in any broker
2) All queues of any name, in any vhost, in a particular broker
3) All queues of any name, in a particular vhost & broker
4) A queue of a specific name, in any vhost, in any broker
5) A queue of a specific name, in any vhost, in a particular broker
6) A queue of a specific name, in a particular vhost & broker
?

To me only (1) makes sense, which leaves instances stateless, so all you
are really doing is grouping a whole bunch of "static" methods under
different headings ("queue", "exchange", etc).

(6) would make some sense too. The Queue constructor would take the
channel and queue name.

Regards,

[james anderson]

On 2011-03-13, at 18:20 , Matthias Radestock wrote:

> Gavin,
>
> Gavin M. Roy wrote:
>> The major change for the driver, from my perspective, is the
>> removal of auto-generated driver mixin rpc command methods to
>> having a object model closer to what is seen in spec.py, with the
>> AMQP methods accessed through their class directly. Think
>> Basic.ack(channel, delivery_tag) instead of channel.basic_ack
>> (delivery_tag).
>
> It's a common misconception about AMQP that the spec's class/method
> terminology implies some sort of sensible mapping to the
> corresponding OO terms. It does not.
>
> What state is associated with instances of the Basic class?
>
> Or with instances of the Queue class?
>
> When I have an instance of Queue in your proposed API, what does it
> represent?
>
> 1) All queues of any name, in any vhost, in any broker
> 2) All queues of any name, in any vhost, in a particular broker
> 3) All queues of any name, in a particular vhost & broker
> 4) A queue of a specific name, in any vhost, in any broker
> 5) A queue of a specific name, in any vhost, in a particular broker
> 6) A queue of a specific name, in a particular vhost & broker?

> ?

any one of those.
it depends on which of its attributes have been constrained by
assigned values.
(1) is the class' interface (as you note below). most likely an
instance would correspond to (3) or (6), but the particulars would
depend on the application's control patterns.

>
> To me only (1) makes sense, which leaves instances stateless, so
> all you are really doing is grouping a whole bunch of "static"
> methods under different headings ("queue", "exchange", etc).
>
> (6) would make some sense too. The Queue constructor would take the
> channel and queue name.

_______________________________________________

[Marek Majkowski]

On Sat, Mar 12, 2011 at 23:12, Gavin M. Roy <g...@myyearbook.com> wrote:
[...]

> The basic sentiment I've picked up on is that
> Callback Passing Style (CPS) is not currently in favor in the Python
> community.

It never has been. That's why Twisted is still perceived as a separate
library (as opposed to, for example EventMachine for ruby, which
often is treated as a core part of the ruby framework).

> The roadmap for changes to Pika 2.0:

[...]

> - Remove existing connection adapter system
> - Implement new pattern for use, behavior based use focused on
> both Asynchronous callback passing style and "Pythonic" development.
>   - Both behaviors available from the same API calling same classes and
> methods
>   - Async:
>     - Merge existing connections into one connection system with IOLoop
> override
>       - Supporting internal IOLoop, tornado, twisted
>   - Pythonic:
>     - high-level blocking on synchronous AMQP commands
>     - Generator for receiving messages from Basic.Publish

That's a quite interesting approach. But non-trivial python
generators are hard to compose. Beware.

> - API notation closer to AMQP spec for implementation.

Sounds good.

> - *.*Ok frames will only be passed back in CPS use.

That means using 'no-wait' whenever possible.
But what if the thing will crash a channel
(trigger an amqp channel-error). How would you notify
a user that the channel can't be used any more?

For example: queue.unbind() with bad parameters may
trigger a channel-wide NotFound error.

[...]

> I am looking for feedback on this direction. Do these changes and the
> example make sense to existing Pika and RabbitMQ uses?  Would you change
> anything about this direction? What would you improve?

You may find this inspiring:
https://github.com/ask/kombu/blob/master/examples/complete_receive.py#L39
https://github.com/majek/puka/blob/master/examples/stress_amqp.py#L60-62

Cheers,
Marek

[Jakub Šťastný]

Just a technical note about Ruby and EventMachine, I would not say it's treated as a core part of the ruby framework. There are some libraries (not many though) which are EventMachine-only, like current AMQP gem, but it's just because it's hard to write library which can run asynchronously with EventMachine a well as synchronously without it. So it's probably the same as in Python with Twisted, I guess.

Jakub

http://www.flickr.com/photos/jakub-stastny

http://twitter.com/botanicus

[Gavin M. Roy]

On Sunday, March 13, 2011 at 1:20 PM, Matthias Radestock wrote:

Gavin,

Gavin M. Roy wrote:

The major change for the driver, from my perspective, is the removal of
auto-generated driver mixin rpc command methods to having a object model
closer to what is seen in spec.py, with the AMQP methods accessed
through their class directly. Think Basic.ack(channel, delivery_tag)
instead of channel.basic_ack(delivery_tag).

It's a common misconception about AMQP that the spec's class/method
terminology implies some sort of sensible mapping to the corresponding
OO terms. It does not.

Right, I am familiar with the differences.

What state is associated with instances of the Basic class?

In what I am proposing to change, there would be no state for Basic class. 

Or with instances of the Queue class?

6) A queue of a specific name, in a particular vhost & broker

More specifically a queue of a specific name on a specific channel.

(6) would make some sense too. The Queue constructor would take the
channel and queue name.

*nod*

I want to reduce the work involved for the developer in areas where we can denote some level of state.

This is obviously a big change from the current client which is one of the reasons I raise the concept here.

Regards,

Gavin

[Michael Klishin]

This is how queue instances work in Ruby's amqp gem and after 2+ years I can tell it works pretty well (causes no confusion

to developers). Just my 2¢.

2011/3/14 Gavin M. Roy <g...@myyearbook.com>

More specifically a queue of a specific name on a specific channel.

-- 
MK