AMQP bridge

[Tim Fox]

A couple of different people have expressed an interest in integrating vert.x with Rabbit so I hacked together a rough AMQP bridge busmod yesterday.

It's very much a prototype, but is functional.

Code is in branch amqp_bridge:

https://github.com/purplefox/vert.x/blob/amqp_bridge/src/main/java/org/vertx/java/busmods/amqp/AmqpBridge.java

Here's an example using it from JS (of course, you can use it from Groovy, Ruby or Java too)

https://github.com/purplefox/vert.x/blob/amqp_bridge/src/examples/javascript/amqp/amqp.js

Basically it's just a busmod http://vertx.io/wsver-0002/bus_mods_manual.html which sits on the event bus and listens for JSON messages.

It currently does two things:

1) When it receives a create-consumer message it creates a pubsub consumer on a "topic".

In AMQP that's a bit convoluted and involves creating a queue, binding it to an exchange then creating a consumer on the queue. But all that nasty stuff is hidden in the bridge so the user doesn't have to worry about it.

When it receives a message on the AMQP consumer it forwards it to the event bus handler specified in the message.

2) Sends a message on to AMQP world.

What's missing:

1) Consumers for message queues. Currently only works for pubsub since creates the queue each time

2) All the various message properties and headers

3) Binary message bodies - currently only supports strings as message bodies

3) Transactions - (personally I don't care about this too much). 

It shouldn't be too much work to bring the busmod to a production level.

Would anyone like to volunteer to take it and run with it? :)

[Mark Fisher]

Tim, as with Redis, there is a thorough and well-used implementation of AMQP support in a Spring project (Spring AMQP). I'm wondering if it makes sense to consider a general busmod for using any Spring Integration Channel Adapters so that you get all of the supported transports in one shot?

Regards,
Mark

[Tim Fox]

On Monday, April 9, 2012 1:53:45 PM UTC+1, markfisher wrote:

Tim, as with Redis, there is a thorough and well-used implementation of AMQP support in a Spring project (Spring AMQP). I'm wondering if it makes sense to consider a general busmod for using any Spring Integration Channel Adapters so that you get all of the supported transports in one shot?

Right. Isn't this the generic Spring event bus adapter I suggested to you guys in a thread some weeks ago?

[Mark Fisher]

On Mon, Apr 9, 2012 at 9:03 AM, Tim Fox <timv...@gmail.com> wrote:

On Monday, April 9, 2012 1:53:45 PM UTC+1, markfisher wrote:

Tim, as with Redis, there is a thorough and well-used implementation of AMQP support in a Spring project (Spring AMQP). I'm wondering if it makes sense to consider a general busmod for using any Spring Integration Channel Adapters so that you get all of the supported transports in one shot?

 

Right. Isn't this the generic Spring event bus adapter I suggested to you guys in a thread some weeks ago?

Perhaps, but that's not quite how I was reading your suggestion. That to me sounded like something that would be configured in a Spring app to be able to use Spring's ApplicationEvent model to send/receive from the vert.x Event Bus. I'm actually talking about the fact that we have support for a long list of transports that you seem to be building from scratch, and I'm wondering if you could use a common wrapping approach to bring all of those in as busmods - even for a user who doesn't necessarily use Spring themselves. I guess it's the inverse of what you were suggesting - in this case, using Spring Integration as a library while vert.x is the runtime.
 

[Tim Fox]

It's definitely something to look at. I do have a few concerns though

Don't want to pull in a huge list of Spring jar dependencies. This is a big turn off.

Ideally our integrations are non blocking - e.g. the redis client I wrote is 100% non blocking. This is pretty important for scalabilty. I believe the Spring redis client is blocking (?) (This is why I wrote it myself and didn't just use jedis)

Don't want to expose the AMQP api to vert.x users. IMHO amqp is a horrible monstrosity of an API and it should be hidden and sanitised behind a nicer simpler API.

[Tim Fox]

On Monday, April 9, 2012 2:26:44 PM UTC+1, Tim Fox wrote:

It's definitely something to look at. I do have a few concerns though

Don't want to pull in a huge list of Spring jar dependencies. This is a big turn off.

Ideally our integrations are non blocking - e.g. the redis client I wrote is 100% non blocking. This is pretty important for scalabilty. I believe the Spring redis client is blocking (?) (This is why I wrote it myself and didn't just use jedis)

Don't want to expose the AMQP api to vert.x users. IMHO amqp is a horrible monstrosity of an API and it should be hidden and sanitised behind a nicer simpler API.

Correction, AMQP is a protocol not an API. But the various client apis from different projects tend to reflect the complexity of the protocol.

[Mark Fisher]

On Mon, Apr 9, 2012 at 9:26 AM, Tim Fox <timv...@gmail.com> wrote:

It's definitely something to look at. I do have a few concerns though

Don't want to pull in a huge list of Spring jar dependencies. This is a big turn off.

I don't understand this one. We work hard to minimize our own dependencies, so for example using the "spring-integration-amqp" JAR will only transitively depend upon the necessary underlying libs. Why is it different (as in a "big turn off") than other dependencies that you do build upon?
 

Ideally our integrations are non blocking - e.g. the redis client I wrote is 100% non blocking. This is pretty important for scalabilty. I believe the Spring redis client is blocking (?) (This is why I wrote it myself and didn't just use jedis)

We have a MessageChannel abstraction that supports asynchronous sends along with a replyChannel for callbacks, and we have a AsyncMessagingTemplate that returns Futures for send-and-receive operations. Something non-blocking should be possible with those.
 

Don't want to expose the AMQP api to vert.x users. IMHO amqp is a horrible monstrosity of an API and it should be hidden and sanitised behind a nicer simpler API.

Spring AMQP provides a higher-level API (e.g. convertAndSend("someQueue", someObject)) over the AMQP model while still allowing all the necessary knobs to be tweaked by those who do want to work at that level.
 

[Tim Fox]

On 09/04/2012 14:38, Mark Fisher wrote:

On Mon, Apr 9, 2012 at 9:26 AM, Tim Fox <timv...@gmail.com> wrote:

It's definitely something to look at. I do have a few concerns though

Don't want to pull in a huge list of Spring jar dependencies. This is a big turn off.

I don't understand this one. We work hard to minimize our own dependencies, so for example using the "spring-integration-amqp" JAR will only transitively depend upon the necessary underlying libs. Why is it different (as in a "big turn off") than other dependencies that you do build upon?
 

Ideally our integrations are non blocking - e.g. the redis client I wrote is 100% non blocking. This is pretty important for scalabilty. I believe the Spring redis client is blocking (?) (This is why I wrote it myself and didn't just use jedis)

We have a MessageChannel abstraction that supports asynchronous sends along with a replyChannel for callbacks, and we have a AsyncMessagingTemplate that returns Futures for send-and-receive operations. Something non-blocking should be possible with those.

I don't know much about Spring redis, but does it use a 3rd party client or is it a native async client itself?

From the website it seems to be a wrapper over the actual redis client (jedis or whatever). Last time I looked all the java redis clients were blocking...

[Tim Fox]

Regarding amqp... one way of thinking about this: All busmods are integrated over the event bus, i.e. the "api" is sending and receiving JSON messages.

So, we wouldn't be exposing the Spring, or Rabbit client api directly to the vert.x user anyway. This means the Spring or Rabbit API becomes an implementation detail of the vert.x AMQP bridge.

Then the question comes down to, what does it add to use the Spring API in our bridge rather than the Rabbit client directly (or code our own non blocking client)?

If the Spring api is basically just a wrapper around the rabbit client then it doesn't really add much, apart from extra jars, afaict.

[Mark Fisher]

On Mon, Apr 9, 2012 at 11:16 AM, Tim Fox <timv...@gmail.com> wrote:

Regarding amqp... one way of thinking about this: All busmods are integrated over the event bus, i.e. the "api" is sending and receiving JSON messages.

So, we wouldn't be exposing the Spring, or Rabbit client api directly to the vert.x user anyway. This means the Spring or Rabbit API becomes an implementation detail of the vert.x AMQP bridge.

Then the question comes down to, what does it add to use the Spring API in our bridge rather than the Rabbit client directly (or code our own non blocking client)?

If the Spring api is basically just a wrapper around the rabbit client then it doesn't really add much, apart from extra jars, afaict.

To those of us who have spent considerable time building that library and supporting its use in some rather significant enterprise environments, it certainly seems like we've added more than "extra jars".

The only reason I suggested that you consider it is that you mentioned yours was "hacked together" and that it *could* be taken to "production level" if someone volunteers.
 

[Jon Brisbin]

If the Spring api is basically just a wrapper around the rabbit client then it doesn't really add much, apart from extra jars, afaict.

Are you meaning the Spring-AMQP "template" abstraction or the Spring Integration functionality that Mark was talking about? If the former, then you could make a case that it's simply a matter of the module author's preference as to which API style suits them and then the "it just adds dependencies and not functionality" argument could play here (personally I would argue that even the spring-amqp abstraction is worth using over the driver itself, which is why I made that choice in previous projects when it would have sufficed to use the plain driver). If the latter, then it opens a whole slew of functionality because Spring Integration has a bazillion adapters, transformers, and the like that could tie a simple JSON message into an entire workflow. I'm not trying to put words in Mark's mouth, but it seems to me he was referring to a much wider swath of functionality than that limited by the specificity of the Java AMQP client.

w.r.t. the asynchronicity (or lack thereof) of various drivers: I think one must be realistic and assume that the effort involved in re-writing perfectly functional and production-quality drivers to use non-blocking IO rather than the traditional blocking IO they currently use (whether Redis, AMQP, or name your JDBC driver) is only a good idea in the theoretical sense. Just as one example, the manpower needed to rewrite the AMQP driver from scratch to be 100% non-blocking would be quite expensive, without really getting much return on that investment. If the majority of enterprise customers started saying they really needed a non-blocking AMQP driver for specific operational reasons, I think the argument could be re-framed and priorities would obviously be shifted. But until that happens, I think it's realistic to expect any network drivers to be blocking and to provide user code with event-based callback functionality to adapt that driver to the evented gestalt.

Thanks!

Jon Brisbin
http://about.me/jonbrisbin

[Tim Fox]

On 09/04/2012 16:29, Mark Fisher wrote:

On Mon, Apr 9, 2012 at 11:16 AM, Tim Fox <timv...@gmail.com> wrote:

Regarding amqp... one way of thinking about this: All busmods are integrated over the event bus, i.e. the "api" is sending and receiving JSON messages.

So, we wouldn't be exposing the Spring, or Rabbit client api directly to the vert.x user anyway. This means the Spring or Rabbit API becomes an implementation detail of the vert.x AMQP bridge.

Then the question comes down to, what does it add to use the Spring API in our bridge rather than the Rabbit client directly (or code our own non blocking client)?

If the Spring api is basically just a wrapper around the rabbit client then it doesn't really add much, apart from extra jars, afaict.

To those of us who have spent considerable time building that library and supporting its use in some rather significant enterprise environments, it certainly seems like we've added more than "extra jars".

The only reason I suggested that you consider it is that you mentioned yours was "hacked together" and that it *could* be taken to "production level" if someone volunteers.

Mark, with respect, I think you're missing the point here.

AIUI the value in Spring-AMQP, Spring-Redis etc is in the API. This is the real benefit to the user, and I'm sure there is much value there. Your enterprise users will be using that Spring API directly and that's a big win for those users.

But there's a difference here. Here we're definitely not going to be exposing that Spring API to the vert.x user - we'd just be using it as an implementation detail, and providing our own API (the event bus). So all the goodness that Spring has added over and above the Rabbit API is not really relevant any more, since we're not exposing it anyway.

Then it comes down to which api that I (or whoever writes the AMQP busmod) feels more comfortable coding against. In such a case it probably makes sense to get as close to the protocol as possible, i.e. use the Rabbit client. (Actually... further down the road we will probably ditch the rabbit client too, since it's a blocking client, and write our own async client).

[Tim Fox]

On 09/04/2012 16:35, Jon Brisbin wrote:

---

If the Spring api is basically just a wrapper around the rabbit client then it doesn't really add much, apart from extra jars, afaict.

Are you meaning the Spring-AMQP "template" abstraction or the Spring Integration functionality that Mark was talking about? If the former, then you could make a case that it's simply a matter of the module author's preference as to which API style suits them and then the "it just adds dependencies and not functionality" argument could play here (personally I would argue that even the spring-amqp abstraction is worth using over the driver itself, which is why I made that choice in previous projects when it would have sufficed to use the plain driver). If the latter, then it opens a whole slew of functionality because Spring Integration has a bazillion adapters, transformers, and the like that could tie a simple JSON message into an entire workflow. I'm not trying to put words in Mark's mouth, but it seems to me he was referring to a much wider swath of functionality than that limited by the specificity of the Java AMQP client.

Agreed. And for things other than AMQP and Redis that probably will never be first class busmods in vert.x, having a generic Spring Integration adaptor seems to have value. That's if we can find some sensible way of mapping a generic spring integration module to the exchange of JSON messages.

w.r.t. the asynchronicity (or lack thereof) of various drivers: I think one must be realistic and assume that the effort involved in re-writing perfectly functional and production-quality drivers to use non-blocking IO rather than the traditional blocking IO they currently use (whether Redis, AMQP, or name your JDBC driver) is only a good idea in the theoretical sense. Just as one example, the manpower needed to rewrite the AMQP driver from scratch to be 100% non-blocking would be quite expensive, without really getting much return on that investment. If the majority of enterprise customers started saying they really needed a non-blocking AMQP driver for specific operational reasons, I think the argument could be re-framed and priorities would obviously be shifted. But until that happens, I think it's realistic to expect any network drivers to be blocking and to provide user code with event-based callback functionality to adapt that driver to the evented gestalt.

Sure, I'm not proposing that we rewrite our own async versions of all drivers. Vert.x worker verticles can cope with blocking apis so it's not a *huge* problem. Although it does make sense for those modules that we think will be heavily used. E.g. redis, and (maybe) AMQP. Actually the redis non blocking client is already written.

[Tim Fox]

On Monday, April 9, 2012 4:57:41 PM UTC+1, Tim Fox wrote:

On 09/04/2012 16:35, Jon Brisbin wrote:

---

If the Spring api is basically just a wrapper around the rabbit client then it doesn't really add much, apart from extra jars, afaict.

Are you meaning the Spring-AMQP "template" abstraction or the Spring Integration functionality that Mark was talking about? If the former, then you could make a case that it's simply a matter of the module author's preference as to which API style suits them and then the "it just adds dependencies and not functionality" argument could play here (personally I would argue that even the spring-amqp abstraction is worth using over the driver itself, which is why I made that choice in previous projects when it would have sufficed to use the plain driver). If the latter, then it opens a whole slew of functionality because Spring Integration has a bazillion adapters, transformers, and the like that could tie a simple JSON message into an entire workflow. I'm not trying to put words in Mark's mouth, but it seems to me he was referring to a much wider swath of functionality than that limited by the specificity of the Java AMQP client.

Agreed. And for things other than AMQP and Redis that probably will never be first class busmods in vert.x, having a generic Spring Integration adaptor seems to have value. That's if we can find some sensible way of mapping a generic spring integration module to the exchange of JSON messages.

I'm going to take a closer look at Spring Integration to see what form this could take. 

[Jon Brisbin]

AIUI the value in Spring-AMQP, Spring-Redis etc is in the API. This is the real benefit to the user, and I'm sure there is much value there. Your enterprise users will be using that Spring API directly and that's a big win for those users.

But there's a difference here. Here we're definitely not going to be exposing that Spring API to the vert.x user - we'd just be using it as an implementation detail, and providing our own API (the event bus). So all the goodness that Spring has added over and above the Rabbit API is not really relevant any more, since we're not exposing it anyway.

Then it comes down to which api that I (or whoever writes the AMQP busmod) feels more comfortable coding against. In such a case it probably makes sense to get as close to the protocol as possible, i.e. use the Rabbit client. (Actually... further down the road we will probably ditch the rabbit client too, since it's a blocking client, and write our own async client).

FWIW- I don't think it's quite the whole story to say that a busmod doesn't expose the functionality (or eccentricities) of the underlying implementation. If one used the AmqpTemplate rather than the driver itself, one would have different workflows for going about sending and receiving messages. That would inherently be exposed to the user of the busmod via what messages it responded to, the format of those messages, and so on because one wouldn't be dealing with the raw AMQP protocol itself, but an additional set of concerns and functionality which have, in my experience, proven to be useful additions to the core functionality. Since AMQP can be quite difficult to use (as you mentioned earlier), IMO it makes sense to at least see how adding additional functionality that's already written and available in production quality could be used in addition to the low-level functionality that some people (but probably not the majority of people) need.