Vert.x 3.0: EventBus Wire Protocol/Gateway

[Weston Price]

Item 33 of the Vertx.3 project plan calls out a need for an EventBus wire protocol gateway. I am starting work on this and I would like to open it up for discussion in this thread. people  Currently the project repository is hosted on GitHub

https://github.com/vert-x3/vertx-eb-wire-protocol

I will be getting things setup to be in line with the other vertx-x3 projects in terms of source layout, maven pom, etc for right now, I'd like to use this thread as a way for people to chime in and let me know what you'd like to see with the protocol, suggestions etc. 

Obviously the first question would be the need for an external protocol, why bother? This one should be pretty easy. While Vert.x supports a variety of development languages, all of those are currently in-VM. If we want to open the doors a bit, allowing for external execution across a wire would make a lot of sense. Here you could envision Vert.x clients no longer being VM dependent or having to traverse another API (REST etc) for verticle communication as well as communicating with Vert.x in general. 

Obviously there can be a lot that goes into a protocol. My initial thoughts on requirements:

1) Protocol Definition

    Currently there is no such thing. Vert.x has it's own internal message format that we use across the wire between nodes in a cluster, but that's about it. Clearly a protocol, message format for external consumption would need to be defined. This of course can be as simple or complex as we can make it :-) I'd prefer to start with the former and address the latter as required.

2) Execution Environment

    Obviously something has to consume the protocol and do something with it. Here would be the runtime engine/layer that would accept protocol requests, do the work etc.

3) Deployment/Management/Configuration

    I'd like to see the external protocol as an 'opt-in' that doesn't come by default. It would be something that can be separately stopped/started/manage/configured independent of the Vertx core. While this may seem to be an obvious requirement, sometimes things have a tendency of 'bleeding into' existing code. 

4) Fun

   I'd like the conception, realization, development of the protocol/runtime to be fun and in which people show an active interest. 

That's pretty much it to start. I'm looking forward to getting things rolling as well as hearing any thoughts, comments etc. 

[Tim Fox]

On 18/10/14 19:30, Weston Price wrote:

Item 33 of the Vertx.3 project plan calls out a need for an EventBus wire protocol gateway. I am starting work on this and I would like to open it up for discussion in this thread. people  Currently the project repository is hosted on GitHub

https://github.com/vert-x3/vertx-eb-wire-protocol

I will be getting things setup to be in line with the other vertx-x3 projects in terms of source layout, maven pom, etc for right now, I'd like to use this thread as a way for people to chime in and let me know what you'd like to see with the protocol, suggestions etc. 

Obviously the first question would be the need for an external protocol, why bother? This one should be pretty easy. While Vert.x supports a variety of development languages, all of those are currently in-VM. If we want to open the doors a bit, allowing for external execution across a wire would make a lot of sense. Here you could envision Vert.x clients no longer being VM dependent or having to traverse another API (REST etc) for verticle communication as well as communicating with Vert.x in general. 

Obviously there can be a lot that goes into a protocol. My initial thoughts on requirements:

1) Protocol Definition

    Currently there is no such thing. Vert.x has it's own internal message format that we use across the wire between nodes in a cluster, but that's about it.

There is another way that Vert.x users sometimes use to talk to the Vert.x event bus from a non Vert.x environment - to use the Vert.x eventbusbridge and talk to it over websockets sending JSON just like eventbus.js does. This works but we could definitely do bette than this

Clearly a protocol, message format for external consumption would need to be defined. This of course can be as simple or complex as we can make it

I vote for simple :)

One basic thing to decide would be whether we go for a binary or text protocol, or maybe both?

:-) I'd prefer to start with the former and address the latter as required.

2) Execution Environment

    Obviously something has to consume the protocol and do something with it. Here would be the runtime engine/layer that would accept protocol requests, do the work etc.

I'm thinking like some kind of gateway server. Kind of like the event bus bridge.

3) Deployment/Management/Configuration

    I'd like to see the external protocol as an 'opt-in' that doesn't come by default. It would be something that can be separately stopped/started/manage/configured independent of the Vertx core. While this may seem to be an obvious requirement, sometimes things have a tendency of 'bleeding into' existing code.

Agreed. I see this living in the "ext" stack which covers the reqirement of not bleeding into core.

4) Fun

   I'd like the conception, realization, development of the protocol/runtime to be fun

Of course. Fun is mandatory and anyone not complying will be severely, and harshly, reprimanded.

and in which people show an active interest. 

That's pretty much it to start. I'm looking forward to getting things rolling as well as hearing any thoughts, comments etc. 

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Fox]

On 18/10/14 21:15, Tim Fox wrote:

On 18/10/14 19:30, Weston Price wrote:

Item 33 of the Vertx.3 project plan calls out a need for an EventBus wire protocol gateway. I am starting work on this and I would like to open it up for discussion in this thread. people  Currently the project repository is hosted on GitHub

https://github.com/vert-x3/vertx-eb-wire-protocol

I will be getting things setup to be in line with the other vertx-x3 projects in terms of source layout, maven pom, etc for right now, I'd like to use this thread as a way for people to chime in and let me know what you'd like to see with the protocol, suggestions etc. 

Obviously the first question would be the need for an external protocol, why bother? This one should be pretty easy. While Vert.x supports a variety of development languages, all of those are currently in-VM. If we want to open the doors a bit, allowing for external execution across a wire would make a lot of sense. Here you could envision Vert.x clients no longer being VM dependent or having to traverse another API (REST etc) for verticle communication as well as communicating with Vert.x in general. 

Obviously there can be a lot that goes into a protocol. My initial thoughts on requirements:

1) Protocol Definition

    Currently there is no such thing. Vert.x has it's own internal message format that we use across the wire between nodes in a cluster, but that's about it.

There is another way that Vert.x users sometimes use to talk to the Vert.x event bus from a non Vert.x environment - to use the Vert.x eventbusbridge and talk to it over websockets sending JSON just like eventbus.js does. This works but we could definitely do bette than this

Clearly a protocol, message format for external consumption would need to be defined. This of course can be as simple or complex as we can make it

I vote for simple :)

One basic thing to decide would be whether we go for a binary or text protocol, or maybe both?

Something to think about:

Binary protocols: Performant but harder to write clients -> less adoption
Text protocols: Slower but easier to write clients -> more adoption

Redis is a good example of this. They have a very simple text based protocol http://redis.io/topics/protocol
It might not be the fastest/most efficient thing in the world, but the simplicity means it's dead easy to write clients for it.
That's why they have more clients than just about any other project I've seen:

http://redis.io/clients


What we *don't* want is another AMQP ;)

[Alexander Lehmann]

Speaking of binary protocols, I read about this project http://spearal.io/ as a "substitute" for JSON.

[KevinPas]

I haven't used vertx a lot,  so I might not understand the philosophy behind the vertx event bus:

"Obviously the first question would be the need for an external protocol"

Don't the tcp servers and clients already handle this.

Two scenarios:

1)  LAN star network of 100 laptops or apps talking to each other via tcp.  Each may be on a different opsys, some with vertx, some not.  But one is connected to a vertx websocket via vertx event bus so that the browser/websocket can talk to any of the other laptops or apps.  And they can talk to that browser via the tcp connection.

2)  Network of 100 apps with 99 of them communicating via vertx event bus.  But one of them is listening to tcp so that the non vertx app can get/set info.  

[Jordan Halterman]

On Oct 18, 2014, at 1:21 PM, Tim Fox <timv...@gmail.com> wrote:

On 18/10/14 21:15, Tim Fox wrote:

On 18/10/14 19:30, Weston Price wrote:

Item 33 of the Vertx.3 project plan calls out a need for an EventBus wire protocol gateway. I am starting work on this and I would like to open it up for discussion in this thread. people  Currently the project repository is hosted on GitHub

https://github.com/vert-x3/vertx-eb-wire-protocol

I will be getting things setup to be in line with the other vertx-x3 projects in terms of source layout, maven pom, etc for right now, I'd like to use this thread as a way for people to chime in and let me know what you'd like to see with the protocol, suggestions etc. 

Obviously the first question would be the need for an external protocol, why bother? This one should be pretty easy. While Vert.x supports a variety of development languages, all of those are currently in-VM. If we want to open the doors a bit, allowing for external execution across a wire would make a lot of sense. Here you could envision Vert.x clients no longer being VM dependent or having to traverse another API (REST etc) for verticle communication as well as communicating with Vert.x in general. 

Obviously there can be a lot that goes into a protocol. My initial thoughts on requirements:

1) Protocol Definition

    Currently there is no such thing. Vert.x has it's own internal message format that we use across the wire between nodes in a cluster, but that's about it.

There is another way that Vert.x users sometimes use to talk to the Vert.x event bus from a non Vert.x environment - to use the Vert.x eventbusbridge and talk to it over websockets sending JSON just like eventbus.js does. This works but we could definitely do bette than this

Clearly a protocol, message format for external consumption would need to be defined. This of course can be as simple or complex as we can make it

I vote for simple :)

One basic thing to decide would be whether we go for a binary or text protocol, or maybe both?

Something to think about:

Binary protocols: Performant but harder to write clients -> less adoption
Text protocols: Slower but easier to write clients -> more adoption

Redis is a good example of this. They have a very simple text based protocol http://redis.io/topics/protocol
It might not be the fastest/most efficient thing in the world, but the simplicity means it's dead easy to write clients for it.
That's why they have more clients than just about any other project I've seen:

http://redis.io/clients


What we *don't* want is another AMQP ;)

This.

[Tim Fox]

On 18/10/14 23:32, Alexander Lehmann wrote:

Speaking of binary protocols, I read about this project http://spearal.io/ as a "substitute" for JSON.

I spent five minutes clicking through the links on that site trying to find some description or specification for the format, couldn't find one, and gave up.

[Alexander Lehmann]

http://architects.dzone.com/articles/beyond-json-introducing?page=0,3

For some reason there is no format spec linked on the project page, strange.

[田传武]

MQTT is a good example of binary protocol.

MQTT is a Client Server publish/subscribe messaging transport protocol. It is light weight, open, simple, and designed so as to be easy to implement.

[Tim Fox]

Someone already wrote an MQTT implementation for Vert.x :)

However, I'm not sure MQTT is the right fit for this... but then again maybe it is, and we don't need any special wire protocol...

[Weston Price]

+1

I've heard of and looked at MQTT a bit in a different context. I think it's an interesting idea and bears looking into as any general wire protocol that we could reuse to meet our needs without a lot of mapping etc. 

Of course, one of the goals of this I think should be supporting multiple protocols with a Vert.x protocol mapping but I consider that long term rather than an immediate need out of the gate. 

[Ivano Pagano]

Is Apache Avro too cumbersome?

it's open-source

It supports:

- binary + json encodings

- embedded schema definition (smoother protocol upgrade)

- compression codecs

- message frames

Client libraries available for:

- Java

- C

- C++

- C#

Ivano

[JC]

Based on your description, I can picture a eventbusbridge upgrade/replacement being more than adequate. Yet this possibility was always supported, so what's the new need now? Just advertising one single standard way? Beautifying the current eventbusbridge interface?

It would be different if you had a real world use case for _every_ module exposing directly this new interface - but this is a need which I cannot imagine at the moment (and opening a possible can of worms for opening bus to outside).

Cheers,
S

[Blake Beaupain]

Hey, I'm not sure how far along you are but I found this thread when looking for EventBus support across other languages (C++, C#, etc). We could totally use this. I highly recommend Protobuf for this for the following reasons:

• Simple, highly organized, and easy to use
• Extremely high performance in terms of both size (bandwidth) and encoding/decoding speed
• Officially supported for Java and C++, unofficially supported for dozens of other languages
  

What's the status of this? Do you need some help developing it?

Sincerely,

Blake Beaupain

Wildcard Software Inc.