Binary protocol support for Vertx communication

[Thimira Darshana]

Hi,

I am planing to move Vertx with my new project. I have encountered following issues while doing POC.

1.) Is there a way to do binary protocol communication using vert.x event bus something like using Thrift or Protobuf? ( This is a must  requirement for my project )
2.) If yes. What are the available resources or examples? (I went through each and every vert.x related projects in Github and couldn't find a good solution)
3.) If currently vert.x does not have binary protocol support with Thrift or Protobuf, what are the necessary steps to achieve this ?


Thanks

Thimz

[Jordan Halterman]

I think this question was asked and answered very recently. Currently you must provide your own serialization for the event bus. This is the case mostly because of Vert.x's polyglot nature. Serializing objects from all the various languages is more complicated than handling just Java objects. 

The event bus supports byte arrays and buffers, so it's simply a matter of handling serialization yourself which is a trivial task anyways. You can even just wrap the default event bus with your own implementation that supports serialization with whatever library you'd like. I have used and would recommend using Jackson since it ships with Vert.x already and since you can parse the resulting JSON in any Vert.x language if necessary.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[qsys]

Check this thread:
https://groups.google.com/forum/#!topic/vertx/pgbmgEgLgK0
In short, as mentioned by Jordan: do your own serialization and send as byte[].

Op woensdag 2 juli 2014 09:44:51 UTC+2 schreef Thimira Darshana:

[Thimira darshana]

Hi Guys,

Binary serialization and supporting Thrift or Protobuf are two things.
To support Thrift event bus should transfer Binary object arrays and additionally structures and IDL files etc with message .
I am worrying about how to pass all these without breaking vert.x existing Event Bus and modular communication.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/_pokDEnIRfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

 ..::Best Regards::..
Thimira Darshana Upatissa

[qsys]

I might misunderstand your question and I've never used protobuf or thrift so far, but checking the docs (protobuf), serializing an object would be

person.SerializeToOstream(&output);

If I get it right, &output is a kind of outputstream, which could be a ByteArrayOutputStream in java.

On the client side:

person.ParseFromIstream(&input);

With &intput being an inputstream, which could be a ByteArrayInputStream in java.

If you need other stuff, well, make sure it's available on the 'client' side (and vice versa), or send it (as a byte array?) and store it.

It looks like if it's pretty similar to sending java objects over the event bus (which makes vert.x a lot less polyglot, but it's possible): The sent java class must be available on both sides. So, while developing your application, make sure you import the same class library. This makes it possible to cast the objects to the right type on both sides. The protobuf case looks pretty similar to me, but again, I never used protobuf, or thrift.

Best of luck,
qsys

PS: I agree with Jordan: why using more complex protobuf/thrift when you you have simple and very well performing serialization mechanisms right there for you, like json, jsonarray, FST ? - Protobuf, Thrift, ..., they are actually 'just' serialization mechanisms, although one can be more complex than the other.



Op woensdag 2 juli 2014 12:59:36 UTC+2 schreef Thimira Darshana:

[Thimira darshana]

1. Firstly we need to communicate with some other modules which are using c++ and .net addition to java.
To communicate with those modules we need high performance binary protocol Trift/Protobuf.
Trift/Protobuf are the best option to achieve that with high performance.

2. Secondly we are exposing POJO's to external parties via API calls.
To do that communication also Trift/Protobuf are giving really good performance results with higher load.

[Tim Fox]

On 02/07/14 14:13, Thimira darshana wrote:

1. Firstly we need to communicate with some other modules which are using c++ and .net addition to java.

So I think you are talking about some non Vert.x code bridging *to* the event bus, not the Vert.x event bus being able to send objects in Thrift format....

You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[田传武]

Hi Tim, 

In Wiki Vert.x-3.0-plan, mentioned "Expose simple wire protocol for event bus", will you adopt MQTT or some existing specifications?

[Tim Fox]

On 02/07/14 15:43, 田传武 wrote:

Hi Tim, 

In Wiki Vert.x-3.0-plan, mentioned "Expose simple wire protocol for event bus", will you adopt MQTT or some existing specifications?

Not sure yet. But note this already possible using websockets and the event bus bridge (search on this google group, it's been discussed a few times before).

Note that this has got nothing to do with custom serialization formats on the event bus which others on this thread have been discussing.

[Jordan Halterman]

Clearly I misunderstood :-)

[Xiaochun Lu]

I've ported Apache Thrift to Vert.x, with Java and JavaScript implementation.

The transport supports EventBus, and you can use built-in binary/compact/json protocols.

Our project is using this EventBus transport, so far so good, it is somewhat battle tested!

Please see my post here: https://groups.google.com/forum/#!topic/vertx/0lY6UbW0sFQ

This project resides on GitHub: https://github.com/XiaochunLU/thrift-for-vertx

[Eranga Samararathna]

Hi Lu,

Great to hear. Definitely I'll have a look. I would suggest you to consider also https://github.com/facebook/swift  

With this library IDL file could be a Java class with set annotations. It has a set of tools including a code generator. 

Era.

[Thimira darshana]

Hi Lu,

Excellent . We will look at your Implementation.

--

You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/_pokDEnIRfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thameem Ansari]

Hi Tim,

I couldn't find support for Protobuf added to vertx 3.0. I am trying to create a tcp server which will receive data in protobuf format. I only see the option to use Buffer and ByteBuf. Parsing these data types creating an error. Is there any other way?

[Ian Andrews]

I use Protobuf with a Vert.x 3.0 TCP server all the time.  When a packet comes across the wire, you convert the Buffer into a byte array and pass that into the builder for the protobuf message that you are expecting.

Keep in mind that you aren't guaranteed that a single Buffer received by the TCP server handler is the whole protobuf message.  The protocol that I use gives me the message size in a header, so I know if I have the whole message already or if I have to cache the portion of the message that I have until I receive the entire message.

[Thameem Ansari]

Thanks Ian. I did the same but missed to identify the length. Is it possible to share some sample code on that area? I will appreciate it.

On Thu, Jul 16, 2015 at 6:40 AM, Ian Andrews <sirgan...@gmail.com> wrote:

I use Protobuf with a Vert.x 3.0 TCP server all the time.  When a packet comes across the wire, you convert the Buffer into a byte array and pass that into the builder for the protobuf message that you are expecting.

Keep in mind that you aren't guaranteed that a single Buffer received by the TCP server handler is the whole protobuf message.  The protocol that I use gives me the message size in a header, so I know if I have the whole message already or if I have to cache the portion of the message that I have until I receive the entire message.

--

You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/_pokDEnIRfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

Visit this group at http://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/91ef264a-cfe5-416b-b17b-220b2197ac8a%40googlegroups.com.

[Tim Fox]

On 16/07/15 14:40, Ian Andrews wrote:

I use Protobuf with a Vert.x 3.0 TCP server all the time.  When a packet comes across the wire, you convert the Buffer into a byte array and pass that into the builder for the protobuf message that you are expecting.

Keep in mind that you aren't guaranteed that a single Buffer received by the TCP server handler is the whole protobuf message.  The protocol that I use gives me the message size in a header, so I know if I have the whole message already or if I have to cache the portion of the message that I have until I receive the entire message.

RecordParser can do all that reassembly of message fragments for you.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at http://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/91ef264a-cfe5-416b-b17b-220b2197ac8a%40googlegroups.com.

[Tim Fox]

On 16/07/15 18:42, Tim Fox wrote:

On 16/07/15 14:40, Ian Andrews wrote:

I use Protobuf with a Vert.x 3.0 TCP server all the time.  When a packet comes across the wire, you convert the Buffer into a byte array and pass that into the builder for the protobuf message that you are expecting.

Keep in mind that you aren't guaranteed that a single Buffer received by the TCP server handler is the whole protobuf message.  The protocol that I use gives me the message size in a header, so I know if I have the whole message already or if I have to cache the portion of the message that I have until I receive the entire message.

RecordParser can do all that reassembly of message fragments for you.

http://vertx.io/docs/vertx-core/java/#_record_parser

[Ian Andrews]

I'd never heard of the RecordParser before.  That is really cool!

[Thameem Ansari]

Thanks for the Tip. In fact I was looking at it. 

My requirement is first 4bytes are network-endian integer length followed by protobuf message. 

If I call RecordParser.newFixed(4) should work?

On Thu, Jul 16, 2015 at 10:59 AM, Ian Andrews <sirgan...@gmail.com> wrote:

I'd never heard of the RecordParser before.  That is really cool!

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/_pokDEnIRfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

Visit this group at http://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/9d4d7ea2-ec4b-49c5-adcd-9fb6fccc621d%40googlegroups.com.

[jordan.h...@gmail.com]

You can do newFixed(4), read the integer, and then called fixedSizeMode(size) with the size of the next record. Then recursively fixedSizeMode(4) and so on.

You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at http://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAHLoCa151Et7uQg2BRPNaXsHxJ8-LhCKpnLxOUw2_3BS9g7wdg%40mail.gmail.com.

[Arnon Lauden]

Hi,

I came across this old thread a couple of weeks ago while looking for some code examples.

If you're looking for simple examples on Vert.x with Protocol buffers communication, please take a look at this repository:

https://github.com/laudena/VertX-ProtoBuf

• In the first example, two verticles are exchanging a message through the event-bus, using proto-buf protocol. 
• In another example, the receiver verticle accepts a communication-message that encapsulate one of two different messages types.
  

Arnon

On Thursday, July 16, 2015 at 8:10:27 PM UTC+3, Thameem Ansari wrote:Thanks Ian. I did the same but missed to identify the length. Is it possible to share some sample code on that area? I will appreciate it.