ROS 2.0 Messages + Serialization

[Benjamin Charrow]

I'd like to start a new thread on the pros/cons of changing ROS' message / serialization layer.  As a lot of people have noticed, the current solution is good but not ideal and there are several open source libraries that attempt to solve the same problem.  The goal for this discussion is to gather feedback / use cases with the aim of creating a concrete specification for messages in ROS 2.0.

Obviously no library is perfect, and so we're going to have to weigh a bunch of different options.  To start off, here is a long and undoubtedly incomplete list of criteria to judge serialization libraries:

1. ease of use (e.g., defining new messages, manipulating message in code)

2. size of messages on wire and in memory

3. ability to evolve message definitions / backwards compatibility

4. speed of serializing / deserializing messages

5. data types supported (e.g., numeric types, strings, lists, dictionaries)

6. effort to maintain library and integrate with ROS

7. number of language bindings and ease of porting to new systems

8. minimum hardware specs (can it run on arm or other embedded devices)

9. size and healthiness of the project

10. reflection (in the CS sense)

11. default values

12. support for namespaces

13. time to compile generated messages

14. ease of providing compatibility with current ROS message definitions

At a high level, it seems like we can do one of two things.  One is to decide on The One True Serialization Layer that ROS uses.  If you don't speak it, you can't communicate with anything in ROS.  The other is to work out a way to support multiple serialization libraries.  One important thing to keep in mind when thinking about these two approaches is what the user level API will look like (e.g.., what type of object is given to callbacks that the typical user writes).

The clear advantage of choosing The One True Serialization Layer is simplicity.  Interoperability between nodes written by different people is made easier, and we can probably pick something that works for almost everyone.  The user level API will be similar to what ROS currently does: users are given direct access to whatever message type we select.  The problem with this approach is that the best serialization library today may not serve our needs in 3 years, and introducing major changes will be difficult.

The other high level choice is to make the serialization layer modular / pluggable so that it can support multiple serialization libraries.  This idea could take on a bunch of different forms.  One would be to change the concept of topics so that the type includes the encoding (e.g., topic /sensors/camera_img is of type protobuf:sensor_msgs/Image and topic /sensors/camera_img2 is of type rosmsg:/sensor_msgs/Image).  A slight variation of this would be to have nodes deal with the details of serialization so that when a subscriber connects to a publisher they negotiate the serialization format in addition to the transport protocol.  Nodes could support multiple formats (e.g., protobuf, msgpack, rosmsg etc.) and users could optionally specify which encoding they'd like for their particular application.

For the multiple serialization solutions, it would probably be a bad idea to directly expose the  fact that there are multiple message formats as that would require users to register a new callback for each message type.  This would place a higher burden on development, and hurt interoperability.  Instead, it seems like if we go the modular route we should define a general "ROS message API" that abstracts away the details of different libraries.  This API would be define what user level callbacks are given.  This approach would let users get the intrinsic advantages of a serialization library (e.g., that it encodes certain types of data very efficiently), but at the cost of hiding specific features because the API doesn't support it (e.g., not every serialization library supports a set datatype or versioning for messages).  I imagine that there would would also be an "abstraction penalty" in any API that supports conversion between multiple underlying message representations, however this penalty might be insignificant.

Given all of this, a couple of questions for people are:

1. What requirements do you have for a message library?

2. Would you prefer to have a single serialization approach or have support for multiple representations?

3. Have you run into any other fundamental problems with the way ROS currently serializes messages?

I’ve started a ROS NG wiki page: http://www.ros.org/wiki/sig/NextGenerationROS  Currently it has a link to this google group and a link to my own analysis of the features provided by various serialization libraries (e.g., protobuf, rosmsg, msgpack):

http://www.ros.org/wiki/sig/NextGenerationROS/MessageFormats

Cheers,

Ben

[Jack O'Quin]

Ben,

Thanks for a good, insightful statement of these issues. My comments and questions are in-line below...

On Wed, May 29, 2013 at 5:59 PM, Benjamin Charrow <bcha...@osrfoundation.org> wrote:

I'd like to start a new thread on the pros/cons of changing ROS' message / serialization layer.  As a lot of people have noticed, the current solution is good but not ideal and there are several open source libraries that attempt to solve the same problem.  The goal for this discussion is to gather feedback / use cases with the aim of creating a concrete specification for messages in ROS 2.0.

To start off, here is a long and undoubtedly incomplete list of criteria to judge serialization libraries:

 

1. reflection (in the CS sense)

I'm not sure what this means. Is it similar to "introspection", in the sense that programs like rostopic and rosbag can read an an arbitrary message and make sense of it?

 

At a high level, it seems like we can do one of two things.  One is to decide on The One True Serialization Layer that ROS uses.  If you don't speak it, you can't communicate with anything in ROS.  The other is to work out a way to support multiple serialization libraries.  One important thing to keep in mind when thinking about these two approaches is what the user level API will look like (e.g.., what type of object is given to callbacks that the typical user writes).

The big advantages of the one true serialization layer (TOTSL) are common IDL and wire-level representations. Since the main point of ROS is interoperability, they are really important.

I will speculate about a possible intermediate option: we might establish standard IDL and wire representations, but generate tailored serialization and deserialization for low-end embedded systems. 

I mention this only because people who know more than I do about real-time and embedded programming say they want Plain Old Data messages to avoid memory allocation and other overhead. Some of those systems don't include the C++ standard template library, so a pure C implementation would work better.

That approach seems difficult to reconcile with higher-level ROS messages like sensor_msgs/PointCloud2, however.

The clear advantage of choosing The One True Serialization Layer is simplicity.  Interoperability between nodes written by different people is made easier, and we can probably pick something that works for almost everyone.  The user level API will be similar to what ROS currently does: users are given direct access to whatever message type we select.  The problem with this approach is that the best serialization library today may not serve our needs in 3 years, and introducing major changes will be difficult.

Changing right now will also be difficult, of course.

 

The other high level choice is to make the serialization layer modular / pluggable so that it can support multiple serialization libraries.  This idea could take on a bunch of different forms.  One would be to change the concept of topics so that the type includes the encoding (e.g., topic /sensors/camera_img is of type protobuf:sensor_msgs/Image and topic /sensors/camera_img2 is of type rosmsg:/sensor_msgs/Image).  A slight variation of this would be to have nodes deal with the details of serialization so that when a subscriber connects to a publisher they negotiate the serialization format in addition to the transport protocol.  Nodes could support multiple formats (e.g., protobuf, msgpack, rosmsg etc.) and users could optionally specify which encoding they'd like for their particular application.

I'm keeping an open mind about it, but the complexity and overhead of that approach is quite unappealing at first impression. 

 

For the multiple serialization solutions, it would probably be a bad idea to directly expose the  fact that there are multiple message formats as that would require users to register a new callback for each message type.  This would place a higher burden on development, and hurt interoperability.  Instead, it seems like if we go the modular route we should define a general "ROS message API" that abstracts away the details of different libraries.  This API would be define what user level callbacks are given.  This approach would let users get the intrinsic advantages of a serialization library (e.g., that it encodes certain types of data very efficiently), but at the cost of hiding specific features because the API doesn't support it (e.g., not every serialization library supports a set datatype or versioning for messages).  I imagine that there would would also be an "abstraction penalty" in any API that supports conversion between multiple underlying message representations, however this penalty might be insignificant.

All this seems worse than what we have right now. 

 

Given all of this, a couple of questions for people are:

1. What requirements do you have for a message library?

2. Would you prefer to have a single serialization approach or have support for multiple representations?

3. Have you run into any other fundamental problems with the way ROS currently serializes messages?

 1. The low-end issues I mentioned above are the main requirements not met by the current implementation.

 2. I prefer TOTSL, if possible.

 3. While it has some limitations, my overall opinion of rosmsg is that it has held up rather well for the past several years. Performance is good. The expressive power of the IDL has proven adequate for a wide range of robotics applications. The ability to handle arbitrary messages in Python is powerful and very useful.

It remains to be seen whether the potential savings in maintenance effort for migrating to a well-supported 3rd-party implementation outweigh the overhead of conversion and maintaining a bridge for existing ROS packages. The community-wide conversion effort should be factored into that decision, not just the ROS core.

--
 joq

[Michael Gratton]

Hey,

On 30/05/13 08:59, Benjamin Charrow wrote:
>
> The clear advantage of choosing The One True Serialization Layer is
> simplicity. Interoperability between nodes written by different people
> is made easier, and we can probably pick something that works for almost
> everyone. The user level API will be similar to what ROS currently
> does: users are given direct access to whatever message type we select.
> The problem with this approach is that the best serialization library
> today may not serve our needs in 3 years, and introducing major changes
> will be difficult.

Another other downside if a third-party library is chosen is that ROS
will likely not have a large degree of control over it. Hence not having
a big say in things like the API, or when to change in some
non-backwards compatible way the internals, its wire format or its API.
This can be remedied in the same way as the second approach - by
defining an abstraction layer, the "ROS message API" over the top of it.
So since this might be desirable anyway, it is probably worth thinking
about.

> Given all of this, a couple of questions for people are:
>

> 1. What requirements do you have for a message library?

Portability, interoperability and API stability, mostly.

> 2. Would you prefer to have a single serialization approach or have
> support for multiple representations?

As long as the above is met, neither really.

> 3. Have you run into any other fundamental problems with the way ROS
> currently serializes messages?

Just the usual, i.e. having to copy and/or send large chunks of data -
e.g. Sending complete map updates from SLAM or sending high-resolution,
high-framerate camera images.

The former very much seems like something that should be handed with
better message design, i.e. not sending the entire occupancy grid every
update.

The latter, especially for tasks such as image processing, is probably
handled by simply avoiding using the network. However, since ROS's only
widely used composition model is via nodes, it would be really nice if
messages on the same system could be passed by shared memory at least.

I think that brings me to what I would like to seem most out of ROS 2.0:
Promoting in-process composition to be first-class along side of
composition via nodes-as-processes. I.e. No difference in the API and
being otherwise transparent when running.

//Mike

--
⊨ Michael Gratton, Percept Wrangler.
⚙ <http://mjog.vee.net/>

[Damon Kohler]

I think it's easy enough to separate the concept of serialization from
ROS. Take zeroMQ for example. They explicitly do not handle
serialization and just ferry binary data around.

The ROS community can decide on a one true serialization library. But,
the underlying ROS API should be ignorant of that. For example, I
agree with Jack that rosmsg be TOTSL. It's a good fit for messages
that don't change often and there's a lot of momentum behind it.

I also think it's easy enough not to rely on a particular transport
layer for ROS. As long as data goes from publisher to subscriber, ROS
shouldn't concern itself with how that happens.

In summary, MHO is that ROS should be a collection of well defined
TOTSL messages and a high level API that describes a bare-bones pubsub
system for arbitrary binary data (i.e. something that easily maps to
several existing transport layers). The rest should be left to ROS
client library developers.

--
Damon Kohler

Google Germany GmbH
Dienerstr. 12, 80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

[Cem Karan]

I think that instead of the one true serialization library (TOTSL), we should concentrate on the one true serialization format (TOTSF). If we have a library, and not a format, then we are going to be tied to limitations imposed by the first one or two languages are used to code up the library. For example, consider integers; if you code them up in C/C++, you'll naturally tend towards char, short, int, long, and long long, as well as the unsigned versions of them. You may also be tempted to use bit vectors that encode into one of those types, which brings up the problems of little-endian vs. big-endian. But if you're using Python, you won't have unsigned integer types, and if both the size and sign are important for some reason, you'll need to fake it somehow. That doesn't even address reading the data out of a bit vector. This is why I was suggesting MessagePack (http://msgpack.org/). It is just a serialization format, although there are libraries that implement it.

That said, MessagePack may not be the best serialization format. HDF5 (http://www.hdfgroup.org/HDF5/) is a much more powerful, if much heavier-weight option, as are many, many other serialization formats.

Perhaps what we should do is make a survey of what formats are available, and make a survey of what data needs to be encoded. For example, MessagePack's support for integer types is strongly influenced by C/C++, but as a Python programmer, I would say that we only need one integer type, and that the C/C++ library will need to have arbitrary precision support via some library (e.g., http://gmplib.org/). That said, MessagePack does support maps (dicts) and nil (NULL, None) which I consider important. Others may consider other types are important.

All that aside, there is one more aspect of this that I think we MUST consider; the semantics of messages. Right now, you can send the same message type on two different ROS topics, with two different meanings. As long as there is only one master, this isn't too big a problem; a single master suggests a single system. However, the moment we have multiple systems that need to communicate, we need to be able to guarantee that the meaning of messages don't shift when transmitted from one master to another. This is of moderate importance when there are multiple independent robots controlled by one organization, but if we want interoperability between robots invented by totally different organizations, we need something better. We could either have some kind of message type review board (I do NOT want to deal with that amount of red tape!), or we could develop a totally decentralized system. My suggestion is that every message type be given its own UUID as specified in http://www.ietf.org/rfc/rfc4122.txt (if you're running Linux or OS X, you should have access to the 'uuidgen' command that create ASCII formatted UUIDs). Anyone can define a new message type, but when they define a new type, they have to give it a new UUID. Every message will have its message type UUID encoded into it, which allows everyone to immediately determine what type of message they are dealing with when they decode it. If a robot doesn't know what the message type it, it can do something appropriate, including telling the programmer about the type. If the message type is documented online, then your favorite search engine should turn it up immediately (if you're curious, generate a UUID, and try searching for it; you shouldn't get any hits, unless you've created a webpage dedicated to that ID). My rough calculations suggest that even if everyone on the planet generated new message types at 1,000,000,000 new types per second, we would still have several million years before we can reasonably expect the first collision between UUIDs, which means that this method is pretty much fully decentralized and robust.

Any thoughts or comments on this?

Thanks,
Cem Karan

> --
> You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Benjamin Charrow]

On Wed, May 29, 2013 at 6:24 PM, Jack O'Quin <jack....@gmail.com> wrote:
>
> On Wed, May 29, 2013 at 5:59 PM, Benjamin Charrow <bcha...@osrfoundation.org> wrote:
>>
>> To start off, here is a long and undoubtedly incomplete list of criteria to judge serialization libraries:
>>
>> reflection (in the CS sense)
>
> I'm not sure what this means. Is it similar to "introspection", in the sense that programs like rostopic and rosbag can read an an arbitrary message and make sense of it?

Depends on what you mean by "by make sense of it" ;-)  As I understand things, ROS messages aren't self-describing and so given raw bytes alone rostopic and rosbag can't decode a message.  By reflection I meant the ability to programatically inspect and manipulate messages.  For example, being able to access the value of a field if you have a string that matches the field's name.  Java and Python have builtin support for this, but C++ does not.  You can add support for it, though; see protobufs:

https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.message

(third code block in the first section,"Same as the last block, but do it dynamically....")

> The big advantages of the one true serialization layer (TOTSL) are common IDL and wire-level representations. Since the main point of ROS is interoperability, they are really important.
>
> I will speculate about a possible intermediate option: we might establish standard IDL and wire representations, but generate tailored serialization and deserialization for low-end embedded systems.
>
> I mention this only because people who know more than I do about real-time and embedded programming say they want Plain Old Data messages to avoid memory allocation and other overhead. Some of those systems don't include the C++ standard template library, so a pure C implementation would work better.

Would these tailored solutions be substantially different than how ROS is currently structured (i.e., an IDL that 1) defines a wire format and 2) makes it possible to auto-generate parsing code)?  Do you mean have a library that generates parsing code aimed at embedded systems, or a more fundamental change?  

Not having much experience with embedded programming myself, I'd also be curious to learn whether ROS' wire format is currently slim enough for most applications.

Also, while I definitely agree that keeping messages simple is a good idea there have definitely been times when I've wished for a bit more in the ROS IDL.  For example, maps would be great and could simplify handling certain types of messages (e.g., sensor_msgs/JointState and diagnostic_msgs/KeyValue).

>> ...the best serialization library today may not serve our needs in 3 years, and introducing major changes will be difficult.

>
> Changing right now will also be difficult, of course.

No doubt about it.  :-)

Cheers,
Ben

[Bob Dean]

Ok, so i guess it is up to me  to chime in for the real-time/embedded crowd. Please  note  that these are my opinions and  others with embedded experience may have different answers. Also, I am a cpp guy and do not know squat about python. 

Let me get Ben's questions answered first:

1) support for key-value types and bitfields. bitfields can  be key for reducing message size and memory usage. 

    serialization should be binary. xml/json/bson are highly inefficient.

    Your list left off the ability to set value ranges. i.e. min and max value for a radian measurement This is a rare capability in IDLs.

2) multiple serialization methods as the infrastructure is key for future flexibility and expansion. 

    if multiple  however, the end user should select how their system is configured. the negotiation of  types makes sense. but development and debugging of a robot or team will be much easier if the dev team can choose 1 and enforce it regardless of where their packages came from.

    Protobuf, thrift, msgpack all look good to me. use of a 3rd party format increases interoperability. So just pick one to start with and prove out the pluggable infrastructure. If it is necessary to transition to another down the road, it will be easier this way.

3) problems with how ros serializes messages? yes. see below.

Thoughts on other peoples posts:

Cem mentioned a couple of important things: unique id's and versioning. I disagree on the use of UUIDs for message ids. To have repeatable UUID creation, name  based UUIDs need to be used which are just a hash of the namespace+name in the 128bit  UUID data format.  I use something similar in hashing the namespace+name into a 32bit unsigned int, and adding another 32bit uint holding the hashed IDL message definition as the version (in my case the IDL is xml).  32bits is large enough to hold the data with few collisions, but small enough to send with each message. why send 128bits when 64bits has more relevant information? It also has the advantage of being usable by switch statements on processors without 128bit int support (unless something happened with 128bit ints that i am unaware of).  Something as simple as using a switch vs cascading if statements can have a performance impact on small systems with limited speed & memory.

I investigated HDF5 for DAQ files (er like a rosbag) and found the api to be cumbersome when you want to serialize dynamic  length arrays, basically it requires you to allocate temporary arrays then serialize the temporary array.

Introspection: From what I can see, protobuf uses a separate definition structure which I assume is compiled into the message code which simplifies the process. If someone  is using C or C++ they most likely will be doing so for efficency, and reflection is not an efficient data access method. It may be good enough to let the development language handle it. Especially for C I highly doubt someone would use introspection. Also, supporting char/short and smaller types is important at the embedded level to reduce memory footprint, as would be bitfields. 

I agree with Damon on separating serialization from transport and from the user level. Industry is starting to implement comm plugins to their middleware layers. We have a version, I hear iRobot also has a version. This is possible if the data and serialization mechanisms are separate. One of my main criticisms of roscpp is that it is not modular at all. There is no reason the message generator could not output a serialization plugin which roscpp then loads. The issue of changing serialization formats then becomes a message generator problem. 

Realtime/embedded:

I think it will be important to define what the scope of ros 2.0 will be w.r.t  realtime/embedded. ROS1.0 specifically states that it does not support real-time operation. And that's ok. There are methods & techniques to handle moving data across the realtime boundary.  In my case,  I feel it is important  to understand  the  concerns as it may yield more efficient code for everyone as how the data is sent has implications upon the serialization approach.

The key is to minimize latency for message access. For realtime, it is an important in order to meet timing constraints. Embedded it is important as lower latency comes from shorter code paths which in turn means less cpu cycles and power usage. One basic method  for this is to never allocate memory after startup. Another is to avoid serialization whenever possible. For example, serialization of high def stereo images is expensive. I continue to run into people  who think GigE cameras are just cameras that have a GigE jack. In reality they require that much bandwidth per camera.

At GDRS we use RCSLIB, an open source comms package maintained by NIST. http://www.isd.mel.nist.gov/projects/rcslib/. Being designed for realtime, NML does nice things such as using shared memory for processes on the same system, using pre-allocated fixed sized buffers, and sending the data from these buffers to remote systems via the appropriate comm layer. All of this is transparent to the user, hidden behind standard  write/read calls (there are no callbacks). RCSLIB also constrains messages to be Plain Old Data.  This means that to send a point cloud between two processes is essentially a memcpy. This of course places an added design burden on the developer w.r.t message definition and how to lay out the communication channels. 

For example: using roscpp, and sending a stereo image pair has multiple memory allocations:

- allocate the message wrapper

- allocate 2 vectors of image data (assuming you use vector::reserve() and don't use push_back() to fill the vector)

- copy the message if nodelets are used

- iterate through the message to calculate its size, allocate  buffer  to hold serialized message

- allocate deserialization buffer on subscriber side

- allocate message wrapper

- allocate each vector of image data to hold de-serialized images

Sending the same message through RCSLIB is zero allocations.

These aspects of pre-allocating memory, PoD, and minimizing serialization have one very important side issue: the majority of existing ros messages were not designed with this in mind. To address this issue would be non-trivial: require a review & modification core ros messages, a codegenerator upgrade, nodehandle updates, updating all nodes which use these messages. Think how common ros::Header is, with its use of string.

[Bob Dean]

also note that embedded systems may be tiny, completely different world. 32bit processor if you are lucky, maybe with a couple MB of ram. The synapticon guys are using chips with <1Mb of RAM. their engineer was rather upset with the use of xmlrpc as his custom stripped down C implementation used a majority of the ram available to hold executable code. 

[Cem Karan]

My comments are inline below.

On May 31, 2013, at 2:15 AM, Bob Dean wrote:

> Ok, so i guess it is up to me to chime in for the real-time/embedded crowd. Please note that these are my opinions and others with embedded experience may have different answers. Also, I am a cpp guy and do not know squat about python.
>
> Let me get Ben's questions answered first:
> 1) support for key-value types and bitfields. bitfields can be key for reducing message size and memory usage.
> serialization should be binary. xml/json/bson are highly inefficient.
> Your list left off the ability to set value ranges. i.e. min and max value for a radian measurement This is a rare capability in IDLs.

Bob, you've just illustrated part of what we need to talk about; I hadn't even thought about value ranges, but clearly you and probably others need them.

However, you've also brought up physical types, which would be useful when dealing with robots. Which ones though? My vote is for SI units and SI derived units, but that still leaves a moderately long list to chose from:

http://en.wikipedia.org/wiki/SI_base_unit
http://en.wikipedia.org/wiki/SI_derived_unit

> 2) multiple serialization methods as the infrastructure is key for future flexibility and expansion.
> if multiple however, the end user should select how their system is configured. the negotiation of types makes sense. but development and debugging of a robot or team will be much easier if the dev team can choose 1 and enforce it regardless of where their packages came from.
> Protobuf, thrift, msgpack all look good to me. use of a 3rd party format increases interoperability. So just pick one to start with and prove out the pluggable infrastructure. If it is necessary to transition to another down the road, it will be easier this way.

Are you asking for multiple serialization types solely for debugging purposes? Could this be accomplished by having a debugger that understood the one true format and decoded it into a human readable format instead? I'm thinking of something like a wireshark plugin that can display the format as it flows past.

> 3) problems with how ros serializes messages? yes. see below.
>
> Thoughts on other peoples posts:
> Cem mentioned a couple of important things: unique id's and versioning. I disagree on the use of UUIDs for message ids. To have repeatable UUID creation, name based UUIDs need to be used which are just a hash of the namespace+name in the 128bit UUID data format. I use something similar in hashing the namespace+name into a 32bit unsigned int, and adding another 32bit uint holding the hashed IDL message definition as the version (in my case the IDL is xml). 32bits is large enough to hold the data with few collisions, but small enough to send with each message. why send 128bits when 64bits has more relevant information? It also has the advantage of being usable by switch statements on processors without 128bit int support (unless something happened with 128bit ints that i am unaware of). Something as simple as using a switch vs cascading if statements can have a performance impact on small systems with limited speed & memory.

One correction; I made a mistake when I wrote my last email, UUIDs are not 40 bytes long, they are 16 bytes long. Thank you Bob for catching that.

I strongly disagree with the above approach. As programmers, we're trained to come up with reasonable, rational variable names that other programmers can guess the meaning of as soon as they look at it. Thus, if I were to create a message type for the pioneer, I might use the namespace 'pioneer'. To make it move, I might create a 'velocity' message, with 'angle' and 'speed' fields. How many other people will come up with similar names and namespaces? Maybe a better namespace would be to use your company name, but what happens if your company is very large? You might have a conflict within the company which requires some work to straighten out. For virtually any scheme that we come up with that involves human-comprehensible names, we significantly increase the likelihood of collisions, simply because we ARE human, and we want to communicate clearly with other people. If we have multiple library vendors writing for the Pioneer, each of which need to interoperate, then these conflicts would need to be settled, otherwise the various robots would become confused as they each misunderstand what others are saying. Moreover, if a message type is ambiguous, as the one above is, a human needs to get in there and figure out what the actual meaning really is. For example, with the above message type, what does 'angle' means? Compass angle? Relative angle? Degrees or radians? Etc. There is probably documentation out there explaining all of this for the given message type, but FINDING said documentation could be quite difficult; if I've received a library as part of a closed source package, it might not be well documented. My only hope might be wireshark, and my favorite search engine. If I know I'm looking at a ROS message, and I know that ROS messages always encode the type into the message, I might search for that type online. If there is a namespace collision, then the hashes will collide, and we might not get anything out of internet. Contrast this with using true UUIDs; we could write a wireshark plugin that automatically searches the web for the given UUID, bringing up wikis, mailing lists, forums, etc., that have all discussed that particular UUID. There is virtually no chance of accidental collision, unlike hashing human-comprehensible names. As a quick test, I generated 5 UUIDs using uuidgen, and searched for them on google. There were no hits for any of them. However, I've just generated the following uuid:

3DE366BC-2AFD-4E41-A517-0C8FBD3F02E7

Use your favorite search engine, and see how many hits you get for it. Now imagine every message type was tagged with its own UUID, and you could search that fast for any documentation on it.

As for handling 128 bits, I'm not sure how others would handle it, but in C/C++ I would decode it to a packed struct of whatever is the native unsigned int size for the platform I'm using. After all, we don't care about integer math for the UUID; the only operator you really need for it is equality. A switch tree would let you cascade through the entire UUID quickly; on a 32 bit system, you'd know which code path to take in 4 clock cycles. Once you're on the right code path, you'd throw away the UUID, which would save some storage as well.

> I investigated HDF5 for DAQ files (er like a rosbag) and found the api to be cumbersome when you want to serialize dynamic length arrays, basically it requires you to allocate temporary arrays then serialize the temporary array.
>
> Introspection: From what I can see, protobuf uses a separate definition structure which I assume is compiled into the message code which simplifies the process. If someone is using C or C++ they most likely will be doing so for efficency, and reflection is not an efficient data access method. It may be good enough to let the development language handle it. Especially for C I highly doubt someone would use introspection. Also, supporting char/short and smaller types is important at the embedded level to reduce memory footprint, as would be bitfields.

I agree that full introspection probably isn't necessary. Once we have the basic types decoded, we won't get too much out of introspection anyways; we're going to need more in-depth documentation to figure out the meanings of the fields anyways, which is where giving each message type a unique ID comes in.

> I agree with Damon on separating serialization from transport and from the user level. Industry is starting to implement comm plugins to their middleware layers. We have a version, I hear iRobot also has a version. This is possible if the data and serialization mechanisms are separate. One of my main criticisms of roscpp is that it is not modular at all. There is no reason the message generator could not output a serialization plugin which roscpp then loads. The issue of changing serialization formats then becomes a message generator problem.

I agree. It would help with debugging as well, and might make bagging data simpler too.

Thanks,
Cem Karan

[Robert Dean]

I suggest a single core binary serialization format, however, it should be implemented as plugins. Then in the future if a new format arises which is a better solution, it can be added in without code rewrite. 

The alternative to uuids I proposed is two parts: id and version. It is the combination of the two which provides the requested protections and searchability for the majority of cases. Similar in use to the current ros md5 sum, but in less space. I have generated all of our types as well as converted all the rosmsg files  in the full ros install to our IDL with zero collisions. If there is a collision within a company, then that is a failure of the company's CMMI process. I think md5 and UUID fall into the category of over-optimization.

"Thus, if I were to create a message type for the pioneer, I might use the namespace 'pioneer'.  To make it move, I might create a 'velocity' message, with 'angle' and 'speed' fields.  How many other people will come up with similar names and namespaces?"

This is why message standardization and use of common data types is important. As well as package documentation and shipping the IDL files with packages, be it open or closed source. Moreover, the ros transport itself could be configured to send or not send the IDL itself. Currently I believe it is sent during the negotiation process, realtime/embedded uses may want the option to turn that off to reduce overhead. 

[Jonathan Bohren]

On Fri, May 31, 2013 at 2:15 AM, Bob Dean <bob....@gmail.com> wrote:

Ok, so i guess it is up to me  to chime in for the real-time/embedded crowd.

Yeah, I think if we're looking to broaden the capabilities of the ros message architecture there are a lot of things in this space that could be done.

For what I do, what's really important is integration with real-time frameworks like orocos. Right now, ros messages need to be converted with orocos typekits for interoperability. I'm not sure if there's something fundamental that could make that easier, but if there is, it should be something worth looking into.

--
Jonathan Bohren
Laboratory for Computational Sensing and Robotics
http://dscl.lcsr.jhu.edu/People/JonathanBohren

[Jack O'Quin]

On Fri, May 31, 2013 at 7:58 AM, Cem Karan <cfka...@gmail.com> wrote:

However, you've also brought up physical types, which would be useful when dealing with robots.  Which ones though?  My vote is for SI units and SI derived units, but that still leaves a moderately long list to chose from:

http://en.wikipedia.org/wiki/SI_base_unit
http://en.wikipedia.org/wiki/SI_derived_unit

 

There is a well-established ROS standard for this, at the message definition level:

  http://ros.org/reps/rep-0103.html

The semantic meaning of the numbers sent is important, but separate from serialization, right?

--
 joq

[Eric Perko]

On Fri, May 31, 2013 at 2:15 AM, Bob Dean <bob....@gmail.com> wrote:

Ok, so i guess it is up to me  to chime in for the real-time/embedded crowd. Please  note  that these are my opinions and  others with embedded experience may have different answers. Also, I am a cpp guy and do not know squat about python. 

Let me get Ben's questions answered first:

1) support for key-value types and bitfields. bitfields can  be key for reducing message size and memory usage. 

    serialization should be binary. xml/json/bson are highly inefficient.

    Your list left off the ability to set value ranges. i.e. min and max value for a radian measurement This is a rare capability in IDLs.

2) multiple serialization methods as the infrastructure is key for future flexibility and expansion. 

    if multiple  however, the end user should select how their system is configured. the negotiation of  types makes sense. but development and debugging of a robot or team will be much easier if the dev team can choose 1 and enforce it regardless of where their packages came from.

    Protobuf, thrift, msgpack all look good to me. use of a 3rd party format increases interoperability. So just pick one to start with and prove out the pluggable infrastructure. If it is necessary to transition to another down the road, it will be easier this way.

3) problems with how ros serializes messages? yes. see below.

Thoughts on other peoples posts:

Cem mentioned a couple of important things: unique id's and versioning. I disagree on the use of UUIDs for message ids. To have repeatable UUID creation, name  based UUIDs need to be used which are just a hash of the namespace+name in the 128bit  UUID data format.  I use something similar in hashing the namespace+name into a 32bit unsigned int, and adding another 32bit uint holding the hashed IDL message definition as the version (in my case the IDL is xml).  32bits is large enough to hold the data with few collisions, but small enough to send with each message. why send 128bits when 64bits has more relevant information? It also has the advantage of being usable by switch statements on processors without 128bit int support (unless something happened with 128bit ints that i am unaware of).  Something as simple as using a switch vs cascading if statements can have a performance impact on small systems with limited speed & memory.

I investigated HDF5 for DAQ files (er like a rosbag) and found the api to be cumbersome when you want to serialize dynamic  length arrays, basically it requires you to allocate temporary arrays then serialize the temporary array.

Introspection: From what I can see, protobuf uses a separate definition structure which I assume is compiled into the message code which simplifies the process. If someone  is using C or C++ they most likely will be doing so for efficency, and reflection is not an efficient data access method. It may be good enough to let the development language handle it. Especially for C I highly doubt someone would use introspection. Also, supporting char/short and smaller types is important at the embedded level to reduce memory footprint, as would be bitfields. 

Protocol Buffers do support reflection in C++, but whether reflection is used for data access depends on how you compiled your .proto files using the C++ generator. See https://developers.google.com/protocol-buffers/docs/proto#options "optimize_for" docs for details - basically if you optimize for speed it generates a bunch of specialized code for data access for each message, but if you optimize for code size it will use the reflection interface for data access.

- Eric

  

I agree with Damon on separating serialization from transport and from the user level. Industry is starting to implement comm plugins to their middleware layers. We have a version, I hear iRobot also has a version. This is possible if the data and serialization mechanisms are separate. One of my main criticisms of roscpp is that it is not modular at all. There is no reason the message generator could not output a serialization plugin which roscpp then loads. The issue of changing serialization formats then becomes a message generator problem. 

Realtime/embedded:

I think it will be important to define what the scope of ros 2.0 will be w.r.t  realtime/embedded. ROS1.0 specifically states that it does not support real-time operation. And that's ok. There are methods & techniques to handle moving data across the realtime boundary.  In my case,  I feel it is important  to understand  the  concerns as it may yield more efficient code for everyone as how the data is sent has implications upon the serialization approach.

The key is to minimize latency for message access. For realtime, it is an important in order to meet timing constraints. Embedded it is important as lower latency comes from shorter code paths which in turn means less cpu cycles and power usage. One basic method  for this is to never allocate memory after startup. Another is to avoid serialization whenever possible. For example, serialization of high def stereo images is expensive. I continue to run into people  who think GigE cameras are just cameras that have a GigE jack. In reality they require that much bandwidth per camera.

At GDRS we use RCSLIB, an open source comms package maintained by NIST. http://www.isd.mel.nist.gov/projects/rcslib/. Being designed for realtime, NML does nice things such as using shared memory for processes on the same system, using pre-allocated fixed sized buffers, and sending the data from these buffers to remote systems via the appropriate comm layer. All of this is transparent to the user, hidden behind standard  write/read calls (there are no callbacks). RCSLIB also constrains messages to be Plain Old Data.  This means that to send a point cloud between two processes is essentially a memcpy. This of course places an added design burden on the developer w.r.t message definition and how to lay out the communication channels. 

For example: using roscpp, and sending a stereo image pair has multiple memory allocations:

- allocate the message wrapper

- allocate 2 vectors of image data (assuming you use vector::reserve() and don't use push_back() to fill the vector)

- copy the message if nodelets are used

- iterate through the message to calculate its size, allocate  buffer  to hold serialized message

- allocate deserialization buffer on subscriber side

- allocate message wrapper

- allocate each vector of image data to hold de-serialized images

Sending the same message through RCSLIB is zero allocations.

These aspects of pre-allocating memory, PoD, and minimizing serialization have one very important side issue: the majority of existing ros messages were not designed with this in mind. To address this issue would be non-trivial: require a review & modification core ros messages, a codegenerator upgrade, nodehandle updates, updating all nodes which use these messages. Think how common ros::Header is, with its use of string.

[Benjamin Charrow]

I definitely think that semantic meaning of fields is separate from serialization.

[Benjamin Charrow]

On Thu, May 30, 2013 at 11:15 PM, Bob Dean <bob....@gmail.com> wrote:

Let me get Ben's questions answered first:

1) support for key-value types and bitfields. bitfields can  be key for reducing message size and memory usage. 

What exactly would you like to see for bitfield support?  I guess I'm thinking that you can do the following in a  ROS message definition:

bitfield.msg:

  uint8_t opts;

  uint8_t READ = 1;

  uint8_t WRITE = 2;

  uint8_t FLUSH = 4;

and then in code have something like

if (bitfield_msg.opts & bitfield_msg.READ) {

  bitfield_msgs.opts |= bitfield_msg.FLUSH

  ...

}

But does this not cover what you're thinking of?

    Your list left off the ability to set value ranges. i.e. min and max value for a radian measurement This is a rare capability in IDLs.

This is an interesting idea.  Currently ROS doesn't have an explicit mechanism to "validate" or limit a field's value.  Do you have a pointer to an IDL that supports something like this?  What happens when your code tries to set a field to an invalid value?

REP 117 http://www.ros.org/reps/rep-0117.html seems like its covering a similar area.  Code that processes range messages like sensor_msgs/LaserScan is supposed to follow well specified application logic.  At first blush, REPs like this seem better than augmenting the IDL as the question of what you do when a message's field has bad data seems application -- not message type -- dependent.

  

I agree with Damon on separating serialization from transport and from the user level. Industry is starting to implement comm plugins to their middleware layers. We have a version, I hear iRobot also has a version. This is possible if the data and serialization mechanisms are separate. One of my main criticisms of roscpp is that it is not modular at all. There is no reason the message generator could not output a serialization plugin which roscpp then loads. The issue of changing serialization formats then becomes a message generator problem. 

The lack of modularity / separation in ROS' client libraries and specification definitely seems like a major them of what people want to change.

 

Realtime/embedded:

I think it will be important to define what the scope of ros 2.0 will be w.r.t  realtime/embedded. ROS1.0 specifically states that it does not support real-time operation. And that's ok. There are methods & techniques to handle moving data across the realtime boundary.  In my case,  I feel it is important  to understand  the  concerns as it may yield more efficient code for everyone as how the data is sent has implications upon the serialization approach.

+100 for making sure ROS has an explicit scope and a set of things that it's designed to be good at.

Cheers,

Ben

[Tully Foote]

On Fri, May 31, 2013 at 3:15 PM, Benjamin Charrow <bcha...@osrfoundation.org> wrote:

On Thu, May 30, 2013 at 11:15 PM, Bob Dean <bob....@gmail.com> wrote:

Let me get Ben's questions answered first:

1) support for key-value types and bitfields. bitfields can  be key for reducing message size and memory usage. 

What exactly would you like to see for bitfield support?  I guess I'm thinking that you can do the following in a  ROS message definition:

bitfield.msg:

  uint8_t opts;

  uint8_t READ = 1;

  uint8_t WRITE = 2;

  uint8_t FLUSH = 4;

and then in code have something like

if (bitfield_msg.opts & bitfield_msg.READ) {

  bitfield_msgs.opts |= bitfield_msg.FLUSH

  ...

}

But does this not cover what you're thinking of?

    Your list left off the ability to set value ranges. i.e. min and max value for a radian measurement This is a rare capability in IDLs.

This is an interesting idea.  Currently ROS doesn't have an explicit mechanism to "validate" or limit a field's value.  Do you have a pointer to an IDL that supports something like this?  What happens when your code tries to set a field to an invalid value?

REP 117 http://www.ros.org/reps/rep-0117.html seems like its covering a similar area.  Code that processes range messages like sensor_msgs/LaserScan is supposed to follow well specified application logic.  At first blush, REPs like this seem better than augmenting the IDL as the question of what you do when a message's field has bad data seems application -- not message type -- dependent.

There are very few cases where the datatype can set these ranges appropriately.  Usually it's specific to the current implementation.  As in the case of the LaserScan the min and max range depend on which scanner you are using. It cannot be hard coded into the message definition the min and max range.

One place this sort of thing is done regularly is in dynamic_reconfigure http://ros.org/wiki/dynamic_reconfigure/Tutorials/HowToWriteYourFirstCfgFile but again this is specific to the runtime of any particular node.  The only real use case I can see for this issue is radian measurements wrapping at 2 pi.  However even something as "generic" as a motor command may or may not be continuous and wrapping so hard coding that any specific angle is invalid if out of the range of 0-2pi is application specific. 

  

  

I agree with Damon on separating serialization from transport and from the user level. Industry is starting to implement comm plugins to their middleware layers. We have a version, I hear iRobot also has a version. This is possible if the data and serialization mechanisms are separate. One of my main criticisms of roscpp is that it is not modular at all. There is no reason the message generator could not output a serialization plugin which roscpp then loads. The issue of changing serialization formats then becomes a message generator problem. 

The lack of modularity / separation in ROS' client libraries and specification definitely seems like a major them of what people want to change.

 

Realtime/embedded:

I think it will be important to define what the scope of ros 2.0 will be w.r.t  realtime/embedded. ROS1.0 specifically states that it does not support real-time operation. And that's ok. There are methods & techniques to handle moving data across the realtime boundary.  In my case,  I feel it is important  to understand  the  concerns as it may yield more efficient code for everyone as how the data is sent has implications upon the serialization approach.

+100 for making sure ROS has an explicit scope and a set of things that it's designed to be good at.

Cheers,

Ben

--

[Bob Dean]

The example idl for both would be JAUS, which is now behind a pay wall.  I have an in house idl I can draw examples from. Hopefully iPhone typing will be legible.

Jays calls them "bit ranges". Our in house idl defines them as such:

<bitrange name="flags" type="unsigned char" >

<bitvalue name="changed" comment="set if object has changed"/>

<bitvalue name="reset" comment="if set, reset device"/>

<bitvalue name="controller id" size="4"/>

</bitrange>

Thus example has two single bit sized fields and one 4bit field. The message generator uses accessors for data access. Setters check range values if necessary and return false. To support bit fields in c++, the appropriate masks and shift operators are generated as are accessors for each but field. Such as 

bool TestMsg::flags() // full field value 

Bool TestMsg::flagsChanged() // get

Bool TestMsg::flagsChanged(bool newValue) // set

Bool TestMsg::flagsControllerId(unsigned char newValue) // set

unsigned char  TestMsg::flagsControllerId() // get

Before I forget, actual enums vs constants would be great. Enums allow extra bounds checking by the compiler, as well as additional namespace scoping of the values. Unlike a bit field enums only define the enum, and the class member is defined separately, allowing enums from one class to be used in another.

<Enum name="ResultEnum" comment ="example enums definition" >

<value name="OK" comment="successful operation"/>

<value name="MEMORY" comment="memory failure during operation"/>

<value name="NOT_FOUND" init="-15" comment="resource was not found"/>

</value>

<field name="result" type="TesultEnum" comment="a result value"/>

Enum values start at 0 and count up until an unit is seen. Values after the unit count up from there. 


Sent from my iPhone

--

[cfka...@gmail.com]

I agree about the plugin architecture and the single core format; plugins will also help with debugging as we can have a format designed for human readability.

I'm still very hesitant about the hashed type + version. My personal experience is that as humans, we tend to clump to certain words.  In addition, we reuse messages that are really different types.  For example, a standard message might have a boolean, and nothing else. The context that the message was sent in defines what the message actually means (e.g., the topic that the message was sent to, or the robot that sent it). Right now, we capture all this context implicitly; each system designer knows the topics he or she is using, the machine they are on, etc. But going across systems is harder. My concern is for larger systems and  for multimaster systems. If we give each type its own UUID, our probability of collision is vastly reduced.

As for good documentation, and failure of process, you're preaching to the choir! I could not agree with you more! Unfortunately, my experience hasn't been that good. I get code which is badly documented, and where the original author isn't responsive to questions; forums and google are my only hope. I want to increase the utility of both by making it easy to tell everyone what we're talking about.  UUIDs do that in a way that minimizes the probability of collision at the cost of 8 extra bytes.

Thanks,
Cem Karan

> On May 31, 2013, at 2:15 AM, Bob Dean wrote:
>
> > Ok, so i guess it is up to me  to chime in for the real-time/embedded
> crowd. Please  note  that these are my opinions and  others with embedded
> experience may have different answers. Also, I am a cpp guy and do not know
> squat about python.
> >

> > Let me get Ben's questions answered first:
> > 1) support for key-value types and bitfields. bitfields can  be key for
> reducing message size and memory usage.

> >     serialization should be binary. xml/json/bson are highly inefficient.

> >     Your list left off the ability to set value ranges. i.e. min and max
> value for a radian measurement This is a rare capability in IDLs.
>

> Bob, you've just illustrated part of what we need to talk about; I hadn't
> even thought about value ranges, but clearly you and probably others need
> them.
>

> However, you've also brought up physical types, which would be useful when
> dealing with robots.  Which ones though?  My vote is for SI units and SI
> derived units, but that still leaves a moderately long list to chose from:
>
> http://en.wikipedia.org/wiki/SI_base_unit
> http://en.wikipedia.org/wiki/SI_derived_unit
>

> > 2) multiple serialization methods as the infrastructure is key for
> future flexibility and expansion.
> >     if multiple  however, the end user should select how their system is
> configured. the negotiation of  types makes sense. but development and
> debugging of a robot or team will be much easier if the dev team can choose
> 1 and enforce it regardless of where their packages came from.
> >     Protobuf, thrift, msgpack all look good to me. use of a 3rd party
> format increases interoperability. So just pick one to start with and prove
> out the pluggable infrastructure. If it is necessary to transition to
> another down the road, it will be easier this way.
>

> Are you asking for multiple serialization types solely for debugging
> purposes? Could this be accomplished by having a debugger that understood
> the one true format and decoded it into a human readable format instead?
>  I'm thinking of something like a wireshark plugin that can display the
> format as it flows past.
>

> > 3) problems with how ros serializes messages? yes. see below.
> >
> > Thoughts on other peoples posts:
> > Cem mentioned a couple of important things: unique id's and versioning.
> I disagree on the use of UUIDs for message ids. To have repeatable UUID
> creation, name  based UUIDs need to be used which are just a hash of the
> namespace+name in the 128bit  UUID data format.  I use something similar in
> hashing the namespace+name into a 32bit unsigned int, and adding another
> 32bit uint holding the hashed IDL message definition as the version (in my
> case the IDL is xml).  32bits is large enough to hold the data with few
> collisions, but small enough to send with each message. why send 128bits
> when 64bits has more relevant information? It also has the advantage of
> being usable by switch statements on processors without 128bit int support
> (unless something happened with 128bit ints that i am unaware of).
>  Something as simple as using a switch vs cascading if statements can have
> a performance impact on small systems with limited speed & memory.
>

> > I investigated HDF5 for DAQ files (er like a rosbag) and found the api
> to be cumbersome when you want to serialize dynamic  length arrays,
> basically it requires you to allocate temporary arrays then serialize the
> temporary array.
> >
> > Introspection: From what I can see, protobuf uses a separate definition
> structure which I assume is compiled into the message code which simplifies
> the process. If someone  is using C or C++ they most likely will be doing
> so for efficency, and reflection is not an efficient data access method. It
> may be good enough to let the development language handle it. Especially
> for C I highly doubt someone would use introspection. Also, supporting
> char/short and smaller types is important at the embedded level to reduce
> memory footprint, as would be bitfields.
>

> I agree that full introspection probably isn't necessary.  Once we have
> the basic types decoded, we won't get too much out of introspection
> anyways; we're going to need more in-depth documentation to figure out the
> meanings of the fields anyways, which is where giving each message type a
> unique ID comes in.
>

> > I agree with Damon on separating serialization from transport and from
> the user level. Industry is starting to implement comm plugins to their
> middleware layers. We have a version, I hear iRobot also has a version.
> This is possible if the data and serialization mechanisms are separate. One
> of my main criticisms of roscpp is that it is not modular at all. There is
> no reason the message generator could not output a serialization plugin
> which roscpp then loads. The issue of changing serialization formats then
> becomes a message generator problem.
>

> I agree.  It would help with debugging as well, and might make bagging
> data simpler too.
>
> Thanks,
> Cem Karan

[cfka...@gmail.com]

Oh, perfect, I didn't know about that! Thank you!

As for being seperate from serialization, it really comes down to what we think of as being native types.  We tend to think of integers and floats as being native types for serialization, but we could have a data format that only understands byte arrays, with interpretation up to the programmer.  Going the other way, if physical types are a part of the format, then the library can enforce strong type rules to prevent certain errors (like what happened with that one Mars probe that crash landed because of a mistake in conversion from metric to standard; in a strongly typed system, the compiler prevents this)

Does that make sense?

Thanks,
Cem Karan

----- Reply message -----
From: "Jack O'Quin" <jack....@gmail.com>
To: <ros-sig...@googlegroups.com>
Subject: [ros-sig-ng-ros] Re: ROS 2.0 Messages + Serialization

Date: Fri, May 31, 2013 12:34 pm

On Fri, May 31, 2013 at 7:58 AM, Cem Karan <cfka...@gmail.com> wrote:

>
> However, you've also brought up physical types, which would be useful when
> dealing with robots.  Which ones though?  My vote is for SI units and SI
> derived units, but that still leaves a moderately long list to chose from:
>
> http://en.wikipedia.org/wiki/SI_base_unit
> http://en.wikipedia.org/wiki/SI_derived_unit

There is a well-established ROS standard for this, at the message
definition level:

 http://ros.org/reps/rep-0103.html

The semantic meaning of the numbers sent is important, but separate from
serialization, right?
--
joq

[cfka...@gmail.com]

If this is going to the list multiple times I apologize; the only email access I have right now is an older not-very-smart phone.

You're right, we don't need to include physical types into the serialization; we could even go as low as byte arrays. The only requirement is that we (as human beings) are able to decide what the type of the whole message is. After that, as long as there is good documentation on how to interpret the contents of the message, we can treat the bytes how we want to. That is why I keep hammering on UUIDs for each message type. Once we have that, the rest is useful but not critical. Physical types in the messages would mean that automatic checkers would prevent certain errors, but that is useful, not necessary.

Thanks,
Cem Karan

----- Reply message -----
From: "Benjamin Charrow" <bcha...@osrfoundation.org>
To: <ros-sig...@googlegroups.com>
Subject: [ros-sig-ng-ros] Re: ROS 2.0 Messages + Serialization

Date: Fri, May 31, 2013 5:40 pm


On Fri, May 31, 2013 at 9:34 AM, Jack O'Quin <jack....@gmail.com> wrote:

> On Fri, May 31, 2013 at 7:58 AM, Cem Karan <cfka...@gmail.com> wrote:
>
>> However, you've also brought up physical types, which would be useful
>> when dealing with robots.  Which ones though?  My vote is for SI units and
>> SI derived units, but that still leaves a moderately long list to chose
>> from:
>>
>> http://en.wikipedia.org/wiki/SI_base_unit
>> http://en.wikipedia.org/wiki/SI_derived_unit
>
>
> There is a well-established ROS standard for this, at the message
> definition level:
>
>   http://ros.org/reps/rep-0103.html
>
> The semantic meaning of the numbers sent is important, but separate from
> serialization, right?
> --
>  joq
>
>

I definitely think that semantic meaning of fields is separate from
serialization.

[Bob Dean]

Question: Do #1 "ease of use" and  #6 "effort to maintain library" include the concept of extending the library?

There are a couple of purposes:
1) defining serialization of existing hand-coded types (for example Time or Header in roscpp)
2) defining serialization for 3rd party library types. For example, an opencv image. (yes there is a ros image message, but direct serialization would skip copying the opencv image into a message in order to send it).
3) defining serialization for new sequence types i.e someone comes up with a new graph representation template.

For example, using msgpack since I have been through their source code:
#1 is easily handled by updating the roscpp headers with the MSGPACK_DEFINE macro.
#2 is a bit more difficult, as i think it would be required to use the C api, and create a serialize/deserialize function for each type (I could be wrong).
#3 Create a new header file containing the >> and << operators for the type, include it from your code. This header could be stored separately from the msgpack source,


also does the library have any limits? i think msgpack tops out at 32 members per class, unless a #define is changed.

[Benjamin Charrow]

On Fri, May 31, 2013 at 7:45 PM, Bob Dean <bob....@gmail.com> wrote:

The example idl for both would be JAUS, which is now behind a pay wall.  I have an in house idl I can draw examples from. Hopefully iPhone typing will be legible.

Jays calls them "bit ranges". Our in house idl defines them as such:

<bitrange name="flags" type="unsigned char" >

<bitvalue name="changed" comment="set if object has changed"/>

<bitvalue name="reset" comment="if set, reset device"/>

<bitvalue name="controller id" size="4"/>

</bitrange> 

 

Thus example has two single bit sized fields and one 4bit field. The message generator uses accessors for data access. Setters check range values if necessary and return false. To support bit fields in c++, the appropriate masks and shift operators are generated as are accessors for each but field. Such as 

bool TestMsg::flags() // full field value 

Bool TestMsg::flagsChanged() // get

Bool TestMsg::flagsChanged(bool newValue) // set

Bool TestMsg::flagsControllerId(unsigned char newValue) // set

unsigned char  TestMsg::flagsControllerId() // get

I spent some time looking at JAUS this weekend; this feature looks like it could definitely save some time when writing heavily space-constrained messages.  One thing worth noting is that it requires method based accessors.  Currently, both roscpp and rospy take a pretty struct oriented approach to messages.  Not sure how other people would feel about the change; I don't really have a preference.

[cfka...@gmail.com]

I vote for an accessor based approach. We'll have greater flexibility that way. That said, I don't know enough about highly constrained systems to know if this will work. Bob, do you have any opinions?

Thanks,
Cem Karan

----- Reply message -----
From: "Benjamin Charrow" <bcha...@osrfoundation.org>
To: <ros-sig...@googlegroups.com>
Subject: [ros-sig-ng-ros] Re: ROS 2.0 Messages + Serialization

[Bob Dean]

I am, in general, always a fan of accessors and mandate them whenever possible. But again, it depends on the language. C++ allows it, C# does it by default, I do not know about python.

The ability to add capability without changing an API is always a good thing.  For example, say a new serialization mechanism is chosen in the future which uses presence vectors to send a subset of information (such as JAUS or Protobuf).  By already having accessors in place, no change to the client code is necessary as the "dirty" bit is set as a side effect of the accessor.  Accessors also allow range checking of values on set operations.  Having a message generator do the heavy lifting takes care of most of the cons to an accessor based approach.

As to use on constrained systems it depends on the programming language. In C++, accessors should be inlined if possible and exist in multiple versions.  For example, to handle const vs non-const requirements of the stl interface. The compiler then is able to take care of the optimizations to make it "as fast as" a struct. Ex:

class Point
{
public:
    inline bool x(double x): // set, return false if range error
    inline double & x() {return x_;}; // get reference, for utility when the type is a another generated type or stl sequence
    inline const double x() {return x_;}; // get const value, necessary for some uses of stl sequence operations
    // etc...
}

if (pointMsg.x() > 1.57) ....

Opinion/rant/pet peeve: I do object strongly to using get/set in accessor names if it is otherwise obvious from the language syntax what is occurring. Using the Point class above, x() obviously returns the value x, and x(123) sets it. There is code out there where substantial portions of the text is either "get" or "set", which is ri-diculous. The api is also closer to the way "struct" version of the class would be used. (I will stop ranting now, but email me if you want to know why _ for member names is potentially more ri-dic-u-lous)

[Jonathan Bohren]

On Tue, Jun 4, 2013 at 9:40 AM, Bob Dean <bob....@gmail.com> wrote:

Opinion/rant/pet peeve: I do object strongly to using get/set in accessor names if it is otherwise obvious from the language syntax what is occurring. Using the Point class above, x() obviously returns the value x, and x(123) sets it. There is code out there where substantial portions of the text is either "get" or "set", which is ri-diculous. The api is also closer to the way "struct" version of the class would be used. (I will stop ranting now, but email me if you want to know why _ for member names is potentially more ri-dic-u-lous)

+1000

[Jack O'Quin]

On Tue, Jun 4, 2013 at 5:24 AM, cfka...@gmail.com <cfka...@gmail.com> wrote:

I vote for an accessor based approach. We'll have greater flexibility that way. That said, I don't know enough about highly constrained systems to know if this will work. Bob, do you have any opinions? 

-10 to making breaking changes to an API affecting every ROS program for non-compelling reasons. "Greater flexibility" is nice, but hardly worth all that effort.

Since we also want a Plain Old Data representation for low-end messages, requiring retro-fitting of accessors seems wrong.

Also, these messages need to be represented in several languages. Let's keep it simple and not get too bogged down in optimizing certain cases for one or two of them.

--
 joq

[Bob Dean]

On Tuesday, June 4, 2013 10:31:24 AM UTC-4, Jack O'Quin wrote:

-10 to making breaking changes to an API affecting every ROS program for non-compelling reasons. "Greater flexibility" is nice, but hardly worth all that effort.

Since we also want a Plain Old Data representation for low-end messages, requiring retro-fitting of accessors seems wrong.

Also, these messages need to be represented in several languages. Let's keep it simple and not get too bogged down in optimizing certain cases for one or two of them.

--
 joq

At some point a line needs to be drawn as to what is ROS1.0 and what is ROS2.0. The only guarantee about that line is that there will be API breaking changes in some fashion with ROS2.0, and a migration path would need to be defined as well as a mechanism to interoperate between the two (such as supporting rosmsg1.0 in ros 2.0).

POD support itself is a API breaking move, common ros messages such as Header and PointCloud2 are not POD. POD also does not negate usefulness accessors for C++, as C structs and C++ classes will have the same memory layout assuming certain requirements are met such as not using virtual methods or dynamic length arrays. 

The message generator for C and for C++ have different outputs, and can be tailored as needed to the needs of the language. On a client by client basis, the generator could add accessors or not. It could also have a hint in the IDL as to the style of the output code, this would allow "new style" messages to be generated with accessors and old to be generated without them, or generate both side by side.

The issue is mitigated by the use of the message generator, assuming the new IDL is flexible enough to provide hints to the generator on what to do.  For example, when ROS is converted to our in house IDL the "publicDataMembers" attribute is set to true, providing an api consistent with current ros C++ messages. In the future, however, we will have an in house ros client as it will our interoperation with ROS. When that time comes the conversion script will be modified to drop the attribute and enforce accessors, as it would be consistent with our software development policies.

[Dirk Thomas]

The focus of ROS 2.0 is to make the best design decisions possible.
API compatibility is not a must-have (that's also why it is called 2.0).
If a different API turns out to be the better choice we should do the change.

And I do agree with the previous posting: having a method based interface enables us to perform whatever mapping / logic / checking is necessary behind that API.
Having that ability without requiring to touch any user land code in the future is one of the big advantages of that approach.

- Dirk

[Jack O'Quin]

On Tue, Jun 4, 2013 at 11:21 AM, Bob Dean <bob....@gmail.com> wrote:

On Tuesday, June 4, 2013 10:31:24 AM UTC-4, Jack O'Quin wrote:

-10 to making breaking changes to an API affecting every ROS program for non-compelling reasons. "Greater flexibility" is nice, but hardly worth all that effort.

Since we also want a Plain Old Data representation for low-end messages, requiring retro-fitting of accessors seems wrong.

Also, these messages need to be represented in several languages. Let's keep it simple and not get too bogged down in optimizing certain cases for one or two of them.

At some point a line needs to be drawn as to what is ROS1.0 and what is ROS2.0. The only guarantee about that line is that there will be API breaking changes in some fashion with ROS2.0, and a migration path would need to be defined as well as a mechanism to interoperate between the two (such as supporting rosmsg1.0 in ros 2.0).

Perhaps, after careful consideration, some API-breaking changes will prove worthwhile. That does not mean we are starting with a "clean slate". Compatibility remains important, and any breakage needs to be justified.

So far, no one has come up with a convincing reason why we should not retain the existing ROS serialization and message definitions. Actually, it is starting to look pretty good. All the other options clearly have problems, too.

POD is one reason to make changes, but probably not compelling all by itself. 

 

POD support itself is a API breaking move, common ros messages such as Header and PointCloud2 are not POD. POD also does not negate usefulness accessors for C++, as C structs and C++ classes will have the same memory layout assuming certain requirements are met such as not using virtual methods or dynamic length arrays.  

It would be unreasonable to completely eliminate dynamic vectors from ROS messages. They are just too pervasively useful. Should we outlaw sensor_msgs/PointCloud2, just because small controllers can't handle it? I don't think so. What would we replace it with, fixed-size point clouds? What size? 

I fully support coming up with a reasonable subset of messages that can be represented via POD. And, that certainly requires finding a solution to the Header frame_id problem, but that issue is mostly independent of the serialization method chosen.

--
 joq

[Tully Foote]

From a CS/elegance side putting the data behind accessors provides great flexibility from the viewpoint of the framework.  It provides the flexibility to change the system at the cost of removing that flexibility from the user.  The user group whom this will most effect is those doing very high bandwidth communications where one of two copies are a significant cost. The use case of image processing and pipelining jumps to my mind as one of the most common use case in this realm.  

We need to keep in mind these tradeoffs in our design. 

Tully

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ros+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ros+unsubscribe@googlegroups.com.

[Jack O'Quin]

On Tue, Jun 4, 2013 at 1:28 PM, Tully Foote <tfo...@osrfoundation.org> wrote:

From a CS/elegance side putting the data behind accessors provides great flexibility from the viewpoint of the framework.  It provides the flexibility to change the system at the cost of removing that flexibility from the user.  The user group whom this will most effect is those doing very high bandwidth communications where one of two copies are a significant cost. The use case of image processing and pipelining jumps to my mind as one of the most common use case in this realm.  

We need to keep in mind these tradeoffs in our design. 

+1

And, don't forget that ROS messages are also data structures. They can usefully be imbedded within other data structures in ROS programs. There are lots of good reasons to do that.

Providing accessors for data fields is fine, but telling all existing packages they may only use accessors is almost impossible to justify. 

--
 joq

[Bob Dean]

I am not sure I understand your point Tully, unless it is my inexperience with python. in C++ the accessors are effectively optimized away or there are methods to construct them such that access is as fast as direct member access. No need to copy data...

[Thibault Kruse]

On Tuesday, June 4, 2013 3:40:31 PM UTC+2, Bob Dean wrote:

Opinion/rant/pet peeve: I do object strongly to using get/set in accessor names if it is otherwise obvious from the language syntax what is occurring. Using the Point class above, x() obviously returns the value x, and x(123) sets it. There is code out there where substantial portions of the text is either "get" or "set", which is ri-diculous. The api is also closer to the way "struct" version of the class would be used. (I will stop ranting now, but email me if you want to know why _ for member names is potentially more ri-dic-u-lous)

See the google styleguide:
http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Access_Control

[Tully Foote]

On Tue, Jun 4, 2013 at 12:15 PM, Bob Dean <bob....@gmail.com> wrote:

I am not sure I understand your point Tully, unless it is my inexperience with python. in C++ the accessors are effectively optimized away or there are methods to construct them such that access is as fast as direct member access. No need to copy data...

My point is that adding abstraction limits the options available to developer/users.  If a design requires accessors to get to data it takes away the ability for users to do low level operations which leverage knowledge of the memory layout, such as using pointer arithmetic and hardware acceleration such as SSE without copying into or out of the messages.  There's a whole spectrum of options were are looking at.  I'm just pointing out a viewpoint which has not been expressed yet in this thread.  

Tully

 

On Tuesday, June 4, 2013 2:28:40 PM UTC-4, Tully Foote wrote:

From a CS/elegance side putting the data behind accessors provides great flexibility from the viewpoint of the framework.  It provides the flexibility to change the system at the cost of removing that flexibility from the user.  The user group whom this will most effect is those doing very high bandwidth communications where one of two copies are a significant cost. The use case of image processing and pipelining jumps to my mind as one of the most common use case in this realm.  

We need to keep in mind these tradeoffs in our design. 

Tully

--
You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.

[cfka...@gmail.com]

I second Bob on this; the fact that we're talking about ROS  2.0 means the API is incompatible.  That said, the API is actually irrelevant. Once the format of the messages is fixed and unambiguous, we can access the contents how we feel like. Lightweight parsers might do exactly enough to ensure the message is well-formed, while a full featured library might have accessors and other support builtin.  In short, even for the same language, there might be multiple libraries out there. This is healthy as we each have different needs, and the different libraries will support them.

So, I'd like to put API discussions on the backburner while we figure out the wire format of the messages themselves.

Thanks,
Cem Karan

----- Reply message -----
From: "Bob Dean" <bob....@gmail.com>
To: <ros-sig...@googlegroups.com>
Subject: [ros-sig-ng-ros] Re: ROS 2.0 Messages + Serialization

[Michael Gratton]

On 06/06/13 20:23, cfka...@gmail.com wrote:
[snip]

> In short, even for the same language, there might be multiple
> libraries out there. This is healthy as we each have different needs,
> and the different libraries will support them.
>
> So, I'd like to put API discussions on the backburner while we
> figure out the wire format of the messages themselves.

Exactly this, yes.

--
⊨ Michael Gratton, Percept Wrangler.
⚙ <http://mjog.vee.net/>

[Jonathan Bohren]

On Thu, Jun 6, 2013 at 6:23 AM, cfka...@gmail.com <cfka...@gmail.com> wrote:

Once the format of the messages is fixed and unambiguous, we can access the contents how we feel like.

So is the consensus that no matter which message architecture we'd like to use, if there isn't an existing API which we like, we would write our own API which wraps or implements it?

[Jack O'Quin]

On Thu, Jun 6, 2013 at 8:39 AM, Jonathan Bohren <jonatha...@gmail.com> wrote:

On Thu, Jun 6, 2013 at 6:23 AM, cfka...@gmail.com <cfka...@gmail.com> wrote:

Once the format of the messages is fixed and unambiguous, we can access the contents how we feel like.

So is the consensus that no matter which message architecture we'd like to use, if there isn't an existing API which we like, we would write our own API which wraps or implements it?

I doubt there is much consensus yet. We don't all agree, even on the meaning of the number 2.0. :-)

Several people with good knowledge and strong opinions about serialization have been educating some of the rest of us about some currently-available options. That is helpful. I have not observed any conclusions on that yet.

There is general agreement that common wire and source representations are extremely desirable. 

There is also consensus that serialization should not be mingled with the transport layer as in the current roscpp implementation.

My personal opinion is that transport issues are much more important than serialization, which is mostly working OK already. Transport and serialization are separate issues. They should be designed and implemented accordingly. Transport delivers bytes, and serialization describes how messages are encoded in those bytes.

--
 joq

[Dirk Thomas]

On 06.06.2013 06:05, Michael Gratton wrote:
> On 06/06/13 20:23, cfka...@gmail.com wrote:
> [snip]
>> In short, even for the same language, there might be multiple
>> libraries out there. This is healthy as we each have different needs,
>> and the different libraries will support them.
>>
>> So, I'd like to put API discussions on the backburner while we
>> figure out the wire format of the messages themselves.
>
> Exactly this, yes.

I can only emphasize that this should not (!) be the goal here.
One of the most important goals for ROS 2.0 is the reuse of existing libraries to reduce long term maintenance.
Defining again our own wire format and implementing again our own serialization is not desirable.

We should either pick one existing serialization library or make the serialization pluggable and being negotiated / configured.
Both approaches have pros and cons (not a thorough list but the major aspect, more has been mentioned in this thread before):

* making the used serialization library pluggable require a generic API as a facade to the various library APIs, some mapping code is required for each serialization library
* a generic solution can not leverage all extrinsic features of a specific serialization library but only the common ones (and of course the specific intrinsic features)

* picking one serialization library might be the "wrong" decision in the near future when the next "better" thing comes out (e.g. protobuf vs. cap-n proto)
* one library might not suite all requirements e.g. regarding supported platforms (library size, suitable for embedded systems)
* exposing the IDL and API of a single serialization library ties every user land code to that solution, upgrading in the future would be a tremendous effort

- Dirk

[Michael Gratton]

On 07/06/13 03:41, Dirk Thomas wrote:
>>> So, I'd like to put API discussions on the backburner while we
>>> figure out the wire format of the messages themselves.
>>
>> Exactly this, yes.
>
> I can only emphasize that this should not (!) be the goal here. One
> of the most important goals for ROS 2.0 is the reuse of existing
> libraries to reduce long term maintenance. Defining again our own
> wire format and implementing again our own serialization is not
> desirable.

I can understand the need to reduce maintenance overhead, however it is
exactly the composition of nodes into a graph and interoperable passing
of messages between them that defines ROS as what it is. If you take
that away ROS becomes just a meta-distribution of vaguely related
software packages, with little value-add.

So it seems strange to to hitch the cart to something that (as you point
out) is inevitably going to require some compromise anyway. Surely if
there is one thing that the project wants to get absolutely right, it is
interoperability and fit-for-purpose of message passing between nodes.
To me that seem to imply maintaining it in-house.

If you think of this as a third option, where the major (only?) downside
is maintenance effort, then that provides a way to evaluate if it is
actually not worthwhile or not, rather than declaring by fiat it is off
the table.

In comparing it with the other two options, making serialisation
pluggable will require a similar order of magnitude of maintenance
effort and greatly harm interoperability, and while choosing one
3rd-party serialisation library is clearly the winner in terms of
maintenance effort, it only maintains interoperability for
supported/suitable platforms and introduces non-trivial risk.

On that basis, it should be seriously considered as an option.

//Mike

[Dirk Thomas]

On 06.06.2013 18:22, Michael Gratton wrote:
> On 07/06/13 03:41, Dirk Thomas wrote:
>>>> So, I'd like to put API discussions on the backburner while we
>>>> figure out the wire format of the messages themselves.
>>>
>>> Exactly this, yes.
>>
>> I can only emphasize that this should not (!) be the goal here. One
>> of the most important goals for ROS 2.0 is the reuse of existing
>> libraries to reduce long term maintenance. Defining again our own
>> wire format and implementing again our own serialization is not
>> desirable.
>
> I can understand the need to reduce maintenance overhead, however it is
> exactly the composition of nodes into a graph and interoperable passing
> of messages between them that defines ROS as what it is. If you take
> that away ROS becomes just a meta-distribution of vaguely related
> software packages, with little value-add.

If you think that any of my proposed options would reduce interoperability than we seem to be not on the same page.
The option with the pluggable serialization would even increase interoperability because it becomes much more easy to integrate systems which might not be able to use the "one-and-only" library we
might end up selecting.
This is why I recommend the pluggable serialization over the one-and-only serialization:
the user land code stays generic and therefore agnostic to the serialization and that enables long term interoperability - even in the case if in the future we would add another serialization library
or switch the default one all existing and future nodes are transparently working together.

> So it seems strange to to hitch the cart to something that (as you point
> out) is inevitably going to require some compromise anyway. Surely if
> there is one thing that the project wants to get absolutely right, it is
> interoperability and fit-for-purpose of message passing between nodes.
> To me that seem to imply maintaining it in-house.
>
> If you think of this as a third option, where the major (only?) downside
> is maintenance effort, then that provides a way to evaluate if it is
> actually not worthwhile or not, rather than declaring by fiat it is off
> the table.

It would be exactly reinventing the wheel.
There are serialization libraries out there - and they are really sophisticated.
Considering that we can just create a better (or even equal) solution is not really realistic without spending significant work into a domain which already has adequate solutions.
Reusing is the key here - and that does not imply any reduced interoperability.

> In comparing it with the other two options, making serialisation
> pluggable will require a similar order of magnitude of maintenance
> effort and greatly harm interoperability, and while choosing one
> 3rd-party serialisation library is clearly the winner in terms of
> maintenance effort, it only maintains interoperability for
> supported/suitable platforms and introduces non-trivial risk.

Providing a standardized API with thin layers of remapping code from that API to a specific serialization library is orders of magnitude less code and less complexity than writing a custom
serialization library.

- Dirk

[Damon Kohler]

I'd just like to stress again that it's possible to have both a
pluggable system and a one-and-only serialization library. If the
underlying ROS pubsub API is only about byte streams (e.g. the way
zeroMQ does it), then you can use the existing ROS serializer or any
other serializer you desire.

There's no need to write a higher level API to wrap all serialization
libraries or to pick one that everyone _has_ to use. Just pick a
preferred one (e.g. the existing serialization) and move on.

This could even be a backwards compatible change.

Protocol negotiation can be built on top of such a system if that
every seems necessary. But I think that should be a separate
discussion.

> --
> You received this message because you are subscribed to the Google Groups
> "ROS SIG NG ROS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ros-sig-ng-ro...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
Damon Kohler

Google Germany GmbH
Dienerstr. 12, 80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

[Jack O'Quin]

On Fri, Jun 7, 2013 at 1:20 AM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On 06.06.2013 18:22, Michael Gratton wrote:

On 07/06/13 03:41, Dirk Thomas wrote:

So, I'd like to put API discussions on the backburner while we
figure out the wire format of the messages themselves.

Exactly this, yes.

I can only emphasize that this should not (!) be the goal here. One
of the most important goals for ROS 2.0 is the reuse of existing
libraries to reduce long term maintenance. Defining again our own
wire format and implementing again our own serialization is not
desirable.

I share Dirk's concern about OSRF maintenance overhead. They have limited resources and a lot on their plate.

But, having The One True Serialization Layer does *not* necesssarily require anyone to invent a new wire format or a new implementation.

 

I can understand the need to reduce maintenance overhead, however it is
exactly the composition of nodes into a graph and interoperable passing
of messages between them that defines ROS as what it is. If you take
that away ROS becomes just a meta-distribution of vaguely related
software packages, with little value-add.

+1 The ease and efficiency of sharing messages between nodes is key to the success of ROS. That is worth serious effort, even at the cost of paying less attention to other areas. 

 

If you think that any of my proposed options would reduce interoperability than we seem to be not on the same page.

With enough layers of abstraction, we can interoperate with almost anything. But there are overhead and complexity costs associated with that approach, which some of us may be reluctant to pay.

 

The option with the pluggable serialization would even increase interoperability because it becomes much more easy to integrate systems which might not be able to use the "one-and-only" library we might end up selecting.

Perhaps, but only if we ignore some significant costs in CPU cycles, configuration complexity and user confusion. Those are some of the reasons I prefer TOTSL, if possible. I want the simplest solution that meets our needs. 

We still have not explicitly defined our requirements and other desiderata. We should do that before going much further.

If our needs can't be met without multiple layers, then we'll have to deal with the extra overhead and complexity. But, no one has really demonstrated that yet.

 

This is why I recommend the pluggable serialization over the one-and-only serialization:
the user land code stays generic and therefore agnostic to the serialization and that enables long term interoperability - even in the case if in the future we would add another serialization library or switch the default one all existing and future nodes are transparently working together.

That addresses only the wire-level issue. The IDL representation and the way messages are represented in code are even more important. ROS nodes need a common language of messages. That's how it all hangs together.

 

So it seems strange to to hitch the cart to something that (as you point
out) is inevitably going to require some compromise anyway. Surely if
there is one thing that the project wants to get absolutely right, it is
interoperability and fit-for-purpose of message passing between nodes.
To me that seem to imply maintaining it in-house.

If you think of this as a third option, where the major (only?) downside
is maintenance effort, then that provides a way to evaluate if it is
actually not worthwhile or not, rather than declaring by fiat it is off
the table.

It would be exactly reinventing the wheel.

Maybe that is a fair criticism Michael's proposal, but I'm not sure. 

There are definitely other alternatives that do not reinvent wheels, however.

 

There are serialization libraries out there - and they are really sophisticated.
Considering that we can just create a better (or even equal) solution is not really realistic without spending significant work into a domain which already has adequate solutions.

+1

 

Reusing is the key here - and that does not imply any reduced interoperability.

Note that sticking with rosmsg is also a form of reuse, and its interoperability for robotics packages has already proven successful. 

At least in the short and medium term, software effort for both OSRF and the overall ROS community would almost certainly be minimized by selecting that option. 

 

In comparing it with the other two options, making serialisation
pluggable will require a similar order of magnitude of maintenance
effort and greatly harm interoperability, and while choosing one
3rd-party serialisation library is clearly the winner in terms of
maintenance effort, it only maintains interoperability for
supported/suitable platforms and introduces non-trivial risk.

Providing a standardized API with thin layers of remapping code from that API to a specific serialization library is orders of magnitude less code and less complexity than writing a custom serialization library.

Dirk is arguing against inventing a *new* serialization library. I agree with him. But, several plausible options have been mentioned already. Surely we can consider each of them as a potential TOTSL without reinventing any wheels? 

 * rosmsg

 * thrift

 * protobuf

 * msgpack

 * JAUS ?

 * DDS ?

Please add others if I forgot to mention one you care about.

--
 joq

[Benjamin Charrow]

I agree with a lot of what Jack said, but one thing in particular stood out:

On Sun, Jun 9, 2013 at 11:41 AM, Jack O'Quin <jack....@gmail.com> wrote:

We still have not explicitly defined our requirements and other desiderata. We should do that before going much further.

Agreed.  Here are some of my higher level requirements (several of which are met by ROS):

Requirements:

1) Easy to define new messages in the IDL.  I don't want to have to specify much except for a field name and a type.

2) Messages are strongly typed.  I want to quickly reference a file that tells me about all the fields in a message.

3) Support for all standard numeric types (fixed width ints and float/double) and strings.

4) Easy to define namespaces.

5) Minimal time to generate code and compile into existing applications.

Desiderata:

1) A KeyValue data structure.  I know this can sometimes run afoul of requirement #3.

2) Versioned messages / fields to enable longer running services.  I get by without this for my own stuff, but for writing a public facing API it would be very nice to have.

Dirk is arguing against inventing a *new* serialization library. I agree with him. But, several plausible options have been mentioned already. Surely we can consider each of them as a potential TOTSL without reinventing any wheels? 

 * rosmsg

 * thrift

 * protobuf

 * msgpack

 * JAUS ?

 * DDS ?

Please add others if I forgot to mention one you care about.

A quick plug for the messageformats page which has some pros/cons analysis:

http://www.ros.org/wiki/sig/NextGenerationROS/MessageFormats 

Ben

[Bob Dean]

Requirements:

1) Easy to define new messages in the IDL.  I don't want to have to specify much except for a field name and a type.

2) Messages are strongly typed.  I want to quickly reference a file that tells me about all the fields in a message.

I suggest that #1 be extended to require a comment, as it directly impacts #2.
 

3) Support for all standard numeric types (fixed width ints and float/double) and strings.

what about bool, and enums?  Enums assist with #2.  Bitfields are discussed elsewhere in the thread.
 

4) Easy to define namespaces.

5) Minimal time to generate code and compile into existing applications.

I think #5 is handled by every modern solution.
 

Desiderata:

1) A KeyValue data structure.  I know this can sometimes run afoul of requirement #3.

2) Versioned messages / fields to enable longer running services.  I get by without this for my own stuff, but for writing a public facing API it would be very nice to have.

To be integrating and receive a "version mismatch error" instead of crashing code is in valuable. Message versions, in addition to message id's, may need to be something which is added onto the serialization solution. Be it md5, 32bit hash, or uuid.  But using a 4 byte id and version is nice to reduce overhead when sending multiple messages across the same comm pipe.  for example when embedding ros messages within a secondary transport, or onto a single socket.

[Michael Gratton]

On 11/06/13 07:11, Benjamin Charrow wrote:
>
> Requirements:
> 1) Easy to define new messages in the IDL. I don't want to have to
> specify much except for a field name and a type.
> 2) Messages are strongly typed. I want to quickly reference a file that
> tells me about all the fields in a message.
> 3) Support for all standard numeric types (fixed width ints and
> float/double) and strings.
> 4) Easy to define namespaces.
> 5) Minimal time to generate code and compile into existing applications.

These look good to me from a requirements-for-defining-messages
perspective. Per Bob Dean's email, support for real enums would be nice,
although bitfields strike me as being a serialisation detail?

> Desiderata:
> 1) A KeyValue data structure. I know this can sometimes run afoul of
> requirement #3.
> 2) Versioned messages / fields to enable longer running services. I get
> by without this for my own stuff, but for writing a public facing API it
> would be very nice to have.

I would suggest that the second should be a requirement, again as Bob
Dean mentioned versioning is terribly useful for public APIs.

On key-value types, I wonder if that is possibly open to abuse of
requirement #2 - in that if your message has a key-value map of some
sort then it can be used to stuff whatever objects in there that you
want. Limiting it to specific key and value types (e.g. a
map<int,string>) might help a bit? Also, if you're going to admit a
key-value map, you may as well add other collections types such as lists
and sets.

Finally, I wonder if there is any value in providing for data stream
messages? Might be worth it for video, or even things as simple as sonar
range data. The idea being that non-stream fields are headers and sent
once, but the stream field data just keeps on coming. Probably not, but
perhaps worth discussing.

> Dirk is arguing against inventing a *new* serialization library. I
> agree with him. But, several plausible options have been mentioned
> already. Surely we can consider each of them as a potential TOTSL
> without reinventing any wheels?

So which do and don't meet these requirements and desiderata?

[CFK]

I'm FINALLY at a real keyboard and can write sensible messages!  Hallelujah!

UUID thoughts:

In an earlier message, I mentioned that if we use UUIDs as type descriptions of messages, various search engines will eventually pick up the UUID, allowing automatic discovery of forums, wikis, etc. that contain information pertinent to the message type.  I then gave the UUID 3DE366BC-2AFD-4E41-A517-0C8FBD3F02E7 as an example to search for.  Google has picked it up, and points to this thread, and only this thread.  The other search engines I've tried haven't yet picked it up, but that might be because we're on googlegroups, and Google has faster access. So we've got one example of UUIDs being useful as a way of finding more information on a message type automatically.

Bob pointed out that if we use fewer bits, we'll save space.  The only problem with this is that the only way to generate UUIDs in a fully distributed fashion is likely to require some kind of randomization, which means that there is some possibility of two independent people choosing the same UUID for two different message types.  Since message types (like standards) don't die, even when they really, really should, that means that when a UUID is used, it can never, ever be reused.  So, this suggests that we need to figure out how likely it is that any pair of randomly chosen UUIDs will collide.  Fortunately, the birthday paradox  answers exactly this question.  The square approximation of the birthday paradox suggests that a collision will occur with 1% probability at sqrt((2**<number of bits>) * 0.01).  For 32 bits, this is about 9000 message types, for 64 bits, this is about 600,000,000 message types, and for 128 bits, this is about 2,600,000,000,000,000,000 message types.   So we could go with 64 bit UUIDs, provided we didn't do versioning (if we do versioning, making only 32 bits randomized, we might increase the probability of collision).  I still strongly prefer real UUIDs (full 128 bit), if only because there are a lot of libraries out there that are able to generate them at this point, and because that particular type of UUID has been well-studied in the literature already, but I'll accept the group decision, provided that it is at least 64 bits in length.

API:

Jonathan Bohren asked about what APIs will be used, and whether or not we'd have to write our own.  Jack O'Quin pointed out that we don't have any consensus yet, not even on the meaning of ROS 2.0.  Dirk Thomas emphasized that we should either pick one serialization library, or make serialization a pluggable architecture.

My thought is that there will likely be several competing libraries/APIs as we hash out what is most easily used for each language/domain.  Pluggable layers will only work if we fix what we're serializing completely.  That means deciding on basic types (e.g., ints, floats, bit vectors, strings, maps, etc.).  Do we have that kind of consensus?  If not, then we could do something crazy, as in the next section:

Serialization format:

I agree with what many others have said about finding a single library/format, and making that THE format.  But, just to play devil's advocate, let's think about what would happen if we went in the exact opposite direction and declare that serialized messages all have exactly two fields:

1. A message type identifier (as everyone can probably guess by now, for me that means a 128 bit UUID)
2. An arbitrarily long byte array. (I'm assuming that there is a length encoding or termination method so we know where the end is)
   

Programmers will be 100% responsible for decoding the byte array into whatever in-memory format that is best suited for their own language, or for finding a library that is able to do it for them. 

Disadvantages:

1. CHAOS!
   
2. Debugging all those serialization/deserialization libraries is going to be a nightmare.
3. Writing them in the first place is going to eat up more time than any of us care to think about, and it will happen over an over again for different languages/platforms.
   
4. It will not be self-describing; if you can't find the UUID in your favorite search engine, then you'll be starting from scratch figuring out what the heck this message actually is.
   
5. Serializing multiple messages into a log/bag file will be a headache.
6. Composing message types may be difficult.
   
7. Fill in your own favorite reasons why this is a Bad Thing (TM)

Advantages:

1. FREEDOM!
   
2. Regardless of the data type, it will never need to be shoehorned into something that wasn't designed to hold it.  That means that bitvectors won't be crammed into the bits of an integer, MPEG video won't get encased into something bigger an bulkier, etc.  This might save space and simplify processing.
   
3. Constrained systems can define message formats that are optimized for them.
4. Every language's ROS-layer API will be trivial.  For any message, you will only have the UUID field, and the byte array field.  That makes adding new languages to the ROS family of supported languages easy.
5. If you have an 'inside' and an 'outside', where all systems that are on the 'inside' are fully under your control, you can write translators that live on the border, converting from one format to another.  For example, if you have several networks on the outside, you may choose different compression/transmission schemes depending on which network you're transmitting on, even though you're sending the same data, while on the inside there is no compression and no error correction.  All messages carry the same information, but the encoding scheme (and therefore message type) changes.
6. Fill in your own favorite reasons why this is a Good Thing (TM)
   

I'm sure that there are other advantages and disadvantages to this approach, which brings up the following two questions:

1. Do the advantages outweigh the disadvantages?
2. Is this approach the best approach?
   

Thanks,
Cem Karan

[Jack O'Quin]

On Wed, Jun 12, 2013 at 9:23 AM, CFK <cfka...@gmail.com> wrote:

I'm FINALLY at a real keyboard and can write sensible messages!  Hallelujah!

UUID thoughts:

I like UUIDs and use them in other ROS messages. Here is a small metapackage that defines a ROS UniqueID message, plus C++ and Python methods for using it.

 http://www.ros.org/wiki/unique_identifier

 

However, I don't understanding something. With pub/sub, why send a type signature in every message? 

Currently, I believe the MD5sum is verified between the subscriber and the publisher at connection time. Isn't that better for most of our purposes?

API:

My thought is that there will likely be several competing libraries/APIs as we hash out what is most easily used for each language/domain.  Pluggable layers will only work if we fix what we're serializing completely.  That means deciding on basic types (e.g., ints, floats, bit vectors, strings, maps, etc.).  Do we have that kind of consensus? 

Not yet. But, I hope we can reach one.

My feeling is that our current modest collection of types has proven adequate. I am happy to listen to those who favor certain additions (bit fields, associative arrays, etc.). Each have their uses. But I am skeptical about adding much to the current list. I suspect most of them end up improving some corner cases, while making many other things harder.

The big advantage of a simple collection of types is that they work cleanly with a wide range of languages.

If we select one of the fancier off-the-shelf serialization libraries, it may provide some additional types already. We should still discuss whether or not to use them within ROS messages. We could agree to limit ourselves to a subset of the possibilities.

If not, then we could do something crazy, as in the next section:

Serialization format:

I agree with what many others have said about finding a single library/format, and making that THE format.  But, just to play devil's advocate, let's think about what would happen if we went in the exact opposite direction and declare that serialized messages all have exactly two fields:

1. A message type identifier (as everyone can probably guess by now, for me that means a 128 bit UUID)
2. An arbitrarily long byte array. (I'm assuming that there is a length encoding or termination method so we know where the end is)
   

...

 

Programmers will be 100% responsible for decoding the byte array into whatever in-memory format that is best suited for their own language, or for finding a library that is able to do it for them.  I'm sure that there are other advantages and disadvantages to this approach, which brings up the following two questions:

1. Do the advantages outweigh the disadvantages?
2. Is this approach the best approach?

1. I doubt it, but will be interested in others' comments.

2. We should consider it only if we fail to find a reasonable TOTSL solution.

--
 joq

[Benjamin Charrow]

On Tue, Jun 11, 2013 at 5:52 PM, Michael Gratton <mi...@vee.net> wrote:

Per Bob Dean's email, support for real enums would be nice,
although bitfields strike me as being a serialisation detail?

I third requiring support for enums and having bools as a base type.

I don't think bitfields should be a requirement because it's pretty easy to use constants as masks and it seems like adding bitmasks would complicate the IDL (e.g., the XML example from JAUS that Bob Dean shared on 5/31).  I do, however, appreciate that my requirements aren't everyone's requirements ;-)

Another requirement that came up with when talking with wjwood: default values for base types.  This has been discussed elsewhere previously, but I think the best example is specifying w=1 in geometry_msgs/Quaternion.  Currently you can't do this, which often leads to bugs (e.g., plotting Markers in rviz).

> Desiderata:
> 1) A KeyValue data structure.  I know this can sometimes run afoul of
> requirement #3.

On key-value types, I wonder if that is possibly open to abuse of
requirement #2 - in that if your message has a key-value map of some
sort then it can be used to stuff whatever objects in there that you
want. Limiting it to specific key and value types (e.g. a
map<int,string>) might help a bit? Also, if you're going to admit a
key-value map, you may as well add other collections types such as lists
and sets.

Ah!  I totally expected maps to be have strongly typed keys/values.  I also think it would be OK if keys were limited to base types as 1) mostly I want to map a string or int to a user defined type and 2) defining equality is easy for base types, but a little less so for user defined types.  The main reason I want maps is I often need to send logically grouped data (e.g., see sensor_msgs/JointState) and I'd like the IDL to support that and generate code that verifies each key has one value.

I didn't list it in my original requirements, but fixed and variable sized arrays should be required (ROS currently does both).

I haven't found myself needing sets, so I won't push for their inclusion.

 

>     Dirk is arguing against inventing a *new* serialization library. I
>     agree with him. But, several plausible options have been mentioned
>     already. Surely we can consider each of them as a potential TOTSL
>     without reinventing any wheels?

So which do and don't meet these requirements and desiderata?

Let's keep on determining what requirements and desiderata we have, and then revisit this question :-)

Ben

[CFK]

On Wed, Jun 12, 2013 at 11:17 AM, Jack O'Quin <jack....@gmail.com> wrote:

On Wed, Jun 12, 2013 at 9:23 AM, CFK <cfka...@gmail.com> wrote:

I'm FINALLY at a real keyboard and can write sensible messages!  Hallelujah!

UUID thoughts:

I like UUIDs and use them in other ROS messages. Here is a small metapackage that defines a ROS UniqueID message, plus C++ and Python methods for using it.

 http://www.ros.org/wiki/unique_identifier

Nice!

 

 

However, I don't understanding something. With pub/sub, why send a type signature in every message? 

pub/sub assumes that we have a connection-oriented transport.  What if we're over UDP, or something else that is similar?  What if we're bagging all messages that all nodes are publishing, and then want to figure out what each message type was afterwards?

 

Currently, I believe the MD5sum is verified between the subscriber and the publisher at connection time. Isn't that better for most of our purposes?

Unfortunately, no.  Assume we have two different message types, each of which contains a single boolean.  One message causes the robot to self-destruct, while the other causes it to take out the garbage.  The MD5 of the types will be the same, so figuring out which we're talking about is impossible.   If we decide to hash the entire message definition, then we'll do better, but I know I've been guilty of writing basic ping-like messages that have no content whatsoever; the information is the topic that it was transmitted on.  With UUIDs, every message type will have a different identifier, and it isn't subject to human failings of various kinds.

 

Thanks,
Cem Karan

[Jack O'Quin]

In my old-fashioned software engineering vocabulary, "requirement" had a very specific meaning: "without this, you have no useful product". Not everyone shares that specialized definition, but it remains an important concept. To make myself clearer, I'll divide some desirable features into three categories: must-have, good-to-have, and nice-to-have. 

I left some good ideas out because I personally don't consider them important enough to worry about at this high level of discussion. I probably left out some others because I forgot. Some items likely require engineering trade-offs to resolve conflicts between them, but hopefully none are totally contradictory. 

must-have:

-------------

 * Commercial-friendly, permissive, open-source license (BSD, Apache, MIT, ...).

 * CPU and storage overhead as good as rosmsg, or close enough.

 * Excellent language support for C++ and Python.

 * Works on Linux, OSX, Windows.

 * Can represent most or all existing ROS messages.  A great deal of thought has gone into many of these messages. They have proven useful for a wide range of robotic systems. That is not a trivial accomplishment.

 * Messages can define fields using messages defined in another package. This follows from the previous requirement, but is worth explicit mention.
 * Handle very large messages, for things like maps, images and point clouds.
 * Can represent a message as an "object" in supported languages.

 * Detect message version inconsistencies somehow.

 * Serialization and deserialization are decoupled from transport of bytes.

 * Clear specifications.

good-to-have:

-----------------

 * Better performance than rosmsg. We want the transport layer free to do clever shared-memory tricks, when possible. Measurements with rosmsg showed that serialization overhead led to disappointing results.

 * Good language support for C, Java, and Lisp. 

 * Works on low-end controllers. (How low?)

 * Can represent fixed-size messages as Plain Old Data structs.

 * Avoid memory allocation for fixed-size messages, where possible. This has both low-end and real-time implications. Is it possible? Worth studying.

 * Somehow deal with the Header frame_id being a variable-length string, even if it involves a hack.

nice-to-have:

----------------

 * Support for additional languages.

 * Message initialization with appropriate default values.
 * Can extend messages by adding fields in a compatible way.

 * Fairly simple IDL.

 * Works on some real-time supervisors. (Which ones?)

--

 joq

[William Woodall]

I would personally put this in the must-have category. This is something that rosmsg, for example, does not address at all but does come up (uninitialized quaternions).

 

 * Can extend messages by adding fields in a compatible way.

 * Fairly simple IDL.

 * Works on some real-time supervisors. (Which ones?)

--

 joq

--

You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
William Woodall

ROS Development Team

wil...@osrfoundation.org

http://williamjwoodall.com/

[Daniel Stonier]

+1 

Not so much for the convenience of it, but ensuring the system has more robustness against humans accidentally making bugs because of ill-specified structures is big point for me.

 

 

 * Can extend messages by adding fields in a compatible way.

 * Fairly simple IDL.

 * Works on some real-time supervisors. (Which ones?)

--

 joq

--
You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
William Woodall

ROS Development Team

wil...@osrfoundation.org

http://williamjwoodall.com/

--
You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Phone : +82-10-5400-3296 (010-5400-3296)
Home: http://snorriheim.dnsdojo.com/

Yujin R&D: http://rnd.yujinrobot.com/

[Cem Karan]

> On Jun 12, 2013, at 7:56 PM, William Woodall wrote:
>
>> On Wed, Jun 12, 2013 at 4:40 PM, Jack O'Quin <jack....@gmail.com> wrote:
>> In my old-fashioned software engineering vocabulary, "requirement" had a very specific meaning: "without this, you have no useful product". Not everyone shares that specialized definition, but it remains an important concept. To make myself clearer, I'll divide some desirable features into three categories: must-have, good-to-have, and nice-to-have.
>>
>> I left some good ideas out because I personally don't consider them important enough to worry about at this high level of discussion. I probably left out some others because I forgot. Some items likely require engineering trade-offs to resolve conflicts between them, but hopefully none are totally contradictory.
>>
>> must-have:
>> -------------
>>
>> * Commercial-friendly, permissive, open-source license (BSD, Apache, MIT, ...).

Do you mean for the wire format of the message, or for the libraries that encode the message type?

Actually, now that I think about it, maybe the format should be copyrighted (copylefted?) as well? I mean, is it possible for someone to claim ownership of the format itself after we've defined it, locking everyone out of using it without paying a fee (like Rambus?)? I'd like to avoid that possibility if at all possible.

>> * CPU and storage overhead as good as rosmsg, or close enough.
>> * Excellent language support for C++ and Python.
>> * Works on Linux, OSX, Windows.
>> * Can represent most or all existing ROS messages. A great deal of thought has gone into many of these messages. They have proven useful for a wide range of robotic systems. That is not a trivial accomplishment.
>> * Messages can define fields using messages defined in another package. This follows from the previous requirement, but is worth explicit mention.
>> * Handle very large messages, for things like maps, images and point clouds.

How large is 'very large'? MessagePack handles maps and lists of up to 2^32 items, is that large enough?

>> * Can represent a message as an "object" in supported languages.

Might be difficult; if I serialize a Python object, and then read it out via a C library, what can I expect?

>> * Detect message version inconsistencies somehow.

Can we consider different versions of messages as different types of messages? In that case, typing via UUIDs or some other mechanism can handle it.

>> * Serialization and deserialization are decoupled from transport of bytes.

+10

>> * Clear specifications.

+infinity

>> good-to-have:
>> -----------------
>>
>> * Better performance than rosmsg. We want the transport layer free to do clever shared-memory tricks, when possible. Measurements with rosmsg showed that serialization overhead led to disappointing results.
>> * Good language support for C, Java, and Lisp.
>> * Works on low-end controllers. (How low?)
>> * Can represent fixed-size messages as Plain Old Data structs.
>> * Avoid memory allocation for fixed-size messages, where possible. This has both low-end and real-time implications. Is it possible? Worth studying.
>> * Somehow deal with the Header frame_id being a variable-length string, even if it involves a hack.
>>
>> nice-to-have:
>> ----------------
>>
>> * Support for additional languages.
>> * Message initialization with appropriate default values.
>
> I would personally put this in the must-have category. This is something that rosmsg, for example, does not address at all but does come up (uninitialized quaternions).

+1

>> * Can extend messages by adding fields in a compatible way.
>> * Fairly simple IDL.
>> * Works on some real-time supervisors. (Which ones?)

Thanks,
Cem Karan

[Jack O'Quin]

On Wed, Jun 12, 2013 at 9:40 PM, Cem Karan <cfka...@gmail.com> wrote:

> On Jun 12, 2013, at 7:56 PM, William Woodall wrote:
>
>> On Wed, Jun 12, 2013 at 4:40 PM, Jack O'Quin <jack....@gmail.com> wrote:
>> In my old-fashioned software engineering vocabulary, "requirement" had a very specific meaning: "without this, you have no useful product". Not everyone shares that specialized definition, but it remains an important concept. To make myself clearer, I'll divide some desirable features into three categories: must-have, good-to-have, and nice-to-have.
>>
>> I left some good ideas out because I personally don't consider them important enough to worry about at this high level of discussion. I probably left out some others because I forgot. Some items likely require engineering trade-offs to resolve conflicts between them, but hopefully none are totally contradictory.
>>
>> must-have:
>> -------------
>>
>>  * Commercial-friendly, permissive, open-source license (BSD, Apache, MIT, ...).

Do you mean for the wire format of the message, or for the libraries that encode the message type?

I was talking about the interface library or headers. Programs using it must not be encumbered by license restrictions. Even the LGPL is probably too much.

 

Actually, now that I think about it, maybe the format should be copyrighted (copylefted?) as well?  I mean, is it possible for someone to claim ownership of the format itself after we've defined it, locking everyone out of using it without paying a fee (like Rambus?)?  I'd like to avoid that possibility if at all possible.

Good point, I didn't think about that. The specification can be openly copyrighted or even placed in the public domain. Once published, it would become "prior art" to any subsequent patent claim. But, I suppose there could be potential "submarine" patents already in the pipeline. We certainly cannot tolerate any patent restrictions.  

  

>>  * CPU and storage overhead as good as rosmsg, or close enough.
>>  * Excellent language support for C++ and Python.
>>  * Works on Linux, OSX, Windows.
>>  * Can represent most or all existing ROS messages.  A great deal of thought has gone into many of these messages. They have proven useful for a wide range of robotic systems. That is not a trivial accomplishment.
>>  * Messages can define fields using messages defined in another package. This follows from the previous requirement, but is worth explicit mention.
>>  * Handle very large messages, for things like maps, images and point clouds.

How large is 'very large'?  MessagePack handles maps and lists of up to 2^32 items, is that large enough?

Probably. We need to do some investigation. I am not sure what the current limits are. 

I have published map messages of a gigabyte or two without problems, and have never heard any complaints about large message difficulties.

 

>>  * Can represent a message as an "object" in supported languages.

Might be difficult; if I serialize a Python object, and then read it out via a C library, what can I expect?

Probably a pointer to a struct. The exact answer depends on which solution we adopt, but it must be well-defined. I used "object" in quotes because some languages are not "object-oriented". In my experience, a good C program can be as "object-oriented" as necessary, but only with careful design and implementation. After all, the reference Python implementation is itself written in C.

 

>>  * Detect message version inconsistencies somehow.

Can we consider different versions of messages as different types of messages?  In that case, typing via UUIDs or some other mechanism can handle it.

I purposefully did not prescribe any details. The overall requirement can be handled in many ways, including UUIDs or MD5 checksums. Some implementations may permit extensions or version numbers. Exactly how it is done does matter, but for now I want to reach consensus on high-level goals.

--
 joq

[Jack O'Quin]

On Wed, Jun 12, 2013 at 4:46 PM, CFK <cfka...@gmail.com> wrote:

On Wed, Jun 12, 2013 at 11:17 AM, Jack O'Quin <jack....@gmail.com> wrote:

However, I don't understanding something. With pub/sub, why send a type signature in every message? 

pub/sub assumes that we have a connection-oriented transport.  What if we're over UDP, or something else that is similar?  What if we're bagging all messages that all nodes are publishing, and then want to figure out what each message type was afterwards?

Pub/sub provides that a publisher "socket" is somehow paired with a subscriber "socket". Although the ROS 1.0 implementation uses TCP, there are other solutions. Zeromq provides pub/sub over PGM reliable multicast, one reason why some of us find that transport option attractive.

 http://www.zeromq.org/whitepapers:0mq-3-0-pubsub

 

I expect that a ROS 2.0 pub/sub implementation should still involve some kind of new "multi-master" protocol for matching up publishers and subscribers using strongly-typed topics.

The fact that topics are strongly typed is the key issue. 

Currently, I believe the MD5sum is verified between the subscriber and the publisher at connection time. Isn't that better for most of our purposes?

Unfortunately, no.  Assume we have two different message types, each of which contains a single boolean.  One message causes the robot to self-destruct, while the other causes it to take out the garbage.  The MD5 of the types will be the same, so figuring out which we're talking about is impossible.   If we decide to hash the entire message definition, then we'll do better, but I know I've been guilty of writing basic ping-like messages that have no content whatsoever; the information is the topic that it was transmitted on.  With UUIDs, every message type will have a different identifier, and it isn't subject to human failings of various kinds.

Actually, the MD5 checksums will differ, unless the field names are identical. But, that is just a detail of the current implementation. More importantly, the message type name (package/Message) must also match. 

Using generic messages like std_msgs/Bool with various divergent meanings is *not* accepted ROS practice. We can't keep people from writing bad code, but we can provide tools that allow them to conveniently write good code.

The main point is that we would prefer to do type checking when advertising or subscribing to a topic, not on every message.

--
 joq

[Michael Gratton]

On 14/06/13 02:01, Jack O'Quin wrote:
>
> On Wed, Jun 12, 2013 at 4:46 PM, CFK <cfka...@gmail.com

> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=cfka...@gmail.com>>

> wrote:
>
> On Wed, Jun 12, 2013 at 11:17 AM, Jack O'Quin <jack....@gmail.com

> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=jack....@gmail.com>>

> wrote:
>
>
> However, I don't understanding something. With pub/sub, why send
> a type signature in every message?
>
>
> pub/sub assumes that we have a connection-oriented transport. What
> if we're over UDP, or something else that is similar? What if we're
> bagging all messages that all nodes are publishing, and then want to
> figure out what each message type was afterwards?
>
>
> Pub/sub provides that a publisher "socket" is somehow paired with
> a subscriber "socket". Although the ROS 1.0 implementation uses TCP,
> there are other solutions. Zeromq provides pub/sub over PGM reliable
> multicast, one reason why some of us find that transport option attractive.
>
> http://www.zeromq.org/whitepapers:0mq-3-0-pubsub
>
> I expect that a ROS 2.0 pub/sub implementation should still involve some
> kind of new "multi-master" protocol for matching up publishers and
> subscribers using strongly-typed topics.

Is there any reason why XMPP & extensions hasn't been suggested for
master & co? If you leave aside the IM aspects, it is a mature,
lightweight, extensible protocol that is well supported, intrinsically
decentralised (i.e. handles multimaster for free), has a large number of
libraries in a number of languages, and existing extensions for
signalling (i.e. negotiation and establishment of out-of-band streams)
and pubsub. It is also an IEEE standard that that seems to be free of
patent and copyright problems.

The downside is poor support for in-band binary data, but if it is used
as a naming/pubsub/signalling solution then that's not a problem, and
maybe is a benefit in the end for people who want to support pluggable
serialisation libraries anyway.

If the idea is that the ROS 2.0 messaging system will be used for naming
and pubsub as well as passing data between nodes then you have a
chicken-egg problem, which can be avoided a dedicated naming service.
Using XMPP as one, nodes establish a connection to their local master on
startup as an XMPP client - master would act as an XMPP server. Nodes
can then do pubsub and access services a similar way as they do now.
Nodes on different masters can be accessed transparently, communicating
via XMPP's built-in server-to-server federation support. This would
actually be a good way to access parameters and have parameter changes
pushed, too.

[Jack O'Quin]

On Thu, Jun 13, 2013 at 8:39 PM, Michael Gratton <mi...@vee.net> wrote:

Is there any reason why XMPP & extensions hasn't been suggested for
master & co? If you leave aside the IM aspects, it is a mature,
lightweight, extensible protocol that is well supported, intrinsically
decentralised (i.e. handles multimaster for free), has a large number of
libraries in a number of languages, and existing extensions for
signalling (i.e. negotiation and establishment of out-of-band streams)
and pubsub. It is also an IEEE standard that that seems to be free of
patent and copyright problems.

The downside is poor support for in-band binary data, but if it is used
as a naming/pubsub/signalling solution then that's not a problem, and
maybe is a benefit in the end for people who want to support pluggable
serialisation libraries anyway.

If the idea is that the ROS 2.0 messaging system will be used for naming
and pubsub as well as passing data between nodes then you have a
chicken-egg problem, which can be avoided a dedicated naming service.
Using XMPP as one, nodes establish a connection to their local master on
startup as an XMPP client - master would act as an XMPP server. Nodes
can then do pubsub and access services a similar way as they do now.
Nodes on different masters can be accessed transparently, communicating
via XMPP's built-in server-to-server federation support. This would
actually be a good way to access parameters and have parameter changes
pushed, too.

You can suggest that. I have not used XMPP. Maybe others can comment on their experiences with it. There does not seem to be much momentum behind it these days.

 http://www.zdnet.com/google-moves-away-from-the-xmpp-open-messaging-standard-7000015918/

I prefer getting rid of XML/RPC for the master protocol and using the serialization we are discussing here, instead. The low-end controller folks would no longer need an XML parser.

I have not thought much about the details. Although ROS topics represent pub/sub "connections", other types of transport will also be supported, probably including datagrams. 

--
 joq

[Cem Karan]

> On Jun 13, 2013, at 12:01 PM, Jack O'Quin wrote:
> <<SNIP>>

I agree that it isn't good/accepted practice, but I also know I've seen bad code on other projects. People are either unaware, or pressed for time and try to cut corners to get the job done. Combating this is a two-step process.

1) Message type definitions are similar (maybe identical?) to what we have now. As humans, we don't put in any UUIDs.
2) The type definitions are precompiled into binary files that includes a generated UUID and all the fields from the message type definition, plus anything else anyone here decides needs to be put in (URL for where to find more info? Author's email? Others?). This file is binary only to discourage people from hacking on it rather than the human-readable format.

When message type definitions are distributed, only the binary format is distributed. This gives a code generator enough to work with that it can generate code for C, C++, Python, etc for the platform it is run on.

Note that I'm NOT trying to make this binary format secret! I want people to be able to extract all the needed metadata, including the UUIDs using whatever tool they feel comfortable with. I'm just going with a binary format because it provides just enough of a speed bump that most people will use the provided tools, which should at least warn them when they are about to cause themselves and others headaches.

> The main point is that we would prefer to do type checking when advertising or subscribing to a topic, not on every message.

In my mind, that means a connection-oriented architecture. What about a message-oriented architecture, such as UDP? I can see a few different solutions, but they all come down to some kind of out-of-band signaling to indicate the topic/sender/etc. Or am I missing something obvious?

> --
> joq

Thanks,
Cem Karan

[Cem Karan]

I'd also like to avoid XML/RPC; I suspect that it is taxing on embedded systems when it doesn't need to be. +1 for datagrams/alternative transports!

Cem

[Michael Gratton]

On 14/06/13 12:08, Jack O'Quin wrote:
>
> You can suggest that. I have not used XMPP. Maybe others can comment
> on their experiences with it. There does not seem to be much
> momentum behind it these days.

XMPP had a lot of momentum before Google adopted it for instant
messaging, and Google was more interested in "embracing and extending"
rather than cooperating anyway. While it is sad for people that use
GTalk, it's not a loss otherwise.

> I prefer getting rid of XML/RPC for the master protocol and using
> the serialization we are discussing here, instead. The low-end
> controller folks would no longer need an XML parser.

I can kind of understand that, but it sounds a bit like "640k should be
enough for anyone", but in reverse. Wasn't the plan to not re-invent the
wheel? While XML isn't a terribly good wire format, it's not like
communication with master constitutes a large amount of typical node
traffic.

[Bob Dean]

please forgive lack of quotations. also, jumping around between topics after reading the last 10 posts or so.

Question: what is the line between serialization and messaging API? I think there is a grey area between the two. There have been requests to not discuss this in this thread, however, there are concepts where api and serialization are tightly coupled.

Message id's (32bit, md5 or uuid) is an example of this. It is likely that the serialization library itself may not provide a message id, or version support, and the ros api would need to add it. Popular approaches would be to calculate these values from the message IDL.  Bit fields are another example, encoding them in serialization is just supporting ints. Using them effectively is an API issue.

As there is likely to be a (hopefully thin) codegen wrapper that adds on to serialization, there are requirements for those tools:
- multi language.
   example :MD5 generation in the current system is python only. I have tried to recreate the md5 calculation in cpp code, and never got it to work. I actually resorted to writing a ruby script to walk the ros install tree, find message files, and parse out the md5 value, as it was faster than the python tools.  A C or CPP library which would provide this functionality without wrapping the python code would be useful.
- minimize client dependencies
- the idl and extensions should be machine readable: http://en.wikipedia.org/wiki/Machine-readable_data  As a government contractor, machine readable is usually a requirement for interoperability.
- Deterministic. if i run the tools in Maryland, and person Y runs them in California, the results should be the same. This is one of the reasons i disagree with unique UUIDs.  Using a named UUID generation scheme is deterministic.  Related to this is that no message is unique, it is always combined with other metadata such as topic name and version information which adds semantic meaning.

Multiplexing messages on the same channel/topic/connection is a common method of interoperability between systems, and is also dependent upon the serialization wrapper functionality we have been discussing. To multiplex, each message needs an additional header which allows the message to be properly delivered.  In general this header requires the message type, message size, connection/destination id, and an optional version field.  Translating this into ros 1.0 land would be the md5, size, topic name, and the version is currently encoded in the md5.  Using ints, this header could be 20 bytes in size. Using ros 1.0 concepts it would be much larger.  In the scope of processing a single message, not a big deal. In the scope of hundreds of thousands of messages per second, the added message size becomes non trivial w.r.t. network load.

[Jack O'Quin]

I think we have been focusing on different parts of the problem, not necessarily disagreeing.

My concern is that most messages in a ROS system flow over topics. Each topic has a type name and hash signature. That API is pub/sub. We want it to work well. Types are checked whenever a subscriber or an advertiser "connects" to the topic, making topics connection-oriented. Not necessarily end-to-end, but certainly end-point to topic.

I am trying to focus on API and avoid transportation terminology here. How to implement pub/sub topics over various transport mechanisms is a very interesting and important question. I prefer to keep it separate from serialization, if possible. So, I avoid discussing TCP, UDP or PGM much in this thread.

While most ROS message traffic is pub/sub, it is not the only communication pattern we need to support. Request/reply is also needed for the master protocol and for ROS services. I can't think of another API pattern that we must support. Maybe I'm forgetting something. But, I would like to focus on a small number of patterns like those two.

For request/reply interfaces, I see no alternative to checking the message types on every request. I think we agree about that. 

But, I don't believe topic traffic needs type checking on every message. My guess is that it will continue be worthwhile to avoid it. I prefer not to indulge in premature optimization until the overall design becomes more concrete. I request only that we keep it open as an option. 

--
 joq

[CFK]

On Mon, Jun 17, 2013 at 10:31 AM, Bob Dean <bob....@gmail.com> wrote:

please forgive lack of quotations. also, jumping around between topics after reading the last 10 posts or so.

Question: what is the line between serialization and messaging API? I think there is a grey area between the two. There have been requests to not discuss this in this thread, however, there are concepts where api and serialization are tightly coupled.

Message id's (32bit, md5 or uuid) is an example of this. It is likely that the serialization library itself may not provide a message id, or version support, and the ros api would need to add it. Popular approaches would be to calculate these values from the message IDL.  Bit fields are another example, encoding them in serialization is just supporting ints. Using them effectively is an API issue.

I see what you're saying, but my fear is that if we start mixing API & format discussions together, we'll end up with a specification that essentially says 'do it like the C++ code does it when compiled using g++ under Unbuntu 13.10 on a 32 bit intel system'.  My hope is that we'll specify the format so clearly that anyone can write a new library to interact with the format, regardless of what system or language they are using.  RFC 1321 specifies both the algorithm and octet ordering of MD5, and RFC 4122 does the same for UUIDs.  As a result API/language/etc. questions are unimportant for these types; each language will have one or more ways of generating the MD5 or UUID it needs, with the API being designed for the language and users of that language.  That way an assembly programmer won't have to deal with the cruft that a Python programmer feels is natural and necessary, and the python programmer isn't bit twiddling when a more Pythonic API can be created.  Why do we want each language to use the same API anyways, what's so important about that???   Let each language evolve an API appropriate to that language.  That way, people that are versed in and used to the idioms of their own language can leverage their knowledge about how the language works to learn the API for their language.

Even your example about bitfields is somewhat ambiguous.  Older systems defined ints as being 16 bit, but now they are mostly 32 bit.  On 64 bit systems, I think they are still 32 bit, but I think that that is compiler dependent.  So when I treat a 64 bit int as a bit vector, what happens when the encoded message is decoded onto a 32 bit or 16 bit system?  What if its on a system that is a different endianess than the encoder?  You need to set the specification to the point where it is absolutely unambiguous FIRST.

As there is likely to be a (hopefully thin) codegen wrapper that adds on to serialization, there are requirements for those tools:
- multi language.
   example :MD5 generation in the current system is python only. I have tried to recreate the md5 calculation in cpp code, and never got it to work. I actually resorted to writing a ruby script to walk the ros install tree, find message files, and parse out the md5 value, as it was faster than the python tools.  A C or CPP library which would provide this functionality without wrapping the python code would be useful.

Do you mean that there need to be multiple generators, each one written in a different language, or that the generator needs to emit native code for the language it is emitting?  I would hope that the tool generated native code for each language; wrapping other languages never quite goes as well as one would hope.

 

- minimize client dependencies
- the idl and extensions should be machine readable: http://en.wikipedia.org/wiki/Machine-readable_data  As a government contractor, machine readable is usually a requirement for interoperability.
- Deterministic. if i run the tools in Maryland, and person Y runs them in California, the results should be the same. This is one of the reasons i disagree with unique UUIDs.  Using a named UUID generation scheme is deterministic.  Related to this is that no message is unique, it is always combined with other metadata such as topic name and version information which adds semantic meaning.

I'm afraid I still have to disagree with you on this.  The UUID needs to be different because while the fields are identical between different message types, the meanings may not be.  As I mentioned before, the canonical example is the empty message.  The meaning may depend on the context in which it is sent (e.g., topic, sender, etc.).  Moreover, a given message may change meaning over time.  This might happen while a system is being developed, with the meaning shifting subtly over time, while the content stays the same.  In this case, we don't want to reuse the message ID, which would happen if we used the hash of the message as the ID.

However, I do see the problem of teams of people working on message types, compiling them, and getting different UUIDs.  This is why I suggested earlier that message compilation be a 2-step process, first going from a human-readable format to a machine-readable format specification that included an automatically generated UUID.  This could be as simple as converting from plain text to some other serialization format; the only reason for the format change is to tack on the automatically generated UUID, and to lightly obscure the contents to discourage hacking on the 'frozen' specification instead of the human-readable one.  This frozen specification would be distributed, either directly or via your favorite version control system and compiled into native code on the robot that runs it.  Composing messages would then be as simple as specifying the UUID of the message type you want to add in.  The message compiler would search your system in the appropriate places for frozen messages, and add them in, creating a new message type with a new UUID. 

 

Multiplexing messages on the same channel/topic/connection is a common method of interoperability between systems, and is also dependent upon the serialization wrapper functionality we have been discussing. To multiplex, each message needs an additional header which allows the message to be properly delivered.  In general this header requires the message type, message size, connection/destination id, and an optional version field.  Translating this into ros 1.0 land would be the md5, size, topic name, and the version is currently encoded in the md5.  Using ints, this header could be 20 bytes in size. Using ros 1.0 concepts it would be much larger.  In the scope of processing a single message, not a big deal. In the scope of hundreds of thousands of messages per second, the added message size becomes non trivial w.r.t. network load. 

A UUID would replace MD5, topic name and version.  Since MD5 is 128 bits, and UUIDs are also 128 bits, we've guaranteed that the new message format is shorter than the old one. 

If you really want to compress everything, we could go for my 'devil's advocate' format of [UUID, byte array], and have individual programmers decide what encoding scheme they're going to use for the byte array.  Assuming that the length of the byte array is always encoded into the message (e.g. [UUID, array size, byte array]), then we can simply stream all messages across the same channel/topic/etc.  The UUID will tell the listener everything they need to know about the incoming message type/topic/etc.  Note that I'm being deliberately vague about what 'everything' is; the author of the message type will need to define what the metadata is, and if it can all be implicitly encoded into the UUID.

Thanks,

Cem Karan

[CFK]

OK, that makes sense to me.  Hmmm... this is a pure straw-man suggestion, beat it up as you will. 

All of the connection-oriented communications systems I know of have some way of deciding that a connection has been broken.  That suggests that a simple protocol over a connection might include a handshake at the start, where both sides determine the types of messages that are being passed along on that connection.  Since this has to happen at the start of a connection, if a connection is broken, all that will happen is another handshake will be forced when the connection is remade.  Once the handshake has been made, both sides can agree to drop the UUIDs altogether, making this like ROS 1.0. 

If we want to make things slightly more complicated, both sides might exchange/agree on substitution dictionaries for canoncial UUIDs; e.g., 23 is an alias for 3D30E9EA-B946-4164-829D-3E9428722D94.  This will allow the two sides to reduce the number of bytes they transmit, while still allowing multiple message types to be exchanged over the same connection (so that we can multiplex like Bob Dean suggested, or exchange multiple message types on the same connection).  The dictionary will need to be ephemeral, existing only as long as the connection does; that way if a connection is broken, we don't accidentally (or maliciously) substitute a different message type for the one that was actually transmitted.

For communications that aren't connection-oriented, no handshaking will happen, and all messages continue to be marked by canonical UUIDs.

Comments?

Thanks,
Cem Karan

[Benjamin Charrow]

On Wed, Jun 19, 2013 at 7:31 AM, CFK <cfka...@gmail.com> wrote:

On Mon, Jun 17, 2013 at 10:31 AM, Bob Dean <bob....@gmail.com> wrote:

- Deterministic. if i run the tools in Maryland, and person Y runs them in California, the results should be the same. This is one of the reasons i disagree with unique UUIDs.  Using a named UUID generation scheme is deterministic.  Related to this is that no message is unique, it is always combined with other metadata such as topic name and version information which adds semantic meaning.

I'm afraid I still have to disagree with you on this.  The UUID needs to be different because while the fields are identical between different message types, the meanings may not be.  As I mentioned before, the canonical example is the empty message.  The meaning may depend on the context in which it is sent (e.g., topic, sender, etc.). 

-1 for uuids as a way of capturing semantic information. +1 for determinism.

tl;dr: I think UUIDs will add complexity, but won't add much in terms of safety.  

longer version:

Currently pubs and subs negotiate and have to agree on the md5sum (representing the message definition) as well as the message type (e.g., sensor_msgs/LaserScan) before exchanging data and I think that is sufficient to eliminate most of the errors you get when using a message passing system.  As Cem has pointed out, a remaining problem is that messages might have the same fields and type, but be semantically different.  However, I think that's where topic names come in.  If an application still really needs to differentiate these messages, they should use something like Jack's UUID library.

 

Moreover, a given message may change meaning over time.  This might happen while a system is being developed, with the meaning shifting subtly over time, while the content stays the same.  In this case, we don't want to reuse the message ID, which would happen if we used the hash of the message as the ID.

Is this also a +1 for requiring the ability to evolve message definitions over time (e.g., optional fields in protobuf)?

Ben

[CFK]

On Wed, Jun 19, 2013 at 12:59 PM, Benjamin Charrow <bcha...@osrfoundation.org> wrote:

On Wed, Jun 19, 2013 at 7:31 AM, CFK <cfka...@gmail.com> wrote:

On Mon, Jun 17, 2013 at 10:31 AM, Bob Dean <bob....@gmail.com> wrote:

- Deterministic. if i run the tools in Maryland, and person Y runs them in California, the results should be the same. This is one of the reasons i disagree with unique UUIDs.  Using a named UUID generation scheme is deterministic.  Related to this is that no message is unique, it is always combined with other metadata such as topic name and version information which adds semantic meaning.

I'm afraid I still have to disagree with you on this.  The UUID needs to be different because while the fields are identical between different message types, the meanings may not be.  As I mentioned before, the canonical example is the empty message.  The meaning may depend on the context in which it is sent (e.g., topic, sender, etc.). 

-1 for uuids as a way of capturing semantic information. +1 for determinism.

tl;dr: I think UUIDs will add complexity, but won't add much in terms of safety.  

longer version:

Currently pubs and subs negotiate and have to agree on the md5sum (representing the message definition) as well as the message type (e.g., sensor_msgs/LaserScan) before exchanging data and I think that is sufficient to eliminate most of the errors you get when using a message passing system.  As Cem has pointed out, a remaining problem is that messages might have the same fields and type, but be semantically different.  However, I think that's where topic names come in.  If an application still really needs to differentiate these messages, they should use something like Jack's UUID library.

I see what you're saying, but as I said earlier MD5 and UUIDs both use 128 bits.  We're not saving anything.  However, with MD5, if the semantic meaning has shifted, but the topic and contents have not, then the MD5 will remain the same.  Hashing is an excellent method of ensuring that your data has not been corrupted, but it can only operate on information that is present; semantic meaning is often just an agreement between programmers.  Take Bob Dean's idea of packing integers into bitfields; a common method in C is to say that some number of bits are used, and that the rest are currently reserved.  This information should be well defined within the documentation of the message type (where the hashing code will pick it up), but sometimes the only place to find it is on a wiki/forum/mailing list/etc.  The problem is that as soon as someone decides they're going to use the formerly reserved bits, they've changed the meaning of the message type, but the MD5 remains the same.  This is what I was trying to get at with the meaning of messages shifting over time.

 

 

Moreover, a given message may change meaning over time.  This might happen while a system is being developed, with the meaning shifting subtly over time, while the content stays the same.  In this case, we don't want to reuse the message ID, which would happen if we used the hash of the message as the ID.

Is this also a +1 for requiring the ability to evolve message definitions over time (e.g., optional fields in protobuf)?

Optional fields only allow you to add more on at a later date.  That is handy, but it still doesn't address the original problem of the shifting of meanings.  I'm trying to make the semantic meaning of a message clear despite the fact that its content, message definition and topic may remain the same.  MD5 won't let us do that.

If you want to go for a pub/sub negotiation method, then use UUIDs instead of MD5; the number of bits are the same, and you get the freedom to change semantic meaning without being forced to change the fields themselves.

Thanks,
Cem Karan

[Jack O'Quin]

On Thu, Jun 20, 2013 at 6:45 PM, CFK <cfka...@gmail.com> wrote:

Optional fields only allow you to add more on at a later date.  That is handy, but it still doesn't address the original problem of the shifting of meanings.  I'm trying to make the semantic meaning of a message clear despite the fact that its content, message definition and topic may remain the same.  MD5 won't let us do that.

I understand what you are trying to accomplish. But, I think it goes too far. The current scheme of package/TypeName plus MD5 works well enough in my experience. The topic name provides additional semantics context. I find that useful and would not like to lose it.

 

If you want to go for a pub/sub negotiation method, then use UUIDs instead of MD5; the number of bits are the same, and you get the freedom to change semantic meaning without being forced to change the fields themselves.

I like UUIDs, but I feel your fixation on that solution is a bit premature.

Let's agree that we need type checking, and that UUIDs are one possible approach, but not the only one. We may end up choosing an existing library that uses some other solution. That will be OK.

--
 joq

[Cem Karan]

You're right, I am getting prematurely fixated on it, and for that reason, I apologize. I keep seeing my own attempt at a messaging system where I used SHA1 to ID the message types... and then got stuck when the meanings shifted, but the types stayed the same. It took quite a bit of hard work to dig myself out of that hole, and I'm worried about falling into it again.

Thanks,
Cem Karan