RFC: MAVLink 1.1 specs - discussion start

[Meier Lorenz]

Hi all,

I've tested yesterday successfully variable-length fields / messages support. It actually works nicely, my implementation is here:
https://github.com/PX4/Firmware/commit/f10453414f215e34624cf9723bf534beb1c78f3b

This will save quite some bandwidth for things like parameters (due to packing into param vectors) and strings.

Its just a prototype, but it shows that its no big drama to add support to the generator. We will support two options: Strings with a max size (NUL-terminated) and arrays where you provide the pointer and the length as argument. This also means that we can do a variable-length param message now that holds anything between 1 byte and a full rotation matrix.

I've a long laundry list of changes intended for 1.1. Please feel free to comment. I'm numbering them consistently, please refer to the numbers as well.


SOME IMPORTANT NOTES:

The intention is to do all these changes by introducing new messages with new IDs and leaving dual support in the GCS (and in some autopilots) for an intermediate transition phase. The switch to 1.2 will then remove the duplicate deprecated messages. This will ensure a smooth transition and allows us to prevent the protocol from aging.

The variable length support is an extra feature and does not alter existing messages or the parser (see the diff for details).



1) Unclear specs on data streams
https://pixhawk.ethz.ch/mavlink/#REQUEST_DATA_STREAM
I think the spec should be changed to say that a data stream is a message ID. This means that you need to fire a data stream message for every message ID you'd like to receive, but this is the only general interface. Most autopilots should anyway be setup sensibly from start and not require.


2) Redudant message: This message should probably be marked as deprecated:
https://pixhawk.ethz.ch/mavlink/#VFR_HUD


3) Missing file transfer protocol:
The common message set has some messages for file transfer, but this has not been implemented. Either implement and spec this or remove them.
https://pixhawk.ethz.ch/mavlink/#FILE_TRANSFER_START
https://pixhawk.ethz.ch/mavlink/#FILE_TRANSFER_DIR_LIST
https://pixhawk.ethz.ch/mavlink/#FILE_TRANSFER_RES


4) Note on message ID #255. This message ID has been reserved for extended messages (e.g. a 2.0 version of MAVLink) I want to keep the reservation in place.


5) Extend the param protocol to a binary, variable-length block. It should at least hold 3x3 float (rotation) matrices. Proposal: Create a new message like param_set:
https://pixhawk.ethz.ch/mavlink/#PARAM_SET
But in contrast spec:
- The param id to be of variable length. Names must not start with numbers. If the ID field starts with a number, it should be interpreted as the index of the param (so parameters can be selected by their ID again)
- The param data contains a union with variable length up to 40 bytes
- The param type supports also complex types like vectors
- We might want to think about optionally supporting vectory-enty names as well (e.g. there is a field_id type in the message, which contains one character per field (so it could contain 'x', 'y', 'z' and the param would then be: vector_test.x, vector_test.y, vector_test.z. This is however optional and might be too tedious to implement.
- A more radical step would be to remove the string during param sending and merge with the string operation proposal from 6), this will in practice however not safe bandwidth and forces users of 5) to implement 6)

6) String operation - this proposal by Andrew Tridgell allows to query strings from the system, mostly to fill human-readable interfaces with labels, descriptions and values. (typical example: get a board or firmware name, get mode display names, store names for an ENUM and allow the GCS to auto-generate a parameter dropdown by querying the enum values)
https://github.com/tridge/mavlink/commit/8041bc387deff0e9cee53f7d79f8d9f138b7d7d4

Some APM-specific things:


7) FENCE_POINT vs SAFETY_SET_ALLOWED_AREA. The fence point notion supports a polygon, while the safety area is just a box. If the polygon support is manageable in APM, it probably makes sense to deprecate the area and support polygons.
https://github.com/mavlink/mavlink/blob/master/message_definitions/v1.0/ardupilotmega.xml#L221
https://pixhawk.ethz.ch/mavlink/#SAFETY_SET_ALLOWED_AREA

8) RADIO packet - with the wide adoption of the 3DR radio, this packet should be moved over to the common message set.
https://github.com/mavlink/mavlink/blob/master/message_definitions/v1.0/ardupilotmega.xml#L280

9) SW version / GIT commit hash / string
We will want to add a proper packet for Firmware versions and use this in the future to deal with new/old compatibility.


10) PIXHAWK vs APM binary blob transmission

PIXHAWK uses one handshake per transmission and then a large data packet. With the variable length support this should be replaced by a better solution using variable length packets.
DATA_TRANSMISSION_HANDSHAKE
https://github.com/mavlink/mavlink/blob/master/message_definitions/v1.0/pixhawk.xml#L175
ENCAPSULATED_DATA
https://github.com/mavlink/mavlink/blob/master/message_definitions/v1.0/pixhawk.xml#L184

APM uses different data packets, which similarly should be replaced by the same new approach in the common message set.
DATA16-DATA96
https://github.com/mavlink/mavlink/blob/master/message_definitions/v1.0/ardupilotmega.xml#L314

11) Redundant HWSTATUS packet, should be replaced with SYSTEM_STATUS and deprecated
https://github.com/mavlink/mavlink/blob/master/message_definitions/v1.0/ardupilotmega.xml#L274
should be replaced with: SYSTEM_STATUS
https://pixhawk.ethz.ch/mavlink/#SYS_STATUS

12) Sensor-level HIL state. The common message set does currently lack a sensor-level HIL message. The APM simstate packet is probably a good base to start with.
SIMSTATE
https://github.com/mavlink/mavlink/blob/master/message_definitions/v1.0/ardupilotmega.xml#L259



Feel free to add more.

-Lorenz


------------------------------------------------------
Lorenz Meier
Institute for Visual Computing
ETH Zurich
http://www.inf.ethz.ch/personal/lomeier/

[Susurrus]

I have some comments inline with your pose, but I wanted to ask about
other changes that are possibly backwards-incompatible. Based on the
suggestions here, some things will break moving to 1.1. Does this mean
other things that should be fixed that would be backwards incompatible
are fair game as well? Assuming it does, here's a list of changes that I
can write the code for directly (including necessary changes to QGC).

In no particular order:
1) Improve field names in LOCAL_POSITION_NED. This would be a
code-breaking change but clarify documentation somewhat. Simple fix for
other projects as well. [https://github.com/mavlink/mavlink/issues/71]

2) `onboard_control_sensors_health` should be changed to
`onboard_control_sensors_healthy` to be consistent. Minor, but being
consistent would be nice. [https://github.com/mavlink/mavlink/issues/18]

3) A major change: move away from using C-style datatypes for the XML
definitions of messages. It's unnecessary and makes less sense as more
programming languages are supported. Use instead `uint8`, `int32`,
`real64`, etc. Would break all .XML files, but since they're all in the
repo, that'd be a pretty straight forward change. This won't actually
change the generated MAVLink code, so this isn't too big of a change.
[https://github.com/mavlink/mavlink/issues/12]

4) Make common.xml inherit from minimal.xml or remove minimal.xml. I
think actually keep minimal.xml around as the minimum dataset you'd need
for groundstations to recognize your vehicle would be worthwhile. Then
everything should inherit from this.
[https://github.com/mavlink/mavlink/issues/11]

5) Resolving how COMMAND_ACK is used. Does it use MAV_CMD_ACK? QGC uses
MAV_RESULT? Depending on what the correct solution to this is, it could
break code. [https://github.com/mavlink/mavlink/issues/105]

On 05/14/2013 09:08 AM, Meier Lorenz wrote:
> 3) Missing file transfer protocol:
> The common message set has some messages for file transfer, but this has not been implemented. Either implement and spec this or remove them.
> https://pixhawk.ethz.ch/mavlink/#FILE_TRANSFER_START
> https://pixhawk.ethz.ch/mavlink/#FILE_TRANSFER_DIR_LIST
> https://pixhawk.ethz.ch/mavlink/#FILE_TRANSFER_RES

If no one's using this, I'd recommend removal. With no one actually
using it, it'd be hard to spec out properly for a real use. Save until
there's a use-case for it.

> 5) Extend the param protocol to a binary, variable-length block. It should at least hold 3x3 float (rotation) matrices. Proposal: Create a new message like param_set:
> https://pixhawk.ethz.ch/mavlink/#PARAM_SET
> But in contrast spec:
> - The param id to be of variable length. Names must not start with numbers. If the ID field starts with a number, it should be interpreted as the index of the param (so parameters can be selected by their ID again)

Does this mean an ASCII numeric value? This will make things more
awkward if you can't refer to a parameter via a proper integer and
instead have to use a string that has only numeric characters. This also
makes having an index in the PARAM_VALUE message odd. I'd propose having
PARAM_SET and PARAM_VALUE only operate using integers with the string
being received for a human interface component. In the system, these
values are probably stored by their integer index anyway.

[swordfishBob]

+1 to keep minimal.xml and inherit it.

Nice to see some progress on variable length packets; I'll have to look into the details..

[Edwin Hayes]

First post; if this is the wrong place, apologies.

I'd like to propose that the should be two different types of heartbeat messages:  one sent out by individual components, and one which is sent out by a complete system (which is what message #0 currently seems to be).  For instance, if a MAV consists of two components, an autopilot and a something-else, then the something-else might need to send out a heartbeat to indicate to the autopilot it is still working, but the something-else might have no idea about things such as the type of MAV it is mounted on, which is a field in the current heartbeat message.  Every component should generally emit a heartbeat indicating its own status (well, 'should' is a bit approximate, but no matter), but almost certainly only one component per MAV should be configured to send out a message indicating the health of the overall system.

I could just add in a custom definition of a 'component heartbeat' message to obtain this functionality myself, but it seems like a pretty fundamental distinction for any system composed of more than one component, so it would be good to see this change considered.  So long as the system heartbeat message remains, I don't think this would break anything?

An alternative is for components like the something-else to use the existing heartbeat message, leaving fields they don't know about unpopulated, and to ensure that either such internal messages never make it off-board to the ground station, or to make sure the ground station is only interested in one particular component's heartbeat as representative of the overall system health, but these options seem uglier.

[crashmatt]

No unnecessary code breaking changes please.  We are tired of maintaining projects around this.  Keep compatibility or make a clean move on to V2.0

/Matt

[crashmatt]

Is anyone interested in supporting flexible mixers?  The mixer functions would not be defined in mavlink.  The messages for discovering different types of available mixers and their parameters would be defined in mavlink.

This would also need the support of the string operations from idea #6.

/Matt

[Susurrus]

What's a flexible mixer?

Bryant

> --
> Sie haben diese Nachricht erhalten, weil Sie der Google Groups-Gruppe
> MAVLink beigetreten sind.
> Um Ihr Abonnement für diese Gruppe zu beenden und keine E-Mails mehr
> von dieser Gruppe zu erhalten, senden Sie eine Email an
> mavlink+u...@googlegroups.com.
> Weitere Optionen: https://groups.google.com/groups/opt_out
>
>

[crashmatt]

Bryant,

Much like an assignable mixer on an RC transmitter but with many different functions available.  Functions can be multistep mixers, decisions, rate limiters, exponential etc. The number of mixers is only limited by the memory space devoted to it.  The advantage is easy experimentation with many different styles of aircraft, not just the standard configurations.

A key problem is having a gui to interface with all the data.  This is where the string operations is important.

Regards Matt

[Gautier Hattenberger]

Hi Lorentz,

I'm one of the Paparazzi developers from ENAC and we are thinking of improving our own message system (unfortunately this project is on hold currently, this old crap was too much work to improve in one step...).
The work that we did last year on this matter was to add support for fixed-length arrays and changing the payload header from 2 bytes (sender and msg id) to 4 bytes in order to have a possible merge (at least some basic compatibility) in the future. The first byte is a packet sequence number. But for the component id, this is not fitting with our software architecture so we have introduced a "class" id instead. This allow you to group messages into groups (standard messages and various custom groups) and extend the possible number of messages to 256^2.
I have no idea how the component id is used, and using this solution for mavlink would probably break everything (same for us, thought). But have you ever consider this option (advantages and drawbacks) ? I see you want to increase the number of messages using a reserved id (and encode the extended id into the payload I guess). Is using classes of messages an option to discuss for a future version of mavlink (2 or more) or is it something you don't want to consider ?

More (outdated) information about our work:
http://paparazzi.enac.fr/wiki/DevGuide/CommunicationsNew
https://github.com/paparazzi/paparazzi/tree/messages

Thanks

Gautier

[Susurrus]

One last thing that I'd like to put on the agenda for MAVLink 1.1 (or maybe 2.0) is to specify in the documentation the proper way to use mavlink_parse_char as no documentation specifies what to do with the final argument `r_mavlink_status`, not the website, not its doxygen comments, not QGC's source code.

From reading the code it appears that mavlink_parse_char stores in `r_mavlink_status` whether there was a parsing error, how many successfully-decoded messages there have been, and the next expected message sequence. The problem here is that most of this is redundant or useless information: the current sequence number can be read from the stores mavlink_status_t struct for the provided channel along with the packet_rx_sucess_count. The parsing error flag is actually stored in the packet_rx_drop_count member, which doesn't make sense, because one call to mavlink_parse_chart can at most result in a single parsing error. And it actually appears that the packet_rx_drop_count for the mavlink_status_t for a given channel is not incremented on failure and cannot be used.

So my proposal is:

• Modify mavlink_parse_char() so that it returns  one of three values: nothing happened, message parsing failure, or message parsing success.
• Remove the last argument (r_mavlink_status) so people use mavlink_get_channel_status() if they want these details
• Modify the parser to actually track the number of dropped packets, which I suspect should really just be a count of missing sequence numbers in the MAVLink data stream (there is existing commented-out code that basically does this, does anyone know what happened with this?)
  

If this seems reasonable the Meier and the core MAVLink team, I'd be happy to whip up a pull request with these changes.

Bryant

On 05/14/2013 09:08 AM, Meier Lorenz wrote:

[Chris Gough]

+1 for adding another byte to message id, and using it as a kind of
message namespace mechanism.

Gautier,

A MavLink Component ID is a finer grained address within a System ID.
It refers to one sub-part of the whole vehicle (this message comes
from the IMU, this one comes from the camera, etc).

Do you think adding a Component ID to the Paparazzi envelope would be
a benign step towards Paparazzi/MavLink integration?

Chris Gough

> --
> Sie haben diese Nachricht erhalten, weil Sie der Google Groups-Gruppe
> MAVLink beigetreten sind.
> Um Ihr Abonnement für diese Gruppe zu beenden und keine E-Mails mehr von
> dieser Gruppe zu erhalten, senden Sie eine Email an
> mavlink+u...@googlegroups.com.
> Weitere Optionen: https://groups.google.com/groups/opt_out
>
>

--
.

[swordfishBob]

I wouldn't want to change the size of the header if it can be avoided. It appears to me that:

- Many messages already imply their component. We have distinct message types for GPS location (final, IMU-integrated) and raw GPS straight from receiver. Couldn't there be a packet type for location+vector, and distinguish its origin and nature by component? I guess location could be a common message, as a camera might theoretically indicate its own location and bearing.

- Most mobile/airborne systems don't have multiple processors/codebases generating separate MAVLink packets which are then routed through the system; please correct me if I'm wrong. If there were, MAVLink could certainly be used on an internal data bus. If we have custom GPS and IMU modules, perhaps they could be made to support MAVLink natively? Would a motor controller or actuator ever talk MAVLink natively?

- Even components which exist in multiples are limited in number. Is there any reason that a component class couldn't be encoded as part of the component number, specifically the high nibble? Some common messages would be applicable to all classes (e.g. heartbeat, read and write variables), but others would be class-specific. This approach could begin without breaking compatibility; there would be no clashes until new messages are introduced. Systems using the new messages would implement the capability to distinguish by component class. Class 0 could be defined as "classless" or "legacy", so compid's 0x00 through 0x0F work as they are.

??

[Chris Gough]

> I wouldn't want to change the size of the header if it can be avoided.

It's quite a drastic change, but what if the alternative is pushing
protocol stuff inside the envelope? yuck.

> It appears to me that:
> - Many messages already imply their component. We have distinct message
> types for GPS location (final, IMU-integrated) and raw GPS straight from
> receiver. Couldn't there be a packet type for location+vector, and
> distinguish its origin and nature by component? I guess location could be a
> common message, as a camera might theoretically indicate its own location
> and bearing.

I agree that for many types of components/message, Component ID and
message type will be strongly correlated. So there's often not a lot
of information in Component ID. But it's there and if people are using
it then taking it away would hurt.

> - Most mobile/airborne systems don't have multiple processors/codebases
> generating separate MAVLink packets which are then routed through the
> system; please correct me if I'm wrong. If there were, MAVLink could
> certainly be used on an internal data bus.

I don't know about most but our system does. The sensor payload and
autopilot communicate with each other and the ground station using
MAVlink like a bus. Also, ground station and payload software is
implemented as MavProxy modules that are effectively agents on the
same bus too. I'm not sure how important component ID is for us right
now though, but we are talking about maybe adding another payload...

> If we have custom GPS and IMU modules, perhaps they could be made
> to support MAVLink natively? Would a motor controller or actuator ever
> talk MAVLink natively?

What if a single autopilot had multiple IMUs, e.g. (e.g.
aeroelastic/vibration analysis, ornithopter, etc)? Or a very large
airframe had multiple GPS's (for attitude)? The individual components
wouldn't need to support MAVLink natively, but the autopilot could
relay information from them (with the appropriate Component ID).

> - Even components which exist in multiples are limited in number. Is there
> any reason that a component class couldn't be encoded as part of the
> component number, specifically the high nibble?

I think this is the key to the semantic mismatch between Paparazzi and
MavLink. The "class" is not bound to components at all (a-priori at
least, there will of course be correlations). Class is a logical
grouping of message types, it is used to set coarse grained policies
on message handling. For example, messages of class "airborne" are not
communicated over telemetry. Messages of class "alert" jump the queue
for the speech agent on the ground segment, etc. Just about any
component could meaningfully issue an Alert, but a ground segment
component would never issue or expect a message of the airborne class.
It's just a convenience feature (and a way to enlarge the message
space).

I don't know if logical groupings of components would be useful,
maybe. That's something else.

Scavenging a message-class nibble off the Component ID header might be
a good solution though. Does anyone really need more than 128
component ids (for the same system id)? 128 message classes might be
plenty. The current messages.xml in Paparazzi master branch only uses
6 classes.

> Some common messages would be applicable to all classes (e.g. heartbeat,
> read and write variables), but others would be class-specific. This approach
> could begin without breaking compatibility; there would be no clashes until
> new messages are introduced.

I don't think there needs to be any clash at all. Different platforms
could start with a small set of proprietary message classes, and then
gradually refactor onto common ones where it makes sense to do so.
Ultimately there might be a small number of core message classes that
are used consistently by everyone, plus "extension" messages classes
that are platform, application, component or tribe specific. Plug-in
pattern.

The point is it's a semantic packaging mechanism, it should help
manage complexity. Maybe there could be a "future" class for
pre-releasing proposed new standard messages, and a few 'reserved
private' classes that are never used by 3rd party packages. etc. etc.

> Systems using the new messages would
> implement the capability to distinguish by component class. Class 0 could
> be defined as "classless" or "legacy", so compid's 0x00 through 0x0F work
> as they are.

Or old-school, pre-class messages could all get Class ID 128
("depricated"), and slowly removed as the housework makes progress.
Other messages could shift to "depricated" as they are updated, to
mitigate pain from backwards-incompatible changes.

Chris Gough

[Oswald Berthold]

Chris Gough writes:

> Scavenging a message-class nibble off the Component ID header might be
> a good solution though. Does anyone really need more than 128
> component ids (for the same system id)? 128 message classes might be
> plenty. The current messages.xml in Paparazzi master branch only uses
> 6 classes.

hi,

maybe the problem should be framed more generally.

basicllay the question is, how many components do you want / need to
address individually _without looking at the payload_. categorization into
system/component/subcomponent etc is only a convention to ease
conceptual handling which reflects the state of convergence in the
development of small flying machines.

it is evident that for most vehicles, there is one system and on the
order of maybe ten components. for certain more experimental
arrangements, the situation is different:
- swarms comprising of more than 256 members? a swarm addressed as one
single system with the individuals as components? what happens to the
components' components then?
- single vehicle seen as a distributed system, think DFA / modular
robotics style
- redundant measurements, multiple GPSs, IMUs, cameras, EMDs, ...
- in addition, software components running on an onboard embedded
system can come in any number.

variable-length address fields come to mind, in the standard case you
just have
- sysid
- componentid

and in the special case you have
- if sysid MSB == 1, treat remaining bits as address length
- any kind of common address part can be interpreted as message class
- do address interval matching (wildcards)

maybe MAVLink should be kept lean and more advanced robotics use cases
should implement their own schemes or use other available protocols and
mechanisms. its a decision on the scope and wether you can or want to
keep both well-hung straightforward applications and more experimental
scenarious under one umbrella.

excuse me if i overwrite or miss something that has been mentioned
already in this discussion.

bst, oswald

[swordfishBob]

On Thursday, 6 June 2013 13:52:52 UTC+10, Chris Gough wrote:

> I wouldn't want to change the size of the header if it can be avoided.

It's quite a drastic change, but what if the alternative is pushing
protocol stuff inside the envelope? yuck.

Agreed, if it is actually protocol stuff. My question was about components really being distinct, but you've answered that and I'm happy to have been corrected.

 

I think this is the key to the semantic mismatch between Paparazzi and 
MavLink. The "class" is not bound to components at all (a-priori at
least, there will of course be correlations). Class is a logical
grouping of message types, it is used to set coarse grained policies
on message handling. For example, messages of class "airborne" are not
communicated over telemetry. Messages of class "alert" jump the queue
for the speech agent on the ground segment, etc. Just about any
component could meaningfully issue an Alert, but a ground segment
component would never issue or expect a message of the airborne class.
It's just a convenience feature (and a way to enlarge the message
space).

Ah ok, the term "class" is hereby clarified. 

I don't know if logical groupings of components would be useful,
maybe. That's something else.

That's what I was thinking in terms of expanding the number of possible message types, without requiring another byte. There are overlaps, but there are messages that are unique by component-type.

 

Scavenging a message-class nibble off the Component ID header might be
a good solution though. Does anyone really need more than 128
component ids (for the same system id)? 128 message classes might be
plenty. The current messages.xml in Paparazzi master branch only uses
6 classes.

 

 So, continuing from Oswald's observation:

- How many systems might be operating together?

- How many components might a system really need, per-type and in total?

- How many classes (ie message routing and priority) - Chris mentioned Paparazzi has 6..

- How many message types are really unique to each type of component? How many messages are really common?

Perhaps resulting in:

* SysId - largely as is. A swarm or fleet has multiple systems, and may recognise e.g. 0=GCS, 1=Master, 255=All.

* CompId - 3 bits for component type (nav, location/orientation, motors/mechanics for motion, payload manipulation, imaging, custom...), 5 bits for component index. Would anyone have more than 32 independently-communicating instances of a component? This is backward-compatible as long as we can renumber some components. Perhaps use variable-length type/address, grouping types that are closely related and have few instances, though that takes more code to then filter type-specific MsgId's.

* MsgId -  1 bit for common vs comp-type-specific messages, 7 bits for message type, of which the 2-3 highest order bits imply message class (mix of latency/losslessness) - so there's still up to 128 common messages and up to 128 per component type, but perhaps only 32 per class.

Again this can be backward compatible with some renumbering..

Just throwing it out there, hoping it's constructive. 

[Gautier Hattenberger]

Hello,

Some more information about the classes of messages in Paparazzi:

What we have currently:
6 classes but only 3 really used
 - telemetry: downlink messages, generating C code and not checking for alignment of data, almost full
 - datalink: uplink messages, generating C code and checking for alignment
 - ground: inter-agent communication on the ground side, no C code generation, no binary format only string over Ivy software bus
Currently, the AP and the GCS are assuming that they receive the correct type of message (datalink and telemetry respectively). Operations like air-to-air communication or communication relay are not really expected or even possible.

With the new system, we wanted to have a "type" defined for each class: uplink, downlink, both up and down (for air-to-air com.), ground (and maybe internal AP communication). It means that the uplink messages for example can be split into several classes like common messages and custom messages. It should also be possible to receive messages from several sources (we have implemented a basic MAVLINK support with this).

We have been thinking of variable length header to integrate a time stamp in the header (depending on a reserved bit in the class id byte, 127 classes should be enough...).

Concerning the addition of a component id to paparazzi, it doesn't make much sense in the current state of the project since the implementation of the subsystems are limiting to only one instance on each AP.

Gautier

[Chris Gough]

Gautier, I'm interested to understand the minimum change to MAVLink that would enable a successful Paparazzi integration. If three message classes are non-essential, and "ground" message classes are only ever used within ivy, then it seems like "datalink" and "telemetry" class messages are the only ones that absolutely must be passed between the autopilot and the ground segment (i.e. via MAVLink). If that's true, would a few more bits of MAVLink message id be sufficient? Does the telemetry/telecommand carrier actually need to know anything about the message class? If MAVLink could dedicate 600 message ids to Paparazzi, as a first step could Paparazzi start supporting MAVLink as a channel, using something like a modem driver. And then, gradually harmonise standard messages by depricating Paparazzi messages that overlap with standard/common MAVLink ones?

Chris Gough

--

[Meier Lorenz]

Hi all,

Thanks for keeping this discussion so active, and I particularly appreciate the interest from the godfathers of UAV research.

First I'd like to make the current objective clear: MAVLink 1.1 will be a functionally equivalent extension to 1.0, and the core requirement is that any client understanding v1.0 will be able to parse the full 1.0 message set, which will still be contained in 1.1.

It will add a bunch of new messages, some with variable length. To successfully decode those a 1.1 parser and message set will be required, 1.0 clients will just dump them as unknown (they will parse successfully, but with unknown ID).
In theory one could harvest some bits off the component ID without introducing complete incompatibility. But I'm not sure we should do that if the intent is to allow more message IDs. It doesn't seem worth the risks.

Also note that there is active work currently done on sMAVLink. This will either be a wrapper / extension of 1.1, or it will use MAVLink v2.0 frames, but not yet with the intended payload encoding. We can discuss header changes for this purpose.


After setting the stage, please allow me to comment on the current discussion: MAVLink is built for aerial robotics and actively used for this purpose. This means that components are already intensively used, and their use will increase further. You have more and more electronics on these devices, and we're pushing currently hard to replace the motor and gimbal buses by CAN. This means that pretty soon you have much more nodes to talk to, and you don't want to keep adding things like ATTITUDE_GIMBAL when you could just send an ATTITUDE message with a different component ID. We have also had redundant IMU setups in the past.

There are also many more messages which lend themselves natively for multi-component use - e.g. think of parameters, and think of a component as one piece of software. We have on our Linux payload already now about ~10 processes where each one has its own parameter set (extremely convenient, you can control the autopilot and the full image processing from within qgroundcontrol). But even for the small PX4 unit there is work under way for scripting, where each script instance then again has a component ID. You need this in order to address encapsulated software, because without it, you would need to keep parameters per system global.

So its not just research or large systems. Its really in use and needed for anything that goes beyond what we have today.


While I see a clear use case for component IDs, its *much* less clear to me if message classes do contribute a benefit. What I got from the current discussion is that they're used to decide about routing. However, I see challenges with this: Are there really *distinct* classes for air and ground segment, or is this not rather a combination of sender and message ID plus application context? In MAVCONN, where we have a similar setup, where we've set up a message filter that allows us to "do the right thing" depending on the message ID. What the right thing is pretty much depends on the context, and it didn't require us to establish classes. The only downside is that you need to add messages manually to the filter, but we could also auto-generate the filter table, without requiring to send the class inside the message.

Mapped to this example:

- telemetry: downlink messages, generating C code and not checking for alignment of data, almost full

The autopilot has anyway manually defined which messages are sent (mapping from data to messages needs to be coded anyway), and unused messages are stripped by the compiler. It's not clear to me what the class here contributes.

- datalink: uplink messages, generating C code and checking for alignment

The GCS also has to explicitly generate the uplink messages, on the same argument its not clear to me why those would need to be part of a class.

- ground: inter-agent communication on the ground side, no C code generation, no binary format only string over Ivy software bus

You certainly want to avoid to relay those to the autopilot, so they need special handling. Since this seems to be a separate bus, one could just add a flag in IVY which encodes ground, air broadcast or one MAV ID (for the case of multiple serial links). You could also create a map from message ID's to classes.


And I'm not sure at all if the number of messages (= with classes or larger message ID range) should be extended. Not on a technical, but on a standardization argument.
For those of you who work with ROS: You might have experienced already that for the same thing ("position", "attitude"), different people use quite different messages from different packages. The limitation to 255 messages in MAVLink is also a commitment to enforce standardization and to keep evolving the protocol.
There are about 30 message IDs left in the common message set, and a few messages are probably due to be phased out / replaced over time. Each project can have on top of this another 100 message IDs.


The whole point about MAVLink is to have something quite simple, which allows interoperability. This is however only possible with a manageable, standardized and reasonably small message set. And its important that the meaning of the message fields is well-defined by the protocol (so no harvesting of payload bytes, no interpretation of other parts of the header depending on context).

For the same reason I believe that the real benefit for Paparazzi will be if the current message set is adopted, maybe first on a 2nd link. Because once you do, you unlock all the GCS and other functionality. Just using a compatible wire format will buy you some commonality in code, but you still don't get the benefits of interoperability.

-Lorenz


------------------------------------------------------
Lorenz Meier
Institute for Visual Computing
ETH Zurich
http://www.inf.ethz.ch/personal/lomeier/

On Jun 7, 2013, at 12:48 PM, Chris Gough <christoph...@gmail.com<mailto:christoph...@gmail.com>> wrote:

Gautier, I'm interested to understand the minimum change to MAVLink that would enable a successful Paparazzi integration. If three message classes are non-essential, and "ground" message classes are only ever used within ivy, then it seems like "datalink" and "telemetry" class messages are the only ones that absolutely must be passed between the autopilot and the ground segment (i.e. via MAVLink). If that's true, would a few more bits of MAVLink message id be sufficient? Does the telemetry/telecommand carrier actually need to know anything about the message class? If MAVLink could dedicate 600 message ids to Paparazzi, as a first step could Paparazzi start supporting MAVLink as a channel, using something like a modem driver. And then, gradually harmonise standard messages by depricating Paparazzi messages that overlap with standard/common MAVLink ones?

Chris Gough

Um Ihr Abonnement für diese Gruppe zu beenden und keine E-Mails mehr von dieser Gruppe zu erhalten, senden Sie eine Email an mavlink+u...@googlegroups.com<mailto:mavlink+u...@googlegroups.com>.
Weitere Optionen: <https://groups.google.com/groups/opt_out> https://groups.google.com/groups/opt_out

--
Sie haben diese Nachricht erhalten, weil Sie der Google Groups-Gruppe MAVLink beigetreten sind.

Um Ihr Abonnement für diese Gruppe zu beenden und keine E-Mails mehr von dieser Gruppe zu erhalten, senden Sie eine Email an mavlink+u...@googlegroups.com<mailto:mavlink+u...@googlegroups.com>.
Weitere Optionen: https://groups.google.com/groups/opt_out

[Christophe De Wagter]

Hi Lorenz,

I like the thinking behind MavLink as described in your last post and clearly understand the reasoning.

Here is my list of problems encountered so far:

1) I had one comment about this interoperability you mention and was wondering if that could be taken into account as well in the next MavLink revision. There seems to be significant differences between the many groundstations in terms of required messages, message order, message logic etc.. Just as an example: QGroundcontrol will already show a new plane with just a HeartBeat while APM will only start if PARAM is sent as well and for things like AndroPilot ... well we don't even know yet as we never managed to see any uav position or even attitude move on the screen with the exact same data stream where QGround and APM work fine. Some ground stations apparently are waiting for a specific message, but not telling you in any way. Defining a message flow, minimal message sets, message work-flows or something like that seem very important to me.

2) On the navigation point of view we only see paparazzi compatibility with MavLink by rewriting the flightplan part. Uploading a list of waypoints is still too far away from the paparazzi-way of flightplan scripting language (Example of one of our flightplans: wait for gps, wait for arming, detect catapult, enable prop 234ms after 3g acceleration detection, start with constant pitch/throttle until speed reaches v-cuise, climbout on current heading, 35m above the ground level deroute to next, when at altitude make a 360 to evaluate the wind, the turn on the camera, start a polygon-survey from downwind to upwind, with 60% photo overlap (computed from airframe camera parameters), at 1mm pixel ground-size and triggering on the straight lines only every 60% overlap meters, and stop triggering in the turns, when turning follow a circle tangent to the exit and entry line, then when the polygon survey is finished, do it again with 50% shift, and when that one is finished, turn off the camera and proceed to downwind, which is computed from the touchdown point, baseleg altitude with the glide properties of the plane and has final in the direction of the take-off, then circle the base-circle until altitude is good and the heading in the direction of final and reduce the airspeed to landing speed, now extend the speed-brakes (e.g. butterfly-crowl) and proceed final, and at 10m above ground reduce the airspeed to landing speed and on impact close the throttle.) What does a groundstation need to do: ARM, set corners of polygon and tough-down spot, define left or right downwind, change image overlap/sidelap/resolution. While flying the autopilot constantly sends down on which line (x1,y1/x2,y2) or circle (x,y,r) segment he is. [http://wiki.paparazziuav.org/wiki/File:PolySurvey30DegreeEx.png]

So if any of these could be considered (1 on the short term) (2 keep in mind for the [hopefully not too] long term) that would help a lot.

Most other implementation specific issues seem to have been discussed before. 

[Gautier Hattenberger]

Hello Lorenz,

Thank you for clarifying the usage of the component id. It really makes sense, especially with the background of the pixhawk team in computer vision (which means carrying linux type processors and all sort of payload). We (at Enac at least) are usually aiming to make really small autopilots for small MAVs and we barely never use extra embedded computers nor redundant components (which may change in future projects...). So our main problem is the huge mess of our messages set. Everyone has added its own messages over the years. Now, useful messages are mixed with custom payload messages and their is not a lot of available ids anymore. So our need was to sort the messages and increase the number of possible messages. That's why I preferred to go with a class id (we already had classes anyway).
Based on what you said, we should probably consider using component id, or splitting the byte into class and component id.

About interoperability, as Christophe said, our way of managing flight plans is currently too different to easily integrate it.

We will probably support mavlink (at least a part of it) some day by actually importing it and using your generated code rather than trying to mix our system with mavlink. Describing the "correct" way to initialize the communication with GCSs would definitely be a great help.

Gautier

[bill king]

Hi, Lorenz

I suggest new mavlink be timestamped

2013/5/15 Meier Lorenz <l...@inf.ethz.ch>

--
Sie haben diese Nachricht erhalten, weil Sie der Google Groups-Gruppe MAVLink beigetreten sind.

[crashmatt]

Should you automatically timestamp the message or should you use the timestamp of the data that is sent?

A standard timestamp field would also add unnecessary burden on bandwidth limited data links.  Maybe something to think about with the variable width version.