log download over MAVLink

[Andrew Tridgell]

Hi Lorenz and Michael,

I've created a set of patches to make it possible to download SD card or
dataflash logs over MAVLink. The idea is to avoid having to boot into
the CLI on APM to download logs from the APM2, and also to make it
possible to pull all the binary logs off a PX4 without having to pull
the microSD card out of the board.

I've implemented the client side of this in MAVProxy for testing, and
I'd now like to get comments from the two of you on the protocol
changes, as it adds 6 new MAVLink messages to common.xml. I include the
new XML code below for you to get an idea of the protocol.

Lorenz, are you happy to have these 6 new messages in common.xml?

MichaelO, are you OK with the protocol structure and do you think you
could implement it reasonably in MissionPlanner? Bill, can you have a
look too?

One of the motivations for this work is it is another step towards
getting rid of the CLI on the APM2, which would save us a lot of flash
space. I also want to fix problems with the text download of logs via
the CLI, which leads to log corruption, doesn't work over a radio and is
generally ugly :-)

Full implementation is here:

https://github.com/tridge/ardupilot/tree/mavlink-log-wip
https://github.com/tridge/MAVProxy/commits/master

Cheers, Tridge


<message id="117" name="LOG_REQUEST_LIST">
<description>Request a list of available logs. On some systems calling this may stop on-board logging until LOG_REQUEST_END is called.</description>
<field type="uint8_t" name="target_system">System ID</field>
<field type="uint8_t" name="target_component">Component ID</field>
<field type="uint16_t" name="start">First log id (0 for first available)</field>
<field type="uint16_t" name="end">Last log id (0xffff for last available)</field>
</message>
<message id="118" name="LOG_ENTRY">
<description>Reply to LOG_REQUEST_LIST</description>
<field type="uint16_t" name="id">Log id</field>
<field type="uint16_t" name="num_logs">Total number of logs</field>
<field type="uint16_t" name="last_log_num">High log number</field>
<field type="uint32_t" name="time_utc">UTC timestamp of log in seconds since 1970, or 0 if not available</field>
<field type="uint32_t" name="size">Size of the log (may be approximate) in bytes</field>
</message>
<message id="119" name="LOG_REQUEST_DATA">
<description>Request a chunk of a log</description>
<field type="uint8_t" name="target_system">System ID</field>
<field type="uint8_t" name="target_component">Component ID</field>
<field type="uint16_t" name="id">Log id (from LOG_ENTRY reply)</field>
<field type="uint32_t" name="ofs">Offset into the log</field>
<field type="uint32_t" name="count">Number of bytes</field>
</message>
<message id="120" name="LOG_DATA">
<description>Reply to LOG_REQUEST_DATA</description>
<field type="uint16_t" name="id">Log id (from LOG_ENTRY reply)</field>
<field type="uint32_t" name="ofs">Offset into the log</field>
<field type="uint8_t" name="count">Number of bytes (zero for end of log)</field>
<field type="uint8_t[90]" name="data">log data</field>
</message>
<message id="121" name="LOG_ERASE">
<description>Erase all logs</description>
<field type="uint8_t" name="target_system">System ID</field>
<field type="uint8_t" name="target_component">Component ID</field>
</message>
<message id="122" name="LOG_REQUEST_END">
<description>Stop log transfer and resume normal logging</description>
<field type="uint8_t" name="target_system">System ID</field>
<field type="uint8_t" name="target_component">Component ID</field>
</message>

[Meier Lorenz]

Hi Tridge,

I think it makes sense to have these in the common message set. I do not understand this comment though - why approximate and how can you manage the transfer, which is offset based, without knowing when you’ve received everything (or do you use some soft matching of sizes at the end)?

> <field type="uint32_t" name="size">Size of the log (may be approximate) in bytes</field>

-Lorenz

------------------------------------------------------
Lorenz Meier
Computer Vision and Geometry Group
Institute for Visual Computing
ETH Zurich
http://www.inf.ethz.ch/personal/lomeier/

> --
> You received this message because you are subscribed to the Google Groups "drones-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Meier Lorenz]

Follow up:

- I’ve put the new messages on MAVLink master
- I’ve started a branch on QGroundControl which has the first commits to integrate this. I plan to finish this later today. I have made it *very* cherry-pick friendly with small orthogonal commits. The ones that could conflict come first, then the main work on the new classes is in the later commits. You’re warmly invited Bill.
https://github.com/mavlink/qgroundcontrol/pull/432

-Lorenz


------------------------------------------------------
Lorenz Meier
Computer Vision and Geometry Group
Institute for Visual Computing
ETH Zurich
http://www.inf.ethz.ch/personal/lomeier/

[Andrew Tridgell]

Hi Lorenz,

> I think it makes sense to have these in the common message set. I do
> not understand this comment though - why approximate and how can you
> manage the transfer, which is offset based, without knowing when
> you’ve received everything (or do you use some soft matching of sizes
> at the end)?

It is approximate as the APM2 has no way to accurately know the exact
size of the log due to the block nature of the dataflash storage. On the
PX4 it is exact.

The end of a log is indicated by receiving a reply to a LOG_DATA message
with less than the full payload size (ie. less than 90 bytes). If the
log is a multiple of 90 bytes long then you should recieve a LOG_DATA
with zero bytes in it at the end.

Cheers, Tridge

[Andrew Tridgell]

I should also mention that it is the responsibility of the GCS to fill
in any missing pieces in the log due to lost packets. I implemented this
in the mavproxy python code by using a set, with each element in the set
corresponding to a 90 byte chunk of the file.

When mavproxy doesn't see a new LOG_DATA message for 0.5 seconds while a
transfer is in progress is starts asking for the missing pieces of the
set, 10 chunks at a time.

This allows log transfers over lossy radio links.

Cheers, Tridge

[Arthur Benemann]

Thanks Tridgel. This has always been a problem, since you need access to the ardupilot board to get the logs.

When you have the protocol defined I can implement it on DroidPlanner. That way we have a very portable way to get those logs.

[Andrew Tridgell]

Hi Arthur,

> When you have the protocol defined I can implement it on
> DroidPlanner. That way we have a very portable way to get those logs.

thanks!

One thing to note is that download over a radio link will be quite slow,
and is only really practical for smaller logs unless you are very
patient. The problem is that when not on a USB cable the code has to
assume the radio link is oversubscribed so it has to slow the transfers,
and you end up losing packets. The GCS can fill in those packets, but it
does take quite a lot longer (I've found it to be about 10 times slower
downloading a log over a radio link than over USB).

Eventually I want to support using USB storage on PX4 to have really
fast log transfer, but this method gives us a good improvement for both
PX4 and APM2, while also getting us closer to getting rid of the CLI. If
we can remove the CLI then that will save us a lot of code space,
allowing the copter code to survive on APM2 for quite a bit longer.

Cheers, Tridge

[Andrew Tridgell]

I've pushed the MAVLink log download support into git master. It also
now guarantees that the logs will have FMT headers even if the dataflash
log has wrapped. This means GCS implementations don't have to worry
about logs without headers any more.

I've also updated the MAVProxy implementation to be a module to make it
easier to use as a reference implementation for the protocol:

https://github.com/tridge/MAVProxy/blob/master/MAVProxy/modules/mavproxy_log.py

Cheers, Tridge

[DanielB]

Hi Tridge.

Sounds great! 

Just a quick question, would this new patch be easily configured to support storing the logs to an Open Log (from SF, uses micro SD card, I use and find very useful/portable on my other controller boards)?

Cheers.
Daniel

[Andrew Tridgell]

Hi Daniel,

> Just a quick question, would this new patch be easily configured to support

> storing the logs to an Open Log (from SF<http://www.sparkfun.com/products/9530>,

> uses micro SD card, I use and find very useful/portable on my other
> controller boards)?

The Open Log is a serial logger. It could be used to log MAVLink by
attaching it to the 2nd telemetry port.

The logging I'm working on at the moment is not serial logging. It is
on-board flash logging, and it makes no sense to use Open Log for that.

Cheers, Tridge

[DanielB]

Hi Tridge.

Thanks for the confirmation and info.. Got it and I agree.  

I'll try attaching the Open log on APM as well.

All the best
Daniel