scheduler boosting in master for ArduPilot

[Andrew Tridgell]

Hi All,

As we have already discussed, the recent merge of the PX4 upstream
greatly improved the scheduling performance of ArduPilot, reducing the
number of scheduling misses. Whereas we frequently used to see some main
loop times of well over 4ms for copter (which aims for 2.5ms loop
times), we now rarely see loops over 3ms.

I've now pushed in another change which improves the scheduling
performance some more, so we now rarely see a loop time over 2.6ms.

The change may seem a little strange at first, so this email is a
heads-up for people who want to understand it. I've CCd David and Lorenz
as it covers some areas they are looking at.

The APM:Copter code now uses about 17% of the CPU on Pixhawk. This means
it finishes each main loop after about 500usec, then it goes to sleep
for around 2ms. The main cause of the delays of around 500usec is when
the hpwork thread decides to wake up and do some work at exactly the
wrong time - if it wakes up just before the 2ms sleep has expired and
then does 400usec of work then we will see the main loop delayed by
400usec.

This happens because the hpwork thread runs at a very high priority of
240, which is higher than the APM main loop (at 180). The hpwork thread
runs the following tasks:

- px4io communications (high speed serial)
- uavcan communications (canbus)
- most I2C communication (mag, airspeed etc)
- ms5611 SPI communication

These operations are quite fast individually, but if they happen to all
want to run at the same time then the APM main loop is delayed.

The solution I've just pushed to master is to add a new AP_HAL scheduler
function called delay_microseconds_boost(). That is like
delay_microseconds(), but when the timeout expires it temporarily boosts
the priority of the APM main thread to 241 for a period of 150
microseconds.

What that does it puts the APM main thread at a higher priority than
hpwork for 150 microseconds at the start of each loop. This allows it to
complete the task of gathering IMU sensor data and start the AHRS update
run without interference from hpwork. This makes a large difference to
loop timing consistency, especially if you have a lot of hpwork sensors
attached.

So if you see the main thread running at a higher priority sometimes,
you now know why it happens!

Cheers, Tridge

[Chris Anderson]

Thrilling news! When is this scheduled (so to speak) to go into a public release candidate for copter and plane?

-c

--
You received this message because you are subscribed to the Google Groups "drones-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Chris Anderson

CEO, 3D Robotics

ch...@3drobotics.com

[Lorenz Meier]

Hi Chris,

I’m happy that  the benefits RTOS scheduling with priorities has now been embraced. Despite the implication of „loop overruns“ being an issue, there is not a direct link to flight performance, rather, it is an arbitrary metric (when we talk about 1-4 ms, not 10s of ms). So I’m not sure what users would gain. I can see that less variance is good, which is an entirely different discussion. Flight-performance wise the SD card fix to not run a busy wait loop in NuttX from a few weeks ago was much more relevant, since it was in the 10s of milliseconds range.

Keep in mind that the propeller needs 10 ms to make a full revolution, and it can’t change speed every quarter rotation. It is effectively a mechanical low pass filter.

I have to constantly bite my tongue for most timing / latency related discussions, because they really should be looked at from the controls perspective (who in this community knows the dead-time of his ESC?), and the discussion is commonly around arbitrary metrics. In fact, the mediocre level of interest in UAVCAN is stunning, given it takes much longer to transmit the PWM pulse than to calculate the control response, and its the quickest way to reduce latency in the system.

If you have seen some recent advertisements for some multicopters with „800 Hz control“ please keep in mind that Hertz are the Megapixels of multicopters (I’m *not* talking about estimation / filtering at high rate, which is entirely different), and are similarly „useful“ in judging performance.

Cheers,

Lorenz

[Chris Anderson]

"Hertz are the Megapixels of Multicopters". Best line of the day!

-c

[Andrew Tridgell]

Hi Chris,

> Thrilling news! When is this scheduled (so to speak) to go into a public
> release candidate for copter and plane?

I expect it will be in a plane release in March. We're in a rapid
development phase at the moment with lots of new fetures going in, so
that will take time to settle.

How soon it gets into copter probably depends on whether the scheduling
changes actually make a really noticeable difference to the flight
characteristics of copters. If it does then I imagine it may be pushed
out a bit faster :-)

I'll be very interested to hear if it does help on copters, especially
smaller copters like Leonard is working on.

Cheers, Tridge

[Andrew Tridgell]

Hi Lorenz,

> Despite the implication of „loop overruns“ being an issue, there is
> not a direct link to flight performance, rather, it is an arbitrary
> metric (when we talk about 1-4 ms, not 10s of ms). So I’m not sure
> what users would gain.

There is reason to think it may make a noticable difference as the
timing jitter may not have been average zero over longer periods of
time. The timing of the various components could line up for many
samples in a row causing the scheduling misses to come in bunches (you
could see this on logic traces). The time between samples is then used
in the gyro integration, so it is quite plausible that this would show
up as both drift and jitter in the gyro bias estimate.

I'll be especially interested to hear from Leonard if it helps with his
very small racing quads.

Cheers, Tridge

[Randy Mackay]

     As for when it'll go out for copter, this will certainly be in AC3.3.  We've still got a fair chunk of work to do to get that out as the official release although we're hoping to get some early release candidates out to beta testers over the next two weeks.

-Randy

Cheers, Tridge

--
You received this message because you are subscribed to the Google Groups "drones-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to drones-discuss+unsub...@googlegroups.com.

[Robert Lefebvre]

Lorenz, you must be very pleased to know that Jeti's new EXbus guarantees 10ms frame-rate to the controller, allowing 100Hz update rates from the pilot's fingers. ;)

[Paul Riseborough]

This is good news for EKF development as we should  now be able to run the various prediction and updates steps concurrently without worrying about over-runs. The need to split operations across multiple frames has driven complexity into the EKF code which makes it harder to maintain and reduces stability. For example we were previously unable to run covariance prediction on the same frame that we fused magnetometer data. I'll check the utilisation and start on a cleanup of the EKF to remove any unnecessary splitting of operations.

-Paul