Possible compass bug, flyaway on ArduCopter master

[mlloyd]

My setup:

* Quad

* Pixhawk

* ArduCopter built from master

* VR uBlox M8 GPS, which includes an HMC5983 compass

* EKF enabled

On my most recent flight, my quad started acting strangely immediately after take-off, then started to fly around a 50 foot circle without my command (in loiter). I was lucky and managed to land safely before it did any damage. Before this flight, I had about 10 completely normal flights with *exactly* the same setup - nothing changed at all.

I looked at the log, and something very strange happened to the compass readings, which I'm pretty sure explains the flyaway. In the normal 10 flights, Mag.Z is positive, around 600-700 (see first graph). In the flyaway, right from the start Mag.Z is negative, around -400. The inverted values persist during the entire flight, even at altitude, so I don't think it was caused by nearby metal or other interference. It looks like a compass driver bug to me, maybe a bad rotation, or perhaps a self-calibration failure in the HMC5983. It doesn't look like failover to the internal compass, AC seems to be using the external compass throughout the flight.

To prevent it happening again, I've added this pre-arm check:

        // check that the Z axis is positive and in the 400-800 range

        if (mag.z < 400 || mag.z > 800) {

            if (display_failure) {

                gcs_send_text_P(SEVERITY_HIGH,PSTR("PreArm: Check mag z"));

            }

            return;

        }

Dataflash logs attached :)

[Paul Riseborough]

This check will not work as the Z component of the earths magnetic field changes sign with hemisphere, being ~0 at the equator.

This type of error could also affect the X or Y readings as well. I've been thinking about doing something about this type of failure in the  EKF which on start-up forms an estimate of the three components of earths magnetic field. This could be compared either values stored in non-volatile memory at the end of the last successful flight (minimum number of seconds in an auto mode?). It could also be compared to values in a database, but local variations might cause an unacceptable percentage of false positives.

[mlloyd]

Such a check would be fantastic. Maybe we could use a parsimonious model like the IGRF, which needs a couple hundred coefficients, to predict the magnetic field as a function of latlon. That might be overkill - comparing against the last flight would be pretty good too, though that would break for frequent travelers.

I wonder whether the underlying cause here was a hardware failure in the HMC5983 or a software bug...

[Randy Mackay]

Mlloyd,

 

      Thanks for this, nice analysis and very interesting.

 

     Some things pulled from the logs:

-          The external compass’s x,y,z are rotated.  X has become Z, Y has become X, Z has become Y.

-          The compass orientation seems fine although we can only see what the orientation was when the vehicle started up.  So if you accidentally changed the orientation after startup, we would not be able to see that.

-          Only the external compass is affected, the internal one is fine

 

     The pre-arm check is a good idea and we will add that if we can’t find the source of the issue.  It could be for example that during start-up it failed to write a setting to the compass and we didn’t catch the failure.

 

-Randy

--
You received this message because you are subscribed to the Google Groups "drones-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Randy Mackay]

Mlloyd,

 

     When Tridge and I were reviewing we found that the hashtag doesn’t match anything from master.  So I guess you’ve made some changes to the software?

[mlloyd]

Randy,

Thanks so much for taking a look!

I'm pretty sure I didn't change the orientation of the compass after start-up. I fly with one of the 3DR folding GPS stands, but I checked after landing that it hadn't accidentally folded up. Other than that I didn't connect a ground station or change any settings after start-up.

I fly with a few of my own additions, nothing which affects the compass logic though. I recently rebased them on top of ardupilot commit e9a9e3328091ab321027d951c7c52600ec60d94c, so that's effectively what I'm flying with. The PX4Firmware and PX4Nuttx versions are directly from the diydrones github repo, commits c3a556d6bb12bfce4d3686b41660182fb48667ea and eba6b56f6a199648fd85498fdebcc3d230929acf respectively. The changes I made are entirely unrelated to the compass - RC sanity checks, send home location to my MinimOSD on GPS lock, add RSSI_RC, add autotune for yaw axis, add high-rate motor/throttle PID logging for offline PID loop tuning, restrict roll/pitch to unit circle, debounce ch7/ch8 switches, an experimental battery state-of-charge estimator, and Paul's recent EKF smoothing PR which I cherry-picked in.

Mlloyd

[Randy Mackay]

Mlloyd,

 

     Cool, thanks for the confirmation.  We didn’t think it would be related but just wanted to check.

 

-Randy

[Randy Mackay]

Mlloyd,

 

     Can you send a link to the repository that you’re using?  We’d just like to review your changes and make sure they’re not related.

 

     Txs again for reporting this!

 

-Randy

 

From: Randy Mackay [mailto:rmac...@yahoo.com]
Sent: August 26, 2014 12:16 PM
To: 'drones-...@googlegroups.com'
Subject: RE: [Bulk] [drones-discuss] Re: Possible compass bug, flyaway on ArduCopter master

 

Mlloyd,

 

     Cool, thanks for the confirmation.  We didn’t think it would be related but just wanted to check.

 

-Randy

 

From: drones-...@googlegroups.com [mailto:drones-...@googlegroups.com] On Behalf Of mlloyd

Sent: August 26, 2014 11:41 AM
To: drones-...@googlegroups.com

[Randy Mackay]

Mlloyd,

 

     On extra bit of info is that I see your comment about using the VRGPS+Compass  which has the HMC5983 compass on it which is different from the more common 3DR GPS + compass module which uses the HMC5883.

 

     The VRGPS+Compass’s field length (see Pixhawk_VRGPSCompass.png) seems to be twice that of the 3DR GPS+compass module (see Pixhawk_3DRGPSCompass.png).  From looking at the datasheets I don’t see why they would be different.

 

-Randy

 

From: Randy Mackay [mailto:rmac...@yahoo.com]
Sent: August 26, 2014 2:02 PM
To: 'drones-...@googlegroups.com'
Subject: RE: [Bulk] [drones-discuss] Re: Possible compass bug, flyaway on ArduCopter master

 

Mlloyd,

 

     Can you send a link to the repository that you’re using?  We’d just like to review your changes and make sure they’re not related.

 

     Txs again for reporting this!

 

-Randy

 

From: Randy Mackay [mailto:rmac...@yahoo.com]

Sent: August 26, 2014 12:16 PM

To: 'drones-...@googlegroups.com'
Subject: RE: [Bulk] [drones-discuss] Re: Possible compass bug, flyaway on ArduCopter master

 

Mlloyd,

 

     Cool, thanks for the confirmation.  We didn’t think it would be related but just wanted to check.

 

-Randy

 

From: drones-...@googlegroups.com [mailto:drones-...@googlegroups.com] On Behalf Of mlloyd

Sent: August 26, 2014 11:41 AM
To: drones-...@googlegroups.com

[Roberto Navoni]

Log in California.

2014-08-26 14:37 GMT+02:00 Roberto Navoni <lase...@gmail.com>:

Hi Randy ,

this is my log in Italy and in California : 

Doing with VR micro Brain and VR GPS 8 that use HMC5983 as mag chipset.

Best

Roberto 

[Roberto Navoni]

Log in Italy

2014-08-26 14:37 GMT+02:00 Roberto Navoni <lase...@gmail.com>:

Hi Randy ,

this is my log in Italy and in California : 

Doing with VR micro Brain and VR GPS 8 that use HMC5983 as mag chipset.

Best

Roberto 

2014-08-26 13:42 GMT+02:00 'Randy Mackay' via drones-discuss <drones-...@googlegroups.com>:

[Randy Mackay]

Roberto,

     Ok, your compass’s magfield seems to be closer to the 500 we expect.

 

Mlloyd,

     Until we find the cause, your idea for a pre-arm check is good so I’ve added a check of the internal vs external compass direction.  If they’re off by more than 45 degrees it will fail the pre-arm compass check.

             https://github.com/diydrones/ardupilot/commit/eb51a8e5da0ac5d53e99104baf9018823c9b7092

 

-Randy

[mlloyd]

Awesome, thanks for adding that pre-arm check, Randy! Great idea on comparing the internal and external compasses. That should definitely prevent a recurrence of my flyaway.

In case it helps, I'm attaching a log from the same site but with my old 3DR GPS + HMC5883, which I was using up until a few days ago. At this site, my mag field length with the HMC5883 was typically around 630. With the HMC5983 it's around 660 (on a normal flight), so it's roughly the same for me. These numbers are using offset adjusted readings, with offsets optimized using a Matlab script, so they're pretty accurate.

From: drones-...@googlegroups.com [mailto:dro...@googlegroups.com] On Behalf Of mlloyd