URGENT: Pixhawk Gyro Drift causing crashes in flight

[Oliver Volkmann]

Dear Developers, 

Chris Anderson sent me here to this group to share with you an issue that we feel is worth escalating since we have had similar crashes with 3 Pixhawks so far. Please see the information that I sent to the help department at 3DR, Brandon Basso and Chris Anderson below. The flight logs can be found here: https://drive.google.com/folderview?id=0B62edZ4l5_rOfkxrZlFzM0F1TElWd0xWd1NEVFVycTVNTVBienpqakNDcFVURWhYbzhwNTA&usp=sharing 

On March 15^th of this year we had a very strange experience where, during an Auto Mission our Pixhawk (FW3.1.5) based SteadiDrone QU4D X started drifting off its flight line increasing in roll angle and speed. We tried to intervene however could not save the vehicle from crashing. We have until now not found out why this happened even with the help from Santiago. I discussed this issue with Andreas Breitenstein (a colleague of mine) who had a similar experience in Namibia on the 13^th of March with a completely different drone equipped with a Pixhawk and FW3.1.5. During his auto mission, the drone did a very similar thing where it deviated from the flight plan and inevitably crashed. Yesterday we were testing some of the new features on another drone with a Pixhawk and FW3.2.1 and noticed that as soon as we put this vehicle down and powered it up, the horizon in mission planners HUD would drift in the roll axis constantly. At first I had thought that since I just put this drone together that perhaps the accelerometer calibration was not done properly and so repeated it in the field. We then flew 3 very nice flights in auto and tested some of the features. At the end of the 3^rd flight, we landed and immediately noticed on the HUD that the horizon had drifted to about 45 degrees and continued to drift. We decided to see then if the pre-arm checks worked and tried to arm the unit which happened successfully even with the horizon at 45 degrees. Andreas then attempted to take-off to see if it was possible and the unit did fly, albeit, as expected it flew sideways and hit the dirt about 3 feet away.

 We then re-started the drone and the horizon went back to normal. I enabled EKF as we have never flown with that and believed it to have far more safety features and that it would produce a far more stable flight. It was quite impressive and I flew the drone around quite a bit in POS-HOLD mode. As we stood there with the drone in POS-HOLD waiting for the battery failsafe to kick in, the drone started rolling to its right and flying away by itself even while I tried to roll left. It continued to fly away at which point I put it in Stabilize to try and prevent it from flying too far away however even Stabilize mode did not seem to help.

 We have looked at the log files of all three flights which you will find attached to this email and have discovered that at the point at which each of these 3 flights went bad, there was a deviation between the two IMUs Gyros in either  X1 to X2 and Y1 to Y2. We are now very concerned because we have a lot of drones out with our customers who have Pixhawks in them and this issue seems to happen randomly. All three drones (multi-rotors) which had the crashes all had different firmware, were flown in different locations and with different hardware/electronic components. The only thing that they share is that they all had Pixhawks on board and that they all experienced this strange Fly Away behavior which could not be stopped even when switching to stabilize.

Could you please take a look at these logs and let us know what is going on? We are pretty confident that this is not a firmware issue nor a stability algorithm issue as the firmware versions vary and we have had this experience with EKF on as well. We are at a loss as to what to do now other than test each and every Pixhawk that we (and our various clients) have on a crash test drone as this issue happens randomly. Sometimes it happens when it is powered via USB on the desk, sometimes before take-off and sometimes during the flights which result in crashes. We don’t know how many flights to do to replicate the issue but will keep the logs of all of the upcoming tests and send them to you as we have them.

 If there are any tests or if there is any other information that you may need such as serial numbers please do not hesitate to let us know. Perhaps these pixhawks are faulty and could be associated with a batch production in which case it may be easier to determine which pixhawks suffer from this strange behavior.

 

Thank you in advance for your prompt attention to this matter and we look forward to hearing back from you soon.

Regards,

Oliver Volkmann

Micro Aerial Projects L.L.C.

[Randy Mackay]

Oliver,

 

     So this is what we call “the leans” when it occurs in the roll-pitch.  It can be caused by:

1.       very bad vibration which leads to inaccurate accelerometer values which leads to the attitude not being corrected properly by the accelerometer values.

2.       Inaccurate gyro bias which can be caused by:

a)      Vehicle being jostled during the calibration at startup or during the first arming

b)      A bug/issue in AC3.1.5 (and earlier versions) in which we did not reset the gyro bias estimate when we redid a gyro calibration.  The effect was most obvious if the vehicle was left for a few minutes between when the battery was plugged in and the first arming/take-off.  During these few minutes DCM/EKF would learn the gyro biases but those biases would shift to zero as part of the first arming calibration meaning DCM/EKF would incorrectly learned biases for another few minutes (until it “unlearn” them).  Normally we only saw this issue in yaw but it could happen in roll/pitch.

c)       Large temperature variations between the calibration and the rest of the flight.

 

     From the logs #1 is not the cause because the vibes appear pretty low so I think it’s 2a, 2b or 2c.

 

     In the AC3.2.1 log it appears to me that it switched automatically from EKF to DCM right at the end of the flight.  DCM’s roll estimate was off by 20deg so that switch wasn’t good although the EKF only gives up and hands back control to DCM if it’s having serious problems.  I’ve asked Paul Riseborough if he can take a look.

 

-Randy

--
You received this message because you are subscribed to the Google Groups "drones-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Randy Mackay]

Andreas,

 

     Paul Riseborough (EKF expert) had a look at the logs and determined that in fact, it’s #1 (vibration).  The high frequency vibration doesn’t show up well in the logs so it can only be seen indirectly which is why I missed it.  The EKF is happy until the end but then at the end it develops a bad Z axis accel bias and the EKF becomes so unhappy with the data consistency that it’s flags itself as unhealthy and hands control to DCM .. which sadly was in even worse shape.

 

     With a lot of vibration we get accelerometer clipping (where the accelerations go outside the 8G accelerometer range) and aliasing.  When this happens DCM/EKF get an incorrect angle from the accels and then can’t correct the attitude estimate proper and we get “the leans”.  Sometimes the clipping/aliasing appears only when the mounting/airframe/motors hit the right resonance which is why it can come and go.  For example a particular throttle speed can be a factor.

    

     There’s two things that could make it better:

Revisit the vehicle’s vibration isolation.  We recommend:

·         3M vibration isolating foam sold by 3dr: https://store.3drobotics.com/products/pixhawk-foam

·         Paul says the pink vibration foam is as good or better than the 3M foam but it requires double sided tape as well

Upgrade to AC3.3 when it’s released (currently in beta testing – you can try it now if you want).  We think this version will be more resilient to vibration because we’ve increased the accel range from 8G to 16G which will reduce the chance of clipping.  We’ve also reworked the filtering (I’m not qualified to explain the new scheme but Paul or Leonard could).

 

     On the code side there’s two things that we could do to help you and other people facing this problem:

·         Add in-flight vibration monitoring (i.e. look for clipping) so that we can alert the pilot that vibes are getting bad hopefully before it affects flight.

·         Stop the EKF from handing control to DCM unless DCM reports it is happy.  In your AC3.2.1 crash log DCM was not happy (i.e. ATT message’s ErrRP field was 0.8 which is bad, 0.1 is good).

 

     So this info from Paul changes things and probably some of your concerns but here’s some answers to your questions in any case:

·         The vibration issue isn’t related (as far as we know) to whether you’re using an APM or Pixhawk.  Theoretically the Pixhawk should be better because it has two IMUs that operate at different frequencies meaning it should be less susceptible to aliasing.

·         Re 2c, this failure is a bit theoretical.  We know temperature affect the gyro drift but we actually in the thousands of logs we’ve looked at, we have not identified a case of this causing a crash.

·         Both DCM and EKF can learn the gyro drift at up to 1deg/sec per minute so generally when we’ve seen problems caused by the gyros, it’s been early in the flight (within the first few minutes) before DCM/EKF have had time to learn the offsets.

 

     So, I think in the short-term looking at the vibration isolation is the best move, then move to AC3.3 when it’s released.

 

     Sorry for your troubles, hope this helps some.

 

-Randy

 

From: Andreas Breitenstein [mailto:and...@aeromapix.com]
Sent: 24-Apr-15 6:15 PM
To: 'Randy Mackay'; drones-...@googlegroups.com
Cc: ch...@3drobotics.com; 'Micro Aerial Projects L.L.C.'
Subject: RE: [drones-discuss] URGENT: Pixhawk Gyro Drift causing crashes in flight

 

Hi Randy,

 

All this is a bit disturbing especially as this is not really documented anywhere and that this can apparently happen during flight at any time.

 

I have some questions concerning the problem that arise on point 2(c).

What do you suggest one can do about this?

 

Also, have you got any Idea if this is also a problem on the Arduplane code?

We often fly for long periods of time (up to 2 hours) and this lean would be a disaster if the plane or hardware heat up or more than likely cool down during flight.

 

Is this fault present on the Pixhawk only? I have some copters on APM and they have been flawless for the past 2-3 years now.

 

Thank for your prompt response.

 

kind regards  

  

 

Andreas Breitenstein

AEROmapix UAV services

www.aeromapix.com

P. O. Box 90725

Windhoek

Tel: 0811294050

---

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.5863 / Virus Database: 4334/9611 - Release Date: 04/24/15

[Radek Voltr]

Is there way how to verify high frequency vibrations before I will turn EKF on ?

[Andy Piper]

On Friday, 24 April 2015 15:03:25 UTC+1, Randy Mackay wrote:

·         Add in-flight vibration monitoring (i.e. look for clipping) so that we can alert the pilot that vibes are getting bad hopefully before it affects flight.

+1000

I'm fairly new to all of this, but clearly bad compass messes you up and bad vibrations or IMU messes you up - both potentially diastrously. Any in-flight diagnosis is a huge step in the right direction. When I look through the code I see a lot "if this thing is bad do this other thing to prevent disaster", which is good from a defensive programming point of view but has two bad consequences:

[Andy Piper]

[I accidently hit POST!]

1. It masks hardware failure. Most people want to know about hardware failure so that they can beat up the supplier - not crashing with hardware failure is good, not knowing is worse IMO.
2. The transition between alternatives can be confusing or lethal, especially if the alternative is only less-bad. Often I would prefer to know that this happened and immediately land, rather than cope and carry on.

[FWIW The roll right fly-away has happened to me and others and my issue is compass rather than vibes.]

andy

[Randy Mackay]

Andreas,

 

      Ok, the bench test issue you’re seeing is related to a gyro calibration and/or gyro drift  (could be 2a or 2b) plus the way we changing the weighting of accels vs gyros when armed vs disarmed.  So the HUD rolling as soon as the vehicle is armed can happen because while disarmed (and landed) we give higher priority to the accelerometers than we do during flight.  While on the ground the correction from the accels can compensate for the bad gyro drift/calibration but once the vehicle is armed and we weaken the accel correction the gyro drift causes the attitude estimate to rotate away from reality.

 

     We do this change in accel/gyro weighting (only in copter) because during flight the accelerometers are more susceptible to vibration than the gyros but while on the ground (during startup or brief periods of being landed) we want to correct the HUD back to reality as quickly as possible.

 

     You’re being careful not to jostle the copter while the pixhawk led is flashing red/blue during startup?

 

@Andy,

     I agree with your point that we should not mask hardware failures.  I think with AC3.2.1 if the MPU6k fails we alert the user by setting the IMU unhealthy with the MP shows as “accel bad”.  For the switch between DCM/EKF you’re right that we need to make it more obvious in the logs when that happens and alert the user that things are going wrong.  AC3.3 is the first version which will have the EKF on by default so we’ve done some work including adding an EKF_STATUS_REPORT message that we hope the GCS people will implement.

            Tower: https://github.com/DroidPlanner/Tower/issues/1437

            AP Planner 2: https://github.com/diydrones/apm_planner/issues/693

            MP: https://github.com/diydrones/MissionPlanner/issues/851

 

-Randy

 

From: Andreas Breitenstein [mailto:and...@aeromapix.com]
Sent: 24-Apr-15 11:29 PM
To: 'Randy Mackay'; drones-...@googlegroups.com
Cc: ch...@3drobotics.com; 'Micro Aerial Projects L.L.C.'
Subject: RE: [drones-discuss] URGENT: Pixhawk Gyro Drift causing crashes in flight

 

Hi Randy,

 

Thank you for the hard work and the detailed answer. I can however not take this as an answer that I can live with. As far as I am concerned, there is something else going on here.

The fact that the copter that crashed on 3.2.1 showed a offset in the HUD by some 30* just as it was connected to telemetry with it standing still on the flying field. The HUD continued to turn up to some 45* until we restarted the copter. On this same day, the same copter stood still on the lawn on the field with the HUD showing level until!! I armed the copter at this point the HUD started rolling as soon as I throttled up it rolled up to 45* and the copter flipped on takeoff. This flip was not by accident, but was induced by me to see if the copter would even take off.

 

That same copter when connected to the USB on my office table would show everything in order then when armed with the radio the HUD will start rolling in a random direction. All this without the Battery even connected.

If it shows level after connecting I leave it connected for some time (maybe 10 minutes) If I look at the Mission Planner again the HUD will show a drift of some 15* to 45* at random.

 

All this has nothing to do with vibration but is a bug somewhere. I suspect that there is a faulty batch of Pixhawks out there.

I will send the Pixhawks that have this problem back to the US with Oliver and maybe you could arrange for someone to look at them.  

 

Thank you

Kind regards

[Al B]

Hi Andreas,

When you wrote "I will send the Pixhawks that have this problem back to the US", do you mean that you have other batch of Pixhawks that don't have this issue?

Also, do you see the same behavior if you load/use the PX4 Flight Control Stack instead of the APM firmware and use QGroundControl instead of Mission Planner?  That might help to isolate the root of this issue from the software standpoint.

[Al B]

Just for clarification. By seem the same behavior using the PX4 Flight Control Stack + QGroundControl, I meant this:

"That same copter when connected to the USB on my office table would show everything in order then when armed with the radio the HUD will start rolling in a random direction. If it shows level after connecting I leave it connected for some time (maybe 10 minutes) If I look at the Mission Planner again the HUD will show a drift of some 15* to 45* at random."

I didn't mean that you should try to flight using the PX4 Flight Control Stack to see if crashes.

[Andy Piper]

FWIW I think its vital that these issues are discernable from the logs. I have found that HUD messages in-flight are easy to miss - and with the compass in 3.2.1, for instance, if you miss them the data is gone. It's also easy to get these messages pre-arm, to make them go away by waiting/rebooting/shorting/doing a rain dance and to think all is well, when in actual fact it's not.

I'm looking forward to trying 3.3, sounds like it will be a big improvement.

My $0.02 :)

andy

From: Randy Mackay [mailto:...@yahoo.com]
Sent: 24 April 2015 07:41
To: drones-...@googlegroups.com
Cc: ch...@3drobotics.com; and...@aeromapix.com; 'Micro Aerial Projects L.L.C.'
Subject:

...

[Oliver Volkmann]

Hi Randy and everyone else, 

We have 3 Pixhawks that do not have this issue and have been extensively flown to test and find out if they do. These units were tested for well over an hours worth of flight time each over the past few days. We have 3 units that DO have this issue and it shows up with EKF on and with EKF off and on different firmware versions. We have tested each of these on exactly the same platform with no difference other than the Pixhawk being used. The vibration dampening has been consistently the 3DR Vibration Dampening Foam hence we really cant see that this is in fact a vibration issue. Randy, what typically happens when you have high vibration on a platform? As far as we can tell, and because we have units that do not have the leans and ones that do we can only conclude that this is in fact a hardware issue as they have all been tested with the same firmware. 

With this being said, there is a failure point somewhere that needs to be addressed. While it could possibly be solved via code, the risk of crashes is still high and concerning, especially considering the locations platforms with Pixhawks are being flown, the weights of these as well as the the evidenced fact that we cannot recover the unit when the leans take place. 

Has 3DR identified the batch of these units that have this hardware issue? If so, can the units be traced via the serial number that is found in the logs/terminal window?

Regards,

Oliver Volkmann

From: Randy Mackay [mailto:...@yahoo.com]
Sent: 24 April 2015 07:41
To: drones-...@googlegroups.com
Cc: ch...@3drobotics.com; and...@aeromapix.com; 'Micro Aerial Projects L.L.C.'
Subject:

...

[Oliver Volkmann]

Hi Al,

I am here with Andreas in Namibia (where we have ample place for the leans to occur - sorry, bad joke) so am responding on his behalf. We have not heard anything about using the PX4 Flight Control Stack nor do we know how to go about that. We are quite familiar with Pixhawk and Mission Planner and of course now our clients are too. We would be willing to try the PX4 Stack however there is a potentially long learning curve and we have no clue as to where to start. Thanks again for mentioning that we dont have to fly to see if it works. Typically that's what we are doing now with every Pixhawk we have and our clients units. So far, the success rate is a bit alarming. If you would be willing to share with us how to do the test you are suggesting we would appreciate it however I do want to get Randy (or anyone else on the code side) opinion on whether or not such a test would help in solving the problem and how it would do that.

Regards,

Oliver & Andreas

[Randy Mackay]

Oliver,

 

     If this can be reproduced on the bench then it would be good to do this:

·         Ensure the Pixhawk has AC3.2.1 loaded on it (or alternatively AC3.3-rc1 if you prefer)

·         Set the LOG_BITMASK to ALL+DisarmedLogging (ie. “131070”)

·         Restart the pixhawk and attempt to recreate the problem (i.e. set the board down, try arming and see if the HUD begins to roll).

·         Download the dataflash logs from the test and send it to Paul and I

    

     I don’t know about the serial numbers of boards with the MPU6k accelerometer issue.  Maybe Vu from 3dr could advise.  So far we haven’t seen that failure in the logs you’ve sent along though so there’s no reason to think it’s that particular MPU6k accelerometer issue that 3DR had from June-2014 to this Feb-2015.

 

-Randy

--

[Al B]

Hi Oliver,

I should have also clarified that I was not suggesting to switch to the PX4 Flight Control Stack for your product line.  I just mentioned that option because ardupilot builds on top of that stack so it would have allowed to determine if the problem manifested on the bench was specific to the ardupilot code; which is the firmware that Randy and this group lead.  However, my suggestion might now be irrelevant after I saw your previous comment saying that you have 3 Pixhawks that do not have this issue and 3 units that DO.  At this point, it is probably better wait until Paul and Randy analyze the dataflash logs they are asking for.

One question though.  When you wrote, "All this without the Battery even connected.", does it mean that you are powering the copter only with the USB from the computer when you run your bench tests?

[Oliver Volkmann]

Hi Al, 

Thanks. Yes, in the case of "All this without the Battery even connected", the pixhawk was powered only via USB.

[Oliver Volkmann]

HI Randy, 

Thanks, we are in the process of doing those tests right now. We did experience it right away with one unit however it was right as we had connected the Pixhawk to the computer to change the log bitmask so it was not recorded. Please find a log from a client of mine attached here which seems to show a complete IMU failure around the 5 minute mark. IMU1 seems to stop working all together... 

Regards,

Oliver

[Oliver Volkmann]

In case the attachment is not visible, here is a link to the folder on Google Drive where you can find the file: https://drive.google.com/folderview?id=0B62edZ4l5_rOfkxrZlFzM0F1TElWd0xWd1NEVFVycTVNTVBienpqakNDcFVURWhYbzhwNTA&usp=sharing 

It's the IMU FAIL one...

[Randy Mackay]

Oliver,

 

     Yes indeed, as you say, this log shows an IMU failure (the MPU6k).  Assuming this Pixhawk was purchased between June-2014 and Feb-2015 it should be returned to 3DR and I suspect they will send a replacement.  My understanding is the MPU6k failures are due to a manufacturing issue so on the software side, all it can do is alert the user and try and cope.  It looks like the fail-over to the 2^nd IMU worked and the pilot got the copter back hopefully in one piece?  If yes, then that’s a success!

 

     By the way, the vibes on the vehicle look pretty high and possibly the pitch tuning is a bit off.  There are places where the desired and actual pitch go off by as much as 20deg although perhaps there were some environmental factors making the control this bad.

[Oliver Volkmann]

Hi Randy, 

Thanks for the feedback on that log. I will send that unit back to 3DR. The vehicle is in one piece still so that is a success! I do know that this copter of theirs is in need of tuning... 

With this issue being an obvious hardware failure, where do we stand with the other 3 pixhawks that we have issues with? Since it happens randomly, are there any other tests we can do? We have been logging a lot now but have yet to catch the lean. Would it help if we sent these units (or one of them) that have had the lean in to 3DR for further analysis?

Best regards,

Oliver

[Randy Mackay]

Oliver,

 

     My guess is that the two issues are not related to each other.  So this most recent log was clearly an accel failure but that failure doesn’t appear in the earlier logs we saw which developed the lean.  I suspect that in this most recent failure the user didn’t see the leans right?

 

     I don’t think that sending the boards into 3DR will help get to the root cause.  With such an intermittent problem, I suspect if they look over the boards (from a hardware point of view) and nothing will turn up.  Of course if a replacement board(s) is good enough then an RMA with 3DR is a quickest solution.  Of course we would then be left with the nagging doubt of what was wrong and could it happen again.

 

     We just need to be able to reproduce the issue to nail the cause down.  My guess is it’s vibration and not being able to reproduce it on the desk is consistent with this (although it’s not proof).  If it is vibes then I suspect AC3.3 will solve the problem and will include faster logging of IMU data and will likely also include warnings of accelerometer clipping.

[Oliver Volkmann]

Hi Randy, 

So we have been running these Pixhawks that were subject to the leans for the last week plus some days and have yet to replicate the issue. I was thinking about what could possibly be causing this to start happening and wanted to run an idea/theory by you to see if you think it is an avenue worth investigating... 

With all 3 units that we have that have demonstrated the leans, they each flew for a bit, then were landed and disarmed and then armed and flown again without power having been removed. Is there a chance that perhaps that is where the issue lies? Perhaps there is something in there where the weighting between Gyro and Acc gets messed up between these multiple take-offs, disarmings and landings? 

I have thrown yet another Pixhawk on a crash testing drone and am doing this test however based on past experiences so far, this issue happens randomly which is making it very frustrating... This is the first time I can honestly say I WANT to see the thing crash so that we can get the data out of it! 

Please let me know your thoughts on my theory.

Thanks, 

Oliver

...

[Oliver Volkmann]

Randy,

In the hopes of finding some reason for the leans starting, I performed the following tests:

Here is what I did: Place quad on the ground, power it up, wait for the blinking green light, press the safety switch button, move the drone to a different location, arm and then fly. On take-off, the drone immediately started flying off to its right. I could still control it using the remote control (I was in Stabilize) however whenever I let go of the lateral control stick, it would try to fly off that way again. I then landed the drone, took power off, then powered back on and repeated the same procedure another 4 times but this time without the drone flying off as if the gyros were calibrated incorrectly. I made absolutely sure that the unit was still when the blue/red LED sequence was running. 

Upon looking at the logs, I found that one of the IMUs does not show up in the logs which is very strange. In the subsequent flight logs both IMUs are present. The pixhawk that I am flying is not one that had the leans before but perhaps it belongs to the batch with the faulty hardware. Could you please take a look at the two logs and let me know what you see? The one with the drone flying away is: 2015-05-11 09-02.log. The flight there-after is: 2015-05-11 12-10-27.log. 

I will now repeat this above test with a Pixhawk that had the leans and get back to you. 

Looking forward to your response.

Kind regards,

Oliver

[Randy Mackay]

Oliver,

 

     Ok, thanks for the determined testing.  You may have uncovered the issue!

 

     If the 1^st IMU (the mpu6k) fails to start-up (like it did in the problem log you provided), the 2ndary lsm303d will becomes the 1^st IMU but the code will use the offsets and scaling meant for the mpu6k.  In AC3.1.5 we had a similar problem with the compasses and we put a lot of effort into AC3.2.1 to ensuring this kind of error could never happen (with the compass).  I don’t think it ever crossed our minds that the same thing could happen with the accels.

 

     I’ve just done a test and confirmed that the issue happens with master.  On my particular pixhawk the lean angle difference between the two accels is only about 2.5deg but on you vehicle’s board I think it’s closer to 7 degrees so this could explain the lean.

 

     so we have a bug to fix – we should certainly ensure that all expected accels are present and if not, fail to arm or at least use the correct accel offsets and scaling.

 

      Thanks very much for this.

--

[Andy Piper]

Is this why some folk see slight lean (myself included) in 3.3 after calibration?

andy

...

[Andy Piper]

Incidentally I was mulling doing a IMU health log patch like I did for the compass. Is this worth it do you think, or is it covered elsewhere?

andy

On Monday, 11 May 2015 20:04:08 UTC+1, Randy Mackay wrote:

...

[Randy Mackay]

Andy,

 

     This IMU failure should be quite rare and I don’t think most people should see it.

 

     Immediately after the first upload of AC3.3 to the board and before the required accel calibration, I’d expect a lean but otherwise I wouldn’t expect it to lean any more than AC3.2.1 I think.

 

     For the logging, yes, we certainly need some kind of logging of accel health.  That could be an event message if an accel becomes unhealthy or it could someone log what the blend of IMU1 vs IMU2 is (that info is in the AHRS).  That blending has changed a bit - at one point it could slide up and down from 0% ~ 100% for each accel but it may be locked at 50/50 now as long as the accels are reporting healthy.

 

-Randy

--

[Oliver Volkmann]

Randy! 

Let me just say... "Phew"! I'm glad that you found something useful there with the logs I sent you... I have probably about 100 more of them but was fighting desperately to get something to go obviously wrong as I am no fundi at reading logs. 

So, with what you have discovered now, how soon do you think that a patch/fix would be available? I am sorry for the question with obvious pressure there-in however we have a lot of clients with Pixhawks who have been grounded (our call) and there are plenty of Pixhawks out there that may have this potentially fatal issue. As you already know, 3 of our drones have crashed so far because of this and the issue was not apparent on take-off but rather happened during flights. 

Thank you for your time, patience and willingness to help here Randy. I really wish I knew more about coding so that I could jump in and help but if there is something else that I can do, please do not hesitate to let me know! 

Looking forward to hearing back from you.

Best regards,

Oliver

...

[Oliver Volkmann]

Sorry, might have gotten too excited here about your findings so just wanted to confirm that this is also the reason for the in-flight failures. Do you think that one of the IMUs might fail or "restart" during flight, then the code switches to the other while the failed/restarted one "re-boots/calibrates" and then back again and hence the copter suddenly thinks that it is off level, resulting in the (dare i say) fly away? 

Regards,

Oliver

...

[Andy Piper]

On Monday, 11 May 2015 20:25:54 UTC+1, Randy Mackay wrote:

Andy,

 

     This IMU failure should be quite rare and I don’t think most people should see it.

I should admit that I have some self-interest here since I have one of the pixhawks from the known bad batch, but have not seen evidence of the known failure. I'm wondering whether the mpu6k failures can be more subtle than simply flat-lining. Could vibration cause very intermittent failure (it was a soldering issue right?) - not enough to trigger the IMU mismatch error but enough to cause subtle errors in flight.
 

 

     Immediately after the first upload of AC3.3 to the board and before the required accel calibration, I’d expect a lean but otherwise I wouldn’t expect it to lean any more than AC3.2.1 I think.

 

     For the logging, yes, we certainly need some kind of logging of accel health.  That could be an event message if an accel becomes unhealthy or it could someone log what the blend of IMU1 vs IMU2 is (that info is in the AHRS).  That blending has changed a bit - at one point it could slide up and down from 0% ~ 100% for each accel but it may be locked at 50/50 now as long as the accels are reporting healthy.

Ok I might try this if only to rule this out as a source of my problems (I'm hoping lightening doesn't strike twice!)

andy
 

 

-Randy

 

[James Harrison]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/05/2015 20:25, 'Randy Mackay' via drones-discuss wrote:
> For the logging, yes, we certainly need some kind of logging of
> accel health. That could be an event message if an accel becomes
> unhealthy
>

As a humble user, I'd certainly expect that any indication of a
hardware failure (in any system, but particularly the IMU) would be
flagged up as both an item in the logs and as an event message for my
GCS to alert me to. Depending on payload/vehicle, a failed IMU at
startup is likely to make me go back to the bench and test, test, test
(or replace) rather than fly. Which is the right outcome!

- --
Cheers,
James Harrison
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQIcBAEBAgAGBQJVURL0AAoJENTyYHL8dmp9r2gP/igt1Cs4k9YY2Tq18qM/uDHG
OWSTLyIqj/i4CCtzVdoAnOFbCJ4BowgsHVXf4H56pLBVYcf+AsrB2OYtguqvY0F7
k/1fkdKOpNgm8k8PEItcIg8xJ/zN68HpmTfGT2zviPaimD9hO/iukyqutbVKK5o/
+A42X5j4uMF0ky40AdsJGus/e3jBGxat/Ml4akBbggPoitqMW+fuucCzoKLxFl4/
TklWzUdMLA7sEKSNGelqJ0uWo4Y9Z5M5IOSvwmIS87t4Q+P6Ota1+YF65Y4Dy9Mc
yFNLOgk1pqHnRGbOphBFua8doGY4BookkcguRirtJH+EDq6UNiIn6eVFdApQtqs4
RUa/hoT5swE4c+T+S+w0ga94job1RCIP+ieYOuL3TZgR2/05n6TsON/XIhJTrTyJ
BfIvMgYfHCdU4ZUHIwcbJSWG8DHrOu9iRDCiCyPrESTlQKaE96T8MH2+kkGOEoqm
ACEV4UzmwEL6Q0CC5eBeAx3bpi5hFBeAxKePBJDJS61XeOlfxrrl3zRRTzfFM9De
FQscWd15YD/KyytNTLFQ57DDJityUtV3BwPNOaZlHb6ejbkNLRnFB6YRV5WOV2Oi
EYrwBHYQ/pkd87MAx3zgfNBgQTFNoCOLu/AajSWSjQPHGOh9TCQdnyxrZfyPIZ5g
xq4D5bb8uG2pdfZ0B4b7
=j4D0
-----END PGP SIGNATURE-----

[Andy Piper]

I'm pretty sure GCS already does this at startup - it's the in-flight logging that is missing. Compass was the same, but not any more :)

andy

[Andrew Tridgell]

Hi Randy,

The reason we accept starting with no MPU6000 is that very early Pixhawk
boards didn't have an MPU6000.

What I suggest we do is have an arming test that checks if more IMUs
have calibration values (from accelcal) than we currently have. If so
then fail to arm saying that the number of IMUs has changed.

So if we have non-zero offsets for Accel2 but only have one accel then
consider that an error and don't arm.

Cheers, Tridge

[Andrew Chapman]

> I have one of the pixhawks from the known bad batch

Is there a known batch? I missed that -- do we know a serial # range or something to be able to check? One of my Pixhawks is very suspect.

--

[Andy Piper]

See http://diydrones.com/forum/topics/pixhawk-bad-accel-health-apm-3-2-collecting-data

[Randy Mackay]

Oliver,

 

     A fix is in master now so it’ll go out with AC3.3-rc4.  This fix is what Tridge described on this thread in which we check that all calibrated accelerometers are present as part of the pre-arm checks for all vehicles.

 

     This particular bug of using the wrong offsets/scaling shouldn’t happen in-flight so it not going to directly fix that problem.

 

     In one of the earlier logs, the EKF was having problems and it gave up and handed back control to an even worse off DCM.  We still plan to make that much less likely to happen for copter at least as part of AC3.3.

 

-Randy

 

From: drones-...@googlegroups.com [mailto:drones-...@googlegroups.com] On Behalf Of Oliver Volkmann

Sent: 12-May-15 4:40 AM
To: drones-...@googlegroups.com

--

[Oliver Volkmann]

Hi Randy,

So I guess a different issue was uncovered than the one that I originally posted about. Could there be a link between the two issues? The leans in flight are of more concern to us than those on the ground for obvious reasons. Based on the in flight logs I sent, do you see and evidence of one of the IMU rebooting or failing momentarily just long enough to get different offsets and cause these leans? We still don't know how the in flight leans start. UT they happen and happen randomly. Any test we can do to find out why it is happening?

Oliver

[Randy Mackay]

Oliver,

Yes, it's likely a different issue.

From our previous analysis we thought it was vibration that was the cause. That cause fits with the evidence in that it can't be reproduced on the bench, and aliasing can happen suddenly just due to the motors hitting the right frequency with other components on the board.

There's a bunch of things in AC3.3 to improve the vibration performance including increasing the accel range, better filtering and fixing the 'whiskers' bug. We want to do a little more including adding detection of clipping and reducing the chance of falling back to DCM. So maybe the problem simply won't happen again once the vehicles move to AC3.3.

I think the best test to do is to capture some logs with the full rate IMU logging using AC3.3-rc4 which could be out as soon as this weekend. Testing that on a problem copter might show us how bad the vibration levels are on the vehicles.

-Randy

-----Original Message-----
From: drones-...@googlegroups.com [mailto:drones-...@googlegroups.com] On Behalf Of Oliver Volkmann
Sent: 12-May-15 8:33 PM
To: drones-...@googlegroups.com
Subject: RE: [drones-discuss] URGENT: Pixhawk Gyro Drift causing crashes in flight

[Brad Bosch]

Randy and Andy,

The relatively small offset after calibration that many people are seeing in 3.3 sounds like the bug I reported and fixed last weekend. It only affects frame configurations with non-zero controller orientations which explains why many don't see it. The developers don't seem to have noticed and triaged my report yet (judging by the lack of tags on the bug and pull request). https://github.com/diydrones/ardupilot/pull/2271

[Andy Piper]

Ok so I patched rc3 to show health of all the IMU's and voila, you can see the result in the attached image. I wasn't able to generate
ErrYaw like I have in other flights, so I'm not sure what this is causing - if anything, but still doesn't seem great.

So I clearly have a problem with my pixhawk in IMU2. So that's one datapoint. But I would bet a beer I am not the only one.

I'll put together a pull request for the IMU health patch.

Cheers

andy

[Andy Piper]

Here is the pull request:

https://github.com/diydrones/ardupilot/pull/2284

[Andy Piper]

For completeness here's the log file in case anyone is interested.

andy

[Andy Piper]

I find it a little strange that there are no follow-ups on this :) I have questions:

1. Why would my Gyro health be bad but not Accel health for the same IMU? Is this likely? Could it indicate software rather than hardware?
2. Is my Gyro issue the same issue Oliver is seeing?
3. Is my Gyro issue grounds for replacement by 3DR?
4. If one of the Gyro's keeps bugging out what is the net effect on flight peformance. Is it linked to how closely the two gyros agree?

I had the exact same issue with my compass which led to several hundred dollars worth of damage, so forgive my paranoia!

andy

[Tom Coyle]

Hi Andy,

I get several errors attempting to Auto Analyze your bin file.

Regards,

Tom C AVD

[Andy Piper]

Yeah I do too. Maybe that's because I added some parameters? Or maybe it's because I'm on 3.3rc3? Not sure, I tried some later logs
and encountered similar issues. I'll try one generated before my patch see if that's the same.

andy

[Andy Piper]

I get this with an earlier log that has vanilla 3.3rc3++ (i.e. without my patch) ergo I think its a problem with 3.3 and/or MP

andy

On Sunday, 17 May 2015 16:00:35 UTC+1, Tom Coyle wrote:

[Oliver Volkmann]

Hi Andy, 

Based on my experience, when the Gyros deviate and continue to do so during the flight, the platform ends up flying away as the value calibrated for "level" shifts... dont know how this works but what happened with us was that all of a sudden, the copter just started leaning and off it went! This happened in POS HOLD as well as AUTO which is really scary considering that GPS was available to keep it in position...

Randy, why would the copter experience the leans and bug off while in POS HOLD or AUTO? 

Regards,

Oliver

[Oliver Volkmann]

Hi Randy, 

I have the latest 3.3 rc4 version on the Pixhawk now and perhaps you can see something interesting here in these log files from desktop tests (powered via USB). I see a drift in the X axis between the two Gryos which is very similar to what happened at the moment that the 3 copters we had have the leans experienced prior to crashing. 

There are 3 logs, all of which can be found here: https://drive.google.com/open?id=0B62edZ4l5_rOfnpaZkRobWRPZENMQVBIbWgyVFduRGZldTJfUzZmVGgwYUFIUVl6aVctQzQ&authuser=0

Please let me know if you see something or if further tests (different ones perhaps) need to be done.

Regards,

Oliver

[Oliver Volkmann]

On another note, I keep on getting "Bad Gyro Health" now since updating to the latest beta firmware version... even after ACC calibrations... 

[Robert Lefebvre]

Exactly which log is that graph generated from?  I had a look at those 4 and none seem to match.

Try plotting those Gyro.X message, along with Baro.Temp (on a different scale, plot it on the right side) and I think you'll see what's going on.

Is this drift causing you a problem?  Or just something you noticed? I have one Pixhawk with gyro drift vs. temp that is bad enough to cause a problem.  But some amount is considered "normal".

Rob

[Andy Piper]

Oliver, it would be worth you trying rc5 when it comes out as it has some additional instrumentation around the IMU health. I think there is a good chance that you are simply seeing one IMU dropping out in flight (as per Randy's initial analysis) and given the differing offsets the leans then set in. My understanding is that the offset issue will be/is fixed, so rc5 may be better for you anyway. But if the health of your IMUs is suspect you still want to know that.

andy

[Andy Piper]

Oliver,

I see the drift, but it's not very large.
The bad gyro health is known - fixed in rc5.
Were these logs after you had calibrated? Your IMU Accel values do not seem to agree very much between the two IMUs.
Is this configured as you would fly it? The logs indicated you are using the internal pixhawk compass, you would be much safer with an external compass.

andy

[Oliver Volkmann]

Hi Rob, 

Did not upload the correct file... sorry :-/ Here is the file that is related to the pictures: https://drive.google.com/open?id=0B62edZ4l5_rOfnpaZkRobWRPZENMQVBIbWgyVFduRGZldTJfUzZmVGgwYUFIUVl6aVctQzQ&authuser=0

This instance of the drift is not causing me a problem but what has happened in the past is that we have had 3 instances where the two gyros suddenly start diverging in flight which I believe is causing the leans because shortly after they diverge, the copters just fly off and are not controllable anymore and have crashed. What I am trying to do now is to replicate what was going on in flight with these individual pixhawks (3) to see if we cant find the source of this very randomly occurring yet fatal problem.

As far as I can tell, there is not a significant correlation between temp and Gyro health in this case. 

Oliver

[Oliver Volkmann]

Hi Andy, 

Okay. Will give RC5 a try when it is out. The ACC Calibration was not performed after the beta firmware upload but was done before hand when the Pixhawk was on a drone. The Pixhawk is pretty much configured as if I was going to fly it except it does not have the external compass or any other peripherals attached as we have seen the "leans" happen on the desk in the past. I think I may just have to bite the bullet and put this Pixhawk that has had the leans back on a drone and fly until it happens again so that we can see what is really going on... Of course I will do all the calibrations prior to flying. Good thing I work near a cemetery where I dont think I'll get any complaints! 

Regards, 

Oliver

[Andy Piper]

Someone else with the leans:

http://diydrones.com/group/iris/forum/topics/iris-massive-yaw-issue

It's worth watching the video. This is on 3.3.rc4, if it happened in mid-flight I can see that there would be problems!

andy

[Randy Mackay]

Andy,

 

     Thanks for the link.  I had a look at the logs he has and I’m pretty sure he’s got two separate issues:

1.       Near fly-away with AC3.2 caused by a GPS glitch  ß he has logs and we can see the GPS glitches, there are no IMU failure.

2.       IMU failure causing the attitude to roll over  ß no logs yet

 

     Hopefully he can provide logs for the 2^nd issue because it would be interesting for Paul to have a look and see why the EKF doesn’t handle it.  We do see the AHRS bad message on the MP HUD though so the FC knows it’s sick.

 

-Randy

There are 3 logs, all of which can be found here: https://drive.google.com/open?id=0B62edZ4l5_rOfnpaZkRobWRPZENMQVBIbWgyVFduRGZldTJfUzZmVGgwYUFIUVl6aVctQzQ&authuser=0

[Andy Piper]

My thoughts exactly! It's nice that he reports that rc5 is better than 3.2.1 - kudos to Paul.

Incidentally I have upgraded to rc5 and am getting the occasional "Bad AHRS" on the bench. Will do some flights this weekend and dig a bit more.

andy

To unsubscribe from this group and stop receiving emails from it, send an email to drones-discuss+unsubscribe@googlegroups.com.

[Andy Piper]

So his new log shows the IMU divergence that Oliver sees:



The troubling thing is that the IMU shows as healthy the whole time, it's just giving bad data. I also don't understand why this much divergence cause such a massive roll, is it just that I am reading the values wrong? Is there any way that this can be detected? Seems like a hard problem, but one worth solving since the effects are so disastrous.

andy

On Saturday, 23 May 2015 07:59:47 UTC+1, Randy Mackay wrote:

[Andy Piper]

Oliver, just to close the loop on this. Other logs I have seen elsewhere show a very similar pattern (significant Gyro drift in one IMU) and I think it's fairly clear that this is also a hardware fault and IMO you should RMA the Pixhawks that show this problem to 3DR.

Unfortunately there will be no indication in the logs that the Gyro is unhealthy (even in rc5), so the only way of spotting it is looking for significant divergence while on the ground. My understanding is that the EKF will learn offsets, but I'm guessing that doesn't work too well if the offset keeps changing. Here is a graph of someone with worse problems than you:

I've no idea whether anything can be done in software other than potentially detecting the situation early. I also have no idea what would cause an IMU to behave in this way. I apparently have a faulty IMU on my Pixhawk which diverges out-of-spec with changing temperature, so maybe it's just that these devices are somewhat fragile.

Hope this helps.

andy

On Thursday, 21 May 2015 14:16:34 UTC+1, Oliver Volkmann wrote:

[Randy Mackay]

 

     Yes, here’s a graph of ErickR’s guys roll-gyro.. He’s two logs that show the same issue (and two that don’t), a quick summary:

·         the IMU6k’s roll-gyro’s offset suddenly changes by about 1.5 deg/sec after the sensor has been on for just over 1 minute.  It’s not clear if it’s always 1 minute, in Erik’s video it starts rolling after much more time has passed.

·         It’s always the same gyro that goes bad for this user.

·         The EKF doesn’t deal with the situation much better than DCM it seems, both roll over

·         No communication errors are recorded, the sensor doesn’t become unhealthy.

     We will ask Tridge and Paul to have a look tomorrow to see if they have any ideas.

     Drones-Discuss thread from Erick R:  http://diydrones.com/forum/topics/iris-massive-yaw-issue

 

-Randy

 

From: drones-...@googlegroups.com [mailto:drones-...@googlegroups.com] On Behalf Of Andy Piper

Sent: 26-May-15 9:17 PM
To: drones-...@googlegroups.com

[Oliver Volkmann]

Hi Andy, 

Thanks for your feedback here. I am in the process of trying to get these unit RMA'd however last time I sent he...@3Drobotics.com the log of one of these crashes they told me that the copter did what it was suppose to do and there was nothing wrong. At that point in time, I did not have the insight into the logs to look at the differences between IMU1 and IMU2. Will let you know how it goes. I wish that we could properly identify what is going on here but I think that hardware might be at play considering that I have a contact who had similar issues and replaced the MPU6k IMU with a new one and the problems disappeared. I will keep one of the 3 Pixhawks for further testing. 

Regards,

Oliver

[François Kneib]

Hi all,

I started to follow this discussion yesterday and I am maybe concerned by the same (or similar) problem. I have a pixhawk since January 2014, but less vibrations than Oliver. I think I have a GyrY offset, causing my copter sometimes to lean. Activating AHRS_EKF_USE = 1 caused to my copter a random, high and sudden pitch lean that I hopefully managed to compensate until now.
You can have a look at the full logs and description at my post on ardupilot forum : http://ardupilot.com/forum/viewtopic.php?f=80&t=12601
Regards,

François.

[Jeremy Hanks]

I contacted 3dr and asked them about the known hardware issue and the person who responded said that they didn't have any "known gyro issues" and that there wasn't a serial number range that is likely affected with a problem.

Is there any definitive way to find out if a genuine pixhawk is at risk of having these issues?

I worry that 3dr is not going to be doing much to remedy the problem before it causes a crash that can be proven that it was a bad gyro.

[Andy Piper]

I looked at your log and posted a response but its still in moderation. Your IMU mismatch is off the chart (3.85), you have "whiskers" in IMU1 and you vibrations look ok so it may be worth trying 3.3 but net is I believe you do have a faulty pixhawk.

andy

[Randy Mackay]

 

     As Andy says, the “whiskers” are really bad in this log (the whisker problem is actually present in AC3.2.1).  Also the PM message shows a much higher than normal number of slow loops (about 10% vs 1% ~ 4%) so something bad is happening.

 

     I agree with Andy that trying AC3.3-rc5 (using MP’s beta firmwares link) would be a good test with.

 

     It doesn’t look like the regular MPU6k failure that Pixhawk’s suffered from between June-2014 and Feb-2015 of last year but it could be an odd variant of that.

 

     So something is wrong but it’s not clear to me what it is.

 

-Randy

--

[François Kneib]

Thanks guys for your answers and log analysis.
- What is the "whisker" bug you are talking about ? I didn't found anything about this. I will try AC3.3-rc5 ASAP and show you the logs (sunny days in France at the moment, perfect for flights :-)
- I have the tools and skills (I hope) to replace the MPU6K on my Pixhawk. I thought it could save me, but I am afraid by Randy's message about slow loops ...
- Do you know how I can have more informations about those slow loops ? Can they be linked to the bad MPU6K ? Or the power module (I see that Oliver's Vcc oscillates between 5.2 and 5.25, while mine between 5.15 and 5.21) ?

[François Kneib]

Ok Randy, I get the "Whiskers" informations from this link on DIY Drones forum, the answer you made to Andrei one month ago.
I just watched an old log I have made with AC3.1.5 (1K loops between two MPs), and I have a low NLoops, between 0 and 4.

[François Kneib]

I'm back with a new AC3.3rc5 log.
I updated, reseted all the settings then made all the usual hardware calibration and configuration.
My first observation was that I was unable to set some parameters, like serial2_baud (forced to 9) or angle_max (forced to 4500). This is not the question here, I think you already know about them.
What it have now:
- my NLon seems normal I think (ouf), below 100 out of 4000 -> max 2.5% ;
- the level accuracy was as good (acceptable, though not perfect) as with AC3.2.1, ahrs_ekf_use enabled ;
- on the log, EKF1.pitch == ATT.pitch (normal because EKF is enabled ?)
- AHR2.pitch is really far from EKF1.pitch (sometimes 10° ...)
Something that is still unclear to me:
- if EKF is disabled, where do ATT and AHR2 values come from ? (respectively, IMU or IMU2 ?)
- if EKF is enabled, it seems to feed ATT which is logic. But then, can we still get the AHR separately from IMU1 and IMU2 ? And where do AHR2 get its data from ? (IMU, IMU2 ?)

Conclusions:
The flight I made today was almost good, but in this situation I can't be sure that the dangerous "high and sudden lean" will not occur any more. Regarding my log and the results of both IMUs, do you think I should try to replace it ?

Again, thanks guys, for the support but also for all the work you do on that code.

Regards,

François.

[Randy Mackay]

 

     That’s a very powerful copter you have there.  With a fresh battery it hovers at just over 30% throttle, pulls 3Gs climbs and at one point climbs at just over 15m/s.  Do the leans occur only after high speed maneuvers or at anytime?

 

     Yes, the timing is looking much better in the logs and the whiskers are gone.

 

     There’s actually a lot of noise coming from the 2^nd IMU, the LSM303d (not the mpu6k) but I’ve checked my own logs and they also show this.  We will look into this a bit more but it may not be related to your problem.

 

     With AC3.3 the EKF is always on by default, in fact, you’ll see there’s no EKF_USE parameter anymore.  For attitude estimation DCM is still running in the background and it’s estimate appears as AHRS2 in the logs.  Both EKF and DCM use all accelerometers and gyros.

 

     Unrelated to you question, I think you may find AC3.3 better than previous versions especially at full throttle because of the improve throttle curve.  I would have expected earlier versions to shake at full throttle but that should be reduced now (maybe).

 

-Randy

--

[Randy Mackay]

Francois,

 

     Paul and I had a look at your logs and there’s definitely some aliasing going on with the accelerometers which is normally a sign of high vibration on the copter.  You’re using vibration isolating foam to attach the flight controller to the frame of course?

 

-Randy

 

From: Randy Mackay [mailto:rmac...@yahoo.com]
Sent: 5-Jun-15 10:50 AM
To: 'drones-...@googlegroups.com'
Subject: RE: [Bulk] Re: [drones-discuss] URGENT: Pixhawk Gyro Drift causing crashes in flight

 

 

     That’s a very powerful copter you have there.  With a fresh battery it hovers at just over 30% throttle, pulls 3Gs climbs and at one point climbs at just over 15m/s.  Do the leans occur only after high speed maneuvers or at anytime?

 

     Yes, the timing is looking much better in the logs and the whiskers are gone.

 

     There’s actually a lot of noise coming from the 2^nd IMU, the LSM303d (not the mpu6k) but I’ve checked my own logs and they also show this.  We will look into this a bit more but it may not be related to your problem.

 

     With AC3.3 the EKF is always on by default, in fact, you’ll see there’s no EKF_USE parameter anymore.  For attitude estimation DCM is still running in the background and it’s estimate appears as AHRS2 in the logs.  Both EKF and DCM use all accelerometers and gyros.

 

     Unrelated to you question, I think you may find AC3.3 better than previous versions especially at full throttle because of the improve throttle curve.  I would have expected earlier versions to shake at full throttle but that should be reduced now (maybe).

 

-Randy

 

From: drones-...@googlegroups.com [mailto:drones-...@googlegroups.com] On Behalf Of François Kneib

Sent: 5-Jun-15 6:37 AM
To: drones-...@googlegroups.com

--

[Andy Piper]

EKF4->SV,SMX,SMY all look really bad. Is this a sign of the aliasing? I'm surprised these are so bad and yet ATT->ErrYaw is averagely bad?

Did you notice the IMU/IMU2->GyrY differences at the points the EKF goes bad? Could it be its just hitting a resonant frequency?

APM Planner is so bad for analysis, it does not show the IMU AccelZ difference at all!

andy

[François Kneib]

Hi Randy &  Andy,

My copter is powerful, yes. I always have to decrease default gains by 40% and autotune was not able to give me good results with AC3.2.1 (do you have a link where I can find solutions for this ?).
Also, I'm coming with a lot of news. My pixhawk was fixed with foam and double-sided tape. I always performed my vibration tests in a hover state, and they were good.
As you suggested a vibration issue I verified that my props are almost balanced (just by making them spinning, one by one, at 75% throttle and touching the arm with my hand). Looks like they were ok, as adding some tape increased the vibration regardless to the side I putted the tape. Then I changed the mounting foam of my pixhawk for a really softer one (I don't have the 3DR foam any more).
Let's go for a flight, here is the log. As you suggested, I tried fast climbs at 3 different throttles (plot RCIN.C3 between lines 16800 and 28000). Now AccZ vibration looks very low. The good thing is that it seems I don't have IMU discrepancies any more, and the copter was able to maintain a good level. Can you confirm that all IMUs are ok now ? If this is the case, it means to my mind that: i) vibration tests have to be done a different climbing rates, ii) with EKF, having high vibrations could cause the sudden lean problem, which can be very hard to recover while in the air ...

Now lets go with the bad news: I reseted my parameters and autotune is not satisfactory. But I forgot to increase the ATC_RATE and ATC_ACCEL parameters. I turned my copter to acro for a couple of flips, but the behavior was not the same as usually (default values for ATC_RATE and ATC_ACCEL are low ...). So I thought it could not flip, try to recover and badly crashed ... Broken 1 LiPo, 1 propeller and the GPS mast (which is an aluminum fuse). For analysis, this is the log file that I cut before triggering acro mode. Have to wait the next week for further test !

Thank you and have a good week-end,

François.

[Andy Piper]

I wouldn't say EKF is any happier. Look at EKF4->SMX/SMY/SMZ and compare with what to expect at http://dev.ardupilot.com/wiki/extended-kalman-filter/ but I am not an expert :)

Are you sure your compass is ok?

The IMU data does look better however - at least on APM planner ;)

andy

[Andy Piper]

I looked at this with MP. Did you see the Error 19? according to the code that's a CPU error, I've never seen that before. The IMU mismatch is borderline IMO - 0.72, but that still could be some vibration thing I guess - they look in pretty good agreement now in the logs. Your compass readings are pretty far apart and there are some weird spikes in the internal compass that are not registering on the external compass. One thing you could try is recalibrate your compasses and then running compassmot to get the internal compass error down and see if you still get these sharp disagreements. If you do then try (carefully) flying with the external compass disabled and see if the EKF errors are so pronounced. You, like me, are pretty far north so it turns out that the vertical component of the compass is pretty important and therefore the mounting of your compass (thanks to Paul for helping me understand this on my rig!) needs to be super accurate.
But your pixhawk looks pretty normal at this point IMO.

On Friday, 5 June 2015 14:59:14 UTC+1, François Kneib wrote:

[Andy Piper]

It would be worth Paul or Randy looking at the log here: http://diydrones.com/group/iris/forum/topics/iris-run-amuk-flyaway
The EKF blows up because of bad compass readings, but the compasses agree very well and it's on an IRIS, so you would expect it to be calibrated pretty well.
Right before the EKF blows up it looks to me like there is gyro drift on GyrY. Am I reading this right? Could this be the cause, or is it just some user configuration issue?

[Craig Elder]

Andy, 

That person is running 3.2.1 with  AHRS_EKF_USE 0

3.2.1 uses the DCM by default and although some of the warnings have EKF in the name they are referring to the DCM solution. i.e. it is the DCM that blows up, not the EKF

If you graph the DCM and the EKF solution against altitude you can see that every time he accelerates to climb that the DCM YAW diverges from the EKF YAW and is about 50 degrees off at the point of impact

Which indicates vibration induced by the high throttle.  If we graph throttle and the accel:

we can see the correlation between throttle and the vibration in the vehicle 

and by graphing Altitude vs Acced we can see the point of impact when the vehicle hits the ground and turns upside down

The vibration of the vehicle adversely affects the DCM solution for Yaw.  This is a known issue.  I suspect if we look at the pilot's stick inputs at the time of the crash we will find that he is trying to steer the vehicle in one direction based on his outside observations while the autopilot is calculating a different value for the vehicle heading

--

[Andy Piper]

Hi Craig,

Thanks - I realised it was DCM after posting, sorry for the confusion!

I'm not seeing the throttle/vibration correlation? I see the vibrations picking up after the throttle has come down e.g. 20440 - 23500 and the level of vibration appears to be well within the limits documented on the wiki?

Also, although the throttle goes up, the radio input doesn't (CH3 on the Iris?) Finally it looks to me like the EKF blowup preceeds the throttle increase (SV). See graph here. So I'll defer to your experience, but remain unconvinced that this is simply vibrations :)

[François Kneib]

 Thanks for pointing that out Andy.
I have seen the error 19, but didn't found anything about it. It happens twice, both conducted to an arming fail (I think I tried to arm, and the fail causes the error report).
On GyrY, I still have a small offset building up during the flight. The consequence is not noticeable thought, so I'm ok with that.
The two magnetometers mismatches are highly correlated to the ThrottleOut as I didn't performed the CompassMot calibration (will do it asap) and my external compass is on a 11cm mast. I'm wondering, why do we spend so much effort on putting the external magnetometer far from electromagnetic interferences if the internal mag is also used ? If I set COMPASS_USE2=0, will I disable the internal one ?

[Andy Piper]

It's not used in 3.3, I'm just speculating that perhaps your external compass is faulty in some way - this would be perhaps a way to figure that out. In my experience if your compass is on a 11cm mast then you will get very little interference - the compassmot is simply to see whether your internal compass will then align with your external.

[Craig Elder]

>>>I'm not seeing the throttle/vibration correlation? I see the vibrations picking up after the throttle has come down e.g. 20440 - 23500 and the level of vibration appears to be well within the limits documented on the wiki?

When we wrote the vibration guidelines in the wiki it was for the APM and the values were empirically arrived at on the basis that the copter did not fly away.  It wasn't until we started flying last fall with the EKF that we could see additional issues with the Gyros being affected by vibration.  the wiki should be updated, but we need to figure out how low to recommend.

Looking back though we used to see the heading shift on other flights as well.  

 

--

[Andy Piper]

Ouch! Two wider points here:

1) If it's getting to the stage where you can only infer from the logs rather than see directly that vibs are a problem then you have a huge support issue. People find it hard enough diagnosing vibrations with the bar set as it is. Setting it higher is going to make it hard for many people (with DIY builds in particular) to ever reach it.

2) Isn't it getting to the stage where mechanically you are asking too much of people? Will Pixhawk 2 resolve these issues? I'm guessing not since this particular case is someone with a pixhawk mounted with the recommended foam and still seeing issues.

If you take the opposite approach and assume that vibrations are inevitable, can anything be done in software to alleviate this?

[Philip Rowse]

Great point Andy...

[François Kneib]

Hi guys, I would like to inform you that I will not be able to give more feedbacks at the moment as the crash I experienced last week has broken a power circuit in my Pixhawk (I'm surprised, as the crash was not so violent). If you are interested, or maybe can help, details can be found there.

[Randy Mackay]

 

     I’m not totally up to speed with the conversation but I think we are going to be forced to add some real-time vibration monitoring into AC3.3.  That would at least include clipping detection but it might also include some other number that tries to capture the level of vibration.

 

-Randy

 

From: drones-...@googlegroups.com [mailto:drones-...@googlegroups.com] On Behalf Of Philip Rowse
Sent: 9-Jun-15 6:06 PM
To: drones-...@googlegroups.com
Cc: craig...@uniserve.com
Subject: Re: [Bulk] Re: [drones-discuss] URGENT: Pixhawk Gyro Drift causing crashes in flight

 

Great point Andy...

[Andy Piper]

+1000

You read my mind :) Having a number which is a measure of vibration for each IMU spat out by the FC is definitely the way to go here.

andy

[Robert Lefebvre]

Ok, just to back up a bit... I had a look at the log, and don't really agree with the statement that this was a vibration problem.  The vibration levels actually look pretty good to me, and and I definitely don't see a connection between high throttle and vibration.  I think what Craig is seeing is actually movement in the Z-axis accelerometer levels which are actually due to the copter moving around and accelerating.  That's signal, not noise.  There are a few areas with noise, but they seem more correlated with GPS speed than throttle.

So I called up Randy and we had a look at this together.  The problem appears to be more likely a gyro problem.  I noticed the GyroY values diverge between IMU1 and 2, just before the problems start.  It's not sudden, and not extreme, but it's enough to cause DCM pitch to diverge from the EKF pitch.  This is what then leads to the uncommanded climb rate, the copter thinks it's falling when it's really not.  The strange thing is, the Gyro divergence gets better, then diverges a bit later on, then the other axis goes off a bit too.

Randy noticed that the IMU1 (MPU6000) has a HUGE Y-Gyro offset on boot up, 14 deg/sec.  This seems very large. It's not clear if it was because it was moved on booting, or perhaps an indication that the MPU hardware is not healthy.  We should investigate this further.  Perhaps review of other logs from this machine might reveal similar high offsets, and we could consider a health check for offsets being reasonable.

The yaw estimation also goes off during the uncontrolled climb, and the EKF yaw estimate can actually be seen to reset, while the DCM does not.  If we assume the EKF reset was correct, we can then assume that the DCM yaw remains wrong, and that would then explain the general navigational errors that lead to it hitting the house.

We'd like to get Paul to have a look at the data and report on what he sees in the EKF data.

We're not saying for sure this wasn't a vibration problem, as it does have many of the symptoms of that.  But the data just doesn't seem to show that at first glance.

Rob

[Robert Lefebvre]

Oh, also, the barometer is registering 56°C, which seems pretty hot.  This might also be a factor, not sure yet.

[Andy Piper]

Excuse me while I take a moment to feel smug ;)

[Robert Lefebvre]

Just... as a point of discussion, here are two logs from my Iris+.  One of them, I'm flying fast and aggressive in Stabilize and Alt Hold, speeds up to 15 m/s, high throttle, bank and yank.  No issues, and no vibration problems evident in the accelerometer logs.  Vibes are a little bit high-ish at times, but not enough to cause a problem.  There is quite some barometer vs. inertial alt discrepancy, but this appears to be clearly related to slowing down from high speed.  Airflow into the underside of the shell.

Then another one, from my T3 Figure 8 race.  WPSpeed is 15 m/s, throttle is over 80% for much of it with points at 100%.  Again, vibration is a bit high-ish at times, but as you can see from the flight path, there are no problems.

If it matters, both were with the long legs installed, and there is no payload.

[john...@gmail.com]

Regardless of cause and issue, having real time vibration messurments with a simple value indicating the vibration level is a very nice thing to have. Zero UAV uses this on their system with live GCS update, and it's great for knowing that very thing is well. Damage a propeller without knowing it or bad bearings, and you instantly see that the vibration level is off the next time you fly.

[linus]

+1

[Andrew Chapman]

Also for quickly comparing between changes when trying to reduce vibes, e.g. this foam tape vs that foam tape, different props, etc. It isn't the easiest thing for a human to look at a noisy IMU graph and determine if one is overall less or more than another.

+1

[Oliver Volkmann]

Hi Guys, I'm back with something interesting again. Not sure if it is the leans but you can see in logs 2015-06-21 09-14-10.log, 2015-06-23 08-58-59.log, 
2015-06-23 11-11-51 L.log, and 2015-06-23 12-26-16 L.log that there is a deviation between IMU1 and IMU2 GyroX. These flights thankfully did not result in a crash, but this one did: 2015-06-23 15-34-47.log. The strange thing is I cannot for the life of me pin-point what happened with this last flight. Could anyone go through this and tell me what happened? The Pixhawk is a brand new one that I got from 3DR about 2 weeks ago. Not sure if there are any real issues in the flights preceding the crash flight however just seeing a deviation in the Gyros again makes me question whether or not this is a case of the leans that got progressively worse over time. 

The above mentioned logs can be found here: https://drive.google.com/folderview?id=0B62edZ4l5_rOfkFYOWlpdkVEN3ZtQWxTeGhyT3FSR3F4ZUEyR2o4V3Jyc0hjcmFDRlRFWUU&usp=sharing

Thanks again for your help and support here everyone! By the way, still trying to crash my crash copter with Pixhawks that have had the leans in the past but not getting the time to get flights in.

On Thursday, April 23, 2015 at 11:57:16 PM UTC-4, Oliver Volkmann wrote:

Dear Developers, 

Chris Anderson sent me here to this group to share with you an issue that we feel is worth escalating since we have had similar crashes with 3 Pixhawks so far. Please see the information that I sent to the help department at 3DR, Brandon Basso and Chris Anderson below. The flight logs can be found here: https://drive.google.com/folderview?id=0B62edZ4l5_rOfkxrZlFzM0F1TElWd0xWd1NEVFVycTVNTVBienpqakNDcFVURWhYbzhwNTA&usp=sharing 

On March 15^th of this year we had a very strange experience where, during an Auto Mission our Pixhawk (FW3.1.5) based SteadiDrone QU4D X started drifting off its flight line increasing in roll angle and speed. We tried to intervene however could not save the vehicle from crashing. We have until now not found out why this happened even with the help from Santiago. I discussed this issue with Andreas Breitenstein (a colleague of mine) who had a similar experience in Namibia on the 13^th of March with a completely different drone equipped with a Pixhawk and FW3.1.5. During his auto mission, the drone did a very similar thing where it deviated from the flight plan and inevitably crashed. Yesterday we were testing some of the new features on another drone with a Pixhawk and FW3.2.1 and noticed that as soon as we put this vehicle down and powered it up, the horizon in mission planners HUD would drift in the roll axis constantly. At first I had thought that since I just put this drone together that perhaps the accelerometer calibration was not done properly and so repeated it in the field. We then flew 3 very nice flights in auto and tested some of the features. At the end of the 3^rd flight, we landed and immediately noticed on the HUD that the horizon had drifted to about 45 degrees and continued to drift. We decided to see then if the pre-arm checks worked and tried to arm the unit which happened successfully even with the horizon at 45 degrees. Andreas then attempted to take-off to see if it was possible and the unit did fly, albeit, as expected it flew sideways and hit the dirt about 3 feet away.

 We then re-started the drone and the horizon went back to normal. I enabled EKF as we have never flown with that and believed it to have far more safety features and that it would produce a far more stable flight. It was quite impressive and I flew the drone around quite a bit in POS-HOLD mode. As we stood there with the drone in POS-HOLD waiting for the battery failsafe to kick in, the drone started rolling to its right and flying away by itself even while I tried to roll left. It continued to fly away at which point I put it in Stabilize to try and prevent it from flying too far away however even Stabilize mode did not seem to help.

 We have looked at the log files of all three flights which you will find attached to this email and have discovered that at the point at which each of these 3 flights went bad, there was a deviation between the two IMUs Gyros in either  X1 to X2 and Y1 to Y2. We are now very concerned because we have a lot of drones out with our customers who have Pixhawks in them and this issue seems to happen randomly. All three drones (multi-rotors) which had the crashes all had different firmware, were flown in different locations and with different hardware/electronic components. The only thing that they share is that they all had Pixhawks on board and that they all experienced this strange Fly Away behavior which could not be stopped even when switching to stabilize.

Could you please take a look at these logs and let us know what is going on? We are pretty confident that this is not a firmware issue nor a stability algorithm issue as the firmware versions vary and we have had this experience with EKF on as well. We are at a loss as to what to do now other than test each and every Pixhawk that we (and our various clients) have on a crash test drone as this issue happens randomly. Sometimes it happens when it is powered via USB on the desk, sometimes before take-off and sometimes during the flights which result in crashes. We don’t know how many flights to do to replicate the issue but will keep the logs of all of the upcoming tests and send them to you as we have them.

 If there are any tests or if there is any other information that you may need such as serial numbers please do not hesitate to let us know. Perhaps these pixhawks are faulty and could be associated with a batch production in which case it may be easier to determine which pixhawks suffer from this strange behavior.

 

Thank you in advance for your prompt attention to this matter and we look forward to hearing back from you soon.

Regards,

Oliver Volkmann

Micro Aerial Projects L.L.C.

[Robert Lefebvre]

Oliver, you're not having much luck.

Yes, the X gyros diverge, and it again causes a roll error (leans).  Looks like the if you'd been using the EKF, it might have been OK.

I notice a 12 degree temperature change on the baro, hot at first, then cools off. I imagine it was sitting in the sun on the ground, then airflow in flight cooled it?  This might be what caused the gyro drift.  But this time it was the second Gyro that had the problem, not the MPU6000.

--

[Andy Piper]

FWIW I also have gyro drift on temperature on IMU2, so maybe they are susceptible. I dunno though - if you plot these and the learned offset it seems pretty bad to me, my drift isn't nearly this bad. The GX spikes also are very weird - the guy who had the rolls had the same thing. Paul had me set EKF_GBIAS_PNOISE to 10x it's default to make the EKF learn faster which seemed to help. I suppose it might help here.

[Oliver Volkmann]

Hi Robert, 

Apparently "luck" and I are not on good terms. This is yet another big drone that has gone down due to an IMU issue, and unfortuantely this one was carrying a $50,000 hyperspectral camera which is now off for repairs. 

If you have a moment, could you please explain to me how and why the platform crashed? I dont understand how the code would allow for the vehicle to act the way it did and then to fall from the sky, especially since the Pixhawk is advertised with "redundant sensors". It seems as if even with EKF (as we have had crashes with EKF enabled due to the leans), that the code is not checking or not switching the priority between the IMUs. The funny thing with this latest crash is that the copter maintained orientation but then started flying backwards after rolling left then right then left again and crashing. How a temperature change can make that happen is news to me. Did you see similar issues in the other logs? I did notice there was a deviation between the two IMUs GyroX however they didn't result in a crash. By the way, this Pixhawk was recently purchased (within the last month) and has a sticker on the bottom of it which I assume is a serial number...

On a side note, the other Pixhawks that I have on my desk that have demonstrated the leans before, 3DR support says they dont see anything wrong in the log files and so I'm having to fight to have them replaced. I have 3 units, now possibly a 4th that need to be repaired/returned.

Best regards and thanks for the help here!

Oliver

[Andy Piper]

I think you need Paul Riseborough to comment. Clearly gyro drift (temperature related or otherwise) is an issue with these sensors since there is code in the EKF designed to measure and compensate. I don't know whether the EKF can be changed/tuned to deal with this level of drift - it seems like a good goal, given that you are not the only one and the consequences are pretty major - but I have no idea whether this is possible. Its possible that to do something better you need 3 sensors - because then you can use consensus to determine which is right.

[Oliver Volkmann]

Andy, 

I agree completely there. I did not have EKF enabled on this drone however it still should not happen. There is an issue with the IMUs whether that be with the manufacturing and/or the code I cannot comment however so far the costs involved just on our side (Andreas Breitenstein and I) are in the region of US$100k so far. We know for a fact that we are not the only ones dealing with this issue and that there are plenty of people out there who still have not experienced this issue however it is only a matter of time. Can anyone suggest a way to escalate this to get it taken more seriously? Customer support at 3DR cant see the things that you guys are and thus are not acknowledging that there is an issue. 

 In my opinion, this is serious enough that further development should be halted until this is fixed and as much as I would love to do it, I sadly lack the skills. My contribution so far is limited to crashing my copters (and my clients mind you) as a result of this and bringing the logs here for analysis. If the solution is to use EKF only, then there should be no option to use DCM (which mind you, prior to 3.1.5 we never had these issues). The concern with EKF for us is all the errors that are listed but hey, if these crashes dont happen and we have all the errors then that's better for us. 

Please do note though, that even in my anger and frustration with this issue, I understand that the Pixhawk is an open source project and that it is considered "experimental" however this issue is going to become far more serious the more of these Pixhawks start being used on platforms for commercial work which is the way things are trending right now. The features and functionalities of this system far surpass those offered by other flight controllers so it is the obvious choice to have on any UAV. That being said, this is going to become an issue like the Phantom Fly-away except it can happen on any platform, from the Iris to 55lbs systems. 

If there is anything I can do to assist, please let me know! I'd like to stop having these crashes...

Regards,

Oliver

[Andy Piper]

I'd forgotten you weren't using EKF. FWIW I think EKF now gives a far more robust experience than DCM does in the presence of these types of sensor issue - I definitely recommend you use it, although with that value of kit you probably should wait for 3.3 GA :)

[George Farris]

Is it correct that EKF is now on by default in 3.3rc6?

On Fri, 2015-06-26 at 03:42 -0700, Andy Piper wrote:

I'd forgotten you weren't using EKF. FWIW I think EKF now gives a far more robust experience than DCM does in the presence of these types of sensor issue - I definitely recommend you use it, although with that value of kit you probably should wait for 3.3 GA :)

--

[Andy Piper]

Yes