RC4 Failing Auto Missions

[Robert Lefebvre]

Hi Randy, you might want to have a look at this.

I flew RC4 a whole bunch today including lots of acro which was really great.  But I also tried a few auto missions, and it failed every single time.  The failure also occurred exactly the same way in almost exactly the same spot every time.

This was with RC4+ compiled from trunk (about a week old-ish?) on an APM2.5 with:

#define FRAME_CONFIG QUAD_FRAME

#define OPTFLOW               DISABLED                      

#define MOUNT                 DISABLED            

#define CLI_ENABLED           ENABLED            

#define AUTOTUNE              ENABLED   

#define COPTER_LEDS           DISABLED

#define CONFIG_SONAR          DISABLED

(I had to do that so it would fit).  

I think I attempted 6 missions.  The first few were a standard test mission file I have saved.  Then I erased all the waypoints, uploaded that to clear the memory, then built a new simple mission and uploaded that.  It still failed on 2 attempts at almost the same place again.

Basically, it seems after passing the first real waypoint, the throttle is cut to minimum and it plummets but with stabilization running.  Investigation of the log file shows the issue appears to be a baro problem.  The baro returns a crazy result like 300m for quite a period of time.  This causes the program to cut the throttle to try and get it to descend to the WPAlt.  You can clearly see this in the logs.  The Baro comes back to reality sometime after switching to Stab.  

I'll attach a sample log, from what I've seen they are all the same.

[Randy Mackay]

Rob,
     Yes, as you say the baro data look very bad.
     Any chance you could go back to test 3.0.1 and see if it has the same problems so we can better determine if it's board or firmware specific?
     Also does it only occur while flying or can you reproduce it while it's sitting on the bench?
     Could it be light hitting the barometer?
-Randy

--
You received this message because you are subscribed to the Google Groups "drones-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[john...@gmail.com]

I would like to advocate simple sanity filters on all sensors, not just the GPS. By sanity I mean delta change triggers on values that clearly break physical limits. Such filter cost little to compute and will catch both code and HW errors as long as the error isn't building gradually. And in addition to decreasing chance of catastrophic failure, it can be used to highlight serious problems by logging the 'sanity' triggers.

- JAB

[Randy Mackay]

JAB,

     Yes, I totally agree.  There's an issue on the to-do list here for it.

     It could probably be done very similarly to the GPS Glitch detection except that we would probably use a maximum velocity instead of a maximum acceleration because I don't think we can get very reliable velocity numbers from the baro without a lot of filtering because it's quite noisy.

     Like with the GPS glitch detection we could simply throw away the baro samples for a little while and when we can't bear to do that anymore because too much time has passed we can reset the inertial nav to the new altitude.

-Randy

[Robert Lefebvre]

Sure thing Randy, I can try that.  I'm pretty sure it's a code problem though, as I can fly around in Alt Hold and Loiter with no problem at all.  

Could light hitting the baro even cause that much of a problem?

I would bet this isn't a main-line code failure, but something weird to do with the fact that I removed a bunch of modules (such as CopterLeds, or Sonar).  Like there's a hole in the logic that occurs when you remove those.

[Randy Mackay]

Rob,

     Well, let's see the results from your test.

     It could be a code issue although the only thing that has changed is we've slightly modified the timing of how the baro is called as part of the performance improvements.

     It seems unlikely to me that there could be any difference depending upon whether you're in Auto vs Loiter or AltHold.  If you look at the logs as well it seems that the glitches happen even when you're in ACRO.

     Did you notice that your board voltage is moving around a bit?  I've included the board voltage from your log vs another person running -rc4.

     Re light affecting the baro, we've done bench tests where we've seen jumps of 17m.  I've seen logs from users of much more than that but I don't think I've seen any as high as 300m.

-Randy

[Robert Lefebvre]

Huh, you're right, it's not just in auto, so it's probably not a timing issue.

I wonder what's going on, because it is *SO* repeatable in auto.  Well, I just noticed that the problem seem highly related to the orientation of the machine.  Right around 330°.  Sun is in the south at this time where I'm at, so it would be hitting the back of the board where the baro is.  The board is naked and I think I may have forgotten to put some foam over the baro.  So I'm betting light is the problem, but that's a crazy amount of error for a sensor that is not supposed to be light sensitive. ;) 

Sorry, this sounds like a false alarm.  Most of my boards are naked and I've never seen anything like this before.

This is the first time I'm using the Power Module, and I've been extremely disappointed in the performance.  The battery voltage measurement is really really terrible.

[dmu...@netgradetech.com]

I had my IRIS dive-bomb into the ground and kill itself with almost the same symptoms on auto, right after hitting waypoint 1. My baro looks good though - looking through now to see if there are any obvious glitches with any of the other channels.

[Robert Lefebvre]

You want to attach the log I can have a look?

I'm pretty sure now this is the light sensitivity issue and not a bug.  

[dmu...@netgradetech.com]

Here ya go.

[Robert Lefebvre]

Wow, that is bizarre.  I looked and looked, and can't see any reason it fell.  It wasn't leaned over too far or anything?  The altitude and climb rates measurements all seem to agree and say it's falling, it's going full throttle, and even the current shot up showing that it wasn't an ESC shutdown.  Yet it fell.  Weird.  Sorry, I can't see anything at all.

[Randy Mackay]

Rob,

     No foam on the baro and you're asking me to investigate baro problems?!

     Re the power module, I can testify that the number of brown-outs has dropped 100x since the power module started being included with the APM kits.  There's a video at the bottom of this wiki page which shows how to calibrate it: http://copter.ardupilot.com/wiki/common-measuring-battery-voltage-and-current-consumption-with-apm/

-Randy

[Jason Short]

Robert,

It could be related to hardware. If you hit wp1 you could send a sudden impulse to the motors causing a hardware failure. 

Jason

Sent from my iPhone

[Robert Lefebvre]

Sorry, didn't know it was a simple baro problem and I'm not sure how I forgot to cover it.  I've never seen a 300 meter jump because of light exposure. That's pretty crazy so I assumed it was a logic problem.  The baro otherwise looks pretty clean or I would have noticed sooner.

The problem I'm having with the PM isn't a brownout problem.  It's the Vcc noise that you pointed out.  And the steady-state voltage measurement is generally OK, but it is showing extremely high variance under load.  Something that I don't think is possible. It's just a 450 quad with 10" props, not even my Octo or 700 heli can drag down these batteries from 15.5V to 12V with a throttle push the way this thing is reporting (same batteries).

If there's a calibration for voltage drop under load, or Vcc control, I don't see it that page.

BTW, the message you replied to was my response to dmurray's log.  He's got a weird one I can't figure out.

[Robert Lefebvre]

Jason, I'm fairly sure this is a light hitting the baro problem.  I knew there was the possibility of a couple meters, but not 300.  But it's pretty clear there's a connection to heading.  The reason It kept happening after waypoint 1 is because that's when it turned north.  

But I am curious why the Vcc is so noisy, and why this little quad is showing that the voltage is dropping instantaneously from 15.5V to 13, even 12V when pushing the throttle to 3/4.  I use these same batteries in my Octo and big heli, and neither of them can do that, so I don't think it's real.

I'm also seeing a +/-0.2V fluctuation when the thing is sitting on the bench.  New battery, motors off, the battery voltage telemetry is up and down between 16.4-16.8 constantly.

[dmu...@netgradetech.com]

Yeah, it has me stumped as well. I even plotted out the delta between GPS messages to see if it suddenly thought it was somewhere else, but didn't see anything there either. It's weird, based on what I saw (and the logs) it appears to have hit WP1 then just lost it's mind. The direction it was heading after WP1 wasn't the right way towards WP2 anyway, then it just DARTED to the right and into the ground. Wish I had the full gopro footage, but I found the camera about 15 meters away with the battery out, and the crash never got written to SD. This was my first time using auto mode, now I'm nervous to try it again.

[Josh Welsh]

Just a thought Rob, maybe one of the ESC’s aren’t healthy, resulting in a larger power drain to maintain the same PWM demand?  I know it’s a stretch but perhaps a bench-test through a wattmeter can verify, or perhaps isolating each motor/esc on the bench and powering up to verify they are all pulling an equal amount of current?  This is likely way overkill in the troubleshooting procedure, but just a thought nonetheless?