Radio and Battery failsafe interaction - possible unwanted mode change from LAND to RTL.

[Olivier ADLER]

I did watch the code and i do not see any mode change protection when a battery failsafe event do occur.


For example, if we have a battery failsafe, we'll get a LAND and set_battery_low will be set.


But if later during LAND we have a radio failsafe (this is possible in practice), then the LAND mode could be changed back to RTL, and induce a crash because of low battery.


To avoid that we should check for set_battery_low flag before to allow a mode change from another failsafe.


Am i right or did i forgot something ?


Olivier

[David Dewey]

Not only possible, seems like it would be extremely likely to go into radio failsafe in LAND if you're a reasonable distance away, since radio tends to be massively attenuated near ground levels. This should definitely be looked into.

[Gareth Owen]

Olivier

This was the point I was making a few weeks ago. The current failsafe code is prone to unexpected behaviour.

I do think it is something which needs some serious attention. Addressing set mode return val fixes some of the problems but as you highlight not all of them.

Best
Gareth

[Randy Mackay]

Olivier,

     yes, you're right that we do not make different failsafe decisions based upon whether there are already other failsafes that have been activated.

-Randy

---

From: Olivier ADLER <cont...@nerim.net>
To: drones-...@googlegroups.com
Sent: Saturday, July 6, 2013 8:50 AM
Subject: [drones-discuss] Radio and Battery failsafe interaction - possible unwanted mode change from LAND to RTL.

--
You received this message because you are subscribed to the Google Groups "drones-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Olivier ADLER]

This failsafe stuff is becoming quite complex and error prone because the failsafe code is spread across the code and there are interaction in the set_mode function with a separate and blind mode filter.

Would be better to manage this in a dedicated state machine object.

Priority between failsafes is not as simple as we could primarily think.


I think we should rework our failsafes strategies on a flow chart, and from here see what would be the best solution to implement it at the programming side.


Olivier

[David Dewey]

I agree with Olivier. First finalize a FS flowchart that works, then build a state machine class to follow exactly that. Actually, once the chart is finalized, programming the state machine class should be quite straightforward.

[Andrew Chapman]

On 2013-07-07 7:48 PM, David Dewey wrote:

I agree with Olivier. First finalize a FS flowchart that works, then build a state machine class to follow exactly that. Actually, once the chart is finalized, programming the state machine class should be quite straightforward.

If we're refactoring failsafe should we also consider making the actions user configurable?

For example I was flying under a canopy of trees yesterday. In that case a LOITER or LAND action upon failsafe would have been a much better option than an RTL rise into the trees. I would pick LOITER so I could walk toward it to try regain control, but other people may prefer LAND in that case.

And to sneak another suggestion in on this topic: should RTL descend toward the height set by RTL_ALT if above it? If you're high up and the low battery failsafe triggers RTL then it's likely to run out of power on the slow descent after reaching home, and you'll be using less power if descending.

Finally, if failsafe is set to complete auto mission rather than RTL, what happens if the user doesn't have any waypoints set? It should fall over to RTL in that case.

AC.

[Olivier ADLER]

Yes Andrew we could have a user choice to force Landing instead of RTL. We should keep that simple with a single parameter, so that users cannot make errors.

I think that we should keep those important things in mind for designing a failsafe flow chart :


1) Failsafe decision has fours primary levels depending about the failsafe importance :


a) Log report only
b) SoftFailsafe (RTL, mode change...)
c) Hardfailsafe (Land, Disarm)
d) Flight recovery (Parachute)


2) We have a set of recovery actions :

- Report Only
- RTL
- LAND
- Alt_Hold + XY Inertial
- Disarm
- Parachute


3) We have actually 4 failsafes sources :

- Radio Failsafe
- GPS Failsafe
- GCS Failsafe
- Battery Failsafe

I think that we should add those ones :

- Bad Attitude failsafe
- Software error detection (APM watchdog)


4) Some failsafes can occur more than one time during flight (GPS, GCS, Radio), other ones can occur only one time (Battery, Bad Attitude).

Failsafe can occur separately or mixed together and according to Failsafe sources there are priorities. For example bad attitude should be the higher priority failsafe followed by the battery failsafe.


Because of that, we must take all those situations into account. This is the tricky part of the global failsafe strategy.



Olivier

[Olivier ADLER]

David, i did start working on a flow chart. I think i will need a few days before i get something working reliably for all situations. I did integrate the parachute and bad attitude detection in it.

Are you interested to work on this with me ? You could write the code as i'm not familiar with C++ object programming. It will ask a code rework with full reliability in mind and dialogue with Randy and Tridge to keep the right code structure.

It's an interesting code and testing challenge.

We'll need to add failsafes simulation as well on channels 6,7 and 8 so that we can test that in the field with multiple failsafe events occurring and clearing.


Perhaps that the bad attitude detection failsafe event could be written by Randy or Leonard, or we could leave this event empty until somebody will write it.


Olivier.

[Randy Mackay]

Olivier,
      very happy to help so I can take care of the review and adding the parachute detection if you like.

      By the way, I put in a temporary fix yesterday re the throttle failsafe kicking off an RTL even though we're already in battery failsafe but it's not a really pretty solution so very happy to have it replaced by something better.

      Thanks for all your help.

-Randy

---

From: Olivier ADLER <cont...@nerim.net>
To: drones-...@googlegroups.com

Cc: Randy Mackay <rmac...@yahoo.com>
Sent: Sunday, July 14, 2013 9:09 AM
Subject: Re: [drones-discuss] Radio and Battery failsafe interaction - possible unwanted mode change from LAND to RTL.

--

[Kevin Hester]

Great idea.  Rather than a flow chart, you might want to consider documenting all these states as a finite state machine (if not familar - the concept is simple and explained in this wikipedia article).  

If there isn't already a little FSM class in the APM code then it might be worth making one and using that class to control all these sorts of modal behaviors/transitions.

Kevin

--

[David Dewey]

Hi Olivier,

Yes, thanks, it sounds like a fun project to work on.

I agree with Kevin about making it easier to code by documenting the states and state changes as an FSM. How hard would it be to write out or draw out a set of states and transitions implementing your flowchart?

-David

[Gareth Owen]

I'm just throwing this out there as an idea - I haven't thought it thtough completely.

How about a lookup table matching failsafe states to mode?  Would be far simpler and readable than a state machine.  Risk of course is scalability as number of failsafe states increases - with each additional failsafe state the table doubles.

e.g.

typedef struct {

   bool fs_rad, fs_bat, fs_gcs;

   uint8_t modeToChangeTo;

} fs_map_t;

fsmap_t fsmap[] = {

     {true, false, false, RTL},

     {false, true, false, LAND},

... etc ...

};

--
You received this message because you are subscribed to a topic in the Google Groups "drones-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/drones-discuss/pj3FhDVQJBU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to drones-discus...@googlegroups.com.

[Gareth Owen]

Of course you could solve the scalability problem by having the rule table store priority and having a quick algorithm to select the most appropriate rule.

[David Dewey]

Nice idea. How about encoding the outputs of the lookup table as boolean values as well, and using a Karnaugh Map Minimizer to create a boolean expression for each output bit? If the encoding was done wisely, then hopefully you'd find that your big complex lookup table would boil down to just a few simple boolean expressions.

-David

[Olivier ADLER]

I think that my flowchart will be mostly a FSM so it should be quite straightforward to transform it to an UML state machine diagram.

I've seen that there is a FSM plugin inside Eclipse that should allow a FSM Diagram to directly translate into C++ code. Would you like to use something like this ?

@kevin, do you have experience programming FSM ? Can you recommend a FSM library ?

I think that for me the first step anyway is to design a flowchart, that we'll be useful as well for the Wiki.

Olivier

[Olivier ADLER]

Gareth yes scalability is a problem here as well as failsafe priority and clearing failsafes event management.

Olivier

[Gareth Owen]

If the problem is reducible to a series of boolean expressions that'd be very elegant.  Certainly a lookup table could be reduced by a number of mechanisms.

Is a FSM really a suitable solution here?  Surely any state can transition to any other state producing a very complex SM (tending toward (O(n^2))- probably more so than a lookup.

Best

Gareth

[Kevin Hester]

@kevin, do you have experience programming FSM ? Can you recommend a FSM library ?

I do - but alas I'm a bit over committed with stuff at the moment.  I typically didn't bother using with any sort of FSM library.  

Rather I always made/used a simple library that looked something like this:

State

  virtual void onEntry() // Establish any the new invariants of this state - this is where you put the interesting code for this state

  virtual void onExit() {} // rarely needs to be overridden

  virtual State *perhapsTransition() // Will be called periodically by StateMachine while in this state, return null for no change or a new state if we should transition out. (implement one clause in this function for each 'arrow' out of that state on the diagram

StateMachine // a tiny class that makes state transitions work

  State *currentState

  StateMachine(State *startState) // constructor will call enter() on the startState

  private void enter(State *state) // call state->onEnter() then set currentState = state 

  void runOnce() // calls perhapsTransition on currentState, and depending on result code perhaps call exit(...) and enter(...)

  void dump() // Print some debugging information about current state

Then to use this micro library just make one subclass/instance for each State in your FSM (LandingState etc...)

--

You received this message because you are subscribed to the Google Groups "drones-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.

[Stefan Gofferje]

That sounds like a great idea!
Battery failsafe could/should have 2 stages: Soft and hard, depending on
remaining battery capacity/voltage. Soft would trigger a configurable
action while hard will always land.

--Stefan

On 07/08/2013 08:48 AM, Andrew Chapman wrote:
> On 2013-07-07 7:48 PM, David Dewey wrote:
>> I agree with Olivier. First finalize a FS flowchart that works, then
>> build a state machine class to follow exactly that. Actually, once the
>> chart is finalized, programming the state machine class should be
>> quite straightforward.
>
> If we're refactoring failsafe should we also consider making the actions
> user configurable?
>
> For example I was flying under a canopy of trees yesterday. In that case

> a LOITER or LAND action upon failsafe would have been a /much/ better

> option than an RTL rise into the trees. I would pick LOITER so I could
> walk toward it to try regain control, but other people may prefer LAND
> in that case.
>
> And to sneak another suggestion in on this topic: should RTL descend
> toward the height set by RTL_ALT if above it? If you're high up and the
> low battery failsafe triggers RTL then it's likely to run out of power
> on the slow descent after reaching home, and you'll be using less power
> if descending.
>
> Finally, if failsafe is set to complete auto mission rather than RTL,
> what happens if the user doesn't have any waypoints set? It should fall
> over to RTL in that case.
>
> AC.
>

[Craig Elder]

Last fall I proposed the following State Machine to describe the failsafe scheme.  This diagram needs to be updated to be current but it was the basis for writing the code at that time.

[Olivier ADLER]

Here is a first preview of a new global failsafe strategy.

I think it is complete enough to be a good work basis before to decide the final strategy.


It should give a clear view of the logic problems we need to manage during failsafe decisions.


The actual v0.8 give almost the same base failsafe strategy as the trunk, with some adds and enhancements :

- Modal failsafes isolation

- New GPS clear failsafe Action if previous mode (before Land) = Auto ( Ascent to safe altitude + auto )

- New failsafe Events : Software error and Bad attitude detection

- New parachute failsafe action

- Radio / GCS failsafes mutual control before failsafe decision



How does work the Radio / GCS multual control :


We have a set of two user parameters :

- RADIO_FS_Check_GCS

- GCS FS_Check_RADIO


If RADIO_FS_Check_GCS is set and a radio failsafe do occur, a check will be done to know if the GCS is actually controlling the copter. If yes, then do not take any failsafe decision.

If GCS FS_Check_RADIO is set and aGCS failsafe do occur, a check will be done to know if the RC Radio is actually controlling the copter. If yes, then do not take any failsafe decision.


The result is that when a radio link is lost, it will not trig a failsafe action if the other link is still alive and actually controlling the copter.

Olivier




On Sunday, July 14, 2013 2:00:00 PM UTC+2, David Dewey wrote:

[Stefan Gofferje]

As said before, I think, we should have 2 battery failsafe levels. One
hard which force-lands and one soft which has a user-defineable action.

Just think if a mission goes e.g. over a lake or a dense forest... To
have only one battery failsafe that force lands would be rather bad.
It would be much better to have a "soft" battery failsafe which can be
set so that the copter e.g. RTLs or flies to a preprogrammed emergency
landing site at a still low but yet safe battery level.

--Stefan

[Olivier ADLER]

You are right Stefan. It would be better to have a two levels failsafe for the battery.

You can add this if you want to the diagrams and share it here. It's a Libre Office Draw document. Libre Office is free and opensource.
A dedicated page for the Battery failsafe would be even better because there is not a lot of room remaining on actual pages.



The most important is to discuss and adopt a  new global strategy structure first, so that we can add those details easily and safely next without intrusive and risky code modifications.

It would be nice if we can do that for version 3.1 so that we can trust a rock solid failsafe strategy basis for future enhancements.


In the actual code the failsafe logic is complex enough to induce undesired "features" because the logic is spread across many locations in the code. It's becoming unmanageable for humans.



Olivier.

[Stefan Gofferje]

Here's my ideas...

[David Dewey]

These are all flowcharts, NOT finite state machines.

Is the idea still to have these drawn out in the form of finite state machines, to make it easy to translate into code?

-David

[Robert Lefebvre]

Stefan, I agree completely.  I don't use battery failsafe, because the first time I did, it was really scary when suddenly, and without warning, it landed where I didn't want it to.

I had started to work on this in my 2.9 branch, but it got lost with the move to 3.0.

While we're at it, I was working on making an optional dual-voltage monitor.  I don't use current monitoring, but I do use two batteries.  I'd like to monitor voltage on both.  This should fit into the failsafe SM easily enough.

[David Dewey]

Definitely agree on two battery failsafe levels, since I may be flying over trees or water and don't want to be landing into them.

-David

[Olivier ADLER]

David, my primary idea was to draw flowcharts so that everyone can clearly see what we need to manage, and check / study the global strategy. In my diagrams there are states (directly linked to actions) and  conditional transitions. Is that not a state machine ?


One of the goal here is to find failsafe attributes that would simplify programming later as well as designing the global strategy : interaction between failsafes and sensors, priorities, user control locks, user preferences, and so on. I would like to have comments about this from the developers team.
For example actually i did use three distinct failsafe events groups : sensors group, overtake group, and modal group and each failsafe or failsafe clear event has a priority attribute. The modal attribute is protecting the failsafe event from overtaking.


It's very important to design things rightly here and we should discuss this before the programming details. I spent a few hours to draw this and to think about the right strategy. It's not so easy.


Let me know if you think i should do some things differently or if you have other ideas to better classify and manage failsafe events.


One of my idea was to separate states and actions, so that we can post manage actions in a post processing area according to flags set during the state sequence or taken from sensors / flight conditions. The same as in a SFC program (CEI 61131-3).

The drawback of this is that it add complexity and because of this it is not so easy to design / read back.


We should keep thing as simple as possible too, for two reasons :

- we are near the flash size limit for the APM

- simpler is better for reliability and less design / programming errors


Olivier.

[Olivier ADLER]

Thanks Stefan. Seems like a good addition.


As a side note, it is desirable during long missions to change the home position using the do_set_home command, so that the battery failsafe can RTL in safe areas not to far.


Olivier.

[Olivier ADLER]

As a side note i'd like to emphasis about the failsafes function : they are last resort mechanisms to try saving the copter and more importantly protect ground life.


In this regard we need to keep things simple, with minimum user configuration.


It is the responsibility of the user to correctly plan its missions so that a battery failsafe will never occur during flight. Same thing for other failsafes.


Olivier

[Robert Lefebvre]

I agree.  But what we need is a battery warning level.  That's what I called it in my code.

https://github.com/R-Lefebvre/ardupilot/commit/5c9f9a392f1a8e59cd5811ab32c16c7f7038077b

--

[David Dewey]

Hi Olivier,

Thanks, I agree with you that we're still at the stage where things are being designed, and we can focus on programming details later. We can later map out the actual state machine(s) based on the agreed upon design, but since the design is still changing, that step can wait.

I'm not sure I agree with you that "it's the responsibility of the user to correctly plan its missions so that a battery failsafe will never occur during flight". The battery failsafe is based on battery voltage, which sometimes can be completely unpredictable due to random fluctuations, a battery suddenly reaching the end of it's life, sudden changes in weather conditions, and many other factors. So it's actually impossible for a user to plan a mission in a way to be absolutely certain that a battery failsafe will "never" occur. So we need to design a battery failsafe that won't cause the loss of the copter over water, in a tall tree, or other inaccessible area.

-David