Better way of writing to the Pixhwak/PX4 SD card?

[Chris Anderson]

Team, 

The current way Pixhawk/PX4 writes to the SD card is fragile, since it datalogs constantly to the card and depends on the SD card being able to close the files and clean up the FAT itself on powerdown. Good SD cards can do this, but poor-quality ones tend to fail with corrupted FATs etc.  We're not quite sure why -- it could have to do with the built-in capacitance of those cards, where the better ones can power themselves long enough to complete a write and close the file, or it could be due to the firmware within the SD cards -- but it's a failure mode that we'd like to eliminate.

This is not a problem with most electronics that use SD card, such as cameras and phones, since they only write to the card occasionally, but autopilots present a unique challenge in that they are constantly datalogging and are often powered-down instantaneously (when a user removes the battery, for example).

It strikes me that there are at least three things we could do to mitigate this risk:

1. Datalog to a flash memory buffer and only write to the SD card on a scheduled basis. This doesn't remove the possibility that a powerdown will happen during a SD card write, but it does lower the probability (from ~100% to ~1%, depending on the size of that buffer). Needless to say, this requires the completion of the NuttX flash driver, which I gather is a few weeks away. 
   
2. Add a supercap to the Pixhawk board to ensure that even if the SD card doesn't power itself long enough during shut-down, the board can
   
3. Modify the NuttX SD card driver to handle write interrupts better and/or be able to rebuild a corrupted FAT at startup, the way a GoPro can. 
   

Regarding #1 and #3, it might be worth looking into other SD-card baed dataloggers do it, such as the Sparkfun Logomatic. 

BTW, right now if the SD card is corrupt, Pixhawk will not boot, since the params are stored on that. But once the NuttX flash driver is compete, the params and other necessary startup info will be stored on flash and a Pixhawk will boot fine with a corrupt or missing SD card (although datalogging will be disabled). Craig thinks that's about a month away. 

What do you guys think of those three methods? Are there better solutions?

Chris

[Roberto Navoni]

Hi Chris,
i confirm that SD card is fragile . I informed Tridge 2 weeks ago
about this kind problem . i doing a long time test .. around 24 hour
of data recording ... 160 mbyte ... during my test i found some times
that we lost the sd card integrity . We are working on Datalogger
based on VRBRain with nuttx as O.S . The VRBrain hardware is similar
to PX4 and i found the same problem. So is not only a problem on not
correct power down but we found that the problem could be related also
at presence of radio frequency trasmission near the sd card . So could
be an hardware related problem ... lenght of wire that connect the
signal from cpu to sd card reader . Could be nice to add a sort of
shield oround the sd card spi / sdio bus.
Best
Roberto Navoni



2014/1/3 Chris Anderson <ch...@3drobotics.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "drones-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to drones-discus...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Gary McCray]

Hi Chris,

Datalogging to a flash buffer seems a good idea, but it also gets the total number of flash writes to a considerably higher level and could compromise Flash longevity.

Possibly a small main memory program ram buffer would work better.

A small supercap would probably solve the problem, but you want to keep the supercap small, because you don't want a unplugged PX4/Pixhawk staying up very long after it is unplugged.

So worst case the cap should hold it up less than a second even if nothing else is connected to the PX4/Pixhawk. (And of course it will drain much quicker if there are other things hooked up to it.)

And the last one: "Modify the NuttX SD card driver to handle write interrupts better and/or be able to rebuild a corrupted FAT at startup, the way a GoPro can." sounds like a no brainer, it needs that ability if at all possible regardless of logs.

Storing the params in flash is an excellent idea (seems like it always should have been done that way anyway) and coupled with an ability to automatically rebuild the FAT, it seems like the SD card problem itself would generally be insignificant.

There will always be marginal SD cards and the correct instruction would normally be get another one, they practically do grow on trees these days.

Pretty much like for any other device where a given SD card proves to be problematic.

Best Regards,

Gary

[Marco Robustini]

Nice question Chris, is not possible to keep the logs in memory like with VR Brain and write to SD only when disarmed?
Of course we would also like something that would verify the successful writing, and if not in case repeat the writing qhen replug the LiPo or microusb, then emptying the flash memory.

Marco

[Robert Lefebvre]

For buffering, that seems like a great idea to me, the only thing that I would worry about is in crashes, we'd hate to lose the last few VERY valuable seconds of data.

Regarding the supercap, we'd have to be careful about this.  I have played a few times with adding a supercap on my APM, and while it seems to be OK most of the time, there's a risk that it does something non-deterministic as it powers down.  Using a supercap lengths the time that it passes through the brown-out window.  I've had a couple time when it gave a blast of full throttle.  This only happens when I used to use a separate battery from the motor and a smaller battery for the APM.  And then it only happens if you disconnect the APM battery first, which is always a bad idea anyway.

But just so that you're aware of the creation of a few additional potential failure modes.  (ie: even if powering normally, if the power supply to the APM failed, you could get a blast of full throttle from the motors).

[Marco Robustini]

I have some questions about this:

1) the parameter file resides only on SD and is loaded at boot time?
2) there is a kind of non-volatile memory in Pixhawk, and if yes what we are saved?
3) a verify test is performed according to a previously stored checksum while we load the parameter set at boot?

Marco

On Friday, January 3, 2014 8:44:33 PM UTC+1, chris wrote:

[Roberto Navoni]

On VRBrain we have eeprom + Dataflash + SD Card
On Pixhawke there is http://www.cypress.com/?docID=44769 128Kbyte FRAM.
So i think that the unique option is FRAM
best
Roberto

2014/1/3 Marco Robustini <robusti...@gmail.com>:

[Marco Robustini]

Ok ,so this FRAM can be used for buffering, but imho should also contain a double checksum (for safety)  of the parameters saved on the SD, so can verify it at boot.
If the SD card is corrupt and the parameter file has some outlier is likely takeoff with a likely crash.
With a double checksum in FRAM could save a double parameter file on the SD, we have 2 Gb minimum of space, there's stuff for check-ups and be more confident of the read data.
Maybe it works well already, but I do not think.
I'm not an engineer but this is a simply logic about data integrity, dealing with business data security believe that these simple steps are essential.
Lorenz or Tridge certainly have definite answers to my "doubts"...

Cheers, Marco

[Bill Bonney]

IMHO the suggestion of using a super cap is not that it powers the motors, but powers the flight controller, like using a separate flight battery. The advantage is that when you plug in your main battery the cap is charged within seconds (no need to remember how much charge is in the flight battery). On power failure you can still run the flight controller and telemetry for a period of time (but the vehicle would still crash) The supercap would be to costly if large. The supercap has a linear voltage decline, so you could trigger last write of SSD on a lower threshold. (Of course there are other interesting scenarios you could use one for, like reserve energy for failsafe) 

I think the best option is to buffer in RAM and only write that buffer if the power supply is still valid, and you have enough time to do so. I did see in another posting that using sequential writes to the SSD is better, than multiple writes. 

I haven’t looked at any datasheets on SSD best practices yet. Though I am guessing (since you still ned to ‘eject’ them to be safe on PCs etc…) that is the the responsibility of the driver not to write when the power is predicted to fail. It’s just that prediction part ;)

Bill

[Robert Lefebvre]

No cap would keep the motors running for any period of time, I don't think that was the idea.  Just the flight controller.  I have used one of the large cap banks that HK sells, and can get a full 6 seconds out of it running an APM.  But IMO, that's too long, as it spends way too long in brown-out territory.  They have another cap, just a single unit, that works for about half a second.  I can *just* unplug and replug the battery while retaining a boot. That's probably way longer than it would take to do the SD card write.  Then I have another APM with a small supercap soldered right onto Vcc, with about the same duration. Not very big, maybe 5mm dia by 3mm thick.

The only risk I was trying to describe is what happens if the power to the Pixhawk is interrupted while the motors are still powered.  That can do bad things.

[Gregory Nutt]

All file system, FAT or otherwise, will keep certain data cached and so when power is unexpectedly removed, there will be an incoherency between the state of the the device and the cached data that was never written to the device.  FAT is  particularly resilient to the sudden-loss-of-power problems, but corruption can still occur.

If the file system is properly unmounted prior to power down, then there show be now problem.  un-mounted causes and synchronization of the cached data so that it is "safe to remove the device."  Of course, you can do the same synchronization at any time by calling fysnc (http://pubs.opengroup.org/onlinepubs/7990989775/xsh/fsync.html).  So perhaps one thing you can do is to call fsync() periodically.

Greg