Any way to recover a crashed UBI master node?

[Andy]

Yesterday I experienced the usual problem:

while my ci20 was busy updating "pi-hole" stuff, it got stuck probably due to overheating.

This time the event was fatal: it was not enough to power-cycle the unit and let fsck do its job;

I had to reconnect an hdmi monitor and a keyboard (I've been running headless for years)

to realize that just after the kernel startup a blocking "ubifs_master_node_recover" failure

was shown, resulting in a kernel panic (VFS couldn't proceed further).

I tried to boot the ci20 using the latest Android Oreo from SD card, since I was attempting

to recover some configs from the crashed UBI. Android was not able to boot (stuck on

the logo animation) forcing me to re-flash the latest Debian 8 beta 2016_02_02.

Is there any tool/mini-distro able to recover from such a situation?

I looked for a "live" buildroot to be run from SD, but the link is actually dangling...

It would be nice having a minimal kernel+initrd on a SD image to perform

data-rescue or backup/restore tasks in those situations.

Any ideas?

A.

[Paul Boddie]

On Friday, May 18, 2018 at 11:01:09 AM UTC+2, Andy wrote:

Yesterday I experienced the usual problem:

while my ci20 was busy updating "pi-hole" stuff, it got stuck probably due to overheating.

Is overheating a common complaint with these boards?

[...]
 

Is there any tool/mini-distro able to recover from such a situation?

I looked for a "live" buildroot to be run from SD, but the link is actually dangling...

It would be nice having a minimal kernel+initrd on a SD image to perform

data-rescue or backup/restore tasks in those situations.

Any ideas?

I don't have any ready-to-go ideas, unfortunately, but I have only used SD-based installations on the CI20 apart from the occasional incursion into the default image to check various things. Non-volatile memory that degrades within a few years isn't really a sustainable solution, in my opinion, so I'd rather leave the NAND flash alone.

I did manage to perform multistrap installations of certain Debian versions for use from SD card. The challenge is often in the initialisation of Debian, and it doesn't help that stuff tends to break, particularly for non-mainstream installation methods (such as multistrap) and, noticeably, around a certain init system that was absolutely not supported by various maintainers in Debian but is now the only thing that anyone wants to look at. (Complain about this and some people get very hostile indeed.)

However, in my multistrap installations I tend to deploy busybox as the init program in order to get the package configuration done. That should be able to get you to a usable prompt very quickly, and I suppose it is then a matter of having the necessary filesystem support available to the kernel and for various tools.

So I can imagine something pretty minimal that might help you out here. Sorry not to be more concrete about it right now, though!

Paul