fsck fails to start on writable partition at boot/startup

[Dave Barndt]

Hi,

I'm hoping that someone has come across this problem before and can point me in the right direction.

I'm trying to troubleshoot a BeagleBone Black with Debian 8 that appears to have a filesystem corruption. The system has two partitions, a read-only rootfs partition and a writable partition for essentially everything else. When the system boots, U-Boot completes and hands control to the kernel, which runs an fsck on the rootfs successfully, but then fails to run an fsck on the writable parition. At that point the startup process appears to simply hang. I cannot seem to break to a console prompt (or get to a login prompt, obviously).

The best hypothesis I have so far is that some sort of power failure caused a corruption, but I'd like to see if I can examine the "footprint" at all. I've never seen a corruption where fsck can't be run at all. Usually fsck can be run and the corruption can be examined and hopefully repaired.

My question is, why can't fsck be run on the partition at all? Can I somehow break to the console prompt when the startup process hangs up? Earlier in the process, I can interrupt U-Boot and run "mmc", "part", and "ls" types of U-Boot commands to look at the partition in question - at that level things appear to look OK. But obviously it can't tell why fsck won't run on the partition.

Could there be anything else going on that I'm not thinking of?

Below is a log of the boot/startup process. Any light anyone can shed would be very helpful.

Thanks for any help,
Dave
-------------------------------------------

U-Boot SPL 2016.11-00002-gab8be1c (Dec 07 2016 - 12:54:09)
Trying to boot from MMC2


U-Boot 2016.11-00002-gab8be1c (Dec 07 2016 - 12:54:09 -0600), Build: jenkins-github_Bootloader-Builder-490

CPU  : AM335X-GP rev 2.1
I2C:   ready
DRAM:  512 MiB
Reset Source: Power-on reset has occurred.
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Using default environment

Board: BeagleBone Black
<ethaddr> not set. Validating first E-fuse MAC
Net:   eth0: MII MODE
cpsw
Press SPACE to abort autoboot in 2 seconds
board_name=[A335BNLT] ...
board_rev=[AIA0] ...
Card did not respond to voltage select!
Card did not respond to voltage select!
Card did not respond to voltage select!
gpio: pin 56 (gpio 56) value is 0
gpio: pin 55 (gpio 55) value is 0
gpio: pin 54 (gpio 54) value is 0
gpio: pin 53 (gpio 53) value is 1
Card did not respond to voltage select!
Card did not respond to voltage select!
switch to partitions #0, OK
mmc1(part 0) is current device
Scanning mmc 1:1...
gpio: pin 56 (gpio 56) value is 0
gpio: pin 55 (gpio 55) value is 0
gpio: pin 54 (gpio 54) value is 0
gpio: pin 53 (gpio 53) value is 1
switch to partitions #0, OK
mmc1(part 0) is current device
gpio: pin 54 (gpio 54) value is 1
Checking for: /uEnv.txt ...
Checking for: /boot.scr ...
Checking for: /boot/boot.scr ...
Checking for: /boot/uEnv.txt ...
gpio: pin 55 (gpio 55) value is 1
1388 bytes read in 28 ms (47.9 KiB/s)
Loaded environment from /boot/uEnv.txt
Checking if uname_r is set in /boot/uEnv.txt...
gpio: pin 56 (gpio 56) value is 1
Running uname_boot ...
loading /boot/vmlinuz-4.4.36-ti-r72 ...
8646448 bytes read in 589 ms (14 MiB/s)
loading /boot/dtbs/4.4.36-ti-r72/am335x-abbbi.dtb ...
61725 bytes read in 39 ms (1.5 MiB/s)
loading /boot/initrd.img-4.4.36-ti-r72 ...
5231183 bytes read in 369 ms (13.5 MiB/s)
debug: [console=ttyO0,115200n8 bone_capemgr.enable_partno=BB-UART5,BB-I2C1 root=UUID=936d3d70-c3c4-4d5b-bb06-2d6d680ae95d ro rootfstype=ext4 rootwait coherent_pool=1M quiet net.ifnames=0 cape_universal=enable] ...
debug: [bootz 0x82000000 0x88080000:4fd24f 0x88000000] ...
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Ramdisk to 8fb02000, end 8ffff24f ... OK
   Loading Device Tree to 8faef000, end 8fb0111c ... OK

Starting kernel ...

[    0.001350] clocksource_probe: no matching clocksources found
[    2.075156] wkup_m3_ipc 44e11324.wkup_m3_ipc: could not get rproc handle
[    2.259213] omap_voltage_late_init: Voltage driver support not added
[    2.270746] PM: Cannot get wkup_m3_ipc handle
[    2.382235] bone_capemgr bone_capemgr: slot #0: No cape found
[    2.426233] bone_capemgr bone_capemgr: slot #1: No cape found
[    2.470229] bone_capemgr bone_capemgr: slot #2: No cape found
[    2.514227] bone_capemgr bone_capemgr: slot #3: No cape found
Loading, please wait...
rootfs: clean, 78785/98304 files, 384993/393216 blocks
[   12.471231] pinctrl-single 44e10800.pinmux: pin 44e108c0.0 already requested by 481aa000.serial; cannot claim for 0-0039
[   12.482304] pinctrl-single 44e10800.pinmux: pin-48 (0-0039) status -22
[   12.488911] pinctrl-single 44e10800.pinmux: could not request pin 48 (44e108c0.0) from group adi_hdmi_bbbi_pins  on device pinctrl-single
[   12.501341] adv7511 0-0039: Error applying setting, reverse things back
[   12.928105] pinctrl-single 44e10800.pinmux: pin 44e108c0.0 already requested by 481aa000.serial; cannot claim for 0-0039
[   12.939198] pinctrl-single 44e10800.pinmux: pin-48 (0-0039) status -22
[   12.945814] pinctrl-single 44e10800.pinmux: could not request pin 48 (44e108c0.0) from group adi_hdmi_bbbi_pins  on device pinctrl-single
[   12.958243] adihdmi 0-0039: Error applying setting, reverse things back
[   14.022183] EDID block is all zeroes
[   14.038732] adihdmi_encoder_get_modes - 788 - No EDID
[   14.279735] adihdmi_encoder_get_modes - 788 - No EDID
[FAILED] Failed to start File System Check on /dev/mmcblk1p2.
See 'systemctl status system...@dev-mmcblk1p2.service' for details.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Load/Save Random Seed.
[DEPEND] Dependency failed for Update UTMP about System Runlevel Changes.
[DEPEND] Dependency failed for /root.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for netfilter persistent configuration.
[DEPEND] Dependency failed for Emergency Shell.
[DEPEND] Dependency failed for Emergency Mode.
[DEPEND] Dependency failed for Update UTMP about System Boot/Shutdown.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[DEPEND] Dependency failed for /srv.
[DEPEND] Dependency failed for /media.
[DEPEND] Dependency failed for /home.
[DEPEND] Dependency failed for /var/local/swapfile.
[DEPEND] Dependency failed for Swap.
         Starting Raise network interfaces...
         Starting File System Check on /dev/mmcblk1p2...
[  OK  ] Reached target Timers.
[  OK  ] Closed Syslog Socket.
[  OK  ] Reached target Login Prompts.
         Starting Create Volatile Files and Directories...
[  OK  ] Reached target Sockets.
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
[FAILED] Failed to start File System Check on /dev/mmcblk1p2.
See 'systemctl status system...@dev-mmcblk1p2.service' for details.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for /root.
[  OK  ] Started Raise network interfaces.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.

(nothing appears after this; unable to break to a console prompt, etc.)

[minde...@gmail.com]

See 'systemctl status systemd-fsck@dev-mmcblk1p2.service' for details.

See 'systemctl status systemd-fsck@dev-mmcblk1p2.service' for details.

[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for /root.
[  OK  ] Started Raise network interfaces.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.

(nothing appears after this; unable to break to a console prompt, etc.)

Hi Dave,

Have you tried booting using an initial ramdisk and using the fsck? I am guessing that your filesystem is in the eMMC.

Thanks,
Gautam.

[Dave Barndt]

Hi Gautam,

Thanks for the reply! I actually had the same idea last night, and did manage to boot the board using an image on an SD card, and was able to run the fsck from there against the bad partition on the eMMC, and saw the details of the corruption. I was able to repair the partition and ultimately reflashed the board. So thanks for the reply and the confirmation!

But, what I'm still baffled about is: Why the fsck couldn't run as part of the kernel startup when the system was booted normally? I assume the partition hasn't been mounted at that point yet, so why would the fsck fail to start? It's really just more of an understanding-type of question.

Anyway, we 're now looking at ways to prevent sudden or unexpected power downs from potentially effecting such behavior/corruptions. Found this reference which looks pretty helpful: https://www.embeddedarm.com/about/resource/preventing-filesystem-corruption-in-embedded-linux ... and the BBB has power down signalling (section 5.10 of the ref manual) that we take advantage of as well.

Thanks again!

Dave

[ta...@thinnect.com]

See 'systemctl status systemd-fsck@dev-mmcblk1p2.service' for details.

See 'systemctl status systemd-fsck@dev-mmcblk1p2.service' for details.

[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for /root.
[  OK  ] Started Raise network interfaces.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.

(nothing appears after this; unable to break to a console prompt, etc.)

Hi Dave, 

Although I have seen a few corrupt file systems, I haven't stumbled on a case where fsck fails to run. With vanilla config fsck usually is started on boot and, in case of errors, patiently waits for you to manually confirm each fix by pressing "y". This can be worked around by adding to the kernel command line "fsck.mode=force fsck.repair=yes".

What's in your /etc/fstab and /boot/uEnv.txt?

--

Kind regards,

Tarmo Kuuse

[minde...@gmail.com]

Hi Dave,

Could you please let me know what the corruption was once you ran fsck from the SD card? I am interested to the see what the corruption was.

In case of fsck failure on startup did you lose your tty terminal once the fsck started? Due to this was it waiting for some confirmation or had it just started and you lost the output messages?

Is this for something which will go into an actual product and is the requirement to have a robust FS during power failure?

Thanks,

Gautam.

[Dave Barndt]

Hi Tarmo,

Thanks for the reply! And we are considering doing something like you're suggesting - adding fsck options to the kernel to do automatic repairs during startup when possible. Thanks for that.

/etc/fstab just has a few entries. Two partitions and then directories bound within the second writable partition (/var):

# /etc/fstab: static file system information.

#

UUID=fe3a62a1-b2ea-45d3-afcb-f1f66161ef0d  /  ext4  ro,noatime,errors=remount-ro  0  1

debugfs  /sys/kernel/debug  debugfs  defaults  0  0

tmpfs  /tmp  tmpfs  defaults,noatime,nodev,nosuid,noexec  0  2

/dev/mmcblk1p2  /var  auto  defaults,noatime,errors=remount-ro  0  2

/var/local/home  /home  none  bind  0  0

/var/local/media  /media  none  bind  0  0

/var/local/root  /root  none  bind  0  0

/var/local/srv  /srv  none  bind  0  0

/var/local/swapfile  none  swap  sw  0  0

/boot/uEnv.txt is as follows - we enable a couple pins but that's it:

#Docs: http://elinux.org/Beagleboard:U-boot_partitioning_layout_2.0

uname_r=4.4.36-ti-r72

#uuid=

#dtb=

##BeagleBone Black/Green dtb's for v4.1.x (BeagleBone White just works..)

##BeagleBone Black: HDMI (Audio/Video) disabled:

#dtb=am335x-boneblack-emmc-overlay.dtb

##BeagleBone Black: eMMC disabled:

#dtb=am335x-boneblack-hdmi-overlay.dtb

##BeagleBone Black: HDMI Audio/eMMC disabled:

#dtb=am335x-boneblack-nhdmi-overlay.dtb

##BeagleBone Black: HDMI (Audio/Video)/eMMC disabled:

#dtb=am335x-boneblack-overlay.dtb

##BeagleBone Black: wl1835

#dtb=am335x-boneblack-wl1835mod.dtb

##BeagleBone Green: eMMC disabled

#dtb=am335x-bonegreen-overlay.dtb

cmdline=coherent_pool=1M quiet net.ifnames=0 cape_universal=enable

#In the event of edid real failures, uncomment this next line:

#cmdline=coherent_pool=1M quiet net.ifnames=0 cape_universal=enable video=HDMI-A-1:1024x768@60e

##Example v3.8.x

#cape_disable=capemgr.disable_partno=

#cape_enable=capemgr.enable_partno=

##Example v4.1.x

#cape_disable=bone_capemgr.disable_partno=

#cape_enable=bone_capemgr.enable_partno=

##Changes to enable serial port /dev/ttyO5 and I2C bus 1

cape_enable=bone_capemgr.enable_partno=BB-UART5,BB-I2C1

##enable Generic eMMC Flasher:

##make sure, these tools are installed: dosfstools rsync

#cmdline=init=/opt/scripts/tools/eMMC/init-eMMC-flasher-v3.sh

uuid=fe3a62a1-b2ea-45d3-afcb-f1f66161ef0d

---------------------------------

Does anything there look strange to you?

Dave

[Dave Barndt]

Hi Gautam,

As you requested, here's the output from running fsck on the writable partition after booting/running from the SD card. Nothing that truly unusual I don't think?

root@bbb-dev:~# fsck -f -v /dev/mmcblk1p2

fsck from util-linux 2.25.2

e2fsck 1.43 (17-May-2016)

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Unattached inode 45

Connect to /lost+found<y>? no

Unattached inode 81

Connect to /lost+found<y>? no

Pass 5: Checking group summary information

Block bitmap differences:  +(96768--97021)

Fix<y>? no

Free blocks count wrong for group #2 (1174, counted=1428).

Fix<y>? no

Free blocks count wrong (1091090, counted=1091344).

Fix<y>? no

rootfs_var: ********** WARNING: Filesystem still has errors **********

        5413 inodes used (1.43%, out of 378256)

          20 non-contiguous files (0.4%)

           2 non-contiguous directories (0.0%)

             # of inodes with ind/dind/tind blocks: 0/0/0

             Extent depth histogram: 5389/10

      419310 blocks used (27.76%, out of 1510400)

           0 bad blocks

           1 large file

        3987 regular files

        1408 directories

           0 character device files

           0 block device files

           0 fifos

           0 links

           8 symbolic links (5 fast symbolic links)

           1 socket

------------

        5402 files

root@bbb-dev:~#

------------------------------------------

I did re-run this and fixed the errors, and then the system could boot again from the eMMC.

If you look at the log I posted in the original note, yes, I did eventually lose ("hang") the serial console terminal session once the fsck failed. I couldn't see it waiting for any confirmation, nor did I ever get to some sort of console or login prompt.

And yes, this is going into an actual product. Our H/W folks are looking at building a circuit that will watch for power failure (or an external power button push) and assert the line that the BBB power button currently does, to send a signal to the processor to initiate a shutdown. There is also some sort of backup battery now planned to provide power during the shutdown.

Dave

[minde...@gmail.com]

Hi Dave,

As you said not unusual but I am not sure. As in the earlier post it is better not to have fsck ask for confirmation. I feel there is some problem in the scripts which might have failed and not in the fsck. What is the filesystem type here?

A battery backup maybe in the form of a super capacitor would be something. Just curious as what sort of power event caused so much of problems? Is it just a normal abrupt power failure or is it some kind of intermittent power failure?

Thanks,
Gautam.

On Thursday, December 7, 2017 at 2:21:00 AM UTC+5:30, Dave Barndt wrote:

Hi Gautam,

[Dave Barndt]

Hi Gautam,

The filesystem is ext4; the O/S is debian 8/jessie, the image is the "officially released" BBB image from last spring (bone-debian-8.7-iot-armhf-2017-03-19-4gb.img.xz). To date we have not modified any of the kernel startup scripts, save for setting a serial port baud rate in /etc/rc.local.

I wish I could tell you the power event that caused this to happen - it was a unit in field test and folks there have been pretty much just killing the power to it at various times. We had discussed putting the circuitry in to allow a more graceful shutdown, but are only able to get to it now. That said, power has been shutdown abruptly (power simply removed) literally thousands of times on dozens of BBBs, but we've only seen a couple corruptions. (Separately, we've also seen "bad block" corruptions occasionally, presumably from wear (many, many writes over time).)

Dave

[minde...@gmail.com]

Hi Dave,

 I too have a similar problem but not have worked directly on the problem. This is in medical devices where the instruments power is pulled off abruptly without shutting down. Also the wear and tear of the eMMC and detection of errors etc. The problems become much more serious and I cannot boot the instrument if the complete system is not working at 100%.  Also the problem of run time monitoring to detect any problems in the system and alert the user or shutdown the system.

 I have been thinking of evaluating datalight filesystem solutions for this problem. If your problem is mission critical I am interested in knowing your solutions.

Thanks,
Gautam.

[Dennis Lee Bieber]

On Thu, 7 Dec 2017 19:17:09 -0800 (PST),
minde...@gmail.com declaimed the
following:

> I too have a similar problem but not have worked directly on the problem.
>This is in medical devices where the instruments power is pulled off
>abruptly without shutting down. Also the wear and tear of the eMMC and
>detection of errors etc. The problems become much more serious and I cannot

Somehow I don't see a medical device being approved for usage if it is
sensitive to the above.

I'd suspect the device will need to have an ancillary power management
circuit (super-cap or such to carry through a shutdown cycle with safety
margin) with an intermediate processor monitoring the main power. On
detection of loss of main power, the intermediate would trigger the safe
shutdown of the BBx. The eMMC would only contain the operating firmware
(ideally, it would be a read-only system), any dynamic data (logs, etc.)
should be placed on a field serviceable SD card.

You'll probably also need a few processes running that just performs
checksums/CRCs of the critical firmware, and in extreme cases, periodic
tests of RAM (tricky to do when the OS can put a program anywhere in RAM).
--
Wulfraed Dennis Lee Bieber AF6VN
wlf...@ix.netcom.com HTTP://wlfraed.home.netcom.com/

[ta...@thinnect.com]

On Wednesday, December 6, 2017 at 10:40:11 PM UTC+2, Dave Barndt wrote:

Does anything there look strange to you?

Can't say it does. Your /var/ file system is writable and gets corrupted - this is expected. fsck failing to run is unexpected. 

Looks like systemd has taken over the responsibility of boot-time fsck-ing. A look at the service which fails (/lib/systemd/system/systemd-fsck@.service) tells me that there is some scant documentation available in "man systemd-fsck". However, it looks like you might have to dig into systemd source to analyze how exactly boot-time fsck-s are triggered and what could go wrong there.

Alternatively, perhaps you'd want to have a quick glance at changelog for systemd, to see if they've fixed any bugs in this area since your installed version. My BBB with Debian 8 is running systemd version "230-7~bpo8+2" (dpkg -l systemd) and they've made 5 releases since.

[minde...@gmail.com]

On Friday, December 8, 2017 at 9:05:02 PM UTC+5:30, Dennis Lee Bieber wrote:

On Thu, 7 Dec 2017 19:17:09 -0800 (PST),
minde...@gmail.com declaimed the
following:

> I too have a similar problem but not have worked directly on the problem.
>This is in medical devices where the instruments power is pulled off
>abruptly without shutting down. Also the wear and tear of the eMMC and
>detection of errors etc. The problems become much more serious and I cannot

        Somehow I don't see a medical device being approved for usage if it is
sensitive to the above.

 

Yes it will not get approved if it is sensitive to all of these. Most of the regulated industries will not approve.

        I'd suspect the device will need to have an ancillary power management
circuit (super-cap or such to carry through a shutdown cycle with safety
margin) with an intermediate processor monitoring the main power. On
detection of loss of main power, the intermediate would trigger the safe
shutdown of the BBx. The eMMC would only contain the operating firmware
(ideally, it would be a read-only system), any dynamic data (logs, etc.)
should be placed on a field serviceable SD card.

I have seen this majority in microcontrollers. Being usually low power the super cap is ideal. There is also good voltage supervisors present to monitor the supply voltage.

        You'll probably also need a few processes running that just performs
checksums/CRCs of the critical firmware, and in extreme cases, periodic
tests of RAM (tricky to do when the OS can put a program anywhere in RAM).

I have seen this being done in implantable medical hardware. Have you done something in similar to this. Would really like to hear your story.

Thanks,

Gautam.

[Dennis Lee Bieber]

On Fri, 8 Dec 2017 10:02:33 -0800 (PST),
minde...@gmail.com declaimed the
following:

<SNIP>

>> I have seen this majority in microcontrollers. Being usually low power the
>super cap is ideal. There is also good voltage supervisors present to
>monitor the supply voltage.
>

<SNIP>

>I have seen this being done in implantable medical hardware. Have you done
>something in similar to this. Would really like to hear your story.
>

Prior to a lay-off, my last four years were avionics -- fortunately
mostly maintenance work for existing software (since my prior 30 years
experience qualified as "mainframe number crunching", so the entire
embedded/real-time world was a new experience).

No experience with use of an OS as Linux in such applications -- from
what I could tell, general-purpose OS with real file-systems was only
approved for things like monitoring/logging systems wherein the loss of the
system would have no effect on the operation of the aircraft itself.

The rest tended to follow two concepts:

A) operation out of Flash memory (similar to what one would find on an
Arduino, TIVA C, and similar boards).

B) operation out of RAM with firmware loaded from Flash memory (closer to
BBx/R-Pi, except the Flash is not a general file system and is read-only)

Both likely used some sort of RTOS providing the core of tasking and
IPC (and the highest level having fixed time-slice assigned to tasks with
watch-dogs for overrun detection). In both, analysts created memory maps
defining what code/data is placed where in memory.

(A) tended to have just one copy of the firmware/data in the layout;
(B) tended to have two copies allowing for swap-over if the boot-up checks
detected corruption (and reducing the odds of "bricking" when loading
updates into one copy)

In both, CRC checks were run during booting before passing control to
the real RTOS/application. Data regions tended to have CRC checks performed
periodically during operation (one assignment involved a board with
expanded Flash memory -- where the person responsible for the CRC found he
had to run it in phases as the new data region was too large to be CRCd in
the allowed time-slot). In (B) the CRC was also run before loading the code
into RAM.

A CRC failure would result in the code going into a "halt" state
(usually implemented as an endless loop) or an attempt to restart the
processor -- while control of the craft went to the second flight computer.

Power-glitches (ever watch all the lights flicker on a passenger jet
while they swap over from ground generator to engine generators?) trigger
state save, and later checks of RAM contents (the restart logic would check
a timer -- capacitor bleed down -- to determine if it needed to do full
reboot or take a fast start path), if reading all the RAM does not trigger
a CRC error, control is passed to the application at the position saved.
While a glitch was considered ~10 seconds with no power, it seems some
boards could keep RAM stable for hours after loss of power.

[minde...@gmail.com]

Thanks a lot for the insight. This is very interesting and something which I have seen in medical implants too. There were a lot of checks with the scheduler and also their queues which I don't seem to recall. Also as you have said I too have seen/worked mostly on microcontrollers with some type of qualified RTOS for such regulated devices. Also as you have noticed usually they relegated the Application processor with Linux for monitoring.

Although I have not explored it, I feel that the design can be made much better now with the availability of Cortex M4 in AM57x and the i.MX7 series. With this you have the Application processor and the microcontroller in the same die thereby shrinking the board to a huge extent and also making development much more compact and also simplifying the implementation of safety to the system.

The safety checks can be implemented during boot-up can be done using the POST framework in the boot loader. I have yet to explore this deeply and the possibilities are good.

-Gautam.