No files after power failure (aufs au_xino_do_write)

[Eric Wick]

Got a power failure, the box waits shutted down. Now after reboot there are no files and folders in filesystem visible except the usb and md0 mounts.

# ls -la /
total 0
# ls -la /etc
total 0

# ls -la /mnt
total 12
drwxr-xr-x    4 root     root            80 Jun 13 22:18 .
drwxr-xr-x   25 root     root          4096 Jun 13 22:18 ..
drwxr-xr-x    9 root     root          4096 Oct 16  2011 md0
drwxr-xr-x    4 root     root          4096 Jun  8 20:56 sdc1

The webif and dropbear is running, minidlna has rescanned and is up. In dmesg i see this

aufs au_xino_write:414:ls[635]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:ls[643]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:ls[643]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:ls[649]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:ls[649]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:ls[652]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:ls[652]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:sh[637]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:sh[637]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:ls[901]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:ls[901]: I/O Error, write failed (-5)

Is this a filesystem error in the aufs? How to fix that?

Eric

[Joao Cardoso]

On Thursday, June 14, 2012 4:53:40 AM UTC+1, Eric Wick wrote:

Got a power failure, the box waits shutted down. Now after reboot there are no files and folders in filesystem visible except the usb and md0 mounts.

What filesystems are missing? What else do you expect?

What do you mean with "there are no files and folders in filesystem visible"? when using samba? using the command line? At the administrative web pages?

 

# ls -la /
total 0
# ls -la /etc
total 0

# ls -la /mnt
total 12
drwxr-xr-x    4 root     root            80 Jun 13 22:18 .
drwxr-xr-x   25 root     root          4096 Jun 13 22:18 ..
drwxr-xr-x    9 root     root          4096 Oct 16  2011 md0
drwxr-xr-x    4 root     root          4096 Jun  8 20:56 sdc1

The webif and dropbear is running, minidlna has rescanned and is up. In dmesg i see this

aufs au_xino_write:414:ls[635]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:ls[643]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:ls[643]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:ls[649]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:ls[649]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:ls[652]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:ls[652]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:sh[637]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:sh[637]: I/O Error, write failed (-5)
aufs au_xino_do_write:378:ls[901]: I/O Error, write failed (4294967268)
aufs au_xino_write:414:ls[901]: I/O Error, write failed (-5)

Is this a filesystem error in the aufs? How to fix that?

This is a problem with 'aufs', that uses the Alt-F folder (notice the caps) to store disk-instalable packages. Read (only) the HowToFixOrCustomizeFirmware wiki.

Please post the output of

   aufs.sh -l

Where is the Alt-F folder located? '/Alt-F' is a symbolic link to it, post the output of

   ls -l /

   ls -l /Alt-F/etc

Is the filesystem where Alt-F resides in read-only (RO) or Read/Write (RW?) mode? Post the output of

   cat /proc/mounts

Have you been "playing" with the Alt-F folder?

I can't understand how can minidlna (or the whole system, as a matter of fact) be running with an empty /etc folder...

Joao

[Eric Wick]

One reboot later, the files and folfers are visible again. I login with ssh to see this.

What filesystems are missing? What else do you expect?

What do you mean with "there are no files and folders in filesystem visible"? when using samba? using the command line? At the administrative web pages?

Directly on the filesystem when login'ed with ssh.

Please post the output of

   aufs.sh -l

# aufs.sh -l
aufs on / type aufs (rw,relatime,si=4193eff1)
/mnt/md0/Alt-F=rw
/rootmnt/root=rw
/rootmnt/rootsq=rr
 

Where is the Alt-F folder located? '/Alt-F' is a symbolic link to it, post the output of

   ls -l /

   ls -l /Alt-F/etc

# ls -l /
total 44
lrwxrwxrwx    1 root     root            14 Jun 14 16:52 Alt-F -> /mnt/md0/Alt-F
lrwxrwxrwx    1 root     root            15 Jun 14 16:52 Backup -> /mnt/md0/Backup
-rw-r--r--    1 root     root         17987 Sep 24  2010 COPYING
-rw-r--r--    1 root     root           245 Sep 24  2010 LICENCE
lrwxrwxrwx    1 root     root            15 Jun 14 16:52 Public -> /mnt/md0/Public
-rw-r--r--    1 root     root           115 Jun 14 16:52 README.txt
drwxr-xr-x    2 root     root          1520 Feb 21 19:34 bin
drwxr-xr-x    2 root     root            40 Feb 21 16:31 boot
drwxr-xr-x    4 root     root          1580 Jun 14 16:52 dev
drwxr-xr-x   11 root     root          4096 Jun 14 17:10 etc
lrwxrwxrwx    1 root     root            14 Jun 14 16:52 home -> /mnt/md0/Users
-rwxr-xr-x    1 root     root          2440 Dec  6  2010 init
drwxr-xr-x    2 root     root           500 Feb 21 19:34 lib
drwxr-xr-x    4 root     root            80 Jun 14 16:52 mnt
dr-xr-xr-x   59 root     root             0 Jan  1  1970 proc
drwxr-x---    2 root     root           100 Jun 14 16:53 root
drwxrwxrwt    4 root     root           100 Jun 14 16:52 rootmnt
drwxr-xr-x    2 root     root          1340 Jun 14 16:52 sbin
drwxr-xr-x   11 root     root             0 Jun 14 16:52 sys
drwxrwxrwt    6 root     root           160 Jun 14 17:13 tmp
drwxrwxrwt    2 root     root            40 Jun 14 16:52 tmproot
drwxr-xr-x   15 root     root          4096 Feb 21 20:27 usr
drwxr-xr-x    6 root     root          4096 Apr 12  2011 var

# ls -l /Alt-F/etc
total 40
-rw-r--r--    1 root     root           853 Jun 14 17:10 blkid.tab
-rw-r--r--    1 root     root           309 Jun 14 17:10 blkid.tab.old
-rw-------    1 root     root           136 Jun 14 16:52 fstab
drwxr-xr-x    2 12       multimed      4096 May  3  2011 fuppes
-rw-r--r--    1 root     root           226 May  3  2011 group-
drwxr-xr-x    2 root     root          4096 Mar 10 12:01 init.d
-rw-r--r--    1 root     root          2046 Jun  8 21:01 minidlna.co-bak
-rw-r--r--    1 minidlna multimed      2048 Jun  8 21:03 minidlna.conf
-rw-r--r--    1 root     root           645 May  3  2011 passwd-
-rw-------    1 root     root           372 May  3  2011 shadow-
 

Is the filesystem where Alt-F resides in read-only (RO) or Read/Write (RW?) mode? Post the output of

   cat /proc/mounts

Seems to be rw now
# cat /proc/mounts
rootfs / rootfs rw 0 0
tmpfs /rootmnt tmpfs rw,relatime 0 0
aufs / aufs rw,relatime,si=4193eff1 0 0
/dev/loop0 /rootmnt/rootsq squashfs ro,relatime 0 0
proc /proc proc rw,relatime 0 0
usbfs /proc/bus/usb usbfs rw,relatime,devgid=9,devmode=664 0 0
sysfs /sys sysfs rw,relatime 0 0
tmpfs /tmp tmpfs rw,relatime,size=106496k 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
/dev/md0 /mnt/md0 ext4 rw,relatime,barrier=1,data=ordered 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
/dev/sdc1 /mnt/sdc1 ext2 rw,relatime,errors=continue 0 0
 

Have you been "playing" with the Alt-F folder?

I can't understand how can minidlna (or the whole system, as a matter of fact) be running with an empty /etc folder...

Only with vi on the /etc/minidlna.conf and /etc/init.d/S61smb, these changes seems to survice full reboots.

The problems seems to go away. Is it possible that we have a race condition when the raid filesystem is dirty and alt-f is booting?

Eric

[Joao Cardoso]

On Thursday, June 14, 2012 5:08:08 PM UTC+1, Eric Wick wrote:

One reboot later, the files and folfers are visible again. I login with ssh to see this.

 

... 

 

Have you been "playing" with the Alt-F folder?

I can't understand how can minidlna (or the whole system, as a matter of fact) be running with an empty /etc folder...

Only with vi on the /etc/minidlna.conf and /etc/init.d/S61smb, these changes seems to survice full reboots.

No problem, as long as you don't edit nothing directly under /Alt-F/<whatever>.

E.g. /etc/minidlna.conf is really /Alt-F/etc/minidlna.conf; you can edit /etc/minidlna.conf and *NEVER* /Alt-F/etc/minidlna.conf

 

The problems seems to go away. Is it possible that we have a race condition when the raid filesystem is dirty and alt-f is booting?

The most probable reason was that md0 was mounted read-only, because of the filesystem being checked after the unclean shutdown, and as such some (controlled) operations that write to /Alt-F/etc/<whatever> failed. Can't say for sure.

Joao

 

Eric

[Bink]

Hi - I seem to have a similar problem. I seem to have lost my filesystem. From the admin panel, I find nothing in the directories menu and when I go to Packages:Alt-F I get:

No ipkg instalation found, install ipkg in:   (*no choices available in dropdown)

There was no power failure and I wasn't editing anything. The only activity on the box was a torrent downloading over night. I've done several restarts and get nothing. Most everything is backed up, but wanted to double-check before give up reformat/get a new drive. Below is the output from the questions you asked Erik.

Thanks for your help. 

 

# aufs.sh -l

aufs on / type aufs (rw,relatime,si=41938bf1)

/rootmnt/root=rw

/rootmnt/rootsq=rr

# 

#  ls -l /

total 28

-rw-r--r--    1 root     root         17987 Sep 25  2010 COPYING

-rw-r--r--    1 root     root           245 Sep 25  2010 LICENCE

drwxr-xr-x    2 root     root          1520 Feb 22 01:34 bin

drwxr-xr-x    2 root     root            40 Feb 21 22:31 boot

drwxr-xr-x    3 root     root          1400 Aug  2 09:56 dev

drwxr-xr-x    9 root     root           980 Aug  2 10:17 etc

-rwxr-xr-x    1 root     root          2440 Dec  7  2010 init

drwxr-xr-x    2 root     root           500 Feb 22 01:34 lib

drwxr-xr-x    4 root     root            80 Aug  2 09:56 mnt

dr-xr-xr-x   45 root     root             0 Jan  1  1970 proc

drwxr-x---    2 root     root           100 Aug 11 14:34 root

drwxrwxrwt    4 root     root           100 Aug  2 09:56 rootmnt

drwxr-xr-x    2 root     root          1320 Aug  2 09:56 sbin

drwxr-xr-x   11 root     root             0 Aug  2 09:56 sys

drwxrwxrwt    6 root     root           180 Aug 11 14:25 tmp

drwxrwxrwt    2 root     root            40 Aug  2 09:56 tmproot

drwxr-xr-x   10 root     root            60 Feb 22 01:34 usr

drwxr-xr-x    4 root     root           180 Aug  2 09:56 var

# 

#  ls -l /Alt-F/etc

ls: /Alt-F/etc: No such file or directory

# 

#  cat /proc/mounts

rootfs / rootfs rw 0 0

tmpfs /rootmnt tmpfs rw,relatime 0 0

aufs / aufs rw,relatime,si=41938bf1 0 0

/dev/loop0 /rootmnt/rootsq squashfs ro,relatime 0 0

proc /proc proc rw,relatime 0 0

usbfs /proc/bus/usb usbfs rw,relatime,devgid=9,devmode=664 0 0

sysfs /sys sysfs rw,relatime 0 0

tmpfs /tmp tmpfs rw,relatime,size=54272k 0 0

devpts /dev/pts devpts rw,relatime,mode=600 0 0

#  

[Joao Cardoso]

On Saturday, August 11, 2012 8:56:10 AM UTC+1, Bink wrote:

Hi - I seem to have a similar problem. I seem to have lost my filesystem. From the admin panel, I find nothing in the directories menu and when I go to Packages:Alt-F I get:

No ipkg instalation found, install ipkg in:   (*no choices available in dropdown)

There was no power failure and I wasn't editing anything. The only activity on the box was a torrent downloading over night. I've done several restarts and get nothing. Most everything is backed up, but wanted to double-check before give up reformat/get a new drive. Below is the output from the questions you asked Erik.

Thanks for your help. 

This seems to be different from the initial report, as  Eric had md0 and its data available, "only" the system files were missing.

Do you have RAID or a "standard" disk layout?
Is the power led blinking with an hearth-beat rhythm? Any disk led activity?
In the Status page, do you see any filesystem maintenance going on? Do disks appear?

Can you please attach the System and aux* logs? System->Utilities->View Logs

[Bink]

Thanks for helping. Here's the info you requested:

• No Raid, just a single drive in left bay
• Power LED not blinking. No disk LED activity
• Status of disks on status page:    left sda WDC WD15EADS-00P8B0 1500.3 GB active or idle 42°C/107.6°F passed
• Logs attached

Thanks!

[Joao Cardoso]

You have a hardware problem (from the System Log), the disk can't be accessed.

You can try to put the drive in another computer, or power-down the box and unplug the power cord; sometimes such problems are intermittent, but I wouldn't have many confidence in the drive (or the box).

Haven't you had previous warnings from SMART (Disk Health)?