DNS-320L-Ax - Alt-F 1.0 - Your md0 raid1 device is degraded after reboot - btrfs

[Paolo Conti]

Hi,

I'm upgrading the disks on my DNS-320L-Ax (Alt-F 1.0) with 2 WD Red 4TB.

Both disks seems good (the health status is "passed" and the smart short test is "Completed without error").

I can create the array (with the wizard, raid1 and filesystem "modern" aka btrfs) and copy files to the NAS without a problem.

BUT, when I reboot the DNS-320L-Ax, the raid1 become degraded (the disk /dev/sda2 is pulled out from the array and only /dev/sdb2 is left working).

If I add again the /dev/sda2 to the array from the Disk->RAID menu, all is working again, until the next reboot.

Parsing the logs, I have found this message: "md: kicking non-fresh sda2 from array!"...

Any hints/suggestions how to fix this?

Now I'm checking the raid1:

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sda2[2] sdb2[1]
      3906064080 blocks super 1.0 [2/2] [UU]
      [==========>..........]  check = 50.7% (1980540544/3906064080) finish=1750.9min speed=18328K/sec

unused devices: <none>

Thanks!

[João Cardoso]

I can reproduce your results (with much smaller disks and filesystems, 80GB)

As I have a serial console attached to the box, I can look at the power off log sequence, and:

md/raid1:md0: Disk failure on sdb2, disabling device.
md/raid1:md0: Operation continuing on 1 devices.

immediately before the btrfs filesystem being unmounted. 

and on reboot:

md: kicking non-fresh sdb2 from array! 
md/raid1:md0: active with 1 out of 2 mirrors

and indeed when mdadm examining sda2/sdb2, their update time and events count are different. Something wrong is happening at the power off sequence.

It is indeed the btrfs initscript that is causing the issue. As 'blkid' reports that sda2 and sdb2 have a btrfs filesystem (which is "true"), that triggers the hotplug removal of sda2/sdb2, which rightly fools mdadm.

I'm afraid that there is no cure for the issue other than stopping the array using Disk->RAID, Stop before doing a poweroff or a reboot; I tried it and it works (off topic: Notice that using the Start button will start the array but will not mount the filesystem, you have to use Disk->Filesystem, FS Operations, Mount for that).

There should be "no problem" with power cuts, as the btrfs initscript is not executed in that case.

So, the 'btrfs' init script should not be used, neither from the command line nor the webUI (Services->System) when btrfs is used as the filesystem of a RAID array.

I will look forward for a definitive fix. If you can, use ext4 on the raid instead.

Thanks

[Paolo Conti]

Thank you João for the detailed explanation.

I'll use the ext4 fs (I'm now moving my data to an external disk before creating the new array).

Thank you again for your amazing support!

[MartenBE]

On Friday, February 8, 2019 at 8:33:14 PM UTC+1, João Cardoso wrote:

I can reproduce your results (with much smaller disks and filesystems, 80GB)

As I have a serial console attached to the box, I can look at the power off log sequence, and:

md/raid1:md0: Disk failure on sdb2, disabling device.
md/raid1:md0: Operation continuing on 1 devices.

immediately before the btrfs filesystem being unmounted. 

and on reboot:

md: kicking non-fresh sdb2 from array! 
md/raid1:md0: active with 1 out of 2 mirrors

and indeed when mdadm examining sda2/sdb2, their update time and events count are different. Something wrong is happening at the power off sequence.

It is indeed the btrfs initscript that is causing the issue. As 'blkid' reports that sda2 and sdb2 have a btrfs filesystem (which is "true"), that triggers the hotplug removal of sda2/sdb2, which rightly fools mdadm.

I'm afraid that there is no cure for the issue other than stopping the array using Disk->RAID, Stop before doing a poweroff or a reboot; I tried it and it works (off topic: Notice that using the Start button will start the array but will not mount the filesystem, you have to use Disk->Filesystem, FS Operations, Mount for that).

There should be "no problem" with power cuts, as the btrfs initscript is not executed in that case.

So, the 'btrfs' init script should not be used, neither from the command line nor the webUI (Services->System) when btrfs is used as the filesystem of a RAID array.

I will look forward for a definitive fix. If you can, use ext4 on the raid instead.

Thanks

Dear João,

I was planning to convert my DNS320 into a BTRFS RAID1, but then I came across this thread. Is it advised to not use a BTRFS RAID1 until this issue is fixed?
Can we assist in anyway to fix this issue?

With kind regards,

Marten

 

[João Cardoso]

Yes, don't use BTRFS with RAID1. This applies only to Alt-F.

Can we assist in anyway to fix this issue?

Thanks, but you can't help (unless you want to take over or fork Alt-F :)

As the bug is in the firmware itself, not on a disk installable package, it can only be fixed on a new firmware release. That's the curse of embedded devices.

With kind regards,

Marten

 

[MartenBE]

On Monday, May 6, 2019 at 7:17:58 PM UTC+2, João Cardoso wrote:

Yes, don't use BTRFS with RAID1. This applies only to Alt-F.

Thanks, but you can't help (unless you want to take over or fork Alt-F :)

As the bug is in the firmware itself, not on a disk installable package, it can only be fixed on a new firmware release. That's the curse of embedded devices.

Ah, I see. I look forward to the next release then :) I know C++ and Linux, so maybe I'll try to see if I can make understand some of the codebase.
I just want to say that I really appreciate your time and effort into this project. I've recently got this secondhand DNS 320, and when I heard Alt-F exists it really made my day :) Thank you very much for this!

With kind regards,

Marten