Fixing errors on a BTRFS partition?

[Nate Bargmann]

I have a Freedom Box Pioneer (hardware is an Olimex A20-OLinuXino-LIME2
unit with a Samsung 128 GB micro-SD card. The micro-SD is partitioned
into 2GB boot ext2 and the remainder as the root partition as BTRFS.

The thing has been crashing for months and now it started giving GPG
signature errors when trying to run 'apt update'. I copied the entire
micro-SD card to an image file with dd so I have a backup. Running
'btrfs check' resulted in a lot of errors so I ran the check and
directed the output to a file which is over 2MB in size! The following
is a small snippet of what it in the file:

[1/7] checking root items
[2/7] checking extents
checksum verify failed on 2337062912 found 00000098 wanted 00000025
checksum verify failed on 2337062912 found 00000098 wanted 00000025
Csum didn't match
owner ref check failed [2337062912 16384]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
checksum verify failed on 2337062912 found 00000098 wanted 00000025
checksum verify failed on 2337062912 found 00000098 wanted 00000025
Csum didn't match
root 11670 inode 1109704 errors 200, dir isize wrong
root 11670 inode 1109705 errors 2001, no inode item, link count wrong
unresolved ref dir 1109704 index 2 namelen 4 name json filetype 1 errors 4, no inode ref
root 11670 inode 1109706 errors 2001, no inode item, link count wrong
unresolved ref dir 1095383 index 2 namelen 11 name 20-json.ini filetype 7 errors 4, no inode ref
root 11670 inode 1109707 errors 2001, no inode item, link count wrong
unresolved ref dir 978863 index 4 namelen 7 name apache2 filetype 2 errors 4, no inode ref
root 11670 inode 1109710 errors 2001, no inode item, link count wrong
unresolved ref dir 1095409 index 2 namelen 11 name 20-json.ini filetype 7 errors 4, no inode ref
root 11670 inode 1109711 errors 2001, no inode item, link count wrong
unresolved ref dir 978863 index 5 namelen 3 name fpm filetype 2 errors 4, no inode ref
root 11670 inode 1109714 errors 2001, no inode item, link count wrong
unresolved ref dir 978864 index 30 namelen 4 name json filetype 1 errors 4, no inode ref
root 11670 inode 1109734 errors 2001, no inode item, link count wrong
unresolved ref dir 45938 index 176 namelen 17 name gschemas.compiled filetype 1 errors 4, no inode ref
root 11670 inode 1109735 errors 2001, no inode item, link count wrong
unresolved ref dir 6679 index 36 namelen 15 name giomodule.cache filetype 1 errors 4, no inode ref
root 11670 inode 1109771 errors 2001, no inode item, link count wrong
unresolved ref dir 295871 index 242 namelen 24 name rsyslog.service.dsh-also filetype 1 errors 4, no inode ref
root 11670 inode 1109784 errors 2001, no inode item, link count wrong
unresolved ref dir 978742 index 31 namelen 12 name readline.ini filetype 1 errors 4, no inode ref
.
.
.
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mmcblk0p2
UUID: ea375ed2-d6e7-49d4-9b19-a624ba09b96c
The following tree block(s) is corrupted in tree 11670:
tree block bytenr: 6562955264, level: 1, node key: (1109704, 96, 3)
found 19331854402 bytes used, error(s) found
total csum bytes: 14201108
total tree bytes: 1242775552
total fs tree bytes: 1160757248
total extent tree bytes: 61292544
btree space waste bytes: 327420862
file data blocks allocated: 182356692992
referenced 113920880640


Everything online hints that attempting repair is particularly
dangerous, but what else am I to do? At the moment the system is pretty
much useless.

All insights appreciated.

- Nate

--
"The optimist proclaims that we live in the best of all
possible worlds. The pessimist fears this is true."
Web: https://www.n0nb.us
Projects: https://github.com/N0NB
GPG fingerprint: 82D6 4F6B 0E67 CD41 F689 BBA6 FB2C 5130 D55A 8819

[piorunz]

On 12/01/2023 13:42, Nate Bargmann wrote:
> I have a Freedom Box Pioneer (hardware is an Olimex A20-OLinuXino-LIME2
> unit with a Samsung 128 GB micro-SD card. The micro-SD is partitioned
> into 2GB boot ext2 and the remainder as the root partition as BTRFS.
>
> The thing has been crashing for months and now it started giving GPG
> signature errors when trying to run 'apt update'. I copied the entire
> micro-SD card to an image file with dd so I have a backup. Running
> 'btrfs check' resulted in a lot of errors so I ran the check and
> directed the output to a file which is over 2MB in size! The following
> is a small snippet of what it in the file:
>
> [1/7] checking root items

> [2/7] checking extents (...)

> Everything online hints that attempting repair is particularly
> dangerous, but what else am I to do? At the moment the system is pretty
> much useless.

Hi Nate, your hardware has failed and corrupted files were detected by
excellent checksumming functionality of Btrfs. Checksums stored does not
match the file read, and error is shown. Additionally, access may be
denied to a corrupted file, creating more problems. Its good that you
created backup of the card, so you can recover things later if you
really need to.

You don't have any redundancy on that single card, therefore recovering
100% of damaged files is impossible, your data is lost. Also, repairing
filesystem to usable state (like deleting corrupted files and so on)
will not fix underlying hardware issue. I suggest obtaining another card
and installing system to it from the scratch.

To satiate your curiosity, you can find out what files are corrupted,
some of the errors are giving filenames. If not, this is my saved
command to obtain filename from inode numbers:
sudo btrfs inspect-internal inode-resolve 50845 /

And obtain filename from logical error:
sudo btrfs inspect-internal logical-resolve -v 540115857408 /

As far as I know, Btrfs may refuse to read file with wrong checksum,
there may be another command to do that.

--
With kindest regards, Piotr.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀⠀⠀

[Dan Ritter]

Nate Bargmann wrote:
> I have a Freedom Box Pioneer (hardware is an Olimex A20-OLinuXino-LIME2
> unit with a Samsung 128 GB micro-SD card. The micro-SD is partitioned
> into 2GB boot ext2 and the remainder as the root partition as BTRFS.
>
> The thing has been crashing for months

For the future: don't let things go this long. I know it's
tempting to say "maybe it won't happen again", but the second
time should be the last time before you take action.

> and now it started giving GPG
> signature errors when trying to run 'apt update'. I copied the entire
> micro-SD card to an image file with dd so I have a backup. Running
> 'btrfs check' resulted in a lot of errors so I ran the check and
> directed the output to a file which is over 2MB in size! The following
> is a small snippet of what it in the file:

...

> Everything online hints that attempting repair is particularly
> dangerous, but what else am I to do? At the moment the system is pretty
> much useless.

1: get a new card, or, much better, replace with a SATA
SSD. (I see the Olimex has a SATA port. Use it!) Here's a
https://www.newegg.com/adata-ultimate-su800-128gb/p/N82E16820215015?Item=9SIAJNUBMB4508
$29 128GB SSD from a reputable manufacturer.

2: Reinstall Debian on the new disk. Don't use btrfs on a
single-drive system; only use btrfs on a mirrored system. In most cases,
use ext4.

3: copy all the data you can from the image file.

-dsr-

[Joseph Loo]

Try this article

https://en.opensuse.org/SDB:BTRFS

[Nito]

On Thu, Jan 12, 2023 at 14:10:22 +0000, piorunz wrote:
> To satiate your curiosity, you can find out what files are corrupted, some
> of the errors are giving filenames. If not, this is my saved command to
> obtain filename from inode numbers:
> sudo btrfs inspect-internal inode-resolve 50845 /
>
> And obtain filename from logical error:
> sudo btrfs inspect-internal logical-resolve -v 540115857408 /
>
> As far as I know, Btrfs may refuse to read file with wrong checksum, there
> may be another command to do that.

I believe mounting with -o rescue=all will allow to read corrupted files,
so the remaining parts of non-replaceable files can be salvaged; though I
fortunately never had to put this to test myself. Specifically the
ignoredatacsums setting, see
https://btrfs.readthedocs.io/en/stable/Administration.html#mount-options

Note that this requires a sufficiently new kernel.

On Thu, Jan 12, 2023 at 08:58:52 -0500, Dan Ritter wrote:
> For the future: don't let things go this long. I know it's
> tempting to say "maybe it won't happen again", but the second
> time should be the last time before you take action.

To add to this: usually it is recommended to regularly run `btrfs scrub`
to detect (and in case of redunancy repair) data degradation. Running also
`btrfs check` regularly to verify the fs structure also isn't a bad idea.
This makes it more likely you'll notice a failing device before it gets
as bad as it is now.

> 2: Reinstall Debian on the new disk. Don't use btrfs on a
> single-drive system; only use btrfs on a mirrored system. In most cases,
> use ext4.

For the record, I don't agree with BTRFS only being useful with RAID.
BTRFS’ scrub and check can detect corruption early, rather then bitrot
silently continuing until it starts crashing and there might be other
features like e.g. transparent compression which might make it worthwhile
for one’s single-device usecases.
(In theory redunancy can also be used with a single device by creating the
FS with the "dup" data profile, but this of course offers les protection
than separate devices.)

[Intense Red]

> Everything online hints that attempting repair is particularly
> dangerous, but what else am I to do?

You sum up my experience with BTRFS. I too was "scared" off from it and
reformatted my BTRFS partitions and went back to ext4 -- it's a known
quantity fit for humans with tons of advice of how to handle problems/errors.


--
"The US has never had a 'foreign policy' but a fanatical domestic policy
which, once it had bled through to the Pacific, sought new hosts on which to
feed." -- Patrick Wilkinson.

[Nate Bargmann]

* On 2023 12 Jan 08:15 -0600, Dan Ritter wrote:
> Nate Bargmann wrote:
> > I have a Freedom Box Pioneer (hardware is an Olimex A20-OLinuXino-LIME2
> > unit with a Samsung 128 GB micro-SD card. The micro-SD is partitioned
> > into 2GB boot ext2 and the remainder as the root partition as BTRFS.
> >
> > The thing has been crashing for months
>
> For the future: don't let things go this long. I know it's
> tempting to say "maybe it won't happen again", but the second
> time should be the last time before you take action.

At one point I replaced another piece of hardware that was on the same
Ethernet switch as this unit and the crashes cleared up for a while.
Then I suspected a flaky power adapted but haven't addressed it and then
I suspected RF from my amateur radio operations and put the power cord
in a ferrite core with no positive results. It wasn't until the 'apt
update' GPG failure this morning (the Freedom box image is setup to auto
update) in a manual update attempt that the light bulb lit up.

> > and now it started giving GPG
> > signature errors when trying to run 'apt update'. I copied the entire
> > micro-SD card to an image file with dd so I have a backup. Running
> > 'btrfs check' resulted in a lot of errors so I ran the check and
> > directed the output to a file which is over 2MB in size! The following
> > is a small snippet of what it in the file:
>
> ...
>
> > Everything online hints that attempting repair is particularly
> > dangerous, but what else am I to do? At the moment the system is pretty
> > much useless.
>
>
> 1: get a new card, or, much better, replace with a SATA
> SSD. (I see the Olimex has a SATA port. Use it!) Here's a
> https://www.newegg.com/adata-ultimate-su800-128gb/p/N82E16820215015?Item=9SIAJNUBMB4508
> $29 128GB SSD from a reputable manufacturer.
>
> 2: Reinstall Debian on the new disk. Don't use btrfs on a
> single-drive system; only use btrfs on a mirrored system. In most cases,
> use ext4.
>
> 3: copy all the data you can from the image file.

Looks like a plan I will do. I had an SATA drive on it to begin with
but then decided to just use the micro-SD card. My guess that a quite
large card with much excess capacity would wear level enough for long
life. Well, maybe 18 months or so is "long life".

For the record, the Freedom Box micro-SD card image formats the root
partition as BTRFS. It wasn't a choice of mine. I've used ext2/3/4 for
many years and that system has always done well for me.

Fortunately this isn't a system that is critical, but it does serve some
purposes.

[Nate Bargmann]

* On 2023 12 Jan 16:58 -0600, Intense Red wrote:
> > Everything online hints that attempting repair is particularly
> > dangerous, but what else am I to do?
>
> You sum up my experience with BTRFS. I too was "scared" off from it and
> reformatted my BTRFS partitions and went back to ext4 -- it's a known
> quantity fit for humans with tons of advice of how to handle problems/errors.

I had experimented with BTRFS some years ago as its virtual partitions
feature is attractive for things like tmp, var, and usr where each is
"separate" but is part of a larger fixed partition. Choosing proper
sizes is eased somewhat. Other than that its not my choice, usually.
In this case the card came already formatted with the root partition as
BTRFS so I left it alone.

An SSD is on order. I still have to use a micro-SD card for the boot
partition as the MICRO cannot boot directly to the SSD as far as I know.

[Jeffrey Walton]

On Thu, Jan 12, 2023 at 8:43 AM Nate Bargmann <n0...@n0nb.us> wrote:
>
> I have a Freedom Box Pioneer (hardware is an Olimex A20-OLinuXino-LIME2
> unit with a Samsung 128 GB micro-SD card. The micro-SD is partitioned
> into 2GB boot ext2 and the remainder as the root partition as BTRFS.
>
> The thing has been crashing for months and now it started giving GPG
> signature errors when trying to run 'apt update'. I copied the entire
> micro-SD card to an image file with dd so I have a backup. Running
> 'btrfs check' resulted in a lot of errors so I ran the check and
> directed the output to a file which is over 2MB in size! The following
> is a small snippet of what it in the file:

The microSD card is failing. Replace it.

Jeff

[Andy Smith]

Hello,

On Thu, Jan 12, 2023 at 04:57:07PM -0600, Intense Red wrote:
> > Everything online hints that attempting repair is particularly
> > dangerous, but what else am I to do?
>
> You sum up my experience with BTRFS. I too was "scared" off from it and
> reformatted my BTRFS partitions and went back to ext4 -- it's a known
> quantity fit for humans with tons of advice of how to handle problems/errors.

I too don't have a lot of love for btrfs, but I think it is a bit
unfair to criticise it in this scenario, which is a failing SD card
with no redundancy. If there'd been redundancy then btrfs should
have noticed the problems and got the data from the other
copy/copies, but here it had no opportunity to do so.

In the same situation, ext4 would have just carried on until it got
read/write errors but this wouldn't have been any better. btrfs got
the same errors and reported more of its own that it noticed from
the incorrect checksums.

It sounds like the OP's use case didn't involve redundancy nor did
it involve any of the other btrfs features such as compression or
snapshots, so btrfs probably wasn't a good choice here. ext4 or XFS
may have been better here just because they are simpler. I can
understand not wanting to have a learning experience when it comes
to one's data.

The btrfs mailing list is pretty helpful. I think if one was not
scared to ask for help there regarding unfamiliar steps, btrfs in a
redundant configuration is pretty safe for your data. My gripes with
it over the years have been user interface, bugs and availability
issues, not resilience i.e. I've never lost data to it.

Having said that, I really do not trust things like SD cards or
CompactFlash cards (remember those? I still have one in service in a
firewall) for main storage unless they will be largely or entirely
read-only. In my mind they are more suited to phones and cameras and
similar devices.

Similarly, USB storage. Attaching a USB drive (or stick!) to an
Intel NUC or Raspberry Pi and exposing that over SMB is what some
people consider network attached storage, and I'm sure there's
people reading this who have done this for years and never had a
problem, but I've had and seen so many problems with non-trivial use
of USB storage. For myself, I'd want any long term setup to be
attached by SATA or NVMe or over network to same.

Cheers,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

[piorunz]

Guys,

Quick offtop question as we are talking about Btrfs:

How do you replace drives in Btrfs Raid10 array?
I am in the process of upgrading 4 drives to twice as big ones. Since
forever, I've been using this (example):

sudo btrfs device add -f /dev/sdf1 /home
sudo btrfs device remove /dev/sdd1 /home

But now I am reading, newer fancy method is:
sudo btrfs replace /dev/sdd1 /dev/sdf1 /home

Which one is better please? For both situation where drive has failed or
is still operational?

[Nate Bargmann]

* On 2023 15 Jan 10:07 -0600, Andy Smith wrote:
> Hello,
>
> On Thu, Jan 12, 2023 at 04:57:07PM -0600, Intense Red wrote:
> > > Everything online hints that attempting repair is particularly
> > > dangerous, but what else am I to do?
> >
> > You sum up my experience with BTRFS. I too was "scared" off from it and
> > reformatted my BTRFS partitions and went back to ext4 -- it's a known
> > quantity fit for humans with tons of advice of how to handle problems/errors.
>
> I too don't have a lot of love for btrfs, but I think it is a bit
> unfair to criticise it in this scenario, which is a failing SD card
> with no redundancy. If there'd been redundancy then btrfs should
> have noticed the problems and got the data from the other
> copy/copies, but here it had no opportunity to do so.
>
> In the same situation, ext4 would have just carried on until it got
> read/write errors but this wouldn't have been any better. btrfs got
> the same errors and reported more of its own that it noticed from
> the incorrect checksums.
>
> It sounds like the OP's use case didn't involve redundancy nor did
> it involve any of the other btrfs features such as compression or
> snapshots, so btrfs probably wasn't a good choice here. ext4 or XFS
> may have been better here just because they are simpler. I can
> understand not wanting to have a learning experience when it comes
> to one's data.

Perhaps this needs to be taken up with the Freedom Box Foundation (it
has close ties to Debian) as it is the image they provide that chose
btrfs to be written to the micro-SD card and used as an appliance on the
Freedom Box Pioneer hardware (Olimex A20-OLinuXino-LIME2).

*I* did not choose this filesystem for this application, just to be
clear. If there is a better choice for what is intended to be an
appliance running from a micro-SD card, then that should be communicated
to the Freedom Box people.

I have an SSD on order and will rebuild with ext4.

[Nate Bargmann]

Well, I didn't fix the errors, but I was able to use 'btrfs replace' to
move the file system to an external HDD. The SDD I ordered apparently
is ping-ponging its way from Kansas City to various area post offices
and back again before they get it on the right truck. Sigh...