How to make btrfs forget a disk?

[Victor Sudakov]

Dear Colleagues,

btrfs thinks that /dev/nvme1n1 has a btrfs:

# btrfs filesystem show
Label: none uuid: 3414ae53-f3d4-43ea-bb88-ffefc9bc86f6
Total devices 1 FS bytes used 1.05TiB
devid 1 size 2.00TiB used 1.33TiB path /dev/nvme0n1

Label: none uuid: 38f74bc8-465d-4866-8ec1-3a144741012c
Total devices 1 FS bytes used 831.16GiB
devid 1 size 3.00TiB used 1.48TiB path /dev/nvme1n1

The problem is that /dev/nvme1n1 is being used for ZFS now, and there is
currently no btrfs thereon. However, there is a btrfs label or something
stuck somewhere, how can I clear it?

I tried to unload/load the btrfs kernel module but it did not help.
It's somewhere on disk, but where?

# blkid | grep nvme1n1
/dev/nvme1n1: UUID="38f74bc8-465d-4866-8ec1-3a144741012c" UUID_SUB="ada72e33-4467-4413-b78a-1a2392f62e62" TYPE="btrfs" PTUUID="d73a33f2-2b34-e64b-bc66-128320256a28" PTTYPE="gpt"
/dev/nvme1n1p1: LABEL="fastdrive" UUID="9760009171611183151" UUID_SUB="5246836986761113023" TYPE="zfs_member" PARTLABEL="zfs-04c563a98b6424bd" PARTUUID="ca163a7a-0150-714c-8e5f-375f57a8df2c"
/dev/nvme1n1p9: PARTUUID="bbc56956-5581-b449-a114-0ece0378a4c9

# Disk /dev/nvme1n1: 3 TiB, 3298534883328 bytes, 6442450944 sectors
Disk model: Amazon Elastic Block Store
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D73A33F2-2B34-E64B-BC66-128320256A28

Device Start End Sectors Size Type
/dev/nvme1n1p1 2048 6442432511 6442430464 3T Solaris /usr & Apple ZFS
/dev/nvme1n1p9 6442432512 6442448895 16384 8M Solaris reserved 1

--
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[David]

On Fri, 12 Mar 2021 at 13:39, Victor Sudakov <v...@sibptus.ru> wrote:

> btrfs thinks that /dev/nvme1n1 has a btrfs:

> # btrfs filesystem show
> Label: none uuid: 3414ae53-f3d4-43ea-bb88-ffefc9bc86f6
> Total devices 1 FS bytes used 1.05TiB
> devid 1 size 2.00TiB used 1.33TiB path /dev/nvme0n1
>
> Label: none uuid: 38f74bc8-465d-4866-8ec1-3a144741012c
> Total devices 1 FS bytes used 831.16GiB
> devid 1 size 3.00TiB used 1.48TiB path /dev/nvme1n1
>
> The problem is that /dev/nvme1n1 is being used for ZFS now, and there is
> currently no btrfs thereon. However, there is a btrfs label or something
> stuck somewhere, how can I clear it?

Hi,

I do not know the answer because I have never done that,
but try reading
man 8 btrfs-device

and then perhaps
btrfs device remove ...

[Victor Sudakov]

"Remove device(s) from a filesystem identified by <path>"

Hmm. /dev/nvme1n1 is not identified by any path because it's not mounted
as a btrfs filesystem.

[Victor Sudakov]

"wipefs -t btrfs -f -a /dev/nvme1n1" did the job.

Still wondering where those labels are stored on disk in Linux.

In FreeBSD, GEOM(4) usually keeps such stuff in the last sector of a
volume/device.

[deloptes]

Victor Sudakov wrote:

> "wipefs -t btrfs -f -a /dev/nvme1n1" did the job.
>
> Still wondering where those labels are stored on disk in Linux.
>

FS Superblock?

> In FreeBSD, GEOM(4) usually keeps such stuff in the last sector of a
> volume/device.

I think it depends on the FS not on the OS.

if search engines are not working where you live, I think this is a good
howto (just found it among the top 10hits in duckduckgo)

https://www.cyberciti.biz/faq/howto-use-wipefs-to-wipe-a-signature-from-disk-on-linux/

[Anssi Saari]

Victor Sudakov <v...@sibptus.ru> writes:

> "wipefs -t btrfs -f -a /dev/nvme1n1" did the job.
>
> Still wondering where those labels are stored on disk in Linux.

Didn't wipefs tell you? I don't have any btrfs but for me wipefs prints
at which offset it found which label. Like this, for a Linux swap
partition:

# wipefs /dev/sdb5
DEVICE OFFSET TYPE UUID LABEL
sdb5 0xff6 swap 8af9fb67-38ca-4c91-bbbd-d53c5ac1f30a

And if I hexdump a bit of /dev/sdb5 I get:

000ff0 00 00 00 00 00 00 53 57 41 50 53 50 41 43 45 32 >......SWAPSPACE2<
001000 01 00 00 00 8a f9 fb 67 38 ca 4c 91 bb bd d5 3c >.....ùûg8ÊL.»½Õ<<
001010 5a c1 f3 0a 00 00 15 00 80 00 ff ff ff ff 03 00 >ZÁó.......ÿÿÿÿ..<
001020 00 00 01 00 00 00 00 00 00 00 3d b5 04 00 00 00 >..........=µ....<

[Victor Sudakov]

deloptes wrote:
>
> > "wipefs -t btrfs -f -a /dev/nvme1n1" did the job.
> >
> > Still wondering where those labels are stored on disk in Linux.
> >
>
> FS Superblock?

Well, the FS (btrfs in this case) was not there already, but the magic
label was still there somewhere.

>
> > In FreeBSD, GEOM(4) usually keeps such stuff in the last sector of a
> > volume/device.
>
> I think it depends on the FS not on the OS.

As I said, the FS had already been replaced by another FS.

I would usually wipe the first several MB of a disk with dd when I
change filesystems or disk partitioning schemes, but this
one was already in production.

>
> if search engines are not working where you live, I think this is a good
> howto (just found it among the top 10hits in duckduckgo)
>
> https://www.cyberciti.biz/faq/howto-use-wipefs-to-wipe-a-signature-from-disk-on-linux/
>

Well, to search Duckduckgo for wipefs, you need to know about wipefs :-)
I found it from reading man blkid and lsblk, after that the information
from wipefs(8) turned out sufficient (and the howto above did not add
any new knowledge).

[David]

On Fri, 12 Mar 2021 at 14:48, Victor Sudakov <v...@sibptus.ru> wrote:
> David wrote:

> > and then perhaps
> > btrfs device remove ...

> "Remove device(s) from a filesystem identified by <path>"

> Hmm. /dev/nvme1n1 is not identified by any path because it's not mounted
> as a btrfs filesystem.

Yeah I expect you're supposed to 'btrfs remove' before
replacing the filesystem with something different.

[Victor Sudakov]

Anssi Saari wrote:
> Victor Sudakov <v...@sibptus.ru> writes:
>
> > "wipefs -t btrfs -f -a /dev/nvme1n1" did the job.
> >
> > Still wondering where those labels are stored on disk in Linux.
>
> Didn't wipefs tell you?

No. It just told me the offset, but I have no idea what is located at
that offset, and this can be important.

> I don't have any btrfs but for me wipefs prints
> at which offset it found which label. Like this, for a Linux swap
> partition:
>
> # wipefs /dev/sdb5
> DEVICE OFFSET TYPE UUID LABEL
> sdb5 0xff6 swap 8af9fb67-38ca-4c91-bbbd-d53c5ac1f30a
>
> And if I hexdump a bit of /dev/sdb5 I get:
>
> 000ff0 00 00 00 00 00 00 53 57 41 50 53 50 41 43 45 32 >......SWAPSPACE2<
> 001000 01 00 00 00 8a f9 fb 67 38 ca 4c 91 bb bd d5 3c >.....ùûg8ÊL.»½Õ<<
> 001010 5a c1 f3 0a 00 00 15 00 80 00 ff ff ff ff 03 00 >ZÁó.......ÿÿÿÿ..<
> 001020 00 00 01 00 00 00 00 00 00 00 3d b5 04 00 00 00 >..........=µ....<
>

So what is that at 0xff6? GPT, MBR, some superblock, some reserved sector?

[David]

On Fri, 12 Mar 2021 at 20:17, Victor Sudakov <v...@sibptus.ru> wrote:
> deloptes wrote:

> > > "wipefs -t btrfs -f -a /dev/nvme1n1" did the job.

> > > Still wondering where those labels are stored on disk in Linux.

> > FS Superblock?

> Well, the FS (btrfs in this case) was not there already, but the magic
> label was still there somewhere.

My new knowledge after 5 seconds of searching, ...

https://en.wikipedia.org/wiki/Btrfs#Superblock
says that there are multiple copies of the superblock.

And the on-disk format is here:
https://btrfs.wiki.kernel.org/index.php/On-disk_Format

And maybe if the drive was once part of a multiple
device filesystem it might be referenced in superblocks
on other devices.

[Victor Sudakov]

David wrote:
> On Fri, 12 Mar 2021 at 20:17, Victor Sudakov <v...@sibptus.ru> wrote:
> > deloptes wrote:
>
> > > > "wipefs -t btrfs -f -a /dev/nvme1n1" did the job.
>
> > > > Still wondering where those labels are stored on disk in Linux.
>
> > > FS Superblock?
>
> > Well, the FS (btrfs in this case) was not there already, but the magic
> > label was still there somewhere.
>
> My new knowledge after 5 seconds of searching, ...
>
> https://en.wikipedia.org/wiki/Btrfs#Superblock
> says that there are multiple copies of the superblock.

Many filesystems have multiple copies of the superblock (e.g. UFS and ext*).

What surprised and worried me is the fact that they persisted after the
disk was converted to ZFS, and btrfs continued to recognize this
filesystem as its own (albeit unmounted).

>
> And the on-disk format is here:
> https://btrfs.wiki.kernel.org/index.php/On-disk_Format

>
> And maybe if the drive was once part of a multiple
> device filesystem it might be referenced in superblocks
> on other devices.
>

No, it was not.

[Victor Sudakov]

David wrote:
>
> > > and then perhaps
> > > btrfs device remove ...
>
> > "Remove device(s) from a filesystem identified by <path>"
>
> > Hmm. /dev/nvme1n1 is not identified by any path because it's not mounted
> > as a btrfs filesystem.
>
> Yeah I expect you're supposed to 'btrfs remove' before
> replacing the filesystem with something different.
>

I dunno. I guess "btrfs device remove" is used to detach a device from
under an operational filesystem, much like "zpool remove".

[deloptes]

Victor Sudakov wrote:

> Well, to search Duckduckgo for wipefs, you need to know about wipefs :-)
> I found it from reading man blkid and lsblk, after that the information
> from wipefs(8) turned out sufficient (and the howto above did not add
> any new knowledge).

I searched for "linux file system signature"

[Andrei POPESCU]

On Vi, 12 mar 21, 09:21:59, Victor Sudakov wrote:
>
> The problem is that /dev/nvme1n1 is being used for ZFS now, and there is
> currently no btrfs thereon. However, there is a btrfs label or something
> stuck somewhere, how can I clear it?

[...]

> It's somewhere on disk, but where?
>
> # blkid | grep nvme1n1
> /dev/nvme1n1: UUID="38f74bc8-465d-4866-8ec1-3a144741012c" UUID_SUB="ada72e33-4467-4413-b78a-1a2392f62e62" TYPE="btrfs" PTUUID="d73a33f2-2b34-e64b-bc66-128320256a28" PTTYPE="gpt"

Look again ;)


Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

[Victor Sudakov]

Andrei POPESCU wrote:
> On Vi, 12 mar 21, 09:21:59, Victor Sudakov wrote:
> >
> > The problem is that /dev/nvme1n1 is being used for ZFS now, and there is
> > currently no btrfs thereon. However, there is a btrfs label or something
> > stuck somewhere, how can I clear it?
>
> [...]
>
> > It's somewhere on disk, but where?
> >
> > # blkid | grep nvme1n1
> > /dev/nvme1n1: UUID="38f74bc8-465d-4866-8ec1-3a144741012c" UUID_SUB="ada72e33-4467-4413-b78a-1a2392f62e62" TYPE="btrfs" PTUUID="d73a33f2-2b34-e64b-bc66-128320256a28" PTTYPE="gpt"
>
> Look again ;)

Beats me!

In what disk structure can this signature of type "btrfs" reside?
The name "nvme1n1" is for the whole disk even, not for a partition thereof.

For example, and for contrast,

/dev/nvme2n1p1: UUID="97fdc843-346f-4f34-a903-c99b22a96050" TYPE="ext4" PARTUUID="1afa8103-4463-1048-9159-7233ce337f3f"

makes perfect sense, it's a GPT partition of the "Linux filesystem" type.

[Andrei POPESCU]

On Du, 14 mar 21, 10:58:02, Victor Sudakov wrote:
> Andrei POPESCU wrote:
> > On Vi, 12 mar 21, 09:21:59, Victor Sudakov wrote:
> > >
> > > The problem is that /dev/nvme1n1 is being used for ZFS now, and there is
> > > currently no btrfs thereon. However, there is a btrfs label or something
> > > stuck somewhere, how can I clear it?
> >
> > [...]
> >
> > > It's somewhere on disk, but where?
> > >
> > > # blkid | grep nvme1n1
> > > /dev/nvme1n1: UUID="38f74bc8-465d-4866-8ec1-3a144741012c" UUID_SUB="ada72e33-4467-4413-b78a-1a2392f62e62" TYPE="btrfs" PTUUID="d73a33f2-2b34-e64b-bc66-128320256a28" PTTYPE="gpt"
> >
> > Look again ;)
>
> Beats me!
>
> In what disk structure can this signature of type "btrfs" reside?
> The name "nvme1n1" is for the whole disk even, not for a partition thereof.

I'm guessing it's in the GPT somewhere. Did you try removing the entire
partition table before switching to ZFS?

By the way, I see the man page for blkid recommends to use lsblk instead
and points to wipefs to erase obsolete magic strings from the device
(which you did eventually).

[Victor Sudakov]

Andrei POPESCU wrote:
> > > >
> > > > The problem is that /dev/nvme1n1 is being used for ZFS now, and there is
> > > > currently no btrfs thereon. However, there is a btrfs label or something
> > > > stuck somewhere, how can I clear it?
> > >
> > > [...]
> > >
> > > > It's somewhere on disk, but where?
> > > >
> > > > # blkid | grep nvme1n1
> > > > /dev/nvme1n1: UUID="38f74bc8-465d-4866-8ec1-3a144741012c" UUID_SUB="ada72e33-4467-4413-b78a-1a2392f62e62" TYPE="btrfs" PTUUID="d73a33f2-2b34-e64b-bc66-128320256a28" PTTYPE="gpt"
> > >
> > > Look again ;)
> >
> > Beats me!
> >
> > In what disk structure can this signature of type "btrfs" reside?
> > The name "nvme1n1" is for the whole disk even, not for a partition thereof.
>
> I'm guessing it's in the GPT somewhere. Did you try removing the entire
> partition table before switching to ZFS?

There had been no partition table, I just ran "mkfs.btrfs /dev/nvme1n1"
on the whole raw volume, and then mounted /dev/nvme1n1.

Later, when switching to ZFS, I ran "zpool create fastdrive /dev/nvme1n1"
again on the whole volume. But ZFS outsmarted me and created a GPT
though I had not asked it to.

The FreeBSD variety of ZFS does not do that, but Solaris AFAIR does
like Linux.

[Andrei POPESCU]

On Du, 14 mar 21, 17:34:40, Victor Sudakov wrote:
> Andrei POPESCU wrote:
> >

> > I'm guessing it's in the GPT somewhere. Did you try removing the entire
> > partition table before switching to ZFS?
>
> There had been no partition table, I just ran "mkfs.btrfs /dev/nvme1n1"
> on the whole raw volume, and then mounted /dev/nvme1n1.
>
> Later, when switching to ZFS, I ran "zpool create fastdrive /dev/nvme1n1"
> again on the whole volume. But ZFS outsmarted me and created a GPT
> though I had not asked it to.

Your blkid output suggests the GPT was created by btrfs ;)

> The FreeBSD variety of ZFS does not do that, but Solaris AFAIR does
> like Linux.

In my understanding it does so to reserve 8MiB of space at the end of
the drive, in case a later replacement for a RAID is slightly smaller.

It also allows to use GPT partition labels on create to get nicer names
for your physical devices.

[Victor Sudakov]

Andrei POPESCU wrote:
> On Du, 14 mar 21, 17:34:40, Victor Sudakov wrote:
> > Andrei POPESCU wrote:
> > >
> > > I'm guessing it's in the GPT somewhere. Did you try removing the entire
> > > partition table before switching to ZFS?
> >
> > There had been no partition table, I just ran "mkfs.btrfs /dev/nvme1n1"
> > on the whole raw volume, and then mounted /dev/nvme1n1.
> >
> > Later, when switching to ZFS, I ran "zpool create fastdrive /dev/nvme1n1"
> > again on the whole volume. But ZFS outsmarted me and created a GPT
> > though I had not asked it to.
>
> Your blkid output suggests the GPT was created by btrfs ;)

Maybe btrfs does create it under some circumstances, but I cannot
reproduce it now:


root@test2-vas:~# blkid /dev/xvdg
root@test2-vas:~# mkfs.btrfs /dev/xvdg
btrfs-progs v4.20.1
See http://btrfs.wiki.kernel.org for more information.

Detected a SSD, turning off metadata duplication. Mkfs with -m dup if you want to force metadata duplication.
Label: (null)
UUID: 7d64c5ab-ae59-4e8d-aa6a-a86d7702f029
Node size: 16384
Sector size: 4096
Filesystem size: 100.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: single 8.00MiB
System: single 4.00MiB
SSD detected: yes
Incompat features: extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 100.00GiB /dev/xvdg

root@test2-vas:~#
root@test2-vas:~# blkid /dev/xvdg
/dev/xvdg: UUID="7d64c5ab-ae59-4e8d-aa6a-a86d7702f029" UUID_SUB="f0868f63-b40d-413a-ba76-6837b67b8ecf" TYPE="btrfs"
root@test2-vas:~#
root@test2-vas:~# fdisk !$
fdisk /dev/xvdg

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

The old btrfs signature will be removed by a write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x43734e7e.

Command (m for help):

[Victor Sudakov]

Victor Sudakov wrote:
>
> btrfs thinks that /dev/nvme1n1 has a btrfs:
>
> # btrfs filesystem show
> Label: none uuid: 3414ae53-f3d4-43ea-bb88-ffefc9bc86f6
> Total devices 1 FS bytes used 1.05TiB
> devid 1 size 2.00TiB used 1.33TiB path /dev/nvme0n1
>
> Label: none uuid: 38f74bc8-465d-4866-8ec1-3a144741012c
> Total devices 1 FS bytes used 831.16GiB
> devid 1 size 3.00TiB used 1.48TiB path /dev/nvme1n1
>
> The problem is that /dev/nvme1n1 is being used for ZFS now, and there is
> currently no btrfs thereon. However, there is a btrfs label or something
> stuck somewhere, how can I clear it?
>
> I tried to unload/load the btrfs kernel module but it did not help.
> It's somewhere on disk, but where?

Found some hints and recipes in
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_to_clean_up_old_superblock_.3F
So the culprit probably *was* the btrfs superblock. Deloptes you were
right.

It's interesting however that "zpool create" created a GPT (which I did
not want), but did not erase a superblock from a previous filesystem.

A lesson? Always dd the first several MB of a disk with zeroes when
changing partitioning schemes, whole disk filesystems etc.