System not booting after upgrade 14.3->14.4 with "ZFS: unsupported feature: com.klarasystems: vdev_zaps_v2"

[Johan Helsingius]

Hi all,

Was doing an upgrade from 14.3-R7 to 14.4, and after the freebsd-update
install, the reboot fails with "ZFS: unsupported feature:
com.klarasystems: vdev_zaps_v2"

Any idea what that feature is, and how to recover?

Julf

[Frank Leonhardt]

On 30/04/2026 10:28, Johan Helsingius wrote:

Was doing an upgrade from 14.3-R7 to 14.4, and after the freebsd-update
install, the reboot fails with "ZFS: unsupported feature:
com.klarasystems: vdev_zaps_v2"

It's an extension form Klara Inc :-)

As no ZFS expert has turned up, I'll have a go.

As you can probably guess, it's not supported in 14.4/OpenZFS. A ZAP is a key/value store and I assume Klara has an enhanced format one, and how you got it on what I assume is your root vdev is anyone's guess.

You might be able to fix it by reinstalling the bootloader. Keep a backup of the bootloader before you reinstall it.

Beyond that you're going beyond anything I've done myself. I assume you've tried everything possible to roll back?
Boot from something else and try importing the zpool with "-o readonly=on"

Try  "zpool get fea...@com.klarasystems:vdev_zaps_v2 zroot" or whatever the pool is called.

Then, if you feel lucky, you may be able to set it to disabled (zpool set fea...@com.klarasystems:vdev_zaps_v2=disabled zroot), export and reboot. This is a general fix for problem features and I have never been in the same situation as you have here with this one.

I suggest you post the results at every stage and it might ring a bell with someone.

Regards, Frank.

P.S. I'm pretty sure this is not on any FreeBSD base system ever released - I wouldn’t know about FreeNAS? All I can think og is that someone ran zpool upgrade when the system was running on a Klara-patched version of ZFS and this set the flag, in which case I'd be fairly confident that turning the feature off as described above would do the trick - unless the enhanced ZAP has been used on the disk, in which case find what did it, boot from that, import your zpool R/O and dump all the important stuff to tape!

[Johan Helsingius]

On 02/05/2026 6:23 pm, Frank Leonhardt wrote:

> As you can probably guess, it's not supported in 14.4/OpenZFS. A ZAP is
> a key/value store and I assume Klara has an enhanced format one, and how
> you got it on what I assume is your root vdev is anyone's guess.

Thanks - I have no idea how it could have gotten there.

> Beyond that you're going beyond anything I've done myself. I assume
> you've tried everything possible to roll back?
> Boot from something else and try importing the zpool with "-o readonly=on"

Booting from installer and doing a freebsd-update after
mounting the zpool seems to have solved the problem.

> Try "zpool get fea...@com.klarasystems:vdev_zaps_v2 zroot" or whatever
> the pool is called.

After the update the feature doesn't seem to be there any more.

> P.S. I'm pretty sure this is not on any FreeBSD base system ever
> released - I wouldn’t know about FreeNAS? All I can think og is that
> someone ran zpool upgrade when the system was running on a Klara-patched
> version of ZFS and this set the flag, in which case I'd be fairly
> confident that turning the feature off as described above would do the
> trick - unless the enhanced ZAP has been used on the disk, in which case
> find what did it, boot from that, import your zpool R/O and dump all the
> important stuff to tape!

i am the only user, and I haven't installed any custom patches,
just kept doing freebsd-updates since initial install many years
ago.

Julf

[Frank Leonhardt]

On 02/05/2026 17:39, Johan Helsingius wrote:
> <snip>

> i am the only user, and I haven't installed any custom patches,
> just kept doing freebsd-updates since initial install many years
> ago.
>

That's really weird, but I'm glad freebsd-update fixed it.

I checked and apparently TrueNAS doesn't use any weird Klana ZFS so my
guess there was wrong. In fact, having RTFM it's part of OpenZFS.

https://openzfs.github.io/openzfs-docs/man/master/7/zpool-features.7.html#com.klarasystems:vdev_zaps_v2

I did some more digging and it was added to OpenZFS 2.2

More info here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275199

Looks like my guess about the bootloader may have been correct, partly
by luck!  (i.e. it was for a pre-2.2 pool).

Regards, Frank.

[Mark G.]

Here are my notes for when I had to solve the klara issue:

Disclaimer: MAKE SURE TO USE YOUR DRIVE NAMES AND PARTITION NUMBERS

6.1 Updating Boot Loaders (ZFS)

I had a failure to boot after an update from FreeBSD 14.2-p3 to 14.2-p4, which had an
upgraded zfs.ko kernel module. Here is the error and boot failure.

=====
Consoles: EFI console

ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2

FreeBSD/amd64 EFI loader, Revision 1.1
...
Failed to find bootable partition
Press any key to interrupt reboot in 3 seconds
can’t load ’kernel’
Type ’?’ for a list of commands, ’help’ for more detailed help.
OK
=====

The fixes required boot into single user mode via an USB bootable drive.
The drive contained FreeBSD 14.2 as an installer with single user
mode as well.

Once the USB is booted into single user mode, set the root partition to be mounted
read-write:

# mount -u /

The change from FreeBSD version 13.x to 14.x required updating the boot loaders (in
various ways) based on zfs-on-root installations. I had failed to do this at the time and
got bit for it.

The FreeBSD 14 release notes 1 document this as quoted below.

[1] https://www.freebsd.org/releases/14.0R/relnotes/

=======
"There have been a number of improvements in the boot loaders, and up-
grading the boot loader on the boot partition is recommended in most cases,
in particular if the system boots via EFI. If the root is on a ZFS file system,
updating the boot loader is mandatory if the pool is to be upgraded, and
the boot loader update must be done first. Note that ZFS pool upgrades are
not recommended for root file systems in most cases, but updating the boot
loader can avoid making the system unbootable if the pool is upgraded in
the future. The bootstrap update procedure depends on the boot method
(EFI or BIOS), and also on the disk partitioning scheme. The next several
sections address each in turn.

Notes for systems that boot via EFI, using either binary or source up-
grades: There are one or more copies of the boot loader on the MS-DOS
EFI System Partition (ESP), used by the firmware to boot the kernel. The
location of the boot loader in use can be determined using the command
efibootmgr -v. The value displayed for BootCurrent should be the number of
the current boot configuration used to boot the system. The corresponding
entry of the output should begin with a + sign, such as

+Boot0000* FreeBSD HD(1,GPT,f859c46d-19ee-4e40-8975-3ad1ab00ac09,
0x800,0x82000)/File(\EFI\freebsd\loader.efi)
nda0p1:/EFI/freebsd/loader.efi (null)

The ESP may already be mounted on /boot/efi. Otherwise, the parti-
tion may be mounted manually, using the partition listed in the efibootmgr
output (nda0p1 in this case): mount msdosfs /dev/nda0p1 /boot/efi. See
loader.efi(8) for another example.

The value in the File field in the efibootmgr -v output, \EFI \freebsd\loader.efi
in this case, is the MS-DOS name for the boot loader in use on the ESP. If the
mount point is /boot/efi, this file will translate to /boot/efi/efi/freebsd/loader.efi.
(Case does not matter on MS-DOSFS file sytems; FreeBSD uses lower case.)
Another common value for File would be \EFI\boot\bootXXX.efi, where
XXX is x64 for amd64, aa64 for aarch64, or riscv64 for riscv64; this is the
default bootstrap if none is configured. Both the configured and default boot
loaders should be updated by copying from /boot/loader.efi to the correct
path in /boot/efi.

For systems that boot via BIOS and use the GPT partition scheme, boot-
strap upgrades are optional unless a ZFS root pool will be upgraded (which
is discouraged). Upgrades are only possible if the freebsd-boot partition is
at least 180K. (512K is now standard.) The device name and the partition
index can be identified using the command gpart show, such as:

=> 34 246162605 ada0 GPT (224G)
34 1024 1 freebsd-boot (512K)

When using ZFS, the bootcode can be updated by running the command
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0. If up-
dating with a UFS root, the bootcode can be updated by running the com-
mand gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0, al-
though this should not be necessary. Note that the freebsd-boot partition
size is running up against a hard limit, and this scheme may not be support-
able in the future. Users of such systems should consider an upgrade to EFI,
possibly by carving space from an existing swap partition.

If the system boots via BIOS and uses the MBR partition scheme, or has
a GPT freebsd-boot partition smaller than 180K, then it is not possible to
update the bootcode, and therefore to upgrade the root ZFS storage pool."
=======

There are two instances that I had to deal with.

6.1.1 UEFI Boot Loader

Most of my newer mini PCs used a UEFI layout for booting and thus have disks
partitioned similar to this:

user@think:~ % gpart show
=> 40 3907029088 nda0 GPT (1.8T)
40 532480 1 efi (260M)
532520 1024 2 freebsd-boot (512K)
533544 984 - free - (492K)
534528 33554432 3 freebsd-swap (16G)
34088960 3872940032 4 freebsd-zfs (1.8T)
3907028992 136 - free - (68K)


The partition of interest is the first one (1) with a type of efi.
This is actually an msdos based partition and can be mounted as such.

user@think:~ % cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/gpt/efiboot0 /boot/efi msdosfs rw 2 2

The /dev/gpt/efiboot0 device is equivalent to /dev/nda0p1, in this case.

root@dmz1:~ # mount /boot/efi
root@dmz1:~ # mount
...
/dev/gpt/efiboot0 on /boot/efi (msdosfs, local)

To update the boot loader to a FreeBSD 14.x version, the above file system should be
mounted and its contents will look like:

user@think:~ % ls -laR /boot/efi
user@think:~ % ls -laR /boot/efi
total 25
drwxr-xr-x 1 root wheel 16384 Dec 31 1979 .
drwxr-xr-x 15 root wheel 71 Jul 6 20:22 ..
drwxr-xr-x 1 root wheel 16384 Feb 12 18:49 efi

/boot/efi/efi:
total 48
drwxr-xr-x 1 root wheel 16384 Feb 12 18:49 .
drwxr-xr-x 1 root wheel 16384 Dec 31 1979 ..
drwxr-xr-x 1 root wheel 16384 Feb 12 18:49 boot
drwxr-xr-x 1 root wheel 16384 Feb 12 18:49 freebsd

/boot/efi/efi/boot:
total 688
drwxr-xr-x 1 root wheel 16384 Feb 12 18:49 .
drwxr-xr-x 1 root wheel 16384 Feb 12 18:49 ..
-rwxr-xr-x 1 root wheel 660992 Feb 12 18:49 bootx64.efi

/boot/efi/efi/freebsd:
total 688
drwxr-xr-x 1 root wheel 16384 Feb 12 18:49 .
drwxr-xr-x 1 root wheel 16384 Feb 12 18:49 ..
-rwxr-xr-x 1 root wheel 660992 Feb 12 18:49 loader.efi

We have to update both paths:

/boot/efi/efi/boot/bootx64.efi
/boot/efi/efi/freebsd/loader.efi

with the file found in /boot/loader.efi. To whit (make some copies of the old ones
first):

root@metal:~ # cp /boot/efi/efi/freebsd/loader.efi loader.efi.from13.x
root@metal:~ # cp /boot/efi/efi/boot/bootx64.efi bootx64.efi.from13.x
root@metal:~ # cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
root@metal:~ # cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi


There is a command called efibootmgr which can supply some information:

root@dmz1:~ # efibootmgr -v
Boot to FW : false
BootCurrent: 0000
Timeout : 1 seconds
BootOrder : 0000, 0001
+Boot0000* FreeBSD HD(1,GPT,9ccacd8f-5c7a-11f0-bfc4-5847ca753c56,
0x28,0x82000)/File(\EFI\FREEBSD\LOADER.EFI)
ada0p1:/EFI/FREEBSD/LOADER.EFI (null)

Boot0001* UEFI OS HD(1,GPT,9ccacd8f-5c7a-11f0-bfc4-5847ca753c56,
0x28,0x82000)/File(\EFI\BOOT\BOOTX64.EFI)
ada0p1:/EFI/BOOT/BOOTX64.EFI (null)

Boot0001* UEFI OS HD(1,GPT,9ccacd8f-5c7a-11f0-bfc4-5847ca753c56,
0x28,0x82000)/File(\EFI\BOOT\BOOTX64.EFI)
ada0p1:/EFI/BOOT/BOOTX64.EFI (null)

The plus (+) sign indicates the file currently in use.


6.1.2 BIOS GPT Boot Loader

The second variant I have is an old Lenovo non-UEFI BIOS based system which has a
partition scheme similar to this:

root@gate:~ % gpart show
=> 40 468862048 ada0 GPT (224G)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 4194304 2 freebsd-swap (2.0G)
4196352 464664576 3 freebsd-zfs (222G)
468860928 1160 - free - (580K)


Note the lack of an efi partition.

The update here is:
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

If drives are mirrored and have the same gpart setup, use the same command for each
drive: ada1, ada2, and so on.

[Johan Helsingius]

Hi Mark,

Great instructions! Thanks!

Julf