ZFS Permanent error <0xffffffffffffffff>:<0x0>

[Andrea Venturoli]

Hello.

Yesterday, out of the blue, on a 13.1/amd64 machine, something went
wrong in one of my zpools, as I discovered from daily run mails.

> # zpool status -vx
> pool: zroo2
> state: ONLINE
> status: One or more devices has experienced an error resulting in data
> corruption. Applications may be affected.
> action: Restore the file in question if possible. Otherwise restore the
> entire pool from backup.
> see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
> scan: scrub repaired 0B in 00:50:25 with 0 errors on Tue Nov 15 04:32:52 2022
> config:
>
> NAME STATE READ WRITE CKSUM
> zroo2 ONLINE 0 0 0
> raidz1-0 ONLINE 0 0 0
> ada0p3 ONLINE 0 0 4
> ada2p3 ONLINE 0 0 4
> ada1p3 ONLINE 0 0 4
>
> errors: Permanent errors have been detected in the following files:
>
> <0xffffffffffffffff>:<0x0>

I don't think there's a "file in question" I can restore, as
"<0xffffffffffffffff>:<0x0>" doesn't seem to identify any.

I searched and found some bugs/discussion, but none lead to a definitive
answer.

From logs and SMART, I don't think I had any hadware problem.

How much should I be worried?
Any hint on how to solve this (apart from obviously recreate the pool
from scracth, after a backup)?

bye & Thanks
av.

P.S.
Further info I *think* might be useful (but feel free to ask for other
data):
> # zpool upgrade
> This system supports ZFS pool feature flags.
>
> All pools are formatted using feature flags.
>
>
> Some supported features are not enabled on the following pools. Once a
> feature is enabled the pool may become incompatible with software
> that does not support the feature. See zpool-features(7) for details.
>
> Note that the pool 'compatibility' feature can be used to inhibit
> feature upgrades.
>
> POOL FEATURE
> ---------------
> zroo2
> userobj_accounting
> encryption
> project_quota
> resilver_defer
> bookmark_v2
> redaction_bookmarks
> redacted_datasets
> bookmark_written
> log_spacemap
> livelist
> device_rebuild
> zstd_compress
> draid

> # zdb -m zroo2
>
> Metaslabs:
> vdev 0
> metaslabs 191 offset spacemap free
> --------------- ------------------- --------------- ------------
> metaslab 0 offset 0 spacemap 8929 free 3.55G
> space map object 8929:
> smp_length = 0x1378c0
> smp_alloc = 0x11caa0000
> metaslab 1 offset 200000000 spacemap 598 free 2.06G
> space map object 598:
> smp_length = 0x12f680
> smp_alloc = 0x17c6c8c00
> metaslab 2 offset 400000000 spacemap 329 free 2.30G
> space map object 329:
> smp_length = 0x134488
> smp_alloc = 0x16ccba000
> metaslab 3 offset 600000000 spacemap 79537 free 3.81G
> space map object 79537:
> smp_length = 0x94460
> smp_alloc = 0x10c5ad400
> metaslab 4 offset 800000000 spacemap 37776 free 2.91G
> space map object 37776:
> smp_length = 0x99910
> smp_alloc = 0x145751000
> metaslab 5 offset a00000000 spacemap 6427 free 2.44G
> space map object 6427:
> smp_length = 0xcc258
> smp_alloc = 0x163a62c00
> metaslab 6 offset c00000000 spacemap 4572 free 1.89G
> space map object 4572:
> smp_length = 0x89318
> smp_alloc = 0x186e8d000
> metaslab 7 offset e00000000 spacemap 4094 free 2.08G
> space map object 4094:
> smp_length = 0x8ccc8
> smp_alloc = 0x17b267400
> metaslab 8 offset 1000000000 spacemap 497 free 3.18G
> space map object 497:
> smp_length = 0x1264d8
> smp_alloc = 0x13475a800
> metaslab 9 offset 1200000000 spacemap 78219 free 2.99G
> space map object 78219:
> smp_length = 0x1902e0
> smp_alloc = 0x140af7c00
> metaslab 10 offset 1400000000 spacemap 79943 free 3.59G
> space map object 79943:
> smp_length = 0x67bd8
> smp_alloc = 0x11a654000
> metaslab 11 offset 1600000000 spacemap 4528 free 2.92G
> space map object 4528:
> smp_length = 0x1498d0
> smp_alloc = 0x1455c3c00
> metaslab 12 offset 1800000000 spacemap 38608 free 1.84G
> space map object 38608:
> smp_length = 0xfcf30
> smp_alloc = 0x18a31fc00
> metaslab 13 offset 1a00000000 spacemap 37203 free 3.01G
> space map object 37203:
> smp_length = 0x1a71e8
> smp_alloc = 0x13f0c7c00
> metaslab 14 offset 1c00000000 spacemap 79406 free 1.81G
> space map object 79406:
> smp_length = 0x3e3a0
> smp_alloc = 0x18c433000
> metaslab 15 offset 1e00000000 spacemap 79790 free 2.67G
> space map object 79790:
> smp_length = 0xd1878
> smp_alloc = 0x154f4f400
> metaslab 16 offset 2000000000 spacemap 605 free 3.49G
> space map object 605:
> smp_length = 0xc1810
> smp_alloc = 0x1207d1400
> metaslab 17 offset 2200000000 spacemap 28207 free 3.21G
> space map object 28207:
> smp_length = 0x1257e8
> smp_alloc = 0x132c76800
> metaslab 18 offset 2400000000 spacemap 81100 free 2.21G
> space map object 81100:
> smp_length = 0x63230
> smp_alloc = 0x172ca7000
> metaslab 19 offset 2600000000 spacemap 21487 free 2.10G
> space map object 21487:
> smp_length = 0xfe3a8
> smp_alloc = 0x1798ae000
> metaslab 20 offset 2800000000 spacemap 38318 free 2.18G
> space map object 38318:
> smp_length = 0x92fd0
> smp_alloc = 0x17442b400
> metaslab 21 offset 2a00000000 spacemap 32057 free 1.10G
> space map object 32057:
> smp_length = 0xa4408
> smp_alloc = 0x1b986f800
> metaslab 22 offset 2c00000000 spacemap 38319 free 2.11G
> space map object 38319:
> smp_length = 0xca088
> smp_alloc = 0x178c44000
> metaslab 23 offset 2e00000000 spacemap 74819 free 1.88G
> space map object 74819:
> smp_length = 0x87c68
> smp_alloc = 0x1878d4800
> metaslab 24 offset 3000000000 spacemap 102136 free 925M
> space map object 102136:
> smp_length = 0x1d3e0
> smp_alloc = 0x1c636cc00
> metaslab 25 offset 3200000000 spacemap 78992 free 2.17G
> space map object 78992:
> smp_length = 0xcb850
> smp_alloc = 0x175076000
> metaslab 26 offset 3400000000 spacemap 32207 free 439M
> space map object 32207:
> smp_length = 0x53c38
> smp_alloc = 0x1e4968000
> metaslab 27 offset 3600000000 spacemap 6338 free 1.76G
> space map object 6338:
> smp_length = 0xdca38
> smp_alloc = 0x18f560c00
> metaslab 28 offset 3800000000 spacemap 79746 free 956M
> space map object 79746:
> smp_length = 0x95cf0
> smp_alloc = 0x1c43bf000
> metaslab 29 offset 3a00000000 spacemap 23941 free 1.11G
> space map object 23941:
> smp_length = 0x5b088
> smp_alloc = 0x1b8eff000
> metaslab 30 offset 3c00000000 spacemap 34028 free 1.18G
> space map object 34028:
> smp_length = 0x4edc8
> smp_alloc = 0x1b4a7b800
> metaslab 31 offset 3e00000000 spacemap 1424 free 2.68G
> space map object 1424:
> smp_length = 0xe1d38
> smp_alloc = 0x154837400
> metaslab 32 offset 4000000000 spacemap 10659 free 1.05G
> space map object 10659:
> smp_length = 0x97640
> smp_alloc = 0x1bceaf400
> metaslab 33 offset 4200000000 spacemap 4592 free 2.70G
> space map object 4592:
> smp_length = 0x11b510
> smp_alloc = 0x153091400
> metaslab 34 offset 4400000000 spacemap 4591 free 2.28G
> space map object 4591:
> smp_length = 0x7c5f8
> smp_alloc = 0x16e3fb800
> metaslab 35 offset 4600000000 spacemap 1026 free 2.12G
> space map object 1026:
> smp_length = 0x613e8
> smp_alloc = 0x178980400
> metaslab 36 offset 4800000000 spacemap 326 free 2.32G
> space map object 326:
> smp_length = 0x115458
> smp_alloc = 0x16b7e8c00
> metaslab 37 offset 4a00000000 spacemap 81775 free 2.33G
> space map object 81775:
> smp_length = 0x72578
> smp_alloc = 0x16a95e400
> metaslab 38 offset 4c00000000 spacemap 21076 free 2.80G
> space map object 21076:
> smp_length = 0x13c528
> smp_alloc = 0x14c931c00
> metaslab 39 offset 4e00000000 spacemap 78373 free 1.06G
> space map object 78373:
> smp_length = 0xbadd0
> smp_alloc = 0x1bc05bc00
> metaslab 40 offset 5000000000 spacemap 43469 free 3.28G
> space map object 43469:
> smp_length = 0x13d168
> smp_alloc = 0x12de0f400
> metaslab 41 offset 5200000000 spacemap 23025 free 1.70G
> space map object 23025:
> smp_length = 0xc0c68
> smp_alloc = 0x19305ec00
> metaslab 42 offset 5400000000 spacemap 28049 free 1.60G
> space map object 28049:
> smp_length = 0x89dd0
> smp_alloc = 0x199837800
> metaslab 43 offset 5600000000 spacemap 75613 free 1.19G
> space map object 75613:
> smp_length = 0x8de20
> smp_alloc = 0x1b41a1400
> metaslab 44 offset 5800000000 spacemap 6146 free 4.94G
> space map object 6146:
> smp_length = 0x129ab8
> smp_alloc = 0xc3b72c00
> metaslab 45 offset 5a00000000 spacemap 7403 free 1.83G
> space map object 7403:
> smp_length = 0x6fac0
> smp_alloc = 0x18ade9800
> metaslab 46 offset 5c00000000 spacemap 43577 free 1.43G
> space map object 43577:
> smp_length = 0x63fd0
> smp_alloc = 0x1a45d2000
> metaslab 47 offset 5e00000000 spacemap 80013 free 1.75G
> space map object 80013:
> smp_length = 0x5b0f0
> smp_alloc = 0x1902b0800
> metaslab 48 offset 6000000000 spacemap 7666 free 3.71G
> space map object 7666:
> smp_length = 0x10bac0
> smp_alloc = 0x112afb000
> metaslab 49 offset 6200000000 spacemap 75442 free 2.64G
> space map object 75442:
> smp_length = 0x490e8
> smp_alloc = 0x156e57c00
> metaslab 50 offset 6400000000 spacemap 8362 free 2.80G
> space map object 8362:
> smp_length = 0xccf78
> smp_alloc = 0x14c92cc00
> metaslab 51 offset 6600000000 spacemap 83247 free 2.05G
> space map object 83247:
> smp_length = 0xa6fc0
> smp_alloc = 0x17cf49400
> metaslab 52 offset 6800000000 spacemap 8190 free 4.20G
> space map object 8190:
> smp_length = 0xdf118
> smp_alloc = 0xf36a6800
> metaslab 53 offset 6a00000000 spacemap 58740 free 891M
> space map object 58740:
> smp_length = 0x60710
> smp_alloc = 0x1c8536000
> metaslab 54 offset 6c00000000 spacemap 8708 free 1.71G
> space map object 8708:
> smp_length = 0xc6fe0
> smp_alloc = 0x19295bc00
> metaslab 55 offset 6e00000000 spacemap 78483 free 2.99G
> space map object 78483:
> smp_length = 0x817f8
> smp_alloc = 0x1405e6400
> metaslab 56 offset 7000000000 spacemap 37151 free 3.91G
> space map object 37151:
> smp_length = 0x17fbb0
> smp_alloc = 0x105ef3400
> metaslab 57 offset 7200000000 spacemap 33915 free 1.96G
> space map object 33915:
> smp_length = 0x870a8
> smp_alloc = 0x182604000
> metaslab 58 offset 7400000000 spacemap 34091 free 2.96G
> space map object 34091:
> smp_length = 0xb0550
> smp_alloc = 0x142647c00
> metaslab 59 offset 7600000000 spacemap 104244 free 1.96G
> space map object 104244:
> smp_length = 0x88b50
> smp_alloc = 0x182cca800
> metaslab 60 offset 7800000000 spacemap 80619 free 2.75G
> space map object 80619:
> smp_length = 0x15c658
> smp_alloc = 0x150316000
> metaslab 61 offset 7a00000000 spacemap 9622 free 4.22G
> space map object 9622:
> smp_length = 0x36fd98
> smp_alloc = 0xf22d8400
> metaslab 62 offset 7c00000000 spacemap 9623 free 2.57G
> space map object 9623:
> smp_length = 0xcf4e8
> smp_alloc = 0x15b4b2400
> metaslab 63 offset 7e00000000 spacemap 23718 free 1.59G
> space map object 23718:
> smp_length = 0x89ce0
> smp_alloc = 0x19a1b2000
> metaslab 64 offset 8000000000 spacemap 77868 free 4.48G
> space map object 77868:
> smp_length = 0xa9ec8
> smp_alloc = 0xe1381c00
> metaslab 65 offset 8200000000 spacemap 11293 free 1.70G
> space map object 11293:
> smp_length = 0x5d920
> smp_alloc = 0x1933b5800
> metaslab 66 offset 8400000000 spacemap 2827 free 1.14G
> space map object 2827:
> smp_length = 0x6dcb0
> smp_alloc = 0x1b6e8f400
> metaslab 67 offset 8600000000 spacemap 13006 free 2.65G
> space map object 13006:
> smp_length = 0x10f560
> smp_alloc = 0x1562be000
> metaslab 68 offset 8800000000 spacemap 9050 free 1.91G
> space map object 9050:
> smp_length = 0xc8a18
> smp_alloc = 0x185afbc00
> metaslab 69 offset 8a00000000 spacemap 33616 free 3.50G
> space map object 33616:
> smp_length = 0x13ae60
> smp_alloc = 0x11fe0b400
> metaslab 70 offset 8c00000000 spacemap 9442 free 3.99G
> space map object 9442:
> smp_length = 0xd1430
> smp_alloc = 0x10081d800
> metaslab 71 offset 8e00000000 spacemap 81777 free 2.45G
> space map object 81777:
> smp_length = 0x136b50
> smp_alloc = 0x1630ae000
> metaslab 72 offset 9000000000 spacemap 10570 free 2.84G
> space map object 10570:
> smp_length = 0xbcb50
> smp_alloc = 0x14a018400
> metaslab 73 offset 9200000000 spacemap 9040 free 2.60G
> space map object 9040:
> smp_length = 0xaecf8
> smp_alloc = 0x15969ec00
> metaslab 74 offset 9400000000 spacemap 80331 free 2.87G
> space map object 80331:
> smp_length = 0xb98f0
> smp_alloc = 0x148690000
> metaslab 75 offset 9600000000 spacemap 9652 free 1.86G
> space map object 9652:
> smp_length = 0x108e58
> smp_alloc = 0x188ac5400
> metaslab 76 offset 9800000000 spacemap 11528 free 3.76G
> space map object 11528:
> smp_length = 0x1700d0
> smp_alloc = 0x10f393800
> metaslab 77 offset 9a00000000 spacemap 11552 free 3.46G
> space map object 11552:
> smp_length = 0xc5508
> smp_alloc = 0x122bc8000
> metaslab 78 offset 9c00000000 spacemap 10658 free 2.06G
> space map object 10658:
> smp_length = 0xa1960
> smp_alloc = 0x17bd80800
> metaslab 79 offset 9e00000000 spacemap 9705 free 2.22G
> space map object 9705:
> smp_length = 0x5bda0
> smp_alloc = 0x17228b400
> metaslab 80 offset a000000000 spacemap 10162 free 4.13G
> space map object 10162:
> smp_length = 0xb9600
> smp_alloc = 0xf7abcc00
> metaslab 81 offset a200000000 spacemap 2696 free 3.70G
> space map object 2696:
> smp_length = 0xee708
> smp_alloc = 0x1136b3000
> metaslab 82 offset a400000000 spacemap 44806 free 1.69G
> space map object 44806:
> smp_length = 0x9ffa0
> smp_alloc = 0x193f2cc00
> metaslab 83 offset a600000000 spacemap 79843 free 1.38G
> space map object 79843:
> smp_length = 0x830a8
> smp_alloc = 0x1a7fa8000
> metaslab 84 offset a800000000 spacemap 78382 free 2.65G
> space map object 78382:
> smp_length = 0x81370
> smp_alloc = 0x1562db400
> metaslab 85 offset aa00000000 spacemap 79865 free 3.41G
> space map object 79865:
> smp_length = 0x179bf8
> smp_alloc = 0x125a07c00
> metaslab 86 offset ac00000000 spacemap 10419 free 2.02G
> space map object 10419:
> smp_length = 0x3ec60
> smp_alloc = 0x17eb39000
> metaslab 87 offset ae00000000 spacemap 75426 free 1.46G
> space map object 75426:
> smp_length = 0xdc160
> smp_alloc = 0x1a2c12000
> metaslab 88 offset b000000000 spacemap 36441 free 4.57G
> space map object 36441:
> smp_length = 0x11cd48
> smp_alloc = 0xdbd1bc00
> metaslab 89 offset b200000000 spacemap 74817 free 1.47G
> space map object 74817:
> smp_length = 0xd74c8
> smp_alloc = 0x1a1ca7c00
> metaslab 90 offset b400000000 spacemap 69882 free 2.54G
> space map object 69882:
> smp_length = 0x717f8
> smp_alloc = 0x15d63b400
> metaslab 91 offset b600000000 spacemap 78158 free 2.99G
> space map object 78158:
> smp_length = 0x1104b0
> smp_alloc = 0x140b5dc00
> metaslab 92 offset b800000000 spacemap 33659 free 2.21G
> space map object 33659:
> smp_length = 0x80f98
> smp_alloc = 0x17274c800
> metaslab 93 offset ba00000000 spacemap 75925 free 3.33G
> space map object 75925:
> smp_length = 0xc8c68
> smp_alloc = 0x12b2c8400
> metaslab 94 offset bc00000000 spacemap 9739 free 3.11G
> space map object 9739:
> smp_length = 0xc0440
> smp_alloc = 0x139399c00
> metaslab 95 offset be00000000 spacemap 34925 free 3.62G
> space map object 34925:
> smp_length = 0x191c70
> smp_alloc = 0x118063400
> metaslab 96 offset c000000000 spacemap 4531 free 2.44G
> space map object 4531:
> smp_length = 0x784a8
> smp_alloc = 0x1640c9800
> metaslab 97 offset c200000000 spacemap 49474 free 2.27G
> space map object 49474:
> smp_length = 0x108db0
> smp_alloc = 0x16e798000
> metaslab 98 offset c400000000 spacemap 49819 free 2.63G
> space map object 49819:
> smp_length = 0xa14c0
> smp_alloc = 0x157710800
> metaslab 99 offset c600000000 spacemap 11122 free 2.30G
> space map object 11122:
> smp_length = 0xcbb88
> smp_alloc = 0x16c9f8800
> metaslab 100 offset c800000000 spacemap 28168 free 1.54G
> space map object 28168:
> smp_length = 0xe3790
> smp_alloc = 0x19d45e400
> metaslab 101 offset ca00000000 spacemap 10970 free 4.08G
> space map object 10970:
> smp_length = 0x104f38
> smp_alloc = 0xfb0ef800
> metaslab 102 offset cc00000000 spacemap 78170 free 4.05G
> space map object 78170:
> smp_length = 0xe9570
> smp_alloc = 0xfd070c00
> metaslab 103 offset ce00000000 spacemap 28255 free 1.99G
> space map object 28255:
> smp_length = 0x9d548
> smp_alloc = 0x1807a0000
> metaslab 104 offset d000000000 spacemap 9947 free 2.87G
> space map object 9947:
> smp_length = 0x146f90
> smp_alloc = 0x1483cb000
> metaslab 105 offset d200000000 spacemap 129332 free 3.24G
> space map object 129332:
> smp_length = 0x1b3c90
> smp_alloc = 0x130601c00
> metaslab 106 offset d400000000 spacemap 34804 free 2.68G
> space map object 34804:
> smp_length = 0xf6178
> smp_alloc = 0x154b3e000
> metaslab 107 offset d600000000 spacemap 17173 free 4.51G
> space map object 17173:
> smp_length = 0xf36d0
> smp_alloc = 0xdf6c7c00
> metaslab 108 offset d800000000 spacemap 80910 free 3.72G
> space map object 80910:
> smp_length = 0xba9c8
> smp_alloc = 0x111b72c00
> metaslab 109 offset da00000000 spacemap 7347 free 2.93G
> space map object 7347:
> smp_length = 0xc91c0
> smp_alloc = 0x144b05800
> metaslab 110 offset dc00000000 spacemap 10565 free 3.25G
> space map object 10565:
> smp_length = 0xf3d50
> smp_alloc = 0x12fe55800
> metaslab 111 offset de00000000 spacemap 46372 free 3.88G
> space map object 46372:
> smp_length = 0xda910
> smp_alloc = 0x1076c2400
> metaslab 112 offset e000000000 spacemap 36176 free 1.85G
> space map object 36176:
> smp_length = 0xcde60
> smp_alloc = 0x1895de800
> metaslab 113 offset e200000000 spacemap 10660 free 1.83G
> space map object 10660:
> smp_length = 0x61bc0
> smp_alloc = 0x18ad87400
> metaslab 114 offset e400000000 spacemap 428 free 2.14G
> space map object 428:
> smp_length = 0xe01e8
> smp_alloc = 0x176e24000
> metaslab 115 offset e600000000 spacemap 35883 free 1.88G
> space map object 35883:
> smp_length = 0x124628
> smp_alloc = 0x187c58c00
> metaslab 116 offset e800000000 spacemap 75081 free 2.59G
> space map object 75081:
> smp_length = 0x13b8c8
> smp_alloc = 0x15a6fd400
> metaslab 117 offset ea00000000 spacemap 4992 free 2.93G
> space map object 4992:
> smp_length = 0xf0d40
> smp_alloc = 0x1445d0c00
> metaslab 118 offset ec00000000 spacemap 10571 free 3.49G
> space map object 10571:
> smp_length = 0xf0e10
> smp_alloc = 0x120ac1400
> metaslab 119 offset ee00000000 spacemap 78993 free 3.97G
> space map object 78993:
> smp_length = 0x1499e0
> smp_alloc = 0x101d2b000
> metaslab 120 offset f000000000 spacemap 10761 free 2.03G
> space map object 10761:
> smp_length = 0xd8248
> smp_alloc = 0x17dcaec00
> metaslab 121 offset f200000000 spacemap 603 free 2.53G
> space map object 603:
> smp_length = 0x12eaa0
> smp_alloc = 0x15e3e8000
> metaslab 122 offset f400000000 spacemap 11642 free 3.99G
> space map object 11642:
> smp_length = 0x163050
> smp_alloc = 0x10099ac00
> metaslab 123 offset f600000000 spacemap 58881 free 4.28G
> space map object 58881:
> smp_length = 0xd51d8
> smp_alloc = 0xee16dc00
> metaslab 124 offset f800000000 spacemap 22721 free 3.17G
> space map object 22721:
> smp_length = 0xea800
> smp_alloc = 0x135355c00
> metaslab 125 offset fa00000000 spacemap 49480 free 2.22G
> space map object 49480:
> smp_length = 0xd9f60
> smp_alloc = 0x1722e3800
> metaslab 126 offset fc00000000 spacemap 10929 free 1.32G
> space map object 10929:
> smp_length = 0x79ea8
> smp_alloc = 0x1ab381000
> metaslab 127 offset fe00000000 spacemap 79922 free 2.00G
> space map object 79922:
> smp_length = 0x6d1d0
> smp_alloc = 0x17ff89400
> metaslab 128 offset 10000000000 spacemap 11649 free 3.10G
> space map object 11649:
> smp_length = 0xc8940
> smp_alloc = 0x13959b000
> metaslab 129 offset 10200000000 spacemap 609 free 2.34G
> space map object 609:
> smp_length = 0xdbd50
> smp_alloc = 0x16a06d400
> metaslab 130 offset 10400000000 spacemap 50191 free 2.51G
> space map object 50191:
> smp_length = 0x1307a8
> smp_alloc = 0x15fa81c00
> metaslab 131 offset 10600000000 spacemap 49493 free 3.57G
> space map object 49493:
> smp_length = 0xe1ce0
> smp_alloc = 0x11bd40400
> metaslab 132 offset 10800000000 spacemap 26255 free 4.57G
> space map object 26255:
> smp_length = 0xc6308
> smp_alloc = 0xdb84a800
> metaslab 133 offset 10a00000000 spacemap 10826 free 2.17G
> space map object 10826:
> smp_length = 0xdccc8
> smp_alloc = 0x1752e6c00
> metaslab 134 offset 10c00000000 spacemap 11410 free 4.39G
> space map object 11410:
> smp_length = 0xa70a0
> smp_alloc = 0xe6f30c00
> metaslab 135 offset 10e00000000 spacemap 39304 free 2.23G
> space map object 39304:
> smp_length = 0xedde8
> smp_alloc = 0x1710b4c00
> metaslab 136 offset 11000000000 spacemap 75425 free 3.41G
> space map object 75425:
> smp_length = 0xe3608
> smp_alloc = 0x125f7f800
> metaslab 137 offset 11200000000 spacemap 11533 free 2.52G
> space map object 11533:
> smp_length = 0xb9f30
> smp_alloc = 0x15eaf3400
> metaslab 138 offset 11400000000 spacemap 11106 free 2.30G
> space map object 11106:
> smp_length = 0x84708
> smp_alloc = 0x16c89e000
> metaslab 139 offset 11600000000 spacemap 21566 free 5.63G
> space map object 21566:
> smp_length = 0xc9448
> smp_alloc = 0x97b47c00
> metaslab 140 offset 11800000000 spacemap 79650 free 4.64G
> space map object 79650:
> smp_length = 0xb48c8
> smp_alloc = 0xd6b95800
> metaslab 141 offset 11a00000000 spacemap 11557 free 2.22G
> space map object 11557:
> smp_length = 0xbc850
> smp_alloc = 0x171ad3400
> metaslab 142 offset 11c00000000 spacemap 84804 free 3.25G
> space map object 84804:
> smp_length = 0x10ea58
> smp_alloc = 0x12fdaa800
> metaslab 143 offset 11e00000000 spacemap 11554 free 1.53G
> space map object 11554:
> smp_length = 0xb1390
> smp_alloc = 0x19e57c400
> metaslab 144 offset 12000000000 spacemap 11553 free 3.01G
> space map object 11553:
> smp_length = 0x148a00
> smp_alloc = 0x13f4f6800
> metaslab 145 offset 12200000000 spacemap 1508 free 3.08G
> space map object 1508:
> smp_length = 0xff598
> smp_alloc = 0x13ac22800
> metaslab 146 offset 12400000000 spacemap 115678 free 2.40G
> space map object 115678:
> smp_length = 0xcd2e0
> smp_alloc = 0x1662ee800
> metaslab 147 offset 12600000000 spacemap 26346 free 2.80G
> space map object 26346:
> smp_length = 0x103808
> smp_alloc = 0x14c800c00
> metaslab 148 offset 12800000000 spacemap 11452 free 4.19G
> space map object 11452:
> smp_length = 0x1a3030
> smp_alloc = 0xf3df3000
> metaslab 149 offset 12a00000000 spacemap 21807 free 1.40G
> space map object 21807:
> smp_length = 0x1359e0
> smp_alloc = 0x1a6483000
> metaslab 150 offset 12c00000000 spacemap 79561 free 1.83G
> space map object 79561:
> smp_length = 0x8d680
> smp_alloc = 0x18a947c00
> metaslab 151 offset 12e00000000 spacemap 258 free 6.22G
> space map object 258:
> smp_length = 0x736a0
> smp_alloc = 0x71d7e400
> metaslab 152 offset 13000000000 spacemap 17569 free 3.65G
> space map object 17569:
> smp_length = 0x10f008
> smp_alloc = 0x1167afc00
> metaslab 153 offset 13200000000 spacemap 17713 free 3.62G
> space map object 17713:
> smp_length = 0x986a8
> smp_alloc = 0x118140000
> metaslab 154 offset 13400000000 spacemap 80974 free 2.23G
> space map object 80974:
> smp_length = 0x9ac40
> smp_alloc = 0x171379400
> metaslab 155 offset 13600000000 spacemap 80441 free 2.00G
> space map object 80441:
> smp_length = 0x112e20
> smp_alloc = 0x17fefc400
> metaslab 156 offset 13800000000 spacemap 73829 free 3.06G
> space map object 73829:
> smp_length = 0x145288
> smp_alloc = 0x13c585400
> metaslab 157 offset 13a00000000 spacemap 17147 free 2.43G
> space map object 17147:
> smp_length = 0x141bf0
> smp_alloc = 0x164695c00
> metaslab 158 offset 13c00000000 spacemap 36996 free 4.20G
> space map object 36996:
> smp_length = 0x172368
> smp_alloc = 0xf2ec9400
> metaslab 159 offset 13e00000000 spacemap 18067 free 2.26G
> space map object 18067:
> smp_length = 0xfd3e0
> smp_alloc = 0x16f8d5000
> metaslab 160 offset 14000000000 spacemap 21659 free 2.10G
> space map object 21659:
> smp_length = 0xe2698
> smp_alloc = 0x179ba3800
> metaslab 161 offset 14200000000 spacemap 18090 free 2.07G
> space map object 18090:
> smp_length = 0x102e20
> smp_alloc = 0x17bc8d800
> metaslab 162 offset 14400000000 spacemap 18331 free 3.23G
> space map object 18331:
> smp_length = 0xe2728
> smp_alloc = 0x1310de000
> metaslab 163 offset 14600000000 spacemap 17739 free 1.84G
> space map object 17739:
> smp_length = 0x809a0
> smp_alloc = 0x18a0cc800
> metaslab 164 offset 14800000000 spacemap 18180 free 3.15G
> space map object 18180:
> smp_length = 0x147f58
> smp_alloc = 0x1364f6000
> metaslab 165 offset 14a00000000 spacemap 62570 free 2.46G
> space map object 62570:
> smp_length = 0xb8ea8
> smp_alloc = 0x162c23400
> metaslab 166 offset 14c00000000 spacemap 17961 free 2.69G
> space map object 17961:
> smp_length = 0x1505f8
> smp_alloc = 0x1539c9c00
> metaslab 167 offset 14e00000000 spacemap 78850 free 2.83G
> space map object 78850:
> smp_length = 0xaeca8
> smp_alloc = 0x14b301400
> metaslab 168 offset 15000000000 spacemap 21553 free 2.83G
> space map object 21553:
> smp_length = 0x16fb88
> smp_alloc = 0x14acbb400
> metaslab 169 offset 15200000000 spacemap 62328 free 3.40G
> space map object 62328:
> smp_length = 0x135d00
> smp_alloc = 0x126348000
> metaslab 170 offset 15400000000 spacemap 21075 free 1.64G
> space map object 21075:
> smp_length = 0x96dd8
> smp_alloc = 0x19703cc00
> metaslab 171 offset 15600000000 spacemap 34815 free 2.94G
> space map object 34815:
> smp_length = 0xba0c0
> smp_alloc = 0x14414f800
> metaslab 172 offset 15800000000 spacemap 78539 free 2.21G
> space map object 78539:
> smp_length = 0xec238
> smp_alloc = 0x172490400
> metaslab 173 offset 15a00000000 spacemap 5280 free 5.17G
> space map object 5280:
> smp_length = 0x8d288
> smp_alloc = 0xb5089c00
> metaslab 174 offset 15c00000000 spacemap 23160 free 2.27G
> space map object 23160:
> smp_length = 0xe84c0
> smp_alloc = 0x16f09ac00
> metaslab 175 offset 15e00000000 spacemap 21555 free 3.68G
> space map object 21555:
> smp_length = 0x126600
> smp_alloc = 0x11465cc00
> metaslab 176 offset 16000000000 spacemap 583 free 2.58G
> space map object 583:
> smp_length = 0x1159d8
> smp_alloc = 0x15a928800
> metaslab 177 offset 16200000000 spacemap 20986 free 2.11G
> space map object 20986:
> smp_length = 0x153ad8
> smp_alloc = 0x17934f800
> metaslab 178 offset 16400000000 spacemap 21948 free 1.47G
> space map object 21948:
> smp_length = 0xa2030
> smp_alloc = 0x1a20b3400
> metaslab 179 offset 16600000000 spacemap 77647 free 4.32G
> space map object 77647:
> smp_length = 0xba610
> smp_alloc = 0xebca0800
> metaslab 180 offset 16800000000 spacemap 37537 free 1.88G
> space map object 37537:
> smp_length = 0x1682c8
> smp_alloc = 0x18780a800
> metaslab 181 offset 16a00000000 spacemap 20079 free 3.42G
> space map object 20079:
> smp_length = 0xf2a50
> smp_alloc = 0x125273c00
> metaslab 182 offset 16c00000000 spacemap 21419 free 1.31G
> space map object 21419:
> smp_length = 0xb67e0
> smp_alloc = 0x1ac114400
> metaslab 183 offset 16e00000000 spacemap 33252 free 4.18G
> space map object 33252:
> smp_length = 0x1459f8
> smp_alloc = 0xf42da800
> metaslab 184 offset 17000000000 spacemap 5279 free 2.27G
> space map object 5279:
> smp_length = 0xe6118
> smp_alloc = 0x16f007800
> metaslab 185 offset 17200000000 spacemap 21773 free 2.78G
> space map object 21773:
> smp_length = 0x8cf40
> smp_alloc = 0x14e072400
> metaslab 186 offset 17400000000 spacemap 21643 free 1.36G
> space map object 21643:
> smp_length = 0x768a8
> smp_alloc = 0x1a93ef000
> metaslab 187 offset 17600000000 spacemap 21663 free 1.89G
> space map object 21663:
> smp_length = 0xdaa08
> smp_alloc = 0x186f8d000
> metaslab 188 offset 17800000000 spacemap 82892 free 2.94G
> space map object 82892:
> smp_length = 0x141750
> smp_alloc = 0x143c02400
> metaslab 189 offset 17a00000000 spacemap 78534 free 1.55G
> space map object 78534:
> smp_length = 0xbc0a0
> smp_alloc = 0x19cb5a000
> metaslab 190 offset 17c00000000 spacemap 4755 free 2.94G
> space map object 4755:
> smp_length = 0x1184c0
> smp_alloc = 0x143988c00

[Graham Perrin]

On 27/11/2022 08:11, Andrea Venturoli wrote:

… Any hint on how to solve this (apart from obviously recreate the pool from scracth, after a backup)? …

First, try a simple restart of the operating system.

(Sometimes: an error that is reportedly permanent becomes transient.)

[Andrea Venturoli]

On 11/27/22 14:36, Graham Perrin wrote:
> On 27/11/2022 08:11, Andrea Venturoli wrote:
>> … Any hint on how to solve this (apart from obviously recreate the
>> pool from scracth, after a backup)? …
>
> First, try a simple restart of the operating system.

Done.
Didn't help.
Thanks anyway.


What's that "<0x............>:<0x....>" supposed to mean? Some sort of
address/location, I suppose. But how do I interpret it?

bye
av.

[David Christensen]

<snip>


I agree that "<0xffffffffffffffff>:<0x0>" does not look like a file name
-- it looks like a signed 64-bit integer value of -1 expressed in
hexadecimal with less-than and greater-than delimiters, followed by a
colon, followed by 0 expressed in hexadecimal with the same delimiters.


But, AIUI, Unix filenames can contain any characters except NUL. So,
there could be a file with that name somewhere on your system. I
suggest looking for the file using find(1); if only as a sanity check.
Be sure to check ".zfs/snapshot" directories also.


David

[Sysadmin Lists]

> ----------------------------------------
> From: Andrea Venturoli <m...@netfence.it>
> Sent: Mon Nov 28 09:39:24 CET 2022
> To: <ques...@freebsd.org>
> Subject: Re: ZFS Permanent error <0xffffffffffffffff>:<0x0>

>
>
> What's that "<0x............>:<0x....>" supposed to mean? Some sort of
> address/location, I suppose. But how do I interpret it?
>
> bye
> av.
>

The construct is <dataset>:<filename>. It looks like you deleted the dataset
the corrupted file was found on, so now ZFS displays it as that string.

Bottom of the page here:
https://docs.oracle.com/cd/E19253-01/819-5461/gbctx/index.html

You can check `zpool history' for what it was called.

--
Sent with https://mailfence.com
Secure and private email

[Andrea Venturoli]

On 11/29/22 02:43, David Christensen wrote:

> I agree that "<0xffffffffffffffff>:<0x0>" does not look like a file name
> -- it looks like a signed 64-bit integer value of -1 expressed in
> hexadecimal with less-than and greater-than delimiters, followed by a
> colon, followed by 0 expressed in hexadecimal with the same delimiters.

Like an address into the spacemap? Only, I don't think such a location
exists.

> But, AIUI, Unix filenames can contain any characters except NUL.  So,
> there could be a file with that name somewhere on your system.  I
> suggest looking for the file using find(1); if only as a sanity check.
> Be sure to check ".zfs/snapshot" directories also.

No file by this name, nowhere.

bye & Thanks anyway
av.

[Andrea Venturoli]

On 11/29/22 06:27, Sysadmin Lists wrote:

Hello!

> The construct is <dataset>:<filename>. It looks like you deleted the dataset
> the corrupted file was found on, so now ZFS displays it as that string.

This is useful info.
Could the meaning of "dataset" include snapshots too?
I don't think I deleted a dataset, but I've got automatical periodical
snapshotting (of course deleting old ones).

Scratching my head here...
If I had a corrupted file on a dataset: first, why was it not detected
earlier?
Second, if I delete that dataset, why didn't the (corrupted) file go
away with it? (If a dataset does not exist anymore, how can it still
have files?)

> You can check `zpool history' for what it was called.

Unfortunately, it's quite hard:
# zpool history zroo2 | grep "2022-11-2[56].*destroy"|wc
802 3229 61694

> Bottom of the page here:
> https://docs.oracle.com/cd/E19253-01/819-5461/gbctx/index.html

Sorry, but I don't get it.
Unless I'm missing something, it describes what to do when a file (or
its metadata) is corrupted; you say the corrupted file was on a
destroyed dataset. So I do I "move" or "delete" it now, if I cannot
address it???

bye & Thanks
av.

[CyberLeo Kitsana]

On 11/27/22 02:11, Andrea Venturoli wrote:
> Hello.
>
> Yesterday, out of the blue, on a 13.1/amd64 machine, something went
> wrong in one of my zpools, as I discovered from daily run mails.

<snip>

>> errors: Permanent errors have been detected in the following files:
>>
>>         <0xffffffffffffffff>:<0x0>
>
> I don't think there's a "file in question" I can restore, as
> "<0xffffffffffffffff>:<0x0>" doesn't seem to identify any.
>
> I searched and found some bugs/discussion, but none lead to a definitive
> answer.
>
> From logs and SMART, I don't think I had any hadware problem.
>
> How much should I be worried?
> Any hint on how to solve this (apart from obviously recreate the pool
> from scracth, after a backup)?

These error reports will persist, even after the underlying problem is
corrected, until you tell ZFS that they have been corrected using 'zpool
clear'.

I would suggest clearing the errors using zpool clear and then running a
scrub to make sure there are no other errors lurking. If the errors
don't return during a scrub, they were probably either transient or
already corrected by the underlying hardware.

--
Fuzzy love,
-CyberLeo


Technical Administrator

CyberLeo.Net Webhosting
http://www.CyberLeo.Net

Element9 Communications
http://www.Element9.net


Furry Peace! - http://www.fur.com/peace/

[Sysadmin Lists]

> ----------------------------------------
> From: Andrea Venturoli <m...@netfence.it>

> Sent: Tue Nov 29 09:55:04 CET 2022
> To: Sysadmin Lists <sysadmi...@mailfence.com>
> Cc: <ques...@freebsd.org>

> Subject: Re: ZFS Permanent error <0xffffffffffffffff>:<0x0>
>
>

The rules appear to be:

1. If the file metadata is intact, it shows the full mount path to the file:
/path/to/filename

2. If the metadata is corrupted but the dataset still exists, it shows this:
full/dataset/name:<object number>

3. If the dataset has been deleted, it shows this:
<dataset number>:<object number>
or
'<metadata>':<object number>

IIRC, snapshots are valid entries. Perhaps the corrupted file is still
associated with other snapshots. You might need to use `zdb' to find it and
delete it from the filesystem, or maybe a `find /dataset/mount/point/' will
trigger an abort on the inode that is corrupted. You can search inside the
'/dataset/mount/point/.zfs/snapshot/' directory.

[Steve O'Hara-Smith]

On Wed, 30 Nov 2022 07:54:00 +0100 (CET)
Sysadmin Lists <sysadmi...@mailfence.com> wrote:

> 1. If the file metadata is intact, it shows the full mount path to the
> file: /path/to/filename
>
> 2. If the metadata is corrupted but the dataset still exists, it shows
> this: full/dataset/name:<object number>
>
> 3. If the dataset has been deleted, it shows this:
> <dataset number>:<object number>
> or
> '<metadata>':<object number>

Right but what the OP sees is <-1>:<0> which rather implies that
neither dataset nor object number exist now. It appears to be corruption in
a place that is no longer used - clearing the error message and running a
scrub is the right thing to do.

--
Steve O'Hara-Smith <st...@sohara.org>

[Sysadmin Lists]

> ----------------------------------------
> From: Steve O'Hara-Smith <st...@sohara.org>
> Sent: Wed Nov 30 10:00:41 CET 2022
> To: Sysadmin Lists <sysadmi...@mailfence.com>
> Cc: <ques...@freebsd.org>, Andrea Venturoli <m...@netfence.it>

> Subject: Re: ZFS Permanent error <0xffffffffffffffff>:<0x0>
>
>

Sure, it's the right thing to try first, but doesn't always work.

[Andrea Venturoli]

On 11/29/22 17:45, CyberLeo Kitsana wrote:

> I would suggest clearing the errors using zpool clear and then running a
> scrub to make sure there are no other errors lurking.

Hello.

"zpool clear zroo2" does absolutely nothing.
After issuing it, "zpool status" still reports the same error.
I tried a scrub, which "repaired 0B in 00:25:13 with 0 errors".
Still the same situation.

bye & Thanks
av.

[Andrea Venturoli]

On 11/30/22 07:54, Sysadmin Lists wrote:

> IIRC, snapshots are valid entries. Perhaps the corrupted file is still
> associated with other snapshots. You might need to use `zdb' to find it and
> delete it from the filesystem

Thanks for your explanation, but this is beyond my skills currently
(i.e. unless I find a good tutorial on zdb).
What should I be looking for?

If the file is associated to another snapshot, why doesn't it say the
name of this snapshot, instead of the deleted one?

In any case, you are saying my storage isn't exploding and I can live
with this harmless warning?
If it's so, well, maybe it will go away someday as older automatical
snapshots are deleted.

bye & Thanks
av.

[Graham Perrin]

On 27/11/2022 08:11, Andrea Venturoli wrote:

> … I don't think there's a "file in question" I can restore, as
> "<0xffffffffffffffff>:<0x0>" doesn't seem to identify any. …


Can something like this be a valid command?

zdb -d <0xffffffffffffffff>:<0x0>

Guesswork from
<https://openzfs.github.io/openzfs-docs/man/8/zdb.8.html#EXAMPLES>
example 3.


Also, taking a hint from <https://github.com/openzfs/zfs/issues/3476>
and <https://github.com/openzfs/zfs/issues/4736> (but not assuming a
hardware issue in your case):

zdb -mc zroo2

[Andrea Venturoli]

On 12/2/22 00:21, Graham Perrin wrote:

> Can something like this be a valid command?
>
> zdb -d <0xffffffffffffffff>:<0x0>

# zdb -d "<0xffffffffffffffff>:<0x0>"
zdb: can't open '<0xffffffffffffffff>:<0x0>': No such file or directory

> Also, taking a hint from <https://github.com/openzfs/zfs/issues/3476> and <https://github.com/openzfs/zfs/issues/4736> (but not assuming a hardware issue in your case):

I had read those, but they didn't add much.

I also read https://github.com/openzfs/zfs/issues/3094: maybe I don't
understand it fully, but it didn't lead to any definitive solution.



> zdb -mc zroo2

Thìs didn't fix the error.
Do you want me to copy all the output here?
And yes, I see some leaked space (see #3094), but I'm not sure the two
issues are related.



bye & Thanks
av.

[Sysadmin Lists]

> ----------------------------------------
> From: Andrea Venturoli <m...@netfence.it>

> Sent: Thu Dec 01 16:50:09 CET 2022
> To: <sysadmi...@mailfence.com>

> Cc: <ques...@freebsd.org>
> Subject: Re: ZFS Permanent error <0xffffffffffffffff>:<0x0>
>
>

> On 11/30/22 07:54, Sysadmin Lists wrote:
>
> > IIRC, snapshots are valid entries. Perhaps the corrupted file is still
> > associated with other snapshots. You might need to use `zdb' to find it and
> > delete it from the filesystem
>
> Thanks for your explanation, but this is beyond my skills currently
> (i.e. unless I find a good tutorial on zdb).
> What should I be looking for?

Graham's suggestions look intriguing.

> If the file is associated to another snapshot, why doesn't it say the
> name of this snapshot, instead of the deleted one?

I don't know. I'm not an OpenZFS developer, I'm just going off personal
experience with similar errors. IIRC, a snapshot was still holding onto the
corrupted file.

> In any case, you are saying my storage isn't exploding and I can live
> with this harmless warning?
> If it's so, well, maybe it will go away someday as older automatical
> snapshots are deleted.

I'd run a long smartctl test on the disks in your pool if I were you.

> bye & Thanks
> av.

[Andrea Venturoli]

On 12/2/22 08:15, Sysadmin Lists wrote:

> I'd run a long smartctl test on the disks in your pool if I were you.

"Completed without error" on all three drives.

bye & Thanks
av.

[Sysadmin Lists]

> ----------------------------------------
> From: Andrea Venturoli <m...@netfence.it>

> Sent: Fri Dec 02 08:27:30 CET 2022
> To: <ques...@freebsd.org>

> Subject: Re: ZFS Permanent error <0xffffffffffffffff>:<0x0>
>
>

That's good news. You should check overall health with:
smartctl -a or smartctl -x

If they show low bad counters and no read-errors, odds are you're safe.

[Graham Perrin]

On 02/12/2022 06:58, Andrea Venturoli wrote:

On 12/2/22 00:21, Graham Perrin wrote:

Can something like this be a valid command?

zdb -d <0xffffffffffffffff>:<0x0>

# zdb -d "<0xffffffffffffffff>:<0x0>"
zdb: can't open '<0xffffffffffffffff>:<0x0>': No such file or directory

For your zroo2 pool, try this: 

zdb -d zroo2 '0xffffffffffffffff'

Again, that's guesswork, and for what it's worth, I got this (for my august pool), trial and error:

root@mowa219-gjp4-8570p-freebsd:~ # zdb -d <0xffffffffffffffff>:<0x0>
Ambiguous input redirect.
root@mowa219-gjp4-8570p-freebsd:~ # sh

# zdb -d <0xffffffffffffffff>:<0x0>

sh: Syntax error: newline unexpected (expecting word)
# zdb -d august <0xffffffffffffffff>:<0x0>
sh: Syntax error: newline unexpected (expecting word)
# zdb -d august '<0xffffffffffffffff>:<0x0>'
zdb: Bad object or range: '<0xffffffffffffffff>:<0x0>': Invalid characters in start object ID

# zdb -d august '0xffffffffffffffff:0x0'
zdb: Bad object or range: '0xffffffffffffffff:0x0': Start object ID may not exceed end object ID

# zdb -d august '0xffffffffffffffff'
Dataset mos [META], ID 0, cr_txg 4, 924M, 1683 objects

   Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
zdb: dmu_object_info() failed, errno 2
# exit
root@mowa219-gjp4-8570p-freebsd:~ # zdb -d august 0xffffffffffffffff
Dataset mos [META], ID 0, cr_txg 4, 924M, 1685 objects

   Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
zdb: dmu_object_info() failed, errno 2
root@mowa219-gjp4-8570p-freebsd:~ #

[Graham Perrin]

In parallel: it's remarkable that you have the same number of checksum
errors (four) on every drive.

[Graham Perrin]

On 02/12/2022 06:58, Andrea Venturoli wrote:

…

zdb -mc zroo2

Thìs didn't fix the error.

In this context, option -m is to display (not to attempt any fix). Please see:

<https://openzfs.github.io/openzfs-docs/man/8/zdb.8.html#m~2>

Do you want me to copy all the output here? …

No, thank you.

Re: my own results, I seek advice in the #storage channel in Discord for FreeBSD.

<https://discord.com/channels/727023752348434432/757305697527398481/1048117129985134642>

– my query there includes reference to your <https://lists.freebsd.org/archives/freebsd-questions/2022-November/002390.html>, although please note that archiving is somewhat broken.

[Andrea Venturoli]

On 12/2/22 08:42, Graham Perrin wrote:

> In parallel: it's remarkable that you have the same number of checksum
> errors (four) on every drive.

Yes, that hit me too.
That's why I think I have no hardware problem, but, rather, a software
issue.

bye & Thanks
av.

[Andrea Venturoli]

On 12/2/22 08:40, Graham Perrin wrote:

> For your /zroo2/ pool, try this:
>
> zdb -d zroo2 '0xffffffffffffffff'

Similar to your:

# zdb -d zroo2 '0xffffffffffffffff'
Dataset mos [META], ID 0, cr_txg 4, 1.15G, 28248 objects

Object lvl iblk dblk dsize dnsize lsize %full type
zdb: dmu_object_info() failed, errno 2

bye & Thanks
av.

[Andrea Venturoli]

On 12/2/22 08:27, Andrea Venturoli wrote:
> On 12/2/22 08:15, Sysadmin Lists wrote:
>
>> I'd run a long smartctl test on the disks in your pool if I were you.
>
> "Completed without error" on all three drives.

The RAM alas... just removed a faulty module.
So I hope I don't get any more troubles, even if I don't know how to
clear/fix the old one.

bye & Thanks
av.

[Sysadmin Lists]

> ----------------------------------------
> From: Andrea Venturoli <m...@netfence.it>

> Sent: Sat Dec 31 11:41:39 CET 2022
> To: <ques...@freebsd.org>

> Subject: Re: ZFS Permanent error <0xffffffffffffffff>:<0x0>
>

> The RAM alas... just removed a faulty module.
> So I hope I don't get any more troubles, even if I don't know how to
> clear/fix the old one.
>

Recreating the pool is what ultimately cleared it for me: Attached a backup
disk, booted into a livecd, took a recursive snapshot of zroot, sent it to the
backup disk, destroyed the old pool, recreated it, sent the backed-up datasets
to the old disk.

I initially did the waiting game like you, but then started getting CAM errors
that made the system unusable. I tried refreshing the disk, but that didn't
help:

dd if=/dev/ada0p3 of=/dev/ada0p3 bs=1m

The ZFS metadata errors vanished after recreating the pool. It's about an
hour-long project.

A recursive snapshot of zroot sends everything over with a single command:
livecd# zfs send -vR zroot@migration | zfs recv -vFud new_zroot

And then back again after recreating the pool:
livecd# zfs send -vR new_zroot@migration | zfs recv -vFud zroot

Just have to make sure you're using a livecd with a ZFS version that's equal or
newer. After all the hassle, I did contemplate setting up a mirror raid.

[Andrea Venturoli]

On 12/2/22 08:54, Graham Perrin wrote:
> On 02/12/2022 06:58, Andrea Venturoli wrote:

Oh! It looks like it autohealed!!!
After the last periodic scrub, the error just went away.
I guess this followed auto removal of old automatic snapshots.

Thanks to all that tried to help.

bye
av.