Cannot find out what uses space in ZFS dataset

[Andrea Venturoli]

Hello.

I've got a urgent trouble I'm trying to solve.
I need to take backups of a server: they should be around 4-5 GBs, but
the end up close to 100GB!!!

The trouble is with the zroot/ROOT/default dataset:
> # cd /
> # du -d 0 -h -x
> 3.3G .

Yet:
> root@chet:/ # zfs list|grep ROOT
> zroot/ROOT 70.9G 1.94T 88K none
> zroot/ROOT/default 70.9G 1.94T 62.1G /

So ho do we go from 3.3G to 62.1G???
There are snapshots, but I'm using plain "zfs send", so they should not
matter...



> # zfs get all zroot/ROOT/default
> NAME PROPERTY VALUE SOURCE
> zroot/ROOT/default type filesystem -
> zroot/ROOT/default creation Sun Dec 15 16:58 2019 -
> zroot/ROOT/default used 70.9G -
> zroot/ROOT/default available 1.94T -
> zroot/ROOT/default referenced 62.1G -
> zroot/ROOT/default compressratio 1.62x -
> zroot/ROOT/default mounted yes -
> zroot/ROOT/default quota none default
> zroot/ROOT/default reservation none default
> zroot/ROOT/default recordsize 128K default
> zroot/ROOT/default mountpoint / local
> zroot/ROOT/default sharenfs off default
> zroot/ROOT/default checksum on default
> zroot/ROOT/default compression lz4 inherited from zroot
> zroot/ROOT/default atime off inherited from zroot
> zroot/ROOT/default devices on default
> zroot/ROOT/default exec on default
> zroot/ROOT/default setuid on default
> zroot/ROOT/default readonly off local
> zroot/ROOT/default jailed off default
> zroot/ROOT/default snapdir hidden default
> zroot/ROOT/default aclmode discard default
> zroot/ROOT/default aclinherit restricted default
> zroot/ROOT/default createtxg 8 -
> zroot/ROOT/default canmount noauto local
> zroot/ROOT/default xattr on default
> zroot/ROOT/default copies 1 default
> zroot/ROOT/default version 5 -
> zroot/ROOT/default utf8only off -
> zroot/ROOT/default normalization none -
> zroot/ROOT/default casesensitivity sensitive -
> zroot/ROOT/default vscan off default
> zroot/ROOT/default nbmand off default
> zroot/ROOT/default sharesmb off default
> zroot/ROOT/default refquota none default
> zroot/ROOT/default refreservation none default
> zroot/ROOT/default guid 13999796839276270198 -
> zroot/ROOT/default primarycache all default
> zroot/ROOT/default secondarycache all default
> zroot/ROOT/default usedbysnapshots 8.76G -
> zroot/ROOT/default usedbydataset 62.1G -
> zroot/ROOT/default usedbychildren 0B -
> zroot/ROOT/default usedbyrefreservation 0B -
> zroot/ROOT/default logbias latency default
> zroot/ROOT/default objsetid 89 -
> zroot/ROOT/default dedup off default
> zroot/ROOT/default mlslabel none default
> zroot/ROOT/default sync standard default
> zroot/ROOT/default dnodesize legacy default
> zroot/ROOT/default refcompressratio 1.56x -
> zroot/ROOT/default written 1.79M -
> zroot/ROOT/default logicalused 114G -
> zroot/ROOT/default logicalreferenced 96.6G -
> zroot/ROOT/default volmode default default
> zroot/ROOT/default filesystem_limit none default
> zroot/ROOT/default snapshot_limit none default
> zroot/ROOT/default filesystem_count none default
> zroot/ROOT/default snapshot_count none default
> zroot/ROOT/default snapdev hidden default
> zroot/ROOT/default acltype nfsv4 default
> zroot/ROOT/default context none default
> zroot/ROOT/default fscontext none default
> zroot/ROOT/default defcontext none default
> zroot/ROOT/default rootcontext none default
> zroot/ROOT/default relatime on default
> zroot/ROOT/default redundant_metadata all default
> zroot/ROOT/default overlay on default
> zroot/ROOT/default encryption off default
> zroot/ROOT/default keylocation none default
> zroot/ROOT/default keyformat none default
> zroot/ROOT/default pbkdf2iters 0 default
> zroot/ROOT/default special_small_blocks 0 default
> zroot/ROOT/default snapshots_changed Thu Sep 18 12:00:02 2025 -
> zroot/ROOT/default prefetch all default
> zroot/ROOT/default autobackup:auto_zroot true inherited from zroot



usedbydataset is 62.1G!!!
Where can this come from if du says 3.3???



bye & Thanks
av.

[Frank Leonhardt]

On 18/09/2025 11:10, Andrea Venturoli wrote:
> Hello.
>
> I've got a urgent trouble I'm trying to solve.
> I need to take backups of a server: they should be around 4-5 GBs, but
> the end up close to 100GB!!!
>
> The trouble is with the zroot/ROOT/default dataset:
>> # cd /
>> # du -d 0 -h -x
>> 3.3G    .
>
> Yet:
>> root@chet:/ # zfs list|grep ROOT
>> zroot/ROOT                     70.9G  1.94T    88K  none
>> zroot/ROOT/default             70.9G  1.94T  62.1G  /
>
> So ho do we go from 3.3G to 62.1G???
> There are snapshots, but I'm using plain "zfs send", so they should
> not matter...
>
>
>
>> # zfs get all zroot/ROOT/default
>> NAME                PROPERTY VALUE                     SOURCE
>> zroot/ROOT/default  type filesystem                -
>> zroot/ROOT/default  creation               Sun Dec 15 16:58 2019     -

>> <snip>

>> zroot/ROOT/default  keyformat none                      default
>> zroot/ROOT/default  pbkdf2iters 0                         default
>> zroot/ROOT/default  special_small_blocks 0                        
>> default
>> zroot/ROOT/default  snapshots_changed      Thu Sep 18 12:00:02 2025  -
>> zroot/ROOT/default  prefetch all                       default
>> zroot/ROOT/default  autobackup:auto_zroot true                     
>> inherited from zroot
>
>
>
> usedbydataset is 62.1G!!!
> Where can this come from if du says 3.3???
>
>
>

First - and I can't say this often enough - ignore du with zfs! It
produces a figure for backward compatibility purposes but it's not good
at dealing with zfs concepts like compression.

Use "zfs list".

The REFERenced column is the amount of data accessible in the dataset
and may be shared with other datasets. The Used column is the amount of
space consumed by the dataset AND ALL IT'S DESCENDENTS and snapshots and
suchlike.

Things like cloned datasets and block deduplication confuse matters even
further. On later zpool versions (I assume yours is recent) you can get
more detail with this:

zfs list -t all -o
name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots

The default zfs list properties are the tip of an iceberg!

Regards, Frank.

[Andrea Venturoli]

On 9/18/25 17:28, Frank Leonhardt wrote:


Thanks a lot for answering.

> First - and I can't say this often enough - ignore du with zfs! It
> produces a figure for backward compatibility purposes but it's not good
> at dealing with zfs concepts like compression.

Well...
I agree with you if the goal is finding out disk occupation.
However what I'm trying to do is "zfs send" this dataset: AFAICT this
ignores compression/deduplication/data sharing/etc... so it should more
or less agree with du.

> The REFERenced column is the amount of data accessible in the dataset
> and may be shared with other datasets.

OK.

> The Used column is the amount of
> space consumed by the dataset AND ALL IT'S DESCENDENTS and snapshots and
> suchlike.

"zfs send" does not include descendents or snapshots, so it's fine.

> Things like cloned datasets and block deduplication confuse matters even
> further.

I don't use these.

> zfs list -t all -o
> name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots

I don't think I'm interested in snapshots, but in any case:
> # zfs list -t all -o name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots|grep default
> zroot/ROOT/default 62.1G 70.9G 0B 62.1G 0B 8.76G
> zroot/ROOT/default@auto_zroot-20241014020000 3.11G 2.95G - - - -
> zroot/ROOT/default@auto_zroot-20241113010000 3.01G 198M - - - -
> zroot/ROOT/default@auto_zroot-20241213010000 3.30G 70.7M - - - -
> zroot/ROOT/default@auto_zroot-20250112010000 3.30G 60.4M - - - -
> zroot/ROOT/default@auto_zroot-20250211010000 3.30G 64.9M - - - -
> zroot/ROOT/default@auto_zroot-20250313010000 3.31G 263M - - - -
> zroot/ROOT/default@auto_zroot-20250412020000 3.32G 149M - - - -
> zroot/ROOT/default@auto_zroot-20250512020000 3.32G 68.6M - - - -
> zroot/ROOT/default@auto_zroot-20250611020000 3.34G 209M - - - -
> zroot/ROOT/default@auto_zroot-20250711020000 62.2G 407M - - - -
> zroot/ROOT/default@auto_zroot-20250810020000 62.1G 107M - - - -
> zroot/ROOT/default@auto_zroot-20250821020000 62.1G 36.2M - - - -
> zroot/ROOT/default@auto_zroot-20250828020000 62.1G 33.9M - - - -
> zroot/ROOT/default@auto_zroot-20250904020000 62.1G 28.1M - - - -
> zroot/ROOT/default@auto_zroot-20250909020000 62.1G 10.1M - - - -
> zroot/ROOT/default@auto_zroot-20250911020000 62.1G 10.3M - - - -
> zroot/ROOT/default@auto_zroot-20250912020000 62.1G 1.84M - - - -
> zroot/ROOT/default@auto_zroot-20250913020000 62.1G 5.97M - - - -
> zroot/ROOT/default@auto_zroot-20250914020000 62.1G 5.79M - - - -
> zroot/ROOT/default@auto_zroot-20250915020000 62.1G 4.75M - - - -
> zroot/ROOT/default@auto_zroot-20250916020000 62.1G 2.48M - - - -
> zroot/ROOT/default@auto_zroot-20250917020000 62.1G 2.08M - - - -
> zroot/ROOT/default@auto_zroot-20250917180000 62.1G 1.27M - - - -
> zroot/ROOT/default@auto_zroot-20250917190000 62.1G 1.44M - - - -
> zroot/ROOT/default@auto_zroot-20250917200000 62.1G 1.21M - - - -
> zroot/ROOT/default@auto_zroot-20250917210000 62.1G 1.18M - - - -
> zroot/ROOT/default@auto_zroot-20250917220000 62.1G 1.54M - - - -
> zroot/ROOT/default@auto_zroot-20250917230000 62.1G 1.36M - - - -
> zroot/ROOT/default@auto_zroot-20250918000000 62.1G 1.44M - - - -
> zroot/ROOT/default@auto_zroot-20250918010000 62.1G 1.42M - - - -
> zroot/ROOT/default@auto_zroot-20250918020000 62.1G 1.65M - - - -
> zroot/ROOT/default@auto_zroot-20250918030000 62.1G 1.95M - - - -
> zroot/ROOT/default@auto_zroot-20250918040000 62.1G 964K - - - -
> zroot/ROOT/default@auto_zroot-20250918050000 62.1G 1.61M - - - -
> zroot/ROOT/default@auto_zroot-20250918060000 62.1G 960K - - - -
> zroot/ROOT/default@auto_zroot-20250918070000 62.1G 1.48M - - - -
> zroot/ROOT/default@auto_zroot-20250918080000 62.1G 1.47M - - - -
> zroot/ROOT/default@auto_zroot-20250918090000 62.1G 1.74M - - - -
> zroot/ROOT/default@auto_zroot-20250918100000 62.1G 2.07M - - - -
> zroot/ROOT/default@auto_zroot-20250918110000 62.1G 2.03M - - - -
> zroot/ROOT/default@auto_zroot-20250918120000 62.1G 892K - - - -
> zroot/ROOT/default@auto_zroot-20250918130000 62.1G 1.19M - - - -
> zroot/ROOT/default@auto_zroot-20250918140000 62.1G 1.62M - - - -
> zroot/ROOT/default@auto_zroot-20250918150000 62.1G 1.81M - - - -
> zroot/ROOT/default@auto_zroot-20250918160000 62.1G 1.58M - - - -
> zroot/ROOT/default@auto_zroot-20250918170000 62.1G 1.86M - - - -

But again, how does this help me?
How do I find out which files take up those 62.1GB if "du" does not
account for them?

bye
av.

[fre...@vanderzwan.org]

> On 18 Sep 2025, at 17:57, Andrea Venturoli <m...@netfence.it> wrote:
>
>> zfs list -t all -o name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots
>
> I don't think I'm interested in snapshots, but in any case:
>> # zfs list -t all -o name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots|grep default
>> zroot/ROOT/default 62.1G 70.9G 0B 62.1G 0B 8.76G
>> zroot/ROOT/default@auto_zroot-20241014020000 3.11G 2.95G - - - -
>> zroot/ROOT/default@auto_zroot-20241113010000 3.01G 198M - - - -
>> zroot/ROOT/default@auto_zroot-20241213010000 3.30G 70.7M - - - -
>> zroot/ROOT/default@auto_zroot-20250112010000 3.30G 60.4M - - - -
>> zroot/ROOT/default@auto_zroot-20250211010000 3.30G 64.9M - - - -
>> zroot/ROOT/default@auto_zroot-20250313010000 3.31G 263M - - - -
>> zroot/ROOT/default@auto_zroot-20250412020000 3.32G 149M - - - -
>> zroot/ROOT/default@auto_zroot-20250512020000 3.32G 68.6M - - - -
>> zroot/ROOT/default@auto_zroot-20250611020000 3.34G 209M - - - -
>> zroot/ROOT/default@auto_zroot-20250711020000 62.2G 407M - - - -

Here the refer jumps from 3.34 to 62.2 GB, so it looks like your missing data is in this snapshot.

I guess 'zfs diff zroot/ROOT/default@auto_zroot-20250611020000 zroot/ROOT/default@auto_zroot-20250711020000' can show what happened.

>> zroot/ROOT/default@auto_zroot-20250810020000 62.1G 107M - - - -
>> zroot/ROOT/default@auto_zroot-20250821020000 62.1G 36.2M - - - -
>> zroot/ROOT/default@auto_zroot-20250828020000 62.1G 33.9M - - - -

Cheers,
Paul

[Andrea Venturoli]

On 9/18/25 18:44, fre...@vanderzwan.org wrote:

>>> zroot/ROOT/default@auto_zroot-20250611020000 3.34G 209M - - - -
>>> zroot/ROOT/default@auto_zroot-20250711020000 62.2G 407M - - - -
>
> Here the refer jumps from 3.34 to 62.2 GB, so it looks like your missing data is in this snapshot.

Hmmm...

> # zfs list -o name,refer,used,usedbysnapshots|grep default
> zroot/ROOT/default 62.1G 70.5G 8.36G

usedbysnapshots is 8.36G, so I guess this single snapshot cannot take 60GB.
Also "zfs send" doesn't transmit snapshots, so, even if this was true,
it should not produce such a big amount of data.

> I guess 'zfs diff zroot/ROOT/default@auto_zroot-20250611020000 zroot/ROOT/default@auto_zroot-20250711020000' can show what happened.

I know what happened on that day: I did a big upgrade of the system.
However, I've done this hundreds of times on several boxes and never
incurred in this problem.




In any case, I tried deleting the 20250711020000 snapshot.
Now the jump in refer just moved to the next:> zfs list -t all -o
name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots|grep
default
> zroot/ROOT/default 62.1G 70.5G 0B 62.1G 0B 8.36G
> ...
> zroot/ROOT/default@auto_zroot-20250611020000 3.34G 215M - - - -
> zroot/ROOT/default@auto_zroot-20250810020000 62.1G 116M - - - -
> ...
Still "zfs send" generates the same huge amount of data.





bye & Thanks
av.

[fre...@vanderzwan.org]

That’s because for some reason the data is still referenced in snapshots.
If you delete all snapshots up to and including 20250810020000 you should see that usage drop to what you expect.
Including % in snapshot name would delete snapshots up to and including the named shapshot, so 'zfs destroy -v zroot/ROOT/default@%auto_zroot-20250611020000’ should work to free space.
Use -n first to verify the correct snapshots are deleted to be sure ;-)



Cheers,
Paul

[Andrea Venturoli]

On 9/18/25 20:00, fre...@vanderzwan.org wrote:

>> In any case, I tried deleting the 20250711020000 snapshot.
>> Now the jump in refer just moved to the next:> zfs list -t all -o name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots|grep default
>>> zroot/ROOT/default 62.1G 70.5G 0B 62.1G 0B 8.36G
>>> ...
>>> zroot/ROOT/default@auto_zroot-20250611020000 3.34G 215M - - - -
>>> zroot/ROOT/default@auto_zroot-20250810020000 62.1G 116M - - - -
>>> ...
>> Still "zfs send" generates the same huge amount of data.
>>
>>
>
> That’s because for some reason the data is still referenced in snapshots.

Then is usedbysnapshot lying???

> If you delete all snapshots up to and including 20250810020000 you should see that usage drop to what you expect.

Unfortunately not.

> # zfs list -t snap|grep default
> zroot/ROOT/default@auto_zroot-20250821020000 41.0M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250828020000 33.9M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250904020000 28.1M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250909020000 10.1M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250911020000 10.3M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250912020000 1.84M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250913020000 5.97M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250914020000 5.79M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250915020000 4.75M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250916020000 2.48M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250917020000 2.09M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250917210000 1.21M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250917220000 1.54M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250917230000 1.36M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918000000 1.44M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918010000 1.42M - 62.1G -

> zroot/ROOT/default@auto_zroot-20250918020000 1.65M - 62.1G -

> zroot/ROOT/default@auto_zroot-20250918030000 1.95M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918040000 964K - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918050000 1.61M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918060000 960K - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918070000 1.48M - 62.1G -

> zroot/ROOT/default@auto_zroot-20250918080000 1.47M - 62.1G -

> zroot/ROOT/default@auto_zroot-20250918090000 1.74M - 62.1G -

> zroot/ROOT/default@auto_zroot-20250918100000 2.07M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918110000 2.03M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918120000 892K - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918130000 1.19M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918140000 1.62M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918150000 1.81M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918160000 1.58M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918170000 1.88M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918180000 1.99M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918190000 1.12M - 62.1G -
> zroot/ROOT/default@auto_zroot-20250918200000 972K - 62.1G -

> # zfs list -o name,refer,used,usedbysnapshots|grep default

> zroot/ROOT/default 62.1G 62.4G 263M





> One more thing, can you show the output of ’zfs list -o space |grep zroot’ to list all datasets in the pool ?

> # zfs list -o space |grep zroot
> zroot 1.95T 1.50T 536K 88K 0B 1.50T
> zroot/ROOT 1.95T 62.4G 0B 88K 0B 62.4G
> zroot/ROOT/default 1.95T 62.4G 263M 62.1G 0B 0B
> zroot/ezjail 1.95T 1.40T 328K 128K 0B 1.40T
> zroot/ezjail/backup 1.95T 33.9G 9.00G 1.29G 0B 23.7G
> zroot/ezjail/backup/cache 1.95T 1.38G 1.20G 182M 0B 0B
> zroot/ezjail/backup/postgres 1.95T 22.3G 17.7G 4.55G 0B 0B
> zroot/ezjail/backup/tmp 1.95T 128K 0B 128K 0B 0B
> zroot/ezjail/basejail 1.95T 6.82G 5.52G 1.31G 0B 0B
> zroot/ezjail/dc 1.95T 4.52G 2.59G 793M 0B 1.16G
> zroot/ezjail/dc/cache 1.95T 1.16G 1.04G 119M 0B 0B
> zroot/ezjail/dc/tmp 1.95T 128K 0B 128K 0B 0B
> zroot/ezjail/fs 1.95T 1.21T 5.73G 28.8M 0B 1.21T
> zroot/ezjail/fs/cache 1.95T 498M 462M 36.3M 0B 0B
> zroot/ezjail/fs/images 1.95T 118G 0B 118G 0B 0B
> zroot/ezjail/fs/log 1.95T 89.3G 56.6G 32.7G 0B 0B
> zroot/ezjail/fs/shares 1.95T 481G 59.1G 422G 0B 0B
> zroot/ezjail/fs/tmp 1.95T 144K 0B 144K 0B 0B
> zroot/ezjail/fs/usr 1.95T 547G 1.18G 305M 0B 546G
> zroot/ezjail/fs/usr/home 1.95T 546G 31.9G 514G 0B 0B
> zroot/ezjail/ids 1.95T 16.2G 2.55G 584M 0B 13.1G
> zroot/ezjail/ids/cache 1.95T 96.5M 0B 96.5M 0B 0B
> zroot/ezjail/ids/logs 1.95T 12.6G 0B 12.6G 0B 0B
> zroot/ezjail/ids/spool 1.95T 364M 0B 364M 0B 0B
> zroot/ezjail/ids/tmp 1.95T 128K 0B 128K 0B 0B
> zroot/ezjail/mail 1.95T 128G 3.31G 662M 0B 124G
> zroot/ezjail/mail/cache 1.95T 558M 449M 110M 0B 0B
> zroot/ezjail/mail/clamav 1.95T 414M 0B 414M 0B 0B
> zroot/ezjail/mail/imap 1.95T 123G 17.2G 106G 0B 0B
> zroot/ezjail/mail/tmp 1.95T 344K 0B 344K 0B 0B
> zroot/ezjail/newjail 1.95T 70.8M 16K 70.8M 0B 0B
> zroot/ezjail/proxy 1.95T 6.32G 737M 222M 0B 5.39G
> zroot/ezjail/proxy/tmp 1.95T 152K 0B 152K 0B 0B
> zroot/ezjail/proxy/var 1.95T 5.39G 122M 22.5M 0B 5.24G
> zroot/ezjail/proxy/var/cache 1.95T 328M 294M 33.1M 0B 0B
> zroot/ezjail/proxy/var/clamav 1.95T 414M 0B 414M 0B 0B
> zroot/ezjail/proxy/var/log 1.95T 3.55G 2.54G 1.02G 0B 0B
> zroot/ezjail/proxy/var/squid 1.95T 989M 0B 989M 0B 0B
> zroot/home 1.95T 22.6G 22.6G 18.8M 0B 0B
> zroot/tmp 1.95T 134M 0B 134M 0B 0B
> zroot/usr 1.95T 11.8G 0B 88K 0B 11.8G
> zroot/usr/obj 1.95T 7.84G 0B 7.84G 0B 0B
> zroot/usr/src 1.95T 3.93G 0B 3.93G 0B 0B
> zroot/var 1.95T 5.32G 0B 88K 0B 5.32G
> zroot/var/audit 1.95T 88K 0B 88K 0B 0B
> zroot/var/cache 1.95T 213M 0B 213M 0B 0B
> zroot/var/clamav 1.95T 425M 0B 425M 0B 0B
> zroot/var/crash 1.95T 88K 0B 88K 0B 0B
> zroot/var/dumps 1.95T 88K 0B 88K 0B 0B
> zroot/var/log 1.95T 4.69G 3.74G 978M 0B 0B
> zroot/var/mail 1.95T 640K 528K 112K 0B 0B
> zroot/var/tmp 1.95T 88K 0B 88K 0B 0B



> And the output of ‘zpool list’.

> # zpool list
> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
> vm 446G 361G 85.4G - - 54% 80% 1.00x ONLINE -
> zroot 3.56T 1.50T 2.06T - - 19% 42% 1.00x ONLINE -




bye & Thanks
av.

[fre...@vanderzwan.org]

> On 18 Sep 2025, at 20:41, Andrea Venturoli <m...@netfence.it> wrote:
>
> On 9/18/25 20:00, fre...@vanderzwan.org wrote:
>
>>> In any case, I tried deleting the 20250711020000 snapshot.
>>> Now the jump in refer just moved to the next:> zfs list -t all -o name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots|grep default
>>>> zroot/ROOT/default 62.1G 70.5G 0B 62.1G 0B 8.36G
>>>> ...
>>>> zroot/ROOT/default@auto_zroot-20250611020000 3.34G 215M - - - -
>>>> zroot/ROOT/default@auto_zroot-20250810020000 62.1G 116M - - - -
>>>> ...
>>> Still "zfs send" generates the same huge amount of data.
>>>
>>>
>> That’s because for some reason the data is still referenced in snapshots.
>
> Then is usedbysnapshot lying???
>
>
>
>> If you delete all snapshots up to and including 20250810020000 you should see that usage drop to what you expect.
>
> Unfortunately not.

My bad, I had to leave my desk and before I got back I realized myself I was wrong.

Those look pretty normal.
One thing that could explain this difference between du and zfs used is if you mount a small/empty filesystem on top of a very large directory.
In that case the content of the directory would be invisible because the mount masks it.
Does the output of the mount command show any strange mounts that could cause this ?
About the du output what does ‘du -mx / |sort -n |tail ‘ show ?

Cheers,
Paul

>
>
> bye & Thanks
> av.
>

[David Christensen]

Understand that ZFS datasets are hierarchical -- e.g. filesystems within
filesystems. It looks like you have root-on-ZFS. This wiki page
describes the filesystems the FreeBSD installer likely created within zpool:

https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot#Create_the_ZFS_file_system_hierarchy


Run the following command to see the filesystems on zpool:

# zfs list -r -t filesystem zpool


Datasets can have snapshots. This is why ZFS reports more space used
than du reports. Run the following command to see the snapshots on zpool:

# zfs list -r -t snapshot zpool


freebsd-update takes snapshots of the root filesystem prior to upgrades.
If your system has been upgraded many times, you will have many
snapshots. These snapshots will consume space. That is why the
"referenced" values reported by zfs(8) are larger than the usage values
reported by du(1).


To do a full backup, you need to send the entire dataset hierarchy and
all of its snapshots someplace -- a file, another pool, etc.. This is
known as "replication".


If you first destroy snapshots you do not want (e.g. "prune snapshots"),
the replication stream will be smaller.


Given the 1 plus 16 filesystems listed above, doing replication by hand
will be tedious and error prone. I have written a few generations of
scripts to automate this chore, first in Bourne shell and later in Perl.
Alternatively, it looks like there is at least one packaged solution
available (untested):

2025-09-18 13:49:04 toor@f5 ~
# pkg search replicate | grep -i zfs
zfs-replicate-2.0.2 ZFS Snapshot Replication Script


Michael W. Lucas wrote several texts that are worth owning:

https://mwl.io/nonfiction/os#af3e

https://mwl.io/nonfiction/os#fmse

https://mwl.io/nonfiction/os#fmzfs

https://mwl.io/nonfiction/os#fmaz


Klara Systems has several articles that are worth reading:

https://klarasystems.com/articles/?topics=zfs

https://klarasystems.com/articles/introduction-to-zfs-replication/


David

[David Christensen]

On 9/18/25 08:28, Frank Leonhardt wrote:
<snip>


Is your mail client modifying the Subject line by prefixing "[List]"
when you reply?


I typically configure my mail client to sort messages by Subject and
then by reverse (sent?) Date. The "[List]" Subject prefix breaks the
thread into two groups -- those with the original Subject and those with
"[List]" Subject.


Furthermore, changing the Subject may adversely impact automated
archiving and searching solutions on the Internet.


Whoever is doing this, please stop.


Thank you,

David

[David Christensen]

On 9/18/25 08:57, Andrea Venturoli wrote:
> On 9/18/25 17:28, Frank Leonhardt wrote:
>
> Thanks a lot for answering.
>
>> First - and I can't say this often enough - ignore du with zfs! It
>> produces a figure for backward compatibility purposes but it's not
>> good at dealing with zfs concepts like compression.
>
> Well...
> I agree with you if the goal is finding out disk occupation.
> However what I'm trying to do is "zfs send"

> <snip>


Please post the console session. zfs-send(8) has several modes of
operation, depending upon options and arguments.

>> zfs list -t all -o
>> name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots

Root-on-ZFS includes a hierarchy of nested filesystems. RTFM
zfs-list(8) you need the option "-r" (recursive) to see them all:

# zfs list -r -t all -o
name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots

On 9/18/25 10:39, Andrea Venturoli wrote:
> On 9/18/25 18:44, fre...@vanderzwan.org wrote:
>

>>>> zroot/ROOT/default@auto_zroot-20250611020000 3.34G
>>>> 209M - - - -
>>>> zroot/ROOT/default@auto_zroot-20250711020000 62.2G
>>>> 407M - - - -
>>

>> Here the refer jumps from 3.34 to 62.2 GB, so it looks like your
>> missing data is in this snapshot.
>
> Hmmm...
>

>> # zfs list -o name,refer,used,usedbysnapshots|grep default

>> zroot/ROOT/default 62.1G 70.5G 8.36G
>
> usedbysnapshots is 8.36G, so I guess this single snapshot cannot take
60GB.

RTFM zfsprops(7):

usedbysnapshots The amount of space consumed by snapshots of
this dataset. In particular, it is the
amount of space that would be freed if all of
this dataset's snapshots were destroyed.
Note that this is not simply the sum of the
snapshots' used properties because space can
be shared by multiple snapshots.


AIUI 8.36G is the amount of space used by snapshots of the dataset
"zroot/ROOT/default" alone. It does not include the amount of space
used by snapshots of child datasets.

> Also "zfs send" doesn't transmit snapshots,

RTFM zfs-send(8):

zfs send [-DLPVRbcehnpvw] [[-I|-i] snapshot] snapshot
Creates a stream representation of the second snapshot, which is
written to standard output. The output can be redirected to a
file or to a different system (for example, using ssh(1)). By
default, a full stream is generated.


AIUI `zfs send ...` transmits snapshots.

> I tried deleting the 20250711020000 snapshot.

> <snip>

> Still "zfs send" generates the same huge amount of data.

Please post the console session. My guess is that you destroyed the
snapshot in the dataset "zroot/ROOT/default", but that you did not
destroy the snapshots in the child datasets.


David

[David Christensen]

On 9/18/25 17:53, David Christensen wrote:
> > I tried deleting the 20250711020000 snapshot.
> > <snip>
> > Still "zfs send" generates the same huge amount of data.
>
>
> Please post the console session.  My guess is that you destroyed the
> snapshot in the dataset "zroot/ROOT/default", but that you did not
> destroy the snapshots in the child datasets.

That guess is wrong -- zroot/ROOT/default should not have child datasets:

https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot#Create_the_ZFS_file_system_hierarchy


The console sessions for `zfs destroy ...` and `zfs send ...` should
provide the needed facts.


David

[Andrea Venturoli]

On 9/18/25 21:50, fre...@vanderzwan.org wrote:

> One thing that could explain this difference between du and zfs used is if you mount a small/empty filesystem on top of a very large directory.
> In that case the content of the directory would be invisible because
the mount masks it.

Doh!
I hadn't thought about this! That's quite obvious and it even happened
to me a few times in the past!

Following a suggestion on a forum, I tried:
> mkdir /mnt/test
> mount -t nullfs / /mnt/test
but what I see under /mnt/test is identical to what I see under /.

The suggestion above was for UFS. I guess it works with ZFS too. Doesn't it?

> Does the output of the mount command show any strange mounts that could cause this ?

Nothing.

> About the du output what does ‘du -mx / |sort -n |tail ‘ show ?

> # du -mx / |sort -n |tail
> 238 /usr/local/lib/python3.11
> 442 /usr/lib/debug/boot
> 539 /usr/local/lib
> 567 /usr/lib/debug/usr/bin
> 758 /usr/lib/debug/usr
> 1229 /usr/local
> 1235 /usr/lib/debug
> 1371 /usr/lib
> 3051 /usr
> 3394 /

That's coherent with my earlier du output.

bye & Thanks
av.

[Andrea Venturoli]

On 9/18/25 23:22, David Christensen wrote:

> ...

> If your system has been upgraded many times, you will have many
> snapshots.  These snapshots will consume space.  That is why the
> "referenced" values reported by zfs(8) are larger than the usage values
> reported by du(1).

All agreed and known up to here.

> To do a full backup, you need to send the entire dataset hierarchy and
> all of its snapshots someplace -- a file, another pool, etc..  This is
> known as "replication".

Agree again, but I'm not doing "replication".
Replication is done with "zfs send -R"; I'm using "zfs send -p" with a
single snapshot I created.

> will be tedious and error prone.  I have written a few generations of
> scripts to automate this chore, first in Bourne shell and later in Perl.

I've got my own scripts :)



bye & Thanks
av.

[Andrea Venturoli]

On 9/19/25 02:53, David Christensen wrote:

> Please post the console session.

> # cd /tmp/
> zfs snap zroot/ROOT/default@dump
> zfs send -v -p zroot/ROOT/default@dump>a
> full send of zroot/ROOT/default@dump estimated size is 96.8G
> total estimated size is 96.8G
> TIME SENT SNAPSHOT zroot/ROOT/default@dump
> 08:45:27 114M zroot/ROOT/default@dump
> 08:45:28 180M zroot/ROOT/default@dump
> 08:45:29 298M zroot/ROOT/default@dump
> 08:45:30 459M zroot/ROOT/default@dump
> ...> 09:09:27 96.8G zroot/ROOT/default@dump
> 09:09:28 96.9G zroot/ROOT/default@dump
> 09:09:29 96.9G zroot/ROOT/default@dump
> 09:09:30 96.9G zroot/ROOT/default@dump
> #

> My guess is that you destroyed the
> snapshot in the dataset "zroot/ROOT/default", but that you did not
> destroy the snapshots in the child datasets.

Again, I'm not "zfs send"ing child datasets (I'm not using "-R").



bye & Thanks
av.

[fre...@vanderzwan.org]

> On 19 Sep 2025, at 09:22, Andrea Venturoli <m...@netfence.it> wrote:
>
> On 9/18/25 21:50, fre...@vanderzwan.org wrote:
>
>> One thing that could explain this difference between du and zfs used is if you mount a small/empty filesystem on top of a very large directory.
> > In that case the content of the directory would be invisible because the mount masks it.
>
> Doh!
> I hadn't thought about this! That's quite obvious and it even happened to me a few times in the past!
>
> Following a suggestion on a forum, I tried:
>> mkdir /mnt/test
>> mount -t nullfs / /mnt/test
> but what I see under /mnt/test is identical to what I see under /.
>
> The suggestion above was for UFS. I guess it works with ZFS too. Doesn't it?
>
>

Yes it's the way mounts work independent of the filesystem type ( except I guess for unionfs ).

>
>> Does the output of the mount command show any strange mounts that could cause this ?
>
> Nothing.
>

Strange. I'm afraid that was the last thing I could think of.

>
>
>> About the du output what does ‘du -mx / |sort -n |tail ‘ show ?
>
>> # du -mx / |sort -n |tail
>> 238 /usr/local/lib/python3.11
>> 442 /usr/lib/debug/boot
>> 539 /usr/local/lib
>> 567 /usr/lib/debug/usr/bin
>> 758 /usr/lib/debug/usr
>> 1229 /usr/local
>> 1235 /usr/lib/debug
>> 1371 /usr/lib
>> 3051 /usr
>> 3394 /
>
> That's coherent with my earlier du output.
>

Agree. So that leaves us the mystery why zfs says the dataset is using 60GB ;-(
With what we know now I have no idea why.


Cheers,
Paul

[fre...@vanderzwan.org]

On 19 Sep 2025, at 09:22, Andrea Venturoli <m...@netfence.it> wrote:

What you could do is to find largest objects in the dataset using zdb:

zdb -ddd zroot/ROOT/default 0:-1:A |sort -h -b -k 5 |tail

that should show you the largest objects in the dataset.

First column is the object id which is also the inode number.

You can find more info on the objects using:

zdb -ddd zroot/ROOT/default OBJECTID

You can try to find the name using :

find / -xdev -inum OBJECTID

But if du cannot see it I doubt find can find it.

But this would at least show you if there are large objects in the dataset using lot of space.

Cheers,

Paul

[Andrea Venturoli]

On 9/19/25 12:08, fre...@vanderzwan.org wrote:

> What you could do is to find largest objects in the dataset using zdb:
> zdb -ddd zroot/ROOT/default 0:-1:A |sort -h -b -k 5 |tail
>
> that should show you the largest objects in the dataset.
> First column is the object id which is also the inode number.
>
> You can find more info on the objects using:
> zdb -ddd zroot/ROOT/default OBJECTID
>
> You can try to find the name using :
> find / -xdev -inum OBJECTID

Here:

> # zdb -ddd zroot/ROOT/default 0:-1:A |sort -h -b -k 5 |tail
> 186544 2 128K 128K 31.3M 512 57.5M 100.00 ZFS plain file
> 5738 2 128K 128K 33.2M 512 51.9M 100.00 ZFS plain file
> 0 6 128K 16K 39.6M 512 176M 29.37 DMU dnode
> 186545 2 128K 128K 48.4M 512 93.5M 99.60 ZFS plain file
> 186426 2 128K 128K 53.1M 512 101M 100.00 ZFS plain file
> 186668 3 128K 128K 76.2M 512 178M 100.00 ZFS plain file
> 186209 3 128K 128K 91.9M 512 253M 100.00 ZFS plain file
> 186671 3 128K 128K 135M 512 327M 100.00 ZFS plain file
> 186427 3 128K 128K 140M 512 334M 100.00 ZFS plain file
> 360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file
> # zdb -ddd zroot/ROOT/default 360
> Dataset zroot/ROOT/default [ZPL], ID 89, cr_txg 8, 62.1G, 105870 objects
>
> Object lvl iblk dblk dsize dnsize lsize %full type
> 360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file
>
> # find / -xdev -inum 360
> #

Does this make any sense to you?



Can it be that a process has opened a huge file, which was later
deleted, but is still there due to the process being still alive?
Would du or the abobe find show this?
Any way to track either the file or the process?
(Just an hypotesis of course).

bye & Thanks
av.

[fre...@vanderzwan.org]

That's a 90GB file taking up 58.8 GB of space. Definitely a candidate for your excessive disk usage.

>
> Can it be that a process has opened a huge file, which was later deleted, but is still there due to the process being still alive?
> Would du or the abobe find show this?
> Any way to track either the file or the process?
> (Just an hypotesis of course).
>

A reboot would kill any process and release the space. But it would not solve the riddle.
You could try lsof, I think the NODE column refers to the inode/object id if the filedescriptor is connected to a file on disk (VREG in the TYPE column)..
Run lsof as root and grep for 360.

Cheers,
Paul

[Andrea Venturoli]

On 9/19/25 12:59, fre...@vanderzwan.org wrote:

> A reboot would kill any process and release the space.

That's one thing I was planning to do, though rebooting a server 80km
away is always something I'm not at ease with.
I'll have to arrange with the customer and probably it will be a few days.

> But it would not solve the riddle.

Also true.

> You could try lsof, I think the NODE column refers to the inode/object id if the filedescriptor is connected to a file on disk (VREG in the TYPE column)..
> Run lsof as root and grep for 360.

No luck here :(
Also tried fstat.



bye & Thanks
av.

[fre...@vanderzwan.org]

Maybe procstat -af and look for a regular file with a huge value in OFFSET column. That assumes the file offset is at EOF for that huge file.

Cheers,
Paul

[David Christensen]

On 9/18/25 03:10, Andrea Venturoli wrote:

> # cd /
> # du -d 0 -h -x
> 3.3G .

> ...

> root@chet:/ # zfs list|grep ROOT
> zroot/ROOT 70.9G 1.94T 88K none
> zroot/ROOT/default 70.9G 1.94T 62.1G /

On 9/18/25 08:57, Andrea Venturoli wrote:

> # zfs list -t all -o
name,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots|grep
default
> zroot/ROOT/default 62.1G
> 70.9G 0B 62.1G 0B 8.76G

> <snip>

> zroot/ROOT/default@auto_zroot-20250611020000 3.34G
> 209M - - - -
> zroot/ROOT/default@auto_zroot-20250711020000 62.2G
> 407M - - - -

On 9/19/25 00:23, Andrea Venturoli wrote:
> # cd /tmp/
> zfs snap zroot/ROOT/default@dump
> zfs send -v -p zroot/ROOT/default@dump>a
> full send of zroot/ROOT/default@dump estimated size is 96.8G
> total estimated size is 96.8G
> TIME        SENT   SNAPSHOT zroot/ROOT/default@dump
> 08:45:27    114M   zroot/ROOT/default@dump
> 08:45:28    180M   zroot/ROOT/default@dump
> 08:45:29    298M   zroot/ROOT/default@dump
> 08:45:30    459M   zroot/ROOT/default@dump
> ...
> 09:09:27   96.8G   zroot/ROOT/default@dump
> 09:09:28   96.9G   zroot/ROOT/default@dump
> 09:09:29   96.9G   zroot/ROOT/default@dump
> 09:09:30   96.9G   zroot/ROOT/default@dump
> #

> <snip>

> Again, I'm not "zfs send"ing child datasets (I'm not using "-R").

Thank you for that information. Sorry for my confusion and noise.

On 9/19/25 03:59, fre...@vanderzwan.org wrote:
> # zdb -ddd zroot/ROOT/default 0:-1:A |sort -h -b -k 5 |tail

> <snip>

> 360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS
plain file

It looks like you and Paul have found a good clue, but have not been
able to identify the file or process (?). This is beyond my knowledge,
so I will refrain from making more noise.


I agree with the suggestion to reboot. As the server is remote: if a
remote reboot fails and you must visit the site, you might want to bring
a remote console solution with you and install it.


David

[Andrea Venturoli]

On 9/19/25 15:28, fre...@vanderzwan.org wrote:

> Maybe procstat -af and look for a regular file with a huge value in OFFSET column. That assumes the file offset is at EOF for that huge file.

Didn't help.

I finally had the chance to reboot, but not even this solved!!! :(

I still get the same figures:

> # du -d 0 -h -x
> 3.3G .

> # zdb -ddd zroot/ROOT/default 0:-1:A |sort -h -b -k 5 |tail
> 186544 2 128K 128K 31.3M 512 57.5M 100.00 ZFS plain file
> 5738 2 128K 128K 33.2M 512 51.9M 100.00 ZFS plain file

> 0 6 128K 16K 39.5M 512 176M 29.37 DMU dnode

> 186545 2 128K 128K 48.4M 512 93.5M 99.60 ZFS plain file
> 186426 2 128K 128K 53.1M 512 101M 100.00 ZFS plain file
> 186668 3 128K 128K 76.2M 512 178M 100.00 ZFS plain file
> 186209 3 128K 128K 91.9M 512 253M 100.00 ZFS plain file
> 186671 3 128K 128K 135M 512 327M 100.00 ZFS plain file
> 186427 3 128K 128K 140M 512 334M 100.00 ZFS plain file
> 360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file

bye & Thanks
av.

[David Christensen]

Have you done a scrub recently?


I had the idea to use diff(1) to compare the snapshots when the size
jumped, but did not post it because if du(1) cannot see problem file(s)
then diff(1) should not:

# diff -r /.zfs/snapshot/auto_zroot-20250611020000
/.zfs/snapshot/auto_zroot-20250711020000


An improved idea that might work would be to use zfs-diff(8) to compare
the snapshots. Consider adding option "-H" and redirecting the output
to a file for further analysis, issue tracking, etc.:

# zfs diff @auto_zroot-20250611020000
zroot/ROOT/default@auto_zroot-20250711020000


Perhaps booting into single user mode, doing a scrub, and investigating/
trouble-shooting?


Perhaps booting live media, doing a scrub, and investigating/
trouble-shooting??


Do you have a saved zfs-send(8) backup stream or raw disk image from
prior to the issue that you can restore?


If all else fails, backup/ wipe/ install/ restore the OS disk. I prefer
the oldest supported version of an OS (e.g. FreeBSD-13.5-RELEASE), as it
should have fewer bugs than newer versions of an OS.


David

[fre...@vanderzwan.org]

> On 21 Sep 2025, at 19:07, Andrea Venturoli <m...@netfence.it> wrote:
>
> On 9/19/25 15:28, fre...@vanderzwan.org wrote:
>
>> Maybe procstat -af and look for a regular file with a huge value in OFFSET column. That assumes the file offset is at EOF for that huge file.
>
> Didn't help.
>
> I finally had the chance to reboot, but not even this solved!!! :(
>

So we know it’s not a deleted but open file. That means it must have a name…

> I still get the same figures:
>> # du -d 0 -h -x
>> 3.3G .
>> # zdb -ddd zroot/ROOT/default 0:-1:A |sort -h -b -k 5 |tail
>> 186544 2 128K 128K 31.3M 512 57.5M 100.00 ZFS plain file
>> 5738 2 128K 128K 33.2M 512 51.9M 100.00 ZFS plain file
>> 0 6 128K 16K 39.5M 512 176M 29.37 DMU dnode
>> 186545 2 128K 128K 48.4M 512 93.5M 99.60 ZFS plain file
>> 186426 2 128K 128K 53.1M 512 101M 100.00 ZFS plain file
>> 186668 3 128K 128K 76.2M 512 178M 100.00 ZFS plain file
>> 186209 3 128K 128K 91.9M 512 253M 100.00 ZFS plain file
>> 186671 3 128K 128K 135M 512 327M 100.00 ZFS plain file
>> 186427 3 128K 128K 140M 512 334M 100.00 ZFS plain file
>> 360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file
>
>

Can you try ‘zdb -dddd zroot/ROOT/default 360’ ?
( so 4 d’s )

If I try that on a normal file in my homedir it gives me the ’stat’ information, and even the filename.
Even if it does not give a name the owner/group and the timestamps might give a hint where to find it.
It will also give a list of the indirect blocks which will be huge but the interresting output should be at the start.

Cheers,
Paul

[fre...@vanderzwan.org]

On 22 Sep 2025, at 07:12, fre...@vanderzwan.org wrote:

You need zdb with 5 d’s to get the name. So 'zdb -ddddd zroot/ROOT/default  360’

Cheers

 Paul

[Andrea Venturoli]

On 9/22/25 07:15, fre...@vanderzwan.org wrote:
>
>
>> On 22 Sep 2025, at 07:12, fre...@vanderzwan.org wrote:
>>
>>
>
> You need zdb with 5 d’s to get the name. So 'zdb -ddddd zroot/ROOT/
> default  360’

Here it is:

> Dataset zroot/ROOT/default [ZPL], ID 89, cr_txg 8, 62.1G, 105865 objects, rootbp DVA[0]=<0:a861f85000:1000> DVA[1]=<0:311e6461000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=800L/800P birth=>

>
> Object lvl iblk dblk dsize dnsize lsize %full type

> 360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file

> 168 bonus System attributes
> dnode flags: USED_BYTES USERUSED_ACCOUNTED
> dnode maxblkid: 737498
> path on delete queue
> uid 0
> gid 0
> atime Wed Jun 25 14:50:38 2025
> mtime Wed Jun 25 15:02:31 2025
> ctime Wed Jun 25 15:02:31 2025
> crtime Wed Jun 25 14:50:38 2025
> gen 35226387
> mode 100644
> size 96665427448
> parent 87
> links 0
> pflags 40800000004
> Indirect blocks:
> 0 L2 0:ad0c152000:9000 20000L/9000P F=737499 B=35226471/35226471 cksum=00000cf77c955f23:00f9a3345e9c1658:c3f286e3bc2c2a99:42b808e738c1f99e
> 0 L1 0:ad0042d000:b000 20000L/b000P F=1024 B=35226388/35226388 cksum=00001115fae7065c:019128fbafb5a2bc:0701bd8b4bd05900:3417736fdef1ad6c
> 0 L0 0:19007486000:1c000 20000L/1c000P F=1 B=35226387/35226387 cksum=00002fe112352d05:0a003c91dad6a92b:ddd312e24d14537f:b7df15b3fff44a4c
> 20000 L0 0:19007872000:8000 20000L/8000P F=1 B=35226387/35226387 cksum=00001019b66eb88a:0133b13699ec22cf:cc2e684318f9738e:57bbf713d899e173
> 40000 L0 0:1900e2d9000:2000 20000L/2000P F=1 B=35226387/35226387 cksum=00000160676ba5c7:00078f59aec453a5:165bf3de05b5543e:8c0213aacf6abbe2
> 60000 L0 0:190075fb000:c000 20000L/c000P F=1 B=35226387/35226387 cksum=00001515331d9638:01f977317121e464:b07853739191bafd:85837809e155832f
> 80000 L0 0:190285a7000:20000 20000L/20000P F=1 B=35226387/35226387 cksum=00003ec18e9e1e3e:0fad53262d5c9c67:a0632107020a024c:6b5dffbf9d374376
>...
> 1681b20000 L0 0:28bc1027000:20000 20000L/20000P F=1 B=35226471/35226471 cksum=00003e64c67aad30:0f930395d4c3b7f8:eaff9bb33545afd3:12c9a6423c2b5b5a
> 1681b40000 L0 0:28b9df38000:d000 20000L/d000P F=1 B=35226471/35226471 cksum=0000175a6f6b02e3:029bcd30daa5d07b:edd314a460c8181d:5b6ad7fe80996b0a
>
> segment [0000000000000000, 0000001681b60000) size 90.0G


No filename unfortunately, but "path on delete queue" gives an hint!
I think I've read some thread on "delete queue" not being worked on...
I'll have to search again.


bye & Thanks
av.

[fre...@vanderzwan.org]

The delete queue should be gone when pool is re-mounted after reboot, so that's weird.
I have no idea if/how you can try to force it, or if it's a matter of more patience..

Good luck ;-)

Paul

[Andrea Venturoli]

On 9/21/25 23:21, David Christensen wrote:

> Have you done a scrub recently?

The pool was last scubbed by periodic a month ago.
I've launched a new scrub manually now, but I guess it won't do much,
given the problem arised long ago.

> I had the idea to use diff(1) to compare the snapshots when the size
> jumped, but did not post it because if du(1) cannot see problem file(s)
> then diff(1) should not:
>
> # diff -r /.zfs/snapshot/auto_zroot-20250611020000 /.zfs/snapshot/
> auto_zroot-20250711020000

Didn't try, but tried a find on /.zfs/snapshot/ and the file doesn't
come up.

> An improved idea that might work would be to use zfs-diff(8) to compare
> the snapshots.  Consider adding option "-H" and redirecting the output
> to a file for further analysis, issue tracking, etc.:
>
> # zfs diff @auto_zroot-20250611020000 zroot/ROOT/
> default@auto_zroot-20250711020000

I already deleted those snapshots, remember?

I wonder if deleting all the others would solve...

> Perhaps booting into single user mode, doing a scrub, and investigating/
> trouble-shooting?

I'll be able to put my hands on the box in 1-2 weeks and I was already
thinking about trying single user mode.
A scrub in SU, however, is out of the question, as it would take several
hours and this is a production server.

> Do you have a saved zfs-send(8) backup stream or raw disk image from
> prior to the issue that you can restore?

Probably not: it would be a very old version of the system and it would
be more trouble than now.

> If all else fails, backup/ wipe/ install/ restore the OS disk.  I prefer
> the oldest supported version of an OS (e.g. FreeBSD-13.5-RELEASE), as it
> should have fewer bugs than newer versions of an OS.

Again, this is a production server, so this is close to impossible.
The problem is not that severe right now: the most disturbing thing is
that I cannot do a "dump" and send it over the Internet, because it
would be too big for a slow line.

Eventually I was thinking about creating a different ROOT dataset,
rsyncyng everything to it and switching to it (although I'd be curious
to sort this out, i.e. find a solution, not a workaround).



I'm also planning the upgrade from 14.2 to 14.3... who knows if it could
help?



bye & Thanks
av.

[Andrea Venturoli]

On 9/23/25 16:14, Andrea Venturoli wrote:
> On 9/21/25 23:21, David Christensen wrote:
>
>> Have you done a scrub recently?
>
> The pool was last scubbed by periodic a month ago.
> I've launched a new scrub manually now, but I guess it won't do much,
> given the problem arised long ago.

As expected, scrub recovered no space.

> I already deleted those snapshots, remember?
>
> I wonder if deleting all the others would solve...

Deleting *all* snapshots didn't help either.



bye
av.