du reports far less usage than data on disk

[Guan Xin]

Hi All,

We have recently set up a beegfs-7.3.2 cluster on openzfs-2.1.9 with zstd compression enabled.

After storing some 100TB large files that are compressible by about 1/3, df reports correctly that 67TB of disk space has been used, and du -s --apparent-size shows the correct amount of data before compression.

However, du -s without --apparent-size reports far less data, something between 20 and 30TB. du on single large files mostly reports between 4k and 1GB of disk usage, while the actually data gone to the hard disks under zfs should be around 1.4GB for each file.

Is this a bug or a feature?

Greetings,

Guan

[Lorenzo BARALDI]

Dear Guan,

have you maybe found a solution to this?

We are encountering the same problem you were reporting. It seems that there are chunks which are not assigned to any file, and beegfs-fsck does not discover them.

Any help will be appreciated!

Lorenzo.

[Fuzzy Rogers]

So - looks like my metadata server filled up its inode database. At least, thats my current theory. Logs says “no space left on device” , when, in fact, there is plenty of space left on device.

We’ve got backups of the filesystem on good old trusty ZFS, so I don’t really mind nuking users that haven’t been on in ages. My thought is to simply clear files, opening up inodes.

Does that seem reasonable? I’m not going to tear down and rebuild my metadata server (though we probably should, eventually). Total space is about 750TB.

Thanks-
Fuzzy

[Guan Xin]

Dear Lorenzo,

I have updated that filesystem from beegfs 7.3.2 to 7.3.4,

and the behavior stays the same.

This is surely something easy to fix because du and du --apparent-size of the underlying zfs reports the correct sizes.

However, I have no time to investigated into this minor bug at present, and the non-free license of beegfs discourages any contribution from users, so I've put this to the lowest priority.

Guan