bogus Disk quota exceeded on chown

[Dr. Thomas Orgis]

Hi,

having settled the big crash issue, I now face a select group of users
(6 out of 264) triggering a quota error when root chowns a file to them:

# no issue if the user creates a file …
user@host$ echo Hello > file.1

# But when root hands one over …
root@host# echo Hello > file.1
root@host# chown user file.1
chown: changing ownership of ‘test.1’: Disk quota exceeded


When I check the quota, there is lots of free space. We set
--sizelimit, no --inodelimit. There really is lots of space free in the
quota, and the user may fill it without issues … just root cannot hand
over files.

Is this a known bug in beegfs? A workaround? Setting the quota again
did not change the state. Also, our XFS partitions below the
beegfs-storage got their quota checks after xfs_repair on the next
mount.


Alrighty then,

Thomas

--
Dr. Thomas Orgis
HPC @ Universität Hamburg

[John DeSantis]

Thomas,

> Is this a known bug in beegfs? A workaround? Setting the quota again
> did not change the state. Also, our XFS partitions below the
> beegfs-storage got their quota checks after xfs_repair on the next
> mount.

We ran into this issue, too.  Unfortunately, this let us to perform a file system check which corrupted it: https://groups.google.com/g/fhgfs-user/c/UTmqk3uDMfE/m/e5opgUcJAQAJ

The only work-around was a restart of the metadata service(s) on the node(s) that reported the "error". 

John DeSantis

[Dr. Thomas Orgis]

Am Tue, 9 Nov 2021 11:38:27 -0800 (PST)
schrieb John DeSantis <fusion...@gmail.com>:

> We ran into this issue, too. Unfortunately, this let us to perform a file
> system check which corrupted it:
> https://groups.google.com/g/fhgfs-user/c/UTmqk3uDMfE/m/e5opgUcJAQAJ

What you have there looks a lot more serious. You say you had my issue
before the check and now the bigger mess?

> The only work-around was a restart of the metadata service(s) on the
> node(s) that reported the "error".

Hm. Indeed, a restart of the metadata server proces fixes the bogus
chown error and only those users remain who really are over quota.
Thanks for the hint!

Now it's of course a bit unfortunate that such bad info can get stuck
in beegfs-meta. This user wasn't even near the quota in the recent
past. Unsettling.

[John DeSantis]

Thomas,

> What you have there looks a lot more serious. You say you had my issue
> before the check and now the bigger mess?

That's right. What started out as incorrect quota reporting led me to perform the ill-fated beegfs-fsck.

> Hm. Indeed, a restart of the metadata server proces fixes the bogus
> chown error and only those users remain who really are over quota.
> Thanks for the hint!

No problem at all! I'd hate to see another site go down the same path we did.

> Now it's of course a bit unfortunate that such bad info can get stuck
> in beegfs-meta. This user wasn't even near the quota in the recent
> past. Unsettling.

Agreed. That was the exact case with our production deployment, too.

John DeSantis

On Wednesday, November 10, 2021 at 3:31:22 AM UTC-5 Dr. Thomas Orgis wrote:

Am Tue, 9 Nov 2021 11:38:27 -0800 (PST)

schrieb John DeSantis <>: