/proj quota not released after ~50 GB / ~123k file deletions (project a2234, Utah)

[Jinchun He]

All writes to the project share /proj/a2234-PG0 fail with Disk quota exceeded, including a 1-byte file, and the condition does not recover after we delete large amounts of data. Reading and SSH are unaffected. We deleted ~123,000 files / ~50 GB and the share still reports Free 0 blocks and Free 0 inodes.

The core problem: deletions do not restore free space

[Mike Hibler]

Both of the Utah clusters (Cloudlab Utah, Emulab) create daily snapshots of
filesystems in /proj that last for a couple of days. These snapshots count
against the filesystem quota, so even though you deleted some files, those
files were still present in the snapshot. I have adjusted your quota to
account for this.

Not being able to delete a file in ZFS (the filesystem underlying /proj)
when the quota is reached is a "feature". Deleteing a file requires additional
metadata. You can first `truncate` a file (releasing the contents) and then
delete it.

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 066c2d6b-c46c-461a-990f-10b7f4bdeaacn%40googlegroups.com.