Missing Data After Storage Quota Recovery

[Fang]

Hi CloudLab admins,

I previously contacted the admins regarding a storage quota issue, but I am still experiencing problems with missing data.

It seems that my storage quota was exceeded, and unfortunately most of the files under my project directory disappeared afterward.

Would it be possible to help rollback or recover the storage state if there are any available snapshots or backups?

More information:
- Project directory: /proj/gbusys0-PG0/
- Region: Utah
- Last experiment name: fzh0u-303805
- Initial issue report date: 5/14/2026
- More details: The disk capacity has since been restored, but the files and experiment data are still missing.

These data are important for my ongoing experiments, so I would greatly appreciate any help or suggestions.

Thank you very much!

Best,
Fang

[Mike Hibler]

Where under /proj/gbusys0-PG0/ was the data stored? The only place our software
would have removed it, is if you put it under /proj/gbusys0-PG0/exp. There is
a directory for every active experiment under there and we remove those
directories when an experiment terminates. You should not put files into or
remove the pre-existing directories; deltas, exp, groups, images, logs, rpms,
tarfiles, templates, or tiplogs. You should always create your own directories
under /proj/<pid>.

Running out of disk space will not remove any files, it will just cause writes
from applications to fail with an "out of space" error.

Keep in mind that all active experiments in the project have access to the
/proj dir and anyone on any experiment node could use "sudo" and remove any
files from your /proj directory. So if someone else noticed the out of space
condition, maybe they tried to clean up and make space.

I did point you to the .zfs/snapshot directory which has two days worth of
snapshots (on Utah cluster only). Did you try looking there?

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 9e947c53-544f-4823-b07e-1b616584d6e0n%40googlegroups.com.