upspin-audit tool

[Andrew Gerrand]

Hi Upspinners,

We recently added an upspin-audit tool to the core repo (and binary distributions) that allows server administrators to find and delete store blocks that are not referred to by a specific set of Upspin trees. In other words, a kind of garbage collection.

The documentation explains how to use it.

We encourage the use of the -backup flag when running 'upspin audit delete-garbage', so that you can recover your data in case you make a mistake.

Note that for this to work you need to be running a recent upspinserver that supports storage.Lister. At present, that is the local disk, Google Cloud Storage, OpenStack, and (soon) B2 storage implementations. So please upgrade your upspinserver before trying upspin-audit.

Cheers,

Andrew

[Frank Rehwinkel]

Used this to reclaim some local disk space used by an upbox system. It worked, but it took me a few minutes to figure out how to get the storage info to the two sub commands that are storeserver related.

Am testing with upboxes on various VMs and would like to reclaim the garbage from the storeserver storage on each. I don't use these VMs in a way where I could just roll back their image.

One upspin-audit storage subcommand accepts an -endpoint argument to get to the storeserver, the second one does not.

upbox creates the storeserver config but omits the actual storeserver endpoint in this file, leaves it at 'inprocess'. A few manual steps needed to make things copacetic. Don't know how to script the change needed to the config file using upspin tools. Maybe good old sed.

[Frank Rehwinkel]

Also interesting that if your script deletes all files and directories after testing, the scan-dir step does not create a file and thus the find-garbage step fails saying the scan-dir step needs to be run first. Not a problem if you know about it. Create a single empty file after deleting everything gets everything back on track or if you purposely removed everything, just removing the storage directory reclaims the space too. Also, not a real world problem where garbage collection is being applied to a user's real data in the cloud.