In Chronopolis, we perform a manual restore on any files that ACE
detects as corrupt. We maintain three copies of all data at three
different sites (UMD,SDSC,NCAR) and at UMD perform local, nightly
backups using IBM's TSM product.
In the event of corruption, we do the folling at UMD:
1. Re-run the audit to verify that the file is indeed corrupt and it's
not an intermittent failure.
2. Attempt to restore from local backup if the file is indeed corrupt
3. If local restore fails, we'll copy from one of the peer sites
4. Re-run ACE against the failed files and test to see if they are no
longer corrupt.
-Mike