Compression of E01 files

[Dorothy Waugh]

Hi there,

We're in the process of migrating our reposited AFF files to E01, which has raised anew some questions around compression. 

Is anyone else choosing to (or, just as significantly, choosing not to) store compressed E01 files as preservation copies? I'm interested in different institutional policy on why you do or do not choose to store compressed E01 files and, if you do, how you select the level of compression. My instinct has always been to store uncompressed files so as to keep preservation copies of data as simple as possible--however, we're being encouraged to consider compression as it would make some of our larger, more unwieldy files more manageable. Are my concerns about applying compression algorithms to preservation copies of files unwarranted?

I would really appreciate your thoughts! Thank you so much!

Dorothy Waugh

Digital Archivist

Stuart A. Rose Manuscript, Archives, and Rare Book Library, Emory University

[Bertram Lyons]

Dorothy --

E01 files use an open compression method (zlib/deflate) to compress both the E01 metadata and the blocks of image bytes stored within the E01 container. This is a lossless (reversible) compression method that does not in itself cause loss of data. And the method to decompress is widely known and possible with many open source libraries. However, the danger in storing compressed data is when (if) any of the stored bytes are corrupted it will compound the damage done to the compressed data. In this case, the danger of storing compressed data is less about the difficulty of decompression and more about the possibility of multiplying damage if compressed bytes are corrupted.

A risk/cost analysis would likely be the best approach here. How much data are you taking about? What would be the cost-benefit analysis of saving storage space versus (small) potential that compressed bytes get corrupted? 

I am interested to hear how you proceed.

Thanks, and good luck!

- Bert

______________________________________
 
Bertram Lyons, CA
AVPreserve
400 Interlake Drive
Madison, Wisconsin 53716
 
office: 202-430-4457

http://www.avpreserve.com
Facebook.com/AVPreserve
twitter.com/AVPreserve

--
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-curati...@googlegroups.com.
To post to this group, send email to digital-...@googlegroups.com.
Visit this group at https://groups.google.com/group/digital-curation.
For more options, visit https://groups.google.com/d/optout.