Important Update: Enhancing Data Durability with Erasure Coding
Nathan Tallman
Dear APTrust Community,
I hope this message finds you well. We’re reaching out to share an important update on our data storage practices, reflecting our commitment to transparency and continual improvement.
What We’ve Learned
Over the past decade, we’ve communicated that our High Assurance storage class maintained six copies of each file—three “hot” in AWS S3 and three “cold” in AWS S3 Glacier. We recently discovered that AWS actually employs erasure coding for data durability, which differs from the replication approach we previously understood. As a result, our High Assurance storage includes one hot copy in S3 and one cold copy in S3 Glacier. Our other storage classes each maintain one copy.
Why Erasure Coding Matters
Erasure coding offers several advantages over traditional replication:
Higher Durability: It distributes data fragments across multiple locations, ensuring that the original data can still be reconstructed even if some fragments are lost or corrupted.
Efficient Storage: It uses storage space more efficiently, providing high durability with reduced overhead.
Improved Performance: Spreading data across various locations enhances data retrieval, leading to faster access and better fault tolerance.
Scalability: It adapts easily to growing storage needs without significant reconfiguration.
Our Commitment
We want to assure you that our dedication to preserving digital content remains as strong as ever. Erasure coding meets and exceeds our durability needs. We’ve updated our documentation and website (blog post, FAQ) to reflect this change and are here to address any questions you might have.
Join Us for a Webinar
We’re hosting a webinar on August 26 at 2 PM Eastern to discuss these changes in more detail and answer your questions. We encourage you to join us and participate in the Q&A session to gain a deeper understanding of how erasure coding enhances our preservation strategy.
Thank you for your continued support and engagement. We value your partnership as we advance digital preservation practices together.
Best regards,
Nathan
Feel free to reach out if you have any immediate questions or need further information. We look forward to connecting with you soon.