Recommend file amount or file size for AIPs

Nicole Critchley

unread,

Oct 14, 2020, 4:11:48 PM10/14/20

to archivematica

Are there any best practices for the amount of files or size for a single AIP?

Context: I have a group of 3382 TIF files (362 GB) that keeps timing out at the compression stage. These files are one part of a large photographic negative collection and I'd rather not break it apart. But will if needed.

How big is everyone making their AIPs? What is your institutional standard or policy?

Cheers,

Nicole Critchley

Asst. Archivist

University Archives, Texas State University

Sean Kalynuk

unread,

Oct 14, 2020, 4:50:38 PM10/14/20

to archiv...@googlegroups.com

Hi Nicole,

We have tested the creation of single AIPs greater than 300GB in size, and also AIPs that were smaller in overall size (~2GB), but had over 14,000 files. They were created using “7zip with bzip2” and “5 – normal compression” on a four-core VM with 16GB of memory and 1TB of space for /var/archivematica. I don’t think we’ve tested something that was both >300GB and with thousands of files though.

You might try changing the compression level to “1 – fastest compression”:

https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/administer/dashboard-admin/#select-compression-level

Alternatively, you might want to try a different compression algorithm:

https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/administer/dashboard-admin/#select-compression-algorithm

Be aware of one known issue though:

https://github.com/archivematica/Issues/issues/917

The issue also has some info on the different compression methods.

--

Sean Kalynuk, M.Sc.

Technical Specialist/Developer

Elizabeth Dafoe Library

University of Manitoba

From: <archiv...@googlegroups.com> on behalf of Nicole Critchley <ne...@txstate.edu>
Reply-To: "archiv...@googlegroups.com" <archiv...@googlegroups.com>
Date: Wednesday, October 14, 2020 at 3:11 PM
To: archivematica <archiv...@googlegroups.com>
Subject: [archivematica] Recommend file amount or file size for AIPs

Caution: This message was sent from outside the University of Manitoba.

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/a0ca6b27-6816-4c7f-a3d8-47e1bb305fb1n%40googlegroups.com.

Sarah Romkey

unread,

Oct 15, 2020, 9:04:39 AM10/15/20

to archiv...@googlegroups.com

Just a note also that your "mileage may vary" depending on your hardware set-up and configurations. You'll find some helpful info here: https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/installation-setup/customization/scaling-archivematica/#scaling-archivematica

and https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/installation-setup/installation/installation/#recommended-minimum-production-requirements

Cheers,

Sarah

Sarah Romkey, MAS,MLIS

Archivematica Program Manager

She/her
Artefactual Systems
604-527-2056

@archivematica / @accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/88CCBF6D-63CF-41C7-8D4F-879CD4F2CE63%40umanitoba.ca.

Nicole Critchley

unread,

Dec 14, 2020, 9:29:13 AM12/14/20

to archivematica

To follow up, we have successfully created our AIP. While we were already on a low compression level, I changed the compression level to “1 – fastest compression.” And we ended up to having to change our wait times as well as add more RAM and memory.