Non-user, looking to see if AtoM and Archivematica are right for us
183 views
Skip to first unread message
Gregory Pierce
Feb 13, 2020, 9:33:05 AM2/13/20
to AtoM Users
Hey everyone!
I'm an archivist with a background in metadata and digitization. My current gig has me putting feelers out for systems to preserve and display our content. We're a public library with a decent IT group, but we're looking to host our storage AND our software.
I'm curious to know what your workflow looks like? I'm wondering if Archivematica and AtoM working together could be our all-in-two solution. In the past, I've batch uploaded .tif files that are in a folder with their MODS and DC .xml files, and would like to know, first, if AtoM allows for that, or if there's a unique workflow that I should prepare for. I'll take any reviews, pros & cons, and/or workflows that anyone can offer!
Dan Gillean
Feb 13, 2020, 11:13:53 AM2/13/20
to ICA-AtoM Users
Hi Gregory,
I'm hoping that some of our community members will chime in and share their own experiences and perspectives, but in the meantime, I wanted to share some additional information that may help you evaluate whether or not AtoM and Archivematica might meet your needs.
General links and information
You can find all of the AtoM project documentation here:
We also have a detailed introductory information sheet on AtoM on the Artefactual website, here:
If you would like to explore some publicly accessible AtoM sites that are being used in production by other institutions, you might find our example Users list on the AtoM wiki useful:
Many of these sites have added their own custom themes - some developed by Artefactual, some created in-house - so this is also a great way to get a sense of how you can customize AtoM’s look and feel to match your institution’s home page. You might also find the following slide deck of interest - it outlines some community AtoM sites who are using AtoM in interesting ways. See:
You'll find a number of other slide decks from past AtoM Camps, conference presentations, or webinars, on our SlideShare account.
The easiest way to get a sense of what you can do with AtoM is to play around with it yourself - why not take a look at our public demo site? See:
The login credentials are listed right on the homepage. The site is loaded with sample data, and you are free to make whatever modifications you like - the site’s contents will automatically reset on an hourly basis. Please note that for security purposes, all uploads and imports have been disabled.
If you would like a way to set up your own test instance on a local computer or laptop, we do have a Vagrant box available, which can be installed with minimal technical knowledge. For more information, see:
AtoM, Archivematica, and batch uploads
If you are intending to use AtoM with Archivematica, then the process is a bit different than just uploading a folder of files and metadata.
Archivematica is a digital preservation workflow management tool that is closely aligned with the OAIS reference model. It's basically an AIP and DIP maker, if you are familiar with OAIS terminology - so you would generally run your master images through Archivematica as a transfer to generate an Archival Information Package (AIP) for long-term preservation. Please note that Archivematica is not a digital repository, though it can be used in conjunction with one - it is storage agnostic, and will send your content to wherever you tell it for long-term storage, but it is not intended to be a preservation storage solution on its own.
During the workflow you will also have the opportunity to generate access copies, in the form of a Dissemination Information Package (DIP). Archivematica has a format policy registry that can be customized by users to generate these access copies in whatever formats you prefer - but the general goal is to make something smaller and easier to view on the web for your access copies (so the default access derivative rule for TIFF images is likely JPG, for example). It is this DIP that you will send to AtoM via Archivematica.
You can include descriptive metadata in your transfer in 2 ways in Archivematica - either using a web form in Archivematica during processing to add general Dublin Core metadata for the entire transfer, or by including an accompanying CSV with additional descriptive metadata. See:
Depending on the workflow that you follow, your aggregate Dublin Core metadata will be transferred to AtoM as a parent level. Alternatively, you can also do some arrangement by fetching the levels of description from AtoM, and using them to assign a basic fonds/collection structure to the transfer before sending the DIP to AtoM, so that same structure will be recreated during the DIP upload. Some links:
Unfortunately, at this time, descriptive metadata included in an accompanying CSV is not passed to AtoM. It will be stored with your AIP in the accompanying METS file generated during processing (and will still be included as a CSV in the final AIP), but you would need to re-add it in AtoM.
AtoM does have the ability to handle a CSV import as an update to existing descriptions, however, so it would technically be possible to update your descriptions via import later. There are some gotchas with this functionality so it would take some practice to figure out a workflow that will work for your needs, but we can always help you with that via this forum if needed.
There are many ways we hope to improve the integration between Archivematica and AtoM in the future. Thus far, unless members of our community decide to sponsor work to improve the integration, it has been difficult for Artefactual to take on the development required to enhance the relationship between the two applications (mostly because we are a small company and we give away all our core work for free) without some kind of community support. You can read more about how we maintain and develop AtoM (and Archivematica) on the AtoM wiki, here:
However, we are also undergoing a lot of internal restructuring and trying to find ways to better harmonize our projects while maintaining our commitment to openness, and keeping our service offerings reasonably priced. I hope to see further work better integrating the two systems in the future.
If your organization is not yet at the point that digital preservation is a priority concern, it is of course possible to use AtoM without Archivematica, and you could upload TIFF files directly to AtoM. However, AtoM is an access system - it is also not a digital repository. I would recommend that you back up your master digital objects outside of the application, and you may save yourself some space and effort if you generate lower-res access copies outside of AtoM and upload those instead.
In these cases, you have many more options for bulk uploading both files and metadata, though the most effective of these for large/bulk transfers involve access to the command-line interface of the application. If you are intending to host the application locally, then this should not be an issue.
You can explore our import and export documentation here:
I encourage you to test out AtoM yourself, and see if it will meet your needs!
Cheers,
he / him
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/ae0ac38d-5dd5-4e4e-ab08-d74b839aec75%40googlegroups.com.
Creighton Barrett
Feb 14, 2020, 11:09:42 AM2/14/20
to ica-ato...@googlegroups.com
Hi Gregory,
As you can see, AtoM has the world's best open source program manager. There are lots of options out there, but only one comes with Dan Gillean :) You'll find the same level of support if you ping this question over to the Archivematica user forum.
I just wanted to add some additional perspective as a community member. Digitization workflows sometimes seem "lost" in conversations about digital preservation, Archivematica/AtoM integration, file normalization, DIPs, etc. So I'm hopeful that your question will prompt some good discussion!
We adopted AtoM in January 2015. And in October 2018, we started working with Artefactual Systems to configure a digital preservation system using Archivematica. The impact on our digitization workflow is an ongoing question for us.
Our digitization workflow assumes that items are partially or fully catalogued in our AtoM catalogue. In other words, we don't digitize unprocessed material. We use our box-folder "reference codes" as file prefixes. And we use those codes to help organize preservation masters and access copies in our storage environment. The reference code is pretty important to the whole process.
When we launched our AtoM site, we weren't using Archivematica. So we uploaded our access copies directly to AtoM, as Dan mentioned. But we don't upload TIFFs because we want to manage the TIFFs in a different environment. Instead, we create access copies during the digitization process, using Adobe tools (primarily Photoshop and Acrobat Pro, but we also use Bridge to do batch file renaming when necessary). Our policy is that OCR'd PDF/A files are the access copies for text and JPEGs are the access copies for graphic material. In both cases, preservation masters are TIFF files.
I mention this because, although Archivematica can certainly convert TIFFs to JPEGs, and it can also transcribe files with Tesseract, it doesn't provide an intuitive workflow for compiling a batch of TIFFs into a single PDF and transcribe the file. Sometimes, we need to create temporary mezzanine files to help reduce the size of the final access copy. With audivisual formats, we always ask our vendor to produce a preservation master and an access copy.
For us, all of this is much easier done outside of Archivematica or AtoM.
You can definitely transfer preservation masters and access copies into Archivematica and skip file normalization. See the Archivematica docs on "manual normalization": https://www.archivematica.org/en/docs/archivematica-1.10/user-manual/ingest/manual-normalization/
The pre-existing access files would be sent to AtoM. Which is really awesome if that works for you! The main point here is that you will need to consider when you perform file normalization. Archivematica provides some great tools, and AtoM can also render access JPEGs from TIFF files. But we are probably going to stick with our manual normalization process for a variety of reasons that make sense for us. And that is all good.
I think the bigger challenge is around metadata. Dan provided lots of info there, so I won't revisit that. But I think it is worth highlighting that Archivematica cannot currently send an access copy to an existing archival description in AtoM. AtoM maintains a 1:1 relationship between digital object and "information object" (i.e., a single archival description). When you send a DIP to AtoM, Archivematica asks you for a "slug" (part of the URL that points to an existing archival description) and then it nests the digital files underneath that URL. In other words, it creates one or more new archival descriptions in AtoM that will then require revision.
This represents a huge departure from our current workflow. Our team is accustomed to just navigating to the right archival description, uploading the digital object, and updating the metadata as necessary.
We use the digitization workflow as an opportunity to enhance our metadata, so it is no problem to update an existing record after a digital object is uploaded. But we don't need to pass that metadata to AtoM from Archivematica. And if we use Archivematica to upload an access copy to a new archival description in AtoM, we would introduce several extra steps to the workflow. We would need to reorganize the finding aid after the DIPs are received in AtoM. Or build a new finding aid and delete old archival descriptions as we go. This then brings into question how and when we regenerate the "slugs" that form part of the URLs that Archivematica used to send the DIPs to AtoM.
All this to say, we haven't really figured out how to fully bring our digitization workflow into an integrated Archivematica/AtoM environment because of how we use existing metadata in AtoM to support the digitization process. Maybe other institutions have taken a different approach. I'd love to hear!
For now, we are probably going to ingest our preservation masters and access copies (i.e., no normalization in Archivematica) and create the AIP but not send anything to AtoM. And we'll just keep uploading access copies to AtoM as we do. We will use the reference codes in our naming conventions within Archivematica so there will be "implicit" connections between the AIP and the DIP, but we won't see the DIP URL in the Access tab of Archivematica. It would be nice to have it all, but this approach is perfectly fine for us right now. It's all a big step forward in terms of digital preservation activities and it minimally disrupts the day-to-day work of our digitization team.
Hope this helps!
Creighton Barrett
Gregory Pierce
Feb 14, 2020, 11:36:28 AM2/14/20
to ica-ato...@googlegroups.com
Thank you for this amazing response, Creighton! So, using AtoM and Archivematica together do not make a fully integrated DAMS?
There's got to be something out there that takes a SIP containing a MODS.xml file and a TIFF and converts it to an AIP and sends it to the hosted archive, while creating a DIP using the MODS record and an access jpg, and sending it to AtoM for public browsing and searching. I can easily create access jpegs using actions and batch automation in Photoshop, so that's not even a big deal. I could even come up with a workflow that would include the creation of separate folders for those jpegs and uploading them into 2 different places. HOWEVER, I want a way to upload (either to both at once, or to each: web portal and archive separately) 1,000 images with their metadata files, and not have to connect those dots afterward. Is that possible?
Greg
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAHueW_XiidNrbmjhz%3DvCEukznA2sLKKo5Sd-eJNj2a3ioVt5JQ%40mail.gmail.com.
Creighton Barrett
Feb 16, 2020, 2:23:15 PM2/16/20
to ica-ato...@googlegroups.com
You're welcome, Gregory!
I don't know much about DAMS and what they might offer that Archivematica and AtoM doesn't offer. What you want seems like it could be possible with Archivematica and AtoM, but you'll need to think about what integration requirements you have, specific outcomes you want, and how to best achieve them with these applications. Your ideal workflow will probably require some additional scripting outside of Archivematica and AtoM.
I think Dan's initial response provides a great overview of what works out of the box and what might require some configuration, workflow development, etc. There are lots of options when it comes to metadata, but descriptive metadata integration between the two applications is somewhat limited and
bulk updates in AtoM can be complicated. I only highlighted our situation to show how our current digitization workflow (which uses existing metadata in AtoM) would be hard to maintain if we decide that sending digitization DIPs from Archivematica to AtoM is a priority. And I'm hoping other users will touch on how their digitization workflows work within Archivematica and AtoM.
Either way it's probably a good idea to play around with the demo sites or set up a local test environment to see if you can do what you want to do.
Good luck!
Creighton
Reply all
Reply to author
Forward
0 new messages