Metadata-only DIP upload to AtoM not working
49 views
Skip to first unread message
Julie S
Mar 26, 2024, 1:08:49 PMMar 26
to archivematica
Hello,
I would like to use the Metadata-only DIP upload to AtoM feature in our production environment (v 1.14.1) but am receiving the following error in the Archival Storage tab when I try to upload the metadata: "Metadata-only DIP upload failed, check the logs for more details".
In the Archivematica logs, I see that the AIP is downloaded and the METS is extracted (or at least the extraction process is started), but no more logs are provided after that in both debug and regular modes. I'm not sure if I should be seeing more? I also don't see the directory where the logs say the METS file is being extracted to.
I see the same error when I try the feature in the Archivematica and AtoM sandboxes but can't confirm if the log messages are the same or not.
Are Archivematica and/or AtoM supposed to be configured in a certain way to enable this feature? I've confirmed that the AtoM credentials in the Administration > AtoM/Binder page are correct for our environment.
Many thanks for any help you can provide!
Julie
Julie S
Apr 5, 2024, 11:58:07 AMApr 5
to archivematica
Hello,
Just wondering if anyone has any suggestions for this?
As an update, I tried this again twice with our dev Archivematica instance and the AtoM sandbox, using "/archivematica-metadata-upload" and "archivematica-metadata-upload" as the slug. The dashboard still returns: "Metadata-only DIP upload failed, check the logs for more details", and I see these same messages in the logs for both attempts:
dashboard.debug.log
INFO 2024-04-05 15:42:10 elasticsearch:base:log_request_success:194: GET http://localhost:9200/aips/_search?_source=name%2Csize%2Ccreated%2Cstatus%2CfilePath%2Cencrypted [status:200 request:0.017s]
DEBUG 2024-04-05 15:42:10 archivematica.dashboard:atom:upload_dip_metadata_to_atom:77: Extracting file xmlImportDemo-39d1f939-22ef-4b68-9f55-6d5d54d8a8c1/data/METS.39d1f939-22ef-4b68-9f55-6d5d54d8a8c1.xml into /tmp/tmp8ahkrsst
INFO 2024-04-05 15:42:13 archivematica.dashboard:atom:upload_dip_metadata_to_atom:91: Creating file container with slug /archivematica-metadata-upload and title xmlImportDemo
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property size from METS: 10860
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property format_name from METS: JPEG
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property format_version from METS: 1.01
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property format_registry_name from METS: PRONOM
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property format_registry_key from METS: fmt/43
INFO 2024-04-05 15:42:13 archivematica.dashboard:atom:upload_dip_metadata_to_atom:114: Creating child with title acorns.jpeg
ERROR 2024-04-05 15:42:13 agentarchives.atom.client:client:_request:67: Response code: 429
ERROR 2024-04-05 15:42:13 agentarchives.atom.client:client:_request:68: Response body: <html>
<head><title>429 Too Many Requests</title></head>
<body bgcolor="white">
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx</center>
</body>
</html>
ERROR 2024-04-05 15:42:13 archivematica.dashboard:views:view_aip:744: Unexepected error during metadata-only DIP upload (UUID: 39d1f939-22ef-4b68-9f55-6d5d54d8a8c1)
Traceback (most recent call last):
File "/usr/share/archivematica/dashboard/components/archival_storage/atom.py", line 116, in upload_dip_metadata_to_atom
parent_slug=file_slug, title=title, level="Item"
File "/usr/share/archivematica/virtualenvs/archivematica/lib/python3.6/site-packages/agentarchives/atom/client.py", line 752, in add_child
expected_response=201,
File "/usr/share/archivematica/virtualenvs/archivematica/lib/python3.6/site-packages/agentarchives/atom/client.py", line 106, in _post
expected_response=expected_response,
File "/usr/share/archivematica/virtualenvs/archivematica/lib/python3.6/site-packages/agentarchives/atom/client.py", line 69, in _request
raise CommunicationError(response.status_code, response)
agentarchives.atom.client.CommunicationError: AtoM server responded with status code 429 (URL: https://sandbox.accesstomemory.org/api/informationobjects)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/share/archivematica/dashboard/components/archival_storage/views.py", line 732, in view_aip
name, uuid, form_upload.cleaned_data["slug"]
File "/usr/share/archivematica/dashboard/components/archival_storage/atom.py", line 121, in upload_dip_metadata_to_atom
raise AtomMetadataUploadError
components.archival_storage.atom.AtomMetadataUploadError
DEBUG 2024-04-05 15:42:10 archivematica.dashboard:atom:upload_dip_metadata_to_atom:77: Extracting file xmlImportDemo-39d1f939-22ef-4b68-9f55-6d5d54d8a8c1/data/METS.39d1f939-22ef-4b68-9f55-6d5d54d8a8c1.xml into /tmp/tmp8ahkrsst
INFO 2024-04-05 15:42:13 archivematica.dashboard:atom:upload_dip_metadata_to_atom:91: Creating file container with slug /archivematica-metadata-upload and title xmlImportDemo
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property size from METS: 10860
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property format_name from METS: JPEG
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property format_version from METS: 1.01
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property format_registry_name from METS: PRONOM
DEBUG 2024-04-05 15:42:13 archivematica.dashboard:atom:_load_premis:64: Extracted property format_registry_key from METS: fmt/43
INFO 2024-04-05 15:42:13 archivematica.dashboard:atom:upload_dip_metadata_to_atom:114: Creating child with title acorns.jpeg
ERROR 2024-04-05 15:42:13 agentarchives.atom.client:client:_request:67: Response code: 429
ERROR 2024-04-05 15:42:13 agentarchives.atom.client:client:_request:68: Response body: <html>
<head><title>429 Too Many Requests</title></head>
<body bgcolor="white">
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx</center>
</body>
</html>
ERROR 2024-04-05 15:42:13 archivematica.dashboard:views:view_aip:744: Unexepected error during metadata-only DIP upload (UUID: 39d1f939-22ef-4b68-9f55-6d5d54d8a8c1)
Traceback (most recent call last):
File "/usr/share/archivematica/dashboard/components/archival_storage/atom.py", line 116, in upload_dip_metadata_to_atom
parent_slug=file_slug, title=title, level="Item"
File "/usr/share/archivematica/virtualenvs/archivematica/lib/python3.6/site-packages/agentarchives/atom/client.py", line 752, in add_child
expected_response=201,
File "/usr/share/archivematica/virtualenvs/archivematica/lib/python3.6/site-packages/agentarchives/atom/client.py", line 106, in _post
expected_response=expected_response,
File "/usr/share/archivematica/virtualenvs/archivematica/lib/python3.6/site-packages/agentarchives/atom/client.py", line 69, in _request
raise CommunicationError(response.status_code, response)
agentarchives.atom.client.CommunicationError: AtoM server responded with status code 429 (URL: https://sandbox.accesstomemory.org/api/informationobjects)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/share/archivematica/dashboard/components/archival_storage/views.py", line 732, in view_aip
name, uuid, form_upload.cleaned_data["slug"]
File "/usr/share/archivematica/dashboard/components/archival_storage/atom.py", line 121, in upload_dip_metadata_to_atom
raise AtomMetadataUploadError
components.archival_storage.atom.AtomMetadataUploadError
We also see this with a production Archivematica instance connected to a production AtoM instance, and would very much like to make use of this feature in this instance!
Any suggestions are much appreciated!
Julie
Amaya Rodrigo
Apr 9, 2024, 9:39:42 AMApr 9
to archiv...@googlegroups.com
Julie,
I don't think our sandbox AtoM instance is configured to allow any sort of uploads, including DIP uploads.
Sorry about that!
--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/6a78dda9-fc98-44a1-8b32-c4963c367c64n%40googlegroups.com.
--
Amaya M Rodrigo Sastre (she/her/bofh) - sysadmin@artefactual
Julie S
Apr 9, 2024, 10:09:56 AMApr 9
to archivematica
Hi Amaya,
Thank you for your reply! We are also seeing the same error message and log output with a production Archivematica instance that we are trying to connect to a production AtoM instance (for the Artefactual sandboxes, the API key for the admin user in the AtoM demo instance is input for the Archivematica sandbox).
From the log output, we are unsure if the issue is on the Archivematica or the AtoM side: "
agentarchives.atom.client.CommunicationError: AtoM server responded with status code 429 (URL: https://sandbox.accesstomemory.org/api/informationobjects)". Are you able to help point us in a direction to investigate further? We are using Archivematica 1.14.x and AtoM 2.x
Thank you so much!
Julie
Julie S
Apr 9, 2024, 6:00:47 PMApr 9
to archivematica
Apologies, Amaya, I forgot to link the AtoM description that I've been testing with in the AtoM sandbox: https://sandbox.accesstomemory.org/archivematica-metadata-upload.
It appears that Archivematica is in fact connecting to AtoM as one item level description from 2 separate attempts is added. However, the upload does not seem to complete as there is more than one file in each package that I attempted to upload metadata from and no digital object metadata is assigned to the two items that were added. There is an image file uploaded but I imported that directly through the AtoM interface to test what the digital object metadata section should look like.
Best,
Julie
Amaya Rodrigo
Apr 16, 2024, 12:54:30 PMApr 16
to archiv...@googlegroups.com
Julie,
So sorry it took so long to get back to you (down with the flu, doing better now).
What I am seeing in the logs you pasted is:
ERROR 2024-04-05 15:42:13 agentarchives.atom.client:client:_request:67: Response code: 429
ERROR 2024-04-05 15:42:13 agentarchives.atom.client:client:_request:68: Response body: <html>
<head><title>429 Too Many Requests</title></head>
<body bgcolor="white">
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx</center>
</body>
</html>
This makes me think it is the AtoM web server that is configured to only allow for a certain number of requests per second.
Is this log from your am instance?
Can you also check the atom web server logs (usually /var/log/ngnix/) and see if there's more info in there?
Thanks!
To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/02d11ec7-39ed-46c7-bb29-57d5549da106n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages