500 error on trying to link digital object to archival description

[Carolyn Sullivan]

Hello all,

I am trying to link this digital object:

https://static.scholarsportal.info/ottawa-permafrost/199e/c574/6832/42b7/9f73/4af0/a3df/a846/ACC-2024-06-6105ea59-4f85-46a2-a304-f8eef6ecf6c1/objects/b31a6630-5515-4ba8-ba3b-410bf3bce469-ACC-2024-06-M1.mp3

to this archival description:

https://arcs-atom.uottawa.ca/index.php/presentation-to-engl-3911-2

Every time I try, I get a 500 error.  I've read through the documentation and previous errors reported in the Atom Users Group on this issue seem to revolve around folks not providing the URL to the actual bitstream, so I'm a bit confused as to what the issue would be.

However... I did try linking this record, which is 90 MB on download, to an archival description in our sandbox, which temporarily has its upload limits lifted to 100 MB (our production instance has its upload limits set at 64 MB), and while the browser did eventually time out, the file DID attach.  So I have a couple questions:

(1) When you link a digital object with the URL, is AtoM simply showing material that is hosted offsite (which is what we always believed), or is it actually UPLOADING that material from that site to ours (which is what this seems to indicate).

(2) If yes to (1), is there a commandline tool that would allow me to upload larger files directly from external sites, or do I have to download the file and then upload it using the existing csv commandline tool?

Thanks,

Carolyn.

[Maria Papanikolaou]

Hi Carolyn, 

I think I am a champion in the face of 500 errors in the last four months. 

Dan has very generously provided the following documentation to help us tackle the problem: 

https://groups.google.com/g/ica-atom-users/c/wVrhL-TAaDY/m/Isu3jUY-AAAJ 

Hopefully it will be helpful. 

Best regards, 

Maria 

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/a408b341-2c54-4b72-976b-c68401c5fc7cn%40googlegroups.com.

[Dan Gillean]

Hi Carolyn, 

I haven't tried it myself, but you could certainly be hitting a timeout error in your browser (which will typically have a 1-minute timeout default, to prevent requests from consuming all available resources), but that doesn't usually lead directly to a 500 error - it would be worth looking in the logs to see what the error message says. As Maria has kindly provided the link, you can certainly use the summary post on dealing with 500 errors I assembled for her in another thread as a good reference point. 

As you your questions: 

(1) When you link a digital object with the URL, is AtoM simply showing material that is hosted offsite (which is what we always believed), or is it actually UPLOADING that material from that site to ours (which is what this seems to indicate).

Because of the derivatives, it's a bit of both. When you upload via URL, then instead of actually uploading a copy of the original digital object as would normally happen, AtoM will store the URI path. When users who have permissions try to click through the reference image to access the master, they are simply redirected to the source URL. So you are saving a lot of storage space and not uploading. 

However, AtoM DOES still need to access and fetch a temporary copy of the remote digital object, so that it can generate derivatives - the reference display copy on the view page of the related description and the thumbnail shown in search/browse results. These are both generated by AtoM's tools and stored locally in your uploads directory, even when using URLs to link objects remotely. So, it's still possible for remote digital objects to lead to timeouts. 

(2) If yes to (1), is there a commandline tool that would allow me to upload larger files directly from external sites, or do I have to download the file and then upload it using the existing csv commandline tool?

You can actually use URIs in the digital object load task, if I recall correctly. So, no need to download it to use the CLI. See: 

• https://www.accesstomemory.org/docs/2.8/admin-manual/maintenance/cli-import-export/#load-digital-objects-via-the-command-line
  

The filename column expects a digital object path, ending in the file extension.  That can either be a local path to a directory of images for upload that AtoM can access, or it can be a URL - provided it still meets AtoM's requirements for remote object linking: 

1. Must be available on the public web - no password protection, VPN requirements, etc
2. Must be an HTTP or HTTPS link - no FTP or SFTP links, no local share drives, etc
3. Must end in the file extension

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--

[C S]

Hello Dan,

Thank you so much for the information (thanks to Maria as well for the general reference page).  

I tried to run the command sudo php symfony digitalobject:load /home/pam_library/UploadDigitalObject_040224_test.csv from the atom directory.

with the csv attached (it should be UTF-8 encoded and have Linux EOF).

Unfortunately, for some reason I got this response:

>> digital-object Load digital objects from /home/pam_library/UploadDigitalObject_040224_test.csv...
Couldn't read file 'https://static.scholarsportal.info/ottawa-permafrost/199e/c574/6832/42b7/9f73/4af0/a3df/a846/ACC-2024-06-6105ea59-4f85-46a2-a304-f8eef6ecf6c1/objects/b31a6630-5515-4ba8-ba3b-410bf3bce469-ACC-2024-06-M1.mp3'
Couldn't read file 'https://static.scholarsportal.info/ottawa-permafrost/199e/c574/6832/42b7/9f73/4af0/a3df/a846/ACC-2024-06-6105ea59-4f85-46a2-a304-f8eef6ecf6c1/objects/b7c8f72d-aa65-40c5-a8cf-8d8f271b57e2-ACC-2024-06-M2.mp3'
>> digital-object Successfully Loaded 0 digital objects.
>> digital-object Please update the search index manually to reflect any changes

I'm running Atom 2.7.x .  The files don't have any firewalls in front of them and are https links that end in the file extension.  Any idea why this isn't working?

Thanks,

Carolyn.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZKj6FjmTQ-iJRG%2BK77CsWNv%3DqzxFarE9%3DhNxL9sF1_dbQ%40mail.gmail.com.

[C S]

Note: the same error appears if I try running this command with the -s option, ie: sudo php symfony digitalobject:load -s /home/pam_library/UploadDigitalObject_040224_test.csv

[Dan Gillean]

Hi Carolyn, 

Hmmm. I tried out your CSV with my own slugs and got the same error. I then tried with 2 different URIs to make sure it wasn't those specific files, this time using one MP3 and one JPG link I found online. I still got the same error. 

I have asked for some input from our developers as there are some complicating factors here. I could have sworn we have used the task with external URIs before, but I am confirming this, given that it is not officially documented as such. Meanwhile, I am testing in a Docker VM, which may not be able to access external URIs outside the container it is running in when executing the task - so it could be me.... and if your AtoM instance is behind a VPN, firewall, or similar, it could be both of us?

Or of course, the most likely option - that we've encountered a regression affecting how AtoM handles external URLs. 

I will update you when I know more. 

Cheers, 

 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CA%2BpcS1TAVNoAaB2KxDVy-Bdono7KOB-73CGeFfZ%2BLKd1RYATjA%40mail.gmail.com.