Skip to first unread message
Josh Smith
Feb 11, 2020, 6:24:02 PM2/11/20
to AtoM Users
Hello all,
I'm running AtoM 2.5.3 - 172 on Ubuntu 18.04. According to AtoM, file size upload limits are preventing me from uploading a PDF through the browser interface. The file in question is 73.2MB. I've uploaded other files up to 58.8MB with no issues. On the Import multiple digital objects page I get the following message when I select the file: "This file couldn't be uploaded because of file size upload limits"
I've looked through past posts and the documentation but I have been unable to find where this upload limit is being defined. I followed what I found to make some changes, but nothing I've tried has worked. Here's what I did:
- php -i | grep php.ini to figure out what php.ini is in use
- changed
post_max_size = 512Mandupload_max_filesize = 128M - ran sudo systemctl restart php7.2-fpm and php symfony cc
When this didn't work I did the same in the other php.ini file just in case, but still nothing. I also checked memory_limit in both files and it is set higher than post_max_size in both places, which is correct from what I've read.
I also checked config/app.yml and upload_limit is set to -1 so there should be no issue. I'm confused because none of the places I know to look are defining the upload limit above 58.8MB but below 73.2MB, but that's my limit somehow. In fact before I made changes the php.ini files had limits much lower than what I've changed them to and files up to 58.8MB were uploading fine.
Any ideas?
Thanks!
Dan Gillean
Feb 12, 2020, 10:41:51 AM2/12/20
to ICA-AtoM Users
Hi Josh,
If you want to upload large files, it is currently best accomplished via the command-line. You could use the digital object load task instructions we have in our documentation, which allows you to use a simple 2-column CSV and a directory of files placed on the server to upload digital objects to existing description. See:
In this previous forum thread, I have also outlined how you might use the 2 digital object related columns in the description's CSV import template to add digital objects. See:
In the future, we could do further development so that the upload through the user interface is performed as a job, using the job scheduler that has been included in AtoM to handle long-running tasks. This would allow the upload to be executed asynchronously in the background as a job, managed by the job scheduler - this way, you would not experience timeout errors when trying to upload large files to AtoM. There are some good libraries that support large file uploads by hashing and chunking them as well, that would work in concert with the job scheduler - I'm currently quite partial to uppy, and would love to see support for it added to AtoM in a future release. However, this development has not yet been sponsored for inclusion in AtoM.
If this functionality sounds important to you and your institution might be interested in sponsoring it for inclusion in a future list, please feel free to contact me off-list, and Artefactual could prepare a development estimate for you.
It's clear that you've looked through the documentation and the forum for solutions before posting - thank you! A couple of further suggestions to try:
First, don't forget that the atom.conf PHP pool file you created during the installation process also has execution limit values in it. These may be overriding the global php.ini values, and our default config in the installation instructions sets a 64MB upload limit, so that might fit with what you're seeing. If you've followed our recommended instructions, this is typically found at /etc/php/7.2/fpm/pool.d/atom.conf - see:
Remember to clear the cache and restart PHP-FPM if you change this file.
Additionally, don't forget there is an Admin setting that can globally limit file uploads, as well as a per-repository setting - might as well make sure these are both set to -1 as well. See:
There's a small chance that you've edited the wrong php.ini file on your system as well - there can be more than 1! Our docs here have some suggestions on how to find them:
Finally: while I'm hopeful that the issue is the PHP pool file, please be aware that you may still run into the browser timeout limits for a 72MB file upload via the user interface. When you upload something through the user interface, everything must happen via the browser synchronously - that is, on demand in real time. Most browsers have a built-in default timeout limit of about 1 minute - so anything that takes longer to upload will time out and fail.
If you want to upload large files, it is currently best accomplished via the command-line. You could use the digital object load task instructions we have in our documentation, which allows you to use a simple 2-column CSV and a directory of files placed on the server to upload digital objects to existing description. See:
In this previous forum thread, I have also outlined how you might use the 2 digital object related columns in the description's CSV import template to add digital objects. See:
In the future, we could do further development so that the upload through the user interface is performed as a job, using the job scheduler that has been included in AtoM to handle long-running tasks. This would allow the upload to be executed asynchronously in the background as a job, managed by the job scheduler - this way, you would not experience timeout errors when trying to upload large files to AtoM. There are some good libraries that support large file uploads by hashing and chunking them as well, that would work in concert with the job scheduler - I'm currently quite partial to uppy, and would love to see support for it added to AtoM in a future release. However, this development has not yet been sponsored for inclusion in AtoM.
If this functionality sounds important to you and your institution might be interested in sponsoring it for inclusion in a future list, please feel free to contact me off-list, and Artefactual could prepare a development estimate for you.
Hopefully this will help!
Cheers,
he / him
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/8240ce80-d351-4996-9559-e6015550352f%40googlegroups.com.
Josh Smith
Feb 12, 2020, 1:27:09 PM2/12/20
to AtoM Users
Thanks for the assistance, Dan. Editing atom.conf as you suggested got rid of the error I described in my first message, but a new problem arose. AtoM would show the progress bar that it was uploading, but then would say that the upload failed. Looking at the browser console I saw that the site was throwing a 413 Request Entity Too Large error when the upload finished.
I was able to resolve this by editing the atom file for my site (/etc/nginx/sites-available/atom), changing the client_max_body_size in the server block to a value appropriate for my needs. I didn't see this mentioned in any posts or documentation, so it might be worth adding somewhere as another thing to check.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-ato...@googlegroups.com.
Dan Gillean
Feb 12, 2020, 1:44:57 PM2/12/20
to ICA-AtoM Users
Thank you for updating the thread, and sharing what worked for you, Josh!
I have filed an issue in our documentation code repository, to remind us to improve the documentation for resolving these kinds of issues (including mentioning the client_max_body_size parameter you have had to adjust). See:
Cheers,
he / him
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/9a2cee86-d728-4102-b59a-003e222e968d%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages