Upload limit

[brzic...@gmail.com]

We are running v 2.2.1-123 and want to upload a 4GB video...

Atom has a -1GB file limit (unlimited) set but the uploads still hang.

Which values should I change in the php config ? Assuming that is where it should be changed...

The upload interface is telling me that there is a 512 MB limit. Where is that value being read from ?

Thanks

[Dan Gillean]

Hi there, 

There are a few places to check, and quite a few factors that can impact the size of uploads. In addition to the option in Admin > Settings, there is also a limit in the config/app.yml configuration file. See: 

• https://www.accesstomemory.org/docs/latest/admin-manual/customization/config-files/#config-app-yml
  

Note that there are also other settings in PHP that might affect your ability to import digital objects - to change these you will have to locate your php.ini file. You might want to check on values usch as Max execution time, Memory limit, Max file size, etc. This PHP page includes some tips and common pitfalls for uploads: 

• http://php.net/manual/en/features.file-upload.common-pitfalls.php
  
• We also have some tips here: https://www.accesstomemory.org/docs/latest/admin-manual/installation/execution-limits/
  
• See also: https://www.accesstomemory.org/docs/2.5/admin-manual/maintenance/troubleshooting/#why-can-t-i-upload-large-digital-objects

If you are using AtoM in multi-repository environment, you might also want to make sure there is not a specific limit on your repository. See: 

• https://www.accesstomemory.org/docs/latest/user-manual/add-edit-content/archival-institutions/#set-digital-object-upload-limit-for-an-archival-institution
  

Finally, remember that if you are trying to upload a file bigger than 64MB via the user interface, the upload likely will not work anyway at the moment. When you upload something through the user interface, everything must happen via the browser. Most browsers have a timeout limit of about 1 minute - so anything that takes longer to upload will time out and fail. A 4GB video almost certainly will not upload via the user interface. 

If you want to upload larger files, it is best done via the command-line. You could use the digital object import CSV instructions we have in our documentation, here: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-import-export/#load-digital-objects-via-the-command-line
  

In this previous forum thread, I also outline how you might use the 2 digital object related columns in the description's CSV import template to add digital objects. See: 

• https://groups.google.com/d/msg/ica-atom-users/tShUsjLd8qg/S3iUdiTtAwAJ
  

In the future, we could do further development so that the upload through the user interface is performed as a job, using the job scheduler that has been included in AtoM to handle long-running tasks since the 2.2 release. This would allow the upload to be executed asynchronously in the background as a job, managed by the job scheduler - this way, you would not experience timeout errors when trying to upload large files to AtoM. However, this development has not yet been sponsored for inclusion in AtoM. 

If this functionality sounds important to you and your institution might be interested in sponsoring it for inclusion in a future list, please feel free to contact me off-list, and Artefactual could prepare a development estimate for you. 

Let us know how it goes!

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/19b3a6f9-442a-48a8-ba77-1d079db121a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[brzic...@gmail.com]

Hi Dan,

Looks like command-line it will be....

Thanks...

[brzic...@gmail.com]

Hi Dan,

I looked for app.yml. I have a total of 10 iterations of it. Which one matters ?

Thanks

[Dan Gillean]

Hi, 

Wow! Do you have other applications installed on the same server? If you are just grepping to find app.yml, I could see that returning a lot, but as far as I am aware, AtoM should just have the one, in the config subdirectory. If you followed our recommended installation instructions, then the full path would be /usr/share/nginx/atom/config/app.yml. 

Be sure to check your PHP execution limits as well - there is an upload_max_filesize variable set there, which by default is 64MB I believe, so that is likely the limit you've been hitting. You may want to adjust other variables as well, like post_max_size, memory_limit, and max_execution_time. 

See: 

•  https://www.accesstomemory.org/docs/latest/admin-manual/installation/execution-limits/
  

Remember to clear the cache and restart PHP-FPM and memcached after! There are instructions on how to do so at the bottom of the Execution limits documentation page. 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e4cde7eb-d94b-4875-a7bc-90bb812f9db6%40googlegroups.com.

[brzic...@gmail.com]

Hi Dan,

I am not the archivist. I actually looked at this closely. The upload is not a problem after a half-dozen changes to nginx and php.

It is the actual import process. I guess that means the .flv file creation. Any ideas about that ?

Thanks

[Dan Gillean]

Hello, 

You will need to ensure that all of AtoM's additional packages are installed for digital object management - see: 

• https://www.accesstomemory.org/docs/latest/admin-manual/installation/requirements/#other-dependencies
  
• https://www.accesstomemory.org/docs/latest/admin-manual/installation/linux/ubuntu-xenial/#other-packages
  

Other than that, if you have checked all of the various upload limit locations I mentioned and adjusted any parameters needed, then AtoM should be able to import and play the file, as far as I'm aware. ImageMagick, the library that handles much of our image processing on upload (to generate reference copy and thumbnail derivatives), uses ffmpeg under the hood, which can definitely process and convert .flv. 

The main concern remains actually adding a file as big as 4GB to AtoM. As I mentioned, doing so via the user interface is unlikely to work, because the browser will time out before the entire file can be processed, aborting the upload and conversion. Instead, work with your archivist(s) to upload large digital objects via the command-line, using either the dedicated digital object import task, or the CSV import. Please see my first message in this thread for links. 

It's also possible I'm not understanding exactly what the issue you're encountering is. Can you please describe to me in greater detail: 

• What you found in the config/app.yml file (was the upload limit already set to -1 there?)
• Any changes you made to the php.ini file as per the Execution limits documentation I sent you
• If you found and changed any other possible places where there might be upload limits, like the global upload limit setting or the per-repository upload limit settings
• The exact steps you are taking to try to import your file, and what you're seeing in response
• Whether you've tried any of the command-line options yet
• Anything else that you think might be helpful in understanding and diagnosing the issue. 

Thanks! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/40408d28-ef77-499d-a66f-cbd2c857b285%40googlegroups.com.

[brzic...@gmail.com]

Hi Dan,

- 1 was already set.

- I changed upload_max_filesize and post_max_size to 5000M in...

/etc/php5/cli/php.ini

/etc/php5/fpm/php.ini

/etc/php5/fpm/pool.d/atom.conf

/etc/nginx/nginx.conf added client_max_body_size=5000M

I don't think there are issues with the object management components...

Did the following workaround test:

Uploaded a small .mp4 file that generated an .flv and .jpeg caption correctly.

Identified the actual folder location in the file system.

Overwrote the .mp4, .flv and .jpg files with identically named files that actually had the correct content.

Tested. Everything works fine:

-flowplayer plays big .flv just fine

-download of big .mp4 works fine

-only issue is that the file size metadata is wrong.

My archivist is going to read up on command-line uploads. I will likely have to give her an atom ubuntu server account and the needed sql credentials...