Error uploading large video

[Geovano Lago Quatrin]

Hello everyone,

I'm getting a error after uploading a large video (more than an hour of recording) file as a digital object. After the video upload, the page stalls and after a period of time the browser reports error 504 Gateway Timeout.
The application instance become inaccessible after this, not accepting new conections. Seeing the running processes at the server there is a set of ffmpeg threads doing the conversion of the video, I guess, to a AtoM standard format.  

There is a setup to make this conversion on the background in another process, without affecting the php-fpm process pool? I'm guessing the entire php-fpm pool become inaccessible because any new connections to the instance return the 504 error.

Thanks

[Dan Gillean]

Hi Geovano, 

Can you tell us a bit more about your installation, and the upload process? Such as: 

• How are you uploading this video - via the command-line (such as the digital object load task, or as part of a CSV import), or via the user interface?
• What is the total size of the video file?
• What is your full AtoM version number, as shown in Admin > Settings?
• Does your installation meet or exceed the recommended hardware requirements listed here in the documentation?
• Processor: 2 vCPUs @ 2.3GHz
  Memory: 7GB
  Disk space (processing): 50GB at a minimum for AtoM’s core stack plus more storage would be required for supporting any substantial number of digital objects.
  
• Have you made any adjustments to the PHP execution limits in your php.ini file and/or the PHP pool set up during installation and configuration?

I have recently shared some information about the limitations of AtoM's digital object import - particularly when done via the user interface - and what can happen when this process fails, along with some suggestions on how to fix the issue in another forum thread. Please see: 

• https://groups.google.com/g/ica-atom-users/c/POpm4nyK9ZQ/m/qhb0oPxKAQAJ
  

Given that your large file import timed out mid-process, it's possible that this has left an incomplete entry in the digital object table of the database - meaning you should follow the links and instructions in that thread to make a backup of your data, and then check for data corruption. 

For large digital objects, we strongly recommend performing these imports via the command-line - in your case, if you are trying to attach this video to an existing description, then I would suggest trying the digital object load task.

Depending on the size of the digital object, you may still need to a) confirm that your server has enough resources (i.e. memory, CPU, disk space), and b) you may still want to increase some of the  PHP execution limits in your php.ini file and/or the PHP pool set up during installation on your server - particularly values such as the max_execution_time and the upload_max_file_size. Remember to restart PHP-FPM after making such changes! 

Good luck, and let us know how everything goes!

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/590a2f33-c42e-487e-80e8-34f7f6b9ab47n%40googlegroups.com.

[Geovano Lago Quatrin]

Hello Dan, 

Follow the requested information: 

• The video is uploaded by the user interface
• The video size is 110 MB
  
• Version: 2.7.3 - 192
  
• Processor: 12vCPU Memory: 16GB
• PHP limits in /etc/php/7.4/fpm/php.ini 
  
  • max_execution_time = 2400
  • memory_limit = 128M
  • post_max_size = 800
  • upload_max_filesize = 800M 

The PHP limits were already tweaked to allow bigger uploads. 

I've not reported in the previous message, but after ~15 minutes the application become responsive again and the uploaded content appear as "Versão preliminar" (preliminar version). 

Apperently the format conversion runs during the inresponsive time of the application. 

The users of our AtoM instance doesn't have CLI access to the server for digital object upload, only to the user interface. 

Thanks for the response, I'll think about some way to solve. 

[Geovano Lago Quatrin]

Hello, 

I found 2 parameters to increase the time for the script finish and avoid the 504 error. At the nginx default configuration had to set the parameters fastcgi_read_timeout and fastcgi_read_timeout to allow the ffmpeg command finish the conversion and return control to the PHP script . Modifications made in /etc/nginx/sites-available/atom , as reported here. 

   location ~* ^/uploads/r/(.*)$ {
      include /etc/nginx/fastcgi_params;
      set $index /index.php;
      fastcgi_param SCRIPT_FILENAME $document_root$index;
      fastcgi_param SCRIPT_NAME $index;
      fastcgi_pass atom;
      fastcgi_read_timeout 3600s;
      fastcgi_send_timeout 3600s;
   }

   location ~ ^/(index|qubit_dev)\.php(/|$) {
      include /etc/nginx/fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_split_path_info ^(.+\.php)(/.*)$;
      fastcgi_pass atom;
      fastcgi_read_timeout 3600s;
      fastcgi_send_timeout 3600s;
   }

Thanks. 

[Dan Gillean]

Thanks for sharing what worked for you, Geovano! 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/fb21420a-8a34-442e-97a7-215233857b93n%40googlegroups.com.