CSV importing and 504 Gateway Time-outs

[ty...@lre-tech.com]

Hi all,

I am attempting to import a large set of CSV data to a test instance of AtoM 2.  I find with CSVs exceeding about 50 KB or 400 items, I get a 504 Gateway Time-out.  What could the issue be and what is the best way to go about solving this?  Is there a parameter causing this that can be changed? Or is this server-related?

I do not have issues with smaller CSVs containing 200 items, around 35 KB in size.

I noticed a slightly similar discussion here and this part of the FAQ, but I don't think that's the same issue since those are error 500 and not 504.

The ultimate goal would be to reliably import CSVs containing hundreds of thousands of items, so I don't think breaking them up into small files is an option.

Thanks in advance for your help.  The sharing of knowledge in this group is much appreciated.

Tyler

[Hutchinson, Tim]

Hi Tyler,

 

Interesting – I haven’t seen that particular error, but it does sound similar to the other timeout issues.

 

I’m mostly wondering whether you’re using the user interface, or the command line. For larger files, I’d definitely suggest using the command line:

https://www.qubit-toolkit.org/wiki/CSV_import

 

Tim

 

Tim Hutchinson
Head, University Archives & Special Collections
University Library, University of Saskatchewan

Tel: (306) 966-6028  Fax: (306) 966-6040

Email: tim.hut...@usask.ca

Web: http://library.usask.ca/archives/

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at http://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/fa54bbf6-1bb2-43f9-8867-41508b64a1fe%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dan Gillean]

Hi Tyler,

If you are using the user interface, it's possible that you've been running up against the PHP script execution limits. There are 2 settings that you might want to consider adjusting - the max_execution_time which affects the HTTP request limit, and the memory_limit, which sets a cap on the maximum amount of memory a script can consume. More information here: https://www.qubit-toolkit.org/wiki/PHP_script_execution_limits

That said, I think that Tim's suggestion of using the command line is your best bet for large imports.

Let us know how it goes!

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6B26EA65847C3D47A5A1450ABE5A06884A014F22%40CAMPUSMB3.usask.ca.

[David Juhasz]

Hi Tyler,

If you are using nginx as a webserver, then likely you are running into the nginx configuration limit for fcgi (via the FastCGI Gateway) script to run.
https://rtcamp.com/tutorials/php/increase-script-execution-time/

Apache and other webservers usually have a similar limit on how long they will wait for an response from an fcgi request.

Regards,

David

--

David Juhasz
Director, AtoM Technical Services

Artefactual Systems Inc.
www.artefactual.com

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZ%2BU3MGs2Np%2Bbxjyj%2BE1%3DcBRVMKjY_Pu5efA-SGttc2PeA%40mail.gmail.com.

[Gabriel Batista da Silva Nogueira]

Hi, 

I am having a problem with error 504 time out,
From what I've seen, the error is when sending too many images, I tested it by sending 3 images and everything went well, but when sending 12 images, this error occurs, I don't know what it could be, I've tried to change the nginx.conf, but still nothing did not work.

I added this on http: 
 proxy_read_timeout 3600;
proxy_connect_timeout 3600;
proxy_send_timeout 600;
client_max_body_size 120M;

and 

server {

                proxy_read_timeout 3600;
                proxy_connect_timeout 3600;
                proxy_send_timeout 3600;
                fastcgi_read_timeout 3600s;
                fastcgi_send_timeout 3600s;


                location /upload {

                        proxy_read_timeout 6000;
                        proxy_connect_timeout 6000;
                        proxy_send_timeout 6000;

                }

                location / {
                        proxy_http_version 1.1;
                        proxy_set_header Connection "";
                }
                location ~ \.php$ {
                        fastcgi_read_timeout 240;
                }
        }

but it's still not working.

I checked the log and the error reported was this:

2023/03/08 20:01:04 [error] 166622#166622: *5437 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxxxxxxxxxx, server: _, request: "POST /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", upstream: "fastcgi://unix:/run/php7.2-fpm.atom.sock", host: "xxxxxxxxxxxx", referrer: "http://xxxxxxxxxxxxx/informationobject/multiFileUpload".

I'm trying all the solutions that were found but none of them managed to resolve.

[Dan Gillean]

Hi Gabriel, 

From the looks of the error message, this is not a CSV import, but rather, it looks like an attempt to use the multi-uploader via the user interface (i.e. "Upload Digital Objects" in the More button menu) - is that right?

If so, then the first limit you are likely encountering is the browser timeout limit. Most browsers are configured to time out long-running requests after about 1 minute.  In the long term, it would be great for us to implement a solution that can do chunking during large file uploads, and use the job scheduler to handle the request asynchronously in the background - however, such work is not yet implemented. 

In the meantime, if you are uploading a lot of images and/or large images, your best approach will be to use the command-line import options, rather than trying to find and adjust every timeout limit. There are a couple options for attaching digital objects to existing descriptions. 

The first and likely easiest approach is to use the Digital Object Load task. See: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-import-export/#load-digital-objects-via-the-command-line
  

Essentially, you prepare a folder of digital objects for import, and a very simple CSV with 2 columns - one with the full path and filename of the target digital object, and the other with information about the associated target description to which you want to attach your digital object. The second column about the description can use either the identifier of the description, the slug (i.e. the permalink, or unique part of the URL - for example, in the URL www.example.com/my-description, my-description would be the slug), or the object ID (i.e. the unique ID used in the AtoM database. 

I recommend using the slug - it's guaranteed to be unique per description (unlike the identifier), and it doesn't require using SQL queries to get the value (like the information_object_id requires).  

You place your folder of digital objects somewhere on your AtoM server - I suggest just below the root installation directory. You then make sure that the paths in the CSV point to that location - so for example: 

• Make sure each digital object has a unique name (.e.g image1.jpg, image2.jpg, or whatever
• Make a folder called import-images and put all the digital objects inside
• Place the folder with your images on your AtoM server at /usr/share/nginx/atom
• Your import CSV will now have paths in the filename column like: /usr/share/nginx/atom/import-images/image1.jpg
• Add the corresponding slugs in the CSV for the target descriptions
• Use the task to import the CSV

Otherwise.... maybe just stick to 3 images at a time?

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

You received this message because you are subscribed to the Google Groups "AtoM Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c826c194-beef-4d4a-85a8-3df836087646n%40googlegroups.com.

[Gabriel Batista da Silva Nogueira]

Hi, 

it was working normally, but after the user indicated this error of sending the files, we started to check the errors.
Basically the user sends about 50 images, each one with more or less 2mb, I tried to do it with 5 of those images that this user usually sends, but even then it doesn't work, it continues with the error 504 gateway time out.

[Dan Gillean]

Hi again Gabriel, 

As I have suggested, using the command-line to handle your imports will be a much better solution. If the user doesn't have command-line access, then ideally they can send you a directory of images and the CSV for you to run the import. Further development needs to be done to make bulk digital object import work at scale via the browser. 

Some browsers allow for the default timeout limit to be changed, but I don't recommend this. If you want to try to extend some of AtoM's timeout times, see the PHP execution limits documentation - though I suspect that the browser will still be the bottleneck. 

Good luck! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/77f02e7b-9656-4f3b-8e10-554d3708250fn%40googlegroups.com.

[Gabriel Batista da Silva Nogueira]

Hi again Dan, I had tried but not tried to upload via command line but forgot to reply sorry.
The version we are using I believe is 2.6, we are waiting to update to the latest one, but at the moment the user needs to send a lot of files, in the current scenario we are trying to solve this error.

doing some research and looking at the error log:
[error] 278453#278453: *41603 FastCGI sent in stderr: "PHP message: Empty module and/or action after parsing the URL "/atom/index.php/xxxxxxxxxx/reports" (/)" while reading response header from upstream , client: xxx.xxx.xxx.xx, server: _, request: "GET /atom/index.php/xxxxxxxxxx/reports HTTP/1.1", upstream: "fastcgi://unix:/run/php7.2- fpm.atom.sock:", host: "xxx.xxxx.xx.xxx.xx"

which could end up influencing the timeout error.

[Dan Gillean]

Hi Gabriel, 

That error appears to be about the Reports module: 

• https://www.accesstomemory.org/docs/2.6/#reports-printing
  

To resolve it, I would first suggest that you: 

1) Make sure the job scheduler is properly configured and running as expected. Installation and configuration instructions for the job scheduler in 2.6 can be found here: 

• https://www.accesstomemory.org/docs/2.6/admin-manual/installation/asynchronous-jobs/
  

You can restart the job scheduler with: 

• sudo systemctl restart atom-worker

You can check the status of the atom-worker with:

• sudo systemctl status atom-worker

If your attempts to restart it are not working, then it may be that the worker has hit the restart limit configured during installation. This limit - which is 3 restarts within 24 hours - is there so that when the worker fails and attempts to automatically restart itself, it does not become caught in an endless retry loop if it cannot recover.  To reset the fail counter back to zero so you can restart the worker, use: 

• sudo systemctl reset-failed atom-worker

All of this is described in the Job Scheduler documentation at the end of this section, as well as here in the Troubleshooting docs: 

• https://www.accesstomemory.org/docs/2.6/admin-manual/maintenance/troubleshooting/#restarting-the-job-scheduler
  

After that, I would suggest running some of the common AtoM maintenance tasks, such as: 

• Clear the application cache
• Restart PHP-FPM
• Rebuild the nested set
• Generate slugs
• Populate the search index

Further instructions and links for all of these tasks can be found from the Troubleshooting page: 

• https://www.accesstomemory.org/docs/2.6/admin-manual/maintenance/troubleshooting/

While tasks such as the nested set rebuild and the slug generation should not necessarily be invovled in this issue, a) running them will not hurt, and b) they can resolve many common issues, including those we have not yet diagnosed which could be contributing to the empty module error you saw. 

After that, check to see if you're able to access the reports page that previously triggered the error. 

Note that none of this should be affecting timeouts when trying to import multiple digital objects via the browser! So don't expect that issue to change - but hopefully this will at least resolve any outstanding issues with the reports. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/39e66c82-a5b2-4f0d-aaf0-5597145df273n%40googlegroups.com.

[Gabriel Batista da Silva Nogueira]

I followed the examples above, corrected the error when sending files, but when I went to delete it, it also exceeded the time limit, and gave the error 504 time out, some requests that take time are returning this error.

I believe that any request that takes time will generate this error, I configured nginx and php.ini as I saw in some tutorials, but even so it was not possible to solve it, I don't know if I may have changed the wrong file the file I changed was /etc/nginx/nginx.conf, I used the command sudo systemctl reload nginx, sudo systemctl restart nginx, but currently not sure if these settings were useful or not.

http{

        proxy_read_timeout 3600;

        proxy_connect_timeout 3600;
        proxy_send_timeout 600;

        send_timeout 600;
        client_max_body_size 120M;
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;

[Dan Gillean]

Hi Gabriel, 

Remember in AtoM 2.x, we create a PHP pool with a number of these variables during installation, at /etc/php/7.2/fpm/pool.d/atom.conf

• https://www.accesstomemory.org/docs/2.6/admin-manual/installation/linux/ubuntu-bionic/#php
  

If you're still trying to configure longer timeout values on the server side, then I would suggest you start here, and in your php.ini file. 

We also define an Nginx configuration block at /etc/nginx/sites-available/atom, in this section of the docs: 

• https://www.accesstomemory.org/en/docs/2.6/admin-manual/installation/linux/ubuntu-bionic/#nginx
  

However, I'm less familiar with configuring Nginx and I'm not sure how much of what you've copied from a 9-year old forum thread will apply. I recommend reviewing current Nginx documentation to see what is and isn't possible. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/d9a69da3-4d34-4d81-804d-8f75e1216d8en%40googlegroups.com.

[Gabriel Batista da Silva Nogueira]

Hi again Dan, 

I'm updating to the latest (2.7.1 recomended version) I hope everything is back to normal, as soon as I'm back with the result.

[Gabriel Batista da Silva Nogueira]

Hi, again.

After updating, I did the test again.

The error persisted, I ran the tests as follows:
I tried to send 4 files totaling 958KB and it returned the 504 error.
I removed a file and it was left with 3 files and a total of 664KB and it sent normally.
for another test, I took the image that the client usually sends, which would be the one of 1.74MB, and sent it normally.
again I tried with 3 large files about 1.70MB each, and again everything worked fine.

I think the error is either in the number of files, but I could be wrong.

[Dan Gillean]

Hi again Gabriel, 

Interesting. I've tested myself, and was successfully able to upload 11 images at once, since all were small and the total combined size was about 1.1MB. I firmly believe this is a configuration issue. 

Did you review the settings in the PHP pool  I mentioned in my last message? There are all these values, which have the following defaults in the AtoM docs: 

php_admin_value[memory_limit] = 512M
php_admin_value[max_execution_time] = 120
php_admin_value[post_max_size] = 72M
php_admin_value[upload_max_filesize] = 64M
php_admin_value[max_file_uploads] = 10

There is also more information in the PHP execution limits documentation page I linked previously. That's all I can think of at the moment, but I will ask our team if they have further ideas. 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/0df23da3-ff35-49aa-95c2-11597b972570n%40googlegroups.com.

[Gabriel Batista da Silva Nogueira]

Hi,

I tried to use this solution but still the error occurred:

[error] 2152203#2152203: *28 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xx.x.x.xx, server: _, request: "POST /index.php/xxxxxxxxxx/informationobject/multiFileUpload HTTP/1.1", upstream: "fastcgi://unix:/run/php7.4-fpm.atom.sock", host: "xxx.xxx.xxx.xx", referrer: "http://xxx.xxx.xxx.xx/index.php/xxxxxxxxxx/informationobject/multiFileUpload"

I tried with 4 images of 250kb.

[Dan Gillean]

Hi Gabriel, 

Because I don't think this is a bug with AtoM's code, I have tried searching your most recent error message online generally, and found a few suggestions. For example, this StackExchange thread user suggests that the error "happens because your upstream takes too long to answer the request and NGINX thinks the upstream already failed in processing the request, so it responds with an error."

Most answers I have seen suggest trying to increase the various timeout settings you can add to the location block of your Nginx config, such as proxy_read_timeout, fastcgi_read_timeout, and proxy_connect_timeout. There's also another proposed solution in the StackExchange link above that also involves Nginx configuration. However, you will likely have the best success if you try to identify and resolve the root cause. What is making your webserver take so long to respond to requests, and/or limiting the number of concurrent requests?

I'm curious to know - does your server meet or exceed the suggestions our docs make in the Technical Requirements?

Processor: 2 vCPUs @ 2.3GHz
Memory: 7GB
Disk space (processing): 50GB at a minimum for AtoM’s core stack plus more storage would be required for supporting any substantial number of digital objects.

I am wondering if your server is perhaps under-resourced and this is causing the limitations. 

In any case, there is a limit to how much we can help you via the user forum for something that seems to be related to your specific deployment, so I would suggest that you do some research online with the error message, review the Nginx documentation, and consider posting in one of the Nginx forums if necessary. 

Good luck! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/4245ddf0-901c-451a-8dc2-4fad42882985n%40googlegroups.com.

[Gabriel Batista da Silva Nogueira]

Hi,

I managed to resolve the error.
Error: fastcgi was limited:

upstream timed out (110: Connection timed out) while reading response header from upstream, client: xx.x.x.xx, server: _, request: "POST /index.php/teste/informationobject/multiFileUpload HTTP/1.1", upstream: "fastcgi://unix:/run/php7.4-fpm.atom.sock", host: "192.0.0.2", referrer: "http://192.0.0.2/index.php/teste/informationobject/multiFileUpload"

Solution: I just accessed the "nginx.conf" file and added these settings:

http {
    ...
    fastcgi_connect_timeout 3600s;
    fastcgi_send_timeout 3600s;
    fastcgi_read_timeout 3600s;
    ...
}

leaving the fastcgi timeout with one hour in total, it may not be the best solution, but it was the one I was able to use to resolve the error.

after solving the error I tested again with 48 files, each about 1.7MB, total of 67MB, it took a long time, but it went correctly without returning the error 504.

now I have made it available again to the customer, now I am waiting for his response, until then I believe that the main 504 error has been resolved.

[Dan Gillean]

Glad you got it solved - thanks for letting us know what worked!

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/5b3e4658-3431-4323-9ab5-934f75b256a0n%40googlegroups.com.

[Gabriel Batista da Silva Nogueira]

Thanks, I believe that the solution you gave me about php.ini configuration has influenced the solution. thank you.