504 Gateway Timeout - nginx <-> tomcat7 errors

[Sanket Gupte]

Hello X-perts

We are using nginx as proxy server to tomcat7.
When uploading large image sets (anything larger than say 300 MB ) nginx reports 504 Gateway Timeout error .
So, we tried uploading different means, and this happens only when we send large compressed files, that xnat uncompresses .
For e.g. if we use the 'Compressed Uploader' form the xnat webapp, then it goes till 98% of the 'Extract/Review' progress bar, and then gives 504 Gateway Timeout.
We think tomcat7 is not returning status information to nginx while it extracts the imageset, coz the upload completes.

We tried tweaking a bunch parameters in nginx but that didnt help.

nginx conf file now looks like this:

server {
    listen 80;
    listen 443 ssl;
    server_name myserver.nimh.nih.gov;
    ssl_certificate /etc/nginx/ssl/xnat.crt;
    ssl_certificate_key /etc/nginx/ssl/xnat.key;

    client_max_body_size 0;

        # Our endpoint for tomcat reverse-proxy, assuming your endpoint java-servlet knows
        # how to handle http://localhost/gadgets  requests
        location / {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Url-Scheme $scheme;
            proxy_pass       http://localhost:8080;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_connect_timeout 300s;
            proxy_send_timeout    300s;
            proxy_read_timeout    300s;
            proxy_next_upstream_timeout 0;
            client_max_body_size 0;
        }
}

[Sanket Gupte]

Oops. Got sent by mistake without a closure :)
----

Anyone has any suggestions or sample configurations for nginx ?

We have been struggling with tweaking this for the last two weeks with no success.

Thanks

Sanket

--
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/GDw7jpkXn1s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Sanket Gupte]

Oops. Got sent by mistake without a closure :)
----

Anyone has any suggestions or sample configurations for nginx ?

We have been struggling with tweaking this for the last two weeks with no success.

Thanks

Sanket

[Herrick, Rick]

Have you tried increasing the various proxy timeout settings? I think you’re probably right that you’re getting no communication between Tomcat and nginx while Tomcat is busy processing the incoming file. It’s possible that a big file just takes too long to get extracted and processed and nginx gets tired of waiting around. Try increasing each of those settings to something like 400s or 500s. If you see an increase in the size of file you can upload before you hit the 504, then that’s your problem.

Really, there should be an option to do the uncompress asynchronously, but that’s currently not available.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 362-1882

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.

To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Sanket Gupte]

Yeah, as seen in the above code snipped from my nginx config, I tried tweaking several of those proxy timeouts. I set it to 500s now, still timing out.
And yes, small files do succeed.
So question, I know Rick you did anticipate this from my post, and did respond that asynchronous uncompression isn't available, but is there any way around it?

The only thing that I can think of at this point is, cutting up the big tar files into smaller baby tar files and then sending them.

[Adam Harding]

A related data point for posterity:

We have an old installation running locally behind mod_proxy on an old VM (4GB & 1 CPU unthrottled, low contention, mostly JVM heap, increased PermGen) and just ran into a timeout requesting ~91000 files totaling 2GB served over NFS. Grabbing via REST with format=zip succeeded without timeout. The default Apache config (in httpd.conf) has "Timeout 120" by default, and mod_proxy uses this unless overridden.

Setting "ProxyTimeout 480" allows requests like this to complete on this system; about 5.5 minutes for the case here.

Adam

On 5/2/16 3:04 PM, Herrick, Rick wrote:
> Have you tried increasing the various proxy timeout settings? I think you’re probably right that you’re getting no communication between Tomcat and nginx while Tomcat is busy processing the incoming file. It’s possible that a big file just takes too long to get extracted and processed and nginx gets tired of waiting around. Try increasing each of those settings to something like 400s or 500s. If you see an increase in the size of file you can upload before you hit the 504, then that’s your problem.
>
> Really, there should be an option to do the uncompress asynchronously, but that’s currently not available.
>
> --
> Rick Herrick
> Sr. Programmer/Analyst
> Neuroinformatics Research Group
> Washington University School of Medicine
> (314) 362-1882
>

> To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com<mailto:xnat_discussi...@googlegroups.com>.
> To post to this group, send email to xnat_di...@googlegroups.com<mailto:xnat_di...@googlegroups.com>.

[Sanket Gupte]

Yes. I had a talk with Rick and Tim at the workshop past June, and they recommended making the value really big. After experimenting with several values "600s"  worked for us.

> To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com<mailto:xnat_discussion+unsubscribe@googlegroups.com>.