vmc push standalone app ERROR: 504 Gateway Time-out from nginx

[Hongliang Sun]

Hi, all,

      Recently，  I have to push a weblogic app to my old Cloud Foundry v1 cluster. And I push this app as a standalone app via vmc.

      Since my standalone app is somewhat large (about 600MB), I changed variable client_max_body_size in nginx.conf into 1024M to avoid too large entity error. And I changed three nginx.conf as all: one for load balance, one in the router(ruby version), last one at front of cloud_controller. As well, I modified the max size of app with 1024MB in ./cloud_controller/config/app_config.rb.

      After changing configurations above, I ran into a 504 Gateway Time-out error from nginx when pushing this app. When I checked the nginx.access.log,  i found:
       api.cloud.sel - [01/Aug/2014:14:39:15 +0800] "POST /apps/w41/application HTTP/1.1" 504 357 "-" "Ruby" 10.10.103.100 response_time:60.101s


      It seems that when nginx forwards the POST request to cloud controller, it will wait for 60s. If 60s expires, no matter whether cloud controller replies, nginx will response a "504 Gateway Time-out".

      And all my three nginx throw a 504 Gateway Time-out error with response time of 60s. So I wander if I missed some default configurations in nginx.conf, while this configuration will cause a 504 Gateway Time-out error.

      Actually I tried to make following variables larger than 60s: send_timeout, proxy_send_timeout, proxy_read_timeout, keepalive_timeout and so on. But vmc returns the same error 504 Gateway Time-out.

      Anything helps, Thanks in advance.

Hongliang Sun
Zhejiang University, China

     

[Noburou Taniguchi]

Hi, Hongliang,

I have a few question:

1. How many cloud controllers do you have in your environment?

2. Is there any output seemed to be related to this issue in a cloud controller's log?

If the answer to 2 is no, setting proxy_connect_timeout may extend the timeout duration. But I'm not sure whether it resolves the issue because it's strange for me to be unable to connect cloud controller.

So, another question is:

3. Is it ok to push an app in smaller size (like this [1]) to your environment?

[1] https://github.com/cloudfoundry/vcap-test-assets/tree/master/ruby/ruby19

2014年8月1日金曜日 17時33分47秒 UTC+9 Hongliang Sun:

[Hongliang Sun]

Hi，Noburou，

      Thanks for your reply.

      Let me answer your questions first:

1. I have two Cloud Controllers in my environment.

2.There is No Related issue in Cloud Controller's log. What's more, it took about 70s for cloud controller to receive and repack the app source, and nginx did not wait for 

the response from cloud controller and returned an 504 gateway timeout error. So the vmc received the error and would not send the command "start an app". And Cloud Controller

would not do the stager's job and starting job.

To your advice on resetting proxy_connect_timeout, i will try it today and tell you my experiments result.

3. We have pushed a lot of Java web apps(source code of .war) less than 100MB, but we have not pushed standalone apps(source code + runtime) less than 512MB. Now our standalone app is larger than 512MB.

Hongliang Sun

Zhejiang University, China