RestClient::RequestTimeout:Request Timeout when creating service

lasky

unread,

Sep 29, 2013, 3:44:49 AM9/29/13

to vcap...@cloudfoundry.org

Hello everybody:

I have added en enternal service with service broker(v1),but some action may take a long time during the provision.Therefore,when i am trying to create a service,it throws a exception(after one minute):HTTP exception: RestClient::RequestTimeout:Request Timeout.

the cloudcontroller.log:

[2013-09-29 15:00:25.991789] cc - pid=6728 tid=e987 fid=e990 ERROR -- Error talking to gateway: Can't decode gateway response. status code:504, response body:<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor="white">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n

[2013-09-29 15:00:25.992071] cc exception pid=6728 tid=e987 fid=e990 ERROR -- VCAP::Services::Api::ServiceGatewayClient::UnexpectedResponse<<Can't decode gateway response. status code:504, response body:<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor="white">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n:/home/hc/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/lib/ruby/gems/1.9.1/bundler/gems/vcap-common-

…………

the service-broker.log:

DEBUG -- Brokered service provisioned {:configuration=>{:plan=>"free"}, :credentials=>{"conn"=>nil}, :service_id=>"1aab2a3a-b8e3-4c93-a0b4-1cb3b69cf722"}

Actually,the service is still created,we just can not find it by vmc services

Any reply is appreciated, thanks!

Shannon Coen

unread,

Sep 30, 2013, 1:25:30 PM9/30/13

to vcap...@cloudfoundry.org

If provision requests don't return a success in 60s, Cloud Controller will timeout. Since Cloud Controller did not receive a success response with the expected payload, it will not write a row to the service_instances table in ccdb. For this reason, you will not see the service instance in response to 'vmc services'.

Cloud Foundry doesn't yet support long-running service instance provisioning. You can raise the timeout in Cloud Controller, but this not a great workaround.

The core CF Services team have stories in our backlog which will eventually enable asynchronous service instance provisioning, but we're probably several months away.

Best,

Shannon

Dr Nic Williams

unread,

Sep 30, 2013, 5:51:50 PM9/30/13

to vcap...@cloudfoundry.org, vcap...@cloudfoundry.org

Shannon, perhaps the long-running task model (and APIs) in bosh director are useful to borrow? I guess it is an implementation detail. But the API supports the return of a task_id for some API endpoints and then another endpoint for polling for status and any recent events since last poll.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

Shannon Coen

unread,

Sep 30, 2013, 7:29:16 PM9/30/13

to vcap...@cloudfoundry.org

Hi Nic,

Yes, I believe that was mentioned in our discussions. We're a ways out from tackling this; current plan is to start by making the cc api support asynchronous client UX, with enqueued synchronous jobs and retries to service brokers. Then we'll work on adding support to cc for asynchronous calls to service brokers and having cc poll for updates.

Best,

Shannon Coen

Product Manager, Cloud Foundry

Pivotal, Inc.

mobile: 415.640.0272

skype: shannoncoen

lasky

unread,

Oct 7, 2013, 11:11:15 PM10/7/13

to vcap...@cloudfoundry.org

Thanks,Shannon. I am so sorry for didn't reply timely. I have found this also,can we make it possible to support long-running service instance provisioning? I try to change the timeout to -1(never timeout) when post the request(/var/lib/gems/1.9.1/gems/vmc-0.3.18/lib/vmc/ client.rb #request). Then it tell me another error:

[2013-10-08 10:34:35.093389] cc - pid=10877 tid=ca32 fid=51ba ERROR -- Error talking to gateway: Can't decode gateway response. status code:504, response body:<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor="white">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n

[2013-10-08 10:34:35.093668] cc exception pid=10877 tid=ca32 fid=51ba ERROR -- VCAP::Services::Api::ServiceGatewayClient::UnexpectedResponse<<Can't decode gateway response. status code:504, response body:<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor="white">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n:/home/hc/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/lib/ruby/gems/1.9.1/bundler/gems/vcap-common-fd6b6d91b19c/lib/services/api/clients/service_gateway_client.rb:179:in `rescue in perform_request',/home/hc/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/lib/ruby/gems/1.9.1/bundler/gems/vcap-common-fd6b6d91b19c/lib/services/api/clients/service_gateway_client.rb:175:in `perform_request',/home/hc/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/lib/ruby/gems/1.9.1/bundler/gems/vcap-common-fd6b6d91b19c/lib/services/api/clients/service_gateway_client.rb:78:in `provision'

………………

Is there a link between this and nginx? I check the nginx_cc_error.log and find this:

[error] 1306#0: *434157 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.20.8.81, server: _, request: "POST /services HTTP/1.0", upstream: "http://unix:/tmp/cloud_controller.sock:/services", host: "api.vcap.me"

在 2013年10月1日星期二UTC+8上午1时25分30秒，Shannon Coen写道：

Shannon Coen

unread,

Oct 8, 2013, 6:23:20 PM10/8/13

to vcap...@cloudfoundry.org

Hello Iasky,

There's also a hardcoded 60s timeout in cloud controller that you would need to increase.

https://github.com/cloudfoundry/cloud_controller_ng/blob/master/app/models/services/service_broker/v1/http_client.rb#L4

Best,

Shannon

lasky

unread,

Oct 9, 2013, 11:18:53 PM10/9/13

to vcap...@cloudfoundry.org

Hi Shannon,

I am sorry but i can't find the services/service_broker/v1/http_client.rb in my cloudfoundry,it seems that it exits in the new version.

在 2013年10月9日星期三UTC+8上午6时23分20秒，Shannon Coen写道：

lasky

unread,

Oct 9, 2013, 11:19:13 PM10/9/13

to vcap...@cloudfoundry.org

Hi Shannon,

I am sorry but i can't find the services/service_broker/v1/http_client.rb in my cloudfoundry,it seems that it exits in the new version.

在 2013年10月9日星期三UTC+8上午6时23分20秒，Shannon Coen写道：

Hello Iasky,

Reply all

Reply to author

Forward