cc blobstore using swift

299 views

Skip to first unread message

lyuyun

unread,

Jun 10, 2014, 4:43:06 AM6/10/14

to vcap...@cloudfoundry.org

I changed the droplet blobstore setting from nfs_server to swift, then push java app successfuly.

resource_pool:
  minimum_size: 65536
  maximum_size: 536870912
  resource_directory_key: cc-resources

  cdn:
    uri:
    key_pair_id:
    private_key: ""

  fog_connection: {"provider":"Local","local_root":"/var/vcap/nfs/shared"}

packages:
  app_package_directory_key: cc-packages
  max_package_size: 1073741824

  cdn:
    uri:
    key_pair_id:
    private_key: ""

  fog_connection: {"provider":"Local","local_root":"/var/vcap/nfs/shared"}

droplets:
  droplet_directory_key: cc-droplets

  cdn:
    uri:
    key_pair_id:
    private_key: ""

  fog_connection: {"provider":"OpenStack","openstack_username":"test:tester","openstack_api_key":"testing","openstack_auth_url":"http://9.91.13.90:8888/auth/v1.0"}

buildpacks:
  buildpack_directory_key: cc-buildpacks

  cdn:
    uri:
    key_pair_id:
    private_key: ""

  fog_connection: {"provider":"Local","local_root":"/var/vcap/nfs/shared"}

But if change all blobstore settings to swift, then push app failed. what is the problem, any help would be greatly appreciated, thanks.

resource_pool:
  ...

  fog_connection: {"provider":"OpenStack","openstack_username":"test:tester","openstack_api_key":"testing","openstack_auth_url":"http://9.91.13.90:8888/auth/v1.0"}

packages:
  ...

  fog_connection: {"provider":"OpenStack","openstack_username":"test:tester","openstack_api_key":"testing","openstack_auth_url":"http://9.91.13.90:8888/auth/v1.0"}

droplets:
  ...

  fog_connection: {"provider":"OpenStack","openstack_username":"test:tester","openstack_api_key":"testing","openstack_auth_url":"http://9.91.13.90:8888/auth/v1.0"}

buildpacks:
  ...

  fog_connection: {"provider":"OpenStack","openstack_username":"test:tester","openstack_api_key":"testing","openstack_auth_url":"http://9.91.13.90:8888/auth/v1.0"}

the error msg:

{"timestamp":1402426653.7889552,"message":"Request failed: 500: {\"code\"=>10001, \"description\"=>\"Storage must my instantiated with the :openstack_temp_url_key option\", \"error_code\"=>\"CF-ArgumentError\", \"backtrace\"=>[\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/openstack/requests/storage/get_object_https_url.rb:53:in `create_temp_url'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/openstack/models/storage/file.rb:78:in `url'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/blobstore/blob.rb:29:in `download_uri_for_file'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/blobstore/blob.rb:15:in `download_url'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/blobstore/client.rb:83:in `download_uri'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/blobstore/url_generator.rb:47:in `generate_download_url'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/blobstore/url_generator.rb:14:in `app_package_download_url'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_stager_task.rb:94:in `staging_request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_stager_task.rb:81:in `block in stage'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/cache/cf-message-bus-cb951d204bee/lib/eventmachine/schedule_sync.rb:16:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/cache/cf-message-bus-cb951d204bee/lib/eventmachine/schedule_sync.rb:16:in `block in schedule_sync'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:250:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:250:in `block in schedule'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:959:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:959:in `block in run_deferred_callbacks'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:956:in `times'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:956:in `run_deferred_callbacks'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/runner.rb:76:in `run!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:12:in `<main>'\"]}","log_level":"error","source":"cc.api","data":{"request_guid":"4faafdf1-e3a3-474d-5e6e-63ba8ff291b3::ed110868-e46b-4bcf-8b8c-3a2285daf11b"},"thread_id":40334060,"fiber_id":40342920,"process_id":21835,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sinatra/vcap.rb","lineno":53,"method":"block in registered"}

lyuyun

unread,

Jun 10, 2014, 4:57:24 AM6/10/14

to vcap...@cloudfoundry.org

i found the resolution in [1].

1. http://docs.cloudfoundry.org/deploying/openstack/using_swift_blobstore.html.

在 2014年6月10日星期二UTC+8下午4时43分06秒，lyuyun写道：

iamflying

unread,

Sep 24, 2014, 12:58:58 AM9/24/14

to vcap...@cloudfoundry.org

Hi lyuyun,

I followed the link to configure swift. But I got "Error downloading: HTTP status: 404". Any comments?

{"timestamp":1411532557.1222494,"message":"admin-buildpacks.download","log_level":"debug","source":"AdminBuildpackDownloader","data":{"buildpacks":[]},"thread_id":9781380,"fiber_id":28779840,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/staging/admin_buildpack_downloader.rb","lineno":15,"method":"download"}
{"timestamp":1411532557.1224577,"message":"{\"source_dir\"=>\"/tmp/unstaged\", \"dest_dir\"=>\"/tmp/staged\", \"cache_dir\"=>\"/tmp/cache\", \"environment\"=>{\"services\"=>[], \"resources\"=>{\"memory\"=>128, \"disk\"=>1024, \"fds\"=>16384}, \"environment\"=>[\"http_proxy=http://georgew...@137.172.74.222:58080\", \"https_proxy=http://georgew:Xsw2...@137.172.74.222:58080\", \"no_proxy=localhost,mycloud1.com,137.172.74.*\"], \"meta\"=>{\"console\"=>false}}, \"staging_info_name\"=>\"staging_info.yml\", \"buildpack_dirs\"=>[]}","log_level":"debug","source":"Dea::StagingTaskWorkspace","data":{},"thread_id":9781380,"fiber_id":28779840,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/staging/staging_task_workspace.rb","lineno":39,"method":"write_config_file"}
{"timestamp":1411532557.585997,"message":"staging.app-download.starting","log_level":"info","source":"Staging","data":{"app_guid":"52751ef9-f2fa-4afd-96e8-66cd818a6ef1","task_id":"35bbd3caca634f419fc872760b5246e7","uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/52/75/52751ef9%2Df2fa%2D4afd%2D96e8%2D66cd818a6ef1?temp_url_sig=b4a27ebc2f060f987e8202336748a48bf03cbb62&temp_url_expires=1411536156"},"thread_id":9781380,"fiber_id":28968480,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb","lineno":289,"method":"block in promise_app_download"}
{"timestamp":1411532557.5884266,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"Staging","data":{"app_guid":"52751ef9-f2fa-4afd-96e8-66cd818a6ef1","task_id":"35bbd3caca634f419fc872760b5246e7","droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/52/75/52751ef9%2Df2fa%2D4afd%2D96e8%2D66cd818a6ef1?temp_url_sig=b4a27ebc2f060f987e8202336748a48bf03cbb62&temp_url_expires=1411536156","droplet_http_status":404},"thread_id":9781380,"fiber_id":22459260,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411532557.588545,"message":"staging.app-download.failed","log_level":"debug","source":"Staging","data":{"app_guid":"52751ef9-f2fa-4afd-96e8-66cd818a6ef1","task_id":"35bbd3caca634f419fc872760b5246e7","duration":null,"error":"Error downloading: HTTP status: 404","backtrace":null},"thread_id":9781380,"fiber_id":22459260,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb","lineno":295,"method":"block (2 levels) in promise_app_download"}
{"timestamp":1411532557.5887954,"message":"staging.task.failed","log_level":"info","source":"Staging","data":{"app_guid":"52751ef9-f2fa-4afd-96e8-66cd818a6ef1","task_id":"35bbd3caca634f419fc872760b5246e7","error":"Error downloading: HTTP status: 404","backtrace":["/var/vcap/packages/dea_next/lib/dea/promise.rb:81:in `resolve'","/var/vcap/packages/dea_next/lib/dea/promise.rb:31:in `each'","/var/vcap/packages/dea_next/lib/dea/promise.rb:31:in `run_in_parallel'","/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb:534:in `resolve_staging_setup'","/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb:47:in `block in start'","/var/vcap/packages/dea_next/lib/dea/promise.rb:92:in `call'","/var/vcap/packages/dea_next/lib/dea/promise.rb:92:in `block in run'"]},"thread_id":9781380,"fiber_id":28779920,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb","lineno":55,"method":"block in start"}
{"timestamp":1411532557.5893118,"message":"Saving snapshot took: 0.000s","log_level":"debug","source":"Snapshot","data":{},"thread_id":9781380,"fiber_id":28779920,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/snapshot.rb","lineno":40,"method":"save"}
{"timestamp":1411532557.7280037,"message":"Caught exception: Error downloading: HTTP status: 404","log_level":"warn","source":"Dea::Promise","data":{"backtrace":["/var/vcap/packages/dea_next/lib/dea/promise.rb:81:in `resolve'","/var/vcap/packages/dea_next/lib/dea/promise.rb:31:in `each'","/var/vcap/packages/dea_next/lib/dea/promise.rb:31:in `run_in_parallel'","/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb:534:in `resolve_staging_setup'","/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb:47:in `block in start'","/var/vcap/packages/dea_next/lib/dea/promise.rb:92:in `call'","/var/vcap/packages/dea_next/lib/dea/promise.rb:92:in `block in run'"]},"thread_id":9781380,"fiber_id":28779920,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/promise.rb","lineno":22,"method":"rescue in block in resolve"}
{"timestamp":1411532575.28748,"message":"varz access","log_level":"debug","source":"Dea::Bootstrap","data":{},"thread_id":9781380,"fiber_id":22459260,"process_id":1400,"file":"/var/vcap/packages/dea_next/vendor/cache/vcap-common-4ad156de3fcb/lib/vcap/component.rb","lineno":22,"method":"call"}

Ferran Rodenas

unread,

Sep 24, 2014, 1:52:56 AM9/24/14

to vcap...@cloudfoundry.org

Can you check if your Swift endpoint is at "http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/"? Not sure if this is your case, but I've observed that when building the temp url, somewhere is missing the port, in my case, the swift endpoint listens on 8080 and not the implicit 80.

Snippet from your log: "http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/52/75/52751ef9%2Df2fa%2D4afd%2D96e8%2D66cd818a6ef1?temp_url_sig=b4a27ebc2f060f987e8202336748a48bf03cbb62&temp_url_expires=1411536156""

- Ferdy

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/a67049c3-4281-4fa6-850c-26a06e5880ec%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

iamflying

unread,

Sep 24, 2014, 10:01:17 PM9/24/14

to vcap...@cloudfoundry.org

Thanks for pointing out the issue. I checked my openstack. Swift port is 8080 by default. I guess cf should use the default port. Not sure why cf chose port 80. Do you know where and how to specify the port in cf manifest?

On Wednesday, September 24, 2014 3:52:56 PM UTC+10, ferdy wrote:

Can you check if your Swift endpoint is at "http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/"? Not sure if this is your case, but I've observed that when building the temp url, somewhere is missing the port, in my case, the swift endpoint listens on 8080 and not the implicit 80.

Snippet from your log: "http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/52/75/52751ef9%2Df2fa%2D4afd%2D96e8%2D66cd818a6ef1?temp_url_sig=b4a27ebc2f060f987e8202336748a48bf03cbb62&temp_url_expires=1411536156""

- Ferdy

2014-09-23 21:58 GMT-07:00 iamflying <guangc...@gmail.com>:

Hi lyuyun,

I followed the link to configure swift. But I got "Error downloading: HTTP status: 404". Any comments?

{"timestamp":1411532557.1222494,"message":"admin-buildpacks.download","log_level":"debug","source":"AdminBuildpackDownloader","data":{"buildpacks":[]},"thread_id":9781380,"fiber_id":28779840,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/staging/admin_buildpack_downloader.rb","lineno":15,"method":"download"}

{"timestamp":1411532557.1224577,"message":"{\"source_dir\"=>\"/tmp/unstaged\", \"dest_dir\"=>\"/tmp/staged\", \"cache_dir\"=>\"/tmp/cache\", \"environment\"=>{\"services\"=>[], \"resources\"=>{\"memory\"=>128, \"disk\"=>1024, \"fds\"=>16384}, \"environment\"=>[\"http_proxy=http://georgewXsw2cde3@137.172.74.222:58080\", \"https_proxy=http://georgew:Xsw2...@137.172.74.222:58080\", \"no_proxy=localhost,mycloud1.com,137.172.74.*\"], \"meta\"=>{\"console\"=>false}}, \"staging_info_name\"=>\"staging_info.yml\", \"buildpack_dirs\"=>[]}","log_level":"debug","source":"Dea::StagingTaskWorkspace","data":{},"thread_id":9781380,"fiber_id":28779840,"process_id":1400,"file":"/var/vcap/packages/dea_next/lib/dea/staging/staging_task_workspace.rb","lineno":39,"method":"write_config_file"}

iamflying

unread,

Sep 24, 2014, 10:15:46 PM9/24/14

to vcap...@cloudfoundry.org

I rechecked the swift log from /var/log/message. I found the proxy server can get the right URL with port 8080. But it seems cloudfoundry-droplets and cloudfoundry-buildpacks were empty. Any idea?

Sep 25 12:11:33 localhost container-updater: Begin container update sweep
Sep 25 12:11:33 localhost account-server: 137.172.74.130 - - [25/Sep/2014:02:11:33 +0000] "PUT /swiftloopback/230167/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages" 201 - "-" "-" "container-updater 15112" 0.0005 ""
Sep 25 12:11:33 localhost container-updater: Container update sweep completed: 0.06s

Sep 25 12:11:38 localhost @container-server: 137.172.74.130 - - [25/Sep/2014:02:11:38 +0000] "GET /swiftloopback/54673/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages" 200 2 "txe82eeb0ae152423c9d9f5-00542379da" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages?format=json" "proxy-server 18650" 0.0011
Sep 25 12:11:38 localhost @container-server: 137.172.74.130 - - [25/Sep/2014:02:11:38 +0000] "PUT /swiftloopback/54673/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/09/c7/09c72aaf-c92f-4578-b3c4-998ae9e6ca12" 201 - "tx37b92bd082994104bbfef-00542379da" "PUT http://137.172.74.130:8080/swiftloopback/219426/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/09/c7/09c72aaf-c92f-4578-b3c4-998ae9e6ca12" "obj-server 15202" 0.0003
Sep 25 12:11:38 localhost object-server: 137.172.74.130 - - [25/Sep/2014:02:11:38 +0000] "PUT /swiftloopback/219426/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/09/c7/09c72aaf-c92f-4578-b3c4-998ae9e6ca12" 201 - "PUT http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/09/c7/09c72aaf-c92f-4578-b3c4-998ae9e6ca12" "tx37b92bd082994104bbfef-00542379da" "proxy-server 18652" 0.0427
Sep 25 12:11:41 localhost dnsmasq-dhcp[13014]: DHCPREQUEST(tapb2a5b8ae-a3) 10.0.0.18 fa:16:3e:86:b5:58
Sep 25 12:11:41 localhost dnsmasq-dhcp[13014]: DHCPACK(tapb2a5b8ae-a3) 10.0.0.18 fa:16:3e:86:b5:58 host-10-0-0-18
Sep 25 12:11:43 localhost @container-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "GET /swiftloopback/54673/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages" 200 196 "txe8280561238745e9a92cf-00542379df" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages?format=json" "proxy-server 18649" 0.0084
Sep 25 12:11:43 localhost object-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "HEAD /swiftloopback/219426/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/09/c7/09c72aaf-c92f-4578-b3c4-998ae9e6ca12" 200 775 "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/09/c7/09c72aaf-c92f-4578-b3c4-998ae9e6ca12" "txabb5a2e437d14280949df-00542379df" "proxy-server 18651" 0.0003
Sep 25 12:11:43 localhost @container-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "GET /swiftloopback/185089/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets" 200 2 "txc82fa35afe1241ca9223f-00542379df" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets?format=json" "proxy-server 18651" 0.0010
Sep 25 12:11:43 localhost object-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "HEAD /swiftloopback/32462/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets/buildpack_cache/09/c7/09c72aaf-c92f-4578-b3c4-998ae9e6ca12" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets/buildpack_cache/09/c7/09c72aaf-c92f-4578-b3c4-998ae9e6ca12" "tx8b6581a46dd84b48b0c2c-00542379df" "proxy-server 18652" 0.0002
Sep 25 12:11:43 localhost @container-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "GET /swiftloopback/76619/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks" 200 2 "txd512caa7645c4f1b85358-00542379df" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks?format=json" "proxy-server 18650" 0.0009
Sep 25 12:11:43 localhost object-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "HEAD /swiftloopback/68328/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/5d/49/5d49bcec-069d-48d4-9a6e-9efaf0c1aa4b_9a3079cd1c5ffce848d9aa492f73c42b48791d2e" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/5d/49/5d49bcec-069d-48d4-9a6e-9efaf0c1aa4b_9a3079cd1c5ffce848d9aa492f73c42b48791d2e" "tx0cd669077f3846398d577-00542379df" "proxy-server 18650" 0.0002
Sep 25 12:11:43 localhost object-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "HEAD /swiftloopback/111278/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/95/09/9509e809-845c-410d-8d2c-ae6e37c8eef9_0eac241dcc6b8def5066447ae62411bfa70572e0" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/95/09/9509e809-845c-410d-8d2c-ae6e37c8eef9_0eac241dcc6b8def5066447ae62411bfa70572e0" "txb216e09c02f34c19bca6b-00542379df" "proxy-server 18652" 0.0001
Sep 25 12:11:43 localhost object-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "HEAD /swiftloopback/141197/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/6e/3e/6e3e4017-881f-47a9-bebb-14588bfeb432_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/6e/3e/6e3e4017-881f-47a9-bebb-14588bfeb432_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b" "txbdc01696d3244ba0b8fe3-00542379df" "proxy-server 18646" 0.0001
Sep 25 12:11:43 localhost object-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "HEAD /swiftloopback/175213/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/bd/8e/bd8ecaef-c06e-4a47-8472-4a86e3083685_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/bd/8e/bd8ecaef-c06e-4a47-8472-4a86e3083685_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350" "txa0c9cc2a46734dc4b0367-00542379df" "proxy-server 18647" 0.0002
Sep 25 12:11:43 localhost object-server: 137.172.74.130 - - [25/Sep/2014:02:11:43 +0000] "HEAD /swiftloopback/156150/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/92/79/9279b7e0-836c-47bf-ad76-cc0e40f7e904_5ce1f73eeb2359111ee57566f760b650215054e6" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/92/79/9279b7e0-836c-47bf-ad76-cc0e40f7e904_5ce1f73eeb2359111ee57566f760b650215054e6" "txba30dd9538ee42dda17ff-00542379df" "proxy-server 18646" 0.0002
Sep 25 12:11:46 localhost container-replicator: Beginning replication run
Sep 25 12:11:46 localhost container-replicator: Replication run OVER
Sep 25 12:11:46 localhost container-replicator: Attempted to replicate 5 dbs in 0.06802 seconds (73.50783/s)
Sep 25 12:11:46 localhost container-replicator: Removed 0 dbs
Sep 25 12:11:46 localhost container-replicator: 0 successes, 0 failures
Sep 25 12:11:46 localhost container-replicator: no_change:0 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0
Sep 25 12:11:49 localhost account-replicator: Beginning replication run
Sep 25 12:11:49 localhost account-replicator: Replication run OVER
Sep 25 12:11:49 localhost account-replicator: Attempted to replicate 1 dbs in 0.07411 seconds (13.49434/s)
Sep 25 12:11:49 localhost account-replicator: Removed 0 dbs
Sep 25 12:11:49 localhost account-replicator: 0 successes, 0 failures
Sep 25 12:11:49 localhost account-replicator: no_change:0 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0

ferdy

unread,

Sep 25, 2014, 1:21:37 PM9/25/14

to vcap...@cloudfoundry.org

(Resending this, as my reply seems it got lost)

The problem is that CC stores correctly the application bits at the swift blobstore (your cloudfoundry-packages directory), but when it creates the swift temp url for the stager it omits the port. This happens because CC still uses an old version of fog (1.18.0) that omits the port, this has been fixed in fog 1.22.0 (https://github.com/fog/fog/commit/a3da07b6a69ea69b66cf41fa3740396b8ced1b84).

Would you mind creating an issue at the cloud_controller_ng github repo https://github.com/cloudfoundry/cloud_controller_ng?

- Ferdy

Ferran Rodenas

unread,

Sep 25, 2014, 2:27:36 PM9/25/14

to vcap...@cloudfoundry.org

would you mind creating an issue at the cloud_controller_ng github repo https://github.com/cloudfoundry/cloud_controller_ng?

- Ferdy

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/38ea9c68-7ccb-45a7-911d-bbe9b86655d7%40cloudfoundry.org.

iamflying

unread,

Sep 26, 2014, 1:37:06 AM9/26/14

to vcap...@cloudfoundry.org

Hi Ferdy,

Thanks for your kind suggestion. But I found it is not port issue. Maybe, I made you misunderstood by providing the log. Sorry for that.

From the dea_next log, I think, my issue is that DEA can not download the buildpacks during staging even though the buildpacks exist in Swift. I found the requests to download buildpacks sent by DEA seem correct as I can find all the http status in Swift log are 200. (marked pairs in color for example).

My questions are:
1. how to debug DEA to identify why it failed? I injected some output in ruby code, but I can not find it in dea_next.log.
2. Any special steps/config to enable Swift in cf besides the guild http://docs.cloudfoundry.org/deploying/openstack/using_swift_blobstore.html?

///////Paste more Swift information.
[root@ror-openstack ~(keystone_demo)]# swift list -l
    4      5225688 2014-09-23 05:25:27 cloudfoundry-buildpacks
    0            0 2014-09-23 05:22:19 cloudfoundry-droplets
    1       544455 2014-09-23 05:22:14 cloudfoundry-packages
    1       440589 2014-09-23 05:25:21 cloudfoundry-resources
    6      6210732
[root@ror-openstack ~(keystone_demo)]#

[root@ror-openstack ~(keystone_demo)]# swift list cloudfoundry-buildpacks
36/56/365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6
a2/cb/a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b
c2/0d/c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350
e3/97/e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0
[root@ror-openstack ~(keystone_demo)]#

//////////////////dea_next.log

{"timestamp":1411703148.986129,"message":"nats.message.received","log_level":"debug","source":"Dea::Nats","data":{"subject":"staging.0-18f300ce5fc64365a62518ba0f97e484.start","data":{"app_id":"ee6b59f6-892d-4594-b813-2ae0119ad257","task_id":"943463e4f1cb47d7b4c2d73e3c1518a1","properties":{"services":[],"resources":{"memory":256,"disk":1024,"fds":16384},"environment":["http_proxy=http://georgew...@137.172.74.222:58080","https_proxy=http://georgew:Xsw2...@137.172.74.222:58080","no_proxy=localhost,mycloud1.com,137.172.74.*"],"meta":{"console":false}},"download_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748","upload_uri":"http://upload:c1ou...@10.0.0.74:9022/staging/droplets/ee6b59f6-892d-4594-b813-2ae0119ad257/upload","buildpack_cache_download_uri":null,"buildpack_cache_upload_uri":"http://upload:c1ou...@10.0.0.74:9022/staging/buildpack_cache/ee6b59f6-892d-4594-b813-2ae0119ad257/upload","start_message":{"droplet":"ee6b59f6-892d-4594-b813-2ae0119ad257","name":"calendar","uris":["calendar.mycloud.com"],"prod":false,"sha1":null,"executableFile":"deprecated","executableUri":null,"version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","services":[],"limits":{"mem":256,"disk":1024,"fds":16384},"cc_partition":"default","env":["http_proxy=http://georgew...@137.172.74.222:58080","https_proxy=http://georgew:Xsw2...@137.172.74.222:58080","no_proxy=localhost,mycloud1.com,137.172.74.*"],"console":false,"debug":null,"start_command":null,"health_check_timeout":null,"vcap_application":{"limits":{"mem":256,"disk":1024,"fds":16384},"application_version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","application_name":"calendar","application_uris":["calendar.mycloud.com"],"version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","name":"calendar","space_name":"aPaaS","space_id":"973ceec8-c6ee-4a7b-8379-7917406a8c16","uris":["calendar.mycloud.com"],"users":null},"index":0},"admin_buildpacks":[{"key":"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748"},{"key":"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748"},{"key":"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748"},{"key":"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748"}]}},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/nats.rb","lineno":146,"method":"handle_incoming_message"}
{"timestamp":1411703148.9870725,"message":"staging.handle.start","log_level":"debug","source":"Staging","data":{"app_guid":"ee6b59f6-892d-4594-b813-2ae0119ad257","request":"#<StagingMessage:0x00000002fd62e8 @message={\"app_id\"=>\"ee6b59f6-892d-4594-b813-2ae0119ad257\", \"task_id\"=>\"943463e4f1cb47d7b4c2d73e3c1518a1\", \"properties\"=>{\"services\"=>[], \"resources\"=>{\"memory\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"environment\"=>[\"http_proxy=http://georgew...@137.172.74.222:58080\", \"https_proxy=http://georgew:Xsw2...@137.172.74.222:58080\", \"no_proxy=localhost,mycloud1.com,137.172.74.*\"], \"meta\"=>{\"console\"=>false}}, \"download_uri\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748\", \"upload_uri\"=>\"http://upload:c1ou...@10.0.0.74:9022/staging/droplets/ee6b59f6-892d-4594-b813-2ae0119ad257/upload\", \"buildpack_cache_download_uri\"=>nil, \"buildpack_cache_upload_uri\"=>\"http://upload:c1ou...@10.0.0.74:9022/staging/buildpack_cache/ee6b59f6-892d-4594-b813-2ae0119ad257/upload\", \"start_message\"=>{\"droplet\"=>\"ee6b59f6-892d-4594-b813-2ae0119ad257\", \"name\"=>\"calendar\", \"uris\"=>[\"calendar.mycloud.com\"], \"prod\"=>false, \"sha1\"=>nil, \"executableFile\"=>\"deprecated\", \"executableUri\"=>nil, \"version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"services\"=>[], \"limits\"=>{\"mem\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"cc_partition\"=>\"default\", \"env\"=>[\"http_proxy=http://georgew...@137.172.74.222:58080\", \"https_proxy=http://georgew:Xsw2...@137.172.74.222:58080\", \"no_proxy=localhost,mycloud1.com,137.172.74.*\"], \"console\"=>false, \"debug\"=>nil, \"start_command\"=>nil, \"health_check_timeout\"=>nil, \"vcap_application\"=>{\"limits\"=>{\"mem\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"application_version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"application_name\"=>\"calendar\", \"application_uris\"=>[\"calendar.mycloud.com\"], \"version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"name\"=>\"calendar\", \"space_name\"=>\"aPaaS\", \"space_id\"=>\"973ceec8-c6ee-4a7b-8379-7917406a8c16\", \"uris\"=>[\"calendar.mycloud.com\"], \"users\"=>nil}, \"index\"=>0}, \"admin_buildpacks\"=>[{\"key\"=>\"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748\"}, {\"key\"=>\"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748\"}, {\"key\"=>\"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748\"}, {\"key\"=>\"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748\"}]}>"},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/responders/staging.rb","lineno":43,"method":"handle"}
{"timestamp":1411703148.9888422,"message":"admin-buildpacks.download","log_level":"debug","source":"AdminBuildpackDownloader","data":{"buildpacks":[{"url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748","key":"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0"},{"url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748","key":"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b"},{"url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748","key":"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350"},{"url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748","key":"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6"}]},"thread_id":9672840,"fiber_id":24603760,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/staging/admin_buildpack_downloader.rb","lineno":15,"method":"download"}
{"timestamp":1411703148.9920592,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"AdminBuildpackDownloader","data":{"droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411703148.992385,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"AdminBuildpackDownloader","data":{"droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411703148.9926739,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"AdminBuildpackDownloader","data":{"droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411703148.992916,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"AdminBuildpackDownloader","data":{"droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411703149.3930206,"message":"staging.app-download.starting","log_level":"info","source":"Staging","data":{"app_guid":"ee6b59f6-892d-4594-b813-2ae0119ad257","task_id":"943463e4f1cb47d7b4c2d73e3c1518a1","uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748"},"thread_id":9672840,"fiber_id":14408360,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb","lineno":289,"method":"block in promise_app_download"}
{"timestamp":1411703149.3942873,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"Staging","data":{"app_guid":"ee6b59f6-892d-4594-b813-2ae0119ad257","task_id":"943463e4f1cb47d7b4c2d73e3c1518a1","droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}

///////////////////// /var/log/message
[root@ror-openstack ~(keystone_demo)]# tail -f /var/log/messages | grep "AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7"
Sep 26 13:47:54 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources (txn: tx86af1739fe854c70a93a7-005424e1ea)
Sep 26 13:47:54 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources (txn: tx86af1739fe854c70a93a7-005424e1ea)
Sep 26 13:47:54 localhost account-server: 137.172.74.130 - - [26/Sep/2014:03:47:54 +0000] "HEAD /swiftloopback/230167/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7" 204 - "tx86af1739fe854c70a93a7-005424e1ea" "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7" "proxy-server 12118" 0.0009 ""
Sep 26 13:47:54 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:47:54 +0000] "GET /swiftloopback/7225/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" 404 - "tx86af1739fe854c70a93a7-005424e1ea" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources?format=json" "proxy-server 12118" 0.0007
Sep 26 13:47:54 localhost proxy-server: STDOUT: swift.account/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7 (txn: tx3274371543384b8dbaaa0-005424e1ea)
Sep 26 13:47:54 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources (txn: tx3274371543384b8dbaaa0-005424e1ea)
Sep 26 13:47:54 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources (txn: tx3274371543384b8dbaaa0-005424e1ea)
Sep 26 13:47:54 localhost account-server: 137.172.74.130 - - [26/Sep/2014:03:47:54 +0000] "PUT /swiftloopback/230167/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" 201 - "tx3274371543384b8dbaaa0-005424e1ea" "PUT http://137.172.74.130:6001/swiftloopback/7225/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" "container-server 12113" 0.0007 ""
Sep 26 13:47:54 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:47:54 +0000] "PUT /swiftloopback/7225/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" 201 - "tx3274371543384b8dbaaa0-005424e1ea" "PUT http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" "proxy-server 12118" 0.0110
Sep 26 13:47:54 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0 (txn: tx2cd6c13261794a86a2f3e-005424e1ea)
Sep 26 13:47:54 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43%2Fb4%2F43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0 (txn: tx2cd6c13261794a86a2f3e-005424e1ea)
Sep 26 13:47:54 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:47:54 +0000] "HEAD /swiftloopback/7225/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" 204 - "tx2cd6c13261794a86a2f3e-005424e1ea" "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" "proxy-server 12118" 0.0010
Sep 26 13:47:54 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:47:54 +0000] "HEAD /swiftloopback/84976/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0" "tx2cd6c13261794a86a2f3e-005424e1ea" "proxy-server 12118" 0.0002
Sep 26 13:47:57 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources (txn: tx3ef55fea22234081bcbe3-005424e1ed)
Sep 26 13:47:57 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources (txn: tx3ef55fea22234081bcbe3-005424e1ed)
Sep 26 13:47:57 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:47:57 +0000] "GET /swiftloopback/7225/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" 200 2 "tx3ef55fea22234081bcbe3-005424e1ed" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources?format=json" "proxy-server 12125" 0.0011
Sep 26 13:47:57 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0 (txn: tx716695ea9b5d4a4d93818-005424e1ed)
Sep 26 13:47:57 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43%2Fb4%2F43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0 (txn: tx716695ea9b5d4a4d93818-005424e1ed)
Sep 26 13:47:57 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:47:57 +0000] "HEAD /swiftloopback/84976/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0" "tx716695ea9b5d4a4d93818-005424e1ed" "proxy-server 12125" 0.0002
Sep 26 13:47:57 localhost proxy-server: STDOUT: swift.account/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7 (txn: tx7f10e04acd5b4b0ab5ea7-005424e1ed)
Sep 26 13:47:57 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0 (txn: tx7f10e04acd5b4b0ab5ea7-005424e1ed)
Sep 26 13:47:57 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43%2Fb4%2F43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0 (txn: tx7f10e04acd5b4b0ab5ea7-005424e1ed)
Sep 26 13:47:57 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:47:57 +0000] "PUT /swiftloopback/7225/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0" 201 - "tx7f10e04acd5b4b0ab5ea7-005424e1ed" "PUT http://137.172.74.130:8080/swiftloopback/84976/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0" "obj-server 12135" 0.0003
Sep 26 13:47:57 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:47:57 +0000] "PUT /swiftloopback/84976/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0" 201 - "PUT http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources/43/b4/43b42733298e82b1e6da8aee6c4d3e6c4f58f5b0" "tx7f10e04acd5b4b0ab5ea7-005424e1ed" "proxy-server 12125" 0.0163
Sep 26 13:47:57 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages (txn: txd0f59e90f4674d11a4ddf-005424e1ed)
Sep 26 13:47:57 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages (txn: txd0f59e90f4674d11a4ddf-005424e1ed)
Sep 26 13:47:58 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:47:58 +0000] "GET /swiftloopback/54673/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages" 200 2 "txd0f59e90f4674d11a4ddf-005424e1ed" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages?format=json" "proxy-server 12125" 0.0010
Sep 26 13:47:58 localhost proxy-server: STDOUT: swift.account/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7 (txn: txd3024a42e28744689261e-005424e1ee)
Sep 26 13:47:58 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257 (txn: txd3024a42e28744689261e-005424e1ee)
Sep 26 13:47:58 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee%2F6b%2Fee6b59f6-892d-4594-b813-2ae0119ad257 (txn: txd3024a42e28744689261e-005424e1ee)
Sep 26 13:47:58 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:47:58 +0000] "PUT /swiftloopback/54673/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257" 201 - "txd3024a42e28744689261e-005424e1ee" "PUT http://137.172.74.130:8080/swiftloopback/41578/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257" "obj-server 12135" 0.0003
Sep 26 13:47:58 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:47:58 +0000] "PUT /swiftloopback/41578/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257" 201 - "PUT http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257" "txd3024a42e28744689261e-005424e1ee" "proxy-server 12126" 0.0124
Sep 26 13:48:00 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages (txn: tx923f5a83225246eda59bd-005424e1f0)
Sep 26 13:48:00 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages (txn: tx923f5a83225246eda59bd-005424e1f0)
Sep 26 13:48:00 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "GET /swiftloopback/54673/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages" 200 199 "tx923f5a83225246eda59bd-005424e1f0" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages?format=json" "proxy-server 12117" 0.0087
Sep 26 13:48:00 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257 (txn: tx47ba9dc714e0482c9c4c7-005424e1f0)
Sep 26 13:48:00 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee%2F6b%2Fee6b59f6-892d-4594-b813-2ae0119ad257 (txn: tx47ba9dc714e0482c9c4c7-005424e1f0)
Sep 26 13:48:00 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "HEAD /swiftloopback/41578/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257" 200 544455 "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257" "tx47ba9dc714e0482c9c4c7-005424e1f0" "proxy-server 12118" 0.0003
Sep 26 13:48:00 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets (txn: txbf60eaa1fd9c44d1aaa59-005424e1f0)
Sep 26 13:48:00 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets (txn: txbf60eaa1fd9c44d1aaa59-005424e1f0)
Sep 26 13:48:00 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "GET /swiftloopback/185089/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets" 200 2 "txbf60eaa1fd9c44d1aaa59-005424e1f0" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets?format=json" "proxy-server 12120" 0.0010
Sep 26 13:48:00 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets/buildpack_cache/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257 (txn: tx88555d41ccfc4546b02c6-005424e1f0)
Sep 26 13:48:00 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets/buildpack_cache%2Fee%2F6b%2Fee6b59f6-892d-4594-b813-2ae0119ad257 (txn: tx88555d41ccfc4546b02c6-005424e1f0)
Sep 26 13:48:00 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "HEAD /swiftloopback/102061/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets/buildpack_cache/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257" 404 - "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-droplets/buildpack_cache/ee/6b/ee6b59f6-892d-4594-b813-2ae0119ad257" "tx88555d41ccfc4546b02c6-005424e1f0" "proxy-server 12120" 0.0002
Sep 26 13:48:00 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks (txn: txbf256aad9f3d4122a8d3b-005424e1f0)
Sep 26 13:48:00 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks (txn: txbf256aad9f3d4122a8d3b-005424e1f0)
Sep 26 13:48:00 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "GET /swiftloopback/76619/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks" 200 960 "txbf256aad9f3d4122a8d3b-005424e1f0" "GET http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks?format=json" "proxy-server 12124" 0.0011
Sep 26 13:48:00 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0 (txn: tx2ea48fc61d2040a3be076-005424e1f0)
Sep 26 13:48:00 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3%2F97%2Fe397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0 (txn: tx2ea48fc61d2040a3be076-005424e1f0)
Sep 26 13:48:00 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "HEAD /swiftloopback/132055/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0" 200 136315 "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0" "tx2ea48fc61d2040a3be076-005424e1f0" "proxy-server 12125" 0.0003
Sep 26 13:48:00 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b (txn: tx139ae8aa2a154d3eb0277-005424e1f0)
Sep 26 13:48:00 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2%2Fcb%2Fa2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b (txn: tx139ae8aa2a154d3eb0277-005424e1f0)
Sep 26 13:48:00 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "HEAD /swiftloopback/152488/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b" 200 27720 "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b" "tx139ae8aa2a154d3eb0277-005424e1f0" "proxy-server 12125" 0.0002
Sep 26 13:48:00 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "HEAD /swiftloopback/87235/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350" 200 228807 "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350" "tx14c5d507b51544ffaeb62-005424e1f0" "proxy-server 12125" 0.0002
Sep 26 13:48:00 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6 (txn: tx0526cd8312c7415d9e5b9-005424e1f0)
Sep 26 13:48:00 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36%2F56%2F365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6 (txn: tx0526cd8312c7415d9e5b9-005424e1f0)
Sep 26 13:48:00 localhost object-server: 137.172.74.130 - - [26/Sep/2014:03:48:00 +0000] "HEAD /swiftloopback/247309/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6" 200 4832846 "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6" "tx0526cd8312c7415d9e5b9-005424e1f0" "proxy-server 12126" 0.0002

Ferran Rodenas

unread,

Sep 26, 2014, 1:07:34 PM9/26/14

to vcap...@cloudfoundry.org

Have you tried to manually curl that URL from inside the DEA to see what's the reponse status code?

- Ferdy

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/2028b305-aac5-471f-8060-ffe8870d033e%40cloudfoundry.org.

iamflying

unread,

Sep 28, 2014, 10:40:21 PM9/28/14

to vcap...@cloudfoundry.org

Ferdy,

You seems right. I tried the URL logged in dea_next.log with curl, like "http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748". I got 404 error.

You mentioned it will resolve the port issue by updating fog version on DEA VM.However, the issue still exists after I updated fog to 1.23. So, I submitted an issue for the swift download issue. (https://github.com/cloudfoundry/cloud_controller_ng/issues/299)

Meantime, if I added the port in that URL, the request will reach to Swift server. but I got another error 401 Unauthorized. That's because expired time generated on DEA was always earlier (almost 30 seconds) than the time generated by openstack server. Any idea how to sync the time between CF VMs and openstack server?

Sep 29 22:38:00 localhost proxy-server: STDOUT: current time:1411994280.23 (txn: tx75306df060f745fcae309-00542952a8)
Sep 29 22:38:00 localhost proxy-server: STDOUT: temp_url_expires:1411961884 (txn: tx75306df060f745fcae309-00542952a8)

{"timestamp":1411703148.986129,"message":"nats.message.received","log_level":"debug","source":"Dea::Nats","data":{"subject":"staging.0-18f300ce5fc64365a62518ba0f97e484.start","data":{"app_id":"ee6b59f6-892d-4594-b813-2ae0119ad257","task_id":"943463e4f1cb47d7b4c2d73e3c1518a1","properties":{"services":[],"resources":{"memory":256,"disk":1024,"fds":16384},"environment":["http_proxy=http://georgew...@137.172.74.222:58080","https_proxy=http://georgew:Xsw2...@137.172.74.222:58080","no_proxy=localhost,mycloud1.com,137.172.74.*"],"meta":{"console":false}},"download_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748","upload_uri":"http://upload:c1ou...@10.0.0.74:9022/staging/droplets/ee6b59f6-892d-4594-b813-2ae0119ad257/upload","buildpack_cache_download_uri":null,"buildpack_cache_upload_uri":"http://upload:c1oudc0w@10.0.0.74:9022/staging/buildpack_cache/ee6b59f6-892d-4594-b813-2ae0119ad257/upload","start_message":{"droplet":"ee6b59f6-892d-4594-b813-2ae0119ad257","name":"calendar","uris":["calendar.mycloud.com"],"prod":false,"sha1":null,"executableFile":"deprecated","executableUri":null,"version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","services":[],"limits":{"mem":256,"disk":1024,"fds":16384},"cc_partition":"default","env":["http_proxy=http://georgew...@137.172.74.222:58080","https_proxy=http://georgew:Xsw2...@137.172.74.222:58080","no_proxy=localhost,mycloud1.com,137.172.74.*"],"console":false,"debug":null,"start_command":null,"health_check_timeout":null,"vcap_application":{"limits":{"mem":256,"disk":1024,"fds":16384},"application_version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","application_name":"calendar","application_uris":["calendar.mycloud.com"],"version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","name":"calendar","space_name":"aPaaS","space_id":"973ceec8-c6ee-4a7b-8379-7917406a8c16","uris":["calendar.mycloud.com"],"users":null},"index":0},"admin_buildpacks":[{"key":"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748"},{"key":"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748"},{"key":"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748"},{"key":"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748"}]}},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/nats.rb","lineno":146,"method":"handle_incoming_message"}
{"timestamp":1411703148.9870725,"message":"staging.handle.start","log_level":"debug","source":"Staging","data":{"app_guid":"ee6b59f6-892d-4594-b813-2ae0119ad257","request":"#<StagingMessage:0x00000002fd62e8 @message={\"app_id\"=>\"ee6b59f6-892d-4594-b813-2ae0119ad257\", \"task_id\"=>\"943463e4f1cb47d7b4c2d73e3c1518a1\", \"properties\"=>{\"services\"=>[], \"resources\"=>{\"memory\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"environment\"=>[\"http_proxy=http://georgewXsw2cde3@137.172.74.222:58080\", \"https_proxy=http://georgew:Xsw2...@137.172.74.222:58080\", \"no_proxy=localhost,mycloud1.com,137.172.74.*\"], \"meta\"=>{\"console\"=>false}}, \"download_uri\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748\", \"upload_uri\"=>\"http://upload:c1ou...@10.0.0.74:9022/staging/droplets/ee6b59f6-892d-4594-b813-2ae0119ad257/upload\", \"buildpack_cache_download_uri\"=>nil, \"buildpack_cache_upload_uri\"=>\"http://upload:c1oudc0w@10.0.0.74:9022/staging/buildpack_cache/ee6b59f6-892d-4594-b813-2ae0119ad257/upload\", \"start_message\"=>{\"droplet\"=>\"ee6b59f6-892d-4594-b813-2ae0119ad257\", \"name\"=>\"calendar\", \"uris\"=>[\"calendar.mycloud.com\"], \"prod\"=>false, \"sha1\"=>nil, \"executableFile\"=>\"deprecated\", \"executableUri\"=>nil, \"version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"services\"=>[], \"limits\"=>{\"mem\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"cc_partition\"=>\"default\", \"env\"=>[\"http_proxy=http://georgew...@137.172.74.222:58080\", \"https_proxy=http://georgew:Xsw2...@137.172.74.222:58080\", \"no_proxy=localhost,mycloud1.com,137.172.74.*\"], \"console\"=>false, \"debug\"=>nil, \"start_command\"=>nil, \"health_check_timeout\"=>nil, \"vcap_application\"=>{\"limits\"=>{\"mem\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"application_version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"application_name\"=>\"calendar\", \"application_uris\"=>[\"calendar.mycloud.com\"], \"version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"name\"=>\"calendar\", \"space_name\"=>\"aPaaS\", \"space_id\"=>\"973ceec8-c6ee-4a7b-8379-7917406a8c16\", \"uris\"=>[\"calendar.mycloud.com\"], \"users\"=>nil}, \"index\"=>0}, \"admin_buildpacks\"=>[{\"key\"=>\"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748\"}, {\"key\"=>\"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748\"}, {\"key\"=>\"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748\"}, {\"key\"=>\"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748\"}]}>"},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/responders/staging.rb","lineno":43,"method":"handle"}

...

Ferran Rodenas

unread,

Sep 28, 2014, 11:19:58 PM9/28/14

to vcap...@cloudfoundry.org

Synchronize them using NTP and use UTC as the default timezone. At the OpenStack boxes you must enable manually NTP if it's not already. On BOSH, if you add the below lines when deploying microBOSH, then it will sync all VMS using the ntp servers provided. Bosh by default sets UTC in all vms.

apply_spec:

properties:

ntp:

- 0.north-america.pool.ntp.org

- 1.north-america.pool.ntp.org

- 2.north-america.pool.ntp.org

- 3.north-america.pool.ntp.org

How have you updated the fog gem? CC (or any other job) doesn't use the default system gemset, so you'll need to update it at the job packages directory (/var/vcap/packages/<package-name>/...). Remember that the temp url is generated by CC, not the DEA, the DEA will use the generated tmp url to download the app bits and generate a droplet (app bits + buildpack).

BTW, thanks for submitting the issue!

- Ferdy

--

You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/20ad7129-0614-4752-9f61-e25dbd63ce61%40cloudfoundry.org.

iamflying

unread,

Sep 29, 2014, 1:24:19 AM9/29/14

to vcap...@cloudfoundry.org

On CC VM, I added the port in the temp url according to the https://github.com/fog/fog/commit/a3da07b6a69ea69b66cf41fa3740396b8ced1b84. However, it seems not to take effective when I push the application. I found the modified code was not invoked even I put wrong code on purpose. Anything else to do to make sure the port added?

My fix code is below.

/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/openstack/requests/storage#

get_object_https_url.rb

          hmac = Fog::HMAC.new('sha1', @openstack_temp_url_key)
          sig = sig_to_hex(hmac.sign(string_to_sign))

          port = '8080'
          case schema
          when 'http' then port = ":#{@port}" unless @port == 80
          when 'https' then port = ":#{@port}" unless @port == 443
          else port = ":#{@port}"
          end
          port = ":#{@port}"

          "#{scheme}://#{@host}#{port}#{object_path_escaped}?temp_url_sig=#{sig}&temp_url_expires=#{expires}"

{"timestamp":1411703148.986129,"message":"nats.message.received","log_level":"debug","source":"Dea::Nats","data":{"subject":"staging.0-18f300ce5fc64365a62518ba0f97e484.start","data":{"app_id":"ee6b59f6-892d-4594-b813-2ae0119ad257","task_id":"943463e4f1cb47d7b4c2d73e3c1518a1","properties":{"services":[],"resources":{"memory":256,"disk":1024,"fds":16384},"environment":["http_proxy=http://georgewXsw2cde3@137.172.74.222:58080","https_proxy=http://georgew:Xsw2cde3@137.172.74.222:58080","no_proxy=localhost,mycloud1.com,137.172.74.*"],"meta":{"console":false}},"download_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748","upload_uri":"http://upload:c1ou...@10.0.0.74:9022/staging/droplets/ee6b59f6-892d-4594-b813-2ae0119ad257/upload","buildpack_cache_download_uri":null,"buildpack_cache_upload_uri":"http://upload:c1oudc0w@10.0.0.74:9022/staging/buildpack_cache/ee6b59f6-892d-4594-b813-2ae0119ad257/upload","start_message":{"droplet":"ee6b59f6-892d-4594-b813-2ae0119ad257","name":"calendar","uris":["calendar.mycloud.com"],"prod":false,"sha1":null,"executableFile":"deprecated","executableUri":null,"version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","services":[],"limits":{"mem":256,"disk":1024,"fds":16384},"cc_partition":"default","env":["http_proxy=http://georgewXsw2cd...@137.172.74.222:58080","https_proxy=http://georgew:Xsw2...@137.172.74.222:58080","no_proxy=localhost,mycloud1.com,137.172.74.*"],"console":false,"debug":null,"start_command":null,"health_check_timeout":null,"vcap_application":{"limits":{"mem":256,"disk":1024,"fds":16384},"application_version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","application_name":"calendar","application_uris":["calendar.mycloud.com"],"version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","name":"calendar","space_name":"aPaaS","space_id":"973ceec8-c6ee-4a7b-8379-7917406a8c16","uris":["calendar.mycloud.com"],"users":null},"index":0},"admin_buildpacks":[{"key":"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748"},{"key":"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748"},{"key":"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748"},{"key":"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748"}]}},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/nats.rb","lineno":146,"method":"handle_incoming_message"}

...

Ferran Rodenas

unread,

Sep 29, 2014, 1:33:02 AM9/29/14

to vcap...@cloudfoundry.org

Have you restart the cloud controller? "sudo monit restart cloud_controller_ng"

- Ferdy

--

You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/1ac09de5-9532-44ee-a89d-c964870284da%40cloudfoundry.org.

iamflying

unread,

Sep 29, 2014, 3:09:42 AM9/29/14

to vcap...@cloudfoundry.org

Ferdy,

Thanks for your great help. It works after I restarted the cloud controller with your command. The sample nodejs application is running in my local cloud foundry.

{"timestamp":1411703148.986129,"message":"nats.message.received","log_level":"debug","source":"Dea::Nats","data":{"subject":"staging.0-18f300ce5fc64365a62518ba0f97e484.start","data":{"app_id":"ee6b59f6-892d-4594-b813-2ae0119ad257","task_id":"943463e4f1cb47d7b4c2d73e3c1518a1","properties":{"services":[],"resources":{"memory":256,"disk":1024,"fds":16384},"environment":["http_proxy=http://georgewXsw2cde3@137.172.74.222:58080","https_proxy=http://georgew:Xsw2cde3@137.172.74.222:58080","no_proxy=localhost,mycloud1.com,137.172.74.*"],"meta":{"console":false}},"download_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748","upload_uri":"http://upload:c1ou...@10.0.0.74:9022/staging/droplets/ee6b59f6-892d-4594-b813-2ae0119ad257/upload","buildpack_cache_download_uri":null,"buildpack_cache_upload_uri":"http://upload:c1oudc0w@10.0.0.74:9022/staging/buildpack_cache/ee6b59f6-892d-4594-b813-2ae0119ad257/upload","start_message":{"droplet":"ee6b59f6-892d-4594-b813-2ae0119ad257","name":"calendar","uris":["calendar.mycloud.com"],"prod":false,"sha1":null,"executableFile":"deprecated","executableUri":null,"version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","services":[],"limits":{"mem":256,"disk":1024,"fds":16384},"cc_partition":"default","env":["http_proxy=http://georgewXsw2cde3@137.172.74.222:58080","https_proxy=http://georgew:Xsw2cde3@137.172.74.222:58080","no_proxy=localhost,mycloud1.com,137.172.74.*"],"console":false,"debug":null,"start_command":null,"health_check_timeout":null,"vcap_application":{"limits":{"mem":256,"disk":1024,"fds":16384},"application_version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","application_name":"calendar","application_uris":["calendar.mycloud.com"],"version":"d9a443dd-af45-4421-a2c1-7d60f51ec6f5","name":"calendar","space_name":"aPaaS","space_id":"973ceec8-c6ee-4a7b-8379-7917406a8c16","uris":["calendar.mycloud.com"],"users":null},"index":0},"admin_buildpacks":[{"key":"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748"},{"key":"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748"},{"key":"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748"},{"key":"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6","url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748"}]}},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/nats.rb","lineno":146,"method":"handle_incoming_message"}

{"timestamp":1411703148.9870725,"message":"staging.handle.start","log_level":"debug","source":"Staging","data":{"app_guid":"ee6b59f6-892d-4594-b813-2ae0119ad257","request":"#<StagingMessage:0x00000002fd62e8 @message={\"app_id\"=>\"ee6b59f6-892d-4594-b813-2ae0119ad257\", \"task_id\"=>\"943463e4f1cb47d7b4c2d73e3c1518a1\", \"properties\"=>{\"services\"=>[], \"resources\"=>{\"memory\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"environment\"=>[\"http_proxy=http://georgewXsw2cde3@137.172.74.222:58080\", \"https_proxy=http://georgew:Xsw2...@137.172.74.222:58080\", \"no_proxy=localhost,mycloud1.com,137.172.74.*\"], \"meta\"=>{\"console\"=>false}}, \"download_uri\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748\", \"upload_uri\"=>\"http://upload:c1ou...@10.0.0.74:9022/staging/droplets/ee6b59f6-892d-4594-b813-2ae0119ad257/upload\", \"buildpack_cache_download_uri\"=>nil, \"buildpack_cache_upload_uri\"=>\"http://upload:c1oudc0w@10.0.0.74:9022/staging/buildpack_cache/ee6b59f6-892d-4594-b813-2ae0119ad257/upload\", \"start_message\"=>{\"droplet\"=>\"ee6b59f6-892d-4594-b813-2ae0119ad257\", \"name\"=>\"calendar\", \"uris\"=>[\"calendar.mycloud.com\"], \"prod\"=>false, \"sha1\"=>nil, \"executableFile\"=>\"deprecated\", \"executableUri\"=>nil, \"version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"services\"=>[], \"limits\"=>{\"mem\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"cc_partition\"=>\"default\", \"env\"=>[\"http_proxy=http://georgew...@137.172.74.222:58080\", \"https_proxy=http://georgew:Xsw2...@137.172.74.222:58080\", \"no_proxy=localhost,mycloud1.com,137.172.74.*\"], \"console\"=>false, \"debug\"=>nil, \"start_command\"=>nil, \"health_check_timeout\"=>nil, \"vcap_application\"=>{\"limits\"=>{\"mem\"=>256, \"disk\"=>1024, \"fds\"=>16384}, \"application_version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"application_name\"=>\"calendar\", \"application_uris\"=>[\"calendar.mycloud.com\"], \"version\"=>\"d9a443dd-af45-4421-a2c1-7d60f51ec6f5\", \"name\"=>\"calendar\", \"space_name\"=>\"aPaaS\", \"space_id\"=>\"973ceec8-c6ee-4a7b-8379-7917406a8c16\", \"uris\"=>[\"calendar.mycloud.com\"], \"users\"=>nil}, \"index\"=>0}, \"admin_buildpacks\"=>[{\"key\"=>\"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748\"}, {\"key\"=>\"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748\"}, {\"key\"=>\"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748\"}, {\"key\"=>\"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6\", \"url\"=>\"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748\"}]}>"},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/responders/staging.rb","lineno":43,"method":"handle"}
{"timestamp":1411703148.9888422,"message":"admin-buildpacks.download","log_level":"debug","source":"AdminBuildpackDownloader","data":{"buildpacks":[{"url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748","key":"e397e50d-a271-498d-aed6-cd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0"},{"url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748","key":"a2cb8756-6343-4370-826b-1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b"},{"url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748","key":"c20dc546-36a9-44ef-960f-c6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350"},{"url":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748","key":"365630d3-1576-4d10-a77a-a6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6"}]},"thread_id":9672840,"fiber_id":24603760,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/staging/admin_buildpack_downloader.rb","lineno":15,"method":"download"}
{"timestamp":1411703148.9920592,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"AdminBuildpackDownloader","data":{"droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/e3/97/e397e50d%2Da271%2D498d%2Daed6%2Dcd7a4ba2f57c_0eac241dcc6b8def5066447ae62411bfa70572e0?temp_url_sig=cc00fcccb01927be6139c5a2d12f8dc41eaeffcf&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411703148.992385,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"AdminBuildpackDownloader","data":{"droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/c2/0d/c20dc546%2D36a9%2D44ef%2D960f%2Dc6f0133bb462_6a77ede2993aeb18d6e8c17c18ec4aa2080b1350?temp_url_sig=f4f63289952f6a4996aaebc6e6ea56b14832ac98&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411703148.9926739,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"AdminBuildpackDownloader","data":{"droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/36/56/365630d3%2D1576%2D4d10%2Da77a%2Da6fc68fc5315_5ce1f73eeb2359111ee57566f760b650215054e6?temp_url_sig=25315fdd459dfdfe4daa413d84d3b6be2813c352&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411703148.992916,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"AdminBuildpackDownloader","data":{"droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-buildpacks/a2/cb/a2cb8756%2D6343%2D4370%2D826b%2D1e12149048ed_e648ebc4eea0e9e78ac0fcd56e382c1a7637625b?temp_url_sig=ae4b1ac29238c57e5c7961ff557774641d212616&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}
{"timestamp":1411703149.3930206,"message":"staging.app-download.starting","log_level":"info","source":"Staging","data":{"app_guid":"ee6b59f6-892d-4594-b813-2ae0119ad257","task_id":"943463e4f1cb47d7b4c2d73e3c1518a1","uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748"},"thread_id":9672840,"fiber_id":14408360,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb","lineno":289,"method":"block in promise_app_download"}
{"timestamp":1411703149.3942873,"message":"Error downloading: HTTP status: 404","log_level":"warn","source":"Staging","data":{"app_guid":"ee6b59f6-892d-4594-b813-2ae0119ad257","task_id":"943463e4f1cb47d7b4c2d73e3c1518a1","droplet_uri":"http://137.172.74.130/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-packages/ee/6b/ee6b59f6%2D892d%2D4594%2Db813%2D2ae0119ad257?temp_url_sig=faea722589770cadcbd657cdd66d5534540f77c4&temp_url_expires=1411706748","droplet_http_status":404},"thread_id":9672840,"fiber_id":23473200,"process_id":1609,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":71,"method":"block in download!"}

///////////////////// /var/log/message
[root@ror-openstack ~(keystone_demo)]# tail -f /var/log/messages | grep "AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7"
Sep 26 13:47:54 localhost proxy-server: STDOUT: PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources (txn: tx86af1739fe854c70a93a7-005424e1ea)
Sep 26 13:47:54 localhost proxy-server: STDOUT: RAW_PATH_INFO : /v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources (txn: tx86af1739fe854c70a93a7-005424e1ea)
Sep 26 13:47:54 localhost account-server: 137.172.74.130 - - [26/Sep/2014:03:47:54 +0000] "HEAD /swiftloopback/230167/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7" 204 - "tx86af1739fe854c70a93a7-005424e1ea" "HEAD http://137.172.74.130:8080/v1/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7" "proxy-server 12118" 0.0009 ""

Sep 26 13:47:54 localhost @container-server: 137.172.74.130 - - [26/Sep/2014:03:47:54 +0000] "GET /swiftloopback/7225/AUTH_cfe9f5b6e3bf4bacaec2bfad67d21ba7/cloudfoundry-resources" 404 - "tx86af1739fe854c70a93a7-00542</u
...

Reply all

Reply to author

Forward

0 new messages