Login & Api Jobs error-out with non CA Certified SSL certs

161 views
Skip to first unread message

equa....@gmail.com

unread,
Oct 20, 2014, 12:49:00 PM10/20/14
to vcap...@cloudfoundry.org

Issue Description for Login Job:

  1. Deployed CF on AWS using BOSH AWS Bootstrap.
  2. Used a cert that was not signed by a third party at the load balancer. (Internal poc\test deployment)
  3. Used uaac to create users.
  4. Pointed cf cli to api target using --skip-ssl-validation. This worked.
  5. Used cf login with --skip-ssl-validation.
  6. Got prompted to enter One time code (OTC). Directed to obtain the code fromhttps://login.xx.xxxx.com/passcode
  7. Go to https://login.xx.xxxx.com/passcode. Prompted to login. Put in credentials - FAILED with error code 500 and not message.
  8. Retried login from CLI again using cf auth - FAILED with error code 500 and not message.

Steps taken to debug

Checked up logs of all Login Jobs. Found the SSL Handshake Error :unable to find valid certification path to requested target.

Temporary Work Around

  1. Downloaded the cert from https://uaa.**.******.com.
  2. SCPed it to all the Login Jobs instances using bosh scp.
  3. SSHed into the Login Jobs instances and added the cert as trusted to the java keystore using the java keytool.
  4. Restarted all login jobs using bosh restart.

May be there is a better solution. Please let me know if so.

cf push failure

cf push failing with error. See error snapshots below:

cf push <app> -f cf-manifest.yml -p target/<app>.war
Using manifest file cf-manifest.yml

Updating app <app> in org <org> / space dev as <user>...
OK

Uploading <app>...
FAILED
Error uploading application.
Server error, status code: 500, error code: 10001, message: An unknown error occurred.

I also have taken care of skip this SSL cert verification during deployment using the following in the cf deployment manifest:

skip_cert_verify: true



After this I disabled HTTPS completely. That is by setting the no_ssl to true in the deployment manifest. After this login went through fine without issues but got stuck with cf push:

Snapshot with CF_TRACE=true:


$ CF_TRACE=true  cf push fb-user -f cf-manifest.yml -p target/fb-user.war  -b https://github.com/cloudfoundry/java-buildpack.git

VERSION:
6.1.0-homebrew

Using manifest file cf-manifest.yml


REQUEST: [2014-10-20T12:02:11-04:00]
POST /oauth/token HTTP/1.1
Host: login.cf.<DOMAIN>.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/x-www-form-urlencoded
User-Agent: go-cli 6.1.0-homebrew / darwin

grant_type=refresh_token&refresh_token=<TOKEN>

RESPONSE: [2014-10-20T12:02:11-04:00]
HTTP/1.1 200 OK
Content-Length: 2467
Cache-Control: no-cache
Cache-Control: no-store
Cache-Control: no-store
Connection: keep-alive
Content-Type: application/json;charset=UTF-8
Date: Mon, 20 Oct 2014 16:02:11 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Pragma: no-cache
Server: Apache-Coyote/1.1
X-Cf-Requestid: e2eb2267-f740-47ba-723f-e84138bba474
X-Cf-Requestid: e4e74a95-56c3-40f4-6992-58aa4041f29e
X-Frame-Options: DENY

{"access_token":"[PRIVATE DATA HIDDEN]","token_type":"bearer","refresh_token":"[PRIVATE DATA HIDDEN]","expires_in":599,"scope":"scim.userids scim.read cloud_controller.admin password.write scim.write openid cloud_controller.write cloud_controller.read","jti":"453a5e85-6c67-4862-82e7-2cc9bf34b493"}

REQUEST: [2014-10-20T12:02:12-04:00]
GET /v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/apps?q=name%3Afb-user&inline-relations-depth=1 HTTP/1.1
Host: api.cf.<DOMAIN>.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.1.0-homebrew / darwin



RESPONSE: [2014-10-20T12:02:12-04:00]
HTTP/1.1 200 OK
Content-Length: 4275
Connection: keep-alive
Content-Type: application/json;charset=utf-8
Date: Mon, 20 Oct 2014 16:02:12 GMT
Server: nginx
X-Cf-Requestid: 78b975cc-b600-48c0-59d7-395f3e8c2529
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 532a1dd5-e29f-4f83-5db5-240d2ae48cf2::a9976ca6-ed33-4b3f-b454-b7805923e6c5

{
  "total_results": 1,
  "total_pages": 1,
  "prev_url": null,
  "next_url": null,
  "resources": [
    {
      "metadata": {
        "guid": "a532840c-c524-4bf7-8f29-58cc89b5bf3a",
        "url": "/v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a",
        "created_at": "2014-10-18T21:46:11+00:00",
        "updated_at": "2014-10-20T16:01:47+00:00"
      },
      "entity": {
        "name": "fb-user",
        "production": false,
        "space_guid": "635b4998-59ae-4982-b51e-e844335fb447",
        "stack_guid": "bb41d432-2c27-4fcb-a9c6-7fb2eafaa46e",
        "buildpack": "https://github.com/cloudfoundry/java-buildpack.git",
        "detected_buildpack": null,
        "environment_json": {

        },
        "memory": 512,
        "instances": 2,
        "disk_quota": 1024,
        "state": "STOPPED",
        "version": "c5e7a5ef-710e-4cd8-abc4-52b0b238e5fd",
        "command": null,
        "console": false,
        "debug": null,
        "staging_task_id": null,
        "package_state": "PENDING",
        "health_check_timeout": null,
        "staging_failed_reason": null,
        "docker_image": null,
        "package_updated_at": null,
        "detected_start_command": "",
        "space_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447",
        "space": {
          "metadata": {
            "guid": "635b4998-59ae-4982-b51e-e844335fb447",
            "url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447",
            "created_at": "2014-10-18T21:40:09+00:00",
            "updated_at": null
          },
          "entity": {
            "name": "dev",
            "organization_guid": "ca62f4a4-30bf-4b8b-8111-4d596337dcb6",
            "space_quota_definition_guid": null,
            "organization_url": "/v2/organizations/ca62f4a4-30bf-4b8b-8111-4d596337dcb6",
            "developers_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/developers",
            "managers_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/managers",
            "auditors_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/auditors",
            "apps_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/apps",
            "routes_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/routes",
            "domains_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/domains",
            "service_instances_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/service_instances",
            "app_events_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/app_events",
            "events_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/events",
            "security_groups_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/security_groups"
          }
        },
        "stack_url": "/v2/stacks/bb41d432-2c27-4fcb-a9c6-7fb2eafaa46e",
        "stack": {
          "metadata": {
            "guid": "bb41d432-2c27-4fcb-a9c6-7fb2eafaa46e",
            "url": "/v2/stacks/bb41d432-2c27-4fcb-a9c6-7fb2eafaa46e",
            "created_at": "2014-10-18T05:35:32+00:00",
            "updated_at": null
          },
          "entity": {
            "name": "lucid64",
            "description": "Ubuntu 10.04 on x86-64"
          }
        },
        "events_url": "/v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a/events",
        "service_bindings_url": "/v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a/service_bindings",
        "service_bindings": [

        ],
        "routes_url": "/v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a/routes",
        "routes": [
          {
            "metadata": {
              "guid": "32ed4006-dbc8-4aad-af5a-4825d2595798",
              "url": "/v2/routes/32ed4006-dbc8-4aad-af5a-4825d2595798",
              "created_at": "2014-10-18T21:46:12+00:00",
              "updated_at": null
            },
            "entity": {
              "host": "fbuser",
              "domain_guid": "b20ae471-c9f5-49e1-ab88-df5832e9d621",
              "space_guid": "635b4998-59ae-4982-b51e-e844335fb447",
              "domain_url": "/v2/domains/b20ae471-c9f5-49e1-ab88-df5832e9d621",
              "space_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447",
              "apps_url": "/v2/routes/32ed4006-dbc8-4aad-af5a-4825d2595798/apps"
            }
          }
        ]
      }
    }
  ]
}
Updating app fb-user in org social / space dev as <USER>...

REQUEST: [2014-10-20T12:02:12-04:00]
PUT /v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a?async=true&inline-relations-depth=1 HTTP/1.1
Host: api.cf.<DOMAIN>.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.1.0-homebrew / darwin

{"name":"fb-user","instances":2,"memory":512,"buildpack":"https://github.com/cloudfoundry/java-buildpack.git","environment_json":{}}

RESPONSE: [2014-10-20T12:02:12-04:00]
HTTP/1.1 201 Created
Content-Length: 3792
Connection: keep-alive
Content-Type: application/json;charset=utf-8
Date: Mon, 20 Oct 2014 16:02:12 GMT
Server: nginx
X-Cf-Requestid: 55030daa-0e2c-4200-4230-54223f44fb8e
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 5c94fa87-dbc0-417c-5523-e5ff3ea641af::8b002c12-b4e3-42c6-80c1-9d29d842b8be

{
  "metadata": {
    "guid": "a532840c-c524-4bf7-8f29-58cc89b5bf3a",
    "url": "/v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a",
    "created_at": "2014-10-18T21:46:11+00:00",
    "updated_at": "2014-10-20T16:02:12+00:00"
  },
  "entity": {
    "name": "fb-user",
    "production": false,
    "space_guid": "635b4998-59ae-4982-b51e-e844335fb447",
    "stack_guid": "bb41d432-2c27-4fcb-a9c6-7fb2eafaa46e",
    "buildpack": "https://github.com/cloudfoundry/java-buildpack.git",
    "detected_buildpack": null,
    "environment_json": {

    },
    "memory": 512,
    "instances": 2,
    "disk_quota": 1024,
    "state": "STOPPED",
    "version": "c5e7a5ef-710e-4cd8-abc4-52b0b238e5fd",
    "command": null,
    "console": false,
    "debug": null,
    "staging_task_id": null,
    "package_state": "PENDING",
    "health_check_timeout": null,
    "staging_failed_reason": null,
    "docker_image": null,
    "package_updated_at": null,
    "detected_start_command": "",
    "space_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447",
    "space": {
      "metadata": {
        "guid": "635b4998-59ae-4982-b51e-e844335fb447",
        "url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447",
        "created_at": "2014-10-18T21:40:09+00:00",
        "updated_at": null
      },
      "entity": {
        "name": "dev",
        "organization_guid": "ca62f4a4-30bf-4b8b-8111-4d596337dcb6",
        "space_quota_definition_guid": null,
        "organization_url": "/v2/organizations/ca62f4a4-30bf-4b8b-8111-4d596337dcb6",
        "developers_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/developers",
        "managers_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/managers",
        "auditors_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/auditors",
        "apps_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/apps",
        "routes_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/routes",
        "domains_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/domains",
        "service_instances_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/service_instances",
        "app_events_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/app_events",
        "events_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/events",
        "security_groups_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447/security_groups"
      }
    },
    "stack_url": "/v2/stacks/bb41d432-2c27-4fcb-a9c6-7fb2eafaa46e",
    "stack": {
      "metadata": {
        "guid": "bb41d432-2c27-4fcb-a9c6-7fb2eafaa46e",
        "url": "/v2/stacks/bb41d432-2c27-4fcb-a9c6-7fb2eafaa46e",
        "created_at": "2014-10-18T05:35:32+00:00",
        "updated_at": null
      },
      "entity": {
        "name": "lucid64",
        "description": "Ubuntu 10.04 on x86-64"
      }
    },
    "events_url": "/v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a/events",
    "service_bindings_url": "/v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a/service_bindings",
    "service_bindings": [

    ],
    "routes_url": "/v2/apps/a532840c-c524-4bf7-8f29-58cc89b5bf3a/routes",
    "routes": [
      {
        "metadata": {
          "guid": "32ed4006-dbc8-4aad-af5a-4825d2595798",
          "url": "/v2/routes/32ed4006-dbc8-4aad-af5a-4825d2595798",
          "created_at": "2014-10-18T21:46:12+00:00",
          "updated_at": null
        },
        "entity": {
          "host": "fbuser",
          "domain_guid": "b20ae471-c9f5-49e1-ab88-df5832e9d621",
          "space_guid": "635b4998-59ae-4982-b51e-e844335fb447",
          "domain_url": "/v2/domains/b20ae471-c9f5-49e1-ab88-df5832e9d621",
          "space_url": "/v2/spaces/635b4998-59ae-4982-b51e-e844335fb447",
          "apps_url": "/v2/routes/32ed4006-dbc8-4aad-af5a-4825d2595798/apps"
        }
      }
    ]
  }
}
OK

Uploading fb-user...

REQUEST: [2014-10-20T12:02:13-04:00]
PUT /v2/resource_match HTTP/1.1
Host: api.cf.<DOMAIN>.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.1.0-homebrew / darwin

[{"fn":"META-INF","sha1":"0","size":136},{"fn":"META-INF/MANIFEST.MF","sha1":"c23824f6fc50dc8d8bd4ce49d8c5f6edab520bcd","size":132},{"fn":"META-INF/maven","sha1":"0","size":102},{"fn":"META-INF/maven/fb-user","sha1":"0","size":102},{"fn":"META-INF/maven/fb-user/fb-user","sha1":"0","size":136},{"fn":"META-INF/maven/fb-user/fb-user/pom.properties","sha1":"e666767cd5161019f138672763d679fdcc93fc57","size":109},{"fn":"META-INF/maven/fb-user/fb-user/pom.xml","sha1":"94698e7ba2c038e83dada815605ef7c69299ad7e","size":6591},{"fn":"WEB-INF","sha1":"0","size":238},{"fn":"WEB-INF/classes","sha1":"0","size":102},{"fn":"WEB-INF/classes/info","sha1":"0","size":102},{"fn":"WEB-INF/classes/info/owaism","sha1":"0","size":102},{"fn":"WEB-INF/classes/info/owaism/social","sha1":"0","size":102},{"fn":"WEB-INF/classes/info/owaism/social/fb","sha1":"0","size":102},{"fn":"WEB-INF/classes/info/owaism/social/fb/user","sha1":"0","size":204},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/config","sha1":"0","size":136},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/config/FBUserApplicationInitializer.class","sha1":"c878ff749ab1820a4bbb8ab5493aab80b67386ae","size":2204},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/config/WebConfig.class","sha1":"55bfe33275b418419488109513de0e55f9313827","size":3726},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/constant","sha1":"0","size":102},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/constant/EnvironmentConstants.class","sha1":"b15451e6f46a2d5b0df2380b8f5bb8c21c3191a7","size":953},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/controller","sha1":"0","size":136},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/controller/HealthCheckResource.class","sha1":"891885de27574e7b866cb709e7fa163204c15eea","size":1750},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/controller/LoginController.class","sha1":"f2d9a653516f073d7cef34a91ce45787df06e46c","size":3615},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/util","sha1":"0","size":136},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/util/RandomStringGenerator.class","sha1":"1273f4604648575efbb16d2ad930f1c810e5ba86","size":1195},{"fn":"WEB-INF/classes/info/owaism/social/fb/user/util/ViewableUtil.class","sha1":"aac2c6f7ad96bb2145c694e083811c9a3081a9da","size":895},{"fn":"WEB-INF/css","sha1":"0","size":136},{"fn":"WEB-INF/css/normalize.mustache","sha1":"012a14eaa9619413d05153ec2a68f3f79af30846","size":9515},{"fn":"WEB-INF/css/style.mustache","sha1":"affb8432f60a63bc3d258619b23b8e624ecc5db6","size":1887},{"fn":"WEB-INF/lib","sha1":"0","size":1530},{"fn":"WEB-INF/lib/aopalliance-1.0.jar","sha1":"0235ba8b489512805ac13a8f9ea77a1ca5ebe3e8","size":4467},{"fn":"WEB-INF/lib/aopalliance-repackaged-2.3.0-b10.jar","sha1":"8970184d22d7ec9f9bd2fbddf861e342f8019a84","size":14766},{"fn":"WEB-INF/lib/classmate-1.0.0.jar","sha1":"434efef28c81162b17c540e634cffa3bd9b09b4c","size":60282},{"fn":"WEB-INF/lib/commons-logging-1.1.3.jar","sha1":"f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f","size":62050},{"fn":"WEB-INF/lib/fb.sdk-MASTER-SNAPSHOT.jar","sha1":"9304e682773ba0ab10ba6c8cd9cc6b59b1526d09","size":29773},{"fn":"WEB-INF/lib/guava-18.0.jar","sha1":"cce0823396aa693798f8882e64213b1772032b09","size":2256213},{"fn":"WEB-INF/lib/hibernate-validator-5.1.2.Final.jar","sha1":"8aecb33d3587f32c877827c2bdddae0d227a40ed","size":633104},{"fn":"WEB-INF/lib/hk2-api-2.3.0-b10.jar","sha1":"4971a6e2e87c4bee3c842d099977969a42a0ad64","size":167109},{"fn":"WEB-INF/lib/hk2-locator-2.3.0-b10.jar","sha1":"40977d562f667b553bed14cae64ad596ecf8d54a","size":167389},{"fn":"WEB-INF/lib/hk2-utils-2.3.0-b10.jar","sha1":"66f81d01d2f732a8c3204d8120488507a2834bca","size":88159},{"fn":"WEB-INF/lib/jackson-annotations-2.4.2.jar","sha1":"6bb52af09372d5064206d47d7887d41671f00f7d","size":38604},{"fn":"WEB-INF/lib/jackson-core-2.4.2.jar","sha1":"ceb72830d95c512b4b300a38f29febc85bdf6e4b","size":225316},{"fn":"WEB-INF/lib/jackson-databind-2.4.2.jar","sha1":"8e31266a272ad25ac4c089734d93e8d811652c1f","size":1075759},{"fn":"WEB-INF/lib/jackson-jaxrs-base-2.3.2.jar","sha1":"14adfd8e7d9caacd97e1c0627486dd1cc70ff211","size":27584},{"fn":"WEB-INF/lib/jackson-jaxrs-json-provider-2.3.2.jar","sha1":"8e635e82db2437804e19e746f460392d791856cf","size":16511},{"fn":"WEB-INF/lib/jackson-module-jaxb-annotations-2.3.2.jar","sha1":"0228f4ed947cf3403c5f4315143b6c372c28e90f","size":31636},{"fn":"WEB-INF/lib/javassist-3.18.1-GA.jar","sha1":"d9a09f7732226af26bf99f19e2cffe0ae219db5b","size":714194},{"fn":"WEB-INF/lib/javax.annotation-api-1.2.jar","sha1":"479c1e06db31c432330183f5cae684163f186146","size":26366},{"fn":"WEB-INF/lib/javax.el-2.2.4.jar","sha1":"a50914ff519682e185bca4385b4313b8c8a81775","size":117583},{"fn":"WEB-INF/lib/javax.el-api-2.2.4.jar","sha1":"1287562cc3f0ff5439ded6f2949e73ce1c0edaab","size":38863},{"fn":"WEB-INF/lib/javax.inject-2.3.0-b10.jar","sha1":"48656679e624eb41140abcd7b52b34808cf1852d","size":5950},{"fn":"WEB-INF/lib/javax.ws.rs-api-2.0.1.jar","sha1":"104e9c2b5583cfcfeac0402316221648d6d8ea6b","size":115534},{"fn":"WEB-INF/lib/jboss-logging-3.1.3.GA.jar","sha1":"64499e907f19e5e1b3fdc02f81440c1832fe3545","size":57183},{"fn":"WEB-INF/lib/jersey-client-2.13.jar","sha1":"c3aa96e822e440176540c9fc78961095082a8f4d","size":155089},{"fn":"WEB-INF/lib/jersey-common-2.13.jar","sha1":"2142596b05bbf3c171173843b132eeea0a0f08d5","size":729165},{"fn":"WEB-INF/lib/jersey-guava-2.13.jar","sha1":"e4b560eba2e072038de80ea2868e891acddf753c","size":962867},{"fn":"WEB-INF/lib/jersey-media-json-jackson-2.13.jar","sha1":"45f1b4e598c8175b3cb2b6f02602f7712dcadf8e","size":6677},{"fn":"WEB-INF/lib/jmustache-1.2.jar","sha1":"2b771de4df29b7be133a2093d0ecd2a18f1fbfc8","size":30492},{"fn":"WEB-INF/lib/logback-classic-1.1.2.jar","sha1":"b316e9737eea25e9ddd6d88eaeee76878045c6b2","size":270750},{"fn":"WEB-INF/lib/logback-core-1.1.2.jar","sha1":"2d23694879c2c12f125dac5076bdfd5d771cc4cb","size":427729},{"fn":"WEB-INF/lib/mustache-spring-view-1.0.jar","sha1":"dc06f814b7b56abf065a719b368c7078637ac037","size":6538},{"fn":"WEB-INF/lib/osgi-resource-locator-1.0.1.jar","sha1":"4ed2b2d4738aed5786cfa64cba5a332779c4c708","size":20235},{"fn":"WEB-INF/lib/security-support-MASTER-SNAPSHOT.jar","sha1":"4a7b831a1a36b15f26c0d2d4b07599ec475a8543","size":5639},{"fn":"WEB-INF/lib/slf4j-api-1.7.7.jar","sha1":"2b8019b6249bb05d81d3a3094e468753e2b21311","size":29257},{"fn":"WEB-INF/lib/spring-aop-4.1.0.RELEASE.jar","sha1":"be535101d0cdd395484f13506bd8334042f35743","size":359025},{"fn":"WEB-INF/lib/spring-beans-4.1.0.RELEASE.jar","sha1":"3c7d7df651e172ec7ce99bd7e19c9c015ebb596a","size":701334},{"fn":"WEB-INF/lib/spring-context-4.1.0.RELEASE.jar","sha1":"0b09f59878c2757490cd6bd1365afe686230ad18","size":1023191},{"fn":"WEB-INF/lib/spring-core-4.1.0.RELEASE.jar","sha1":"cd3ba08b0fe1b2cb9dc50e9762c4ba4fae5d8401","size":1005039},{"fn":"WEB-INF/lib/spring-expression-4.1.0.RELEASE.jar","sha1":"39d5734c5ca1f0b5c3bd9decd7f9277059d1300c","size":249627},{"fn":"WEB-INF/lib/spring-oxm-4.1.0.RELEASE.jar","sha1":"9f2678e153641b05fe9358d278350d112c0b195f","size":82147},{"fn":"WEB-INF/lib/spring-web-4.1.0.RELEASE.jar","sha1":"5755c5ac57e38b46c3d9893fd540f4a5a99e22fb","size":697454},{"fn":"WEB-INF/lib/spring-webmvc-4.1.0.RELEASE.jar","sha1":"7c4f66e33b75a4a4a320b88633ff8acf640d44ed","size":773814},{"fn":"WEB-INF/lib/validation-api-1.1.0.Final.jar","sha1":"8613ae82954779d518631e05daa73a6a954817d5","size":63777},{"fn":"WEB-INF/views","sha1":"0","size":204},{"fn":"WEB-INF/views/footer.mustache","sha1":"06511637470dd52af341f2c2bec54fa8a624a6d7","size":110},{"fn":"WEB-INF/views/get.mustache","sha1":"4636b9ad67f3ed7152c35d39294a518caf8246ab","size":168},{"fn":"WEB-INF/views/home.mustache","sha1":"41f42b313490e04f2e1c8b49b7c0d02bb5ccee14","size":296},{"fn":"WEB-INF/views/profile.mustache","sha1":"db11484f5c11a81b1c3e60227f48d1dba58dfb20","size":540},{"fn":"WEB-INF/web.xml","sha1":"4a2eb166021a683da0cd1370ba9f8ba8e4e92742","size":249}]

RESPONSE: [2014-10-20T12:02:13-04:00]
HTTP/1.1 500 Internal Server Error
Content-Length: 86
Connection: keep-alive
Content-Type: application/json;charset=utf-8
Date: Mon, 20 Oct 2014 16:02:13 GMT
Server: nginx
X-Cf-Requestid: 4676a67b-1e02-410d-6d86-e0ea0b37bed9
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 173f0dd4-4a14-4e72-4b03-ec74e778f55b::588f85e3-da81-408f-ac1a-c13cd58e7cdb

{"error_code":"UnknownError","description":"An unknown error occurred.","code":10001}

FAILED
Error uploading application.
Server error, status code: 500, error code: 10001, message: An unknown error occurred.
FAILED
Error uploading application.
Server error, status code: 500, error code: 10001, message: An unknown error occurred.


On further analysis of the api_z1 logs found the following in the cloud controller logs:

{ "data" : {  },
  "fiber_id" : 31079300,
  "file" : "/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/app/jobs/exception_catching_job.rb",
  "lineno" : 31,
  "log_level" : "error",
  "message" : "Request failed: 500: {\"code\"=>10001, \"description\"=>\"hostname does not match the server certificate (OpenSSL::SSL::SSLError)\", \"error_code\"=>\"CF-SocketError\", \"backtrace\"=>[\"/var/vcap/packages/ruby/lib/ruby/1.9.1/openssl/ssl-internal.rb:129:in `post_connection_check'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/ssl_socket.rb:129:in `initialize'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:375:in `new'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:375:in `socket'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:105:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/mock.rb:47:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/instrumentor.rb:22:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:230:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:253:in `rescue in request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:201:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:253:in `rescue in request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:201:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:253:in `rescue in request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:201:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/xml/sax_parser_connection.rb:35:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/xml/connection.rb:17:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/aws/storage.rb:535:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/aws/requests/storage/get_bucket.rb:51:in `get_bucket'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/aws/models/storage/directories.rb:22:in `get'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/directory.rb:14:in `get'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/idempotent_directory.rb:9:in `get_or_create'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/client.rb:131:in `dir'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/client.rb:108:in `files'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/client.rb:60:in `block in cp_to_blobstore'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/client.rb:55:in `open'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/client.rb:55:in `cp_to_blobstore'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/upload_buildpack.rb:20:in `upload_buildpack'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/app/jobs/runtime/buildpack_installer.rb:19:in `perform'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/app/jobs/timeout_job.rb:6:in `block in perform'\", \"/var/vcap/packages/ruby/lib/ruby/1.9.1/timeout.rb:69:in `timeout'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/app/jobs/timeout_job.rb:5:in `perform'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/app/jobs/request_job.rb:9:in `perform'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/app/jobs/exception_catching_job.rb:7:in `perform'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/backend/base.rb:105:in `block in invoke_job'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/backend/base.rb:102:in `invoke_job'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:206:in `block (2 levels) in run'\", \"/var/vcap/packages/ruby/lib/ruby/1.9.1/timeout.rb:69:in `timeout'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:206:in `block in run'\", \"/var/vcap/packages/ruby/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:205:in `run'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:267:in `block in reserve_and_run_one_job'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:267:in `reserve_and_run_one_job'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:189:in `block in work_off'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:188:in `times'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:188:in `work_off'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:153:in `block (4 levels) in start'\", \"/var/vcap/packages/ruby/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:152:in `block (3 levels) in start'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:151:in `block (2 levels) in start'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:150:in `loop'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:150:in `block in start'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/plugins/clear_locks.rb:7:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `block (2 levels) in add'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `block in add'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/delayed_job-4.0.0/lib/delayed/worker.rb:149:in `start'\", \"lib/tasks/jobs.rake:39:in `start_working'\", \"lib/tasks/jobs.rake:10:in `block (2 levels) in <top (required)>'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:240:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:240:in `block in execute'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:235:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:235:in `execute'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'\", \"/var/vcap/packages/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:172:in `invoke_with_call_chain'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:165:in `invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:150:in `invoke_task'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:106:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:106:in `block in top_level'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:115:in `run_with_threads'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:100:in `top_level'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:78:in `block in run'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:176:in `standard_exception_handling'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:75:in `run'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/bin/rake:33:in `<top (required)>'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/bin/rake:23:in `load'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/bin/rake:23:in `<main>'\"]}",
  "method" : "log_error",
  "process_id" : 1782,
  "source" : "cc.background",
  "thread_id" : 6600700,
  "timestamp" : 1413815785.5855165
}

Just a note: This is with SSL turned off.


Any help on this issue will be greatly appreciated.






equa....@gmail.com

unread,
Oct 20, 2014, 12:50:09 PM10/20/14
to vcap...@cloudfoundry.org, equa....@gmail.com
I have also opened an issue @ Cloud Foundry Releases github: 

Filip Hanik

unread,
Oct 20, 2014, 12:51:33 PM10/20/14
to vcap...@cloudfoundry.org

Temporary Work Around

  1. Downloaded the cert from https://uaa.**.******.com.
  2. SCPed it to all the Login Jobs instances using bosh scp.
  3. SSHed into the Login Jobs instances and added the cert as trusted to the java keystore using the java keytool.
  4. Restarted all login jobs using bosh restart.

Yes, the communication between Login Server and UAA is dictated by the property

properties.login.protocol



 

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/fb3631e2-94b7-46cf-b8b0-53cdbdd80bad%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

equa....@gmail.com

unread,
Oct 20, 2014, 1:00:18 PM10/20/14
to vcap...@cloudfoundry.org, fha...@pivotal.io
Thanks.

I changed everything from https to http in the deployment manifest. I still get the last problem with cf push.
catching_job.rb",
  "lineno" : 31,
  "log_level" : "error",
  "message" : "Request failed: 500: {\"code\"=>10001, \"description\"=>\"hostname does not match the server certificate (OpenSSL::SSL::SSLError)\", \"error_code\"=>\"CF-SocketError\", \"backtrace\"=>[\"/var/vcap/packages/ruby/lib/ruby/1.9.1/openssl/ssl-internal.rb:129:in `post_connection_check'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/ssl_socket.rb:129:in `initialize'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:375:in `new'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:375:in `socket'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:105:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/mock.rb:47:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/instrumentor.rb:22:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:230:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:253:in `rescue in request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:201:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:253:in `rescue in request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:201:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \&
...

equa....@gmail.com

unread,
Oct 20, 2014, 1:03:39 PM10/20/14
to vcap...@cloudfoundry.org, fha...@pivotal.io, equa....@gmail.com
If you see the rest of my initial post. I have also set "no_ssl" to true in the deployment manifest.

All My cf and uaac targets are now pointing to http rather than https. But still when I upload a java webapp (war) I get the 


{"error_code":"UnknownError","description":"An unknown error occurred.","code":10001}

And in the logs of api_z1

{ "data" : {  },
  "fiber_id" : 31079300,
  "file" : "/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/app/jobs/exception_
catching_job.rb",
  "lineno" : 31,
  "log_level" : "error",
  "message" : "Request failed: 500: {\"code\"=>10001, \"description\"=>\"hostname does not match the server certificate (OpenSSL::SSL::SSLError)\", \"error_code\"=>\"CF-SocketError\", \"backtrace\"=>[\"/var/vcap/packages/ruby/lib/ruby/1.9.1/openssl/ssl-internal.rb:129:in `post_connection_check'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/ssl_socket.rb:129:in `initialize'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:375:in `new'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:375:in `socket'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:105:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/mock.rb:47:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/instrumentor.rb:22:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:15:in `request_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:230:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:253:in `rescue in request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:201:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:253:in `rescue in request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:201:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/idempotent.rb:22:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/middlewares/base.rb:10:in `error_call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:253:in `rescue in request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.39.6/lib/excon/connection.rb:201:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/xml/sax_parser_connection.rb:35:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/xml/connection.rb:17:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/aws/storage.rb:535:in `request'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/aws/requests/storage/get_bucket.rb:51:in `get_bucket'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.23.0/lib/fog/aws/models/storage/directories.rb:22:in `get'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/directory.rb:14:in `get'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/idempotent_directory.rb:9:in `get_or_create'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/client.rb:131:in `dir'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac5cee35abdef1.1-0f7d98fa86cd6175b035f5a35d6927a7fd11f104/cloud_controller_ng/lib/cloud_controller/blobstore/client.rb:108:in `files'\", \"/var/vcap/data/packages/cloud_controller_ng/23a297c0c272d1e5a1eaee9d98ac


Details provided in the original post.

Filip Hanik

unread,
Oct 20, 2014, 1:41:01 PM10/20/14
to equa....@gmail.com, vcap...@cloudfoundry.org

equa....@gmail.com

unread,
Oct 20, 2014, 2:29:17 PM10/20/14
to vcap...@cloudfoundry.org, equa....@gmail.com, fha...@pivotal.io
Looks very similar to it.

The solution in that thread was to set the fog connection with the provider as local rather than it look up s3. Something like below:

fog_connection:
        aws_access_key_id: XXXX
        aws_secret_access_key: XXXXX
        provider: local
        local_root: /var/vcap/store/cc_blobstore


Made changes to the Deployment manifest and deploying it right now.

In the mean time it would save me some time if it was possible for you to provide me an answer to this question also:
The above fog connection setting is for buildpacks. There are fog connection settings for droplets and packages. Would you suppose it to be the same for those also?

I will anyway update you on the progress. Thanks for the points Filip.
...

Owais Mohamed

unread,
Oct 20, 2014, 3:41:44 PM10/20/14
to vcap...@cloudfoundry.org, equa....@gmail.com, fha...@pivotal.io
Had deleted the deployment. So re-deployed CF...

I think the directory structure has changed in CF-releases since that thread. 

This was what the 'generate_deployment_manifest' produced for me:

buildpacks:
      buildpack_directory_key: cf.<domain>.com-cc-buildpacks
      cdn: null
      fog_connection:
        aws_access_key_id: xxxxxx
        aws_secret_access_key: xxxxxx
        provider: aws

I see that the build packs have not been uploaded to S3. So the buildpack_directory_key does not hold good. I know that the buildpacks where upload during the 'bosh upload release' phase. But not sure where? 

I checked up the director's file system and found that the drive mounted on '/var/vcap/store/' took up 7.2 gigs. It has a directory 'blobstore' within it. 

$ ls -lrt /var/vcap/store/blobstore/
total 8
drwxr-xr-x   3 vcap vcap 4096 2014-10-18 01:44 tmp
drwx------ 157 vcap vcap 4096 2014-10-20 18:27 store


$ ls -lrt /var/vcap/store/blobstore/store/
total 620
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 a4
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 1f
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 23
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 e3
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 27
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 f8
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 77
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 33
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 71
drwx------ 2 vcap vcap 4096 2014-10-18 03:48 96



Based on this I dont know what should be the 'buildpack_directory_key' and the 'local_root'.

Would be helpful if someone can point me documentation related to this. 

Regards,
Owais
...

Filip Hanik

unread,
Oct 20, 2014, 3:55:47 PM10/20/14
to Owais Mohamed, vcap...@cloudfoundry.org
this is out of my expertise area, I was able to help you with login-server and UAA cause I'm more familiar with those.
But an example from http://docs.cloudfoundry.org/deploying/ec2/deploy_aws_cf.html suggests that maybe you don't need to specify a directory?

Owais Mohamed

unread,
Oct 21, 2014, 2:00:27 PM10/21/14
to vcap...@cloudfoundry.org, equa....@gmail.com, fha...@pivotal.io
Thanks a lot Filip. Your pointers helped a lot.

Resolution to this problem - for future readers:

  1. If see a SSL handshake error while pushing an app it could be because of connectivity to amazon S3 as described in thread https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/4B88mmKF2s0/R00aaSul1OsJ
  2. For this you have two options either:
    1. You can opt not to use S3 buckets at all. For this you need to change your FOG connection to:
      fog_connection:
              provider: Local
              local_root: /var/vcap/data
      You need to change all fog connections for packages, buildpacks, droplets and cc-resources.
      The local_root above should point to a folder  on api_z1 job which has enough disk storage. For this you need to bosh ssh into api_z1 job and check out `df -h` to see which mounted drives have enough spaces. For me it was `/var/vcap/data`
    2. You can use S3 buckets but add `path_style: true` to your fog_connection every where. I have not tried this but the hint was provided by `da...@castlelaing.com`. In the other thread that I have pointed to.  What this would do is hit `https://s3.amazonaws.com/<BUCKET_NAME>/` instead of `https://<BUCKET_NAME>.s3.amazonaws.com/`.
Reply all
Reply to author
Forward
0 new messages