Certificate error on blobstore's connection on AWS

[tchass...@ippon.fr]

I installed the last CloudFoundry release + MySQL release on my AWS environnement following the instructions here : http://docs.cloudfoundry.com/docs/running/deploying-cf/ec2/

After a few mistakes coming from me I was able to connect and I wanted to deploy a simple application (Spring petclinic demo)

I'm able to configure the deployment of my application (nb of instance, memory, service binding) but I have a certificate error when I try to upload the war file with the following stacktrace :

Uploading ippon-petclinic.  >>>
REQUEST: PUT http://api.cloudfoundry.ippon.fr/v2/resource_match
REQUEST_HEADERS:
  Accept : application/json
  Authorization : [PRIVATE DATA HIDDEN]
  Content-Length : 25379
  Content-Type : application/json
REQUEST_BODY: [{"size":137,"sha1":"42d52903d2b144abead4fda4cdf14a83b81f175b","fn":"/tmp/.cf_a977f9c7-3bb2-4358-bb0b-73d42776618a_files/META-INF/maven/org.springframework.samples/spring-petclinic/pom.properties"},{"size":14452,"sha1":"1a54db9b8b310cbc4bd0a831b3dfc4bc61ef6a4e","fn":"/tmp/.cf_a977f9c7-3bb2-4358-bb0b-73d42776618a_files/META-INF/maven/org.springframework.samples/spring-petclinic/pom.xml"},{"size":130,"sha1":"6a4738852dd8efffdb390c018d9a6c53fefaf33b","fn":"/tmp/.cf_a977f9c7-3bb2-4358-bb0b-73d42776618a_files/META-INF/MANIFEST.MF"},{"size":2643,"sha1":"8faae7d439a3c24a11285373f487f2aacdeb4f7a","fn":"/tmp/.cf_a977f9c7-3bb2-4358-bb0b-73d42776618a_files/WEB-INF/classes/org/springframework/samples/petclinic/model/Vet.class""}]
   RESPONSE: [500]
RESPONSE_HEADERS:
  connection : keep-alive
  content-length : 14595
  content-type : application/json;charset=utf-8
  date : Mon, 20 Jan 2014 08:31:07 GMT
  server : nginx
  x-content-type-options : nosniff
  x-vcap-request-id : c3a46b9b-6e61-40f2-a256-c64f32faa567
RESPONSE_BODY:
{"code":10001,"description":"hostname does not match the server certificate (OpenSSL::SSL::SSLError)","error_code":"CF-SocketError","types":["SocketError","Error"],"backtrace":["/var/vcap/packages/ruby/lib/ruby/1.9.1/openssl/ssl-internal.rb:129:in `post_connection_check'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/ssl_socket.rb:85:in `initialize'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:373:in `new'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:373:in `socket'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:122:in `request_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/agent/instrumentation/excon/middleware.rb:28:in `request_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/mock.rb:42:in `request_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/instrumentor.rb:22:in `request_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:15:in `request_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:15:in `request_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:15:in `request_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:251:in `request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/idempotent.rb:12:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:10:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:10:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:268:in `rescue in request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:217:in `request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/idempotent.rb:12:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:10:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:10:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:268:in `rescue in request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:217:in `request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/idempotent.rb:12:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:10:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/middlewares/base.rb:10:in `error_call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:268:in `rescue in request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/excon-0.28.0/lib/excon/connection.rb:217:in `request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/core/connection.rb:57:in `request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/core/deprecated/connection.rb:20:in `request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/aws/storage.rb:526:in `request'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/aws/requests/storage/put_bucket.rb:36:in `put_bucket'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/aws/models/storage/directory.rb:104:in `save'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.18.0/lib/fog/core/collection.rb:52:in `create'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/blobstore/blobstore.rb:104:in `dir'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/blobstore/blobstore.rb:90:in `files'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/resource_pool.rb:103:in `resource_known?'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/resource_pool.rb:33:in `block in match_resources'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/resource_pool.rb:33:in `select'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/resource_pool.rb:33:in `match_resources'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/runtime/resource_matches_controller.rb:7:in `match'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/base.rb:95:in `dispatch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:16:in `block in define_route'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `block in compile!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `[]'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (3 levels) in route!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:966:in `route_eval'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/agent/instrumentation/sinatra.rb:133:in `route_eval_with_newrelic'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (2 levels) in route!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:987:in `block in process_route'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `catch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `process_route'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/agent/instrumentation/sinatra.rb:116:in `process_route_with_newrelic'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:948:in `block in route!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `each'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `route!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1059:in `block in dispatch!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1056:in `dispatch!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/agent/instrumentation/sinatra.rb:151:in `dispatch_and_notice_errors_with_newrelic'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/agent/instrumentation/sinatra.rb:146:in `block in dispatch_with_newrelic'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:339:in `perform_action_with_newrelic_trace'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/agent/instrumentation/sinatra.rb:145:in `dispatch_with_newrelic'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `block in call!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `call!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/rack/error_collector.rb:50:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/rack/browser_monitoring.rb:23:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.7.1.180/lib/new_relic/rack/agent_hooks.rb:28:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/xss_header.rb:18:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/path_traversal.rb:16:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/json_csrf.rb:18:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/frame_options.rb:31:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:175:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1949:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/commonlogger.rb:33:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:212:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:81:in `block in pre_process'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:79:in `catch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:79:in `pre_process'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `block in spawn_threadpool'"]}
<<<
... FAILED
Upload failed. Try again with 'cf push'.
CFoundry::ServerError: 10001: hostname does not match the server certificate (OpenSSL::SSL::SSLError)

I used Bosh to deploy AWS and the only keys I had to configure were for the login part of CloudFoundry. Maybe something is missing in my deployment descriptor ?

After a few research it looks like it's around fog storage configuration. What are the default values for an AWS based deployment ?

Thx in advance

[tchass...@ippon.fr]

No ideas at all ?

[Aram Price]

hi,

I have seen this at least once on a development AWS installation of CF.

In our case the root cause was that the URLs being generated for the S3 buckets were in the format:

* https://<BUCKET_NAME>.s3.amazonaws.com/

instead of 

* https://s3.amazonaws.com/<BUCKET_NAME>/

We found that the hostname addressing of the bucket (BUCKET.s3....) exhibited certificate validation errors.

Unfortunately I can't remember what we changed to resolve this but maybe this will help with your investigation.

aram

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[tchass...@ippon.fr]

The default blobstore is the local one and not S3.

Any ideas for skipping the SSL certificate verification when trying to upload the application's files in the blobstore ?

[Aram Price]

Given than the stack trace contains "lib/fog/aws/requests/storage/put_bucket.rb:36" it seems likely that fog is somehow thinking that it aught to be connecting to S3.

Fog is using the "fog_connection:" hash in your cloud_controller_ng.yml on the CC box as the connection configuration.

aram

[cpt...@gmail.com]

Hi,

Facing the same issue ... did u manage to have it working ?

[tchass...@ippon.fr]

Can you send us your fog_connection configuration section.

Mine was like this :

fog_connection:

        aws_access_key_id: XXXX

        aws_secret_access_key: XXXXX

        provider: local

        local_root: /var/vcap/store/cc_blobstore

Thibault

[da...@castlelaing.com]

I think adding path_style: true to fog_config in your deployment manifest changes the S3 urls to have a bucket_name suffix - see https://github.com/cloudfoundry/cf-release/pull/312/files