SSL: CERTIFICATE_VERIFY_FAILED error while installing cloud environment

[Tomasz Osiński]

Hello,

I cannot install my OpenStack environment using Cloudify, because of SSL certificate verification error. My OpenStack endpoint is accessible only with HTTPS, but I have tried to use various insecure modes. With no success. I'm using the pre-packaged Cloudify 4.2 image and try to install OpenStack environment with following command:

cfy install cloudify-environment-setup-latest/openstack-blueprint.yaml -i cloudify-environment-setup-latest/inputs/openstack.yaml

using GitHub repo example: https://github.com/cloudify-examples/cloudify-environment-setup

The error is:

File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 766, in post
    return self.request(url, 'POST', **kwargs)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/positional/__init__.py", line 108, in inner
    return wrapped(*args, **kwargs)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
    resp = send(**kwargs)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 678, in _send_request
    raise exceptions.SSLError(msg)
SSLError: SSL exception connecting to https://-url-/v2.0/tokens: HTTPSConnectionPool(host='-url-', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),))


I've tried already different workarounds:
1) Set "export CLOUDIFY_SSL_TRUST_ALL=true;" does not bring any result.
2) By setting custom configuration in openstack-blueprint.yaml:

custom_configuration:
    nova_client:
     insecure: true
    keystone_client:
      insecure: true
    neutron_client:
      insecure: true
    cinder_client:
      insecure: true

The result is that requests are sent to HTTP instead of HTTPS, but this is useless as my endpoint is not accesible via HTTP.

What I only need is to set flag "--insecure" / "-k" as it is possible with cURL. With cURL I'm able to access my endpoint using -k option. I would like to have the same behaviour with Cloudify..

Please, help me with this annoying issue.

[Tadej Borovšak]

On Mon, Feb 05, 2018 at 03:01:23AM -0800, Tomasz Osiński wrote:
> Hello,
>
> I cannot install my OpenStack environment using Cloudify, because of SSL
> certificate verification error. My OpenStack endpoint is accessible only
> with HTTPS, but I have tried to use various insecure modes. With no
> success. I'm using the pre-packaged Cloudify 4.2 image and try to install
> OpenStack environment with following command:
>
> cfy install cloudify-environment-setup-latest/openstack-blueprint.yaml -i
> cloudify-environment-setup-latest/inputs/openstack.yaml
>
> using GitHub repo example:
> https://github.com/cloudify-examples/cloudify-environment-setup

If my memory serves me correctly, OpenStack plugin can be instructed to trust
all certificates by setting "insecure" configuration parameter to true. In
your specific use case, that means adding this line

insecure: true

to the file [1] after the line 140. This should resolve your problem.

Cheers,
Tadej


[1] https://github.com/cloudify-examples/cloudify-environment-setup/blob/latest/openstack-blueprint.yaml

--
Tadej Borovšak
tade...@gmail.com
tadej.b...@gmail.com

[Tomasz Osiński]

Thanks for your quick answer. However, the fix you mentioned is covered by workaround 2) that I tried to do. The result is that instead of HTTPS Cloudify requests HTTP. As my OpenStack endpoint is not accesible via HTTP, the result is as follows:

File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 766, in post
    return self.request(url, 'POST', **kwargs)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/positional/__init__.py", line 108, in inner
    return wrapped(*args, **kwargs)
  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
    resp = send(**kwargs)

  File "/opt/mgmtworker/env/plugins/default_tenant/cloudify-openstack-plugin-2.5.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 690, in _send_request
    raise exceptions.ConnectFailure(msg)
ConnectFailure: Unable to establish connection to http://-url-:5000/v2.0/tokens: ('Connection aborted.', BadStatusLine("''",))

[Tadej Borovšak]

> ConnectFailure: Unable to establish connection to *http*://-url-:5000/v2.0/tokens:
> ('*Connection aborted*.', BadStatusLine("''",))

This is strange. OpenStack clients usually do not transform the urls at all.
You can try to connect to your OpenStack from python by running next sequence
of commands:

$ cd /tmp
$ virtualenv venv
$ . venv/bin/activate
$ pip install 'keystoneauth1<3'
$ python
>>> from keystoneauth1 import loading, session
>>> auth_opts = dict(auth_url="https://my.keystone.com:5000/v2.0",
>>> username="myuser",
>>> password="mypassword",
>>> tenant_name="proj",
>>> )
>>> loader = loading.get_plugin_loader("password")
>>> auth = loader.load_from_options(**auth_opts)
>>> sess = session.Session(auth=auth, verify=False)
>>> sess.get_token()

If this is working, then the cloudify plugin should work too.

Cheers,
Tadej

[Tomasz Osiński]

It's really strange.. Something is unstable with keystoneauth libraries. When I run your python script for the first time, it failed as for cloudify plugin. Then I've made a change from verify=False to True and do sess.get_token(). As expected it raises SSLError. Then, I changed again to verify=False and perform sess.get_token(). Surpisingly, it works then. And what's strange this behaviour is repeatable, it means for the first time sess.get_token() always changes https -> http and requests fail. Unfortunately, cloudify plugin still don't work...

[Earthman T]

Hello,

I notice that you are using the plugin version 2.5.0. If you update your version, this issue is addressed in version 2.5.2.

I would suggest upgrading to 2.6.0.

https://github.com/cloudify-cosmo/cloudify-openstack-plugin/releases/tag/2.6.0

Trammell

[Tomasz Osiński]

Hello,

Thanks for your answers. I finally solve the problem. The problem laid in our OpenStack deployment, where Keystone returns HTTP link (instead of HTTPS) as a "self" link in response for GET request from client to Keystone service.