Openstack servers with cinder volumes fail

[Patrick Vinas]

I've got both Openstack and AWS environments in an account in scalr (5.11.22 Community, scalarizr agent v. 4.8.2 (stable) and 4.9.9 (latest) )

The only issue I'm having with launching a farm is with roles that are launching in Openstack with a storage volume attached. All AWS roles, and all Openstack roles without cinder storage volumes, launch successfully. 

The error in the UI Servers->Initialization progress is "SSL exception connecting to https://<controller>:5000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)"

First tried adding my internal CA cert to the system trust for the image, to no effect. I can ssh into the failed instances, and I've verified that I can connect to the keystone service at that port (curl https://<controller>:5000/v3 returns as expected). There aren't any errors in the keystone or cinder logs, and the Scalr internal messaging and system logs look fine. If I turn off SSL verification of endpoints in the environment settings, this error persists. 

Does anyone have any ideas for further troubleshooting? 

[Marc O'Brien]

Hi Patrick,

Could you attach the full error log here?  There was previously a similar issue related to Python version that has since been resolved in Enterprise Scalr 6.0.1.  Error log should hopefully provide a bit more context for us.

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[Igor Savchenko]

We will need /var/log/scalarizr_debug.log file from the failed VM. You
can send it to igor [at] scalr.com

Thanks,
Igor

> --
> You received this message because you are subscribed to the Google Groups
> "scalr-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scalr-discus...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Patrick Vinas]

Thanks, Igor and Marc. Debug log is attached, with domain obfuscated.

[Marc O'Brien]

Hi Patrick,

Can you provide a screenshot of your Openstack cloud credentials configuration? You can mask your credentials.  We are looking for the current status of the checkbox for "Enable SSL certificate verification for Keystone endpoints."

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[Patrick Vinas]

As specified in my original post, the issue persists whether "Enable SSL certificate verification" is checked or unchecked (it's currently unchecked).

[Marc O'Brien]

Hi Patrick,

It appears that this is a bug with the current version of Open Source Scalr that has since been resolved in Enterprise Scalr as well as Hosted Scalr.  When testing using the latest agent and insecure OpenStack this issue does not present in Hosted or Enterprise Scalr. 

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[Randy Black]

Is this going to be back ported or is there a work around proposed?  Either a cinder client upgrade on the scalr host or something of that nature?

Thanks!

[Marc O'Brien]

Hi Randy,

An update for Open Source Scalr will be released in the near future.  We do not have a public release timeline for Open Source Scalr, but bug fixes from Enterprise Scalr will periodically be rolled in to Open Source Scalr after some delay.  If this is a blocker for you, we would recommend that you consider Enterprise or Hosted Scalr to ensure you have access to the latest bug fixes at all times.

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[slop...@gmail.com]

I am having this same problem myself. I attached a python script I wrote to work around it (using the OpenStack API), with the credentials in masked and locked environment-level global variables to keep them secure(ish). You add a global variable "volumes" (with a space-separated list of volume sizes) to either the Farm scope (if you want all the systems in the farm to have the same number and size of volumes) or the Farm-role scope.