Cloudify on cloudlab.us new bootstrap error: volume is in error state

56 views

Skip to first unread message

Roberto Bandini

unread,

Nov 27, 2015, 3:15:39 PM11/27/15

to cloudify-users, Federica Paganelli, Ing. Francesca Paradiso

Hi all,

I'm trying to bootstrap Cloudify manager on cloudlab.us platform.
I'm using "Openstack" profile with Kilo version.

After some tests I've understand that unlike HP Helion, with Cloudlab I have to specify also URLs about: Nova, Neutron, Cinder.
To specify Cinder URL I have to use bypass_url in openstack_configuration in openstack-manager-blueprint.yaml.

http://getcloudify.org/guide/3.2/plugin-openstack.html#openstack-configuration

But using CLI 3.2.1 and then Openstack plugin 1.2.1, the method "set_management_url" is missing in cinderclient/client.py file in the class HTTPclient.

This bug was corrected in last version, then I added it manually and I tried to do the bootstrap.

https://github.com/openstack/python-cinderclient

So, now the bootstrap process goes on, validation is ok, the manager VM is created and started, but I get another error while cinder trying to create a volume: "Task failed 'cinder_plugin.volume.create' -> Volume 121da02c-827f-4013-aff5-6342cc3990f7 is in error state".

I paste here the entire error:

2015-11-27 20:33:39 CFY <manager> [volume_22444.create] Task started 'cinder_plu
gin.volume.create'
2015-11-27 20:33:43 LOG <manager> [volume_22444.create] ERROR: Exception raised
on operation [cinder_plugin.volume.create] invocation
Traceback (most recent call last):
  File "D:\Programmi\Cloudify\lib\site-packages\cloudify\decorators.py", line 12
5, in wrapper
    result = func(*args, **kwargs)
  File "D:\Programmi\Cloudify\lib\site-packages\openstack_plugin_common\__init__
.py", line 547, in wrapper
    return f(*args, **kw)
  File "D:\Programmi\Cloudify\lib\site-packages\cinder_plugin\volume.py", line 7
4, in create
    status=VOLUME_STATUS_AVAILABLE)
  File "D:\Programmi\Cloudify\lib\site-packages\openstack_plugin_common\__init__
.py", line 547, in wrapper
    return f(*args, **kw)
  File "D:\Programmi\Cloudify\lib\site-packages\cinder_plugin\volume.py", line 9
2, in wait_until_status
    "Volume {0} is in error state".format(volume_id))
NonRecoverableError: Volume 121da02c-827f-4013-aff5-6342cc3990f7 is in error sta
te
2015-11-27 20:33:43 CFY <manager> [volume_22444.create] Task failed 'cinder_plug
in.volume.create' -> Volume 121da02c-827f-4013-aff5-6342cc3990f7 is in error sta
te [attempt 1/6]
2015-11-27 20:33:43 CFY <manager> 'install' workflow execution failed: Workflow
failed: Task failed 'cinder_plugin.volume.create' -> Volume 121da02c-827f-4013-a
ff5-6342cc3990f7 is in error state
bootstrap failed!
executing teardown due to failed bootstrap

Any idea about how to resolve this error?

Thank you all,

Roberto

Roberto Bandini

unread,

Nov 30, 2015, 2:59:15 PM11/30/15

to cloudify-users, federica....@unifi.it, francesca...@unifi.it

Hi all,

based on this previous answer
https://groups.google.com/d/msg/cloudify-users/JG_yg-fPb4U/DlddyNcwBQAJ
I tried to change my "openstack-manager-blueprint.yaml" getting the following version. Is it correct?

tosca_definitions_version: cloudify_dsl_1_0

imports:
  - http://www.getcloudify.org/spec/cloudify/3.2.1/types.yaml
  - http://www.getcloudify.org/spec/openstack-plugin/1.2.1/plugin.yaml
  - http://www.getcloudify.org/spec/fabric-plugin/1.2.1/plugin.yaml


inputs:
  keystone_username:
    default: ''
    type: string

  keystone_password:
    default: ''
    type: string

  keystone_tenant_name:
    default: ''
    type: string

  keystone_url:
    default: ''
    type: string

  region:
    default: ''
    type: string

  nova_url:
    default: ''
    type: string

  neutron_url:
    default: ''
    type: string

  manager_public_key_name:
    type: string

  agent_public_key_name:
    type: string

  image_id:
    type: string

  flavor_id:
    type: string

  external_network_name:
    type: string

  use_existing_manager_keypair:
    type: boolean
    default: false

  use_existing_agent_keypair:
    type: boolean
    default: false

  manager_server_name:
    default: cloudify-manager-server
    type: string

  manager_server_user:
    default: ubuntu
    type: string

  manager_private_key_path:
    default: ~/.ssh/cloudify-manager-kp.pem
    type: string

  agent_private_key_path:
    default: ~/.ssh/cloudify-agent-kp.pem
    type: string

  agents_user:
    default: ubuntu
    type: string

  resources_prefix:
    default: ''
    type: string

  management_network_name:
    default: cloudify-management-network
    type: string

  management_subnet_name:
    default: cloudify-management-network-subnet
    type: string

  management_router:
    default: cloudify-management-router
    type: string

  manager_security_group_name:
    default: cloudify-sg-manager
    type: string

  agents_security_group_name:
    default: cloudify-sg-agents
    type: string

  manager_port_name:
    default: cloudify-manager-port
    type: string

node_templates:

  management_keypair:
    type: cloudify.openstack.nodes.KeyPair
    properties:
      use_external_resource: { get_input: use_existing_manager_keypair }
      resource_id: { get_input: manager_public_key_name }
      private_key_path: { get_input: manager_private_key_path }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }

  agent_keypair:
    type: cloudify.openstack.nodes.KeyPair
    properties:
      use_external_resource: { get_input: use_existing_agent_keypair }
      resource_id: { get_input: agent_public_key_name }
      private_key_path: { get_input: agent_private_key_path }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }

  management_network:
    type: cloudify.openstack.nodes.Network
    properties:
      resource_id: { get_input: management_network_name }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }

  management_subnet:
    type: cloudify.openstack.nodes.Subnet
    properties:
      resource_id: { get_input: management_subnet_name }
      subnet:
        ip_version: 4
        cidr: 10.67.79.0/24
      openstack_config: { get_property: [openstack_configuration, openstack_config] }
    relationships:
      - target: management_network
        type: cloudify.relationships.contained_in
      - target: router
        type: cloudify.openstack.subnet_connected_to_router

  manager_port:
    type: cloudify.openstack.nodes.Port
    properties:
      openstack_config: { get_property: [openstack_configuration, openstack_config] }
      resource_id: { get_input: manager_port_name }
    relationships:
      - type: cloudify.relationships.contained_in
        target: management_network
      - type: cloudify.relationships.depends_on
        target: management_subnet
      - type: cloudify.openstack.port_connected_to_security_group
        target: management_security_group

  router:
    type: cloudify.openstack.nodes.Router
    properties:
      resource_id: { get_input: management_router }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }
    relationships:
      - target: external_network
        type: cloudify.relationships.connected_to

  external_network:
    type: cloudify.openstack.nodes.Network
    properties:
      use_external_resource: true
      resource_id: { get_input: external_network_name }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }

  agents_security_group:
    type: cloudify.openstack.nodes.SecurityGroup
    properties:
      resource_id: { get_input: agents_security_group_name }
      security_group:
        description: Security group for Cloudify agent VMs
      rules:
        - port: 22
          remote_ip_prefix: { get_property: [management_subnet, subnet, cidr] }
        - port: 5985
          remote_ip_prefix: { get_property: [management_subnet, subnet, cidr] }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }

  management_security_group:
    type: cloudify.openstack.nodes.SecurityGroup
    properties:
      resource_id: { get_input: manager_security_group_name }
      security_group:
        description: Security group for Cloudify Manager VM
      rules:
        - port: 80
          remote_ip_prefix: 0.0.0.0/0
        - port: 443
          remote_ip_prefix: 0.0.0.0/0    
        - port: 22
          remote_ip_prefix: 0.0.0.0/0
        - port: 8101
          remote_ip_prefix: { get_property: [management_subnet, subnet, cidr] }
        - port: 5672
          remote_ip_prefix: { get_property: [management_subnet, subnet, cidr] }
        - port: 53229
          remote_ip_prefix: { get_property: [management_subnet, subnet, cidr] }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }

  manager_server_ip:
    type: cloudify.openstack.nodes.FloatingIP
    properties:
      floatingip:
        floating_network_name: { get_input: external_network_name }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }

  manager_server:
    type: cloudify.openstack.nodes.Server
    properties:
      resource_id: { get_input: manager_server_name }
      install_agent: false
      server:
        image: { get_input: image_id }
        flavor: { get_input: flavor_id }
      management_network_name: { get_property: [management_network, resource_id] }
      openstack_config: { get_property: [openstack_configuration, openstack_config] }
    relationships:
      - target: manager_server_ip
        type: cloudify.openstack.server_connected_to_floating_ip
      - target: management_keypair
        type: cloudify.openstack.server_connected_to_keypair
      - target: manager_port
        type: cloudify.openstack.server_connected_to_port


  openstack_configuration:
    type: openstack_configuration
    properties:
      openstack_config:
        username: { get_input: keystone_username }
        password: { get_input: keystone_password }
        tenant_name: { get_input: keystone_tenant_name }
        auth_url: { get_input: keystone_url }
        region: { get_input: region }
        nova_url: { get_input: nova_url }
        neutron_url: { get_input: neutron_url }
        custom_configuration:
          cinder_client:
            bypass_url: http://......

  manager:
    type: cloudify.nodes.CloudifyManager
    properties:
      cloudify_packages:
        agents:
          ubuntu_agent_url: http://gigaspaces-repository-eu.s3.amazonaws.com/org/cloudify3/3.2.1/ga-RELEASE/cloudify-ubuntu-agent_3.2.1-ga-b212_amd64.deb
          centos_agent_url: http://gigaspaces-repository-eu.s3.amazonaws.com/org/cloudify3/3.2.1/ga-RELEASE/cloudify-centos-final-agent_3.2.1-ga-b212_amd64.deb
          windows_agent_url: http://gigaspaces-repository-eu.s3.amazonaws.com/org/cloudify3/3.2.1/ga-RELEASE/cloudify-windows-agent_3.2.1-ga-b212_amd64.deb
        docker:
           docker_url: http://gigaspaces-repository-eu.s3.amazonaws.com/org/cloudify3/3.2.1/ga-RELEASE/cloudify-docker_3.2.1-ga-b212.tar

      cloudify:
        resources_prefix: { get_input: resources_prefix }

        cloudify_agent:
          min_workers: 0
          max_workers: 5
          remote_execution_port: 22
          user: { get_input: agents_user }

        workflows:
          task_retries: -1  # this means forever
          task_retry_interval: 30

        policy_engine:
          start_timeout: 30

        security:
          #########################################################################
          # Setting 'enabled' to true will activate security on the REST service,
          # according to the below configuration. Each request received by the
          # REST service will be authenticated, and communication can be limited
          # to SSL.
          enabled: false

          #########################################################################
          # A list of one or more authentication providers. Cloudify will use
          # these providers in the order specified in the list (Mandatory).
          # The list below sets Flask-secuREST's password-based and token-based
          # authentication providers.
          # ** Note: Passwords are usually not store as plaintext. Set passowrd_hash
          #          to match the hash scheme used in the selected datastore.
          #          Acceptable values: 'bcrypt', 'des_crypt', 'pbkdf2_sha256',
          #          pbkdf2_sha512', 'sha256_crypt' and 'sha512_crypt'.
          authentication_providers:
            - name: password
              implementation: flask_securest.authentication_providers.password:PasswordAuthenticator
              properties:
                password_hash: plaintext
            - name: token
              implementation: flask_securest.authentication_providers.token:TokenAuthenticator
              properties:
                secret_key: my_secret

          #########################################################################
          # A user-store implementation is used to find a matching user
          # and load its details (Mandatory unless you specified a custom
          # authentication provider that does not require it).
          # The configuration below will use Flask-secuREST's "simple userstore"
          # implementation, with the users listed inline.
          userstore_driver:
            implementation: flask_securest.userstores.simple:SimpleUserstore
            properties:
              userstore:
                user1:
                  username: example_user1
                  password: example_password1
                  email: example_user1@your_domain.dom
                user2:
                  username: example_user2
                  password: example_password2
                  email: example_user2@your_domain.dom
                user3:
                  username: example_user3
                  password: example_password3
                  email: example_user3@your_domain.dom
              identifying_attribute: username

          #########################################################################
          # REST calls to "MANAGER_IP/tokens" will generate and return
          # authentication tokens if an auth_token_generator is set (Optional).
          # The below configuration sets Flask_secuREST's token module as the
          # token generator.
          # ** Note: Set the same secret key as in your token authenticator
          auth_token_generator:
            implementation: flask_securest.authentication_providers.token:TokenAuthenticator
            properties:
              secret_key: my_secret
              expires_in_seconds: 600

          #########################################################################
          # Enabling SSL limits communication with the server to SSL only (Optional).
          # If enabled, both certificate and private key are mandatory.
          ssl:
            enabled: false
            certificate_path: ""
            private_key_path: ""

          #########################################################################
          # Security log file properties (Optional, the default values are as
          # shown below).
          # ** Note: The log file is located on the rest-service container
          #
          audit_log_file: /var/log/cloudify/rest-security-audit.log
          audit_log_level: INFO # values: CRITICAL, ERROR, WARNING, INFO, DEBUG
          audit_log_file_size_MB: 100
          audit_log_files_backup_count: 20

    relationships:
      - target: manager_server
        type: cloudify.relationships.contained_in

    interfaces:
      cloudify.interfaces.lifecycle:
        configure:
          implementation: fabric.fabric_plugin.tasks.run_task
          inputs:
            tasks_file: scripts/configure.py
            task_name: configure
            task_properties:
              openstack_config: { get_property: [openstack_configuration, openstack_config] }
            fabric_env:
              user: { get_input: manager_server_user }
              key_filename: { get_property: [management_keypair, private_key_path] }
              host_string: { get_attribute: [manager_server_ip, floating_ip_address] }
        start:
          implementation: fabric.fabric_plugin.tasks.run_module_task
          inputs:
            task_mapping: cloudify_cli.bootstrap.tasks.bootstrap_docker
            task_properties:
              cloudify_packages: { get_property: [manager, cloudify_packages] }
              agent_local_key_path: { get_property: [agent_keypair, private_key_path] }
              provider_context: { get_attribute: [manager, provider_context] }
            fabric_env:
              user: { get_input: manager_server_user }
              key_filename: { get_property: [management_keypair, private_key_path] }
              host_string: { get_attribute: [manager_server_ip, floating_ip_address] }
        stop:
          implementation: fabric.fabric_plugin.tasks.run_module_task
          inputs:
            task_mapping: cloudify_cli.bootstrap.tasks.stop_manager_container
            fabric_env:
              user: { get_input: manager_server_user }
              key_filename: { get_property: [management_keypair, private_key_path] }
              host_string: { get_attribute: [manager_server_ip, floating_ip_address] }
        delete:
          implementation: fabric.fabric_plugin.tasks.run_module_task
          inputs:
            task_mapping: cloudify_cli.bootstrap.tasks.stop_docker_service
            fabric_env:
              user: { get_input: manager_server_user }
              key_filename: { get_property: [management_keypair, private_key_path] }
              host_string: { get_attribute: [manager_server_ip, floating_ip_address] }

      cloudify.interfaces.validation:
        creation:
          implementation: cli.cloudify_cli.bootstrap.tasks.creation_validation
          inputs:
            cloudify_packages: { get_property: [manager, cloudify_packages] }

node_types:
  openstack_configuration:
    derived_from: cloudify.nodes.Root
    properties:
      openstack_config: {}

plugins:
  cli:
    install: false
    executor: central_deployment_agent

outputs:
  manager_ip:
    value: { get_attribute: [manager_server_ip, floating_ip_address] }

Anyway, I tried again to bootstrap with that and now I get a different error:

2015-11-30 20:45:17 CFY <manager> [manager_ddaa3.configure] Sending task 'fabric
_plugin.tasks.run_task' [attempt 6/6]
2015-11-30 20:45:17 CFY <manager> [manager_ddaa3.configure] Task started 'fabric
_plugin.tasks.run_task' [attempt 6/6]
2015-11-30 20:45:17 LOG <manager> [manager_ddaa3.configure] INFO: running task:
configure from scripts/configure.py
2015-11-30 20:45:17 LOG <manager> [manager_ddaa3.configure] INFO: preparing fabr
ic environment...
2015-11-30 20:45:17 LOG <manager> [manager_ddaa3.configure] INFO: environment pr
epared successfully
2015-11-30 20:46:07 LOG <manager> [manager_ddaa3.configure] ERROR: Exception rai
sed on operation [fabric_plugin.tasks.run_task] invocation

Traceback (most recent call last):
  File "D:\Programmi\Cloudify\lib\site-packages\cloudify\decorators.py", line 12
5, in wrapper
    result = func(*args, **kwargs)

  File "D:\Programmi\Cloudify\lib\site-packages\fabric_plugin\tasks.py", line 74
, in run_task
    return _run_task(task, task_properties, fabric_env)
  File "D:\Programmi\Cloudify\lib\site-packages\fabric_plugin\tasks.py", line 95
, in _run_task
    return task(**task_properties)
  File "<string>", line 38, in configure
  File "<string>", line 49, in _copy_openstack_configuration_to_manager
  File "D:\Programmi\Cloudify\lib\site-packages\fabric\network.py", line 639, in
 host_prompting_wrapper
    return func(*args, **kwargs)
  File "D:\Programmi\Cloudify\lib\site-packages\fabric\operations.py", line 346,
 in put
    ftp = SFTP(env.host_string)
  File "D:\Programmi\Cloudify\lib\site-packages\fabric\sftp.py", line 30, in __i
nit__
    self.ftp = connections[host_string].open_sftp()
  File "D:\Programmi\Cloudify\lib\site-packages\fabric\network.py", line 151, in
 __getitem__
    self.connect(key)
  File "D:\Programmi\Cloudify\lib\site-packages\fabric\network.py", line 143, in
 connect
    self[key] = connect(user, host, port, cache=self)
  File "D:\Programmi\Cloudify\lib\site-packages\fabric\network.py", line 565, in
 connect
    raise NetworkError(msg, e)
NetworkError: Timed out trying to connect to 128.110.155.130 (tried 5 times)
2015-11-30 20:46:07 CFY <manager> [manager_ddaa3.configure] Task failed 'fabric_
plugin.tasks.run_task' -> Timed out trying to connect to 128.110.155.130 (tried
5 times) [attempt 6/6]
2015-11-30 20:46:07 CFY <manager> 'install' workflow execution failed: Workflow
failed: Task failed 'fabric_plugin.tasks.run_task' -> Timed out trying to connec
t to 128.110.155.130 (tried 5 times)

bootstrap failed!
executing teardown due to failed bootstrap

What could be the problem? Can you help us?

Thank you,

Roberto

tram...@gigaspaces.com

unread,

Dec 6, 2015, 4:48:01 AM12/6/15

to cloudify-users, federica....@unifi.it, francesca...@unifi.it

Hi Roberto,

We've seen this on a few Openstack installations. Generally this error indicates that something is not right with the Openstack installation.

Can you check if you can reproduce the error manually, and discuss with your Openstack provider if you do manage to reproduce it manually?

NonRecoverableError: Volume 121da02c-827f-4013-aff5-6342cc3990f7 is in error state

Roberto Bandini

unread,

Dec 6, 2015, 9:14:21 AM12/6/15

to tram...@gigaspaces.com, cloudify-users, Federica Paganelli, Francesca Paradiso

Hi Trammel,

we've received this answer, now it seems it's ok:

https://groups.google.com/d/msg/cloudlab-users/rOI8EZWH1hg/b_4Dl-0yBAAJ

Thank you,

Roberto

--
You received this message because you are subscribed to a topic in the Google Groups "cloudify-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloudify-users/Bh_gJOZa8X0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cloudify-user...@googlegroups.com.
To post to this group, send email to cloudif...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloudify-users/f848c232-6a0e-4cf7-a49f-2871d3d23670%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward

0 new messages