500: Internal Server Error during executions

[Grzegorz Panek]

Hello, I'm stuggling with the following problem:

[centos@cfm-manager kubernetes-cluster-TOSCA]$ cfy deployments create -b k8s-cluster k8s
Creating new deployment from blueprint k8s-cluster...
Deployment created. The deployment's id is k8s
[centos@cfm-manager kubernetes-cluster-TOSCA]$ cfy executions start install -d k8s
Executing workflow install on deployment k8s [timeout=900 seconds]
An error occurred on the server: 500: Internal Server Error

Error output: (/var/log/cloudify/rest)

04/07/2018 08:18:09 [ERROR] [manager_rest.server] Internal Error
Traceback (most recent call last):
  File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/manager/env/lib/python2.7/site-packages/flask_restful/__init__.py", line 261, in wrapper
    resp = resource(*args, **kwargs)
  File "/opt/manager/env/lib/python2.7/site-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/opt/manager/env/lib/python2.7/site-packages/flask_restful/__init__.py", line 341, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/security/secured_resource.py", line 33, in wrapper
    response = func(*args, **kwargs)
  File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/rest/rest_decorators.py", line 97, in wrapper
    return func(*args, **kwargs)
  File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/security/authorization.py", line 78, in wrapper
    return func(*args, **kwargs)
  File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/rest/rest_decorators.py", line 137, in wrapper
    response = f(*args, **kwargs)
  File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/rest/resources_v1/executions.py", line 121, in post
    dry_run=dry_run, bypass_maintenance=bypass_maintenance)
  File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/resource_manager.py", line 438, in execute_workflow
    self._verify_deployment_environment_created_successfully(deployment_id)
  File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/resource_manager.py", line 1208, in _verify_deployment_environment_created_successfully
    deployment_id, env_creation.error))
RuntimeError: Can't launch executions since environment creation for deployment k8s has failed: Traceback (most recent call last):
  File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 548, in _remote_workflow_child_thread
    workflow_result = self._execute_workflow_function()
  File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 577, in _execute_workflow_function
    result = self.func(*self.args, **self.kwargs)
  File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify_system_workflows/deployment_environment.py", line 100, in create
    return graph.execute()
  File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks_graph.py", line 129, in execute
    self._handle_terminated_task(task)
  File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks_graph.py", line 203, in _handle_terminated_task
    raise RuntimeError(message)
RuntimeError: Workflow failed: Task failed 'cloudify_agent.operations.install_plugins' -> Central deployment agent managed plugins can only be installed using the REST plugins API. [{u'distribution_release': u'core', u'archive_name': u'cloudify_openstack_plugin-2.6.0-py27-none-linux_x86_64-centos-Core.wgn', u'package_name': u'cloudify-openstack-plugin', u'distribution_version': u'7.3.1611', u'tenant_name': u'default_tenant', u'package_version': u'2.6.0', u'distribution': u'centos', u'visibility': u'tenant', u'file_server_path': u'', u'resource_availability': u'tenant', u'package_source': u'../cloudify-openstack-plugin/', u'id': u'4f387340-2f5b-4d16-99a2-8c51e1797fb9', u'yaml_url_path': u'plugin:cloudify-openstack-plugin?version=2.6.0&distribution=centos', u'supported_py_versions': [u'py27'], u'uploaded_at': u'2018-07-03T13:28:03.811Z', u'created_by': u'admin', u'private_resource': False, u'supported_platform': u'linux_x86_64', u'wheels': [u'keystoneauth1-2.21.0-py2.py3-none-any.whl', u'python_novaclient-7.0.0-py2.py3-none-any.whl', u'python_keystoneclient-3.5.0-py2.py3-none-any.whl', u'python_neutronclient-6.0.0-py2.py3-none-any.whl', u'python_cinderclient-1.9.0-py2.py3-none-any.whl', u'python_glanceclient-2.5.0-py2.py3-none-any.whl', u'six-1.11.0-py2.py3-none-any.whl', u'stevedore-1.28.0-py2.py3-none-any.whl', u'iso8601-0.1.12-py2.py3-none-any.whl', u'pbr-3.1.1-py2.py3-none-any.whl', u'requests-2.18.4-py2.py3-none-any.whl', u'oslo.utils-3.35.0-py2.py3-none-any.whl', u'oslo.serialization-2.23.0-py2.py3-none-any.whl', u'Babel-2.5.3-py2.py3-none-any.whl', u'oslo.i18n-3.19.0-py2.py3-none-any.whl', u'debtcollector-1.19.0-py2.py3-none-any.whl', u'oslo.config-5.2.0-py2.py3-none-any.whl', u'os_client_config-1.28.0-py2.py3-none-any.whl', u'osc_lib-1.9.0-py2-none-any.whl', u'cliff-2.11.0-py2-none-any.whl', u'netaddr-0.7.19-py2.py3-none-any.whl', u'requests_toolbelt-0.8.0-py2.py3-none-any.whl', u'certifi-2018.1.18-py2.py3-none-any.whl', u'chardet-3.0.4-py2.py3-none-any.whl', u'idna-2.6-py2.py3-none-any.whl', u'urllib3-1.22-py2.py3-none-any.whl', u'pytz-2017.3-py2.py3-none-any.whl', u'pyparsing-2.2.0-py2.py3-none-any.whl', u'funcsigs-1.0.2-py2.py3-none-any.whl', u'monotonic-1.4-py2.py3-none-any.whl', u'rfc3986-1.1.0-py2.py3-none-any.whl', u'appdirs-1.4.3-py2.py3-none-any.whl', u'requestsexceptions-1.3.0-py2.py3-none-any.whl', u'openstacksdk-0.10.0-py2.py3-none-any.whl', u'jsonschema-2.6.0-py2.py3-none-any.whl', u'jsonpatch-1.21-py2.py3-none-any.whl', u'jmespath-0.9.3-py2.py3-none-any.whl', u'os_service_types-1.1.0-py2-none-any.whl', u'decorator-4.2.1-py2.py3-none-any.whl', u'futures-3.2.0-py2-none-any.whl', u'jsonpointer-2.0-py2.py3-none-any.whl', u'cloudify_plugins_common-3.4.1-py2-none-any.whl', u'cloudify_openstack_plugin-2.6.0-py2-none-any.whl', u'IPy-0.81-py2-none-any.whl', u'cloudify_rest_client-3.4.1-py2-none-any.whl', u'pika-0.9.14-py2-none-any.whl', u'networkx-1.8.1-py2-none-any.whl', u'proxy_tools-0.1.0-py2-none-any.whl', u'bottle-0.12.7-py2-none-any.whl', u'Jinja2-2.7.2-py2-none-any.whl', u'positional-1.2.1-py2.py3-none-any.whl', u'simplejson-3.13.2-cp27-cp27mu-linux_x86_64.whl', u'prettytable-0.7.2-py2-none-any.whl', u'warlock-1.2.0-py2.py3-none-any.whl', u'MarkupSafe-1.0-cp27-cp27mu-linux_x86_64.whl', u'wrapt-1.10.11-cp27-cp27mu-linux_x86_64.whl', u'netifaces-0.10.6-cp27-cp27mu-linux_x86_64.whl', u'msgpack_python-0.5.1-cp27-cp27mu-linux_x86_64.whl', u'PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl', u'cmd2-0.7.9-py2-none-any.whl', u'unicodecsv-0.14.1-py2-none-any.whl', u'munch-2.2.0-py2-none-any.whl', u'ipaddress-1.0.19-py2-none-any.whl', u'dogpile.cache-0.6.4-py2.py3-none-any.whl', u'deprecation-1.0.1-py2-none-any.whl', u'pyperclip-1.6.0-py2-none-any.whl', u'functools32-3.2.3.post2-py2-none-any.whl'], u'excluded_wheels': []}]

Instalation process:
https://github.com/cloudify-cosmo/cloudify-manager-install

Cloudify Version:

[centos@cfm-manager ~]$ cfy --version
Community edition
Cloudify CLI 18.6.28
Cloudify Manager 18.6.28~community [ip=localhost]

Moreover I've modificated /etc/cloudify/config.yaml in order to connect with CFY agent on external machine:

  cloudify_agent: &id001
  # A dict of network names and IP addresses associated with them.
  # By default, there is only a "default" network, with the manager's
  # private IP associated with it. This network can be overwritten
    networks: {default: 192.168.1.43, external: 10.254.188.158}

Does anyone struggled with this also? or know how can i manage this problem?

Thanks!

[Trammell -]

Can you try updating the plugin to 2.9.2?

https://github.com/cloudify-cosmo/cloudify-openstack-plugin/releases/tag/2.9.2

[Grzegorz Panek]

The same behaviour as previously:

[Trammell -]

Can you provide the entire log? The screenshot is cut off.

[Grzegorz Panek]

I uploaded logs file, you can see error at the end of the logs.

[Trammell -]

Is your Cloudify Manager from a QCow image or a Docker Image or did you install from RPM? What is your OS/Platform/Distribution?

[Grzegorz Panek]

Cloudify Manager is installed on CentOS 7 from RPM

[Trammell -]

Hi

It appears that there is a bug in the Cloudify Agent for this release. Can you check with version 18.2.28?

Thanks.

[Grzegorz Panek]

Okay, now it works fine with version 18.7.12

[id...@cloudify.co]

hello.

sorry for the inconvenience, there is probably a glitch in the build released this week.

fixed build will be published next week.