Adding storage failed (automatically mount the third storage)

[Brant Fortest]

Hi, Team,

Need your help! Thanks!

Adding more than two storages failed.

First of all, adding Persistent Storage 1 is fine, keep it.

Then, adding Ephemeral0 is fine, keep it as well.

At last, adding neither Ephemeral1 / Persistent Storage 2 failed.

--------------------------------------------------------------------

[20:06:46]		Create Server record in Scalr
[20:07:44]		Provision Server in Cloud Platform
[20:08:23]		Wait for OS to finish booting
[20:08:49]		Wait for Scalarizr Agent to update and start
[20:09:18]		Wait for Agent HostInit phase to complete
		Wait for Agent BeforeHostUp phase to complete timed out
		Wait for Agent HostUp phase to complete
		Done

 

--------------------------------------------------------------------

 

Error in Debug Log

--------------------------------------------------------------------

2016-06-16 20:12:54,069+00:00 - ERROR - scalarizr.ops.system.init - Operation "system.init" (id: fc66c1c5-48b4-45a9-83cf-fbe686fa407d) failed. Reason: timed out

Traceback (most recent call last):

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/api/operation.py", line 273, in _in_progress

    self._completed(self.func(self, *self.func_args, **self.func_kwds))

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/handlers/lifecycle.py", line 441, in handler

    bus.fire("host_init_response", message)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/libs/bases.py", line 33, in fire

    ln(*args, **kwargs)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/handlers/block_device.py", line 103, in on_host_init_response

    self._plug_new_style_volumes(volumes)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/handlers/block_device.py", line 127, in _plug_new_style_volumes

    vol.ensure(mount=bool(vol.mpoint), mkfs=True)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/storage2/volumes/base.py", line 88, in ensure

    self._ensure()

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/storage2/volumes/ebs.py", line 462, in _ensure

    kms_key_id=self.kms_key_id)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/storage2/volumes/ebs.py", line 527, in _create_volume

    encrypted, kms_key_id)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/platform/__init__.py", line 88, in __call__

    return self.invoke(*args, **kwds)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/platform/ec2/__init__.py", line 46, in invoke

    return super(Ec2ConnectionProxy, self).invoke(*args, **kwds)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.6.1-py2.7.egg/scalarizr/platform/__init__.py", line 99, in invoke

    return self.obj(*args, **kwds)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/boto/ec2/connection.py", line 2322, in create_volume

    return self.get_object('CreateVolume', params, Volume, verb='POST')

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/boto/connection.py", line 1192, in get_object

    response = self.make_request(action, params, path, verb)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/boto/connection.py", line 1116, in make_request

    return self._mexe(http_request)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/boto/connection.py", line 913, in _mexe

    self.is_secure)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/boto/connection.py", line 705, in get_http_connection

    return self.new_http_connection(host, port, is_secure)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/boto/connection.py", line 747, in new_http_connection

    connection = self.proxy_ssl(host, is_secure and 443 or 80)

  File "/opt/scalarizr/embedded/lib/python2.7/site-packages/boto/connection.py", line 792, in proxy_ssl

    int(self.proxy_port)), timeout)

  File "/opt/scalarizr/embedded/lib/python2.7/socket.py", line 575, in create_connection

    raise err

timeout: timed out

 

--------------------------------------------------------------------

[Marc O'Brien]

Hi Brant,

It appears that the agent is timing out when making this call to the AWS API.  As a first step for investigation we would need to look at the Scalarizr agent logs.  Can you attach a zip of the contents of /var/log for this instance?

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support


On Thursday, June 16, 2016 at 2:38:53 PM UTC-6, Brant Fortest wrote:

Hi, Team,

Nee your help! Thanks!

Adding the third storage failed.

1. adding Persistent Storage 1 is fine.

2. adding Ephemeral0 is fine.

3. but adding Ephemeral1 / Persistent Storage 2 failed.

[Brant Fortest]

Hi, Marc,

I wanted to provide our scalarizr logs: /var/log/scalarizr_*.log as well as the role_builder log in the same folder on the instance? But there're some customized info in it, I cannot upload here.

Can you just let me know how to debug this issue? The key checkpoints? Thanks a lot!

Best Wishes!
Brant

[Marc O'Brien]

Hi Brant,

It may be difficult to provide guidance without the logs.  We would be looking for context related to the timeout error such as other errors, timeouts, or connection issues.  It would also be useful to test on AWS US vs your AWS China deployment to see if there is any difference in behavior.

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support