Issue with Scalarizr on Windows 2012

[Jeff Gojkovich]

I am trying to launch an instance in an Openstack environment utilizing Scalr and it hangs at "Waiting for OS to finish booting".  It looks like the scalarizr agent isn't automatically started on the instance and when I manually start it, the build still does not finish.  Let me know what steps I can take to further troubleshoot the issue.

This is the scalarizr log

2016-09-20 21:43:19,257+00:00 - INFO - scalarizr.app - [pid: 1840] Starting scalarizr 4.11.2

2016-09-20 21:43:19,273+00:00 - INFO - scalarizr.node - State: unknown

2016-09-20 21:43:19,305+00:00 - INFO - scalarizr.app - Polling UpdateClient state...

2016-09-20 21:44:20,039+00:00 - WARNING - scalarizr.app - Failed to talk to UpdateClient: Failed to get UpdateClient status. Timeout: 60 seconds reached

2016-09-20 21:45:20,555+00:00 - ERROR - scalarizr.app - UpdateClient(scalr-init) completion flag C:\opt\scalarizr\var\run\result.json still not available after 60 seconds.

Check that 'ScalrUpdClient' service has no errors in log.

This is in the scalarizr_update.log file

2016-09-20 21:43:25,539 - INFO - agent.scalrinit.metadata - Meta-data providers that fit: [<ConfigDriveProvider>, <DefaultProvider>]

2016-09-20 21:43:25,539 - DEBUG - agent.scalrinit.metadata.configdrive - Found meta_data.json

2016-09-20 21:43:25,539 - INFO - agent.scalrinit.metadata - Best match for Instance-Id: <ConfigDriveProvider>

2016-09-20 21:43:25,539 - DEBUG - agent.scalrinit.metadata.configdrive - Found meta_data.json

2016-09-20 21:43:25,539 - INFO - agent.scalrinit.metadata - Best-match for User-data: <ConfigDriveProvider>

2016-09-20 21:43:25,539 - ERROR - agent.scalrinit.app - No Instance-Id: User-data file /mnt/config\openstack\latest\meta_data.json do not exist or not accessible.

2016-09-20 21:43:25,539 - INFO - upd.client - Stopping UpdateClient

2016-09-20 21:43:25,539 - INFO - upd.client - Stopped

2016-09-20 21:43:25,539 - ERROR - upd.client - Detailed exception information below:

Traceback (most recent call last):

  File "C:\opt\scalarizr\4.11.2\embedded\lib\site-packages\scalarizr-4.11.2-py2.7.egg\scalarizr\updclient\app.py", line 184, in do_start

    agent.scalrinit.app.main_start()

  File "C:\opt\scalarizr\4.11.2\embedded\lib\site-packages\fatmouse_agent-0.1.26ad7587c-py2.7.egg\agent\scalrinit\app.py", line 26, in main_start

    instance_id = metadata.instance_id()

  File "C:\opt\scalarizr\4.11.2\embedded\lib\site-packages\fatmouse_agent-0.1.26ad7587c-py2.7.egg\common\utils\mixutil.py", line 297, in wrapper

    func._memorized = func(*args, **kwds)

  File "C:\opt\scalarizr\4.11.2\embedded\lib\site-packages\fatmouse_agent-0.1.26ad7587c-py2.7.egg\agent\scalrinit\metadata.py", line 578, in instance_id

    return create_provider(get_datasource()['instance_id']).instance_id

  File "C:\opt\scalarizr\4.11.2\embedded\lib\site-packages\fatmouse_agent-0.1.26ad7587c-py2.7.egg\agent\scalrinit\metadata.py", line 333, in instance_id

    return json.loads(self.user_data)['uuid']

  File "C:\opt\scalarizr\4.11.2\embedded\lib\site-packages\fatmouse_agent-0.1.26ad7587c-py2.7.egg\agent\scalrinit\metadata.py", line 326, in user_data

    .format(self._metadata_path))

NoUserdataError: User-data file /mnt/config\openstack\latest\meta_data.json do not exist or not accessible.

[Marc O'Brien]

Hi Jeff,

What version of Scalr are you using?  Have Openstack Windows 2012 instances previously launched successfully?  Are you seeing these issues with any other instances?  Copies of the following logs from C:\opt\scalarizr\var\log may be helpful:

scalarizr.log

scalarizr_debug.log

scalarizr_update.log

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[Jeff Gojkovich]

Hey Marc

We are using 5.11.22 (Community Edition)

I am attaching the entire log directory.  We have never got a Windows instance to launch successfully with the scalarizr agent.  We have got CentOS and RedHat instances to launch successfully.

[Marc O'Brien]

Hi Jeff,

It appears that the instance is unable to connect to the Openstack meta-data server and eventually gives up:
 
agent.scalrinit.metadata.openstack - Does not fit: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /openstack/latest/meta_data.json (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x00000000020EB278>, 'Connection to 169.254.169.254 timed out.
...

ERROR - agent.scalrinit.app - No Instance-Id: User-data file /mnt/config\openstack\latest\meta_data.json do not exist or not accessible.

....
ERROR - upd.client ......  NoUserdataError: User-data file /mnt/config\openstack\latest\meta_data.json do not exist or not accessible.


We are calling http://169.254.169.254/openstack/latest/meta_data.json which should contain the user-data.  Are you able to execute the following from the impacted instance?  Are you able to do the same from your successfully launched CentOS or RHEL instances?

curl http://169.254.169.254/openstack/latest/meta_data.json

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[Jeff Gojkovich]

Mark,

That does not seem to return on any instance type. I have tried a curl on centos, redhat, and windows and it doesn't work on any of them.  Although the CentOS and RedHat images run the scalarizr agent just fine.

Any recommendations?  We are running a modified instance of Openstack and it doesn't support the metadata service.  It does look like it supports config drive though.

[Jeff Gojkovich]

It also appears that it is not pointing to the config-drive that is being mounted.

"NoUserdataError: User-data file /mnt/config\openstack\latest\meta_data.json do not exist or not accessible."

The config-drive is D: rather than /mnt/config like it is in the *nix systems.  Is that a config item within the scalarizr application?