New Scalr and nodes stuck in pending, scalarizer not listenning on 8013,8014

281 views
Skip to first unread message

Enrico Kern

unread,
Jun 26, 2015, 2:34:48 PM6/26/15
to scalr-...@googlegroups.com
Hello,

i just deployed the latest Scalr Open Source and try to launch an image.

I created a fresh Ubuntu 14.04 image (also tested with centos) using add image, added it to a farm and role and deployed it. The server is always tuck as "Boot OS" and pending. However i can ssh into the server, so provisioning works.

I found out that scalarizer is not listenning on port 8013 and 8014 and the update service constantly returns this error:

2015-06-26 18:28:59,948 - DEBUG - requests.packages.urllib3.connectionpool - Starting new HTTP connection (1): $hostname
2015-06-26 18:29:00,061 - WARNING - scalarizr.queryenv - QueryEnv failed: 403 Client Error: Forbidden
2015-06-26 18:29:00,061 - WARNING - scalarizr.queryenv - Sleep 30 seconds before next attempt...

(i replaced $hostname)

In the web logs on the scalr server i see:

==> web.proxy.access.log <==
xx.xxx.xxx.xx  (for "-") - - [26/Jun/2015:18:32:31 +0000] "GET /query-env/2012-07-01/get-latest-version?operation=get-latest-version&version=2012-07-01 HTTP/1.1" 403 23 "-" "python-requests/2.5.3 CPython/2.7.8 Linux/2.6.32-358.2.1.el6.x86_64"


no errors nothing.  What i find strange is also that if i try to curl that /message url i get this:


<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://127.0.0.1:6270/messaging/">here</a>.</p>
</body></html>


I really have no idea anymore and its just a standard setup. Maybe someone can shed some light. Any help is appreciated.

Daniele Testa

unread,
Jun 27, 2015, 1:48:23 AM6/27/15
to scalr-...@googlegroups.com
Hi!

Please tell me the steps you took when you installed Scalr.

What version of Scalarizer is installed on the server?

Also, could you provide the following files:

/etc/scalr-server/scalr-server.rb
/etc/scalr-server/scalr-server-local.rb
/var/log/scalarizr_debug.log
/var/log/scalarizr_update.log

Regards,
Daniele

Enrico Kern

unread,
Jun 27, 2015, 4:22:33 AM6/27/15
to scalr-...@googlegroups.com
Hello Daniele,

5.3.2 (Open Source Edition)

fa7c962 (Thu, 18 Jun 2015 22:36:35 +0300)


the only difference is that afterwards i moved the databases to our mysql cluster and adjusted that.


/etc/scalr-server/scalr-server.rb (i have no scalr-server-local.rb)

routing[:endpoint_host] = "scalr.mydomain.com"


# Point the app to the Scalr main DB server.
app[:mysql_scalr_host] = 'x.x.x.x'   # Change this to the hostname / IP of your Scalr main DB server.
app[:mysql_scalr_port] = 3306   # Make sure this matches the MySQL bind port.

# Point the app to the Scalr Cost Analytics DB server.
app[:mysql_analytics_host] = 'x.x.x.x'   # Change this to the hostname / IP of your Scalr main DB server.
app[:mysql_analytics_port] = 3306   # Make sure this matches the MySQL bind port.


Scalarizer Log right after provisioning shows this (strange that it is terminated)

 cat /var/log/scalarizr.log
2015-06-27 06:53:55,797+00:00 - INFO - scalarizr.app - [pid: 3873] Starting scalarizr 3.7.13
2015-06-27 06:53:55,798+00:00 - INFO - scalarizr.app - Configuring Scalarizr. This can take a few minutes...
2015-06-27 06:54:15,331+00:00 - INFO - scalarizr.messaging.p2p.consumer - Building message consumer server on 0.0.0.0:8013
2015-06-27 06:54:15,331+00:00 - INFO - scalarizr.app - Starting API server on http://0.0.0.0:8010
2015-06-27 06:54:15,340+00:00 - INFO - scalarizr.handlers.lifecycle - Server will be imported into Scalr
2015-06-27 06:54:16,531+00:00 - INFO - scalarizr.handlers - Detecting supported behaviors...
2015-06-27 06:54:16,547+00:00 - INFO - scalarizr.handlers - Scalr built-in automation: checking for supported software.
If installed software isn't detected, review the Scalr Wiki: https://scalr-wiki.atlassian.net/wiki/x/IoB1
2015-06-27 06:54:16,563+00:00 - INFO - scalarizr.handlers - app: Unavailable. Not installed.
2015-06-27 06:54:17,745+00:00 - INFO - scalarizr.handlers - chef: Available. Installed version: 11.18.0
2015-06-27 06:54:17,777+00:00 - INFO - scalarizr.handlers - haproxy: Unavailable. Not installed.
2015-06-27 06:54:17,813+00:00 - INFO - scalarizr.handlers - mariadb: Unavailable. Not installed.
2015-06-27 06:54:17,814+00:00 - INFO - scalarizr.handlers - memcached: Unavailable. Not installed.
2015-06-27 06:54:17,814+00:00 - INFO - scalarizr.handlers - mysql: Unavailable. Not installed.
2015-06-27 06:54:17,815+00:00 - INFO - scalarizr.handlers - mysql2: Unavailable. Not installed.
2015-06-27 06:54:17,816+00:00 - INFO - scalarizr.handlers - percona: Unavailable. Not installed.
2015-06-27 06:54:17,830+00:00 - INFO - scalarizr.handlers - postgresql: Unavailable. Not installed.
2015-06-27 06:54:17,833+00:00 - INFO - scalarizr.handlers - rabbitmq: Unavailable. Not installed.
2015-06-27 06:54:17,846+00:00 - INFO - scalarizr.handlers - redis: Unavailable. Not installed.
2015-06-27 06:54:17,858+00:00 - INFO - scalarizr.handlers - tomcat: Unavailable. Not installed.
2015-06-27 06:54:17,859+00:00 - INFO - scalarizr.handlers - www: Unavailable. Not installed.
2015-06-27 06:54:35,139+00:00 - INFO - scalarizr.scripts.reboot - Starting reboot script...
2015-06-27 06:54:35,157+00:00 - INFO - scalarizr.app - [pid: 3873] Stopping scalarizr 3.7.13
2015-06-27 06:54:36,381+00:00 - INFO - scalarizr.app - [pid: 3873] Scalarizr terminated
2015-06-27 07:03:03,566+00:00 - INFO - scalarizr.scripts.udev - Starting udev script...
2015-06-27 07:03:03,568+00:00 - INFO - scalarizr.scripts.udev - Starting udev script...
2015-06-27 07:03:08,116+00:00 - INFO - scalarizr.scripts.udev - Starting udev script...


Debug log:


However i think that logfile is stil from the template creation. If i restart scalarizer on the server all i get is this:


seems to be stuck at 

2015-06-27 08:18:50,364+00:00 - DEBUG - scalarizr.app - Fetching UpdateClient status...



To note this is all on an FRESHLY created image which was attached to a role and farm. 

If i restart scalr-update service and then scalarizer it seems to work and the machine gets to show up as running. So something is wrong with image creation?


(i replaced the domain in the logs by scalr.mydomain.com)

Igor Savchenko

unread,
Jun 27, 2015, 4:32:41 AM6/27/15
to scalr-...@googlegroups.com
Can you please provide /var/log/scalarizr_update.log

Thanks,
Igor

--
You received this message because you are subscribed to the Google Groups "scalr-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalr-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Enrico Kern

unread,
Jun 27, 2015, 5:27:27 AM6/27/15
to scalr-...@googlegroups.com
right after first machine is started with freshly created template and in "pending" phase:

 cat /var/log/scalarizr_update.log
2015-06-27 09:19:43,160 - INFO - upd.client - Starting UpdateClient (pid: 1439)
2015-06-27 09:19:43,161 - INFO - upd.client - Starting API on port 8008
2015-06-27 09:19:43,175 - DEBUG - scalarizr.util - system: ('/bin/rpm', '-qa', 'scalarizr*', '--queryformat', '%{NAME}|%{VERSION}\n')
2015-06-27 09:19:44,005 - DEBUG - root - stdout: scalarizr-ec2|3.7.13
scalarizr|3.7.13

2015-06-27 09:19:44,005 - INFO - scalarizr.updclient.api - Getting System ID
2015-06-27 09:19:44,006 - DEBUG - scalarizr.util - system: dmidecode -s system-uuid
2015-06-27 09:19:44,084 - DEBUG - root - stdout: EC29496A-2EDD-2A38-EE06-611ABB9652AA

2015-06-27 09:19:44,085 - DEBUG - scalarizr.updclient.api - Status file /etc/scalr/private.d/update.status not exists
2015-06-27 09:19:44,085 - INFO - scalarizr.updclient.api - Initializing UpdateClient...
2015-06-27 09:19:44,085 - INFO - scalarizr.util.metadata - Getting user-data
2015-06-27 09:19:44,091 - DEBUG - scalarizr.util.metadata.cloudstack - Use DHCP leases file: /var/lib/dhclient/dhclient-eth0.leases
2015-06-27 09:19:44,091 - DEBUG - scalarizr.util.metadata.cloudstack - Use DHCP server: 169.254.1.0
2015-06-27 09:19:44,091 - DEBUG - scalarizr.util.metadata.verizon - system: vmtoolsd --cmd "info-get guestinfo.personality.count"
2015-06-27 09:19:44,093 - DEBUG - scalarizr.util.metadata.openstack-xenbus - Try '/proc/xen/xenbus': not exists
2015-06-27 09:19:44,093 - DEBUG - scalarizr.util.metadata.file - Try '/etc/.scalr-user-data': not exists
2015-06-27 09:19:44,093 - DEBUG - scalarizr.util.metadata.file - Try '/etc/scalr/private.d/.user-data': not exists
2015-06-27 09:19:44,094 - DEBUG - scalarizr.util.metadata - GcePvd.vote raised: <urlopen error [Errno -2] Name or service not known>
2015-06-27 09:19:44,095 - DEBUG - scalarizr.util.metadata.openstack-query - Try 'http://169.254.169.254/openstack/latest/meta_data.json': HTTP Error 404: Not Found
2015-06-27 09:19:44,096 - DEBUG - scalarizr.util.metadata.ec2 - matched instance_id
2015-06-27 09:19:44,097 - DEBUG - scalarizr.util.metadata.ec2 - matched user_data
2015-06-27 09:19:44,100 - DEBUG - scalarizr.util.metadata.verizon - stderr: /bin/sh: vmtoolsd: command not found

2015-06-27 09:19:44,100 - DEBUG - scalarizr.util.metadata - VerizonPvd.vote raised: vmtoolsd (code: 127) <out>:  <err>: /bin/sh: vmtoolsd: command not found <args>: vmtoolsd --cmd "info-get guestinfo.personality.count"
2015-06-27 09:19:54,101 - DEBUG - scalarizr.util.metadata.cloudstack - Try 'http://169.254.1.0/latest/instance-id': <urlopen error timed out>
2015-06-27 09:19:54,101 - DEBUG - scalarizr.util.metadata - provider for 'instance_id': <scalarizr.util.metadata.Ec2Pvd object at 0x7fb0b538ad10>
2015-06-27 09:19:54,101 - DEBUG - scalarizr.util.metadata - provider for 'user_data': <scalarizr.util.metadata.Ec2Pvd object at 0x7fb0b538ad10>
2015-06-27 09:19:54,101 - INFO - scalarizr.updclient.api - Applying configuration from user-data
2015-06-27 09:19:54,117 - INFO - scalarizr.updclient.api - Testing that crypto key works (file: /etc/scalr/private.d/keys/default)
2015-06-27 09:19:54,117 - DEBUG - scalarizr.updclient.api - Initializing database connection
2015-06-27 09:19:54,120 - DEBUG - scalarizr.util - Wait 0.10 seconds before the next attempt
2015-06-27 09:19:54,228 - DEBUG - scalarizr.updclient.api - Initializing QueryEnv
2015-06-27 09:19:54,231 - DEBUG - scalarizr.queryenv - Call QueryEnv: http://scalr.glispa.com/query-env/2012-07-01/get-latest-version
2015-06-27 09:19:54,231 - DEBUG - scalarizr.queryenv - QueryEnv request: {'operation': 'get-latest-version', 'version': u'2012-07-01'}
2015-06-27 09:19:54,280 - DEBUG - requests.packages.urllib3.connectionpool - Starting new HTTP connection (1): scalr.glispa.com
2015-06-27 09:19:54,591 - WARNING - scalarizr.queryenv - QueryEnv failed: 403 Client Error: Forbidden
2015-06-27 09:19:54,591 - WARNING - scalarizr.queryenv - Sleep 30 seconds before next attempt...
2015-06-27 09:20:24,621 - DEBUG - scalarizr.queryenv - QueryEnv request: {'operation': 'get-latest-version', 'version': u'2012-07-01'}
2015-06-27 09:20:24,622 - DEBUG - requests.packages.urllib3.connectionpool - Starting new HTTP connection (1): scalr.glispa.com
2015-06-27 09:20:24,868 - WARNING - scalarizr.queryenv - QueryEnv failed: 403 Client Error: Forbidden
2015-06-27 09:20:24,868 - WARNING - scalarizr.queryenv - Sleep 30 seconds before next attempt...


now if i restart scalr-update service and scalarizer at least 2 times (and then server shows as initializing and then running) the log is that:



so it really seems that something is broken when images are created :(  I can work around this if i create images by login in to pending machines and restart updater and scalarizer and then taking a snapshot of a machine and using this as images. Images from "fixed" snapshots work as usual but i cant explain that to my users i think.

Enrico Kern

unread,
Jun 28, 2015, 6:48:11 AM6/28/15
to scalr-...@googlegroups.com
nobody got an idea? :(

Elvin Abordo

unread,
Jun 29, 2015, 3:49:23 PM6/29/15
to scalr-...@googlegroups.com
I'm actually experiencing the same thing at the moment. Any help would be appreciated. This was working well a week an a half ago. I'm not exactly sure what changed. 

Elvin Abordo

unread,
Jun 29, 2015, 4:30:54 PM6/29/15
to scalr-...@googlegroups.com
Enrico - 

I believe you're correct in saying that during image creation there is an issue. If you follow the scalr documentation on setting up the scalr agent on an instance, it will prompt you to install the debian repositories. By default it uses the latest repository instead of the stable repository. 

there maybe an issue in the 3.7 agent vs. the 3.6 agent. I just created a new image and role using the stable repository try using the stable repo instead of the latest repo when you are setting up a new image. 

In debian you can navigate to /etc/apt/sources.list.d/ and comment out the line that is in the latest .list file and enable the stable repo in the stable .list file. Continue with the scalr docs found here:


That should get you up and running. I'll leave it up to scalr folks to pin down what went wrong in their latest branch. 

Kevin Sample

unread,
Jun 30, 2015, 3:26:41 PM6/30/15
to scalr-...@googlegroups.com
Switching to the scalr-stable (3.6) repo in RHEL 6 fixed this issue for me for deployments to Openstack Icehouse.  No service restarts required.

Thanks Elvin!

ks

Daniele Testa

unread,
Jul 1, 2015, 8:44:20 AM7/1/15
to scalr-...@googlegroups.com
Hi all!

Sorry for the late response.

We have been investigating the reported issue and found the cause.

It was indeed a problem with the 3.7 branch of the Scalr Agent and has been fixed in the latest (3.7.15) version.

We really appreciate that you guys made us aware of this!

Please try to import your images again and let me know if you still have problems. Thanks!

Regards,
Daniele
Reply all
Reply to author
Forward
0 new messages