2016-08-17 19:33:30,698+00:00 - INFO - scalarizr.app - [pid: 9486] Starting scalarizr 4.8.2 2016-08-17 19:33:30,700+00:00 - DEBUG - scalarizr.util - system: ('/etc/init.d/scalr-upd-client', 'status') 2016-08-17 19:33:30,739+00:00 - DEBUG - root - stdout: ● scalr-upd-client.service - SYSV: Scalarizr is a part of Scalr project Loaded: loaded (/etc/rc.d/init.d/scalr-upd-client) Active: active (running) since Wed 2016-08-17 19:32:50 UTC; 40s ago Docs: man:systemd-sysv-generator(8) CGroup: /system.slice/scalr-upd-client.service ├─3436 /opt/scalarizr/embedded/bin/python /opt/scalarizr/bin/scalr-upd-client --daemonize --pid-file /var/run/scalr-upd-client.pid --log-file /var/log/scalarizr_update.log ├─9459 /bin/sh /etc/init.d/scalarizr start └─9465 /bin/systemctl start scalarizr.service Aug 17 19:32:47 scalarizr.novalocal systemd[1]: Starting SYSV: Scalarizr is a part of Scalr project... Aug 17 19:32:50 ncad9etl1 scalr-upd-client[851]: [42B blob data] Aug 17 19:32:50 ncad9etl1 systemd[1]: Started SYSV: Scalarizr is a part of Scalr project. 2016-08-17 19:33:30,739+00:00 - INFO - scalarizr.app - Polling UpdateClient state... 2016-08-17 19:33:30,741+00:00 - DEBUG - scalarizr.app - UpdateClient state: completed/wait-ack 2016-08-17 19:33:30,742+00:00 - DEBUG - scalarizr.app - UpdateClient state: completed/wait-ack 2016-08-17 19:33:30,742+00:00 - INFO - scalarizr.app - UpdateClient state: completed/wait-ack 2016-08-17 19:33:30,743+00:00 - INFO - scalarizr.app - UpdateClient completed update and should be restarted, restarting 2016-08-17 19:33:30,743+00:00 - INFO - scalarizr.util.initdv2 - Stopping scalr-upd-client 2016-08-17 19:33:30,744+00:00 - DEBUG - scalarizr.util - system: ('/etc/init.d/scalr-upd-client', 'stop') 2016-08-17 19:33:31,814+00:00 - DEBUG - root - stdout: Stopping scalr-upd-client (via systemctl): [ OK ] 2016-08-17 19:33:31,815+00:00 - INFO - scalarizr.util.initdv2 - Starting scalr-upd-client 2016-08-17 19:33:31,816+00:00 - DEBUG - scalarizr.util - system: ('/etc/init.d/scalr-upd-client', 'start') 2016-08-17 19:33:32,310+00:00 - DEBUG - root - stdout: Starting scalr-upd-client (via systemctl): [ OK ] 2016-08-17 19:33:32,311+00:00 - DEBUG - agent.scalrinit.datadir - msg 2016-08-17 19:33:32,311+00:00 - DEBUG - common.utils.mixutil - Trying until at 0x7f751afdd8c0> (timeout: 60 interval: 1 times: unlimited) 2016-08-17 19:33:32,311+00:00 - DEBUG - common.utils.mixutil - /var/run/scalarizr/result.json not exists (1) ... 2016-08-17 19:33:32,312+00:00 - DEBUG - scalarizr.node - Reloading /etc/scalr/private.d/config.ini 2016-08-17 19:33:32,313+00:00 - INFO - scalarizr.app - This image was bundled with cloud API call. Cleauping ancestor server data 2016-08-17 19:33:32,314+00:00 - DEBUG - scalarizr.util - system: sync 2016-08-17 19:33:33,664+00:00 - INFO - scalarizr.node - State: bootstrapping 2016-08-17 19:33:33,665+00:00 - DEBUG - scalarizr.app - Applying user-data to configuration 2016-08-17 19:33:33,666+00:00 - DEBUG - scalarizr.app - User-data (Instance): {u'behaviors': u'base', u'cloud_storage_path': u'swift://', u'env_id': u'4', u'farm_roleid': u'94', u'farmid': u'22', u'hash': u'bedb486e9941a3', u'httpproto': u'https', u'message_format': u'json', u'p2p_producer_endpoint': u'https://scalr1.domain.com/messaging', u'platform': u'openstack', u'queryenv_url': u'https://scalr1.domain.com/query-env', u'realrolename': u'CentOS7-NCA-etl', u'region': u'r1', u'role': u'base', u'roleid': u'18', u'server_index': u'1', u'serverid': u'cad34d9b-15d7-4aa3-86e5-1f192149b788', u'szr_key': u'mPhQo9r1Gkve+V5MeFDPc6n6rkxf83qBG7FdkXYz'} 2016-08-17 19:33:33,667+00:00 - DEBUG - scalarizr.config - Updating configuration file /etc/scalr/private.d/config.ini 2016-08-17 19:33:33,668+00:00 - DEBUG - scalarizr.app - Reloading configuration after user-data applying 2016-08-17 19:33:33,668+00:00 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=True) 2016-08-17 19:33:33,668+00:00 - DEBUG - scalarizr.config - Loading main configuration 2016-08-17 19:33:33,669+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/config.ini 2016-08-17 19:33:33,670+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/private.d/config.ini 2016-08-17 19:33:33,670+00:00 - DEBUG - scalarizr.config - Loading platform configuration 2016-08-17 19:33:33,671+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/openstack.ini 2016-08-17 19:33:33,671+00:00 - DEBUG - scalarizr.config - Loading behaviours configuration 2016-08-17 19:33:33,671+00:00 - DEBUG - scalarizr.config - Loading handlers configuration 2016-08-17 19:33:33,672+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/ip_list_builder.ini 2016-08-17 19:33:33,672+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/script_executor.ini 2016-08-17 19:33:33,673+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/hooks.ini 2016-08-17 19:33:33,673+00:00 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=True) 2016-08-17 19:33:33,673+00:00 - DEBUG - scalarizr.config - Loading main configuration 2016-08-17 19:33:33,674+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/config.ini 2016-08-17 19:33:33,674+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/private.d/config.ini 2016-08-17 19:33:33,675+00:00 - DEBUG - scalarizr.config - Loading platform configuration 2016-08-17 19:33:33,675+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/openstack.ini 2016-08-17 19:33:33,676+00:00 - DEBUG - scalarizr.config - Loading behaviours configuration 2016-08-17 19:33:33,676+00:00 - DEBUG - scalarizr.config - Loading handlers configuration 2016-08-17 19:33:33,676+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/ip_list_builder.ini 2016-08-17 19:33:33,677+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/script_executor.ini 2016-08-17 19:33:33,677+00:00 - DEBUG - scalarizr.config - Reading configuration file /etc/scalr/public.d/hooks.ini 2016-08-17 19:33:33,678+00:00 - DEBUG - scalarizr.app - Initialize platform 2016-08-17 19:33:34,316+00:00 - DEBUG - scalarizr.app - Database doesn't exist, creating new one from script 2016-08-17 19:33:34,317+00:00 - DEBUG - scalarizr.app - Open SQLite database (file: /etc/scalr/private.d/db.sqlite) 2016-08-17 19:33:34,349+00:00 - DEBUG - scalarizr.util - system: sync 2016-08-17 19:33:34,418+00:00 - DEBUG - scalarizr.app - Open SQLite database (file: /etc/scalr/private.d/db.sqlite) 2016-08-17 19:33:34,419+00:00 - DEBUG - scalarizr.util - Wait 0.10 seconds before the next attempt 2016-08-17 19:33:34,591+00:00 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=False) 2016-08-17 19:33:34,591+00:00 - DEBUG - scalarizr.config - Bootstrap INI configuration (reload=False) 2016-08-17 19:33:34,595+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.lifecycle 2016-08-17 19:33:35,226+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.lifecycle 2016-08-17 19:33:35,226+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.ip_list_builder 2016-08-17 19:33:35,229+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.ip_list_builder 2016-08-17 19:33:35,229+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.script_executor 2016-08-17 19:33:35,229+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.script_executor 2016-08-17 19:33:35,230+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.hooks 2016-08-17 19:33:35,231+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.hooks 2016-08-17 19:33:35,232+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.ssh_auth_keys 2016-08-17 19:33:35,234+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.ssh_auth_keys 2016-08-17 19:33:35,234+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.deploy 2016-08-17 19:33:35,235+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.deploy 2016-08-17 19:33:35,235+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.openstack.vol 2016-08-17 19:33:35,239+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.openstack.vol 2016-08-17 19:33:35,239+00:00 - DEBUG - scalarizr.config - Importing module scalarizr.handlers.openstack.rebundle 2016-08-17 19:33:35,247+00:00 - DEBUG - scalarizr.config - Lookup config container in module scalarizr.handlers.openstack.rebundle 2016-08-17 19:33:35,248+00:00 - DEBUG - scalarizr.node - Reloading /etc/scalr/private.d/config.ini 2016-08-17 19:33:35,249+00:00 - DEBUG - scalarizr.app - Got scalr url: 'https://scalr1.domain.com' 2016-08-17 19:33:35,439+00:00 - DEBUG - scalarizr.app - Initialize QueryEnv client 2016-08-17 19:33:35,440+00:00 - DEBUG - scalarizr.node - Reloading /etc/scalr/public.d/config.ini 2016-08-17 19:33:35,444+00:00 - DEBUG - scalarizr.queryenv - Call QueryEnv: https://scalr1.domain.com/query-env/2012-04-17/get-latest-version 2016-08-17 19:33:35,445+00:00 - DEBUG - scalarizr.queryenv - QueryEnv request: {'operation': 'get-latest-version', 'version': '2012-04-17'} 2016-08-17 19:33:35,730+00:00 - DEBUG - scalarizr.queryenv - QueryEnv response (parsed): 2015-04-10 2016-08-17 19:33:35,799+00:00 - DEBUG - scalarizr.queryenv - Call QueryEnv: https://scalr1.domain.com/query-env/2015-04-10/get-global-config 2016-08-17 19:33:35,799+00:00 - DEBUG - scalarizr.queryenv - QueryEnv request: {'operation': 'get-global-config', 'version': '2015-04-10'} 2016-08-17 19:33:36,030+00:00 - DEBUG - scalarizr.queryenv - QueryEnv response (parsed): {'scalr.id': '1fe38615', 'scalr.version': '5.11.22', 'dns.static.endpoint': 'example-dns.net'} 2016-08-17 19:33:36,032+00:00 - DEBUG - scalarizr.queryenv - Call QueryEnv: https://scalr1.domain.com/query-env/2015-04-10/list-farm-role-params-json 2016-08-17 19:33:36,032+00:00 - DEBUG - scalarizr.queryenv - QueryEnv request: {'operation': 'list-farm-role-params-json', 'version': '2015-04-10', 'farm-role-id': '94'} 2016-08-17 19:33:36,543+00:00 - DEBUG - scalarizr.queryenv - QueryEnv response (parsed): {u'base': {u'reboot_after_hostinit_phase': 0, u'disable_firewall_management': 0, u'abort_init_on_script_fail': 1, u'hostname': u'ncad9etl1.domain.com', u'update': {u'deb_repo_url': u'http://repo.scalr.net/apt-plain stable/', u'repository': u'stable', u'server_url': u'http://update.scalr.net/', u'api_port': u'', u'rpm_repo_url': u'http://repo.scalr.net/rpm/stable/rhel/$releasever/$basearch', u'client_mode': u'client', u'win_repo_url': u'http://repo.scalr.net/win/stable'}, u'api_port': u'8010', u'messaging_port': u'8013', u'scalr_labs_fatmouse_update': 0, u'resume_strategy': u'reboot', u'union_script_executor': 1, u'keep_scripting_logs_time': u'36000'}, u'volumes': [{u'scalr_storage_id': u'1c4e68a4-4476-41b0-b207-2b4faba4a8d5', u'mpoint': u'', u'type': u'cinder', u'fstype': u'', u'size': u'20'}]} 2016-08-17 19:33:36,544+00:00 - DEBUG - scalarizr.app - Initialize messaging 2016-08-17 19:33:36,546+00:00 - DEBUG - scalarizr.app - Initialize message handlers 2016-08-17 19:33:36,546+00:00 - DEBUG - scalarizr.handlers - Initializing message listener 2016-08-17 19:33:36,547+00:00 - DEBUG - scalarizr.handlers - Keywords for each Handler::accept {'behaviour': [], 'dist': ('centos', '7.2.1511', 'Core'), 'os': ('Linux', 'ncad9etl1', '3.10.0-327.28.2.el7.x86_64', '#1 SMP Wed Aug 3 11:11:39 UTC 2016', 'x86_64', 'x86_64'), 'platform': 'openstack'} 2016-08-17 19:33:36,548+00:00 - DEBUG - scalarizr.handlers - Collecting message handlers... 2016-08-17 19:33:36,549+00:00 - DEBUG - scalarizr.messaging.p2p.store - Add rotate messages table task for periodical executor 2016-08-17 19:33:36,697+00:00 - DEBUG - scalarizr.handlers - Message handlers chain: [, , , , , ] 2016-08-17 19:33:36,745+00:00 - DEBUG - scalarizr.handlers.block_device - Ensure udev rule for EBS devices 2016-08-17 19:33:36,746+00:00 - DEBUG - scalarizr.util - system: ('mount', '-a') 2016-08-17 19:33:36,758+00:00 - DEBUG - scalarizr.handlers.lifecycle - Adding iptables rules for scalarizr ports 2016-08-17 19:33:36,759+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--list-rules', 'INPUT') 2016-08-17 19:33:36,767+00:00 - DEBUG - root - stdout: -P INPUT ACCEPT 2016-08-17 19:33:36,768+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--list-rules', 'INPUT') 2016-08-17 19:33:36,774+00:00 - DEBUG - root - stdout: -P INPUT ACCEPT 2016-08-17 19:33:36,775+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--list-rules', 'INPUT') 2016-08-17 19:33:36,781+00:00 - DEBUG - root - stdout: -P INPUT ACCEPT 2016-08-17 19:33:36,782+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--list-rules', 'INPUT') 2016-08-17 19:33:36,787+00:00 - DEBUG - root - stdout: -P INPUT ACCEPT 2016-08-17 19:33:36,788+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--list-rules', 'INPUT') 2016-08-17 19:33:36,793+00:00 - DEBUG - root - stdout: -P INPUT ACCEPT 2016-08-17 19:33:36,794+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--list-rules', 'INPUT', '--table', 'filter') 2016-08-17 19:33:36,799+00:00 - DEBUG - root - stdout: -P INPUT ACCEPT 2016-08-17 19:33:36,800+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--protocol', 'udp', '--match', 'udp', '--jump', 'ACCEPT', '--dport', '8014', '--insert', 'INPUT') 2016-08-17 19:33:36,819+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--protocol', 'tcp', '--match', 'tcp', '--jump', 'ACCEPT', '--dport', '8013', '--insert', 'INPUT') 2016-08-17 19:33:36,828+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--protocol', 'tcp', '--match', 'tcp', '--jump', 'ACCEPT', '--dport', '8012', '--insert', 'INPUT') 2016-08-17 19:33:36,834+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--protocol', 'tcp', '--match', 'tcp', '--jump', 'ACCEPT', '--dport', '8010', '--insert', 'INPUT') 2016-08-17 19:33:36,840+00:00 - DEBUG - scalarizr.util - system: ('/sbin/iptables', '--protocol', 'tcp', '--match', 'tcp', '--jump', 'ACCEPT', '--dport', '8008', '--insert', 'INPUT') 2016-08-17 19:33:36,847+00:00 - DEBUG - scalarizr.handlers.hooks - Hook on 'init'() {} 2016-08-17 19:33:36,848+00:00 - INFO - scalarizr.messaging.p2p.consumer - Building message consumer server on 0.0.0.0:8013 2016-08-17 19:33:36,848+00:00 - INFO - scalarizr.app - Starting API server on http://0.0.0.0:8010 2016-08-17 19:33:36,853+00:00 - DEBUG - scalarizr.util.PeriodicalExecutor - Executing task Rotate messages sqlite table 2016-08-17 19:33:36,854+00:00 - DEBUG - scalarizr.queryenv - Call QueryEnv: https://scalr1.domain.com/query-env/2015-04-10/list-farm-role-params-json 2016-08-17 19:33:36,854+00:00 - DEBUG - scalarizr.queryenv - QueryEnv request: {'operation': 'list-farm-role-params-json', 'version': '2015-04-10', 'farm-role-id': '94'} 2016-08-17 19:33:36,860+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Starting message consumer http://0.0.0.0:8013 2016-08-17 19:33:36,861+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Starting message handler 2016-08-17 19:33:37,113+00:00 - DEBUG - scalarizr.queryenv - QueryEnv response (parsed): {u'base': {u'reboot_after_hostinit_phase': 0, u'disable_firewall_management': 0, u'abort_init_on_script_fail': 1, u'hostname': u'ncad9etl1.domain.com', u'update': {u'deb_repo_url': u'http://repo.scalr.net/apt-plain stable/', u'repository': u'stable', u'server_url': u'http://update.scalr.net/', u'api_port': u'', u'rpm_repo_url': u'http://repo.scalr.net/rpm/stable/rhel/$releasever/$basearch', u'client_mode': u'client', u'win_repo_url': u'http://repo.scalr.net/win/stable'}, u'api_port': u'8010', u'messaging_port': u'8013', u'scalr_labs_fatmouse_update': 0, u'resume_strategy': u'reboot', u'union_script_executor': 1, u'keep_scripting_logs_time': u'36000'}, u'volumes': [{u'scalr_storage_id': u'1c4e68a4-4476-41b0-b207-2b4faba4a8d5', u'mpoint': u'', u'type': u'cinder', u'fstype': u'', u'size': u'20'}]} 2016-08-17 19:33:37,122+00:00 - INFO - scalarizr.handlers.lifecycle - Starting initialization 2016-08-17 19:33:37,123+00:00 - DEBUG - scalarizr.handlers.hooks - Hook on 'before_host_init'(,) {} 2016-08-17 19:33:37,123+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Message consumer locked 2016-08-17 19:33:37,124+00:00 - DEBUG - scalarizr.messaging.p2p.producer - Sending message 'HostInit' into queue 'control' 2016-08-17 19:33:37,158+00:00 - DEBUG - scalarizr.node - Reloading /etc/scalr/private.d/config.ini 2016-08-17 19:33:37,160+00:00 - DEBUG - scalarizr.messaging.p2p.producer - Delivering message 'HostInit' {'local_ip': '192.168.155.57', 'seconds_since_boot': 55.81, 'seconds_since_start': 6.42, 'behaviour': [], 'crypto_key': '8i4pWd/nL+7ayq9deuOasfzqyr51Fk+mkx7GHQniQUwmQGHz8iqD0Q==\n', 'role_name': 'CentOS7-NCA-etl', 'remote_ip': '192.168.155.57'}. Json: True, Headers: {'Content-Type': 'application/json'} 2016-08-17 19:33:37,160+00:00 - DEBUG - scalarizr.messaging.p2p.security - Encrypting message 2016-08-17 19:33:37,829+00:00 - DEBUG - scalarizr.messaging.p2p.producer - Message 'HostInit' delivered (message_id: 109c25cd-0fae-4777-a2f4-6621cd72f65e) 2016-08-17 19:33:37,860+00:00 - DEBUG - scalarizr.util.PeriodicalExecutor - Executing task Rotate scripting logs 2016-08-17 19:33:37,861+00:00 - DEBUG - scalarizr.util.PeriodicalExecutor - Executing task Rotate tasks data 2016-08-17 19:33:37,863+00:00 - DEBUG - scalarizr.handlers.lifecycle - Writing new crypto key to disk 2016-08-17 19:33:37,864+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Message consumer unlocked 2016-08-17 19:33:37,864+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Waiting message acknowledge event: HostInit 2016-08-17 19:34:21,736+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Request timed out: timeout('timed out',) 2016-08-17 19:34:31,747+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Request timed out: timeout('timed out',) 2016-08-17 19:36:16,562+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Received ingoing message in queue: 'control' 2016-08-17 19:36:16,564+00:00 - DEBUG - scalarizr.messaging.p2p.security - Decrypting message 2016-08-17 19:36:16,580+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Decoding message: { "body": { "server_id": "cad34d9b-15d7-4aa3-86e5-1f192149b788", "server_index": "1", "farm_role_id": "", "local_ip": "", "event_id": "9558a34b-918a-49ea-922f-cea62039965c", "handlers": [ "base" ], "cloud_location": "r1", "farm_crypto_key": "LuMlzOHPscVgdi62aB3mKatx52LHcmVwu3AHx2rr", "behaviour": "", "role_name": "", "base": { "reboot_after_hostinit_phase": 0, "disable_firewall_management": 0, "abort_init_on_script_fail": 1, "hostname": "ncad9etl1.domain.com", "update": { "deb_repo_url": "http://repo.scalr.net/apt-plain stable/", "repository": "stable", "server_url": "http://update.scalr.net/", "api_port": "", "rpm_repo_url": "http://repo.scalr.net/rpm/stable/rhel/$releasever/$basearch", "client_mode": "client", "win_repo_url": "http://repo.scalr.net/win/stable" }, "api_port": "8010", "messaging_port": "8013", "scalr_labs_fatmouse_update": 0, "resume_strategy": "reboot", "union_script_executor": 1, "keep_scripting_logs_time": "36000" }, "volumes": [ { "scalr_storage_id": "1c4e68a4-4476-41b0-b207-2b4faba4a8d5", "mpoint": "", "type": "cinder", "fstype": "", "size": "20" } ], "global_variables": [ { "value": "d0eebd63-3651-41c9-8aea-ef99c4811cfe", "name": "SCALR_IMAGE_ID", "system": 1 }, { "value": "10.2.209.158", "name": "SCALR_EXTERNAL_IP", "system": 1 }, { "value": "192.168.155.57", "name": "SCALR_INTERNAL_IP", "system": 1 }, { "value": "CentOS7-NCA-etl", "name": "SCALR_ROLE_NAME", "system": 1 }, { "value": "", "name": "SCALR_ISDBMASTER", "system": 1 }, { "value": "1", "name": "SCALR_INSTANCE_INDEX", "system": 1 }, { "value": "2", "name": "SCALR_INSTANCE_FARM_INDEX", "system": 1 }, { "value": "4bd3cb5f-41c4-45e2-8150-87997c08cece", "name": "SCALR_SERVER_TYPE", "system": 1 }, { "value": "ncad9etl1.domain.com", "name": "SCALR_SERVER_HOSTNAME", "system": 1 }, { "value": "cad34d9b-15d7-4aa3-86e5-1f192149b788", "name": "SCALR_SERVER_ID", "system": 1 }, { "value": "22", "name": "SCALR_FARM_ID", "system": 1 }, { "value": "94", "name": "SCALR_FARM_ROLE_ID", "system": 1 }, { "value": "CentOS7-NCA-etl", "name": "SCALR_FARM_ROLE_ALIAS", "system": 1 }, { "value": "ncad9_openstack", "name": "SCALR_FARM_NAME", "system": 1 }, { "value": "bedb486e9941a3", "name": "SCALR_FARM_HASH", "system": 1 }, { "value": "", "name": "SCALR_FARM_OWNER_EMAIL", "system": 1 }, { "value": "devops", "name": "SCALR_FARM_TEAM", "system": 1 }, { "value": "base", "name": "SCALR_BEHAVIORS", "system": 1 }, { "value": "4", "name": "SCALR_ENV_ID", "system": 1 }, { "value": "nca", "name": "SCALR_ENV_NAME", "system": 1 }, { "value": "r1", "name": "SCALR_CLOUD_LOCATION", "system": 1 }, { "value": "501258dc-5747-4eb0-869c-2cb69087a0e1", "name": "SCALR_CLOUD_SERVER_ID", "system": 1 }, { "value": "nova", "name": "SCALR_CLOUD_LOCATION_ZONE", "system": 1 }, { "value": "3f54770e-bf1a-11e3-92c5-000feae9c516", "name": "SCALR_COST_CENTER_ID", "system": 1 }, { "value": "CC-00", "name": "SCALR_COST_CENTER_BC", "system": 1 }, { "value": "Cost center 01", "name": "SCALR_COST_CENTER_NAME", "system": 1 }, { "value": "30c59dba-fc9b-4d0f-83ec-4b5043b12f72", "name": "SCALR_PROJECT_ID", "system": 1 }, { "value": "PR-00-00", "name": "SCALR_PROJECT_BC", "system": 1 }, { "value": "Project 01", "name": "SCALR_PROJECT_NAME", "system": 1 }, { "value": "etl", "name": "NODETYPE" }, { "value": "ncad9", "name": "BASENAME" }, { "value": "4.5.3", "name": "VERSION" } ], "message_id": "604c4c39-51e1-4f64-9d50-5e7cc2aa1429", "remote_ip": "" }, "meta": { "server_id": "cad34d9b-15d7-4aa3-86e5-1f192149b788", "scalr_version": "5.11.22" }, "name": "HostInitResponse", "id": "604c4c39-51e1-4f64-9d50-5e7cc2aa1429" } 2016-08-17 19:36:16,580+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Received message 'HostInitResponse' (message_id: 604c4c39-51e1-4f64-9d50-5e7cc2aa1429, format: json) 2016-08-17 19:36:16,614+00:00 - DEBUG - scalarizr.messaging.p2p.store - Commiting put_ingoing 2016-08-17 19:36:16,615+00:00 - DEBUG - scalarizr.messaging.p2p.store - Commited put_ingoing 2016-08-17 19:36:16,615+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - "POST /control HTTP/1.1" 201 - 2016-08-17 19:36:27,569+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Received ingoing message in queue: 'control' 2016-08-17 19:36:27,570+00:00 - DEBUG - scalarizr.messaging.p2p.security - Decrypting message 2016-08-17 19:36:27,581+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Decoding message: { "body": { "server_id": "cad34d9b-15d7-4aa3-86e5-1f192149b788", "server_index": "1", "farm_role_id": "94", "handlers": [ "base" ], "event_id": "9558a34b-918a-49ea-922f-cea62039965c", "farm_role_alias": "CentOS7-NCA-etl", "local_ip": "192.168.155.57", "cloud_location": "r1", "ssh_pub_key": "", "behaviour": [ "base" ], "crypto_key": "", "debug_scripting_count": 0, "role_name": "CentOS7-NCA-etl", "debug_scripting_error": "", "base": { "reboot_after_hostinit_phase": 0, "disable_firewall_management": 0, "abort_init_on_script_fail": 1, "hostname": "ncad9etl1.domain.com", "update": { "deb_repo_url": "http://repo.scalr.net/apt-plain stable/", "repository": "stable", "server_url": "http://update.scalr.net/", "api_port": "", "rpm_repo_url": "http://repo.scalr.net/rpm/stable/rhel/$releasever/$basearch", "client_mode": "client", "win_repo_url": "http://repo.scalr.net/win/stable" }, "api_port": "8010", "messaging_port": "8013", "scalr_labs_fatmouse_update": 0, "resume_strategy": "reboot", "union_script_executor": 1, "keep_scripting_logs_time": "36000" }, "scripts": [], "global_variables": [ { "value": "d0eebd63-3651-41c9-8aea-ef99c4811cfe", "name": "SCALR_IMAGE_ID", "system": 1 }, { "value": "10.2.209.158", "name": "SCALR_EXTERNAL_IP", "system": 1 }, { "value": "192.168.155.57", "name": "SCALR_INTERNAL_IP", "system": 1 }, { "value": "CentOS7-NCA-etl", "name": "SCALR_ROLE_NAME", "system": 1 }, { "value": "", "name": "SCALR_ISDBMASTER", "system": 1 }, { "value": "1", "name": "SCALR_INSTANCE_INDEX", "system": 1 }, { "value": "2", "name": "SCALR_INSTANCE_FARM_INDEX", "system": 1 }, { "value": "4bd3cb5f-41c4-45e2-8150-87997c08cece", "name": "SCALR_SERVER_TYPE", "system": 1 }, { "value": "ncad9etl1.domain.com", "name": "SCALR_SERVER_HOSTNAME", "system": 1 }, { "value": "cad34d9b-15d7-4aa3-86e5-1f192149b788", "name": "SCALR_SERVER_ID", "system": 1 }, { "value": "22", "name": "SCALR_FARM_ID", "system": 1 }, { "value": "94", "name": "SCALR_FARM_ROLE_ID", "system": 1 }, { "value": "CentOS7-NCA-etl", "name": "SCALR_FARM_ROLE_ALIAS", "system": 1 }, { "value": "ncad9_openstack", "name": "SCALR_FARM_NAME", "system": 1 }, { "value": "bedb486e9941a3", "name": "SCALR_FARM_HASH", "system": 1 }, { "value": "", "name": "SCALR_FARM_OWNER_EMAIL", "system": 1 }, { "value": "devops", "name": "SCALR_FARM_TEAM", "system": 1 }, { "value": "base", "name": "SCALR_BEHAVIORS", "system": 1 }, { "value": "4", "name": "SCALR_ENV_ID", "system": 1 }, { "value": "nca", "name": "SCALR_ENV_NAME", "system": 1 }, { "value": "r1", "name": "SCALR_CLOUD_LOCATION", "system": 1 }, { "value": "501258dc-5747-4eb0-869c-2cb69087a0e1", "name": "SCALR_CLOUD_SERVER_ID", "system": 1 }, { "value": "nova", "name": "SCALR_CLOUD_LOCATION_ZONE", "system": 1 }, { "value": "3f54770e-bf1a-11e3-92c5-000feae9c516", "name": "SCALR_COST_CENTER_ID", "system": 1 }, { "value": "CC-00", "name": "SCALR_COST_CENTER_BC", "system": 1 }, { "value": "Cost center 01", "name": "SCALR_COST_CENTER_NAME", "system": 1 }, { "value": "30c59dba-fc9b-4d0f-83ec-4b5043b12f72", "name": "SCALR_PROJECT_ID", "system": 1 }, { "value": "PR-00-00", "name": "SCALR_PROJECT_BC", "system": 1 }, { "value": "Project 01", "name": "SCALR_PROJECT_NAME", "system": 1 }, { "value": "etl", "name": "NODETYPE" }, { "value": "ncad9", "name": "BASENAME" }, { "value": "4.5.3", "name": "VERSION" } ], "message_id": "7534ce52-4807-4efa-bf4e-a94043c82be3", "remote_ip": "10.2.209.158" }, "meta": { "server_id": "cad34d9b-15d7-4aa3-86e5-1f192149b788", "scalr_version": "5.11.22" }, "name": "HostInit", "id": "7534ce52-4807-4efa-bf4e-a94043c82be3" } 2016-08-17 19:36:27,582+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Received message 'HostInit' (message_id: 7534ce52-4807-4efa-bf4e-a94043c82be3, format: json) 2016-08-17 19:36:27,616+00:00 - DEBUG - scalarizr.messaging.p2p.store - Commiting put_ingoing 2016-08-17 19:36:27,617+00:00 - DEBUG - scalarizr.messaging.p2p.store - Commited put_ingoing 2016-08-17 19:36:27,617+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - "POST /control HTTP/1.1" 201 - 2016-08-17 19:36:27,660+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Going to handle_one_message. Thread: MessageHandler 2016-08-17 19:36:27,661+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Notify message listeners (message_id: 7534ce52-4807-4efa-bf4e-a94043c82be3) 2016-08-17 19:36:27,662+00:00 - DEBUG - scalarizr.handlers - Handle 'HostInit' 2016-08-17 19:36:27,664+00:00 - DEBUG - scalarizr.handlers - Scalr version: (5, 11, 22) 2016-08-17 19:36:27,665+00:00 - DEBUG - scalarizr.handlers - Call handler BaseBehaviorHandler 2016-08-17 19:36:27,666+00:00 - DEBUG - scalarizr.handlers - Call handler LifeCycleHandler 2016-08-17 19:36:27,666+00:00 - DEBUG - scalarizr.handlers.lifecycle - HI.body.base: {u'reboot_after_hostinit_phase': 0, u'disable_firewall_management': 0, u'abort_init_on_script_fail': 1, u'hostname': u'ncad9etl1.domain.com', u'update': {u'deb_repo_url': u'http://repo.scalr.net/apt-plain stable/', u'repository': u'stable', u'server_url': u'http://update.scalr.net/', u'api_port': u'', u'rpm_repo_url': u'http://repo.scalr.net/rpm/stable/rhel/$releasever/$basearch', u'client_mode': u'client', u'win_repo_url': u'http://repo.scalr.net/win/stable'}, u'api_port': u'8010', u'messaging_port': u'8013', u'scalr_labs_fatmouse_update': 0, u'resume_strategy': u'reboot', u'union_script_executor': 1, u'keep_scripting_logs_time': u'36000'} 2016-08-17 19:36:27,667+00:00 - DEBUG - agent.utils.celeryutil - task_pre_run task_id=0abc11c8bd3a45bc9bce41c622701ddd 2016-08-17 19:36:27,668+00:00 - DEBUG - agent.utils.celeryutil - func_pre_run func_name=sys.set_hostname 2016-08-17 19:36:27,670+00:00 - DEBUG - common.utils.subprocess2 - Executing subprocess: ('hostname ncad9etl1.domain.com',) 2016-08-17 19:36:27,780+00:00 - DEBUG - agent.tasks.sys - Reloading syslog service 2016-08-17 19:36:27,781+00:00 - DEBUG - common.utils.subprocess2 - Executing subprocess: ('pkill -HUP syslogd',) 2016-08-17 19:36:27,810+00:00 - DEBUG - agent.utils.celeryutil - func_post_run stack=['001-sys.set_hostname'] 2016-08-17 19:36:27,810+00:00 - DEBUG - agent.utils.celeryutil - task_post_run task_id=0abc11c8bd3a45bc9bce41c622701ddd 2016-08-17 19:36:27,810+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Mark message (message_id: 7534ce52-4807-4efa-bf4e-a94043c82be3) as handled 2016-08-17 19:36:27,847+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Completed handle_one_message. Thread: MessageHandler 2016-08-17 19:36:27,848+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Found a message and continue message handler 2016-08-17 19:36:27,848+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Fired message acknowledge event: HostInit 2016-08-17 19:36:27,850+00:00 - INFO - scalarizr.config - State: initializing 2016-08-17 19:36:27,850+00:00 - DEBUG - scalarizr.handlers.hooks - Hook on 'host_init'() {} 2016-08-17 19:36:27,850+00:00 - DEBUG - scalarizr.handlers.hooks - Hook on 'start'() {} 2016-08-17 19:36:27,950+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Notify message listeners (message_id: 604c4c39-51e1-4f64-9d50-5e7cc2aa1429) 2016-08-17 19:36:27,950+00:00 - DEBUG - scalarizr.handlers - Handle 'HostInitResponse' 2016-08-17 19:36:27,952+00:00 - DEBUG - scalarizr.handlers - Scalr version: (5, 11, 22) 2016-08-17 19:36:27,953+00:00 - DEBUG - scalarizr.handlers - Call handler BaseBehaviorHandler 2016-08-17 19:36:27,953+00:00 - DEBUG - scalarizr.handlers - Call handler LifeCycleHandler 2016-08-17 19:36:27,954+00:00 - DEBUG - scalarizr.handlers.lifecycle - bus.init_op: 2016-08-17 19:36:27,955+00:00 - DEBUG - scalarizr.util.flag - Touch file '/etc/scalr/private.d/.hir' 2016-08-17 19:36:27,956+00:00 - DEBUG - scalarizr.handlers.lifecycle - HIR.body.base: {u'reboot_after_hostinit_phase': 0, u'disable_firewall_management': 0, u'abort_init_on_script_fail': 1, u'hostname': u'ncad9etl1.domain.com', u'update': {u'deb_repo_url': u'http://repo.scalr.net/apt-plain stable/', u'repository': u'stable', u'server_url': u'http://update.scalr.net/', u'api_port': u'', u'rpm_repo_url': u'http://repo.scalr.net/rpm/stable/rhel/$releasever/$basearch', u'client_mode': u'client', u'win_repo_url': u'http://repo.scalr.net/win/stable'}, u'api_port': u'8010', u'messaging_port': u'8013', u'scalr_labs_fatmouse_update': 0, u'resume_strategy': u'reboot', u'union_script_executor': 1, u'keep_scripting_logs_time': u'36000'} 2016-08-17 19:36:27,956+00:00 - INFO - scalarizr.ops.system.init - Configuring storage volumes 2016-08-17 19:36:27,957+00:00 - DEBUG - scalarizr.queryenv - Call QueryEnv: https://scalr1.domain.com/query-env/2015-04-10/list-ebs-mountpoints 2016-08-17 19:36:27,957+00:00 - DEBUG - scalarizr.queryenv - QueryEnv request: {'operation': 'list-ebs-mountpoints', 'version': '2015-04-10'} 2016-08-17 19:36:28,206+00:00 - DEBUG - scalarizr.queryenv - QueryEnv response (parsed): [] 2016-08-17 19:36:28,223+00:00 - INFO - scalarizr.ops.system.init - Ensure cinder: create volume 2016-08-17 19:36:28,224+00:00 - DEBUG - scalarizr.util - Caught: 'thread._local' object has no attribute 'current' 2016-08-17 19:36:28,225+00:00 - DEBUG - scalarizr.util - Creating new object... 2016-08-17 19:36:28,226+00:00 - DEBUG - scalarizr.util - Created 2016-08-17 19:36:28,226+00:00 - DEBUG - scalarizr.util - Added weakref 2016-08-17 19:36:28,227+00:00 - DEBUG - scalarizr.storage2 - Creating Cinder volume (zone: None size: 20 snapshot: None volume_type: None) 2016-08-17 19:36:28,228+00:00 - DEBUG - keystoneclient.auth.identity.v3.base - Making authentication request to https://cloud.domain.com:5000/v3/auth/tokens 2016-08-17 19:38:42,356+00:00 - DEBUG - keystoneclient.auth.identity.v3.base - Making authentication request to https://cloud.domain.com:5000/v3/auth/tokens 2016-08-17 19:40:43,573+00:00 - DEBUG - keystoneclient.auth.identity.v3.base - Making authentication request to https://cloud.domain.com:5000/v3/auth/tokens 2016-08-17 19:42:43,651+00:00 - ERROR - scalarizr.ops.system.init - Operation "system.init" (id: d224be1e-cc91-4ca3-a268-0b51995115ca) failed. Reason: Unable to establish connection to https://cloud.domain.com:5000/v3/auth/tokens Traceback (most recent call last): File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/api/operation.py", line 187, in _in_progress self._completed(self.func(self, *self.func_args, **self.func_kwds)) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/handlers/lifecycle.py", line 446, in handler bus.fire("host_init_response", message) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/libs/bases.py", line 33, in fire ln(*args, **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/handlers/block_device.py", line 103, in on_host_init_response self._plug_new_style_volumes(volumes) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/handlers/block_device.py", line 127, in _plug_new_style_volumes vol.ensure(mount=bool(vol.mpoint), mkfs=True) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/storage2/volumes/base.py", line 88, in ensure self._ensure() File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/storage2/volumes/cinder.py", line 160, in _ensure volume_type=self.volume_type) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/storage2/volumes/cinder.py", line 211, in _create_volume **kwds) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/platform/__init__.py", line 88, in __call__ return self.invoke(*args, **kwds) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/platform/openstack.py", line 186, in invoke return super(CinderConnectionProxy, self).invoke(*args, **kwds) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/scalarizr-4.8.2-py2.7.egg/scalarizr/platform/__init__.py", line 99, in invoke return self.obj(*args, **kwds) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 263, in create return self._create('/volumes', body, 'volume') File "/opt/scalarizr/embedded/lib/python2.7/site-packages/cinderclient/base.py", line 299, in _create resp, body = self.api.client.post(url, body=body) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/cinderclient/client.py", line 106, in post return self._cs_request(url, 'POST', **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/cinderclient/client.py", line 100, in _cs_request return self.request(url, method, **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/cinderclient/client.py", line 91, in request **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request return self.session.request(url, method, **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner return func(*args, **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/session.py", line 305, in request auth_headers = self.get_auth_headers(auth) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/session.py", line 618, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/auth/base.py", line 142, in get_headers token = self.get_token(session) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 202, in get_token return self.get_access(session).auth_token File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 248, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/session.py", line 502, in post return self.request(url, 'POST', **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner return func(*args, **kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/session.py", line 387, in request resp = send(**kwargs) File "/opt/scalarizr/embedded/lib/python2.7/site-packages/keystoneclient/session.py", line 431, in _send_request raise exceptions.ConnectionRefused(msg) ConnectionRefused: Unable to establish connection to https://cloud.domain.com:5000/v3/auth/tokens 2016-08-17 19:42:43,663+00:00 - DEBUG - scalarizr.messaging.p2p.producer - Sending message 'OperationResult' into queue 'control' 2016-08-17 19:42:43,701+00:00 - DEBUG - scalarizr.messaging.p2p.security - Encrypting message 2016-08-17 19:42:43,991+00:00 - DEBUG - scalarizr.messaging.p2p.consumer - Mark message (message_id: 604c4c39-51e1-4f64-9d50-5e7cc2aa1429) as handled