I, [2016-04-26T13:35:33.269990 #2255] INFO -- : ==== Starting AppController ==== I, [2016-04-26T13:35:33.291052 #2255] INFO -- : Starting iaas_manager on ip 172.19.192.3, port 17444 with start command [/usr/bin/python2 /root/appscale/InfrastructureManager/infrastructure_manager_service.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/InfrastructureManager/infrastructure_manager_service.py /usr/bin/python2] I, [2016-04-26T13:35:33.298931 #2255] INFO -- : Started InfrastructureManager successfully! I, [2016-04-26T13:35:33.299057 #2255] INFO -- : Restoring AppController state I, [2016-04-26T13:35:33.299152 #2255] INFO -- : No recovery data found - skipping recovery process I, [2016-04-26T13:35:33.991393 #2255] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T13:35:34.338493 #2255] INFO -- : Converting 'verbose' with value 'False'. I, [2016-04-26T13:35:34.338899 #2255] INFO -- : Converting 'alter_etc_resolv' with value 'False'. I, [2016-04-26T13:35:34.339091 #2255] INFO -- : Converting 'ips' with value '[{"ip": "172.19.192.2", "disk": null, "jobs": ["memcache", "zookeeper", "db_master", "appengine", "taskqueue_slave"]}]'. I, [2016-04-26T13:35:34.339272 #2255] INFO -- : Converting 'table' with value 'cassandra'. I, [2016-04-26T13:35:34.339429 #2255] INFO -- : Converting 'user_commands' with value '[]'. I, [2016-04-26T13:35:34.339574 #2255] INFO -- : Converting 'autoscale' with value 'True'. I, [2016-04-26T13:35:34.339724 #2255] INFO -- : Converting 'replication' with value '1'. I, [2016-04-26T13:35:34.339901 #2255] INFO -- : Converting 'hostname' with value 'domain.com'. I, [2016-04-26T13:35:34.340052 #2255] INFO -- : Converting 'flower_password' with value 'appscale'. I, [2016-04-26T13:35:34.340196 #2255] INFO -- : Converting 'max_memory' with value '400'. I, [2016-04-26T13:35:34.340340 #2255] INFO -- : Converting 'clear_datastore' with value 'False'. I, [2016-04-26T13:35:34.340503 #2255] INFO -- : Converting 'keyname' with value 'appscale10eadeb855034ef981a8426735383816'. I, [2016-04-26T13:35:34.340644 #2255] INFO -- : Converting 'appengine' with value '1'. I, [2016-04-26T13:35:39.410796 #2255] INFO -- : Spawning/setting up other nodes. I, [2016-04-26T13:35:39.451898 #2255] INFO -- : Copying SSH keys to node at IP address 172.19.192.2 I, [2016-04-26T13:35:50.835024 #2255] INFO -- : Starting AppController for 172.19.192.2 returned * Starting system appscale-controller daemon ...done. . I, [2016-04-26T13:35:54.004409 #2255] INFO -- : Parameters set on node at 172.19.192.2 returned . I, [2016-04-26T13:35:54.004683 #2255] INFO -- : Done initializing nodes. I, [2016-04-26T13:35:55.282782 #2255] INFO -- : Starting nginx on ip 172.19.192.3, port 9999 with start command [/usr/bin/service nginx start] and stop command [/usr/bin/service nginx stop] I, [2016-04-26T13:35:55.643387 #2255] INFO -- : Starting Load Balancer I, [2016-04-26T13:35:55.765517 #2255] INFO -- : Starting haproxy on ip 172.19.192.3, port 9999 with start command [/usr/sbin/service haproxy start] and stop command [/usr/sbin/service haproxy stop] I, [2016-04-26T13:35:55.909551 #2255] INFO -- : Waiting for DB services ... I, [2016-04-26T13:36:06.952776 #2255] INFO -- : Done configuring zookeeper. I, [2016-04-26T13:36:25.339659 #2255] INFO -- : UserAppServer is ready. I, [2016-04-26T13:36:25.339912 #2255] INFO -- : Waiting for all services to finish starting up I, [2016-04-26T13:36:25.340011 #2255] INFO -- : Starting TaskQueue Master I, [2016-04-26T13:36:25.429387 #2255] INFO -- : Starting appmanagerserver on ip 172.19.192.3, port 49934 with start command [/usr/bin/python2 /root/appscale/AppManager/app_manager_server.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppManager/app_manager_server.py /usr/bin/python2] I, [2016-04-26T13:36:25.631023 #2255] INFO -- : Starting rabbitmq on ip 172.19.192.3, port 9999 with start command [/usr/sbin/rabbitmq-server -detached -setcookie 507e7846eb05f89483226683315a0570cbf18990] and stop command [/usr/sbin/rabbitmqctl stop] I, [2016-04-26T13:36:25.899691 #2255] INFO -- : Starting taskqueue on ip 172.19.192.3, port 64839 with start command [/usr/bin/python2 /root/appscale/AppTaskQueue/taskqueue_server.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppTaskQueue/taskqueue_server.py /usr/bin/python2] I, [2016-04-26T13:36:26.076766 #2255] INFO -- : Starting ejabberd on ip 172.19.192.3, port 9999 with start command [bash /root/appscale/AppController/lib/../scripts/start_ejabberd.sh] and stop command [/etc/init.d/ejabberd stop] I, [2016-04-26T13:36:37.298698 #2255] INFO -- : API services have started on this node I, [2016-04-26T13:36:37.299123 #2255] INFO -- : Starting AppDashboard I, [2016-04-26T13:36:37.299271 #2255] INFO -- : Starting Hermes service. I, [2016-04-26T13:36:37.411384 #2255] INFO -- : Starting hermes on ip 172.19.192.3, port 4378 with start command [/usr/bin/python2 /root/appscale/Hermes/hermes.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/Hermes/hermes.py /usr/bin/python2] I, [2016-04-26T13:36:37.436186 #2255] INFO -- : Done starting Hermes service. I, [2016-04-26T13:36:37.469548 #2255] INFO -- : Starting flower on ip 172.19.192.3, port 5555 with start command [/usr/local/bin/flower --basic_auth=appscale:appscale] and stop command [/bin/ps ax | /bin/grep flower | /bin/grep -v grep | /usr/bin/awk '{print $1}' | xargs kill -9] I, [2016-04-26T13:36:38.524166 #2255] INFO -- : Starting controller on ip 172.19.192.3, port 17443 with start command [/usr/bin/ruby -w /root/appscale/AppController/djinnServer.rb] and stop command [/usr/sbin/service appscale-controller stop] I, [2016-04-26T13:36:38.789969 #2255] INFO -- : Waiting for nodes to finish loading I, [2016-04-26T13:36:38.792252 #2255] INFO -- : Node at domain.com has finished loading. I, [2016-04-26T13:36:38.806493 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:36:49.509781 #2255] INFO -- : Starting python27 app appscaledashboard I, [2016-04-26T13:36:58.809572 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:37:14.138116 #2255] INFO -- : Priming AppDashboard's cache I, [2016-04-26T13:37:18.822025 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:37:19.544896 #2255] INFO -- : Starting taskqueue worker for appscaledashboard I, [2016-04-26T13:37:24.049781 #2255] INFO -- : Starting TaskQueue worker for app appscaledashboard: {"error"=>false} I, [2016-04-26T13:37:24.049984 #2255] INFO -- : Starting cron service for appscaledashboard I, [2016-04-26T13:37:24.141815 #2255] INFO -- : Parsed cron URL: /status/refresh I, [2016-04-26T13:37:24.987615 #2255] INFO -- : Writing crontab for [appscaledashboard]: MAILTO=\"\" */1 * * * * root curl -sSH "X-Appengine-Cron:true" -H "X-AppEngine-Fake-Is-Admin:aad799a56263bbc9fb164282dc5c5f1fbc57aea3" -k -L "http://domain.com:1080/status/refresh" 2>&1 >> /var/apps/appscaledashboard/log/cron.log I, [2016-04-26T13:37:39.274121 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:37:59.425308 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:38:19.430622 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:38:57.004045 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T13:39:57.807530 #2255] WARN -- : Saw a transient ZooKeeper error: response for meth: :get, args: [34, "/appscale/deployment_id", nil, nil], not received within 30 seconds /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/lib/zookeeper/continuation.rb:117:in `block in value' /usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/lib/zookeeper/continuation.rb:107:in `value' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/ext/c_zookeeper.rb:231:in `submit_and_block' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/ext/c_zookeeper.rb:40:in `get' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/lib/zookeeper/client_methods.rb:40:in `get' /root/appscale/AppController/lib/zkinterface.rb:690:in `block in exists?' /root/appscale/AppController/lib/zkinterface.rb:669:in `run_zookeeper_operation' /root/appscale/AppController/lib/zkinterface.rb:689:in `exists?' /root/appscale/AppController/djinn.rb:1386:in `deployment_id_exists' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/router.rb:458:in `call' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/router.rb:458:in `call' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/router.rb:174:in `route' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/soaplet.rb:87:in `block in do_POST' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/mapping/mapping.rb:533:in `protect_threadvars' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/soaplet.rb:84:in `do_POST' /usr/lib/ruby/1.9.1/webrick/httpservlet/abstract.rb:106:in `service' /usr/lib/ruby/1.9.1/webrick/httpserver.rb:138:in `service' /usr/lib/ruby/1.9.1/webrick/httpserver.rb:94:in `run' /usr/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread' W, [2016-04-26T13:40:41.456404 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T13:40:41.456875 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T13:40:41.567226 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T13:43:02.120076 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>1, :rc=>-101, :data=>nil, :stat=>#}. W, [2016-04-26T13:43:03.114830 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>2, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T13:43:03.122619 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:43:23.128035 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:43:31.458600 #2255] INFO -- : Received request to update with apps: [shell-php] I, [2016-04-26T13:43:43.139839 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:44:03.152421 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:44:23.164214 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:44:43.203047 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:45:03.205732 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:45:23.321938 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:45:43.583930 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:46:03.595413 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:46:23.604725 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:46:43.624986 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:47:03.670227 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:47:23.705002 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:47:43.796321 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:48:03.801420 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:48:23.805321 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:48:43.815504 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:49:03.838226 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:49:23.845795 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:49:43.855931 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:50:04.466626 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:50:17.149365 #2255] INFO -- : Received request to update with apps: [shell-php] I, [2016-04-26T13:50:24.472324 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:50:48.173376 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:51:08.701414 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:51:29.730406 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:51:52.884669 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:52:13.671794 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T13:52:52.367733 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T13:52:52.368002 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T13:53:51.358149 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. I, [2016-04-26T13:54:06.583978 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T13:54:30.816941 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>4, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T13:54:43.555846 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T13:54:43.556866 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>7, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T13:55:03.773779 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:55:26.685554 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:55:54.523412 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T13:55:54.656787 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>16, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T13:56:15.362934 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:56:36.188480 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:56:56.220356 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:57:24.723209 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T13:57:24.830643 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>27, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T13:57:45.259064 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:58:07.021479 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:58:29.870624 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:59:20.149656 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T13:59:42.160368 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:00:05.540931 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T14:00:08.208139 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>43, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T14:00:27.028415 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:00:54.749070 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T14:00:58.961844 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>49, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T14:01:48.424517 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T14:01:48.556587 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T14:02:12.699270 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T14:02:56.759226 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T14:03:48.294770 #2255] WARN -- : Saw a transient ZooKeeper error: response for meth: :create, args: [0, "/appcontroller/nodes/172.19.192.3/live", "", nil, [#>], 1], not received within 30 seconds /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/lib/zookeeper/continuation.rb:117:in `block in value' /usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/lib/zookeeper/continuation.rb:107:in `value' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/ext/c_zookeeper.rb:231:in `submit_and_block' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/ext/c_zookeeper.rb:40:in `create' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/ext/zookeeper_base.rb:157:in `create' /var/lib/gems/1.9.1/gems/zookeeper-1.4.11/lib/zookeeper/client_methods.rb:103:in `create' /root/appscale/AppController/lib/zkinterface.rb:484:in `block in set_live_node_ephemeral_link' /root/appscale/AppController/lib/zkinterface.rb:669:in `run_zookeeper_operation' /root/appscale/AppController/lib/zkinterface.rb:483:in `set_live_node_ephemeral_link' /root/appscale/AppController/lib/zkinterface.rb:139:in `reinitialize' /root/appscale/AppController/lib/zkinterface.rb:676:in `rescue in run_zookeeper_operation' /root/appscale/AppController/lib/zkinterface.rb:668:in `run_zookeeper_operation' /root/appscale/AppController/lib/zkinterface.rb:689:in `exists?' /root/appscale/AppController/djinn.rb:1386:in `deployment_id_exists' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/router.rb:458:in `call' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/router.rb:458:in `call' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/router.rb:174:in `route' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/soaplet.rb:87:in `block in do_POST' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/mapping/mapping.rb:533:in `protect_threadvars' /var/lib/gems/1.9.1/gems/soap4r-ruby1.9-2.0.5/lib/soap/rpc/soaplet.rb:84:in `do_POST' /usr/lib/ruby/1.9.1/webrick/httpservlet/abstract.rb:106:in `service' /usr/lib/ruby/1.9.1/webrick/httpserver.rb:138:in `service' /usr/lib/ruby/1.9.1/webrick/httpserver.rb:94:in `run' /usr/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread' W, [2016-04-26T14:03:49.581820 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>1, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T14:03:49.585225 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:04:09.594682 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:04:29.599374 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:04:49.615194 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:05:09.628002 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:05:30.029680 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:05:50.179027 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:06:11.073206 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:06:59.235033 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T14:07:02.818442 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T14:07:02.821747 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. W, [2016-04-26T14:07:03.835141 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>2, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T14:07:19.326856 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:07:22.330129 #2255] INFO -- : Received request to update with apps: [shell-php] I, [2016-04-26T14:07:39.334234 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:07:59.344882 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:08:19.383120 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:08:39.391487 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:08:59.619113 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:09:20.477275 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:09:41.344563 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:10:04.563608 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T14:10:27.515661 #2255] WARN -- : (get_deployment_id) failed talking to zookeeper with Failed to get /appscale/deployment_id, with info {:req_id=>34, :rc=>-101, :data=>nil, :stat=>#}. I, [2016-04-26T14:10:29.403721 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:11:26.099965 #2255] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. W, [2016-04-26T14:11:53.007131 #2255] WARN -- : Lost our ZooKeeper connection - making a new connection and trying again. I, [2016-04-26T14:14:03.068782 #1616] INFO -- : ==== Starting AppController ==== I, [2016-04-26T14:14:03.193532 #1616] INFO -- : Starting iaas_manager on ip 172.19.192.3, port 17444 with start command [/usr/bin/python2 /root/appscale/InfrastructureManager/infrastructure_manager_service.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/InfrastructureManager/infrastructure_manager_service.py /usr/bin/python2] I, [2016-04-26T14:14:03.277913 #1616] INFO -- : Started InfrastructureManager successfully! I, [2016-04-26T14:14:03.278098 #1616] INFO -- : Restoring AppController state I, [2016-04-26T14:14:03.278265 #1616] INFO -- : No recovery data found - skipping recovery process I, [2016-04-26T14:14:04.119327 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:09.120120 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:14.125146 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:19.127797 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:24.128318 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:29.134908 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:34.144881 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:39.154866 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:44.164877 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:49.194870 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:54.204959 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:14:59.214919 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:15:04.245085 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:15:09.265199 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:15:14.277991 #1616] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:15:17.189085 #1616] INFO -- : Converting 'verbose' with value 'False'. I, [2016-04-26T14:15:17.189271 #1616] INFO -- : Converting 'alter_etc_resolv' with value 'False'. I, [2016-04-26T14:15:17.189360 #1616] INFO -- : Converting 'ips' with value '[{"ip": "172.19.192.2", "disk": null, "jobs": ["memcache", "db_master", "appengine", "taskqueue_slave"]}]'. I, [2016-04-26T14:15:17.189495 #1616] INFO -- : Converting 'table' with value 'cassandra'. I, [2016-04-26T14:15:17.189573 #1616] INFO -- : Converting 'user_commands' with value '[]'. I, [2016-04-26T14:15:17.189653 #1616] INFO -- : Converting 'autoscale' with value 'True'. I, [2016-04-26T14:15:17.189768 #1616] INFO -- : Converting 'replication' with value '1'. I, [2016-04-26T14:15:17.189855 #1616] INFO -- : Converting 'hostname' with value 'domain.com'. I, [2016-04-26T14:15:17.189943 #1616] INFO -- : Converting 'flower_password' with value 'appscale'. I, [2016-04-26T14:15:17.190017 #1616] INFO -- : Converting 'max_memory' with value '400'. I, [2016-04-26T14:15:17.190105 #1616] INFO -- : Converting 'clear_datastore' with value 'False'. I, [2016-04-26T14:15:17.190180 #1616] INFO -- : Converting 'keyname' with value 'appscale10eadeb855034ef981a8426735383816'. I, [2016-04-26T14:15:17.190254 #1616] INFO -- : Converting 'appengine' with value '1'. I, [2016-04-26T14:15:19.676469 #1616] INFO -- : Spawning/setting up other nodes. I, [2016-04-26T14:15:19.738522 #1616] INFO -- : Copying SSH keys to node at IP address 172.19.192.2 I, [2016-04-26T14:15:39.947215 #1616] INFO -- : Starting AppController for 172.19.192.2 returned appscale-controller already running. . I, [2016-04-26T14:15:40.125383 #1616] INFO -- : Parameters set on node at 172.19.192.2 returned . I, [2016-04-26T14:15:40.125875 #1616] INFO -- : Done initializing nodes. I, [2016-04-26T14:15:41.204155 #1616] INFO -- : Starting nginx on ip 172.19.192.3, port 9999 with start command [/usr/bin/service nginx start] and stop command [/usr/bin/service nginx stop] I, [2016-04-26T14:15:41.407237 #1616] INFO -- : Starting Load Balancer I, [2016-04-26T14:15:41.734576 #1616] INFO -- : Starting haproxy on ip 172.19.192.3, port 9999 with start command [/usr/sbin/service haproxy start] and stop command [/usr/sbin/service haproxy stop] I, [2016-04-26T14:15:41.792370 #1616] INFO -- : Waiting for DB services ... I, [2016-04-26T14:15:41.792561 #1616] INFO -- : Starting zookeeper. I, [2016-04-26T14:15:41.846504 #1616] INFO -- : Starting ZooKeeper I, [2016-04-26T14:15:44.273906 #1616] INFO -- : Starting zookeeper on ip 172.19.192.3, port 2181 with start command [/usr/sbin/service zookeeper start] and stop command [/usr/sbin/service zookeeper stop] I, [2016-04-26T14:15:44.291015 #1616] INFO -- : Started ZooKeeper I, [2016-04-26T14:15:50.415698 #1616] INFO -- : Done configuring zookeeper. W, [2016-04-26T14:16:10.506673 #1616] WARN -- : [get_all_apps] SOAP call to 172.19.192.3 timed out I, [2016-04-26T14:16:13.590997 #1616] INFO -- : UserAppServer is ready. I, [2016-04-26T14:16:13.591288 #1616] INFO -- : Waiting for all services to finish starting up I, [2016-04-26T14:16:13.591704 #1616] INFO -- : Starting TaskQueue Master I, [2016-04-26T14:16:13.646610 #1616] INFO -- : Starting appmanagerserver on ip 172.19.192.3, port 49934 with start command [/usr/bin/python2 /root/appscale/AppManager/app_manager_server.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppManager/app_manager_server.py /usr/bin/python2] I, [2016-04-26T14:16:13.667121 #1616] INFO -- : Starting backup_recovery_service on ip 172.19.192.3, port 8423 with start command [/usr/bin/python2 /root/appscale/AppDB/backup/backup_recovery_service.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppDB/backup/backup_recovery_service.py /usr/bin/python2] I, [2016-04-26T14:16:13.699922 #1616] INFO -- : Starting rabbitmq on ip 172.19.192.3, port 9999 with start command [/usr/sbin/rabbitmq-server -detached -setcookie 157292eeccb2d5a8205e014cced4cb448f68fb20] and stop command [/usr/sbin/rabbitmqctl stop] I, [2016-04-26T14:16:13.756397 #1616] INFO -- : Starting taskqueue on ip 172.19.192.3, port 64839 with start command [/usr/bin/python2 /root/appscale/AppTaskQueue/taskqueue_server.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppTaskQueue/taskqueue_server.py /usr/bin/python2] I, [2016-04-26T14:16:13.780512 #1616] INFO -- : Starting ejabberd on ip 172.19.192.3, port 9999 with start command [bash /root/appscale/AppController/lib/../scripts/start_ejabberd.sh] and stop command [/etc/init.d/ejabberd stop] I, [2016-04-26T14:16:18.907467 #1616] INFO -- : API services have started on this node I, [2016-04-26T14:16:18.907706 #1616] INFO -- : Starting AppDashboard I, [2016-04-26T14:16:18.907811 #1616] INFO -- : Starting Hermes service. I, [2016-04-26T14:16:19.044207 #1616] INFO -- : Starting hermes on ip 172.19.192.3, port 4378 with start command [/usr/bin/python2 /root/appscale/Hermes/hermes.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/Hermes/hermes.py /usr/bin/python2] I, [2016-04-26T14:16:19.062569 #1616] INFO -- : Done starting Hermes service. I, [2016-04-26T14:16:19.101804 #1616] INFO -- : Starting flower on ip 172.19.192.3, port 5555 with start command [/usr/local/bin/flower --basic_auth=appscale:appscale] and stop command [/bin/ps ax | /bin/grep flower | /bin/grep -v grep | /usr/bin/awk '{print $1}' | xargs kill -9] I, [2016-04-26T14:16:19.770414 #1616] INFO -- : Starting controller on ip 172.19.192.3, port 17443 with start command [/usr/bin/ruby -w /root/appscale/AppController/djinnServer.rb] and stop command [/usr/sbin/service appscale-controller stop] I, [2016-04-26T14:16:20.000526 #1616] INFO -- : Waiting for nodes to finish loading I, [2016-04-26T14:16:20.036299 #1616] INFO -- : Node at domain.com has finished loading. I, [2016-04-26T14:16:20.038925 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:16:25.773425 #1616] INFO -- : Starting python27 app appscaledashboard I, [2016-04-26T14:16:40.048269 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:16:42.608753 #1616] INFO -- : Priming AppDashboard's cache I, [2016-04-26T14:16:43.592550 #1616] INFO -- : Starting taskqueue worker for appscaledashboard I, [2016-04-26T14:16:45.752760 #1616] INFO -- : Starting TaskQueue worker for app appscaledashboard: {"error"=>false} I, [2016-04-26T14:16:45.752852 #1616] INFO -- : Starting cron service for appscaledashboard I, [2016-04-26T14:16:45.753743 #1616] INFO -- : Parsed cron URL: /status/refresh I, [2016-04-26T14:16:45.821633 #1616] INFO -- : Writing crontab for [appscaledashboard]: MAILTO=\"\" */1 * * * * root curl -sSH "X-Appengine-Cron:true" -H "X-AppEngine-Fake-Is-Admin:330a4b1d3ad6f7e04117bb7f73fd04dabb918b0a" -k -L "http://domain.com:1080/status/refresh" 2>&1 >> /var/apps/appscaledashboard/log/cron.log I, [2016-04-26T14:17:00.058439 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:17:20.071174 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:17:38.589466 #1616] INFO -- : Received request to update with apps: [shell-php] I, [2016-04-26T14:17:40.079243 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:18:00.088375 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:18:20.099794 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:18:40.108617 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:19:00.121093 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:19:20.128908 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:19:40.142903 #1616] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:20:25.626399 #10023] INFO -- : ==== Starting AppController ==== I, [2016-04-26T14:20:25.636536 #10023] INFO -- : Starting iaas_manager on ip 172.19.192.3, port 17444 with start command [/usr/bin/python2 /root/appscale/InfrastructureManager/infrastructure_manager_service.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/InfrastructureManager/infrastructure_manager_service.py /usr/bin/python2] I, [2016-04-26T14:20:25.642464 #10023] INFO -- : Started InfrastructureManager successfully! I, [2016-04-26T14:20:25.642573 #10023] INFO -- : Restoring AppController state I, [2016-04-26T14:20:25.642648 #10023] INFO -- : No recovery data found - skipping recovery process I, [2016-04-26T14:20:25.691784 #10023] INFO -- : Starting nginx on ip 172.19.192.3, port 9999 with start command [/usr/bin/service nginx start] and stop command [/usr/bin/service nginx stop] I, [2016-04-26T14:20:25.734324 #10023] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:20:26.308931 #10023] INFO -- : Converting 'verbose' with value 'False'. I, [2016-04-26T14:20:26.309205 #10023] INFO -- : Converting 'alter_etc_resolv' with value 'False'. I, [2016-04-26T14:20:26.309363 #10023] INFO -- : Converting 'ips' with value '[{"ip": "172.19.192.2", "disk": null, "jobs": ["db_master"]}]'. I, [2016-04-26T14:20:26.309520 #10023] INFO -- : Converting 'table' with value 'cassandra'. I, [2016-04-26T14:20:26.309650 #10023] INFO -- : Converting 'user_commands' with value '[]'. I, [2016-04-26T14:20:26.309776 #10023] INFO -- : Converting 'autoscale' with value 'True'. I, [2016-04-26T14:20:26.309936 #10023] INFO -- : Converting 'replication' with value '1'. I, [2016-04-26T14:20:26.310094 #10023] INFO -- : Converting 'hostname' with value 'domain.com'. I, [2016-04-26T14:20:26.310230 #10023] INFO -- : Converting 'flower_password' with value 'appscale'. I, [2016-04-26T14:20:26.310351 #10023] INFO -- : Converting 'max_memory' with value '400'. I, [2016-04-26T14:20:26.310499 #10023] INFO -- : Converting 'clear_datastore' with value 'False'. I, [2016-04-26T14:20:26.310619 #10023] INFO -- : Converting 'keyname' with value 'appscale10eadeb855034ef981a8426735383816'. I, [2016-04-26T14:20:26.310734 #10023] INFO -- : Converting 'appengine' with value '1'. I, [2016-04-26T14:20:30.869034 #10023] INFO -- : Spawning/setting up other nodes. I, [2016-04-26T14:20:30.977542 #10023] INFO -- : Copying SSH keys to node at IP address 172.19.192.2 I, [2016-04-26T14:20:40.220954 #10023] INFO -- : Starting AppController for 172.19.192.2 returned * Starting system appscale-controller daemon ...done. . I, [2016-04-26T14:20:41.419702 #10023] INFO -- : Parameters set on node at 172.19.192.2 returned . I, [2016-04-26T14:20:41.420194 #10023] INFO -- : Done initializing nodes. I, [2016-04-26T14:20:42.558065 #10023] INFO -- : Starting nginx on ip 172.19.192.3, port 9999 with start command [/usr/bin/service nginx start] and stop command [/usr/bin/service nginx stop] I, [2016-04-26T14:20:42.818744 #10023] INFO -- : Starting Load Balancer I, [2016-04-26T14:20:42.890659 #10023] INFO -- : Starting haproxy on ip 172.19.192.3, port 9999 with start command [/usr/sbin/service haproxy start] and stop command [/usr/sbin/service haproxy stop] I, [2016-04-26T14:20:42.996499 #10023] INFO -- : Waiting for DB services ... I, [2016-04-26T14:20:42.996653 #10023] INFO -- : Starting zookeeper. I, [2016-04-26T14:20:43.018811 #10023] INFO -- : Starting ZooKeeper I, [2016-04-26T14:20:43.360214 #10023] INFO -- : Starting zookeeper on ip 172.19.192.3, port 2181 with start command [/usr/sbin/service zookeeper start] and stop command [/usr/sbin/service zookeeper stop] I, [2016-04-26T14:20:43.366619 #10023] INFO -- : Started ZooKeeper I, [2016-04-26T14:20:44.398086 #10023] INFO -- : Done configuring zookeeper. W, [2016-04-26T14:21:04.467927 #10023] WARN -- : [get_all_apps] SOAP call to 172.19.192.3 timed out I, [2016-04-26T14:21:05.556757 #10023] INFO -- : UserAppServer is ready. I, [2016-04-26T14:21:05.557356 #10023] INFO -- : Waiting for all services to finish starting up I, [2016-04-26T14:21:05.622086 #10023] INFO -- : Starting TaskQueue Master I, [2016-04-26T14:21:05.639915 #10023] INFO -- : Starting up memcache I, [2016-04-26T14:21:05.651585 #10023] INFO -- : Starting appmanagerserver on ip 172.19.192.3, port 49934 with start command [/usr/bin/python2 /root/appscale/AppManager/app_manager_server.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppManager/app_manager_server.py /usr/bin/python2] I, [2016-04-26T14:21:05.659795 #10023] INFO -- : Starting blobstore on ip 172.19.192.3, port 6107 with start command [/usr/bin/python2 /root/appscale/AppDB/blobstore/blobstore_server.py -d 172.19.192.3:8888 -p 6107] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppDB/blobstore/blobstore_server.py /usr/bin/python2] I, [2016-04-26T14:21:05.679544 #10023] INFO -- : Starting backup_recovery_service on ip 172.19.192.3, port 8423 with start command [/usr/bin/python2 /root/appscale/AppDB/backup/backup_recovery_service.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppDB/backup/backup_recovery_service.py /usr/bin/python2] I, [2016-04-26T14:21:05.734729 #10023] INFO -- : Starting memcached on ip 172.19.192.3, port 11211 with start command [/usr/bin/memcached -m 64 -p 11211 -u root] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /usr/bin/memcached 11211] I, [2016-04-26T14:21:05.769752 #10023] INFO -- : Starting rabbitmq on ip 172.19.192.3, port 9999 with start command [/usr/sbin/rabbitmq-server -detached -setcookie 57c6c5fbe8b3b20d586e767c6550c5a30c57ecbf] and stop command [/usr/sbin/rabbitmqctl stop] I, [2016-04-26T14:21:05.906939 #10023] INFO -- : Starting taskqueue on ip 172.19.192.3, port 64839 with start command [/usr/bin/python2 /root/appscale/AppTaskQueue/taskqueue_server.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppTaskQueue/taskqueue_server.py /usr/bin/python2] I, [2016-04-26T14:21:05.931357 #10023] INFO -- : Starting ejabberd on ip 172.19.192.3, port 9999 with start command [bash /root/appscale/AppController/lib/../scripts/start_ejabberd.sh] and stop command [/etc/init.d/ejabberd stop] I, [2016-04-26T14:21:09.073811 #10023] INFO -- : API services have started on this node I, [2016-04-26T14:21:09.074043 #10023] INFO -- : Starting AppDashboard I, [2016-04-26T14:21:09.074110 #10023] INFO -- : Starting Hermes service. I, [2016-04-26T14:21:09.128709 #10023] INFO -- : Starting hermes on ip 172.19.192.3, port 4378 with start command [/usr/bin/python2 /root/appscale/Hermes/hermes.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/Hermes/hermes.py /usr/bin/python2] I, [2016-04-26T14:21:09.168579 #10023] INFO -- : Done starting Hermes service. I, [2016-04-26T14:21:09.238649 #10023] INFO -- : Starting flower on ip 172.19.192.3, port 5555 with start command [/usr/local/bin/flower --basic_auth=appscale:appscale] and stop command [/bin/ps ax | /bin/grep flower | /bin/grep -v grep | /usr/bin/awk '{print $1}' | xargs kill -9] I, [2016-04-26T14:21:09.935628 #10023] INFO -- : Starting python27 app appscaledashboard I, [2016-04-26T14:21:10.210509 #10023] INFO -- : Starting controller on ip 172.19.192.3, port 17443 with start command [/usr/bin/ruby -w /root/appscale/AppController/djinnServer.rb] and stop command [/usr/sbin/service appscale-controller stop] I, [2016-04-26T14:21:10.486306 #10023] INFO -- : Waiting for nodes to finish loading I, [2016-04-26T14:21:10.489055 #10023] INFO -- : Node at domain.com has finished loading. I, [2016-04-26T14:21:10.490252 #10023] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:21:26.189896 #10023] INFO -- : Priming AppDashboard's cache I, [2016-04-26T14:21:27.904322 #10023] INFO -- : Starting taskqueue worker for appscaledashboard I, [2016-04-26T14:21:30.081665 #10023] INFO -- : Starting TaskQueue worker for app appscaledashboard: {"error"=>false} I, [2016-04-26T14:21:30.081754 #10023] INFO -- : Starting cron service for appscaledashboard I, [2016-04-26T14:21:30.082639 #10023] INFO -- : Parsed cron URL: /status/refresh I, [2016-04-26T14:21:30.129045 #10023] INFO -- : Writing crontab for [appscaledashboard]: MAILTO=\"\" */1 * * * * root curl -sSH "X-Appengine-Cron:true" -H "X-AppEngine-Fake-Is-Admin:49e885f1bf12d03f9cf698263669f74e29cc2853" -k -L "http://domain.com:1080/status/refresh" 2>&1 >> /var/apps/appscaledashboard/log/cron.log I, [2016-04-26T14:21:30.500412 #10023] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:21:47.630300 #10023] INFO -- : Received request to update with apps: [shell-php] I, [2016-04-26T14:21:50.510471 #10023] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:22:10.525367 #10023] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:22:30.528757 #10023] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:22:50.538617 #10023] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:25:02.659720 #1989] INFO -- : ==== Starting AppController ==== I, [2016-04-26T14:25:02.668138 #1989] INFO -- : Starting iaas_manager on ip 172.19.192.3, port 17444 with start command [/usr/bin/python2 /root/appscale/InfrastructureManager/infrastructure_manager_service.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/InfrastructureManager/infrastructure_manager_service.py /usr/bin/python2] I, [2016-04-26T14:25:02.673857 #1989] INFO -- : Started InfrastructureManager successfully! I, [2016-04-26T14:25:02.673966 #1989] INFO -- : Restoring AppController state I, [2016-04-26T14:25:02.674045 #1989] INFO -- : No recovery data found - skipping recovery process I, [2016-04-26T14:25:02.787237 #1989] INFO -- : Waiting for data from the load balancer or cmdline tools I, [2016-04-26T14:25:04.428808 #1989] INFO -- : Converting 'verbose' with value 'False'. I, [2016-04-26T14:25:04.428994 #1989] INFO -- : Converting 'alter_etc_resolv' with value 'False'. I, [2016-04-26T14:25:04.429089 #1989] INFO -- : Converting 'ips' with value '[{"ip": "172.19.192.2", "disk": null, "jobs": ["memcache", "zookeeper", "db_master", "appengine", "taskqueue_slave"]}]'. I, [2016-04-26T14:25:04.429191 #1989] INFO -- : Converting 'table' with value 'cassandra'. I, [2016-04-26T14:25:04.429272 #1989] INFO -- : Converting 'user_commands' with value '[]'. I, [2016-04-26T14:25:04.429356 #1989] INFO -- : Converting 'autoscale' with value 'True'. I, [2016-04-26T14:25:04.429441 #1989] INFO -- : Converting 'replication' with value '1'. I, [2016-04-26T14:25:04.429534 #1989] INFO -- : Converting 'hostname' with value 'domain.com'. I, [2016-04-26T14:25:04.429625 #1989] INFO -- : Converting 'flower_password' with value 'appscale'. I, [2016-04-26T14:25:04.429701 #1989] INFO -- : Converting 'max_memory' with value '400'. I, [2016-04-26T14:25:04.429783 #1989] INFO -- : Converting 'clear_datastore' with value 'False'. I, [2016-04-26T14:25:04.429871 #1989] INFO -- : Converting 'keyname' with value 'appscale10eadeb855034ef981a8426735383816'. I, [2016-04-26T14:25:04.429952 #1989] INFO -- : Converting 'appengine' with value '1'. I, [2016-04-26T14:25:07.893250 #1989] INFO -- : Spawning/setting up other nodes. I, [2016-04-26T14:25:07.981965 #1989] INFO -- : Copying SSH keys to node at IP address 172.19.192.2 I, [2016-04-26T14:25:17.294519 #1989] INFO -- : Starting AppController for 172.19.192.2 returned * Starting system appscale-controller daemon ...done. . I, [2016-04-26T14:25:18.537158 #1989] INFO -- : Parameters set on node at 172.19.192.2 returned . I, [2016-04-26T14:25:18.537355 #1989] INFO -- : Done initializing nodes. I, [2016-04-26T14:25:19.606869 #1989] INFO -- : Starting nginx on ip 172.19.192.3, port 9999 with start command [/usr/bin/service nginx start] and stop command [/usr/bin/service nginx stop] I, [2016-04-26T14:25:19.847645 #1989] INFO -- : Starting Load Balancer I, [2016-04-26T14:25:19.918738 #1989] INFO -- : Starting haproxy on ip 172.19.192.3, port 9999 with start command [/usr/sbin/service haproxy start] and stop command [/usr/sbin/service haproxy stop] I, [2016-04-26T14:25:20.021709 #1989] INFO -- : Waiting for DB services ... I, [2016-04-26T14:25:30.100145 #1989] INFO -- : Done configuring zookeeper. I, [2016-04-26T14:25:44.546768 #1989] INFO -- : UserAppServer is ready. I, [2016-04-26T14:25:44.547281 #1989] INFO -- : Waiting for all services to finish starting up I, [2016-04-26T14:25:44.547488 #1989] INFO -- : Starting TaskQueue Master I, [2016-04-26T14:25:44.611421 #1989] INFO -- : Starting appmanagerserver on ip 172.19.192.3, port 49934 with start command [/usr/bin/python2 /root/appscale/AppManager/app_manager_server.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppManager/app_manager_server.py /usr/bin/python2] I, [2016-04-26T14:25:44.638792 #1989] INFO -- : Starting rabbitmq on ip 172.19.192.3, port 9999 with start command [/usr/sbin/rabbitmq-server -detached -setcookie 285c2f1f1c6451c3522c394f5bc7bfab009dcaf8] and stop command [/usr/sbin/rabbitmqctl stop] I, [2016-04-26T14:25:44.719643 #1989] INFO -- : Starting taskqueue on ip 172.19.192.3, port 64839 with start command [/usr/bin/python2 /root/appscale/AppTaskQueue/taskqueue_server.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/AppTaskQueue/taskqueue_server.py /usr/bin/python2] I, [2016-04-26T14:25:44.729588 #1989] INFO -- : Starting ejabberd on ip 172.19.192.3, port 9999 with start command [bash /root/appscale/AppController/lib/../scripts/start_ejabberd.sh] and stop command [/etc/init.d/ejabberd stop] I, [2016-04-26T14:25:51.789638 #1989] INFO -- : API services have started on this node I, [2016-04-26T14:25:51.789879 #1989] INFO -- : Starting AppDashboard I, [2016-04-26T14:25:51.789948 #1989] INFO -- : Starting Hermes service. I, [2016-04-26T14:25:51.844518 #1989] INFO -- : Starting hermes on ip 172.19.192.3, port 4378 with start command [/usr/bin/python2 /root/appscale/Hermes/hermes.py] and stop command [/usr/bin/python2 /root/appscale/scripts/stop_service.py /root/appscale/Hermes/hermes.py /usr/bin/python2] I, [2016-04-26T14:25:51.882194 #1989] INFO -- : Done starting Hermes service. I, [2016-04-26T14:25:51.948955 #1989] INFO -- : Starting flower on ip 172.19.192.3, port 5555 with start command [/usr/local/bin/flower --basic_auth=appscale:appscale] and stop command [/bin/ps ax | /bin/grep flower | /bin/grep -v grep | /usr/bin/awk '{print $1}' | xargs kill -9] I, [2016-04-26T14:25:52.676871 #1989] INFO -- : Starting python27 app appscaledashboard I, [2016-04-26T14:25:53.078070 #1989] INFO -- : Starting controller on ip 172.19.192.3, port 17443 with start command [/usr/bin/ruby -w /root/appscale/AppController/djinnServer.rb] and stop command [/usr/sbin/service appscale-controller stop] I, [2016-04-26T14:25:53.596159 #1989] INFO -- : Waiting for nodes to finish loading I, [2016-04-26T14:25:53.609550 #1989] INFO -- : Node at domain.com has finished loading. I, [2016-04-26T14:25:53.618177 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:26:10.075243 #1989] INFO -- : Priming AppDashboard's cache I, [2016-04-26T14:26:11.041778 #1989] INFO -- : Starting taskqueue worker for appscaledashboard I, [2016-04-26T14:26:13.149228 #1989] INFO -- : Starting TaskQueue worker for app appscaledashboard: {"error"=>false} I, [2016-04-26T14:26:13.149316 #1989] INFO -- : Starting cron service for appscaledashboard I, [2016-04-26T14:26:13.150202 #1989] INFO -- : Parsed cron URL: /status/refresh I, [2016-04-26T14:26:13.197982 #1989] INFO -- : Writing crontab for [appscaledashboard]: MAILTO=\"\" */1 * * * * root curl -sSH "X-Appengine-Cron:true" -H "X-AppEngine-Fake-Is-Admin:55e9d584a7e3fd4dd7f66fb5cd43ea17947f7087" -k -L "http://domain.com:1080/status/refresh" 2>&1 >> /var/apps/appscaledashboard/log/cron.log I, [2016-04-26T14:26:13.628331 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:26:33.638080 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:26:53.648120 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:27:13.651110 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:27:33.658388 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:27:53.684486 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:28:13.688597 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:28:33.698465 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:28:53.708646 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:29:13.718910 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:29:33.728669 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:29:53.732347 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:30:13.748499 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:30:33.817036 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:30:53.828327 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:31:13.838380 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:31:33.841516 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:31:53.844402 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:32:13.848360 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:32:33.857901 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:32:53.867799 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:33:13.878115 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:33:33.887955 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:33:53.928687 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:34:13.951588 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:34:33.957592 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:34:53.967764 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:35:13.977630 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:35:33.987662 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:35:53.992477 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish. I, [2016-04-26T14:36:14.046270 #1989] INFO -- : Node at 172.19.192.2 has not yet finished loading - will wait for it to finish.