instance creation failure - Jump BeforeHostUp and HostUp - did not send 'hostUp' event - Two timestamps found in scalarizr.log
66 views
Skip to first unread message
Brant Fortest
Dec 14, 2016, 11:09:42 PM12/14/16
to scalr-discuss
Hi, Team,
What's wrong with this issue? Happened several times! Zombie process on Scalr server? Why?
Server 'f61ed795-1be0-4c78-9ba8-b9e651f197f9' did not send 'hostUp'
event in 2400 seconds after launch (Try increasing timeouts in role
settings). Considering it broken. Terminating instance.
Two timestamps found in scalarizr.log
# cat /var/log/scalarizr.log;
2016-07-01 00:53:07,639+00:00 - INFO - scalarizr.app - [pid: 3953] Starting scalarizr 4.7.12
2016-07-01 00:53:07,640+00:00 - INFO - scalarizr.app - Configuring Scalarizr. This can take a few minutes...
2016-07-01 00:53:30,542+00:00 - INFO - scalarizr.messaging.p2p.consumer - Building message consumer server on 0.0.0.0:8013
2016-07-01 00:53:30,543+00:00 - INFO - scalarizr.app - Starting API server on http://0.0.0.0:8010
2016-07-01 00:53:30,551+00:00 - INFO - scalarizr.handlers.lifecycle - Server will be imported into Scalr
2016-07-01 00:53:30,553+00:00 - INFO - scalarizr.handlers - Detecting supported behaviors...
2016-07-01 00:53:30,567+00:00 - INFO - scalarizr.handlers - Scalr built-in automation: checking for supported software.
If installed software isn't detected, review the Scalr Wiki: https://scalr-wiki.atlassian.net/wiki/x/IoB1
2016-07-01 00:53:30,582+00:00 - INFO - scalarizr.handlers - app: Unavailable. Not installed.
2016-07-01 00:53:31,206+00:00 - INFO - scalarizr.handlers - chef: Available. Installed version: 12.1.2
2016-07-01 00:53:31,235+00:00 - INFO - scalarizr.handlers - haproxy: Unavailable. Not installed.
2016-07-01 00:53:31,275+00:00 - INFO - scalarizr.handlers - mariadb: Unavailable. Not installed.
2016-07-01 00:53:31,276+00:00 - INFO - scalarizr.handlers - memcached: Unavailable. Not installed.
2016-07-01 00:53:31,276+00:00 - INFO - scalarizr.handlers - mysql: Unavailable. Not installed.
2016-07-01 00:53:31,277+00:00 - INFO - scalarizr.handlers - mysql2: Unavailable. Not installed.
2016-07-01 00:53:31,278+00:00 - INFO - scalarizr.handlers - percona: Unavailable. Not installed.
2016-07-01 00:53:31,291+00:00 - INFO - scalarizr.handlers - postgresql: Unavailable. Not installed.
2016-07-01 00:53:31,294+00:00 - INFO - scalarizr.handlers - rabbitmq: Unavailable. Not installed.
2016-07-01 00:53:31,307+00:00 - INFO - scalarizr.handlers - redis: Unavailable. Not installed.
2016-07-01 00:53:31,316+00:00 - INFO - scalarizr.handlers - tomcat: Unavailable. Not installed.
2016-07-01 00:53:31,316+00:00 - INFO - scalarizr.handlers - www: Unavailable. Not installed.
2016-07-01 00:54:15,949+00:00 - INFO - scalarizr.scripts.reboot - Starting reboot script...
2016-07-01 00:54:15,964+00:00 - INFO - scalarizr.app - [pid: 3953] Stopping scalarizr 4.7.12
2016-07-01 00:54:16,623+00:00 - INFO - scalarizr.app - [pid: 3953] Scalarizr terminated
2016-12-15 03:36:49,407+00:00 - INFO - scalarizr.scripts.udev - Starting udev script...
2016-12-15 03:36:52,911+00:00 - INFO - scalarizr.scripts.udev - Starting udev script...
2016-12-15 03:38:35,992+00:00 - INFO - scalarizr.app - [pid: 1560] Starting scalarizr 4.12.0
2016-12-15 03:38:35,993+00:00 - INFO - scalarizr.node - State: unknown
2016-12-15 03:38:36,001+00:00 - INFO - scalarizr.app - Polling UpdateClient state...
2016-12-15 03:38:36,001+00:00 - INFO - scalarizr.app - UpdateClient state: in-progress/install
2016-12-15 03:38:38,117+00:00 - INFO - scalarizr.app - UpdateClient completed update and should be restarted, restarting
2016-12-15 03:38:38,118+00:00 - INFO - scalarizr.util.initdv2 - Stopping scalr-upd-client
2016-12-15 03:38:39,140+00:00 - INFO - scalarizr.util.initdv2 - Starting scalr-upd-client
2016-12-15 03:38:39,401+00:00 - INFO - scalarizr.app - This image was bundled with cloud API call. Cleanuping ancestor server data
2016-12-15 03:38:39,922+00:00 - INFO - scalarizr.node - State: bootstrapping
2016-12-15 03:38:52,295+00:00 - INFO - scalarizr.app - Starting API server on http://0.0.0.0:8010
2016-12-15 03:38:52,295+00:00 - INFO - scalarizr.messaging.p2p.consumer - Building message consumer server on 0.0.0.0:8013
2016-12-15 03:39:02,376+00:00 - INFO - agent.tasks.service.tasks - Service started: chef-client
2016-12-15 03:39:02,377+00:00 - INFO - scalarizr.handlers.lifecycle - Starting initialization
Igor Savchenko
Dec 14, 2016, 11:21:23 PM12/14/16
to scalr-...@googlegroups.com
Nothing bad here. First timestamp from the log that was bundled into
the server (when AMI was created). The second one is when a new server
started from the AMI.
> --
> You received this message because you are subscribed to the Google Groups
> "scalr-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scalr-discus...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
the server (when AMI was created). The second one is when a new server
started from the AMI.
> You received this message because you are subscribed to the Google Groups
> "scalr-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scalr-discus...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Igor Savchenko
Dec 14, 2016, 11:22:42 PM12/14/16
to scalr-...@googlegroups.com
"Server 'f61ed795-1be0-4c78-9ba8-b9e651f197f9' did not send 'hostUp'
event in 2400 seconds after launch (Try increasing timeouts in role
settings). Considering it broken. Terminating instance." - means that
event in 2400 seconds after launch (Try increasing timeouts in role
initialization took longer than a timeout. We need a full log to
figure out why it took so much time (Maybe orchestration script or
something else). You can increase timeout as well on Advanced tab in
the Farm Designer.
Regards,
Igor
Message has been deleted
Brant Fortest
Dec 15, 2016, 3:09:04 AM12/15/16
to scalr-discuss
Hi,
I increased timeout from 2400s to 4800s. There's still same errors."Terminate instance if it will not send 'hostUp' or 'hostInit' event after launch in 4800 seconds"
Igor Savchenko
Dec 15, 2016, 7:51:58 AM12/15/16
to scalr-...@googlegroups.com
Can you please provide a screenshot from Scalr Internal Messaging page
for this server.
for this server.
Message has been deleted
Igor Savchenko
Dec 15, 2016, 6:43:25 PM12/15/16
to scalr-...@googlegroups.com
Seems like your scalr is not performing well. Can you provide an
output of scalr-server-manage status command.
Thanks,
Igor
output of scalr-server-manage status command.
Thanks,
Igor
Brant Fortest
Dec 15, 2016, 9:08:34 PM12/15/16
to scalr-discuss
# scalr-server-manage status
crond RUNNING pid 1155, uptime 3:09:04
httpd RUNNING pid 1132, uptime 3:09:04
memcached RUNNING pid 1154, uptime 3:09:04
rrd RUNNING pid 1138, uptime 3:09:04
service-analytics_poller RUNNING pid 1144, uptime 3:09:04
service-analytics_processor RUNNING pid 1153, uptime 3:09:04
service-dbqueue RUNNING pid 1136, uptime 3:09:04
service-msgsender RUNNING pid 1141, uptime 3:09:04
service-plotter RUNNING pid 1151, uptime 3:09:04
service-poller RUNNING pid 1147, uptime 3:09:04
service-szrupdater RUNNING pid 1145, uptime 3:09:04
zmq_service RUNNING pid 1156, uptime 3:09:04
# scalr-server-manage status
crond RUNNING pid 1246, uptime 3:17:58
mysql RUNNING pid 1247, uptime 3:17:58
Reply all
Reply to author
Forward
0 new messages