Failed to get updated incarnation from Monit

22 views
Skip to first unread message

张婕

unread,
Jan 14, 2015, 9:16:12 AM1/14/15
to bosh-...@cloudfoundry.org

I am trying to deploy cloud foundry on open stack with bosh, and after bosh deploy I got some errors .


-----------

Preparing deployment

  binding deployment (00:00:00)                                                                     

  binding releases (00:00:00)                                                                       

  binding existing deployment (00:00:00)                                                            

  binding resource pools (00:00:00)                                                                 

  binding stemcells (00:00:00)                                                                      

  binding templates (00:00:00)                                                                      

  binding properties (00:00:00)                                                                     

  binding unallocated VMs (00:00:00)                                                                

  binding instance networks (00:00:00)                                                              

Done                    9/9 00:00:00                                                                


Preparing package compilation

  finding packages to compile (00:00:00)                                                            

Done                    1/1 00:00:00                                                                


Preparing DNS

  binding DNS (00:00:00)                                                                            

Done                    1/1 00:00:00                                                                


Creating bound missing VMs

  small/0 (00:00:50)                                                                                

  small/1 (00:00:51)                                                                                

  small/3 (00:00:59)                                                                                

  small/2 (00:01:00)                                                                                

Done                    4/4 00:01:00                                                                


Binding instance VMs

  data/0 (00:00:01)                                                                                 

  core/0 (00:00:01)                                                                                 

  api/0 (00:00:01)                                                                                  

  dea/0 (00:00:01)                                                                                  

Done                    4/4 00:00:01                                                                


Preparing configuration

  binding configuration (00:00:01)                                                                  

Done                    1/1 00:00:01                                                                


  Started updating job data: data/0 (canary)

     Done updating job data: data/0 (canary)

  Started updating job core: core/0 (canary)

     Done updating job core: core/0 (canary)

  Started updating job api: api/0 (canary)  


Failed updating job api: api/0 (canary): Failed to get updated incarnation from Monit: ["/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/monit.rb:147:in `reload'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:178:in `reload_monit'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:77:in `apply'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:10:in `process'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:265:in `process'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:250:in `process_long_running'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:177:in `block in process_in_thread'", "<internal:prelude>:10:in `synchronize'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:175:in `process_in_thread'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:155:in `block in handle_message'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `block in spawn_threadpool'"]

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:80:in `rescue in apply'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:72:in `apply'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:10:in `process'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:265:in `process'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:250:in `process_long_running'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:177:in `block in process_in_thread'

<internal:prelude>:10:in `synchronize'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:175:in `process_in_thread'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:155:in `block in handle_message'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `block in spawn_threadpool'


Error 450001: Failed to get updated incarnation from Monit: [

"/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/monit.rb:147:in `reload'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:178:in `reload_monit’",

 "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:77:in `apply’",

 "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:10:in `process’",

 "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:265:in `process'”, 

"/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:250:in `process_long_running'”, 

"/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:177:in `block in process_in_thread'", "<internal:prelude>:10:in `synchronize'", "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:175:in `process_in_thread’",

 "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:155:in `block in handle_message’",

 "/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'”, 

"/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `block in spawn_threadpool'"]

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:80:in `rescue in apply'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:72:in `apply'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/message/apply.rb:10:in `process'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:265:in `process'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:250:in `process_long_running'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:177:in `block in process_in_thread'

<internal:prelude>:10:in `synchronize'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:175:in `process_in_thread'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib/bosh_agent/handler.rb:155:in `block in handle_message'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'

/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `block in spawn_threadpool'


Task 160 error



For a more detailed error report, run: bosh task 160 --debug

-----------------------------


I Checked monitrc in node api and find monitrc file for gorouter and cloud controller,

It seems something wrong with cloud controller, but I cannot figure out how to fix the problem.

can someone help me with this? 


/var/vcap/monit/job# vi 0000_api.gorouter.monitrc 

check process gorouter with pidfile /var/vcap/sys/run/gorouter/gorouter.pid start program "/var/vcap/jobs/gorouter/bin/gorouter_ctl start" with timeout 60 seconds stop program "/var/vcap/jobs/gorouter/bin/gorouter_ctl stop" group vcap mode manual

 

/var/vcap/monit/job# vi 0001_api.cloud_controller_ng.monitrc  

check process cloud_controller_ng with pidfile /var/vcap/sys/run/cloud_controller_ng/cloud_controller_ng.pid start program "/var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_ng_ctl start" stop program "/var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_ng_ctl stop" group vcap depends on nfs_mounter if totalmem > 2250 Mb for 3 cycles then alert if totalmem > 2250 Mb for 15 cycles then exec "/var/vcap/jobs/cloud_controller_ng/bin/restart_drain" if totalmem > 2450 Mb for 3 cycles then exec "/var/vcap/jobs/cloud_controller_ng/bin/restart_drain" if failed host 192.168.100.16 port 9022 protocol http and request '/v2/info' with timeout 60 seconds for 5 cycles then restart mode manual check process cloud_controller_worker_local_1 with pidfile /var/vcap/sys/run/cloud_controller_ng/cloud_controller_worker_1.pid start program "/var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl start 1" stop program "/var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl stop 1" depends on cloud_controller_ng group vcap if totalmem > 2250 Mb for 3 cycles then alert if totalmem > 2250 Mb for 15 cycles then restart if totalmem > 2450 Mb for 3 cycles then restart mode manual check process cloud_controller_worker_local_2 with pidfile /var/vcap/sys/run/cloud_controller_ng/cloud_controller_worker_2.pid start program "/var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl start 2" stop program "/var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl stop 2" depends on cloud_controller_ng group vcap if totalmem > 2250 Mb for 3 cycles then alert if totalmem > 2250 Mb for 15 cycles then restart if totalmem > 2450 Mb for 3 cycles then restart mode manual check process nginx_cc with pidfile /var/vcap/sys/run/nginx_cc/nginx.pid start program "/var/vcap/jobs/cloud_controller_ng/bin/nginx_ctl start" stop program "/var/vcap/jobs/cloud_controller_ng/bin/nginx_ctl stop" depends on cloud_controller_ng group vcap mode manual


Dmitriy Kalinin

unread,
Jan 14, 2015, 12:25:58 PM1/14/15
to bosh-...@cloudfoundry.org
Please use one of the supported version of stemcells from: http://bosh-artifacts.cfapps.io/

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

Reply all
Reply to author
Forward
0 new messages