`director/0' is not running after update

98 views

Skip to first unread message

Guangcai Wang

unread,

Jul 18, 2014, 3:14:24 AM7/18/14

to bosh-...@cloudfoundry.org

I am trying to installing cloudfoundry on openstack following the steps http://docs.cloudfoundry.org/deploying/openstack/.

On step 6, Deploying BOSH using Micro BOSH, I got an error -- `director/0' is not running after update, which led to task failure. Any idea?

E, [2014-07-18T04:04:20.528175 #3565] [canary_update(director/0)] ERROR -- : Error updating canary instance: #<Bosh::Director::AgentJobNotRunning: `director/0' is not running after update>
/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.2624.0/lib/bosh/director/instance_updater.rb:85:in `update'
/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.2624.0/lib/bosh/director/job_updater.rb:74:in `block (2 levels) in update_canary_instance'
/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh_common-1.2624.0/lib/common/thread_formatter.rb:46:in `with_thread_name'
/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.2624.0/lib/bosh/director/job_updater.rb:72:in `block in update_canary_instance'
/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.2624.0/lib/bosh/director/event_log.rb:83:in `call'
/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.2624.0/lib/bosh/director/event_log.rb:83:in `advance_and_track'
/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.2624.0/lib/bosh/director/job_updater.rb:71:in `update_canary_instance'

Then I logged in the director VM, and tried to start it by

monit start director

and found the following exception under /var/vcap/sys/log/director

/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.0000.0/bin/bosh-director-scheduler:3:in `require': cannot load such file -- bosh/director
from /var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.0000.0/bin/bosh-director-scheduler:3:in `<top (required)>'
from /var/vcap/packages/director/bin/bosh-director-scheduler:16:in `load'
from /var/vcap/packages/director/bin/bosh-director-scheduler:16:in `<main>'

==================below is my environment information========

ubuntu@ubuntu12:~/bosh-workspace/deployments$ gem list bosh

*** LOCAL GEMS ***

bosh-director-core (1.2641.0)
bosh-registry (1.2641.0)
bosh-stemcell (1.2641.0)
bosh_aws_cpi (1.2641.0)
bosh_cli (1.2641.0)
bosh_cli_plugin_micro (1.2641.0)
bosh_common (1.2641.0)
bosh_cpi (1.2641.0)
bosh_openstack_cpi (1.2641.0)
bosh_vcloud_cpi (0.5.4)
bosh_vsphere_cpi (1.2641.0)
ubuntu@ubuntu12:~/bosh-workspace/deployments$

ubuntu@ubuntu12:~/bosh-workspace/deployments$ bosh status
Config
/home/ubuntu/.bosh_config

Director
Name       microbosh-openstack
URL        https://10.0.0.12:25555
Version    1.2624.0 (00000000)
User       admin
UUID       a11193e9-99ff-4f85-8dc0-8a06509aeed2
CPI        openstack
dns        enabled (domain_name: microbosh)
compiled_package_cache disabled
snapshots disabled

Deployment
Manifest /home/ubuntu/bosh-workspace/deployments/bosh-openstack/bosh-openstack.yml
ubuntu@ubuntu12:~/bosh-workspace/deployments$
ubuntu@ubuntu12:~/bosh-workspace/deployments$
ubuntu@ubuntu12:~/bosh-workspace/deployments$ bosh vms
Deployment `bosh-openstack'

Director task 25

Task 25 done

VMs total: 8
ubuntu@ubuntu12:~/bosh-workspace/deployments$

ubuntu@ubuntu12:~/bosh-workspace/deployments$ bosh stemcells

+---------------------------+---------+--------------------------------------+
| Name | Version | CID |
+---------------------------+---------+--------------------------------------+
| bosh-openstack-kvm-ubuntu | 2427* | 174deff1-4855-4fc7-8e28-6476a8b638ef |
+---------------------------+---------+--------------------------------------+

(*) Currently in-use

Stemcells total: 1
ubuntu@ubuntu12:~/bosh-workspace/deployments$

ubuntu@ubuntu12:~/bosh-workspace/deployments$ bosh releases

+------+-----------+-------------+
| Name | Versions | Commit Hash |
+------+-----------+-------------+
| bosh | 91+dev.1 | 00000000 |
| | 91+dev.2* | 00000000 |
+------+-----------+-------------+
(*) Currently deployed

Releases total: 1
ubuntu@ubuntu12:~/bosh-workspace/deployments$

ubuntu@ubuntu12:~/bosh-workspace/deployments$ cat bosh-openstack/bosh-openstack.yml
---
name: bosh-openstack
director_uuid: a11193e9-99ff-4f85-8dc0-8a06509aeed2 # CHANGE

release:
name: bosh
version: latest

compilation:
workers: 3
network: default
reuse_compilation_vms: true
cloud_properties:
instance_type: m1.small # CHANGE

update:
canaries: 1
canary_watch_time: 3000-120000
update_watch_time: 3000-120000
max_in_flight: 4

networks:
- name: floating
    type: vip
    cloud_properties: {}
- name: default
    type: dynamic
    cloud_properties: {}

resource_pools:
- name: common
    network: default
    size: 8
    stemcell:
      name: bosh-openstack-kvm-ubuntu
      version: latest
    cloud_properties:
      instance_type: m1.small # CHANGE

jobs:
- name: nats
    template: nats
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

- name: redis
    template: redis
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

- name: postgres
    template: postgres
    instances: 1
    resource_pool: common
    persistent_disk: 5120
    networks:
      - name: default
        default: [dns, gateway]

- name: powerdns
    template: powerdns
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
      - name: floating
        static_ips:
          - 137.172.74.83 # CHANGE

- name: blobstore
    template: blobstore
    instances: 1
    resource_pool: common
    persistent_disk: 5120
    networks:
      - name: default
        default: [dns, gateway]

- name: director
    template: director
    instances: 1
    resource_pool: common
    persistent_disk: 5120
    networks:
      - name: default
        default: [dns, gateway]
      - name: floating
        static_ips:
          - 137.172.74.84 # CHANGE

- name: registry
    template: registry
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

- name: health_monitor
    template: health_monitor
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

properties:
nats:
    address: 0.nats.default.bosh-openstack.microbosh
    user: nats
    password: nats

redis:
address: 0.redis.default.bosh-openstack.microbosh
password: redis

postgres: &bosh_db
    host: 0.postgres.default.bosh-openstack.microbosh
    user: postgres
    password: postgres
    database: bosh

dns:
    address: 137.172.74.83 # CHANGE
    db: *bosh_db
    recursor: 137.172.74.82 # CHANGE

blobstore:
    address: 0.blobstore.default.bosh-openstack.microbosh
    agent:
      user: agent
      password: agent
    director:
      user: director
      password: director

director:
    name: bosh
    address: 0.director.default.bosh-openstack.microbosh
    db: *bosh_db

registry:
    address: 0.registry.default.bosh-openstack.microbosh
    db: *bosh_db
    http:
      user: registry
      password: registry

hm:
    http:
      user: hm
      password: hm
    director_account:
      user: admin
      password: admin
    resurrector_enabled: true

ntp:
- 0.north-america.pool.ntp.org
- 1.north-america.pool.ntp.org

openstack:
    auth_url: http://137.172.74.130:5000/v2.0 # CHANGE
    username: demo # CHANGE
    api_key: <password> # CHANGE
    tenant: demo # CHANGE
    region: RegionOne # CHANGE
    default_security_groups: ["default", "ssh", "bosh", "cf-public", "cf-private"] # CHANGE
    default_key_name: microbosh # CHANGE
ubuntu@ubuntu12:~/bosh-workspace/deployments$

Adam Stegman

unread,

Jul 18, 2014, 3:24:52 PM7/18/14

to bosh-...@cloudfoundry.org

It looks like you built the bosh release yourself. How did you create it? It looks like your logs have two different version numbers, which is pretty strange. Can you show us the output of `ls /var/vcap/data/packages/director`?

Also, can you show us the output of `bosh vms` and `monit summary`? We'd like to see what exactly is not running and what is running. The debug log from /var/vcap/sys/log/director would also be helpful.

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

Guangcai Wang

unread,

Jul 21, 2014, 10:02:03 PM7/21/14

to bosh-...@cloudfoundry.org

I checked the modules in director VM. It seems really not to have the bosh-director module on director VM. Please check the details.

I also found there was no bosh-director on BOSH server. After I installed bosh-director via "gem install bosh-director", there is no error for the testing.

ubuntu@ubuntu12:~/bosh-workspace/deployments$ irb
irb(main):001:0> require 'bosh/director'
=> true
irb(main):002:0>

So, I am wondering if I need to install these modules on director VM from source code according the step in "Deploying BOSH with Micro BOSH" (http://docs.cloudfoundry.org/deploying/openstack/deploying_bosh.html).

because it will solve all the dependencies required by "gem install bosh-director"

vcap@790e0e30-b5ae-475e-8525-d27917ca3c2d:~$ gem install bosh-director

WARNING: Error fetching data: SocketError: getaddrinfo: Temporary failure in name resolution (http://rubygems.org/prerelease_specs.4.8.gz)
ERROR: While executing gem ... (Gem::DependencyError)
Unable to resolve dependencies: sys-filesystem requires ffi (>= 0); rufus-scheduler requires tzinfo (>= 0.3.22); resque requires redis-namespace (~> 1.3), vegas (~> 0.1.2), multi_json (~> 1.0); sinatra requires rack (~> 1.4), tilt (>= 1.3.4, ~> 1.3), rack-protection (~> 1.4); rack-test requires rack (>= 1.0); nats requires json_pure (>= 1.8.0), daemons (>= 1.1.9); thin requires rack (>= 1.0.0), daemons (>= 1.0.9); bosh_vcloud_cpi requires builder (~> 3.1.4), rest-client (~> 1.6.7), nokogiri (~> 1.5.6); bosh_warden_cpi requires warden-protocol (>= 0), warden-client (>= 0); bosh_vsphere_cpi requires pg (~> 0.15.1), mysql2 (~> 0.3.11), builder (~> 3.1.4), nokogiri (~> 1.5.10); bosh_aws_cpi requires bosh-registry (~> 1.2652.0); bosh_openstack_cpi requires bosh-registry (~> 1.2652.0); bosh-core requires gibberish (~> 1.2.0); blobstore_client requires multi_json (~> 1.1), ruby-atmos-pure (~> 1.0.5); fog requires builder (>= 0), excon (~> 0.25.0), formatador (~> 0.2.0), multi_json (~> 1.0), mime-types (>= 0), net-scp (~> 1.1), net-ssh (>= 2.1.3), nokogiri (~> 1.5), ruby-hmac (>= 0); aws-sdk requires nokogiri (>= 1.4.4)

>>>>on BOSH server:

ubuntu@ubuntu12:~/bosh-workspace/deployments$ bosh releases

+------+-----------+-------------+
| Name | Versions | Commit Hash |
+------+-----------+-------------+

| bosh | 91+dev.4* | 00000000 |
+------+-----------+-------------+
(*) Currently deployed

Releases total: 1
ubuntu@ubuntu12:~/bosh-workspace/deployments$

ubuntu@ubuntu12:~/bosh-workspace/deployments$ bosh deployments

Deployments total: 1
ubuntu@ubuntu12:~/bosh-workspace/deployments$

ubuntu@ubuntu12:~/bosh-workspace/deployments$ bosh vms
Deployment `bosh-openstack'

Director task 44

Task 44 done

+------------------+---------+---------------+---------------+
| Job/index | State | Resource Pool | IPs |
+------------------+---------+---------------+---------------+

VMs total: 8

>>> on director server:

root@fbf850d4-6462-47f4-8ffd-0512f338c977:~# tail -f /var/vcap/sys/log/director/scheduler.stderr.log

from /var/vcap/packages/director/bin/bosh-director-scheduler:16:in `load'
from /var/vcap/packages/director/bin/bosh-director-scheduler:16:in `<main>'

/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.0000.0/bin/bosh-director-scheduler:3:in `require': cannot load such file -- bosh/director (LoadError)

from /var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.0000.0/bin/bosh-director-scheduler:3:in `<top (required)>'
from /var/vcap/packages/director/bin/bosh-director-scheduler:16:in `load'
from /var/vcap/packages/director/bin/bosh-director-scheduler:16:in `<main>'

/var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.0000.0/bin/bosh-director-scheduler:3:in `require': cannot load such file -- bosh/director (LoadError)

root@fbf850d4-6462-47f4-8ffd-0512f338c977:~# ruby -v
ruby 1.9.3p545 (2014-02-24 revision 45159) [x86_64-linux]
root@fbf850d4-6462-47f4-8ffd-0512f338c977:~#

root@fbf850d4-6462-47f4-8ffd-0512f338c977:~# gem -v
1.8.24
root@fbf850d4-6462-47f4-8ffd-0512f338c977:~#

root@fbf850d4-6462-47f4-8ffd-0512f338c977:~# cat /var/vcap/packages/director/gem_home/ruby/1.9.1/gems/bosh-director-1.0000.0/bin/bosh-director-scheduler
#!/usr/bin/env ruby

require 'bosh/director'
require 'bosh/director/scheduler'

config_file = ::File.expand_path('../../config/bosh-director.yml', __FILE__)

opts = OptionParser.new do |op|
op.on('-c', '--config [ARG]', 'Configuration File') do |opt|
config_file = opt
end
end

root@fbf850d4-6462-47f4-8ffd-0512f338c977:/var/vcap/data/sys/log/director# irb
irb(main):001:0> require 'bosh/director'
LoadError: cannot load such file -- bosh/director
from /var/vcap/bosh/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /var/vcap/bosh/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from (irb):1
from /var/vcap/bosh/bin/irb:12:in `<main>'
irb(main):002:0>

Reply all

Reply to author

Forward

0 new messages