Deploy bosh with microbosh in openstack: instances error out

89 views
Skip to first unread message

Tom Sherrod

unread,
Aug 21, 2014, 1:48:08 PM8/21/14
to bosh-...@cloudfoundry.org
Problem: Following http://docs.cloudfoundry.org/deploying/openstack/deploying_bosh.html  The "bosh deploy" errors out. From the "bosh task # --debug" ERROR state is error, expected active. Watching instances in openstack, I see the status go to error. bosh cleans up and then errors out. 

Error 100: Bosh::Clouds::VMCreationFailed

The image bosh is trying to boot, boots fine manually.

Any pointers most appreciated.

Thanks,
Tom

Information:

- bosh gem versions:
bosh-director-core (1.2690.0)
bosh-registry (1.2690.0)
bosh-stemcell (1.2690.0)
bosh-template (1.2690.0)
bosh_aws_cpi (1.2690.0)
bosh_cli (1.2690.0)
bosh_cli_plugin_micro (1.2690.0)
bosh_common (1.2690.0)
bosh_cpi (1.2690.0)
bosh_openstack_cpi (1.2690.0)
bosh_vcloud_cpi (0.5.5)
bosh_vsphere_cpi (1.2690.0)

- bosh director info:

Config
             /home/ubuntu/.bosh_config
Director
  Name       openstack-microbosh
  URL        https://10.124.11.22:25555
  Version    1.2657.0 (00000000)

  User       admin
  UUID       d6e4bde6-751d-4554-b8ad-e2162e395e04
  CPI        openstack
  dns        enabled (domain_name: microbosh)
  compiled_package_cache disabled
  snapshots  disabled
Deployment
  Manifest   /home/ubuntu/bosh-workspace/deployments/bosh-openstack/bosh-openstack.yml

- stemcell:

bosh-stemcell-2657-openstack-kvm-centos-go_agent.tgz
http://bosh-jenkins-artifacts.s3.amazonaws.com/bosh-stemcell/openstack/bosh-stemcell-latest-openstack-kvm-ubuntu.tgz

- the release:
Whatever version is from the documentation above. Don't think I've gotten there yet.

- deployment manifest:

---

name: bosh-openstack

director_uuid: d6e4bde6-751d-4554-b8ad-e2162e395e04 # CHANGE

release:
  name: bosh
  version: latest

compilation:
  workers: 3
  network: default
  reuse_compilation_vms: true
  cloud_properties:
    instance_type: Medium # CHANGE

update:
  canaries: 1
  canary_watch_time: 3000-120000
  update_watch_time: 3000-120000
  max_in_flight: 4

networks:
  - name: floating
    type: vip
    cloud_properties: {}
  - name: default
    type: manual
    subnets:
      - name: private
        range: 192.168.1.0/24 #<subnet_cidr> # CHANGE
        gateway: 192.168.1.1 #<subnet_gateway> # CHANGE
        reserved:
          - 192.168.1.2 - 192.168.1.25 #<first_reserved_ip_address> - <last_reserved_ip_address> # CHANGE
        static:
          - 192.168.1.26 - 192.168.1.50 #<first_ip_address_for_vms> - <last_ip_address_for_vms> # CHANGE
        cloud_properties:
          net_id: ced682c3-c212-4b27-91c2-a60e757cbcab #<network_uuid> # CHANGE

resource_pools:
  - name: common
    network: default
    size: 8
    stemcell:
      name: bosh-openstack-kvm-ubuntu
      version: latest
    cloud_properties:
      instance_type: Medium # CHANGE

jobs:
  - name: nats
    template: nats
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
        static_ips:
          - 192.168.1.26 #<ip_address_for_nats> # CHANGE

  - name: redis
    template: redis
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
        static_ips:
          - 192.168.1.27 #<ip_address_for_redis> # CHANGE

  - name: postgres
    template: postgres
    instances: 1
    resource_pool: common
    persistent_disk: 16384
    networks:
      - name: default
        default: [dns, gateway]
        static_ips:
          - 192.168.1.28 #<ip_address_for_postgres> # CHANGE

  - name: powerdns
    template: powerdns
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
        static_ips:
          - 192.168.1.29 #<ip_address_for_powerdns> # CHANGE

  - name: blobstore
    template: blobstore
    instances: 1
    resource_pool: common
    #persistent_disk: 51200
    networks:
      - name: default
        default: [dns, gateway]
        static_ips:
          - 192.168.1.30 #<ip_address_for_blobstore> # CHANGE

  - name: director
    template: director
    instances: 1
    resource_pool: common
    persistent_disk: 16384
    networks:
      - name: default
        default: [dns, gateway]
        static_ips:
          - 192.168.1.31 #<ip_address_for_director> # CHANGE
      - name: floating
        static_ips:
          - 10.124.11.23 #<allocated_floating_ip_1> # CHANGE

  - name: registry
    template: registry
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
        static_ips:
          - 192.168.1.32 #<ip_address_for_registry> # CHANGE

  - name: health_monitor
    template: health_monitor
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
        static_ips:
          - 192.168.1.33 #<ip_address_for_health_monitor> # CHANGE

properties:
  nats:
    address: 192.168.1.26 #<ip_address_for_nats> # CHANGE
    user: nats
    password: nats

  redis:
    address: 192.168.1.27 #<ip_address_for_redis> # CHANGE
    password: redis

  postgres: &bosh_db
    host: 192.168.1.28 #<ip_address_for_postgres> # CHANGE
    user: postgres
    password: postgres
    database: bosh

  dns:
    address: 192.168.1.29 #<ip_address_for_powerdns> # CHANGE
    db: *bosh_db
    recursor: 192.168.1.5 # Should this be floating? <microbosh_ip_address> # CHANGE

  blobstore:
    address: 192.168.1.30 #<ip_address_for_blobstore> # CHANGE
    agent:
      user: agent
      password: agent
    director:
      user: director
      password: director

  director:
    name: bosh
    address: 192.168.1.31 #<ip_address_for_director> # CHANGE
    db: *bosh_db

  registry:
    address: 192.168.1.32 #<ip_address_for_registry> # CHANGE
    db: *bosh_db
    http:
      user: registry
      password: registry

  hm:
    http:
      user: hm
      password: hm
    director_account:
      user: admin
      password: admin
    resurrector_enabled: true

  ntp:
    - 0.north-america.pool.ntp.org
    - 1.north-america.pool.ntp.org

openstack:
    auth_url: #http://<identity_server>:5000/v2.0 # CHANGE
    username: #<username> # CHANGE
    api_key: #<password> # CHANGE
    tenant: #<tenant> # CHANGE
    region: #<region> # CHANGE
    default_security_groups: ["default", "bosh" ] #<microbosh_security_group>] # CHANGE
    default_key_name: microbosh #<microbosh_keypair> # CHANGE

Dmitriy Kalinin

unread,
Aug 21, 2014, 2:14:20 PM8/21/14
to bosh-...@cloudfoundry.org
Does your `bosh deploy` create multiple VMs at the same time? It's possible that rate-limiting is preventing VM creation. Try re-running bosh deploy to see if it creates more VMs successfully.

Also latest bosh release 2693 should have `bosh task X --cpi` command working with OpenStack CPI. It might contain more details about the error.

Dmitriy Kalinin

unread,
Aug 21, 2014, 3:10:35 PM8/21/14
to bosh-...@cloudfoundry.org
Just realized we did not publish OpenStack microbosh stemcell for 2693 so you cannot use --cpi then.

Tom Sherrod

unread,
Aug 21, 2014, 3:30:11 PM8/21/14
to bosh-...@cloudfoundry.org
Thanks...just double checked before responding and wondered what I was missing regarding version.

Ran the deploy twice more.
Watching horizon I see 3 instances fire up, tasks and get ip, go error, deleting and 2 instances fire up and then 1 more...all error and then are deleted. I'm missing something obvious here.

I check all the quota limits and don't see how any those would interfere.

What are the recommended sizes for these vms?

Tom


To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

Dmitriy Kalinin

unread,
Aug 21, 2014, 5:18:13 PM8/21/14
to bosh-...@cloudfoundry.org
Any size should work. Usually sizes depend on specific jobs.

Note: currently if you do not have an ephemeral disk for your flavor, BOSH will not expand root partition to possible full size. This feature is being currently worked on.

Tom Sherrod

unread,
Aug 24, 2014, 1:56:40 PM8/24/14
to bosh-...@cloudfoundry.org
I now see the problem: 
NovaException: Error mounting /var/lib/nova/instances/0d813980-94ed-440d-b1c5-8d4f8b9e6a59/disk with libguestfs (could not parse /etc/fstab or empty file)


Any other input welcome! Can something else be done and avoid the nova.conf change?
I only see this from cf posts with Grizzly...appears to still exist in Havana, Openstack.

Thanks,
Tom

Guangcai Wang

unread,
Aug 24, 2014, 8:34:44 PM8/24/14
to bosh-...@cloudfoundry.org
I did as the bug solution: change libvirt_inject_partition = 1 in nova.conf and then restart nova-compute service.
 
It works after the nova.conf change.


To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

Tom Sherrod

unread,
Aug 25, 2014, 12:11:21 PM8/25/14
to bosh-...@cloudfoundry.org
We changed the nova.conf and it broke the ability to start other instances.

Next up, try a basic fstab and see if we can get by this.

Tom

Tom Sherrod

unread,
Aug 25, 2014, 4:22:59 PM8/25/14
to bosh-...@cloudfoundry.org
I got past this issue. 

I used a modified command to upload a newer version, replace latest with 2657-openstack-kvm-centos-go_agent.tgz.

I now have a full bosh deployed. Now on to the next steps.
Reply all
Reply to author
Forward
0 new messages