Some units not started when bootstrapping with cloud-init

[Holger Amann]

Hi,

I'm trying to setup dc/os on AWS with terraform and coreos (766.5.0). For that I created a cloud-config which should create and start following tasks/units:

1. Apply additional ssh keys (update-additional-ssh-keys.service)

2. Create a docker.tar.gz (docker-tarball.service)

3. Start consul in docker container (consul.service)

4. Download dcos (dcos-download.service)

5. Setup dcos (dcos-setup.service)

6. Do something with /etc/profile (dcos-link-env.service)

I started with dc/os stuff (4-6) and everything worked fine so far when I created an ec2 instance.

Then I added additional units (1-3) and they are working when bootstrapping, but the dc/os stuff won't get started until I do a reboot.

Can someone please help me to get every step running when bootstrapping a new instance?

This is my cloud config

#cloud-config

"coreos":

  "units":

  - "command": |-

      start

    "content": |

      [Unit]

      Description=AWS Setup: Formats the /var/lib ephemeral drive

      Before=var-lib.mount dbus.service

      [Service]

      Type=oneshot

      RemainAfterExit=yes

      ExecStart=/bin/bash -c "(blkid -t TYPE=ext4 | grep xvdb) || (/usr/sbin/mkfs.ext4 -F /dev/xvdb)"

    "name": |-

      format-var-lib-ephemeral.service

  - "command": |-

      start

    "content": |

      [Unit]

      Description=AWS Setup: Mount /var/lib

      Before=dbus.service

      [Mount]

      What=/dev/xvdb

      Where=/var/lib

      Type=ext4

    "name": |-

      var-lib.mount

  - "command": |-

      stop

    "mask": !!bool |-

      true

    "name": |-

      etcd.service

  - "command": |-

      stop

    "mask": !!bool |-

      true

    "name": |-

      update-engine.service

  - "command": |-

      stop

    "mask": !!bool |-

      true

    "name": |-

      locksmithd.service

  - "command": |-

      stop

    "name": |-

      systemd-resolved.service

  - "command": |-

      restart

    "name": |-

      systemd-journald.service

  - "command": |-

      restart

    "name": |-

      docker.service

  - "command": |-

      start

    "content": |

      [Unit]

      Description=Update additional ssh keys

      [Service]

      Type=oneshot

      ExecStart=/usr/bin/update-ssh-keys

      StandardOutput=journal+console

      StandardError=journal+console

      User=core

    "enable": !!bool |-

      true

    "name": |-

      update-additional-ssh-keys.service

  - "command": |-

      start

    "content": |

      [Unit]

      Description=Create docker.tar.gz for marathon

      [Service]

      Type=oneshot

      ExecStart=/usr/bin/tar -czf /home/core/docker.tar.gz -C /home/core .docker

      StandardOutput=journal+console

      StandardError=journal+console

      User=core

    "enable": !!bool |-

      true

    "name": |-

      docker-tarball.service

  - "command": |-

      start

    "content": |

      [Unit]

      Description=Consul Agent

      Requires=docker.service

      After=docker.service

      [Service]

      Restart=always

      ExecStartPre=/usr/bin/mkdir -p /media/data/consul

      ExecStartPre=/usr/bin/chown 100 /media/data/consul

      ExecStart=/usr/bin/docker run --name=consul --net=host -v /media/data/consul:/consul/data:rw -e "CONSUL_LOCAL_CONFIG={\"leave_on_terminate\": true}" -e "CONSUL_BIND_INTERFACE=eth0" -e "CONSUL_CLIENT_INTERFACE=eth0" -e "ATLAS_TOKEN=${atlas_token}" consul:v0.6.4 agent -atlas ${atlas_env} -atlas-join

      ExecStop=/usr/bin/docker stop -t 2 consul ; /usr/bin/docker rm -f consul

    "enable": !!bool |-

      true

    "name": |-

      consul.service

  - "command": |-

      start

    "content": |

      [Unit]

      Before=dcos.target

      [Service]

      Type=oneshot

      StandardOutput=journal+console

      StandardError=journal+console

      ExecStartPre=/usr/bin/mkdir -p /etc/profile.d

      ExecStart=/usr/bin/ln -sf /opt/mesosphere/environment.export /etc/profile.d/dcos.sh

    "name": |-

      dcos-link-env.service

  - "content": |

      [Unit]

      Description=Pkgpanda: Download DC/OS to this host.

      After=network-online.target

      Wants=network-online.target

      ConditionPathExists=!/opt/mesosphere/

      [Service]

      EnvironmentFile=/etc/mesosphere/setup-flags/bootstrap-id

      Type=oneshot

      StandardOutput=journal+console

      StandardError=journal+console

      ExecStartPre=/usr/bin/curl -fLsSv --retry 20 -Y 100000 -y 60 -o /tmp/bootstrap.tar.xz https://downloads.dcos.io/dcos/EarlyAccess/bootstrap/$${BOOTSTRAP_ID}.bootstrap.tar.xz

      ExecStartPre=/usr/bin/mkdir -p /opt/mesosphere

      ExecStart=/usr/bin/tar -axf /tmp/bootstrap.tar.xz -C /opt/mesosphere

      ExecStartPost=-/usr/bin/rm -f /tmp/bootstrap.tar.xz

    "name": |-

      dcos-download.service

  - "command": |-

      start

    "content": |

      [Unit]

      Description=Pkgpanda: Specialize DC/OS for this host.

      Requires=dcos-download.service

      After=dcos-download.service

      [Service]

      Type=oneshot

      StandardOutput=journal+console

      StandardError=journal+console

      EnvironmentFile=/opt/mesosphere/environment

      ExecStart=/opt/mesosphere/bin/pkgpanda setup --no-block-systemd

      [Install]

      WantedBy=multi-user.target

    "enable": !!bool |-

      true

    "name": |-

      dcos-setup.service

  "update":

    "reboot-strategy": |-

      off

"write_files":

- "content": |

    https://downloads.dcos.io/dcos/EarlyAccess

  "owner": |-

    root

  "path": |-

    /etc/mesosphere/setup-flags/repository-url

  "permissions": |-

    0644

- "content": |

    BOOTSTRAP_ID=${bootstrap_id}

  "owner": |-

    root

  "path": |-

    /etc/mesosphere/setup-flags/bootstrap-id

  "permissions": |-

    0644

- "content": |

${cluster_packages}

  "owner": |-

    root

  "path": |-

    /etc/mesosphere/setup-flags/cluster-packages.json

  "permissions": |-

    0644

- "content": |

    [Journal]

    MaxLevelConsole=warning

  "owner": |-

    root

  "path": |-

    /etc/systemd/journald.conf.d/dcos.conf

  "permissions": |-

    0644

- "content": |

    MESOS_CLUSTER=${stack_name}

  "path": |-

    /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/mesos-master-provider

- "content": |

    ADMINROUTER_ACTIVATE_AUTH_MODULE=${authentication_enabled}

  "path": |-

    /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/adminrouter.env

- "content": |

    com.netflix.exhibitor.s3.access-key-id=${aws_access_key_id}

    com.netflix.exhibitor.s3.access-secret-key=${aws_secret_access_key}

  "path": |-

    /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/exhibitor.properties

- "content": |

    MASTER_SOURCE=exhibitor

    EXHIBITOR_ADDRESS=${internal_master_lb_dns_name}

    RESOLVERS=${fallback_dns}

  "path": |-

    /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/dns_config

- "content": |

    EXHIBITOR_BACKEND=AWS_S3

    AWS_REGION=${aws_region}

    AWS_S3_BUCKET=${exhibitor_s3_bucket}

    AWS_S3_PREFIX=${stack_name}

  "path": |-

    /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/exhibitor

- "content": |

    {"uiConfiguration":{"plugins":{"banner":{"enabled":false,"backgroundColor":"#1E232F","foregroundColor":"#FFFFFF","headerTitle":null,"headerContent":null,"footerContent":null,"imagePath":null,"dismissible":null},"branding":{"enabled":false},"external-links": {"enabled": false},

    "authentication":{"enabled":false},

    "oauth":{"enabled":${authentication_enabled},"authHost":"https://dcos.auth0.com"},

    "networking":{"enabled":false},"organization":{"enabled":false},"tracking":{"enabled":true}}}}

  "path": |-

    /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/ui-config.json

- "content": |-

    {}

  "path": |-

    /etc/mesosphere/setup-packages/dcos-provider-aws--setup/pkginfo.json

- "content": ""

  "path": |-

    /etc/mesosphere/roles/slave

- "content": ""

  "path": |-

    /etc/mesosphere/roles/aws

- "content": |

    ${additional_ssh_public_keys}

  "path": |-

    /home/core/.ssh/authorized_keys.d/additional

- "content": |

    {"auths": {"https://index.docker.io/v1/": {"auth": "${docker_registry_token}"}}}

  "path": |-

    /home/core/.docker/config.json

systemctl status

core@ip-10-0-0-234 ~ $ systemctl status update-additional-ssh-keys

● update-additional-ssh-keys.service - Update additional ssh keys

   Loaded: loaded (/etc/systemd/system/update-additional-ssh-keys.service; static; vendor preset: disabled)

   Active: inactive (dead)

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal systemd[1]: Starting Update additional ssh keys...

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal update-ssh-keys[750]: Updated /home/core/.ssh/authorized_keys

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal systemd[1]: Started Update additional ssh keys.

core@ip-10-0-0-234 ~ $ systemctl status docker-tarball

● docker-tarball.service - Create docker.tar.gz for marathon

   Loaded: loaded (/etc/systemd/system/docker-tarball.service; static; vendor preset: disabled)

   Active: inactive (dead)

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal systemd[1]: Starting Create docker.tar.gz for marathon...

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal systemd[1]: Started Create docker.tar.gz for marathon.

core@ip-10-0-0-234 ~ $ systemctl status consul

● consul.service - Consul Agent

   Loaded: loaded (/etc/systemd/system/consul.service; static; vendor preset: disabled)

   Active: active (running) since Fri 2016-06-10 08:50:18 UTC; 21min ago

  Process: 768 ExecStartPre=/usr/bin/chown 100 /media/data/consul (code=exited, status=0/SUCCESS)

  Process: 766 ExecStartPre=/usr/bin/mkdir -p /media/data/consul (code=exited, status=0/SUCCESS)

 Main PID: 772 (docker)

   Memory: 2.0M

      CPU: 16ms

   CGroup: /system.slice/consul.service

           └─772 /usr/bin/docker run --name=consul --net=host -v /media/data/consul:/consul/data:rw -e CONSUL_LOCAL_CONFIG={"leave_on_terminate": true} -e CONSUL_BIND_INTERFACE=eth0 -e CONSUL_CLIENT_INTERFAC...

Jun 10 09:09:08 ip-10-0-0-234.us-west-2.compute.internal docker[772]: 2016/06/10 09:09:08 [ERR] agent: failed to sync remote state: No known Consul servers

Jun 10 09:09:24 ip-10-0-0-234.us-west-2.compute.internal docker[772]: 2016/06/10 09:09:24 [ERR] agent: failed to sync remote state: No known Consul servers

core@ip-10-0-0-234 ~ $ systemctl status dcos-setup

● dcos-setup.service - Pkgpanda: Specialize DC/OS for this host.

   Loaded: loaded (/etc/systemd/system/dcos-setup.service; enabled; vendor preset: disabled)

   Active: inactive (dead)

core@ip-10-0-0-234 ~ $ systemctl status dcos-download

● dcos-download.service - Pkgpanda: Download DC/OS to this host.

   Loaded: loaded (/etc/systemd/system/dcos-download.service; static; vendor preset: disabled)

   Active: inactive (dead)

core@ip-10-0-0-234 ~ $ systemctl status dcos-link-env

● dcos-link-env.service

   Loaded: loaded (/etc/systemd/system/dcos-link-env.service; static; vendor preset: disabled)

   Active: inactive (dead)

Logs:

core@ip-10-0-0-234 ~ $ journalctl _EXE=/usr/bin/coreos-cloudinit --no-pager

-- Logs begin at Fri 2016-06-10 08:49:46 UTC, end at Fri 2016-06-10 09:07:56 UTC. --

Jun 10 08:50:16 localhost coreos-cloudinit[623]: Checking availability of "local-file"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: Fetching user-data from datasource of type "local-file"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: Fetching meta-data from datasource of type "local-file"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Parsing user-data as cloud-config

Jun 10 08:50:16 localhost coreos-cloudinit[623]: Merging cloud-config from meta-data and user-data

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Writing file to "/etc/oem-release"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Wrote file to "/etc/oem-release"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Wrote file /etc/oem-release to filesystem

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Writing drop-in unit "10-oem.conf" to filesystem

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Writing file to "/run/systemd/system/etcd.service.d/10-oem.conf"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Wrote file to "/run/systemd/system/etcd.service.d/10-oem.conf"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Wrote drop-in unit "10-oem.conf"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Ensuring runtime unit file "etcd.service" is unmasked

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Writing drop-in unit "10-oem.conf" to filesystem

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Writing file to "/run/systemd/system/etcd2.service.d/10-oem.conf"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Wrote file to "/run/systemd/system/etcd2.service.d/10-oem.conf"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Wrote drop-in unit "10-oem.conf"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Ensuring runtime unit file "etcd2.service" is unmasked

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Masking unit file "user-configdrive.service"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Masking unit file "user-configvirtfs.service"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Writing unit "oem-cloudinit.service" to filesystem

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Writing file to "/run/systemd/system/oem-cloudinit.service"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Wrote file to "/run/systemd/system/oem-cloudinit.service"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Wrote unit "oem-cloudinit.service"

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Ensuring runtime unit file "oem-cloudinit.service" is unmasked

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 /run/systemd/system/oem-cloudinit.service is not null or empty, refusing to unmask

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Ensuring runtime unit file "etcd.service" is unmasked

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Ensuring runtime unit file "etcd2.service" is unmasked

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Ensuring runtime unit file "fleet.service" is unmasked

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Ensuring runtime unit file "locksmithd.service" is unmasked

Jun 10 08:50:16 localhost coreos-cloudinit[623]: 2016/06/10 08:50:16 Calling unit command "restart" on "oem-cloudinit.service"'

Jun 10 08:50:16 localhost coreos-cloudinit[675]: Checking availability of "cloud-drive"

Jun 10 08:50:16 localhost coreos-cloudinit[675]: Checking availability of "ec2-metadata-service"

Jun 10 08:50:16 localhost coreos-cloudinit[675]: Fetching user-data from datasource of type "ec2-metadata-service"

Jun 10 08:50:16 localhost coreos-cloudinit[675]: 2016/06/10 08:50:16 Fetching data from http://169.254.169.254/2009-04-04/user-data. Attempt #1

Jun 10 08:50:16 localhost coreos-cloudinit[675]: Fetching meta-data from datasource of type "ec2-metadata-service"

Jun 10 08:50:16 localhost coreos-cloudinit[675]: 2016/06/10 08:50:16 Fetching data from http://169.254.169.254/2009-04-04/meta-data/public-keys. Attempt #1

Jun 10 08:50:16 localhost coreos-cloudinit[675]: 2016/06/10 08:50:16 Fetching data from http://169.254.169.254/2009-04-04/meta-data/public-keys/0/openssh-key. Attempt #1

Jun 10 08:50:16 localhost coreos-cloudinit[675]: Found SSH key for "dcos-main"

Jun 10 08:50:16 localhost coreos-cloudinit[675]: 2016/06/10 08:50:16 Fetching data from http://169.254.169.254/2009-04-04/meta-data/hostname. Attempt #1

Jun 10 08:50:16 localhost coreos-cloudinit[675]: 2016/06/10 08:50:16 Fetching data from http://169.254.169.254/2009-04-04/meta-data/local-ipv4. Attempt #1

Jun 10 08:50:16 localhost coreos-cloudinit[675]: 2016/06/10 08:50:16 Fetching data from http://169.254.169.254/2009-04-04/meta-data/public-ipv4. Attempt #1

Jun 10 08:50:16 localhost coreos-cloudinit[675]: 2016/06/10 08:50:16 Parsing user-data as cloud-config

Jun 10 08:50:16 localhost coreos-cloudinit[675]: Merging cloud-config from meta-data and user-data

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Set hostname to ip-10-0-0-234.us-west-2.compute.internal

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Authorized SSH keys for core user

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-flags/repository-url"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-flags/repository-url"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-flags/repository-url to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-flags/bootstrap-id"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-flags/bootstrap-id"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-flags/bootstrap-id to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-flags/cluster-packages.json"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-flags/cluster-packages.json"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-flags/cluster-packages.json to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/journald.conf.d/dcos.conf"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/journald.conf.d/dcos.conf"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/systemd/journald.conf.d/dcos.conf to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/mesos-master-provider"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/mesos-master-provider"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/mesos-master-provider to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/adminrouter.env"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/adminrouter.env"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/adminrouter.env to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/exhibitor.properties"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/exhibitor.properties"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/exhibitor.properties to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/dns_config"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/dns_config"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/dns_config to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/exhibitor"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/exhibitor"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/exhibitor to filesystem

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/ui-config.json"

Jun 10 08:50:16 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/ui-config.json"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-packages/dcos-provider-aws--setup/etc/ui-config.json to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/pkginfo.json"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/setup-packages/dcos-provider-aws--setup/pkginfo.json"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/setup-packages/dcos-provider-aws--setup/pkginfo.json to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/roles/slave"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/roles/slave"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/roles/slave to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/mesosphere/roles/aws"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/mesosphere/roles/aws"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/mesosphere/roles/aws to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/home/core/.ssh/authorized_keys.d/additional"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/home/core/.ssh/authorized_keys.d/additional"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /home/core/.ssh/authorized_keys.d/additional to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/home/core/.docker/config.json"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/home/core/.docker/config.json"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /home/core/.docker/config.json to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/coreos/update.conf"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/coreos/update.conf"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file /etc/coreos/update.conf to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/environment"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/environment"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Updated /etc/environment

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing unit "format-var-lib-ephemeral.service" to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/system/format-var-lib-ephemeral.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/system/format-var-lib-ephemeral.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote unit "format-var-lib-ephemeral.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing unit "var-lib.mount" to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/system/var-lib.mount"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/system/var-lib.mount"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote unit "var-lib.mount"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Masking unit file "etcd.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Masking unit file "update-engine.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Masking unit file "locksmithd.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing unit "update-additional-ssh-keys.service" to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/system/update-additional-ssh-keys.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/system/update-additional-ssh-keys.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote unit "update-additional-ssh-keys.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Enabling unit file "update-additional-ssh-keys.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Enabled unit "update-additional-ssh-keys.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing unit "docker-tarball.service" to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/system/docker-tarball.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/system/docker-tarball.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote unit "docker-tarball.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Enabling unit file "docker-tarball.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Enabled unit "docker-tarball.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing unit "consul.service" to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/system/consul.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/system/consul.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote unit "consul.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Enabling unit file "consul.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Enabled unit "consul.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing unit "dcos-link-env.service" to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/system/dcos-link-env.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/system/dcos-link-env.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote unit "dcos-link-env.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing unit "dcos-download.service" to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/system/dcos-download.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/system/dcos-download.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote unit "dcos-download.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing unit "dcos-setup.service" to filesystem

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Writing file to "/etc/systemd/system/dcos-setup.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote file to "/etc/systemd/system/dcos-setup.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Wrote unit "dcos-setup.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Enabling unit file "dcos-setup.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Enabled unit "dcos-setup.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Ensuring runtime unit file "etcd.service" is unmasked

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Ensuring runtime unit file "etcd2.service" is unmasked

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Ensuring runtime unit file "fleet.service" is unmasked

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Ensuring runtime unit file "locksmithd.service" is unmasked

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Masking unit file "locksmithd.service"

Jun 10 08:50:17 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:16 Calling unit command "start" on "format-var-lib-ephemeral.service"'

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Result of "start" on "format-var-lib-ephemeral.service": done

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Calling unit command "start" on "var-lib.mount"'

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Result of "start" on "var-lib.mount": done

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Calling unit command "stop" on "etcd.service"'

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Result of "stop" on "etcd.service": done

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Calling unit command "stop" on "update-engine.service"'

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Result of "stop" on "update-engine.service": done

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Calling unit command "stop" on "locksmithd.service"'

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Result of "stop" on "locksmithd.service": done

Jun 10 08:50:18 ip-10-0-0-234.us-west-2.compute.internal coreos-cloudinit[675]: 2016/06/10 08:50:18 Calling unit command "stop" on "systemd-resolved.service"'

[will.g...@coreos.com]

Holger,

I'm working to recreate your issue. Could you post the original cloud-config where dc/os is working on first run?

Thanks!

Will