OpenStack config drive: CoreOS image not applying user_data

[Gavin McDougall]

Hi All,

We run an OpenStack Grizzly environment that makes use of a config_drive presented to the instance as /dev/vdb for MetaData.

This setup works perfectly for us with all of the major OS's including Windows.

When we deploy the CoreOS OpenStack image, whether the beta or alpha version, the CoreOS instance does 'see' and mount the /dev/vdb config drive to /media/configdrive, however it appears that CoreOS does not apply the user_data presented to it.

So for example, the SSH key is not copied to /home/core/.ssh/authorized_keys, the hostname is not applied and so on, and as such, we cannot SSH to our newly created instances.

From another thread I read how to intercept the kernel boot to force the CoreOS to autologin, so I have been able to observe the following:-

# df -h

Filesystem      Size  Used Avail Use% Mounted on

rootfs           28G  135M   28G   1% /

devtmpfs        237M     0  237M   0% /dev

tmpfs           248M     0  248M   0% /dev/shm

tmpfs           248M  244K  247M   1% /run

tmpfs           248M     0  248M   0% /sys/fs/cgroup

/dev/vda9        28G  135M   28G   1% /

/dev/vda3      1008M  288M  670M  31% /usr

tmpfs           248M     0  248M   0% /media

tmpfs           248M     0  248M   0% /tmp

/dev/vda6       108M   76K   99M   1% /usr/share/oem

/dev/vdb        410K  410K     0 100% /media/configdrive

# ls /media/configdrive/

ec2  openstack

# cat /media/configdrive/openstack/latest/user_data 

#cloud-config

coreos:

  etcd:

    # generate a new token for each unique cluster from https://discovery.etcd.io/new

    discovery: https://discovery.etcd.io/73f5c235b687796047b0cba69c9d1afd

    # multi-region and multi-cloud deployments need to use $public_ipv4

    addr: $private_ipv4:4001

    peer-addr: $private_ipv4:7001

  units:

    - name: etcd.service

      command: start

    - name: fleet.service

      command: start

ssh_authorized_keys:

  # include one or more SSH public keys

  ssh-rsa AAAAB3N........

# ls /home/core/.ssh/

# cat /home/core/.ssh/authorized_keys

cat: /home/core/.ssh/authorized_keys: No such file or directory

Is this something that I am doing wrong, is there anything else that I need to be doing ?

This is a production cloud and we are really keen to start offering our clients the ability to deploy CoreOS on top of it.

Thanks in advance.

Gavin

[Jonathan Boulle]

Can you provide the output of `systemctl status ec2-cloudinit` ?

Thanks

[Brian Harrington]

Gavin,

Can you try putting a " - " in front of your SSH key?  While syntactically this is valid YAML with the line break the rendering (on our end) might be off.

As processed by yamllint.com:

---
coreos:
  etcd:
    addr: "$private_ipv4:4001"
    discovery: "https://discovery.etcd.io/73f5c235b687796047b0cba69c9d1afd"

    peer-addr: "$private_ipv4:7001"
  units:
    -

      command: start
      name: etcd.service
    -
      command: start
      name: fleet.service
ssh_authorized_keys: "ssh-rsa AAAAB3N........"

[Gavin]

Hi Jonathan,

Thanks for your reply, here is the output of that command:-

# systemctl status ec2-cloudinit

● ec2-cloudinit.service - Cloudinit from EC2-style metadata

   Loaded: loaded (/run/systemd/system/ec2-cloudinit.service; static)

   Active: inactive (dead)

Is this the expected output ?

Thanks,

Gavin

[Gavin]

Hi Brian,

Thanks for the information, I updated my cloud-config yaml to include the '-' and parsed it with yamllint.com and it was confirmed to be valid yaml.

My user_data is still not being applied though, see the output from the following commands:-

# systemctl status ec2-cloudinit

● ec2-cloudinit.service - Cloudinit from EC2-style metadata

   Loaded: loaded (/run/systemd/system/ec2-cloudinit.service; static)

   Active: inactive (dead)

# df -h | grep config

/dev/vdb        410K  410K     0 100% /media/configdrive

# cat /media/configdrive/openstack/latest/user_data 

#cloud-config

coreos:

  etcd:

    # generate a new token for each unique cluster from https://discovery.etcd.io/new

    discovery: https://discovery.etcd.io/de709972994c8c38980d84b2e5fe78bd

    # multi-region and multi-cloud deployments need to use $public_ipv4

    addr: $private_ipv4:4001

    peer-addr: $private_ipv4:7001

  units:

    - name: etcd.service

      command: start

    - name: fleet.service

      command: start

ssh_authorized_keys:

  # include one or more SSH public keys

  - ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAA..........

Thanks very much,

Gavin

[Gavin McDougall]

I can also confirm that user_data does get applied when run manually:-

# coreos-cloudinit --from-file=/media/configdrive/openstack/latest/user_data

2014/05/27 09:11:07 Fetching user-data from datasource of type "local-file"

2014/05/27 09:11:07 Parsing user-data as cloud-config

2014/05/27 09:11:07 Authorized SSH keys for core user

2014/05/27 09:11:07 Wrote etcd config file to filesystem

2014/05/27 09:11:07 Calling unit command 'start etcd.service'

2014/05/27 09:11:07 Result of 'start etcd.service': done

2014/05/27 09:11:07 Calling unit command 'start fleet.service'

2014/05/27 09:11:07 Result of 'start fleet.service': done

# cat /home/core/.ssh/authorized_keys

# auto-generated by /usr/bin/update-ssh-keys

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEApy

[alex.c...@coreos.com]

Can you check the status of user-cloudinit@media-configdrive-openstack-latest-user_data.path and user-cloudinit@media-configdrive-openstack-latest-user_data.service?

-Alex

[Gavin]

I get the following:-

# systemctl status user-cloudinit@media-configdrive-openstack-latest-user_data.path

● user-cloudinit@media-configdrive-openstack-latest-user_data.path

   Loaded: not-found (Reason: No such file or directory)

   Active: inactive (dead)

# systemctl status user-cloudinit@media-configdrive-openstack-latest-user_data.service

● user-cloudinit@media-configdrive-openstack-latest-user_data.service - Load cloud-config from /media/configdrive/openstack/latest/user_data

   Loaded: loaded (/usr/lib64/systemd/system/user-cloudinit@media-configdrive-openstack-latest-user_data.service; static)

   Active: inactive (dead)

[alex.c...@coreos.com]

Hmm. You're config drive is getting mounted to /meda/configdrive, but systemd can't find the path to user_data. Can you verify that there is user_data under /media/configdrive? Does the path differ from ./openstack/latest/user_data?

-Alex

[Gavin]

Hi Alex,

The structure that I see on the configdrive layout looks like so:-

/dev/vdb  -->> /media/configdrive/ which only contains two directories:-

/media/configdrive/ec2/

/media/configdrive/openstack/  - Under this folder I find a further three directories:-

/media/configdrive/openstack/2012-08-10/

/media/configdrive/openstack/2013-04-04/

/media/configdrive/openstack/latest/

Under all three paths I find my provided user_data yaml.

/media/configdrive is being presented by /dev/vdb from KVM, I've also tried switching to a cdrom device which was also auto-mounted and accessible under /media/configdrive however we experienced the same results.

[alex.c...@coreos.com]

That makes it appear as though user-cloudinit@.path is missing. Can you double check that there is a user-cloudinit@.service and user-cloudinit@.path under /usr/lib/systemd/system/?

-Alex

[Gavin]

Hi Alex,

It appears that user-cloudinit@.path does not exist under that path:-

# ls /usr/lib/systemd/system | grep 'user-cloudinit'

user-cloudinit-proc-cmdline.service

user-cloudinit@.service

user-cloudinit@media-configdrive-openstack-latest-user_data.service

Thanks,

Gavin

[Alex Crawford]

What version of CoreOS is this? (cat /etc/os-release)

-Alex

--

-Alex

[Gavin]

Hi Alex,

This was running on 324.2.0 (Alpha), and have just since tried the latest release, 324.3.0, however with the exact same results.

Thanks. 

[alex.c...@coreos.com]

We were discussing this in #coreos and it appears as though there is an issue with the way cloud-init parsing metadata. Its looking to connect to the metadata service (I assume you are not running that) and holds up the boot until it can connect. Can you verify that coreos-setup-environment.service is stuck in the running state? (systemctl list-jobs)

-Alex

[Gavin]

Hi Alex,

In OpenStack, the metadata can be presented in the form of an EC2 type
http url or also as a config-drive.

We are only using the config drive type, so access of metadata from
http://169.254.169.254/openstack will not work.

Here is the output of the command you mentioned:-

# systemctl list-jobs
JOB UNIT
TYPE STATE
104 multi-user.target
start waiting
168 system-config.target
start waiting
181 user-config.target
start waiting
184 user-cloudinit-proc-cmdline.service
start waiting
169 coreos-setup-environment.service
start running
351 user-cloudinit@media-configdrive-openstack-latest-user_data.service
start waiting
170 system-cloudinit@usr-share-oem-cloud\x2dconfig.yml.service
start running
559 ec2-ssh-key.service
start running

8 jobs listed.

Thanks and Regards,
Gavin

[Gavin]

Unfortunately for me the / partition is read-only so I can't even
modify the live image if I wanted to.

Is there anything that I can look into, ie: the image build/compile
process, maybe a doc or a guide that can point me in the right
direction ?

Thanks.

[Alex Crawford]

What version of CoreOS are you running and on which platform are you running it? The rootfs was changed a couple months back to be read/write, so you should be able to make this change.

-Alex

---

From: Gavin
Sent: ‎6/‎8/‎2014 1:16
To: coreo...@googlegroups.com
Subject: Re: OpenStack config drive: CoreOS image not applying user_data

[Alex Crawford]

Sorry, there's a typo in my previous message. The *root* is read/write.

-Alex

---

From: Alex Crawford
Sent: ‎6/‎8/‎2014 8:34
To: coreo...@googlegroups.com
Subject: RE: OpenStack config drive: CoreOS image not applying user_data

[The entire original message is not included.]

[Gavin]

Hi Alex,

I'm testing each new Alpha release as they arrive, I'm currently testing 338.

I did manage to discover that I can write to /etc, so after learning a
little about systemd, I have been able to create a workaround (dodgy
hack?) that executes the coreos-cloudinit on system boot.

So for now, we are happy, and we have an image that works for us, now
to start building some clusters. :)

We are running OpenStack Grizzly on top of Ubuntu 12.04.

Thanks very much for all your assistance.

Regards,
Gavin

[Andreas Steffan]

Hallo,

I am a CoreOS newbie,  and I followed http://coreos.com/docs/running-coreos/platforms/libvirt/ .  The "user_data not applied issue here" is hitting me as well. The file valid yaml an residing where it should be. However, CoreOS does not seem to pick it up.

How  do I get the system to pick up that configuration ?

Is it a better idea to start with the current beta ?

regards

Andreas