cloud-init not working on OpenStack (CloudVPS)

[Ralf Schimmel]

Hey there,

CoreOS seems like a perfect match for one of our projects and I would love to try it.

I followed the OpenStack installation, did a YAML lint, checked that I had the latest Alpha release or the beta release, all ok.

When starting a machine (or cluster) with nova, all seems good and the machines come up.

However.... it seems that the cloud-init is not executed, since I cannot login with my key.

Also tried cloud-config with a user with password, no success.

All I want is CoreOS machines on OpenStack on which I can login, other options than the default OpenStack install are also good, if anyone can tell me how :D

I saw a recent post by Jim Walters on the same subject: https://groups.google.com/forum/#!topic/coreos-dev/INnx4AOpoYc

Though it did not end with a solution that worked, also passing the key directly does not work.

Any help is appreciated, the developer at CloudVPS (http://www.cloudvps.com/) are telling me that CoreOS fails to execute the cloud-init (i quote: ".. it seems CoreOS is not executing the group-config...").

btw; booting a CentoOS image with the same cloud-config works fine.

Thanks, Ralf

[Brian Harrington]

Do you know if they're supplying the cloud-config via configdrive or meta-data service?  Either way can you send the output of a `nova console-log` for the image?

--Brian 'redbeard' Harrington

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Harrington]

One other thing, can you send a copy of your cloud-config?  Certain syntax issues like specifying groups which don't exist can cause problems (i.e. If I attempt to create user 'redbeard' in groups 'docker' and 'developers' but have not defined the group 'developers' the user creation will fail).

--Brian 'redbeard' Harrington

On 05/20/2014 12:25 PM, Ralf Schimmel wrote:

[Ralf Schimmel]

They are using config-drive;

The cloud-config:

https://gist.githubusercontent.com/ralfschimmel/bb6eaa5d99914ddcf356/raw/44ad8c17c8ab429587d0cdce6cf5f0497339f0dd/gistfile1.yml

// Image creation: 

$ glance image-create --name CoreOS \

   --container-format bare \

   --disk-format qcow2 \

   --file coreos_production_openstack_image.img \

   --is-public True

// boot

$ nova boot \

--user_data ./cloud-config.yaml \

--image 361753cb-1301-4ae5-a9c6-5fa11d3dff9c \

--flavor 101 \

--security-groups default \

--key_name ralf \

--config-drive=true \

--poll coreos-lb-1

The log:

https://gist.githubusercontent.com/ralfschimmel/3701768a3613c37b8285/raw/bc1a25af7adf3fe5c053787a94a066e39250af87/gistfile1.txt

Op dinsdag 20 mei 2014 21:31:27 UTC+2 schreef Brian Harrington:

[Tristan van Bokkem]

I noticed that when I specified a public_ipv4 addr in the cloud-config like:

    addr: $public_ipv4:4001

    peer-addr: $public_ipv4:7001

And booted the instances with a public ip address, I was not able to login with my key, however when I specified it with a private_ipv4 like:

    addr: $private_ipv4:4001

    peer-addr: $private_ipv4:7001

And then booted the instance with a private network attached and assigned a floating ip to the private ip address I am able to login.

Not sure what this is causing.

[Ralf Schimmel]

Indeed, this enables me to ssh into the machine!

However, the environment is not set.

in /etc/environment the private and public ip are empty, resulting in a bad etcd configuration.

Any ideas how this can happen or be resolved?

Op woensdag 21 mei 2014 13:04:01 UTC+2 schreef Tristan van Bokkem:

[Brandon Philips]

On Wed, May 21, 2014 at 11:25 AM, Ralf Schimmel
<ralf.s...@topicus.nl> wrote:
> Indeed, this enables me to ssh into the machine!
> However, the environment is not set.
>
> in /etc/environment the private and public ip are empty, resulting in a bad
> etcd configuration.
>
> Any ideas how this can happen or be resolved?

This is related to this bug:
https://github.com/coreos/init/pull/98#issuecomment-43692227

Brandon

[Brian Harrington]

Think of building a 'traditional' LAMP application inside an organization's firewall.

You would provision the VM, assign an IP address which that machine will then own. Then you would set up a NAT rule to route traffic from the outside world to the host which is serving the application.  This is essentially the same process inside of OpenStack.

Within OpenStack there is the concept of a fixed and a floating IP.  The fixed IP is the one which you assigned to the VM.  The floating ip is the one on the firewall.  Within OpenStack this translation is done with IPTables SNAT rules the same way that a hardware firewall would. 

The actual knowledge of the IP address(es) and other information comes from the meta-data service.  If you are not using the meta-data service (or it's not configured correctly by your OpenStack hosting provider) then when machines make an HTTP GET to  http://169.254.169.254/2009-04-04/meta-data it will get a list of parameters back.  If the system was not assigned a public IP (via the floating ip mechanism) at the time of instantiation then it won't be available when the system tries to do a GET of  http://169.254.169.254/2009-04-04/meta-data/public-ipv4.  When you specify $public-ipv4 in your cloud-config referencing the meta-data provided by the cloud-config service, not a value that CoreOS provides.

--Brian 'redbeard' Harrington

[Ralf Schimmel]

Thanks Brian,

It indeed seems that the metadata service is returning a HTTP 500.

I've informed the people over at CloudVPS and hope to hear more tomorrow.

Until then;

1. is there a way to manually set the /etc/environment in the boot process, after it was tried by the coreos-setup-environment script?

2. will that fix etcd and fleet?

Cheers, Ralf

[Ralf Schimmel]

So it seems to be a routing issue.

Booting a Ubuntu host outside of my private network does enable it to reach the Metadata service.

Could it be a routing issue on my instance/private network, and thus non coreos related?

Op woensdag 21 mei 2014 21:37:46 UTC+2 schreef Ralf Schimmel:

[Arash Kaffamanesh]

Hi Ralf,

If you're using floating public IP, then you can use:

/usr/share/oem/bin/coreos-setup-environment /etc/environment

to set your floating IP in the /etc/environment.

I'd a similar problem to setup a CoreOS Cluster on my OpenStack Havana environment, which is discussed on coreos-dev here:

https://groups.google.com/forum/#!topic/coreos-dev/INnx4AOpoYc

By the way running:

journalctl -u ec2-cloudinit

might show you what's going wrong.

HTH,

-Arash

[wijnand...@holder.nl]

Hi Ralf,

just curious, have you been able to get CoreOS working on CloudVPS?

I have been trying to get it to work last weekend and before I try again this weekend it would be nice to know if it's worth my time ;-)

-- Wijnand

[Ralf Schimmel]

Hey Wijnand,

It works, though with some workarounds.

 - create a private network, without a router (bug @ cloudvps)

 - create a router

 - create a machine, add your cloud-init file and attach it to the private network

 - add a floating IP to your new machine

 - reboot your machine and it will have a correct /etc/environment file with private and public ip

Good luck!

--

[Jeffrey Schilperoord]

Hey Ralf,

Also a CloudVPS customer here. I even tried using the nova command utility to provide a cloud-config.yaml but no cloud-init executons.

I check the server log and I dont see cloud-init doing something. Do you have any tips ?

Thanks,

Jeffrey

Op zondag 1 juni 2014 16:18:00 UTC+2 schreef Ralf Schimmel:

[Ralf Schimmel]

Hey Jeffrey,

Wijnand his workaround is the way to go, works like a charm, also with a cloud-init config.

Tried it today and works like a charm. What seems to be the problem? What did you try?

Ralf

[Jeffrey Schilperoord]

Hi Ralf,

I created a new network. Added the router and can now login to the server via its external ip using the core user.

The only issue I have now is that the configdrive service somehow can't load the config:

user-configdrive.service loaded failed failed Load cloud-config from /media/configdrive

The mount is there when I check with df -H

Thanks,

Jeffrey

Op woensdag 25 juni 2014 16:12:13 UTC+2 schreef Ralf Schimmel:

[Jonathan Boulle]

What does `systemctl status user-configdrive.service` give you?

[Jeffrey Schilperoord]

Thanks Jonathan. I had an error in my cloudconfig file.

Regards,

Jeffrey

Op woensdag 25 juni 2014 20:49:38 UTC+2 schreef Jonathan Boulle: