Cloud-init integration help?

[Brian King]

Hello Ganeteers!

I'm an SRE with the Wikimedia Foundation, and we have a large Ganeti footprint . I've been searching for a way to make our VM provisioning process act more like a cloud. Specifically, I'd like to start using cloud-init. 

To that end, I've found several promising repos including https://github.com/neicnordic/ganeti-os-nocloud , https://github.com/osuosl/ganeti-instance-image , and https://github.com/ganeti/instance-cloudimage . 

But I'm still pretty new in to Ganeti, and I was wondering if anyone would be willing to help me with a proof of concept create script? I can work through the Google group, email, IRC (inflatador on libera.chat) or whatever you prefer.

As far as requirements, here's what I've got so far:

• Must work with vendor cloud images (qcow2)
• Config-drive created outside of the process. In other words, the script just attaches an already-formed config drive with meta-data, user-data, etc.

Thanks for reading and let me know if you're willing to help!

Best,

Brian King

[Brian King]

Hello all,

I forked ganeti-os-nocloud and was able to get a proof of concept going. If anyone else is interested in contributing, please let me know. PRs welcome!

Best,

Brian

[Petter Urkedal]

Dear Brian,

Good you found ganeti-os-nocloud useful.

A colleague of mine made some changes to the create script to support
Rocky 9 by using a seed image (which we unfortunately haven't committed
back yet). It's not a perfect solution, since it means doing
gnt-instance copyfile on a generated ISO image and passing an argument
to attach it on the first boot, which defies the original design of
automating the VM creation.

From your README and stripped-down create-script, I can see your
approach is to externalize the seeding, which seems reasonable given
this limitation and the fact that the generated seed image may not suit
everybody.

Best regards,
Petter

On 2023-09-02 12:06, Brian King wrote:
> Hello all,

> I forked ganeti-os-nocloud <https://github.com/inflatador/ganeti-os-nocloud>

> and was able to get a proof of concept going. If anyone else is interested
> in contributing, please let me know. PRs welcome!
>
> Best,
>
> Brian
> On Friday, September 1, 2023 at 12:34:42 AM UTC-5 Brian King wrote:
>
> > Hello Ganeteers!
> >
> > I'm an SRE with the Wikimedia Foundation, and we have a large Ganeti

> > footprint <https://wikitech.wikimedia.org/wiki/Ganeti> . I've been

> > searching for a way to make our VM provisioning process

> > <https://wikitech.wikimedia.org/wiki/Ganeti#Create_a_VM> act more like a

> > cloud. Specifically, I'd like to start using cloud-init.
> >
> > To that end, I've found several promising repos including
> > https://github.com/neicnordic/ganeti-os-nocloud ,
> > https://github.com/osuosl/ganeti-instance-image , and
> > https://github.com/ganeti/instance-cloudimage .
> >
> > But I'm still pretty new in to Ganeti, and I was wondering if anyone would
> > be willing to help me with a proof of concept create script? I can work
> > through the Google group, email, IRC (inflatador on libera.chat) or
> > whatever you prefer.
> >
> > As far as requirements, here's what I've got so far:
> >

> > - Must work with vendor cloud images (qcow2)
> > - Config-drive created outside of the process. In other words, the

> > script just attaches an already-formed config drive with meta-data,
> > user-data, etc.
> >
> > Thanks for reading and let me know if you're willing to help!
> >
> > Best,
> > Brian King
> >
>

> --
> You received this message because you are subscribed to the Google Groups "ganeti" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ganeti+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ganeti/76898bfb-34d7-491b-ae2b-ecc4a86d98e2n%40googlegroups.com.

[Rudolph Bott]

Hi Brian,

Each Ganeti instance has a UUID assigned internally by Ganeti. You can use that UUID to request configuration data from an external webservice. I did some testing recently (needed a bleeding edge cloud init version at that time), but I don’t have access to that right now. 

In short: I booted an instance using a simple image-based OS provider and some custom KVM parameters (which are the same for all instances) and cloud init initiated a HTTP call during early boot to an external Webserver, adding the instance‘s UUID to the URL.

In theory one could create a small service that answers to these URLs and retrieves instance information through the provided UUID (via Ganeti RAPI) and additional information from other sources if needed (eg netbox) and returns valid cloud init bootstrapping data.

I’ll try to get my hands on the details and update this thread.

Cheers,

Rudi

--
You received this message because you are subscribed to the Google Groups "ganeti" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ganeti+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ganeti/d6b5bebb-7596-4aa4-af75-4468d56765f6n%40googlegroups.com.

[Brian Candler]

Last time I looked at this, the metadata support in Ganeti was half-baked, with a number of limitations.

Once you'd managed to enable it, the fundamental problem was that it wasn't compatible with any public cloud provider's metadata service. This means that to use it, you'd have to modify cloud-init itself - which defeats the ability to work with unmodified, published VM images.

See previous discussions:

https://groups.google.com/g/ganeti/c/WOGH3uru2GQ

https://groups.google.com/g/ganeti/c/tALQ13az0wg/m/4cLOVlRNDgAJ

I don't have any ganeti clusters any more, but when I was last using it, I was using ganeti-os-nocloud, using its own ability to define cloud-init data. From the notes I made at the time:

==> /etc/ganeti/nocloud/variants.list <==
ubuntu-22.04

==> /etc/ganeti/nocloud/variants/ubuntu-22.04.conf <==
IMAGE_FILE=ubuntu-22.04-server-cloudimg-amd64.img

==> /etc/ganeti/nocloud/user-data/default.yml <==
#cloud-config

# Allow root to login (with ssh keys only)
disable_root: false
ssh_pwauth: false
ssh_authorized_keys:
  - ssh-rsa ... etc

I had also tweaked the DNS and networking settings:

gnt-os modify -O 'dns_nameservers=10.0.0.53\,10.0.0.54,dns_search=example.com' -H kvm:vhost_net=true nocloud+ubuntu-22.04

This allowed me to create instances like this:

gnt-instance add -t plain -n node1 -s 14G -o nocloud+ubuntu-22.04 --net 0:network=net1000,ip=10.0.0.5,name=eth0 --no-name-check --no-ip-check -B vcpus=2,memory=4096m myvm1

However, if you want to provide your own prepared seed disk image to each new instance, I think you might as well use ganeti-os-noop and temporarily attach the second drive on first bootup.

Regards,

Brian (a different one)

[Brian King]

Thanks Petter, Rudi, and other Brian!

I'll definitely check out ganeti-os-noop , that might be enough to do what I want.

I've written an Ansible playbook (very much in progress) that works in a single-node environment (including IP auto-config via cloud-init), but I still haven't gotten to test this on an actual cluster. Do any of you foresee any problems with this approach in a clustered environment? I'm thinking of what might happen when a Ganeti host fails...will the ISO files automatically be on the VM's backup host?

Rudi, 

We do use Netbox so I would be interested in any advice/integrations you have on that topic. I'll probably be interacting with Netbox through Ansible and/or raw Python.

Thanks again everyone!

Brian King

SRE, Search Platform Team

Wikimedia Foundation

IRC: inflatador

You received this message because you are subscribed to a topic in the Google Groups "ganeti" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ganeti/PvWRt_9b0-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ganeti+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ganeti/d48fe1cb-d7ec-4cf8-a958-534cecf6fe69n%40googlegroups.com.

[Brian Candler]

On Tuesday, 5 September 2023 at 21:07:04 UTC+1 Brian King wrote:

I'm thinking of what might happen when a Ganeti host fails...will the ISO files automatically be on the VM's backup host?

No, it's up to you to place the ISO files wherever they're needed. You could either mount a central NFS server, or you could use "gnt-cluster copyfile" to distribute it whenever you add a new one.

 

For Netbox<->VM syncing, there are several ways you could implement it:

1. adding a VM in Ganeti causes it to be added retrospectively to Netbox

2. adding a VM to Netbox causes it to be created in Ganeti (via a webhook)

3. adding a VM in Ganeti fetches the necessary parameters from Netbox (e.g. IP addresses) - i.e. you have to create it first in Netbox, but it doesn't automatically trigger creation

I've implemented (3) for my home network with lxd containers, via some simple scripting.

[Brian King]

No, it's up to you to place the ISO files wherever they're needed. You could either mount a central NFS server, or you could use "gnt-cluster copyfile" to distribute it whenever you add a new one.

 

You know, come to think of it, the ISO files aren't that important after the initial cloud-init run. I'm thinking maybe I just detach the ISO files with gnt-instance modify. Config management (puppet) should reboot the VM after updates, which would apply the detach request. Let me know if you see any problems with that approach.

For Netbox<->VM syncing, there are several ways you could implement it:

After looking a bit more closely at our current automation , it looks like we're going with option 1. So I guess I'll need to use that approach for my proof of concept.

Thanks again for your help!

~B

--

You received this message because you are subscribed to a topic in the Google Groups "ganeti" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ganeti/PvWRt_9b0-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ganeti+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ganeti/47483d3e-bd6d-49f6-9086-1951f04cb00fn%40googlegroups.com.

[Brian Candler]

If you just want to attach the ISO (or floppy or whatever) image on first boot only, then pass it as an argument to gnt-instance start; it won't persist with the VM config. e.g.

gnt-instance start -H boot_order=cdrom,cdrom_image_path=/srv/isos/xxx.iso <VMNAME>

(Well, that's an example where I wanted to *boot* from the cdrom one-time, but you get the idea).

Actually, I believe it will remain attached if you do a reboot *within* the instance. But if you shutdown and restart the instance from the ganeti side, or use gnt-instance reboot, then it won't be attached next time.