Vagrant for generating VMs

[Frédéric Pierret]

Hi,

We had a discussion with Marek yesterday about how we can renew the way we build our templates. For example it would be very much convenient to have sample piece of file like a Dockerfile as an input to generate a VM. This is something I've already experimented and done in other projects. Even if it's very convenient for a building process, it is not in the standard for VMs. My opinion is that we can go further by using for example a more flexible and adopted format with several configuration backends like Vagrant with the use of Vagrantfile.

We see two subjects related to Vagrant:

1) Use a Vagrantfile as an input for what will be inside our templates. We need to adapt our linux-template-builder + pieces in builder-{rpm,debian,archlinux,gentoo}. Here, it's currently out of consideration to use pre-built images from somewhere like we would do with Vagrant or even Docker by pulling external images. We recreate from scratch the "box" then ship it as usual in a RPM as our current templates.

Of course, we would still support the legacy way somehow but this would be the occasion to make our template builder maybe less qubes-builder dependent. Also, it would allow to have a more standalone tool for which any user could benefit in creating their own templates in place of post applying Salt formulas directly on Qubes templates which are not all the time user friendly.

2) Share your AppVM!

We are thinking to add a way to create AppVM easily by using Vagranfile with a Qubes integration. A naive approach would be to use StandaloneVM cloned from Qubes templates then doing what's inside the Vagrantfile but this is something we don't want to do at first. We want to use advantage of AppVM and starting for example by:

- identify instruction about which Qubes template to use,
- identify persistent/non-persistent instructions.

Every instructions like installing packages would be triggered into the underlying requested Qubes template then, the others in the AppVM (mostly configuration or local user installation). Thinking again of Salt formulas, we integrate a way of customization which is from my opinion very convenient (not speaking of Ruby syntax ;) ). But, as Vagrant supports Ansible, Salt, etc. as customization configuration, we don't lost our current Salt formulas at all! For those who want to keep them, just put the trigger to needed ones in the Vagrantfile.

3) Create complex Qubes infrastructure:

We have several topics where we setup service infrastructure into Qubes (e.g. https://github.com/QubesOS/qubes-infrastructure or https://github.com/fepitre/qubes-mgmt-salt-qubes-server/tree/devel-140320). With a focus on services, currently I'm testing and use in production docker-compose to create services (e.g. https://github.com/fepitre/package-rebuilder). Switching this simple piece of file to more Salt formulas to use service in AppVM is not really something I want to do because it's not as user friendly as docker-compose when updating the services code, restarting, etc. That would be another example where using only a Vagrantfile to handle multiple VMs at once could be a benefit to use Qubes as a "secure services platform".


Having a proper Qubes integration, we could think having/using a community hub where we could push/share Vagrantfiles for creating AppVMs/StandaloneVMs from Qubes templates. Once again, only the descriptive instructions would be used and all would be done locally like you would do it currently with Salt formulas.

Any feedback is welcomed :)

Best regards,
Frédéric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Apr 02, 2021 at 10:52:13AM +0200, Frédéric Pierret wrote:
> Hi,
>
> We had a discussion with Marek yesterday about how we can renew the way we build our templates. For example it would be very much convenient to have sample piece of file like a Dockerfile as an input to generate a VM. This is something I've already experimented and done in other projects. Even if it's very convenient for a building process, it is not in the standard for VMs. My opinion is that we can go further by using for example a more flexible and adopted format with several configuration backends like Vagrant with the use of Vagrantfile.

A bit context why "like a Dockerfile" but not Dockerfile specifically:
docker is about containers, not VMs. While most of it would be the same,
containers operate on a bit higher level and lack few parts that we need
in a VM, specifically:
- disk partitioning and creating filesystem
- bootloader, kernel, modules etc

It isn't a deal-breaker, those parts could be added outside of
Dockerfile. But if there is something that would handle it all, it would
be better.

> We see two subjects related to Vagrant:
>
> 1) Use a Vagrantfile as an input for what will be inside our templates. We need to adapt our linux-template-builder + pieces in builder-{rpm,debian,archlinux,gentoo}. Here, it's currently out of consideration to use pre-built images from somewhere like we would do with Vagrant or even Docker by pulling external images. We recreate from scratch the "box" then ship it as usual in a RPM as our current templates.
>
> Of course, we would still support the legacy way somehow but this would be the occasion to make our template builder maybe less qubes-builder dependent. Also, it would allow to have a more standalone tool for which any user could benefit in creating their own templates in place of post applying Salt formulas directly on Qubes templates which are not all the time user friendly.
> 2) Share your AppVM!
>
> We are thinking to add a way to create AppVM easily by using Vagranfile with a Qubes integration. A naive approach would be to use StandaloneVM cloned from Qubes templates then doing what's inside the Vagrantfile but this is something we don't want to do at first. We want to use advantage of AppVM and starting for example by:
>
> - identify instruction about which Qubes template to use,
> - identify persistent/non-persistent instructions.
>
> Every instructions like installing packages would be triggered into the underlying requested Qubes template then, the others in the AppVM (mostly configuration or local user installation). Thinking again of Salt formulas, we integrate a way of customization which is from my opinion very convenient (not speaking of Ruby syntax ;) ). But, as Vagrant supports Ansible, Salt, etc. as customization configuration, we don't lost our current Salt formulas at all! For those who want to keep them, just put the trigger to needed ones in the Vagrantfile.
>
> 3) Create complex Qubes infrastructure:
>
> We have several topics where we setup service infrastructure into Qubes (e.g. https://github.com/QubesOS/qubes-infrastructure or https://github.com/fepitre/qubes-mgmt-salt-qubes-server/tree/devel-140320). With a focus on services, currently I'm testing and use in production docker-compose to create services (e.g. https://github.com/fepitre/package-rebuilder). Switching this simple piece of file to more Salt formulas to use service in AppVM is not really something I want to do because it's not as user friendly as docker-compose when updating the services code, restarting, etc. That would be another example where using only a Vagrantfile to handle multiple VMs at once could be a benefit to use Qubes as a "secure services platform".
>
>
> Having a proper Qubes integration, we could think having/using a community hub where we could push/share Vagrantfiles for creating AppVMs/StandaloneVMs from Qubes templates. Once again, only the descriptive instructions would be used and all would be done locally like you would do it currently with Salt formulas.
>
> Any feedback is welcomed :)

The above is some high level idea how to make building new templates
and custom setups easier. Before talking details, we need to answer some
basic questions:
1. Whether such thing would be useful at all? Or maybe our current
pre-built templates (and template-builder scripts) + Salt is enough?
2. Is Vagrant the right too for the job?

If we'd go in a direction like this, I'd strongly prefer to re-use some
existing well known tool. Integrating anything into our custom
qubes-builder + template-builder has quite high contributor entry
barrier.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmBokV8ACgkQ24/THMrX
1yy7ygf/WaxPxHhU6wF4Q1kwyxmMs9c0c8d1Vadboegxx4SaNfMvJq/UwbLdWlNh
ugsmROItLdmrZy/Ul4gTsEs33cH4cGaZW71jvOzAkXlfz9PHMa5y6/ZBobxXtOSy
6VRNsXCw3InfPFPsk2OrsBb5SqMflrJ5pWxu1enOF9QrT9oZ6NBHngM11eKJzoXL
TwsovCQtVBjG9DBbYa6KDlFI1ykRAR958h1GfMqVcn9nnSr0C7XGluZP8L/Zx1HM
ERHzKR6qAAH+WOAaT7IGCEHGC0HwNBDApwflr7oF2iSPOgSzBWHuQIydBOG1wqOS
/4CQqEtItqo6Es+sew48lqSoQambHQ==
=kMmy
-----END PGP SIGNATURE-----

[Conor Schaefer]

On 4/3/21 9:01 AM, Marek Marczykowski-Górecki wrote:
> On Fri, Apr 02, 2021 at 10:52:13AM +0200, Frédéric Pierret wrote:
>> Hi,
>
>> We had a discussion with Marek yesterday about how we can renew the way we build our templates. For example it would be very much convenient to
have sample piece of file like a Dockerfile as an input to generate a VM.
This is something I've already experimented and done in other projects. Even if it's very convenient for a building process, it is not in the standard for VMs. My opinion is that we can go further by using for example a
more flexible and adopted format with several configuration backends like
Vagrant with the use of Vagrantfile.

Wonderful topic, thank you both for bringing it up on the list. I've
long struggled to understand the qubes-builder logic intimately, and
reusing tools where possible would certainly lower the barrier to entry.

>> Having a proper Qubes integration, we could think having/using a community hub where we could push/share Vagrantfiles for creating AppVMs/StandaloneVMs from Qubes templates. Once again, only the descriptive instructions would be used and all would be done locally like you would do it currently with Salt formulas.

Some common Qubes use cases, such as setting up a Wireguard ProxyVM, are
fairly difficult for newcomers. While qubes-tunnel has fine
documentation, and inclusion in the contrib repo has really helped
getting started, there's still a lot of fussing via manual steps that
begs for a scripting layer to smooth out stringing the pieces together.

>
>> Any feedback is welcomed :)
>
> The above is some high level idea how to make building new templates
> and custom setups easier. Before talking details, we need to answer some
> basic questions:
> 1. Whether such thing would be useful at all? Or maybe our current
> pre-built templates (and template-builder scripts) + Salt is enough?
> 2. Is Vagrant the right too for the job?

My first question at this point is: why Vagrant, and not Packer? My
understanding is that Packer is the de facto standard for building VM
images, whether for local use or on a cloud provider. Vagrant, among
other tools, can then consume the artifacts generated by Packer in a
fairly accessible fashion.

Perhaps I'm making a false dichotomy here, and writing a plugin to
create and package a VM within Qubes is six of one, half a dozen of the
other. In my mind, there's very much room for both: Packer if a VM image
is going to be packaged as an RPM and installed in dom0, Vagrant if a VM
image is going to reuse an existing image on the system (e.g.
"qubes-template-debian-10") and sprinkle a little config magic on top.

Frédéric, consider me an eager tester if you have any early scripts or
implementations to share.