Help using qubes as testing VMs

[Eric W. Biederman]

Can someone tell me if I am missing something?

I do a lot of testing of linux kernels. A bunch of that I have
historically done in qemu with kvm support. Unfortunately nested
hardware virtualization does not work. Which means for testing
for race conditions and the like I need to run the kernels in
their own HVM.

I use an HVM so I can update the kernel in /boot and reboot the qube and
be running the kernel I am testing. It would be nice if I could use a
throw-away qube that just boots with a kernel of my choosing but using
an stand-alone qube is fine.

Where I run into practical problems is when I want to place specific
files into my testing qube. I have not figured out how to ssh into
the qube from another qube, nor have I figured out how to use qvm-copy.
The best I have right now is to have an external machine that I copy
things to and then copy them back, which seems like a real hack.

I also have not figured out how to get a serial console from such a qube
only a graphical one which makes it more difficult than I would like
to capture errors.

I looked at installing the qubes-core-agent package in my testing HVM
but it has too many dependencies and installing it makes it impossible
to test what I would like to test. That is assuming someone has even
packaged it for the distro I need to test on.

Am I missing something? Is there an easier more straight forward way to
setup a testing qube? Is it possible to setup a virtual serial console
to a qube? Is it possible to ssh to a qube from another qube?

Eric

[unman]

Hi Eric

You should probably check out the fine documentation:
https://www.qubes-os.org/doc/managing-vm-kernels/ has information about
using different kernels, including kernels provided by the qube.

https://www.qubes-os.org/doc/firewall has information about enabling
networking between qubes.

If you are using HVMs you can, in some cases, install qubes packages,
and then use tools like qvm-copy. I say, in some cases, because this
wont work with some targets, like Ubuntu standalones.

unman

[Eric W. Biederman]

I don't seem to be missing anything there.

I could theoretically use a standalone PV style cube instead of a
standalone HVM. But going that route I might not be able to see the
output of a boot failure, and the grub timeout is made so small (when
installing qubes-core-agent) I do not have enough time to mess with grub
in case I install a broken kernel.

> https://www.qubes-os.org/doc/firewall has information about enabling
> networking between qubes.

Unfortunately the formula given there to allow networking between hosts
does not work for me and I am not certain why. I am using Qubes 4.0 and
that is supposed to work. When I follow the instructions ping works
fine but tcp connections make it to the firewall vm and I get a "no
route to host" icmp reply.

I am not certain what is the problem. I have been able to completely
disable the qubes firewall and still the ssh packets are returned with a
"no route to host" and icmp packets still make the round trip. It looks
like there is some clever networking configuration that I have not
figured out yet, which is causing the problem.

I am going to spin up a second firewall vm and poke some more, and
see if I can get somewhere.

> If you are using HVMs you can, in some cases, install qubes packages,
> and then use tools like qvm-copy. I say, in some cases, because this
> wont work with some targets, like Ubuntu standalones.

Yes. I have explored using qubes packages. My initial kernel test
configuration is using debian11. Unfortunately the qubes packages make
the HVM unusable for my testing. Pulling in a bunch of stuff I don't
want and taking over configuration I need to control for my tests.

Eric

[Eric W. Biederman]

"Eric W. Biederman" <ebie...@xmission.com> writes:

>> https://www.qubes-os.org/doc/firewall has information about enabling
>> networking between qubes.
>

> I am going to spin up a second firewall vm and poke some more, and
> see if I can get somewhere.

I figured it out. Apparently both iptables rules and nftable
rules are both being configured to prevent qubes from talking
to each other.

The redundancy was a real surprise, as that is just unnecessary
overhead.

Using nftables must be a recent addition and the firewall
Documentation has not caught up.

Eric

[Eric W. Biederman]

It looks like it is some weird fedora34 compatibility thing,
and not the qubes scripts that was causing my problem.

What worked for me was adding the following two lines to
my /rw/config/qubes-firewall-user-script

> # For some reason a duplicate nftables ruleset is getting created
> # that mirrors the iptables ruleset. Flush it so that only iptables
> # needs to be dealt with.
> nft flush ruleset
>
> # Allow my two development machines to talk to each other.
> iptables -I FORWARD 2 -s 10.137.0.33 -d 10.137.0.13 -j ACCEPT
> iptables -I FORWARD 2 -s 10.137.0.13 -d 10.137.0.33 -j ACCEPT

In particular "nft flush ruleset" was needed before any iptables changes
were reflected in the forwarding behavior.

What is the appropriate way to get https://www.qubes-os.org/doc/firewall
updated to reflect that people my have to deal with this?

Eric

[Peter Funk]

Eric W. Biederman schrieb am Monday, den 24.01.2022 um 12:01:
...

> >>> https://www.qubes-os.org/doc/firewall has information about enabling
> >>> networking between qubes.

...
> > nft flush ruleset
...

> In particular "nft flush ruleset" was needed before any iptables changes
> were reflected in the forwarding behavior.

Very interesting! I've a comparable setup in my qubes-firewall-user-script
but since the fedora-34 template receive updates so frequently I've
switched template for my sys-firewall to debian-11. For me this
`nft flush ruleset` command wasn't necessary.

I will try to switch my sys-firewall back to the fedora-34 to see if
this will break things for me and if adding this command will fix it.
Thank you for figuring this out.

Best regards, Peter Funk
--
Peter Funk ✉:Oldenburger Str.86, 27777 Ganderkesee, Germany; 📱:+49-179-640-8878
homeoffice ☎:+49-4222-950270
office ✉: ArtCom GmbH, Haferwende 2, D-28357 Bremen, Germany; ☎:+49-421-20419-0