Networking issue after upgrading to Fedora-33

[mgla...@gmail.com]

Hi everyone,

I'm looking for some help as to how to diagnose some app-VM networking issues. I have 2 vms, both based on the same template with identical config, but one can reach the internet and the other cannot.

Before upgrading:

2 standalone VMs based on Fedora-30. One with a bunch of dev tools installed, one relatively untouched. I had multiple VMs based on these two templates. I also updated my sys-net and sys-firewall to Fedora-33 at the same time.

Upgrade:

I upgraded to Fedora-33, and realised I could rationalise my VMs, so now every appVM is based off the same Fedora-33 template.

The issue:

Some of my migrated VMs are completely fine, others have no network.

I _think_ it is the VMs that used to be based on my old "untouched" vm that have the issue.

VM1:

No networking at all.

VM2:

Networking is completely fine, everything works as expected.

Both VMs are based on the same Fedora-33 template, with the same (default) sys-firewall Networking, both with the firewall configured to allow all outgoing internet connections

vm1$ ping google.com

ping: google.com: Name or service not known

vm1$ dig google.com

; <<>> DiG 9.11.35-RedHat-9.11.35-1.fc33 <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached

vm1$ resolvectl dns

Global: 10.139.1.1 10.139.1.2
Link 2 (eth0):

vm2$ resolvectl dns

Global: 10.139.1.1 10.139.1.2
Link 2 (eth0):
Link 3 (br-11bfb2cd10e9):
Link 4 (docker0):
Link 5 (br-cf58034d074b):
Link 6 (br-f9686c41a7f5):

So there's definitely something wrong, but I don't know enough about Linux/Qubes networking to work out what.

Any pointers gratefully received.

[unman]

On Mon, Sep 27, 2021 at 02:35:34AM -0700, mgla...@gmail.com wrote:
>
> Hi everyone,
>
> I'm looking for some help as to how to diagnose some app-VM networking
> issues. I have 2 vms, both based on the same template with identical
> config, but one can reach the internet and the other cannot.
>
> Before upgrading:
> 2 standalone VMs based on Fedora-30. One with a bunch of dev tools
> installed, one relatively untouched. I had multiple VMs based on these two
> templates. I also updated my sys-net and sys-firewall to Fedora-33 at the
> same time.
>
> Upgrade:
> I upgraded to Fedora-33, and realised I could rationalise my VMs, so now
> every appVM is based off the same Fedora-33 template.
>
> The issue:
> Some of my migrated VMs are completely fine, others have no network.
> I _think_ it is the VMs that used to be based on my old "untouched" vm that
> have the issue.
>
> VM1:
> No networking at all.
>
> VM2:
> Networking is completely fine, everything works as expected.
>
> Both VMs are based on the same Fedora-33 template, with the same (default)
> sys-firewall Networking, both with the firewall configured to allow all
> outgoing internet connections
>

> *vm1$ ping google.com*

> ping: google.com: Name or service not known
>

> *vm1$ dig google.com*

> ; <<>> DiG 9.11.35-RedHat-9.11.35-1.fc33 <<>> google.com
> ;; global options: +cmd
> ;; connection timed out; no servers could be reached
>

> *vm1$ resolvectl dns*

> Global: 10.139.1.1 10.139.1.2
> Link 2 (eth0):
>
>

> *vm2$ resolvectl dns*

> Global: 10.139.1.1 10.139.1.2
> Link 2 (eth0):
> Link 3 (br-11bfb2cd10e9):
> Link 4 (docker0):
> Link 5 (br-cf58034d074b):
> Link 6 (br-f9686c41a7f5):
>
> So there's definitely something wrong, but I don't know enough about
> Linux/Qubes networking to work out what.
>
> Any pointers gratefully received.
>

There is something wrong, but there is nothing in the detail you have
provided that might explain it.
So:
Do you have any custom firewall rules set on vm1? (qvm-firewall vm1)
Can you ping the firewall from vm1, by IP address?
Can you access a host on the internet by IP address?(e.g https://9.9.9.9)
If you create a new qube from the template, does networking work as
expected?
If you change templates on vm1, does networking work?

[mgla...@gmail.com]

Yes, there are custom firewall rules, but the firewall is set to  "Allow all outgoing internet connections". Which should ignore all the rules?

Yes, I can happily ping sys-firewall's IP from both vm1 and vm2.

No, I can't access a host on the internet by IP

vm1$ curl https://9.9.9.9
curl: (7) Failed to connect to 9.9.9.9 port 443: No route to host

Yes, creating a new qube from the same template works fine - networking is exactly as expected.

No, changing templates on vm1 doesn't fix it (I thought it did when I tried a month or so ago, but I just gave up without really trying to get to the bottom of what's wrong. Either way, it doesn't work now)

Changing vm1's network from sys-firewall to sys-net doesn't fix it, either.

But this is interesting though. This is what I get when pinging the IP of sys-net (whilst networking is set to sys-firewall):

vm1$ ping 10.137.0.5
PING 10.137.0.5 (10.137.0.5) 56(84) bytes of data.
From 10.137.0.6 icmp_seq=1 Packet filtered
From 10.137.0.6 icmp_seq=2 Packet filtered
From 10.137.0.6 icmp_seq=3 Packet filtered

[...]

3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2077ms

(0.5 is the IP of sys-net and 0.6 is sys-firewall)

So somehow when I ping sys-net I get a response from sys-firewall. Huh!?

This is what happens on my working VM:

vm2$ ping 10.137.0.5
PING 10.137.0.5 (10.137.0.5) 56(84) bytes of data.
64 bytes from 10.137.0.5: icmp_seq=1 ttl=63 time=0.286 ms
64 bytes from 10.137.0.5: icmp_seq=2 ttl=63 time=0.529 ms
64 bytes from 10.137.0.5: icmp_seq=3 ttl=63 time=0.551 ms

[unman]

On Mon, Sep 27, 2021 at 06:36:16AM -0700, mgla...@gmail.com wrote:
>
> Yes, there are custom firewall rules, but the firewall is set to "Allow
> all outgoing internet connections". Which should ignore all the rules?
>

AFAIK, if you set custom firewall rules, they override the GUI firewall
setting.
Inspect the rules applying on sys-firewall.