firewall/proxy VM not working with Qubes 4.0-rc4

[thorsten...@gmail.com]

I installed Qubes 4.0-rc4 and have a problem with my internet connection.
sys-net itself has a working internet connection but sys-firewall does not. No need to mention that every other VM that uses sys-firewall as netVM does also have no working internet connection.

If I switch the default netVM from sys-firewall to sys-net (for testing), dom0 can use it to update etc. Also any other VM gets internet connection with sys-net as Networking VM.

An update of dom0 from testing-repository did not fix the problem.
Also switching the sys-firewall template from fedora-26 to debian-9 does not help.

I found a similar problem here:
https://github.com/QubesOS/qubes-issues/issues/2141

So I checked the network interfaces and they are like this:

sys-net:
lo
enp0s0
vif2.0

sys-firewall:
eth0
lo

Not sure, but I guess the vif interface is missing in sys-firewall?
How do I fix this problem?

[awokd]

Did the installer run into problems when it was doing the first template
deployment? Did you run it a second time by mistake, possibly?

Try to delete (or rename) your existing sys-firewall, then create a new
AppVM named sys-firewall with the "provides network" box checked.

Then, double-check that your test AppVM is pointing to your new
sys-firewall with qvm-prefs (look for netvm setting), and start it up and
test.

[Alex Dubois]

vif interface will appear when a VM connects to it.

Could you clarify the term no internet.

I had a lot of problems solved once sys-net had the service clocksync enabled (as it should).

[Thorsten Schierer]

Ok, I set up 2 new VMs (sys-net and sys-firewall) in case something went wrong during the setup, but the result was the same as before.

Not sure how to enable the clocksync service in sys-net (fedora-26 template) but the date/time settings are correct, so I assume it already is syncing correctly.

But I did some more research and this is what I found out so far is:

sys-net itself has a working internet connection (I can do "ping www.google.com" in a terminal and everything is fine).
Also other VMs that use sys-net directly as netVM can access the internet (i.e. ping a server etc.).
The only exception is sys-firewall, in which a ping just fails due to no connection.

When sys-firewall starts up, a new vif is created inside sys-net (which was expected), but there is no route created.

When I tried to create a new route it said "Network is down". So it did "ifconfig vif8.0 up" and afterwards added a new route with:

"sudo ip route add 10.137.0.15 dev vif8.0 metric 32752"

"route -v" displays:
10.137.0.15   0.0.0.0   255.255.255.255   UH   32752   0   0   vif8.0

So at this point the ifconfig and route entries look exactly like on my other machine which is working fine out of the box.

Unfortunately sys-firewall still does not have a working internet connection ("ping www.google.com" results in "Name or service not known" due to no DNS connectivity).

So it seems like as soon as I create a new VM with "provides network" checked, it can not use the network connection of sys-net. Any other VM that does not provide network ifself can use sys-net directly and works fine.

I think there is a problem with some kind of proxy setup in sys-firewall or something.

Is there some documentation which steps are done regarding networking during the startup of sys-firewall, so I can try to do those steps manually one by one to see where the problem appears?

--
You received this message because you are subscribed to a topic in the Google Groups "qubes-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/qubes-users/oN204nGh63I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to qubes-users+unsubscribe@googlegroups.com.
To post to this group, send email to qubes...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/46a6952f-6fd5-4aec-93ca-994937a24c5e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[awokd]

On Mon, February 26, 2018 11:17 pm, Thorsten Schierer wrote:

>
> So it seems like as soon as I create a new VM with "provides network"
> checked, it can not use the network connection of sys-net. Any other VM
> that does not provide network ifself can use sys-net directly and works
> fine. I think there is a problem with some kind of proxy setup in
> sys-firewall or something. Is there some documentation which steps are done
> regarding networking during the startup of sys-firewall, so I can try to
> do those steps manually one by one to see where the problem appears?

Can you also try doing this against the template you're using for your
sys-firewall?

qvm-features fedora-26-minimal qubes-firewall 1

[Alex Dubois]

On Monday, 26 February 2018 23:17:41 UTC, Thorsten Schierer wrote:
> Ok, I set up 2 new VMs (sys-net and
> sys-firewall) in case something went wrong during the setup, but the
> result was the same as before.
>
> Not sure how to enable the
> clocksync service in sys-net (fedora-26 template) but the date/time
> settings are correct, so I assume it already is syncing correctly.

Yes probably. For reference, to check (or enable):
- go to start menu/System Tools/Qube Manager
- right click sys-net/Qube Settings/Services tab
- clocksync should be in the list and ticked if not type clocksync and click on +
- I think a full reboot is required. There are probably ways to avoid it...

>
> But I did some more research and this is what I found out so far is:
>
> sys-net itself has a working internet connection (I can do "ping www.google.com"
> in a terminal and everything is fine).
> Also other VMs that use sys-net directly as netVM can access the internet (i.e. ping a server etc.).
> The only exception is sys-firewall, in which a ping just fails due to no connection.
>
> When sys-firewall starts up, a new vif is created inside sys-net (which was expected), but there is no route created.
> When
> I tried to create a new route it said "Network is down". So it did
> "ifconfig vif8.0 up" and afterwards added a new route with:
>
> "sudo ip route add 10.137.0.15 dev vif8.0 metric 32752"
>
>
> "route -v" displays:
> 10.137.0.15   0.0.0.0   255.255.255.255   UH   32752   0   0   vif8.0
>

I am confused, did you do this in sys-net or sys-firewall. Because sys-net would have a default route and a route for your Lan. You may have tripped the info which is fine.

my routing on sys-net looks like this:
-bash-4.4# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 ens5
10.137.0.15 0.0.0.0 255.255.255.255 UH 0 0 0 vif8.0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens5

You should not have needed to ifconfig vifX up. This is something that will need to be looked at later.

on sys-firewall, you are probably going to need to ifconfig eth0 up and you should have something like this:
-bash-4.4# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.137.0.14 0.0.0.0 UG 0 0 0 eth0
10.137.0.14 0.0.0.0 255.255.255.255 UH 0 0 0 eth0

if .14 is the ip of sys-net (ifconfig | grep -i ast)


from sys-firewall, try ping 8.8.8.8 (Google dns) or something else to remove dns resolution from the picture

also arp -an
to check you have connectivity to sys-net and arp resolution

> To unsubscribe from this group and all its topics, send an email to qubes-users...@googlegroups.com.

[thorsten...@gmail.com]

> Can you also try doing this against the template you're using for your sys-firewall?
>
> qvm-features fedora-26-minimal qubes-firewall 1

I did this and restarted everything, no difference.

> Yes probably. For reference, to check (or enable):
> - go to start menu/System Tools/Qube Manager
> - right click sys-net/Qube Settings/Services tab
> - clocksync should be in the list and ticked if not type clocksync and click on +
> - I think a full reboot is required. There are probably ways to avoid it...

clocksync is checked.

> I am confused, did you do this in sys-net or sys-firewall. Because sys-net would have a default route and a route for your Lan. You may have tripped the info which is fine.

In fact I left the default routes away and just focused on the missing one.
When I start sys-firewall a new network interface is added (vifx.0) where x is a number.
"ifconfig -a" displays:

vif3.0: flags=4098<BROADCAST,MULTICAST> mtu 1500
(and also 2 default interfaces: enp0s0 and lo, which are both UP and RUNNING)

I noticed here that "UP" / "RUNNING" is missing for the vif, therefore I have to "up" it myself.
This might be part of the problem, since it has to be running in order to add a new route (which should be done automatically).
So "route" in sys-net displays only the default routes:

Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 enp0s0
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s0

So if I add the route myself it additionally displays:

10.137.0.6 0.0.0.0 255.255.255.255 U 100 0 0 vif3.0

So far so good, the values in sys-net are looking "ok" to me now. Or am I missing something?

> on sys-firewall, you are probably going to need to ifconfig eth0 up and you should have something like this:
> -bash-4.4# netstat -nr
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> 0.0.0.0 10.137.0.14 0.0.0.0 UG 0 0 0 eth0
> 10.137.0.14 0.0.0.0 255.255.255.255 UH 0 0 0 eth0

On sys-firewall eth0 and lo are UP and RUNNING, but "route" takes around 20 seconds to finish and displays:

Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 0 0 0 eth0
gateway 0.0.0.0 255.255.255.255 UH 0 0 0 eth0

The long waiting time before "route" finishes makes me wonder...

I deleted the default routes and recreated them. I also restarted the eth0 interface.

When I try to ping 8.8.8.8 from sys-firewall I get:

From 10.137.0.6 icmp_seq=1 Destination Host Unreachable
From 10.137.0.6 icmp_seq=2 Destination Host Unreachable
...

I also switched the templates of sys-net and sys-firewall to debian-9, but the result is the same (vif down in sys-net, no route for vif).

The more I try to fix this, I get a feeling that the root of the problem lies inside sys-net.
It seems like the vif in sys-net does not get "up", which breaks the setup/initialization script (or maybe it breaks earlier, I don't know).

If I knew, which steps have to be done to set up network between a VM, sys-firewall and sys-net correctly, I could try to pinpoint the problem better.
What happens exactly behind the scenes when sys-firewall starts and uses sys-net as netVM?
Also I was thinking if iptables might be involved here?!

[Alex Dubois]

Yes looks good.

>
>
> > on sys-firewall, you are probably going to need to ifconfig eth0 up and you should have something like this:
> > -bash-4.4# netstat -nr
> > Kernel IP routing table
> > Destination Gateway Genmask Flags MSS Window irtt Iface
> > 0.0.0.0 10.137.0.14 0.0.0.0 UG 0 0 0 eth0
> > 10.137.0.14 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
>
> On sys-firewall eth0 and lo are UP and RUNNING, but "route" takes around 20 seconds to finish and displays:
>
> Destination Gateway Genmask Flags Metric Ref Use Iface
> default gateway 0.0.0.0 UG 0 0 0 eth0
> gateway 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
>
> The long waiting time before "route" finishes makes me wonder...

This is probably just because it tries to resolve the IPs and DNS times out. if you use netstat -nr, it should be fast.

>
> I deleted the default routes and recreated them. I also restarted the eth0 interface.
>
> When I try to ping 8.8.8.8 from sys-firewall I get:
>
> From 10.137.0.6 icmp_seq=1 Destination Host Unreachable
> From 10.137.0.6 icmp_seq=2 Destination Host Unreachable
> ...
>
>
> I also switched the templates of sys-net and sys-firewall to debian-9, but the result is the same (vif down in sys-net, no route for vif).
>
> The more I try to fix this, I get a feeling that the root of the problem lies inside sys-net.

Or the "physical" link between sys-net and sys-firewall. I believe there is a doc page (or maybe a thread here) on how to reconnect after a disconnection.

could you please do the arp -an after the ping 8.8.8.8
If you have a MAC address for sys-net, then you have "wire" connectivity, otherwise, it is where the pb is.

[thorsten...@gmail.com]

A friend was using my PC and forgot to logout, so I accidently posted with his account. So here it goes again:

> This is probably just because it tries to resolve the IPs and DNS times out. if you use netstat -nr, it should be fast.

Yes, using "netstat -nr" I get a result immediately in sys-firewall:

Destination Gateway Genmask Flags MSS Window irtt Iface

0.0.0.0 10.137.0.5 0.0.0.0 UG 0 0 0 eth0
10.137.0.5 0.0.0.0 255.255.255.255 UH 0 0 0 eth0

> could you please do the arp -an after the ping 8.8.8.8

"arp -an" in sys-net displays:
? (192.168.0.2) at xx:xx:xx:xx:xx:xx [ether] on enp0s0

(xx:xx:xx:xx:xx:xx is a valid mac address, I just replaced the actual values with X's)

"arp -an" in sys-firewall displays:

? (10.137.0.5) at <incomplete> on eth0

[Alex Dubois]

Yes, so the problem is that you don't have connectivity between the 2 VMs.
Could you try this:
qvm-prefs sys-firewall | grep netvm
it should say sys-net? Y/N

Based on the info in the Qubes Firewall doc page
Even if it states sys-net, let's try to force it again
qvm-prefs sys-firewall -s netvm sys-net

and try the arp -an in sys-firewall again

[thorsten...@gmail.com]

> Could you try this:
> qvm-prefs sys-firewall | grep netvm
> it should say sys-net? Y/N

yes, result is sys-net

> Even if it states sys-net, let's try to force it again
> qvm-prefs sys-firewall -s netvm sys-net

that command did not work due to wrong syntax, so I did:

qvm-prefs --set sys-firewall netvm sys-net

If sys-firewall is shut down, the command works.
If sys-firewall is running, the command fails with the error:
"no such preoperty: 'netvm'", in addition if sys-firewall is running while I do this, the eth0 interface is removed from sys-firewall.

> and try the arp -an in sys-firewall again

Still the same result:

? (10.137.0.5) at <incomplete> on eth0

Maybe I should try to look into the script(s) that are running when using "qvm-prefs --set sys-firewall netvm sys-net"?

[Alex Dubois]

Yes good idea. I would need to do so too to be able to help now... I'm not familiar with this part.