AppVMs using ProxyVM having DNS problems some days

[Markus Kilås]

Hi,

I am experiencing an issue with DNS queries in my AppVMs in R3.0.

Sometimes after booting up, the AppVMS that are connected to
sys-firewall are unable to do DNS lookups:
user@untrusted ~]$ dig qubes-os.org
; <<>> DiG 9.10.3-P3-RedHat-9.10.3-10.P3.fc23 <<>> qubes-os.org
;; global options: +cmd
;; connection timed out; no servers could be reached

The same command works in sys-firewall and netvm and any AppVM connected
directly to the netvm but not when going through sys-firewall. There are
no firewall rules added in the Qubes VM Manager and changing to allow
all network traffic for 5 minutes makes no difference.

Besides DNS lookups not working, the networking is working:
[user@untrusted ~]$ ping 104.25.119.5
PING 104.25.119.5 (104.25.119.5) 56(84) bytes of data.
64 bytes from 104.25.119.5: icmp_seq=1 ttl=56 time=31.4 ms

If I manually change the nameserver to the same as in sys-firewall the
resolving works also in the AppVM:

With IP from /etc/resolve.conf (sys-firewall):
[user@untrusted ~]$ dig @10.137.2.1 qubes-os.org
; <<>> DiG 9.10.3-P3-RedHat-9.10.3-10.P3.fc23 <<>> @10.137.2.1 qubes-os.org
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Instead with the netvm IP:
[user@untrusted ~]$ dig @10.137.5.1 qubes-os.org
; <<>> DiG 9.10.3-P3-RedHat-9.10.3-10.P3.fc23 <<>> @10.137.5.1 qubes-os.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5804
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;qubes-os.org. IN A

;; ANSWER SECTION:
qubes-os.org. 127 IN A 104.25.119.5
qubes-os.org. 127 IN A 104.25.118.5

;; Query time: 11 msec
;; SERVER: 10.137.5.1#53(10.137.5.1)
;; WHEN: Sun Feb 28 16:03:09 CET 2016
;; MSG SIZE rcvd: 73


Any idea what is going on here?


Cheers,
Markus

[Markus Kilås]

I think I solved this now.

After re-installing with V3.2-rc2 and restoring my VMs (including my old
netvm) I still had this problem from time to time.

So what I did was to start use the new sys-net VM as NetVM instead of my
restored old netvm (I manually copied over the network manager config,
private keys, certificates etc from the old VM to not have to
reconfigure that).

Since then, so far I have not seen the issue again.


Cheers,
Markus

[David Hobach]

Very similar issues here...

> I think I solved this now.
>
> After re-installing with V3.2-rc2 and restoring my VMs (including my old
> netvm) I still had this problem from time to time.
>
> So what I did was to start use the new sys-net VM as NetVM instead of my
> restored old netvm (I manually copied over the network manager config,
> private keys, certificates etc from the old VM to not have to
> reconfigure that).
>
> Since then, so far I have not seen the issue again.

I had renamed the sys-firewall VM back to its old "firewallvm" name
using Qubes manager after a fresh 3.1rc2 install (otherwise restoring my
backup wouldn't have worked: "could not find referenced firewallvm"
...). Maybe the sys-firewall name is hardcoded somewhere? I guess I'll
test renaming it back again soon...

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I think it's this issue:
https://github.com/QubesOS/qubes-issues/issues/1067

> > I think I solved this now.
> >
> > After re-installing with V3.2-rc2 and restoring my VMs (including my old
> > netvm) I still had this problem from time to time.
> >
> > So what I did was to start use the new sys-net VM as NetVM instead of my
> > restored old netvm (I manually copied over the network manager config,
> > private keys, certificates etc from the old VM to not have to
> > reconfigure that).
> >
> > Since then, so far I have not seen the issue again.
>
> I had renamed the sys-firewall VM back to its old "firewallvm" name using
> Qubes manager after a fresh 3.1rc2 install (otherwise restoring my backup
> wouldn't have worked: "could not find referenced firewallvm" ...).

Enable option "ignore missing" during backup restoration. This will use
default VMs in place of missing ones (default netvm, default template
etc).

> Maybe the
> sys-firewall name is hardcoded somewhere? I guess I'll test renaming it back
> again soon...

It shouldn't matter.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXoZ3UAAoJENuP0xzK19cs2+8H/2RvRRp8hJzdTdL25sW9k3pS
fBejELvPrYHyqcodoRRnUUdVzycld598Jgj7nxx3MSt+hwv90ueA7iti7PjtHYLE
f+dnubN/69I2ZaqOS36JmrarCAUPE32NLuE9bw/+cs/5l5X0tnkOODgI0ZWm11zm
9lZC0l/23gAhofxQvdirllvBa+6qYL8YfDrQSpznJq0lQmsrRvquL7P7n1+pKtwd
G0FY8zFJuNX9oEUuytdR0lgwDlZAIKRk2C8W0FWpELoZDDQE4slQUMsy2AEUx4cA
Dad5BBR/pbqgynsSV4NrjfdOF2BIrJ/Bi8N5J9ur46hmTiYiUGenVV3jXt7sNFk=
=bYHd
-----END PGP SIGNATURE-----

[Markus Kilås]

My guess was not that the issue was with the name but rather that my
restored netvm had some configuration (or similar) issue preventing the
resolving from working in some situations.

I have no idea if that makes sense or not, it was just a hypothesis of mine.

But the fact for me is that since I switched to use the stock sys-net VM
I haven't had the problem a single time yet.


Cheers,
Markus

[Markus Kilås]

Unfortunately, I was wrong.

After working perfectly for a few weeks now I have seen the issue again :(

- working networking in sys-net
- working networking in sys-firewall using sys-net
- ping/dig etc not working in AppVM when using sys-firewall
- working networking in AppVM when connecting directly to sys-net

Currently the only workaround I know of is to connect directly to
sys-net or reboot and hope for better luck...

Cheers,
Markus

[Markus Kilås]

On 08/13/2016 10:49 AM, Markus Kilås wrote:
> Currently the only workaround I know of is to connect directly to
> sys-net or reboot and hope for better luck...

Copying the values in /etc/resolv.conf from sys-firewall to the AppVM
as mentioned in the ticket also seems to work as workaround.

// Markus