Sporadic Inter-VM Routing Issues with Qubes Updates Proxy

[Kiwi17]

Hi, I was hoping someone may be able to help make heads or tails of this frustrating issue I'm having.

Background

I use a VPN configured as-per the Qubes recommended config for VPNs (https://www.qubes-os.org/doc/vpn/).

I have used this configuration with the following VM hierarchy for some months without a problem: sys-net -> sys-firewall -> vpn -> vpn-firewall -> *

[where "vpn-firewall" runs the qubes-yum-proxy service (verified TCP listener is showing up in netstat on  0.0.0.0:8082)]

Problem

Recently I have encountered a problem where whenever I go to update a TemplateVM, or dom0 - any VM that is configured to use the qubes update proxy - the dnf update times out. The following is the output of "sudo dnf -vvv --refresh update" on a Fedora 26 TemplateVM:

Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f26&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (28): Timeout was reached for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f26&arch=x86_64 [Connection timed out after 30003 milliseconds].

Error: Failed to synchronize cache for repo 'updates'

If we watch netstat during this attempted update, we see that a SYN is sent to the correct update proxy address of 10.137.255.254:8082, but no SYN-ACK is received:

tcp        0   1 10.137.5.14:57914       10.137.255.254:8082     SYN_SENT

Leaving this running, no TCP connection is ever established with the qubes-updates-proxy service at "vpn-firewall". Similarly, watching for inbound connections on "vpn-firewall" yields no results for an incoming connection from the TemplateVM. During this time, all AppVMs continue to have full network connectivity via the vpn-firewall gateway.

Now here's the weird bit... The problem is sporadic. Sometimes I can reboot my host machine and the updates proxy is broken, other times it works fine.

To my untrained eye, this appears to be a routing issue internal to Xen. Does anyone have some advice on how I can investigate further?

Many thanks in advance,

Alex

Sent with ProtonMail Secure Email.

[wordsw...@gmail.com]

Some thoughts that may or may not be useful:

- qubes-updates-proxy should always be running on the firewall that is closest to the vpn. So if you are doing something like

sys-net->sys-firewall->sys-vpn->sys-firewall-vpn->sys-firewall-work

then qubes-updates-proxy should be running on your sys-firewall-vpn.

- Check that you've enabled the qubes-updates-proxy service on the sys-firewall-vpn Settings in Qubes VM Manager

- Check that the service is running on sys-firewall-vpn

sudo service qubes-updates-proxy status

If you're running your firewall with restricted memory then in my experience tinyproxy *sometimes* fails to start. This minimal memory requirement seems to be higher for Fedora 26 than 25.

- Check your dnf settings "cat /etc/dnf/dnf.conf" on your TemplateVM to confirm that it's set up to use the proxy. There should be a line at the bottom similar to

proxy=10.137.255.254

- Try to update the TemplateVM without using the proxy