TemplateVM updates almost instantly fail when target is VPN qube but dom0 updates run just fine
Sphere
$type:TemplateVM $default allow,target=VPN
As soon as I execute a sudo dnf update to my template VM, it takes a little less than a second for it to go
"Failed to synchronize cache for repo 'updates'"
"Error: Failed to synchronize cache for repo 'updates'"
dom0 updates isn't suffering from this which is why I don't really get what the problem is. At first I thought it's the custom resolv.conf being used for dnscrypt-proxy but that doesn't seem to be the case after trying to use the dns server of the vpn itself
Web browsing and everything else is working as intended through the VPN qube and only TemplatVM updates are failing. I don't think there is a problem with what I set on /etc/qubes-rpc/policy/qubes-UpdatesProxy
But I don't really face this problem when the target is set to the net VM. That however is no good since I'm looking to utilize the VPN for Template updating.
Is TemplateVM updating dependent on whether or not net VM is capable of connecting to the internet and domain name resolutions?
unman
presumably *not* set to VPN.
Without knowing how your VPN qubes is configured, it's difficult to say
what's going on.
Are you able to browse from VPN?
Have you configured VPN qube to provide updateProxy service?
unman
Chris Laprise
earlier updated reliably.
I think it has something to do with my update vm being a qube dedicated
to that purpose: sys-update has to start before the template can receive
updated. However, earlier versions of Fedora would wait instead of
timing-out immediately.
So this looks like a bug that should have an issue opened for it.
--
Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886
Sphere
I'm using a fedora minimal and I don't see any sort of package requirement for a qube to install to provide TemplateVM updates as I read through the Fedora Minimal documentation. Only thing that has a requirement from the looks of it is a qube for dom0 updates.
Thanks for the suggestion about updateProxy service. I suppose I should set it to provide qubes-yum-proxy, is that right?
@Chris Thank you for telling me that as I too am using fedora-30, specifically fedora-30-minimal Templates
I am trying to do a setup where a qube is dedicated for the purpose of updating, something similar to yours I believe.
I'll try to work on this tomorrow and report results back to this topic.
unman
> @unman The dom0 updates setting is set correctly and working as intended through the VPN qube, I haven't tried browsing from the VPN qube itself but I can definitely browse from an AppVM connected to it and I can confirm that all the browsing being done there is tunneled through the VPN.
>
> I'm using a fedora minimal and I don't see any sort of package requirement for a qube to install to provide TemplateVM updates as I read through the Fedora Minimal documentation. Only thing that has a requirement from the looks of it is a qube for dom0 updates.
>
> Thanks for the suggestion about updateProxy service. I suppose I should set it to provide qubes-yum-proxy, is that right?
>
I think the other is a red herring.
Chris Laprise
> On Wed, Jun 26, 2019 at 02:54:04AM -0700, Sphere wrote:
>> @unman The dom0 updates setting is set correctly and working as intended through the VPN qube, I haven't tried browsing from the VPN qube itself but I can definitely browse from an AppVM connected to it and I can confirm that all the browsing being done there is tunneled through the VPN.
>>
>> I'm using a fedora minimal and I don't see any sort of package requirement for a qube to install to provide TemplateVM updates as I read through the Fedora Minimal documentation. Only thing that has a requirement from the looks of it is a qube for dom0 updates.
>>
>> Thanks for the suggestion about updateProxy service. I suppose I should set it to provide qubes-yum-proxy, is that right?
>>
>
> The new name is qubes-updates-proxy service: enable that.
> I think the other is a red herring.
debian and earlier versions of fedora.
Also would like to clarify that my updatevm is a dedicated fedora-30 VM
'sys-update' that is downstream from the VPN VM; sys-update is often in
stopped state and has to start when updates are performed, but not sure
if that matters.
I don't see anything obvious in the fedora-30 log except for a very
large number of 'success' messages like:
Jun 26 09:47:18 fedora-30 audit[1]: SERVICE_START pid=1 uid=0
auid=4294967295 ses=4294967295
msg='unit=qubes-updates-...@13-127.0.0.1:8082-127.0.0.1:45048
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=?
terminal=? res=success'
Jun 26 09:47:18 fedora-30 systemd[1]:
qubes-updates-...@13-127.0.0.1:8082-127.0.0.1:45048.service:
Succeeded.
I wonder if its restarting over and over.
Chris Laprise
little more persistent and may succeed sometimes.
I believe the culprit is with the fedora-30 version of the updates proxy
– it seems to be un-blocking the update procedure before it is ready to
handle traffic.
Sphere
I also noticed that qubes-updates-proxy.service fails by default on startup and I'm unsure if that is a minimal template-only problem but I was able to fix it thanks to it indicating that the problem is a missing folder: /var/run/qubes-service/qubes-updates-proxy
Pretty much the same problem that I get with clocksync service thankfully so I was able to confirm that this service was running as intended
systemctl status qubes-updates-proxy:
qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
Loaded: loaded (/usr/lib/systemd/system/qubes-updates-proxy.service; enabled;
vendor preset: enabled)
Active: active (running) since Thu 2019-06-27 12:06:14 +08; 2s ago
Process: 1603 ExecStartPre=/usr/lib/qubes/iptables-updates-proxy start (code=e
xited, status=0/SUCCESS)
Main PID: 1608 (tinyproxy)
Tasks: 3 (limit: 414)
Memory: 4.1M
CGroup: /system.slice/qubes-updates-proxy.service
├─1608 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
├─1609 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
└─1610 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
Jun 27 12:06:14 redacted systemd[1]: Starting Qubes updates proxy (tinyproxy)...
Jun 27 12:06:14 redacted systemd[1]: Started Qubes updates proxy (tinyproxy).
Jun 27 12:06:14 redacted tinyproxy-wrapper[1608]: Found tinyproxy at /usr/bin/tinyproxy
Despite this however, the problem still persists and still behaves the same even after trying dnf update for 5 times
I think is right about the fact that there is a bug about this
@Chris I think you may be right about the fact that this is a bug and I guess it's time to escalate it into an issue in github. I'm willing to lend a helping hand in making the issue as needed.
My setup is all fully dependent on variations of fedora-30-minimal template that I have tailored depending on use-case of the AppVM that would be using it.
unman
> @unman: thanks for that
> I also noticed that qubes-updates-proxy.service fails by default on startup and I'm unsure if that is a minimal template-only problem but I was able to fix it thanks to it indicating that the problem is a missing folder: /var/run/qubes-service/qubes-updates-proxy
>
> Pretty much the same problem that I get with clocksync service thankfully so I was able to confirm that this service was running as intended
>
> systemctl status qubes-updates-proxy:
> qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
> Loaded: loaded (/usr/lib/systemd/system/qubes-updates-proxy.service; enabled;
> vendor preset: enabled)
> Active: active (running) since Thu 2019-06-27 12:06:14 +08; 2s ago
> Process: 1603 ExecStartPre=/usr/lib/qubes/iptables-updates-proxy start (code=e
> xited, status=0/SUCCESS)
> Main PID: 1608 (tinyproxy)
> Tasks: 3 (limit: 414)
> Memory: 4.1M
> CGroup: /system.slice/qubes-updates-proxy.service
> ??????1609 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
> ??????1610 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
> Jun 27 12:06:14 redacted systemd[1]: Starting Qubes updates proxy (tinyproxy)...
> Jun 27 12:06:14 redacted systemd[1]: Started Qubes updates proxy (tinyproxy).
> Jun 27 12:06:14 redacted tinyproxy-wrapper[1608]: Found tinyproxy at /usr/bin/tinyproxy
>
> Despite this however, the problem still persists and still behaves the same even after trying dnf update for 5 times
>
> I think is right about the fact that there is a bug about this
>
> @Chris I think you may be right about the fact that this is a bug and I guess it's time to escalate it into an issue in github. I'm willing to lend a helping hand in making the issue as needed.
>
> My setup is all fully dependent on variations of fedora-30-minimal template that I have tailored depending on use-case of the AppVM that would be using it.
>
Unlike you and Chris I don't see the behaviour you report.
Let's try to dig in before raising a bug report.
I've tested this with 30-minimal template 201905071541 and 201906241949,
from stable and testing.
I've tested against dom0 stable and dom0 testing: both fully updated.
Test boxes are an old x230 and a custom rig with X-series CPU and 32G RAM.
In all cases, the proxy is started as appropriate, and the update
process (from fedora 29 and 30-minimal) waits until proxy is up and then
proceeds.
What hardware are you, Sphere and Chris, running?
Sphere - if you create a dedicated update qube using the 30-minimal with
qubes-core-agent-networking installed,
enable the qubes-updates-proxy service, route it through
sys-firewall, and edit the policy file appropriately, do you see the
same behaviour? (Almost instant fail)
What if you start the new update proxy before attempting a 'dnf update'?
unman
Sphere
Big update: I was able to solve the problem
What I essentially did:
1. Ensure to run the Update Qube first
2. Confirm and ensure that the qubes-updates-proxy is already running after the qube is started. qubes-updates-proxy was listed and set as checked in the services tab of Qubes Settings GUI before starting the update qube.
checking was done through the `systemctl status qubes-updates-proxy` command.
3. Ensure that qubes.UpdatesProxy policy file is configured correctly before starting the templateVM
4. Ensure that DNS queries are resolving in the update qube
5. Start the templateVM and try to do a dnf update
One big thing to note here is that I encountered the problem after step 4 and was able to solve it by ensuring that my update qube is able to properly resolve DNS queries but I have to say that what's unique in my situation is that I use DNSCrypt for resolving DNS queries.
So basically, the problem was solved after I ran DNSCrypt on the update qube.
Admittedly that was kinda dumb on my part to not realize that the f30 template definitely needs to have DNS resolutions to do updating along with that fact that I have already blocked all plaintext DNS from going out.
However, I can't quite remember whether or not I had DNSCrypt running on the update qube last time I tested it so there's a possibility that strictly doing the first 2 steps that I did contributed greatly in solving the problem.
For the purpose of troubleshooting this problem however, the qube that I used to update and the qube that I used for VPN is one and the same. I guess I'll try to use separate ones next week to see how it goes (I have none to very minimal online activity throughout the weekend).
@Chris: Maybe you could try what I did and see how it goes?
Chris Laprise
templates usually on the first try, but it always takes 3 or more tries
to update fedora-30.
I may try changing my update vm to debian-10 to see if that helps. The
problem started when I switched the update vm from fedora-28 to fedora-30.
Sphere
So far so good there were no problems given the fact that I ensured that the qube responsible for being updates proxy to the Templates were resolving DNS queries just fine and updates proxy service actively running properly in that qube.
If you still got the problem after switching to debian-10 maybe do some network diagnosis in your update vm? I can lend a helping hand with that if you need it