dnf over VPN with qubes-updates-proxy

[Nemo]

I have a set of chained VMs set up like this

Net <- Firewall <- VPN <- Firewall-VPN <- TemplateVMs/AppVMs

While my AppVMs have perfect internet connection, I cannot get the Updates Proxy to work for my TemplateVMs.

Skipping the VPN does work fine:

Net <- Firewall <- TemplateVMs

The Net, Firewall, and VPN VMs are all based on fedora-24-minimal with the packages required for NetVMs (including those blocked by qubes-template-minimal-stub).

I've tried enabling the qubes-updates-proxy service on the VPN and the Firewall-VPN VMs without success. When I enable the service on the VPN dnf times out, and when I enable it on Firewall-VPN it immediately errors out.

The TemplateVM has "Allow Connections to Updates Proxy" checked.

VPN has blocked all traffic in the firewall except for traffic to my VPN ports.

Checking "Allow Connections to Updates Proxy" in VPN and Firewall-VPN doesn't have any effect.

What am I missing? How can I make this work?

[Nemo]

I should clarify - technically this is not dnf *over* VPN, I just want to enable dnf to connect around by VPN using qubes-updates-proxy.

[Chris Laprise]

If you set up the VPN as in the Qubes VPN doc... you could easily tweak
your config to do updates *through* the VPN by disabling the updates
proxy in the VPN and enabling it for firewall-VPN. But that's assuming
your VPN is configured to allow that kind of traffic (general Internet
access).

Going *around* it would have the updates proxy enabled for the VPN
(instead of firewall-VPN) with some modification to allow tinyproxy to
access the external network. For example, having the tinyproxy process
run as group "qvpn", which is the group that has access when using the
doc iptables configuration.

Also keep in mind the Fedora-minimal template has a small problem with
tinyproxy; Installation is normally blocked for some reason. That can
make it seem like the updates proxy refuses to work.

--

Chris Laprise, tas...@openmailbox.org
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Nemo]

Yes, I did follow the official documentation to create the proxy.

The only thing I've borrowed from the Rudd-O version is having Firewall downstream from VPN, and setting the VPN's firewall settings to block all traffic except that on my VPN's port.

Doing updates through the VPN would be perfect if possible.

Adding qubes-updates-proxy service to Firewall-VPN (and installing tinyproxy via tinyproxy.x86_64) causes an immediate connection error from dnf. Is that caused by the firewall rules I've added to VPN? Are they necessary, given a setup via the official documentation?

[Chris Laprise]

On 03/28/2017 08:14 AM, Nemo wrote:
> Yes, I did follow the official documentation to create the proxy.
>
> The only thing I've borrowed from the Rudd-O version is having Firewall
> downstream from VPN, and setting the VPN's firewall settings to block
> all traffic except that on my VPN's port.
>
> Doing updates through the VPN would be perfect if possible.
>
> Adding qubes-updates-proxy service to Firewall-VPN (and installing
> tinyproxy via tinyproxy.x86_64) causes an immediate connection error
> from dnf. Is that caused by the firewall rules I've added to VPN? Are
> they necessary, given a setup via the official documentation?

It depends on where the rules are set, but I think its probable the
added rules are blocking updates. This type of setup, with downstream
proxyVM handling the updates proxy, is working well for me.

Keep in mind the firewall already has a config to prevent any output not
initiated by the VPN client (i.e. OpenVPN, etc) so restricting by port
number may not be adding anything to link security.

[Nemo]

I'm really having a lot of trouble getting consistent results with the updates proxy. I've managed to break it on Firewall as well, despite only removing and then re-adding qubes-updates-proxy (as far as I can tell).

Could you please help me by listing the elements required for it to work?

Eg

* TemplateVM

** Firewall page

*** Allow connections to Updates Proxy: checked

* ProxyVM(can be VPN or Firewall)

** Firewall page

*** Allow access to 10.137.255.254:8082 (or just all)

** Services page

*** qubes-updates-proxy listed and checked

*** yum-updates-proxy must not be listed

** Packages

*** tinyproxy (tinyproxy.x86_64) must be installed

** CLI firewall rules

*** Official VPN documentation rules are fine, other rules might cause problems

* Net

** Must have internet access

Is there anything else?

--
You received this message because you are subscribed to a topic in the Google Groups "qubes-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/qubes-users/nJ8OkyHuqCw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to qubes-users+unsubscribe@googlegroups.com.
To post to this group, send email to qubes...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/46271c9f-ed60-9267-1ecd-8b41e228fdd1%40openmailbox.org.
For more options, visit https://groups.google.com/d/optout.

[Nemo]

> To unsubscribe from this group and all its topics, send an email to qubes-users...@googlegroups.com.

>
> To post to this group, send email to qubes...@googlegroups.com.
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/46271c9f-ed60-9267-1ecd-8b41e228fdd1%40openmailbox.org.
>
> For more options, visit https://groups.google.com/d/optout.

Here are the `--verbose` results from `dnf upgrade` in two scenarios:

Net < VPN < Firewall-VPN (fedora-24 and qubes-updates-proxy) < TemplateVM

[user@fedora-24-minimal-sys ~]$ sudo dnf -v upgrade
cachedir: /var/cache/dnf
Loaded plugins: builddep, noroot, debuginfo-install, needs-restarting, config-manager, copr, reposync, protected_packages, playground, download, qubes-hooks, generate_completion_cache, Query
DNF version: 1.1.10
Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f24&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f24&arch=x86_64 [Failed to connect to 10.137.255.254 port 8082: No route to host].
Error: Failed to synchronize cache for repo 'updates'

Net < VPN (fedora-24-minimal w/ tinyproxy and qubes-updates-proxy) < Firewall-VPN < TemplateVM

[user@fedora-24-minimal-sys ~]$ sudo dnf -v upgrade
cachedir: /var/cache/dnf
Loaded plugins: debuginfo-install, config-manager, reposync, needs-restarting, download, copr, Query, noroot, qubes-hooks, protected_packages, generate_completion_cache, playground, builddep
DNF version: 1.1.10
Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (28): Timeout was reached for https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64 [Connection timed out after 120002 milliseconds].
Error: Failed to synchronize cache for repo 'fedora'

[Chris Laprise]

On 03/28/2017 12:27 PM, Nemo wrote:
> I'm really having a lot of trouble getting consistent results with the
> updates proxy. I've managed to break it on Firewall as well, despite
> only removing and then re-adding qubes-updates-proxy (as far as I can tell).
>
> Could you please help me by listing the elements required for it to work?
>
> Eg
>
> * TemplateVM
> ** Firewall page
> *** Allow connections to Updates Proxy: checked
>
> * ProxyVM(can be VPN or Firewall)
> ** Firewall page

> *** Allow access to 10.137.255.254:8082 <http://10.137.255.254:8082> (or

> just all)
> ** Services page
> *** qubes-updates-proxy listed and checked
> *** yum-updates-proxy must not be listed
> ** Packages
> *** tinyproxy (tinyproxy.x86_64) must be installed
> ** CLI firewall rules
> *** Official VPN documentation rules are fine, other rules might cause
> problems
>
> * Net
> ** Must have internet access
>
> Is there anything else?

Where it says "Allow access to 10.137.255.254:8082"... I would get rid
of that on all proxyVMs and template VMs, at least while testing.
Firewall pages should also be set like default; For proxyVMs that is
"Allow access except + empty list + Allow DNS + Allow ICMP". The
qubes-updates-proxy service is enabled /only/ for the downstream proxyVM.

Once you have it working with default settings, you can try re-adding
your other rules one-by-one while testing them.

[Unman]

> Here are the `--verbose` results from `dnf upgrade` in two scenarios:
>
> Net < VPN < Firewall-VPN (fedora-24 and qubes-updates-proxy) < TemplateVM
>
> [user@fedora-24-minimal-sys ~]$ sudo dnf -v upgrade
> cachedir: /var/cache/dnf
> Loaded plugins: builddep, noroot, debuginfo-install, needs-restarting, config-manager, copr, reposync, protected_packages, playground, download, qubes-hooks, generate_completion_cache, Query
> DNF version: 1.1.10
> Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f24&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f24&arch=x86_64 [Failed to connect to 10.137.255.254 port 8082: No route to host].
> Error: Failed to synchronize cache for repo 'updates'
>
> Net < VPN (fedora-24-minimal w/ tinyproxy and qubes-updates-proxy) < Firewall-VPN < TemplateVM
>
> [user@fedora-24-minimal-sys ~]$ sudo dnf -v upgrade
> cachedir: /var/cache/dnf
> Loaded plugins: debuginfo-install, config-manager, reposync, needs-restarting, download, copr, Query, noroot, qubes-hooks, protected_packages, generate_completion_cache, playground, builddep
> DNF version: 1.1.10
> Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (28): Timeout was reached for https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64 [Connection timed out after 120002 milliseconds].
> Error: Failed to synchronize cache for repo 'fedora'
>

Hi

I think that it would be best to try to focus on one scenario, and get
that working right, rather than thrashing about.

So let's try the

Net < VPN < Firewall-VPN (fedora-24 and qubes-updates-proxy) < TemplateVM

scenario.

I think you have said that this works right for normal qubes attached
to the Firewall-VPN, and that the traffic all goes down the VPN tunnel.

On Firewall-VPN, make sure that you have the qubes-update proxy running:
'systemctl status qubes-updates-proxy' should show "running"
'netstat -tlp' should show tinyproxy listening on port 8082

'iptables -L -nv -t nat' will show you the nat table: there should be a
redirect rule in PR-QBS-SERVICES , taking traffic for destination
10.137.255.254 and sending it to dpt 8082.

'iptables -L -nv' will show you the filter table: you should see a rule
in the INPUT chain allowing traffic to port 8082.

If you append "-Z" to the iptables commands, this will zero the
counters. That means that you should be able to troubleshoot the problem
relatively easily.
Connect a template to the Firewall-VPN, and no other qubes.
Then attempt to update the template.
Run the two iptables commands, and look at the counters - you should be
able to see if you are blocking traffic anywhere, or if there is
something wrong.

Look at the man pages using 'man' for more information on these
commands.

hth

unman

[Nemo]

Thanks unman

I found your previous thread helpful: https://groups.google.com/forum/#!topic/qubes-users/lyaPQjb8Pxo

I've done the following troubleshooting steps on the FirewallVM -

-----------------------

iptables -L -nv -t nat

*** confirms that PR-QBS-SERVICE redirects traffic to dport 8082 (the packets counter increments when I attempt `dnf upgrade`) ***

Chain PR-QBS-SERVICES (1 references)
pkts bytes target prot opt in out source destination
3 156 REDIRECT tcp -- vif+ * 0.0.0.0/0 10.137.255.254 tcp dpt:8082

-----------------------

netstat -plnt

*** shows me tinyproxy listening on port 8082 ***

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8082 0.0.0.0:* LISTEN 550/tinyproxy

-----------------------

systemctl status qubes-updates-proxy

*** shows the proxy running ***

● qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
Loaded: loaded (/usr/lib/systemd/system/qubes-updates-proxy.service; enabled;
Active: active (running) since Tue 2017-03-28 19:31:52 EDT; 21min ago
Process: 522 ExecStartPre=/usr/lib/qubes/iptables-updates-proxy start (code=ex
Main PID: 550 (tinyproxy)
Tasks: 3 (limit: 512)
CGroup: /system.slice/qubes-updates-proxy.service
├─550 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
├─557 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
└─558 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf

Mar 28 19:31:52 sys-firewall-vpn systemd[1]: Starting Qubes updates proxy (tinyp
Mar 28 19:31:52 sys-firewall-vpn systemd[1]: Started Qubes updates proxy (tinypr

--------------------------------------------------------------

To help me understand how qubes-updates-proxy is working, is this more or less accurate?:

The proxy gives the TemplateVM's network connection permission to break through it's own firewall's "Deny All" setting, for the purpose of updates only.

The proxy should be applied on a FirewallVM before hitting a VPN/NetVM. The FirewallVM will block all traffic, but proxy the repo request, which it receives via tinyproxy at 10.137.255.254:8082. The request will pass through the FirewallVM and arrive in the VPN/NetVM as a normal repo request.

Is that right?

------------------------------------------

To (maybe?) confuse things further:

I just realized that the TemplateVMs will not update via Firewall-VPN even if they are set to allow all traffic. Although they will still update via the Net < Firewall < TemplateVM chain either directly or through the proxy without issue.

[Nemo]

Hmmm, actually I missed running `iptables -L -nv`

[user@sys-firewall-vpn ~]$ sudo iptables -L -nv
Chain INPUT (policy DROP 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

0 0 DROP udp -- vif+ * 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Seems I'm filtering all this traffic, which would cause problems...

I tried recreating Firewall-VPN from scratch, and ran `iptables -L -nv` immediately after adding qubes-updates-proxy

[user@sys-firewall-vpn2 ~]$ sudo iptables -L -nv
Chain INPUT (policy DROP 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

0 0 DROP udp -- vif+ * 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
1 52 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Doesn't seem like 8082 is automatically added. How can I add the record?

[Unman]

You can add the rule like this:
'sudo iptables -I INPUT -p tcp --dport 8082 -j ACCEPT'

'-I INPUT' Inserts the rule at the top of the INPUT chain (You can
specify a number here, like '-I INPUT 2' to specify position.)

-p tcp = specifies Protocol is tcp
--dport = Destination PORT

Try that and see if it works for you.
If this is the solution, (and I think it is), the you can add this line
in /rw/config/qubes-user-firewall-script - look at the docs on the Qubes
firewall to help here.

This is a known issue in proxyVMS - in fact I've fixed it and that code
is merged but, I guess, you havent yet got it.

Make sure you clean up any other changes you may have made getting to
this point.

unman

[Zrubi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/28/2017 02:14 PM, Nemo wrote:
> Doing updates through the VPN would be perfect if possible.

For this you can simply skip the updates proxy, and let your template
access the same networks as you appVMs.

You just have to edit the /etc/dnf/dnf.conf in your template, and
comment out the qubes proxy line.

Of course this will disable the original "template protection" where
you can only reach the updates proxy.


- --
Zrubi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJY23AqAAoJEH7adOMCkunmBJMP/RL0i7xF/tY3+xdtvMuXiCUt
W6jsuUcvsC2VlTsW2WSskBnixGEZuZlFaTxo6yA1HXpRCTgOiJm6mHrmLLcyV4im
jbuEpkI/wcoKodSzyhvQ6NBIj1PUBZehuOIbK3NoBkasnzqSDQ/cSB30z72QljOn
S/T/njAQ3PFju3+yA6l7jrE7gJiICXecR2zwN78Y2b9KdxSs/JwnQBVdh+ahvxI4
A4fEF45gvtJ+5QkK3A2zmrnkhl39k/b54NjglN993LL0uUB4j8cbgUUjG6mjnPYj
YEW4MlvSKbAHp8wzwDpWT1yb1zCzDt2F45K3ILDwO0G119TBC1Ur8qwz5YELRwcm
9ZDrDK9gLaVfWxj4aBv4N1ecZZu9pVN8RKtFsFHuvFGkCjLoDPVKjkgpSLMhDNpi
aUAo61cpsVxgE6AocepbHykrbrhyQnncUO0LcZT1Ozl9g9hCyFT5LgMvRN3k9vRb
QwNZVptA2ktps7X9BtjPSjcYUgdhEFxqxxt85Sy0BS61bfH2UlgPzEIr+I7XRxaa
SIj6yUG9SpRTz/HBaP2Rt3IzKkwLzkOaQDRKebNofsl8X+z9fF3Oq7cVOSFqj2X5
LM+t/NXnkwCciwbD0V5m4v9qNorj5GYTdzlauAROLA4tRmAqPddOl7rfhMqnnlXs
S7P4liRGjwwbyYeK02y6
=h4kr
-----END PGP SIGNATURE-----

[Nemo]

Thank you! This did fix it.

Is the proxyVMS update included in the updated qubes-core-vm and qubes-core-vm-systemd packages? If not, how can I make sure I get updates like this in the future?

[Unman]

On Wed, Mar 29, 2017 at 06:02:17AM -0700, Nemo wrote:
> >
> > You can add the rule like this:
> > 'sudo iptables -I INPUT -p tcp --dport 8082 -j ACCEPT'
> >
> > '-I INPUT' Inserts the rule at the top of the INPUT chain (You can
> > specify a number here, like '-I INPUT 2' to specify position.)
> >
> > -p tcp = specifies Protocol is tcp
> > --dport = Destination PORT
> >
> > Try that and see if it works for you.
> > If this is the solution, (and I think it is), the you can add this line
> > in /rw/config/qubes-user-firewall-script - look at the docs on the Qubes
> > firewall to help here.
> >
> > This is a known issue in proxyVMS - in fact I've fixed it and that code
> > is merged but, I guess, you havent yet got it.
> >
> > Make sure you clean up any other changes you may have made getting to
> > this point.
> >
> > unman
>
> Thank you! This did fix it.
>
> Is the proxyVMS update included in the updated qubes-core-vm and qubes-core-vm-systemd packages? If not, how can I make sure I get updates like this in the future?
>

Yes, that update will find its way in to the stable updates, after it's been
through testing. If you make sure you keep your system updated you will
get that fix.