Bug in qubes-firewall.service could result in VPN leak

[skiinglasso2]

There's a bug in qubes-firewall.service. It should pull in and be ordered before network-pre.target such that the firewall rules are guaranteed to be in place before the network is raised.

From man sytemd.special,

network-pre.target

           This passive target unit may be pulled in by services that want to

           run before any network is set up, for example for the purpose of

           setting up a firewall. All network management software orders

           itself after this target, but does not pull it in.

From https://systemd.io/NETWORK_ONLINE/

network-pre.target is used to order services before any network interfaces start to be configured. Its primary purpose is for usage with firewall services that want to establish a firewall before any network interface is up. Services that want to be run before the network is configured should use Before=network-pre.target and Wants=network-pre.target.

I suggest applying this change so that people who are currently relying on this popular guide https://forum.qubes-os.org/t/configuring-a-proxyvm-vpn-gateway/19061 can continue to do so without having to make modifications to systemd themselves.

[qubist]

On Sun, 23 Mar 2025 06:40:23 +0000 'skiinglasso2' via qubes-devel wrote:

> There's a bug in qubes-firewall.service. It should pull in and be
> ordered before network-pre.target such that the firewall rules are
> guaranteed to be in place before the network is raised.

How do you detect the leak?

According to the same link you refer to, there is no established
network connectivity before network-online.target which starts after
network.target:

user@sys-firewall:~ > systemctl cat network-online.target | grep After
After=network.target

qubes-firewall.service starts before network.target, i.e. even earlier:

user@sys-firewall:~ > systemctl cat qubes-firewall.service | grep Before
Before=qubes-network.service
user@sys-firewall:~ > systemctl cat qubes-network.service | grep Before
Before=network.target
user@sys-firewall:~ > systemctl cat network.target | grep After
After=network-pre.target

I don't know if it is not possible (or necessary) to have it
Before=network-pre.target because the virtual interfaces (vif*) are
part of the nft rules. (See /etc/xen/scripts/vif-route-qubes)

[unman]

On Sun, Mar 23, 2025 at 06:40:23AM +0000, 'skiinglasso2' via qubes-devel wrote:
[quote]

There's a bug in qubes-firewall.service. It should pull in and be ordered before network-pre.target such that the firewall rules are guaranteed to be in place before the network is raised.

[/quote]
I dont think this is a bug in practice, but you are right that it would
be better to do this. In fact we have an open issue that covers this.

--
I never presume to speak for the Qubes team.
When I comment in the mailing lists I speak for myself.

[skiinglasso2]

How do you detect the leak?

I haven't attempted to capture the leak.

> According to the same link you refer to, there is no established

> network connectivity before network-online.target

You've misinterpreted the reference. It is saying that services configuring interfaces must run **after** network-pre.target. It is also saying that services that contribute to the network being online must run **before** network-online.target.

> qubes-firewall.service starts before network.target, i.e. even earlier:

The services that contribute to the network being up run before network-online.target, which can also be before network.target.

> I don't know if it is not possible (or necessary) to have it

> Before=network-pre.target because the virtual interfaces (vif*) are

> part of the nft rules. (See /etc/xen/scripts/vif-route-qubes) \

It is possible and necessary to have it before network-pre.target. In fact, it already can/does run before network-pre.target, it just isn't

configured such that this is guaranteed. I have tested this and it works. If this is introducing some invisible problem, then you can create

another service that only runs /rw/config/qubes-firewall-user-script.

> I dont think this is a bug in practice, but you are right that it would

> be better to do this.

I do think it is a bug in practice. Doing things improperly because a path is unlikely to be encountered is bad.

[skiinglasso2]

> In fact we have an open issue that covers this.

I couldn't find one that references this specific issue. Which issue are you referring to? Can you add this information to the issue?

Can you please allow me to post on the forums? My post has been pending for 48 hours even though multiple moderators have been online and responding to other threads, which is not very professional. I want to update the VPN guide as it sounds like this isn't going to be resolved in the near future.

I would suggest doing something to make it easier for people to contribute to this project. It is made very difficult for people using proxies because this mailing list is gated by a Google captcha. The archive link is down pretty much always. The forums prevents signing up with proxies until a moderator accepts it which is apparently slow. GitHub prevents signing up with proxies. Frustration after frustration.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

There is one more missing piece above: qubes-firewall.service is ordered
before qubes-network.service. And it's only the latter that enables IP
forwarding. So, at any point before that, no network traffic is
forwarded.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmfgkFsACgkQ24/THMrX
1yyxbwf+OonjRYcDdVcUP6RM5W6Ne0w3wT5O0yjKIzwsbAJH0dOmZecQQEvhFbga
4TwM0Sq4psTfovvZGs/8HeX8xnkvKa27bf4X1FSoj4N8006UhXysJD4xBONE/6S+
IqTNXkOrJYTB57eX50Mm+qwPDx5CPIaNEqA81NxamZ12KoTbKnhgQn5+QwmgRfwP
JAmB60pKEJClH2jIQTF/D/iWY9DAdkz/kuF3gHy6Xt2LdRQqygy7qYw3Q7584j3T
ZXme1Lg2dAFtKtutaH6uBencVTRgn9pI9ValQj8SF84n57HaUtR12dip3HuVn9d3
8DrordnUStLZaCOaQmdJAqbeTFdf2Q==
=lsTH
-----END PGP SIGNATURE-----

[qubist]

On Sun, 23 Mar 2025 23:51:07 +0100 Marek Marczykowski-Górecki wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Sun, Mar 23, 2025 at 11:32:52AM -0000, qubist wrote:
> > On Sun, 23 Mar 2025 06:40:23 +0000 'skiinglasso2' via qubes-devel wrote:
> >

> > [...]

> > user@sys-firewall:~ > systemctl cat qubes-firewall.service | grep Before
> > Before=qubes-network.service

> > [...]

>
> There is one more missing piece above: qubes-firewall.service is ordered
> before qubes-network.service.

Missing?

[qubist]

On Sun, 23 Mar 2025 22:31:43 +0000 'skiinglasso2' via qubes-devel wrote:

> I haven't attempted to capture the leak.

Then it is not reproducible, i.e. not a bug, so unman is right.

> > According to the same link you refer to, there is no established
> > network connectivity before network-online.target
>
> You've misinterpreted the reference.

Compare what you quoted to the reference:

"Network connectivity has been established: network-online.target"

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ah, indeed it's there. The thing about network-online.target is that
it's ordered after network is up. This means anything that requires
network to work, should be ordered after network-online.target. But
there is no guarantee in the other direction - ordering stuff before
network-online.target doesn't guarantee network is not up yet. For this
theoretically there is network-pre.target as you mentioned later in your
response. Adding Before=network-pre.target to qubes-firewall may work,
but as explained in my response, it isn't really necessary. Note also
that qubes-firewall is only about configuring firewall for forwarded
traffic. Base firewall for the qube itself (input rules etc) is set in
qubes-iptables.service and that has Before=network-pre.target.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmfhQsQACgkQ24/THMrX
1yy4cgf+IxzsS+r1mpmz9q1XIMK+GYoyDpn+v0bnrgIS92kkeVd4V26SYooxn/Fh
/5MJnf/UPDCyTpciSUjsXx9BhK2qhzpJJS8U6skWEawdpk4scAo8wNZH29CPhnXd
tIFD+2peA/b/bF3G9wVckcCyYV+s8uBi7Vjh/p4JdNRfhSKsG/ukzUaDGzaEIcvG
SRMdRQp2+LlWQshEZFBJbJlrDLOsvg4eiKy91EHuVFjRUClSZsh9JZdMl+e7i0aF
gF5rQktrbRvOLVBoNZR5KXbOmVrOA9UsiR440HJXh0+TjvGO0uQbjy9R6PPLe4Jb
YUBKryMoEBzZ61/09gAofh+OmRrDnw==
=OtyS
-----END PGP SIGNATURE-----

[qubist]

On Mon, 24 Mar 2025 12:32:20 +0100 Marek Marczykowski-Górecki wrote:

> Adding Before=network-pre.target to qubes-firewall may work,
> but as explained in my response, it isn't really necessary.

Exactly what I tried to explain too.

[skiinglasso2]

> Then it is not reproducible, i.e. not a bug, so unman is right.

Firstly, I said I didn't attempt to capture it, not that it isn't reproducible. Secondly, not reproducible implies not a bug? Absurd claim. What are you talking about?

> Compare what you quoted to the reference:

> Exactly what I tried to explain too.

My explanation of why you misunderstood the reference is **exactly** the same as what Marek explained. Your initial explanation was relying purely on systemd ordering semantics, not the key detail that Marek added about forwarding.

> Adding Before=network-pre.target to qubes-firewall may work,

> but as explained in my response, it isn't really necessary. Note also

> that qubes-firewall is only about configuring firewall for forwarded

> traffic. Base firewall for the qube itself (input rules etc) is set in

> qubes-iptables.service and that has Before=network-pre.target.

I admit my initial concern is no longer valid, but I think it still make sense to put these rules before network-pre.target. These rules are commonly used for things other than forwarding. Look at the VPN guide I linked, they use these rules to only allow vpn-process traffic out eth0. If the rules are only related to forwarding then at the very least the name of the script and/or a comment within should include this.

On Sunday, March 23rd, 2025 at 6:40 AM, skiinglasso2 <skiing...@proton.me> wrote:

[qubist]

On Mon, 24 Mar 2025 22:06:49 +0000 'skiinglasso2' via qubes-devel wrote:

> > Then it is not reproducible, i.e. not a bug, so unman is right.
>
> Firstly, I said I didn't attempt to capture it, not that it isn't
> reproducible.

No steps to reproduce = nothing to reproduce => not reproducible.

> Secondly, not reproducible implies not a bug? Absurd claim.

I find it absurd to claim that there is "a leak" which you have neither
seen, nor even attempted to detect. A bug report requires steps to
reproduce.

> What are you talking about?

Sanity. Facts. Logic.

> My explanation of why you misunderstood the reference is **exactly**
> the same as what Marek explained. Your initial explanation was
> relying purely on systemd ordering semantics, not the key detail that
> Marek added about forwarding.

Marek simply explained what I was hoping you would understand from the
systemd sequencing I showed. There is no network traffic forwarding
before network.target. If you can explain how a "leak" is possible
without network traffic, that would be a revolution.

> I admit my initial concern is no longer valid, but I think it still
> make sense to put these rules before network-pre.target. These rules
> are commonly used for things other than forwarding.

This is quite different from:

[skiinglasso2]

> I don't know if it is not possible (or necessary) to have it

> Before=network-pre.target

> Marek simply explained what I was hoping you would understand from the

> systemd sequencing I showed.

You didn't know yourself but at the same time you were hoping I would understand from your explanation? Drop the ego, you've already looked a fool in front of your boss.

There's still a leak, just not from the AppVMs but from the VPN VM itself. Meaning processes other than the VPN process are able to send traffic out eth0, for example.

def add(a, b):

   return a - b

If I sent you this function and told you there was a bug, would you be asking me to provide steps to reproduce the bug? The only difference is that this code is more complex. Apparently too complex for you.

You're a fool with a dangerous mindset qubist. Please hire better Qubes.

[qubist]

On Tue, 25 Mar 2025 09:57:01 +0000 'skiinglasso2' via qubes-devel wrote:

> You didn't know yourself but at the same time you were hoping I would understand from your explanation?

My "I don't know" is regarding the necessity for what you are requesting. Marek's explanation confirms that there isn't any. In case you still don't understand, the sequence is:

qubes-iptables.service
network-pre.target

qubes-firewall.service
qubes-network.service <-- No network traffic forwarding before this
network.target
network-online.target <-- Network connectivity has been established

[unman]

On Tue, Mar 25, 2025 at 09:57:01AM +0000, 'skiinglasso2' via qubes-devel wrote:
@skiinglasso2 there's no reason for you to get carried away. Personal
attacks dont help your argument.
I've suggested you propose a change in the documentation to reflect what
you see as an issue. Will you do that?

[Andrew David Wong]

On 3/25/25 2:57 AM, 'skiinglasso2' via qubes-devel wrote:
> [...] in front of your boss. [...] Please hire better Qubes.

1. Personal attacks are a violation of the Code of Conduct:
https://www.qubes-os.org/code-of-conduct/

2. The person you're replying to does not work for the Qubes OS Project. This is a public mailing list. Anyone with an email address can participate.

[skiinglasso2]

> I've suggested you propose a change in the documentation to reflect what

> you see as an issue. Will you do that?

I have updated the entire guide which I will share soon.

Leaving this as a community guide is really not the way this should be done. Any number of changes made by Qubes could break the firewall and cause leaks.

Can someone at Qubes confirm whether it is safer to rely on "eth0" or ifgroup 1 for referring to the upstream interface. Are these expected to remain stable forever?

Another issue I've discovered with qubes-firewall.service is that it will succeed even if the user firewall script fails. If this is something you will change then the guide I will submit will use qubes-firewall.service, otherwise I will have to create a new service that detects failures which makes the guide even more difficult for non-technical users to use.

[Demi Marie Obenour]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Mar 26, 2025 at 06:04:40AM +0000, Qubes OS Development Mailing List wrote:
> > I've suggested you propose a change in the documentation to reflect what
> > you see as an issue. Will you do that?
>
> I have updated the entire guide which I will share soon.
>
> Leaving this as a community guide is really not the way this should be done. Any number of changes made by Qubes could break the firewall and cause leaks.
>
> Can someone at Qubes confirm whether it is safer to rely on "eth0" or ifgroup 1 for referring to the upstream interface. Are these expected to remain stable forever?

Please use ifgroup 1.

> Another issue I've discovered with qubes-firewall.service is that it will succeed even if the user firewall script fails. If this is something you will change then the guide I will submit will use qubes-firewall.service, otherwise I will have to create a new service that detects failures which makes the guide even more difficult for non-technical users to use.

Please submit a PR.
- --
Sincerely,
Demi Marie Obenour (she/her/hers)

Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEopQtqVJW1aeuo9/sszaHOrMp8lMFAmfktaMACgkQszaHOrMp
8lN8phAAmW+lCw1tC+v+FCCyjIi7vzdJXFuAjxSNXtNtKqB8DK5BpHG36p0BaLVR
494bvoaeq0SvG18wNaOw2+CqnDXmwRg0BdHwpICmPlFZQ24qGJ6JlKAqAeJ+SylU
gAzhMS1iyQ1ouyr83dOVXvvenev3JZiHJbtsTaNaX7+WZnIMwr1kdA1pZqaDqZ+p
PGfh6ZROHj1mRy3nTZWf3nmv2XEqsTQuj+S5p8+YtXuujQllaYA/4TDE+YSUxJg9
3NSAmDGw6DAVea/Q2u2OGdAyaVqcMX4mX0uh810QVi54Fn6O3K+hiBnD/RK1xv0Z
tjoCa+C2IKaLnzuLPpbgGJ5uksCQ8/PxFvlaGtdPmljwi7TcnrcckxeE1MJ0gmg0
XbPdMdiOsf6xovvJzlkpiad5Y8sA03c9Hb7dJtA0IADyy/TR7eDmOJy+hKg73LYb
EXH8sqUB5biKqRhbqLIN+v+eSzZWhLUrdCHTXDoKX9oizb7TOsVyRUKjqry9jqHt
gLPBi5cyRPkmkT+8OVz+KdAfRnwwPVJ6ypJWg7vKmJHKca0Z63xrIavL4xu24sgW
OMtbO0YY+UkmFHZAryL+7cLGH7TZlBu1ZtuAHJRimyZiksrjSFh5ldCyJG1t4PkQ
OeUXTS7KY44hLCnp3oB7BnDZMk9/SfSLQON0594s8Pz3ddnq0X0=
=syTI
-----END PGP SIGNATURE-----

[Demi Marie Obenour]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Mar 25, 2025 at 09:57:01AM +0000, Qubes OS Development Mailing List wrote:
> > I don't know if it is not possible (or necessary) to have it
> > Before=network-pre.target
>
> > Marek simply explained what I was hoping you would understand from the
> > systemd sequencing I showed.
>
> You didn't know yourself but at the same time you were hoping I would understand from your explanation? Drop the ego, you've already looked a fool in front of your boss.
>
> There's still a leak, just not from the AppVMs but from the VPN VM itself. Meaning processes other than the VPN process are able to send traffic out eth0, for example.

It is best to not run any process that may send sensitive information in
the VPN VM itself. The VPN must always be able to send information out
eth* interfaces, but qubes behind it do not. The latter can be
expressed via very simple rules, such as:

iifgroup 1 drop
oifgroup 1 drop

in a forward rule.

Instead of requiring users to write custom scripts, it would be much
better to allow users to configure a qube as a VPN qube in the GUI, and
for Qubes OS to set up these rules automatically.

- --
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEopQtqVJW1aeuo9/sszaHOrMp8lMFAmfkuJsACgkQszaHOrMp
8lP+rg/+N4utvKz2kKFZ2XjvJuLsZE2LoCIcpvkgg6voq12fzPqNJuzVe/RRJ+7E
omMkVrrfYHSVJ065Sz3VYKhG2PkOFrwSMJ+gENFg/tPT4B/YVgLN9Oqkelec4sq/
e4mwbpIeMRMfQhpRFCPGykQ9zQY3Q5stu5OBrExG5JKhf94faRJyTWIDDT/Pf/e4
CcuiBtAQb60qu5g+MscXMmrfcIZpJiY2E99+Q/wIt31teGbgZ2Ug3x9rC8WE1HTs
atBNn3UksWD0kvBxp1zfKSJiC5lOUsHfvvCDPbuhmE+5BU6/i4JTnuLwC0j3dRO1
b5qc/vgZ9Bo52zcKlWuvurYha3u6LLrJW20vcbT5HfN+y+5maU+kW7NePHnX+ofO
itEbCW2yFwiqIK74JY7/ytfiLqUsye6+TWK1SS5BfRI9t+iULW7gmGy3jVVzWzLN
VqbVwBENGaFyngGS9OhPqqc8DIKShWht05Qr0SbINp0gsFrScjYxCNVoPJP4naCp
TdDTKs2qm65Hcj1NCQ9/L/McG8MAJGsB3p6u2kd4AWfSZg9fJ+/nMoPacOJwcwX8
tgTquRK7tAHUqOnTSj907I69JljmdNBEV2ZH4tHTGixnxQqaPzEYGfvLSqVwHow7
0hmCKLYfFVN2Fz+zPklpWIsHSwZDVGAX5+RWyVijwMROgUZosD0=
=4gWv
-----END PGP SIGNATURE-----

[skiinglasso2]

I have added new firewall rules to https://forum.qubes-os.org/t/configuring-a-proxyvm-vpn-gateway/19061/58

These rules need to be reviewed. I will do it myself when I have the chance but that will be in a few months.

On Sunday, March 23rd, 2025 at 6:40 AM, skiinglasso2 <skiing...@proton.me> wrote: