Route between sys-net and appvm without masquerading/snat at sys-firewall

[Tom Carroll]

Hello Qubes devs,

I'm working on a project where I don't want the AppVM to be masqueraded
as the sys-firewall from the perspective of sys-net (I desire to see the
ip address of the appvm). This is necessary for my application, as I
desire sys-net to perform per-appvm task, such as (as an example)
modifying the DSCP code point.

In summary (the short version), a ping initiated within the sys-firewall
will route onto the network (as observed from tcpdump on the pif in
sys-net), but a ping initiated in the appvm will not (I see appvm
traffic on the vif in sys-net, but not on the pif).

Here's greater detail. I remove the general MASQUERADE rule from
sys-firewall and clear the raw table in sys-net and sys-firewall. To be
comprehensive, I change the default chain policies to ACCEPT and remove
any drop-all rules. Then, I modify the routes in sys-net, to forward the
appvm back into the sys-firewall.

When I ping an external IP (say gateway) from the appvm, the packets can
be observed on the vif, put not the pif. If I do the same on the
sys-firewall, I observe traffic both on the vif and pif and at the end
device.

To further support diagnosis, I changed the ip addresses to /31 on the
vif between sys-net and sys-fireall, updating the routes where
necessary. Furthermore, I disabled proxy_arp on the vif in sys-net as
this should be the case of conventional routing. Moreover, I also
disabled rp_filter at all location. Even with these changes, I observe
the same behavior in that the sys-firewall will route out, but the appvm
stops at the sys-net.

Does anyone have insight why this doesn't work? I would assume this to
be straightforward routing.

I appreciate your help. Thanks in advance.

~ Tom

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Make sure you've added appropriate routes for returning packets - for
example in sys-net a route to appvm via sys-firewall. Without this,
you'll have two problems:
1. Returning packets will be dropped because of lack of route back to
the source
2. Outgoing packets will be dropped by reverse path filter
(/proc/sys/net/ipv4/conf/all/rp_filter), because it looks like IP
spoofing from sys-net perspective.

If you're using R4.0, check also nftables, see here:
https://www.qubes-os.org/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world

Generally, without MASQUERADE, each node of the network tree needs to
know routes to _every_ node in the subtree - not only those directly
connected. In your case it might be ok, but in general case it gets even
more complicated because you may want some exceptions (Proxy VMs that
tunnel the traffic using VPN of some kind, including Tor). And switching
netvm require route changes in multiple places.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlqhI/sACgkQ24/THMrX
1yxbaAgAibEV4Gevei0p80zgdfY0/pH24Inm0+IrUCHi56fT7MnKM/Caj9ip1FrI
3VGGBrszlOpMP7iTaMos7Jsach0dCSmtLb9R0jtkbuMt5Tm3QgBQIVyqiUViLBUP
b0yWdx2IOXgiof9KnMAuI7RXr4t/3I5AGTJuPaSsFIpNB5z3F5JvOi/G8G3KSNz+
U50Ewa9NNOKTxMO27G1PFWSUC9tMZdXEIVk8/ZCVTPaltFeoIlavNzb4SrmmiT/I
OsysT6QjNlN1tncw5v/JTeIFqSVlCs6srCVU63Jm8oFpCwE14SqiOV+b7XoYr5rF
XIEYet9Mz+1l/FQXBRtMeupGugZz1A==
=s7tZ
-----END PGP SIGNATURE-----

[Tom Carroll]

Marek -

On 03/13/2018 02:41 AM, Marek Marczykowski-Górecki wrote:
> If you're using R4.0, check also nftables, see here:
> https://www.qubes-os.org/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world

Thanks for this tip. It was nftables rulesets making my life hard.

~ Tom