Port Forward using iptables broken?

88 views
Skip to first unread message

joev...@gmail.com

unread,
Feb 9, 2018, 1:50:05 AM2/9/18
to qubes-devel
Fedora templates have a weird issue where the packet counter on the sys-net nat FORWARD chain does not increment. The PREROUTING chain does increment.

The commands work, my configuration is correct, as it was working on R3.2 and does work on R4.0 if using an older debian-8 template for sys-net and sys-firewall.

Using Debian-8 template that came with Qubes 4.0 RC2 or earlier.... does work as expected.
It has iptables 1.4, while fedora has iptables 1.6.
I can go back and forth between fedora-25/26 and debian-8, and it will work when on debian.

Debian-9 has a weird issue where the counter on the sys-net FORWARD chain does get incremented, but nothing is sent to sys-firewall. Verified with tcpdump.


Sys-Net:
iptables -t nat -A PREROUTING -i ens5 -p tcp -d <outside_ip_of_sys-net> --dport 2200 -j DNAT --to-destination <ip_of_sys-firewall> -m comment --comment 'PortFwd'
iptables -I FORWARD 2 -i ens5 -p tcp <ip_of_sys-firewall> --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'

Sys-Firewall:
iptables -t nat -A PREROUTING -i eth0 -p tcp -d <ip_of_sys-firewall> --dport 2200 -j DNAT --to-destination <ip_of_AppVM> -m comment --comment 'PortFwd'
iptables -I FORWARD 2 -i ens5 -p tcp <ip_of_AppVM> --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'

Target AppVM:
iptables -I INPUT 5 -p tcp --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'


SYS-NET
sudo iptables -vnL -t nat | grep 2200 ; sudo iptables -vnL FORWARD | grep 2200

3 180 DNAT tcp -- eth0 * 0.0.0.0/0 10.0.1.53 tcp dpt:2200 /* PortFwd */ to:10.137.0.6
0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 10.137.0.6 tcp dpt:2200 ctstate NEW /* PortFwd */


My configuration works flawlessly for many months on R3.2 and even works today when using debian-8 for my sys-net and sys-firewall.

And yes, of course my net.ipv4.ip_forward is 1

bow...@gmail.com

unread,
Feb 9, 2018, 2:33:02 AM2/9/18
to qubes-devel
On Friday, 9 February 2018 06:50:05 UTC, joev...@gmail.com wrote:
> Fedora templates have a weird issue where the packet counter on the sys-net nat FORWARD chain does not increment. The PREROUTING chain does increment.
>
> The commands work, my configuration is correct, as it was working on R3.2 and does work on R4.0 if using an older debian-8 template for sys-net and sys-firewall.
>
> Using Debian-8 template that came with Qubes 4.0 RC2 or earlier.... does work as expected.
> It has iptables 1.4, while fedora has iptables 1.6.
> I can go back and forth between fedora-25/26 and debian-8, and it will work when on debian.
>
> Debian-9 has a weird issue where the counter on the sys-net FORWARD chain does get incremented, but nothing is sent to sys-firewall. Verified with tcpdump.
>
>
> Sys-Net:
> iptables -t nat -A PREROUTING -i ens5 -p tcp -d <outside_ip_of_sys-net> --dport 2200 -j DNAT --to-destination <ip_of_sys-firewall> -m comment --comment 'PortFwd'
> iptables -I FORWARD 2 -i ens5 -p tcp <ip_of_sys-firewall> --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'

I suspect you have just a typo here in your email (-d is missing)
I usually also put --sport 1024:65535 (but it is not required, and I doubt adding it would fix the problem)

>
> Sys-Firewall:
> iptables -t nat -A PREROUTING -i eth0 -p tcp -d <ip_of_sys-firewall> --dport 2200 -j DNAT --to-destination <ip_of_AppVM> -m comment --comment 'PortFwd'
> iptables -I FORWARD 2 -i ens5 -p tcp <ip_of_AppVM> --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'

Typo also here, the interface is eth0 I think

>
> Target AppVM:
> iptables -I INPUT 5 -p tcp --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'
>
>
> SYS-NET
> sudo iptables -vnL -t nat | grep 2200 ; sudo iptables -vnL FORWARD | grep 2200
>
> 3 180 DNAT tcp -- eth0 * 0.0.0.0/0 10.0.1.53 tcp dpt:2200 /* PortFwd */ to:10.137.0.6
> 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 10.137.0.6 tcp dpt:2200 ctstate NEW /* PortFwd */
>
>
> My configuration works flawlessly for many months on R3.2 and even works today when using debian-8 for my sys-net and sys-firewall.
>
> And yes, of course my net.ipv4.ip_forward is 1

I'll try to have a look today on my Qubes R4 server.

awokd

unread,
Feb 9, 2018, 3:44:13 AM2/9/18
to bow...@gmail.com, qubes-devel
On Fri, February 9, 2018 7:33 am, bow...@gmail.com wrote:
> On Friday, 9 February 2018 06:50:05 UTC, joev...@gmail.com wrote:
>
>> Fedora templates have a weird issue where the packet counter on the
>> sys-net nat FORWARD chain does not increment. The PREROUTING chain does
>> increment.

I saw this too when trying to follow the port forwarding example in
https://www.qubes-os.org/doc/firewall/ . Mentioned it on qubes-users.
@adubois is researching as well.


joev...@gmail.com

unread,
Feb 9, 2018, 11:36:14 AM2/9/18
to qubes-devel
Yes, thanks for pointing out the typos. They are only mistakes in this post. I use a script running in dom0 to generate pretty much everything. The same script works when debian-8 is used. The interface is different depending on the template

bow...@gmail.com

unread,
Feb 9, 2018, 6:59:52 PM2/9/18
to qubes-devel
On Friday, 9 February 2018 16:36:14 UTC, joev...@gmail.com wrote:
> Yes, thanks for pointing out the typos. They are only mistakes in this post. I use a script running in dom0 to generate pretty much everything. The same script works when debian-8 is used. The interface is different depending on the template

I confirm I have the same issue.
Please however note that I have another PCI NIC connected to an AppVM (My qubes also act as a firewall for home network) and we have no issue connecting outbound.
Outbound connection as you know do not need the PRE-ROUTING rules, so also the problem is seen on the FORWARD rule, I suspect more the PRE-ROUTING rule is at fault and does not do its job.
I'll try to dig into this, however I won't have much time this week...

bow...@gmail.com

unread,
Feb 9, 2018, 7:02:09 PM2/9/18
to qubes-devel
Also, could you clarify if you've tested on FirewallVM and if here again Debian is OK and Fedora not. This might rule out issues with physical cards (which I suspect is not the problem as PRE-ROUTING does get the packet).

joev...@gmail.com

unread,
Feb 9, 2018, 7:13:00 PM2/9/18
to qubes-devel
Yes, if the template on sys-net is changed to Debian-8, but sys-firewall (FirewallVM) is left with fedora... sys-net does send the packet to sys-firewall, which then appears the same way... PREROUTING sees it, but FORWARD does not.

Thanks.

P.S.
Debian-9 has issues as well, but I didn't test thoroughly with that. And I think Fedora-25 was working prior to some updates. I do enable testing repos for these templates, but don't know what package is the culprit and don't know how to rollback.

joev...@gmail.com

unread,
Feb 9, 2018, 9:31:36 PM2/9/18
to qubes-devel
https://github.com/QubesOS/qubes-issues/issues/3556

> An idea: Debian don't have nftables installed by default, so
> qubes-firewal fallback to iptables. But not on Fedora - there nftables
> is used. This applies to both sys-net and sys-firewall.
>
> A quick test:
>
> 1. List rules:
>
> nft list table ip qubes-firewall
>
> 2. Add rule accepting traffic from eth0:
>
> nft add rule ip qubes-firewall forward meta iifname eth0 accept
>
> - --
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?

That did it!
Thanks so much for the quick resolve.

This was my results from `nft list table ip qubes-firewall`
```
table ip qubes-firewall {
chain forward {
type filter hook forward priority 0; policy drop;
ct state established,related accept
ip saddr 10.137.0.6 jump qbs-10-137-0-6
}

chain qbs-10-137-0-6 {
accept
drop
}
}
```
`nft add rule ip qubes-firewall forward meta iifname eth0 accept`
adds `iifname eth0 accept` to the bottom of chain forward

Is it intended that fedora uses both iptables and nft?
Are there any security implications for allowing iifname eth0 accept (in my case for fedora-26, ens5)?

joev...@gmail.com

unread,
Feb 9, 2018, 10:44:23 PM2/9/18
to qubes-devel
On Friday, 9 February 2018 03:44:13 UTC-5, awokd wrote:
> On Fri, February 9, 2018 7:33 am, bowabos wrote:
> > On Friday, 9 February 2018 06:50:05 UTC, joev...@gmail.com wrote:
> >
> >> Fedora templates have a weird issue where the packet counter on the
> >> sys-net nat FORWARD chain does not increment. The PREROUTING chain does
> >> increment.
>
> I saw this too when trying to follow the port forwarding example in
> https://www.qubes-os.org/doc/firewall/ . Mentioned it on qubes-users.
> @adubois is researching as well.

https://gist.github.com/Joeviocoe/6c4dc0c283f6d6c5b1a3f5af8793292b
Try this Portfwd script. It was modified to work with new fedora templates using nft.

Created an updated version for Qubes 4.0 (RC4 tested)
Portfwd.sh <vm> <port> <proto> | <vm> clear all

Command line specify the "VM, Port and Protocol"... or just "VM clear all" to undo previous.
Script will recursively configure iptables/nft for all proxyVMs in use.
Now uses comments on iptables to remove previous entries (no duplicates)

Works with Fedora 25/26 which uses nft rules along with iptables
Works with Debian 8/9 too

Alex Dubois

unread,
Feb 10, 2018, 7:31:50 AM2/10/18
to joev...@gmail.com, qubes-devel


Sent from my mobile phone.
Thanks. I may have a look later. I will first validate that it does not work as well with vanilla Fedora 26.

>
> --
> You received this message because you are subscribed to a topic in the Google Groups "qubes-devel" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/qubes-devel/0ixnn8G5dAg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to qubes-devel...@googlegroups.com.
> To post to this group, send email to qubes...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/8a0d97ad-843e-4a44-ae07-86885ad396d1%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Marek Marczykowski-Górecki

unread,
Feb 10, 2018, 4:45:30 PM2/10/18
to joev...@gmail.com, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Feb 09, 2018 at 04:12:57PM -0800, joev...@gmail.com wrote:
> On Friday, 9 February 2018 19:02:09 UTC-5, Alex Dubois wrote:
> > On Friday, 9 February 2018 23:59:52 UTC, Alex Dubois wrote:
> > > On Friday, 9 February 2018 16:36:14 UTC, joev...@gmail.com wrote:
> > > > Yes, thanks for pointing out the typos. They are only mistakes in this post. I use a script running in dom0 to generate pretty much everything. The same script works when debian-8 is used. The interface is different depending on the template
> > >
> > > I confirm I have the same issue.
> > > Please however note that I have another PCI NIC connected to an AppVM (My qubes also act as a firewall for home network) and we have no issue connecting outbound.
> > > Outbound connection as you know do not need the PRE-ROUTING rules, so also the problem is seen on the FORWARD rule, I suspect more the PRE-ROUTING rule is at fault and does not do its job.
> > > I'll try to dig into this, however I won't have much time this week...
> >
> > Also, could you clarify if you've tested on FirewallVM and if here again Debian is OK and Fedora not. This might rule out issues with physical cards (which I suspect is not the problem as PRE-ROUTING does get the packet).
>
> Yes, if the template on sys-net is changed to Debian-8, but sys-firewall (FirewallVM) is left with fedora... sys-net does send the packet to sys-firewall, which then appears the same way... PREROUTING sees it, but FORWARD does not.

An idea: Debian don't have nftables installed by default, so
qubes-firewal fallback to iptables. But not on Fedora - there nftables
is used. This applies to both sys-net and sys-firewall.

A quick test:

1. List rules:

nft list table ip qubes-firewall

2. Add rule accepting traffic from eth0:

nft add rule ip qubes-firewall forward meta iifname eth0 accept

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlp+rHcACgkQ24/THMrX
1ywR9gf/RJFy4TVihhweEh7ZqpwKTTD/JNgYCrl2nelvRwxl8awlqL/sxBBTGo39
byprAaL/Oe+6L4aX3d/tfbmpuJ7plHIJvm9PIxQ4SVj46iEcMRJIm1xQCjV8YtFu
bvAna5vrisuUuaEo/Kx1a7ee4gJTjHNUtTgA8N2ar+oL/csG2Vlz38zCVjAD8isf
HoCn8H35V4zvJoVXNuFTpSBplIlxa4ouryBWT9GQktBnZ1OPqdeiKotgFX2N5sJc
z01XQQ83HWJ+1/x+iGI9OoGidBKHI+izjSNhlyO70SW/9L1Xg+2NkaetJcO1VLHI
TaegOvEhZkvw2X6DVeeG5fGk1nYKXQ==
=evy9
-----END PGP SIGNATURE-----

joev...@gmail.com

unread,
Feb 10, 2018, 10:06:02 PM2/10/18
to qubes-devel

Alex Dubois

unread,
Feb 25, 2018, 11:27:46 AM2/25/18
to qubes-devel

I have opened a thread on the iptables mailing-list to try to go to the bottom of the reason why it stopped working with subject: iptables PREROUTING to-destination hit but no hit in FORWARD (advanced)

Alex Dubois

unread,
Feb 25, 2018, 4:39:00 PM2/25/18
to qubes-devel
On Saturday, 10 February 2018 21:45:30 UTC, Marek Marczykowski-Górecki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Fri, Feb 09, 2018 at 04:12:57PM -0800, joev...@gmail.com wrote:
> > On Friday, 9 February 2018 19:02:09 UTC-5, Alex Dubois wrote:
> > > On Friday, 9 February 2018 23:59:52 UTC, Alex Dubois wrote:
> > > > On Friday, 9 February 2018 16:36:14 UTC, joev...@gmail.com wrote:
> > > > > Yes, thanks for pointing out the typos. They are only mistakes in this post. I use a script running in dom0 to generate pretty much everything. The same script works when debian-8 is used. The interface is different depending on the template
> > > >
> > > > I confirm I have the same issue.
> > > > Please however note that I have another PCI NIC connected to an AppVM (My qubes also act as a firewall for home network) and we have no issue connecting outbound.
> > > > Outbound connection as you know do not need the PRE-ROUTING rules, so also the problem is seen on the FORWARD rule, I suspect more the PRE-ROUTING rule is at fault and does not do its job.
> > > > I'll try to dig into this, however I won't have much time this week...
> > >
> > > Also, could you clarify if you've tested on FirewallVM and if here again Debian is OK and Fedora not. This might rule out issues with physical cards (which I suspect is not the problem as PRE-ROUTING does get the packet).
> >
> > Yes, if the template on sys-net is changed to Debian-8, but sys-firewall (FirewallVM) is left with fedora... sys-net does send the packet to sys-firewall, which then appears the same way... PREROUTING sees it, but FORWARD does not.
>
> An idea: Debian don't have nftables installed by default, so
> qubes-firewal fallback to iptables. But not on Fedora - there nftables
> is used. This applies to both sys-net and sys-firewall.
>
> A quick test:
>
> 1. List rules:
>
> nft list table ip qubes-firewall
>
> 2. Add rule accepting traffic from eth0:
>
> nft add rule ip qubes-firewall forward meta iifname eth0 accept

Shall I test and document firewall.md all using nft if it all works (there are some incompatibility warning in the nftables wiki with iptables for nat that may need us to move fully to nft)?
I'll be happy to try (in my spare time and own pace) to submit PR for all the qubes firewall scripts in sys-net and sys-firewall if you think it is the right way forward.

Alex Dubois

unread,
Feb 28, 2018, 12:22:02 PM2/28/18
to qubes-devel

I've submitted a doc update in this PR 605 (https://github.com/QubesOS/qubes-doc/pull/605)

Reply all
Reply to author
Forward
0 new messages