nftables vs iptables

[mfreemon]

On 1/11/18 3:01 PM, Chris Laprise wrote:
> On 01/10/2018 03:47 PM, Connor Page wrote:
>> The official templates use nftables so shouldn’t be mixed with
iptables. I didn’t have time to learn about nftables, so just removed
nftables package from debian 9 template. YMMV.
>>
>
> Hmmm, I was just thinking how Qubes' own guest scripts still use
> iptables even in fedora-26.
>
> IIUC, iptables and nft are two different interfaces to netfilter. I
> don't know if it really matters, at least for the R4.0 window. I'd
> prefer to put the syntax change (for docs) off until a later release.

I was recently thrown by the mix of both nftables and iptables in R4.

The qubes docs don't clarify much. The qubes firewall scripts use nft.
Most of the discussion on the qubes website documentation is about
iptables, but there are also a few mentions of nft. The upgrade
instructions (going from R3.2 to R4) did not mention converting rules
from iptables to nftables. It looks like other related projects (one
example is qubes-tunnel) is using iptables.

Just reading a few things and trying to come up to speed, I get the
impression that nftables and iptables should not both by used at the
same time. Even if technically possible (i.e. both sets of rules
applied correctly), it strikes me as not a great idea to maintain packet
filtering rules in two different ways.

What is the best practice recommendation on this (for R4, Fedora 28
template)? Are we to be using, exclusively, nftables in R4?

[Chris Laprise]

The last I read about this (for 4.0) is that nftables is used in Fedora
Qubes code, but Debian Qubes is still using iptables. That still appears
to be the case since nftables is not installed in my debian-9 templates.

I've submitted qubes-tunnel to Qubes with iptables commands only, with
the intention to transition to nftables (or that other new interface in
Linux, name escapes me just now) for Qubes 4.1. Someone who is just
starting a project might be better off going with nftables.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Ivan Mitev]

... until yet another packet filtering mechanism replaces nftables (in
that case, bpfilter [1]).

I understand the rationale behind using nftables [2] but given how it is
widespread (hint: close to 0 even amongst seasoned sysadmins) IMHO it
wasn't worth it. The OP's post confirms there's quite some confusion
about how it interacts with iptables, and the official documentation is
far from helpful.
I'm quite proficient with iptables and networking in general but it took
me half an hour to understand how to tweak Qubes' nftables rules last
time I wanted to change something in the firewall, while I would have
done that task in less than one minute with iptables. I could have spent
a few hours learning nftables to improve the official doc but at my age
I prefer to spend time learning tech that significantly improves things
(eg. Qubes OS over standard linux distribution) over loosing time
learning stuff that is only marginally better.
Anyway - I digress :)

[1] https://old.lwn.net/Articles/747551/
[2]
https://github.com/QubesOS/qubes-issues/issues/1815#issuecomment-245109500

[Zrubi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 10/1/18 11:48 PM, mfreemon wrote:
> What is the best practice recommendation on this (for R4, Fedora
> 28 template)? Are we to be using, exclusively, nftables in R4?

The intended benefit was that in case of nftables qubes firewall not
needed to be reloaded all the time.

But: until nftables is not a complete iptables replacement, Qubes is
still needs iptables too.

My personal opinion that this mixed setup causing more confusion, and
do not provide any real benefits at all.

- --
Zrubi
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEmAe1Y2qfQjTIsHwdVjGlenYHFQ0FAluzQnQACgkQVjGlenYH
FQ20CQ//X1uSjJbzce00spcrL/tGtXZaYfP6BGlfzyExTPmdZlQB4TwtjCpe65MM
kaH5iUuw51FWeZgPbvn3Sk09QJQYwCbYuVzBl2+XpEky5ltFPqnRvpIe+HaSIeCM
YJwG4jrnss8uzWS04gJtvdOOB01dLe1HnckIvuU+QQk0aJuke9YqZmrdfl1vlAa9
iJ3+MYYIpBwEbXBcI0riBwLqUK7FqqmL39v1UFqx39IioS2q9XO1kO3RAhupwsre
Jxg781Z+6qbXdmE9lztF4CpNPccHcx+gtKBv4XthS8aqP40FVGvAaoAU4bKCedQC
HrI3Y8ZjcsE1kSQxkf3bHDWX6Q12lKbZC7HWfjGe3J6akgGKTIjkNfx8mbrG5/7N
1Os3OhBCJ4i4R+WEpCunyVPsZlc2CJM0SnKDXgTCWUKRPL0bZW6rm9MjvwJqBP/w
oV8vgR0JMNExMg+kSb5AsFdSKeaPu4sDmOhw3PwYRwNsluNxCTlwtLmyBQZDSiCV
xTH3gjD/Qaf+A9EBD4qt2Ie3d/FpvYhUQlor/PoRRZeJe5dkA1CLT6fUQv6lMwUY
o3tIdiR3w8fyKtlQK62FunBu7yjt7CiMJB3KWY4C1G9lonnJQdcxOV64IBJMUPLI
V3nKirJmAhvekT7HwTvsKqgsOSebD9K385rboyntdjsmBq45jQk=
=vNfP
-----END PGP SIGNATURE-----

[mfreemon]

I'm concerned about the confusion and unnecessary complexity here.

Network packet filtering is certainly (one of) those features that
software such Qubes needs to be solid on (in both design approach and
implementation detail).

Is the Qubes team confident in the current situation, such that users of
Qubes should not be concerned?

nb. This is not meant to be a criticism at all. I very much appreciate
the hard (and complicated) work going into Qubes. I'm just looking to
understand the current situation better so as to judge whether my
concern is warranted or not.

[mfreemon]

As an example: I'm wanting to enable some specific network traffic
between two qubes. The docs say to use iptables
(https://www.qubes-os.org/doc/firewall/#enabling-networking-between-two-qubes).
qubes-firewall-user-script also specifies iptables rules. But
qvm-firewall implements the rules it manages using nftables. So the
firewall VMs have both iptables rules and nftables rules in effect. And
these are different sets of rules. It's not that the iptables command
and the nft command are just two user interfaces showing the same packet
filtering rules. They are different packet filtering rules. This seems
like a receipt for disaster.

Is this the wrong forum for this discussion? Should this be on
qubes-devel, or an issue in qubes-issues at
https://github.com/QubesOS/qubes-issues/issues?

[mfreemon]

s/receipt/recipe/

[Ivan Mitev]

You'll definitely get more visibility on qubes-devel.

FWIW I'm not concerned about the complexity itself: I trust the Qubes
devs not to mess up.
IMHO the problem is that people proficient with iptables are not willing
to spend time learning yet another packet filter tool when iptables
works for 99.99% of the cases (+, as others pointed out, nftables is
still not feature complete wrt. iptables). For those users - an
overwhelming majority - Qubes' nftables firewall is a black box that is
difficult to understand/tweak/debug.

[floasretch]

On Monday, October 1, 2018 4:32 PM, Chris Laprise <tas...@posteo.net> wrote:
> I've submitted qubes-tunnel to Qubes with iptables commands only, with
> the intention to transition to nftables (or that other new interface in
> Linux, name escapes me just now) for Qubes 4.1.

I guess you mean BPF (Berkeley Packet Filter).

[unman]

I think this is the problem. I remember stalwarts hanging on to ipchains
for similar reasons. (I speak as someone who has clung on to iptables for
far too long.)
It seems to me that the few features lacking in nftables are only of
interest to people who are fully capable of learning a new tool. The
extras that nft brings completely outweigh the deficiencies.
nft provides tools to translate your iptables rules in to the new
syntax, so there's really no excuse for not diving in. Even if you have
minimal time, you can write your iptables rules and then translate them
to nft.

Qubes tries to provide a straightforward experience for relatively
inexperienced users, and the nft/iptables mix per distribution is a
compromise to that end.

The docs need to be updated to provide nft rules throughout.

[Illidan Pornrage]

Yeah, I use my qubes for network services a lot and it quadruples the
time I need to get stuff working.
At least explain whats going on in the docs.

[Illidan Pornrage]

^ So do I need to set rules in both or just one of them?

[unman]

I dont recommend mixing them for clarity.
I would use nft throughout.

[Ivan Mitev]

I don't think that's a good reason. Ipchains (and even ipfwadm before)
had major deficiencies - a crippling one being their lack of state.
Iptables addressed those issues and that's why everybody quickly
switched to it: the benefits it provided for people serious about
firewalling infinitely outweighed the time they'd have to spend learning
the tool.

FWIW I've been involved in huge/complicated firewalls the only issue we
had with iptables was when restoring thousands of rules: it took a bit
of time; and even then, the increasing power of PCs made it increasingly
negligible over time. Ipset was later included in the kernel and solved
our "issue".

So, as an advanced user of iptables I don't have/see any issue that
would be solved by nftables. Sure, one could translate its iptables
rules to nftables but the problem is learning how to use nftables later
on. For instance, why did it just take me more than 10 minutes to find
out how to simply list rules ?

`man nft` ? {list | flush} ruleset [family] ;

`nft list` -> error.

How/where am I supposed to find out that I have to type `nft list
tables` (which I found searching on a random post on the web). And how
are "tables" related to a "ruleset" ? etc.

Also, with bpfilter advertised to replace nftables/iptables (maybe
someday), people will be reluctant to learn nftables.

Just to be clear, I'm not arguing with you nor any of the devs - I'm
just stating my experience, which is pretty much the same among all the
"network" guys I know.

> Qubes tries to provide a straightforward experience for relatively
> inexperienced users, and the nft/iptables mix per distribution is a
> compromise to that end.

Probably, but it would be interesting to understand what exactly
nftables provides that iptables can't. Marek's post in the issue I've
linked to in another post mentioned something about whonix.

>
> The docs need to be updated to provide nft rules throughout.

^ this. + sample usage of nftables.

[David Hobach]

On 10/10/18 3:33 PM, unman wrote:

If I recall correctly (and this is what
https://github.com/QubesOS/qubes-issues/issues/1815#issuecomment-245109500
states, too), nftables were introduced in order to *not be used
manually*. I.e. all users should stick to the GUI, qvm-firewall and
iptables wherever possible. That also explains why there's still a lot
more documentation on iptables topics apart from the historical reason.

Since the Qubes firewall is inherently dynamic (rules added and removed
whenever VMs are started and stopped), user scripts sometimes did
interfere with the dynamic changes resulting in unusable rules or even
security issues. Or simply the question "why did my custom rule xyz
disappear?". So all of these dynamic changes went to nftables in 4.0 and
the iptables rules should remain pretty static during the runtime of a
Qubes system.

I added qubes-devel for clarification in case my statement above is
imprecise or plain wrong.

Anyway admittedly I also consider that design decision not perfect as
a) NATting with 2 firewalls can get weird if you're not 100% sure which
one is applied first. I second the aforementioned opinion in the qubes
devs' skills though, i.e. I guess this was thought through.
b) It adds overall complexity, i.e. adds to the chance of having a bug.
c) Some special stuff might still have to go to nftables.

Regarding the overall iptables vs nftables syntax discussion I honestly
don't see the point as pretty much all recent kernel firewall backends
support the iptables syntax for backwards compatibility anyway.

KR
David

[Chris Laprise]

From my standpoint, the only thing wrong with the R3.x firewall was
that (aside from treating custom/tunnel DNS as alien) there's no best
practice established for setting temporary rules outside the
qubes-firewall-user-script. I now realize that creating/describing such
a best practice would have been _easy_ and curtailed about 70% of the
confusion and mishaps... all that was required was a non-racy way to
restart qubes-firewall with the rule changes and documentation saying
this step is required. If I could make the choice, I'd supply a default
qubes-firewall-user-script that looks for ephemeral rules set in a place
like /var/run.

But that's moot and I have to say I also like the R4.0 firewall which
has another way of dealing with that issue.

I think the biggest issue with a transition to nft (besides the
documentation) is the apparent fedora-debian inconsistency. Once debian
has essentially the same nft-based code as fedora then people can choose
their favorite interface without incompatibility worries.