Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

iproute2 fwmark doesn't work (or iptables --set-mark doesn't)

4,080 views
Skip to first unread message

Adam Wysocki

unread,
Dec 3, 2013, 8:45:33 AM12/3/13
to
Hi,

I just subscribed this group. Hello everyone!

I have a problem with setting up routing. In short, I want to mark certain
packets with iptables and then catch these marks with iproute2. However
Linux seems to ignore these marks.

I have a 3.11.4 kernel with CONFIG_IP_ADVANCED_ROUTER and CONFIG_IP_MULTIPLE_TABLES
set. Here's my setup:

- eth0: LAN
- IP 192.168.0.1, netmask /24
- It connects to the LAN

- eth1: WAN
- IP 192.168.1.180, netmask /24, gateway 192.168.1.1
- It connects to the Internet

- tap0: VPN (created with OpenVPN)
- Local IP 172.24.25.4, remote IP (and gateway) 172.24.25.2, netmask /24

I want to pass certain traffic through 192.168.1.1 (eth1) and certain through
172.24.25.2 (tap0) - local traffic as well as routed traffic. Rules will be
somewhat complicated (for locally-generated traffic and routed traffic), now
just let's assume that I want to mark all packets to go through 192.168.1.1.

I have added two new routing tables to /etc/iproute2/rt_tables:

#v+
2 vpn
3 wan
#v-

I've created two of them, so the main table will not have a default gateway.
This way I can avoid errors - if I make a mistake and traffic goes through
the main table, it won't get out.

Tables are then populated:

#v+
ip route flush table main
ip route add 192.168.0.0/24 dev eth0 table main
ip route add 192.168.1.0/24 dev eth1 table main
ip route add 172.24.25.0/24 dev tap0 table main

ip route flush table wan
ip route add 192.168.0.0/24 dev eth0 table wan
ip route add 192.168.1.0/24 dev eth1 table wan
ip route add 172.24.25.0/24 dev tap0 table wan
ip route add default via 192.168.1.1 dev eth1 table wan

ip route flush table vpn
ip route add 192.168.0.0/24 dev eth0 table vpn
ip route add 192.168.1.0/24 dev eth1 table vpn
ip route add 172.24.25.0/24 dev tap0 table vpn
ip route add default via 172.24.25.2 dev tap0 table vpn
#v-

And now rules. I want packets marked with 2 going through WAN and packets
marked with 3 going through VPN.

#v+
ip rule flush

# restore default rules (deleted by flush)
ip rule add from all lookup main prio 32766
ip rule add from all lookup default prio 32767

ip rule add fwmark 2 table wan
ip rule add fwmark 3 table vpn
#v-

Now the cache is reloaded...

#v+
ip route flush table cache
#v-

And iptables flushed:

#v+
iptables -t filter -F
iptables -t nat -F
iptables -t mangle -F
#v-

And now the tricky part. I can't make iptables mark packets (or can't
make iproute2 honor fwmark). I've tried different combinations of:

#v+
iptables -t mangle -A PREROUTING -j MARK --set-mark 2
iptables -t mangle -A POSTROUTING -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -m connmark --mark 2 -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j CONNMARK --set-mark 2
iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
#v-

And nothing. All commands succeeed, but traffic still gets through the
main table. Regarding to this:

http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.netfilter.html

it should work. What am I missing?

Kernel doesn't have CONFIG_IP_ROUTE_FWMARK (it's completely absent), but
I've read that in new kernels it is enabled by default and not found in
config.

Thanks for your time.

Cheers.

AW

Tauno Voipio

unread,
Dec 3, 2013, 9:46:23 AM12/3/13
to
Iptables and routing apply to layer 3 (network).
Your tunnel interface (tap0) is a layer 2 (data link) device.

If you're tunneling at data link layer, the firewall does
not do anything at it.

I'd start by changing the VPN to use tun0 instead.

--

Tauno Voipio

Adam Wysocki

unread,
Dec 3, 2013, 1:07:01 PM12/3/13
to
Tauno Voipio <tauno....@notused.fi.invalid> wrote:

> Iptables and routing apply to layer 3 (network).
> Your tunnel interface (tap0) is a layer 2 (data link) device.
>
> If you're tunneling at data link layer, the firewall does
> not do anything at it.
>
> I'd start by changing the VPN to use tun0 instead.

Hi,

Thanks for the reply, but I don't think it's the case here. Why:

1. For now I'm dealing only with two tables - "main" and "wan".
I'll use "vpn" table when I'll be able to correctly mark packets
to pass through "wan" table instead of "main" table. For now we
can assume that VPN doesn't exist, it will come into play later.

2. tap0 is on layer 2, but it's only an interface, I want to
redirect layer 3 traffic to this tunnel in the same way it
would be done with other, physical datalink interfaces (e.g.
Ethernet), basing on their assigned IP addresses.

Once again thanks for the reply.

Cheers.

AW

Pascal Hambourg

unread,
Dec 3, 2013, 5:58:31 PM12/3/13
to
Tauno Voipio a ᅵcrit :
>
> Iptables and routing apply to layer 3 (network).
> Your tunnel interface (tap0) is a layer 2 (data link) device.

So what ? eth0 and eth1 are layer 2 devices too.

> If you're tunneling at data link layer, the firewall does
> not do anything at it.
>
> I'd start by changing the VPN to use tun0 instead.

Total nonsense. Iptables and ip do not care about the type of interface.
As you wrote, they work at the layer 3.

Pascal Hambourg

unread,
Dec 3, 2013, 6:13:17 PM12/3/13
to
Adam Wysocki a ᅵcrit :
>
> And now the tricky part. I can't make iptables mark packets (or can't
> make iproute2 honor fwmark). I've tried different combinations of:

Which combinations exactly ? You cannot have all those rules together,
it just does not make sense.

> #v+
> iptables -t mangle -A PREROUTING -j MARK --set-mark 2

Fine, that should mark all incoming packets.

> iptables -t mangle -A POSTROUTING -j MARK --set-mark 2

Useless : POSTROUTING is after the routing decision, so it will have no
effet.

> iptables -t mangle -A OUTPUT -j MARK --set-mark 2

Fine, that should mark all locally generated packets.
I'd start with just the above two rules in PREROUTING and OUTPUT, append
a couple of LOG rules to print the packet mark in the kernel log and see
what happens.

> iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j CONNMARK --set-mark 2
> iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

There are a lot of rules with --restore-mark (copy the connection mark
to the packet mark), but only one rule setting the connection mark.

> And nothing. All commands succeeed,

Did you check the actual results by printing the routing tables, routing
rules and iptables ruleset ?

> but traffic still gets through the main table.

How do you know ?

Adam Wysocki

unread,
Dec 7, 2013, 7:36:10 AM12/7/13
to
Pascal Hambourg <boite-...@plouf.fr.eu.org> wrote:

Hi,

Sorry for the late reply, but I finally found some time to fiddle with it...

>> And now the tricky part. I can't make iptables mark packets (or can't
>> make iproute2 honor fwmark). I've tried different combinations of:
>
> Which combinations exactly ? You cannot have all those rules together,
> it just does not make sense.

Ok, I left only these rules that you said.

> I'd start with just the above two rules in PREROUTING and OUTPUT, append
> a couple of LOG rules to print the packet mark in the kernel log and see
> what happens.

It seems that marking is done properly, but iproute2 does not direct
matched packets to proper tables.

iptables -t mangle -A PREROUTING -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -j LOG --log-prefix "prerouting "

iptables -t mangle -A OUTPUT -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -j LOG --log-prefix "output "

It gave log lines with MARK=0x02:

Dec 7 13:27:32 somewhere kernel: [71053.449380] output IN= OUT=eth1 SRC=192.168.1.180 DST=89.72.158.60 LEN=107 TOS=0x00 PREC=0x00 TTL=64 ID=3020 DF PROTO=TCP SPT=43104 DPT=443 WINDOW=28723 RES=0x00 ACK PSH URGP=0 MARK=0x2
Dec 7 13:27:35 somewhere kernel: [71056.520793] prerouting IN=eth1 OUT= MAC=00:50:04:30:e1:90:74:ea:3a:bd:5d:10:08:00 SRC=89.72.158.60 DST=192.168.1.180 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=59395 DF PROTO=TCP SPT=443 DPT=44304 WINDOW=12600 RES=0x00 ACK URGP=0 MARK=0x2
Dec 7 13:27:38 somewhere kernel: [71059.616705] output IN= OUT=eth0 SRC=192.168.0.1 DST=192.168.0.3 LEN=116 TOS=0x10 PREC=0x00 TTL=64 ID=14433 DF PROTO=TCP SPT=22 DPT=33868 WINDOW=330 RES=0x00 ACK PSH URGP=0 MARK=0x2
Dec 7 13:27:38 somewhere kernel: [71059.617382] prerouting IN=eth0 OUT= MAC=00:0a:5e:60:ce:b0:00:19:7e:19:1e:10:08:00 SRC=192.168.0.3 DST=192.168.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58487 DF PROTO=TCP SPT=33868 DPT=22 WINDOW=1001 RES=0x00 ACK URGP=0 MARK=0x2

>> And nothing. All commands succeeed,
>
> Did you check the actual results by printing the routing tables, routing
> rules and iptables ruleset ?

Yes:

[gof@raid ~]$ ip r s
172.24.25.0/24 dev tap0 scope link
192.168.0.0/24 dev eth0 scope link
192.168.1.0/24 dev eth1 scope link

[gof@raid ~]$ ip r s t vpn
default via 172.24.25.2 dev tap0
172.24.25.0/24 dev tap0 scope link
192.168.0.0/24 dev eth0 scope link
192.168.1.0/24 dev eth1 scope link

[gof@raid ~]$ ip r s t wan
default via 192.168.1.1 dev eth1
172.24.25.0/24 dev tap0 scope link
192.168.0.0/24 dev eth0 scope link
192.168.1.0/24 dev eth1 scope link

[gof@raid ~]$ ip ru s
0: from all lookup local
32764: from all fwmark 0x3 lookup vpn
32765: from all fwmark 0x2 lookup wan
32766: from all lookup main
32767: from all lookup default

[gof@raid ~]$ sudo iptables -t mangle -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK set 0x2
LOG all -- 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4 prefix `prerouting '

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK set 0x2
LOG all -- 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4 prefix `output '

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

>> but traffic still gets through the main table.
>
> How do you know ?

Main table doesn't have default route, so:

[gof@raid ~]$ ping 1.2.3.4
connect: Network is unreachable

[gof@raid ~]$ telnet 1.2.3.4 1234
Trying 1.2.3.4...
telnet: Unable to connect to remote host: Network is unreachable

AW

Pascal Hambourg

unread,
Dec 7, 2013, 10:14:26 AM12/7/13
to
Adam Wysocki a ᅵcrit :
>
>>> but traffic still gets through the main table.
>> How do you know ?
>
> Main table doesn't have default route, so:
>
> [gof@raid ~]$ ping 1.2.3.4
> connect: Network is unreachable
>
> [gof@raid ~]$ telnet 1.2.3.4 1234
> Trying 1.2.3.4...
> telnet: Unable to connect to remote host: Network is unreachable

That's the result of the first routing decision which happens while the
packet is being built, before it is passed to iptables' OUTPUT chains
and marked. Thus the main routing table must have a default route
(possibly with a nonexistent gateway so that packets won't actually be
sent out using it) to avoid this error. However routing of incoming
packets should have worked.

Pascal Hambourg

unread,
Dec 7, 2013, 10:21:26 AM12/7/13
to
Pascal Hambourg a ᅵcrit :
>
> That's the result of the first routing decision which happens while the
> packet is being built, before it is passed to iptables' OUTPUT chains
> and marked.

One more side note about this : you can see in the OUTPUT chain log that
locally generated packets have an initial source address and output
interface as a result of this routing decision. The rerouting after the
OUTPUT chain will update the output interface, but not the source
address. So you may need to update it with SNAT or MASQUERADE.

logs...@gmail.com

unread,
Aug 3, 2014, 4:06:28 PM8/3/14
to
I am facing similar issue.

Does locally generated packets reach the OUTPUT chain after routing decision?
Should there be any entries in main table always
Any good solution.

=============
ip route flush cache
ip route flush table main

iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
iptables -t mangle -A OUTPUT -m owner --uid-owner squid -j MARK --set-mark 0x5
iptables -t mangle -A OUTPUT -j LOG --log-prefix "IPtab: "

ip route add 192.168.1.0/24 dev wlan0 src 192.168.1.103 table logu
ip route add default via 192.168.1.1 dev wlan0 table logu
ip rule add fwmark 0x5 table logu

ip route flush cache

========================
As squid user

$ ping 74.125.236.39
connect: Network is unreachable

Adding an entry like this in main table works
$ route add -net 0.0.0.0/0 dev wlan0

Regards
Logu

Pascal Hambourg

unread,
Aug 3, 2014, 5:22:11 PM8/3/14
to
logs...@gmail.com a ᅵcrit :
> I am facing similar issue.

Similar to what ? That thread is so old that my news server does not
have the messages any more.

> Does locally generated packets reach the OUTPUT chain after routing decision?

Yes. There is always a routing decision before the OUTPUT chains. There
may be a rerouting decision after the OUTPUT chains.

> Should there be any entries in main table always

Not necessarily, but the routing rules and routes must be able to route
the original packets before it is mangled in the OUTPUT chains,
otherwise it is discarded immediately.

Anyway, what would be the purpose of flushing the main routing table ?
0 new messages