ipfw nat on 14.4 not working

[Philipp]

Hi

Since I updated to 14.4 release my nat setup only works for icmp. I have
one public interface and two epair interfaces. The epairs (vnet42,
vnet43) are connected to a bridge and on the bridge are a bunch of jails.

I can connect to the service of the jails from the host itself. But the
ports redirects don't work and the jails can't use tcp/udp to hosts on
the internet.

The public interface has the address 80.190.133.201 and the vnet interface
have the address 10.188.42.1/24 and 10.188.43.1/24. the vnet* epairs are
connected to a bridge. The jails are created by ezjail and the jib[0] tool
is used to create epairs and attach to this bridge. I haven't had time to
look at ether_gen_addr so it's off.

the ipfw.rules looks like this:

#!/bin/sh
# Flush out the list before we begin.
ipfw -q -f flush

pif="em1"

#public interface
pif=em1

ipfw nat 123 delete
ipfw nat 123 config ip 80.190.133.201 redirect_port tcp 10.188.42.84:80 80 redirect_port tcp 10.188.42.1:22 22
# redirect_port tcp 10.188.43.54:2222 2222 \
# redirect_port tcp 10.188.43.55:2223 2223 \
# redirect_port tcp 10.188.42.84:443 443 \
# redirect_port tcp 10.188.42.86:25 25 \
# redirect_port tcp 10.188.42.86:587 587 \
# redirect_port tcp 10.188.42.46:993 993 \
# redirect_port tcp 10.188.42.46:4190 4190

#${fwcmd} add 00120 nat 123 ip4 from any to any via $pif
ipfw add 00120 nat 123 ip from 10.0.0.0/8 to any via $pif #out via $pif
ipfw add 00121 nat 123 ip from any to 80.190.133.201 via $pif #in via $pif
ipfw add 00122 check-state
# ICMP
ipfw add 00130 allow icmp from any to any
ipfw add 00131 allow icmp6 from any to any

ipfw add 00132 allow ip from 10.0.0.0/8 to any keep-state

#ipfw add 00140 skipto 200 ipv6 from 2a01:138:9000:bc00::/56 to any out via $pif
#ipfw add 00141 skipto 200 ipv6 from me6 to any out via $pif
#ipfw add 00142 skipto 200 ipv6 from any to 2a01:138:9000:bc00::/56 in via $pif
#ipfw add 00143 skipto 200 ipv6 from any to me6 in via $pif
#ipfw add 00144 skipto 200 ipv4 from any to me in via $pif
#ipfw add 00145 skipto 200 ipv4 from me to any out via $pif
#ipfw add 00150 drop ipv6 from any to any via $pif
#ipfw add 00151 drop all from any to any frag in via $pif
#ipfw add 00152 drop ipv4 from any to any via $pif

ipfw add 00210 allow tcp from any to any 22 via $pif # keep-state
ipfw add 00211 allow tcp from any to any 25 in via $pif # keep-state
ipfw add 00212 allow tcp from any to any 587 in via $pif # keep-state
ipfw add 00213 allow tcp from any to any 993 in via $pif #keep-state
ipfw add 00214 allow tcp from any to any 2222 in via $pif # keep-state
ipfw add 00215 allow tcp from any to any 4190 in via $pif # keep-state
ipfw add 00216 allow tcp from any to any 2223 in via $pif # keep-state
ipfw add 00217 allow udp from any to any 60000-61000 in via $pif
ipfw add 00218 allow tcp from any to any 80 in via $pif
ipfw add 00219 allow tcp from any to any 443 in via $pif
ipfw add 00220 allow all from any to any out via $pif keep-state
#ipfw add 00221 allow all from any to any in via $pif keep-state
#ipfw add 00230 allow tcp from any to 2a01:138:9000:bc00::/56 in via $pif
#ipfw add 00231 allow udp from any to 2a01:138:9000:bc00::/56 in via $pif
#ipfw add 00232 allow tcp from any to 80.190.133.201 in via $pif
#ipfw add 00233 allow udp from any to 80.190.133.201 in via $pif

# vnet42
ipfw add 41999 allow all from 10.188.42.0/24 to any in via vnet42 keep-state
ipfw add 42000 allow all from 2a01:138:9000:bc42::/64 to any in via vnet42 keep-state
#ipfw add 42001 allow tcp from 10.188.42.0/24 to any in via vnet42 keep-state
#ipfw add 42002 allow udp from 10.188.42.0/24 to any in via vnet42 keep-state
ipfw add 42003 allow all from any to any 80 via vnet42 keep-state
ipfw add 42004 allow tcp from any to 2a01:138:9000:bc42::2e 993 out via vnet42 keep-state
#ipfw add 42005 allow tcp from any to 10.188.42.46 993 out via vnet42 keep-state
ipfw add 42006 allow tcp from any to 2a01:138:9000:bc42::15e4:2d4d 587 out via vnet42 keep-state
ipfw add 42007 allow tcp from any to 2a01:138:9000:bc42::15e4:2d4d 25 out via vnet42 keep-state
#ipfw add 42009 allow tcp from any to 10.188.42.86 25 out via vnet42 keep-state
#ipfw add 42010 allow tcp from any to 10.188.42.86 587 out via vnet42 keep-state
ipfw add 42011 allow tcp from any to 2a01:138:9000:bc42::2e 4190 out via vnet42 keep-state
#ipfw add 42012 allow tcp from any to 10.188.42.46 4190 out via vnet42 keep-state
ipfw add 42013 allow all from any to 2a01:138:9000:bc42::53 53 out via vnet42 keep-state
ipfw add 42014 allow tcp from any to 2a01:138:9000:bc42::54 443 out via vnet42 keep-state

# vnet43
#ipfw add 42999 allow all from 10.188.43.0/24 to any in via vnet43 keep-state
ipfw add 43000 allow all from 2a01:138:9000:bc43::/64 to any in via vnet43 keep-state
ipfw add 43002 allow all from 10.188.43.0/24 to any in via vnet43 keep-state
ipfw add 43003 allow udp from any to 10.188.43.0/24 60000-61000 out via vnet43 keep-state
ipfw add 43004 allow udp from any to 2a01:138:9000:bc43::/64 60000-61000 out via vnet43 keep-state
ipfw add 43013 allow tcp from any to 10.188.43.0/24 22 out via vnet43 keep-state
ipfw add 43014 allow tcp from any to 2a01:138:9000:bc43::/64 22 out via vnet43 keep-state
ipfw add 43023 allow tcp from any to 10.188.43.0/24 2223 out via vnet43 keep-state
ipfw add 43024 allow tcp from any to 2a01:138:9000:bc43::/64 2223 out via vnet43 keep-state
# git ssh port
ipfw add 43100 allow tcp from any to 2a01:138:9000:bc43::36 2222 out via vnet43 keep-state
ipfw add 43101 allow tcp from any to 10.188.43.54 2222 out via vnet43 keep-state

# catch all
ipfw add 64999 reject ip6 from any to any
ipfw add 65000 allow tcp from any to any
ipfw add 65001 allow udp from any to any

This is a bit messi at the moment because I had done some changes and
tests because of the nat problem. The strange thing is I have tested
if I redirect the port 22 to the vnet interface of the host and this
works. It just doesn't work for the jails.

Any idea what is going wrong or what change I have missed?

Philipp

[0] /usr/share/examples/jails/jib