NAT Port forwarding not working

321 views
Skip to first unread message

Josh Thorpe

unread,
Jan 6, 2021, 7:40:57 AM1/6/21
to iocage
Hi, I have Truenas server with multiple jails.
They are all on a VNET 172.16.0.x with a NAT that I have port fowarding to the host.
The host is on a local network at 192.168.0.8 and also has a wireguard vpn interface at 10.0.0.8

The host services such as SSH and the web UI can be accessed over the local network at:
or over the wireguard VPN at:

The issue is the iocage jail services are only accessible over the local network:
192.168.0.8:xyz
And the same services are not accessible over the wireguard VPN:
10.0.0.8:xyz

The host and jails have no firewalls, and it appears that the NAT created from iocage is dropping the packets if the origin is not the local network (192.168.0.x).

I have tried wireguard and truenas communities but each keep putting the blame on the other. I have several  docker containers on linux with ports bound to the host and no issues accessing these over wireguard vpn. The issue seems to be with the routing of the nat/vnet/bridge used with the iocage jails.

Josh Thorpe

unread,
Jan 6, 2021, 7:45:19 AM1/6/21
to iocage
$ iocage list
+-----+-----------------+-------+--------------+-------------+
| JID |      NAME       | STATE |   RELEASE    |     IP4     |
+=====+=================+=======+==============+=============+
| 6   | deluge          | up    | 12.1-RELEASE | 172.16.0.2  |
+-----+-----------------+-------+--------------+-------------+
| 2   | nextcloud       | up    | 12.1-RELEASE | 172.16.0.6  |
+-----+-----------------+-------+--------------+-------------+
| 3   | plexmediaserver | up    | 12.1-RELEASE | 172.16.0.10 |
+-----+-----------------+-------+--------------+-------------+

$ ifconfig
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e13abb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
ether d0:50:99:d1:12:76
inet 192.168.0.8 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether d0:50:99:d1:12:77
media: Ethernet autoselect
status: no carrier
nd6 options=9<PERFORMNUD,IFDISABLED>
ix2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether d0:50:99:d1:12:78
media: Ethernet autoselect
status: no carrier
nd6 options=1<PERFORMNUD>
ix3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether d0:50:99:d1:12:79
media: Ethernet autoselect
status: no carrier
nd6 options=1<PERFORMNUD>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog
vnet0.2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: nextcloud as nic: epair0b
options=8<VLAN_MTU>
ether d2:50:99:ba:b5:81
hwaddr 02:16:e2:77:33:0a
inet 172.16.0.5 netmask 0xfffffffc broadcast 172.16.0.7
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=1<PERFORMNUD>
wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
options=80000<LINKSTATE>
inet 10.0.0.8 --> 10.0.0.8 netmask 0xffffffff
groups: tun
nd6 options=101<PERFORMNUD,NO_DAD>
Opened by PID 3206
vnet0.3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: plexmediaserver as nic: epair0b
options=8<VLAN_MTU>
ether d2:50:99:3d:6b:aa
hwaddr 02:53:3b:fb:2d:0a
inet 172.16.0.9 netmask 0xfffffffc broadcast 172.16.0.11
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=1<PERFORMNUD>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:b3:17:a6:42:00
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
       ifmaxaddr 0 port 1 priority 128 path cost 2000
groups: bridge
nd6 options=1<PERFORMNUD>
vnet0.6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: deluge as nic: epair0b
options=8<VLAN_MTU>
ether d2:50:99:56:71:77
hwaddr 02:24:be:e1:2f:0a
inet 172.16.0.1 netmask 0xfffffffc broadcast 172.16.0.3
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=1<PERFORMNUD>

$ netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.1        UGS         ix0
10.0.0.0/24        wg0                US          wg0
10.0.0.8           link#10            UH          wg0
localhost          link#5             UH          lo0
172.16.0.0/30      link#8             U       vnet0.6
172.16.0.1         link#8             UHS         lo0
172.16.0.4/30      link#9             U       vnet0.2
172.16.0.5         link#9             UHS         lo0
172.16.0.8/30      link#11            U       vnet0.3
172.16.0.9         link#11            UHS         lo0
192.168.0.0/24     link#1             U           ix0
192.168.0.8        link#1             UHS         lo0

$  iocage get all plex
CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:0
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:1
boot:1
bpf:0
children_max:0
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:172.16.0.9
defaultrouter6:auto
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:plexmediaserver
host_hostuuid:plexmediaserver
host_time:1
hostid:00000000-0000-0000-0000-d05099d11276
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|172.16.0.10/30
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/plexmediaserver/data
jail_zfs_mountpoint:none
last_started:2021-01-06 11:25:21
localhost_ip:none
login_flags:-f root
mac_prefix:d25099
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:1
nat_backend:ipfw
nat_forwards:tcp(32400:32400)
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:plexmediaserver
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:12.1-RELEASE-p12
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:pluginv2
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:d250993d6baa d250993d6bab
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:auto
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off

Foo JH

unread,
Nov 5, 2021, 10:05:01 PM11/5/21
to iocage
I'm working on a similar use case on a different approach: have you thought about enabling pf in the jail for NAT?
Reply all
Reply to author
Forward
0 new messages