How to force network packets across network instead of loopback

[Sad Clouds]

Hello, I have two network interfaces ix0 and ix1 on the same FreeBSD
host. When transferring data between those interfaces, is there a way
to force FreeBSD to send and receive packets across the physical
network, instead of using loopback? I'd like to use this for NIC
hardware testing.

I tried adding various static routes, but this does not seem to work.

Thanks.

[Frank Leonhardt]

I haven't tested this but like any sane OS it will optimise local traffic this way unless you trick it. One way might be to assign different subnets to each interface. Then add a route between them.

e.g.
ifconfig ix0 inet 192.168.1.2/24
ifconfig ix1 inet 192.168.2.2/24
route add 192.168.2.0/24 192.168.1.1

Enable forwarding. You might also add a firewall rule to stop it:

ipfw add 100 deny ip from 192.168.1.2 to 192.168.2.2 via lo0 or (pf) block in on lo0 from 192.168.1.2 to 192.168.2.2

Another way might be to use a bridge.

Regards, Frank.

[Sad Clouds]

On Thu, 11 Sep 2025 10:52:14 +0100
Frank Leonhardt <freeb...@fjl.co.uk> wrote:

> I haven't tested this but like any sane OS it will optimise local
> traffic this way unless you trick it. One way might be to assign
> different subnets to each interface. Then add a route between them.

Hi, thanks for the suggestion. I think I tried adding different subnets
and also using multiple routing tables with setfib, but none of that
seemed to work properly. I was trying to avoid using packet filters, as
they probably add overhead and latency to network I/O paths.

Then I setup several VNET jails, and that seems to force packets across
the physical network. So I'll be using this method for performance
testing.

On Solaris this setup is quite easy, just enable symmetric routing:

$ ipadm set-prop -p hostmodel=strong ipv4
$ ipadm set-prop -p hostmodel=strong ipv6

I was hoping for something similar on FreeBSD, but apart from
net.inet.ip.rfc1122_strong_es which doesn't quite achieve the same
effect, I could not find anything.

[Frank Leonhardt]

Linux has network namespaces, I believe, would be an obvious answer and
I I was going to suggest jails to get a similar effect, but was less
sure it would work.

Did you try setting up as I suggested and looking at the routing table?
You can always delete any direct (link#x) routes it creates
automatically. FreeBSD doesn't have the global configuration options
like Solaris (as you know) so normally you need to tweak the routing
tables to get what you want.

[Frank Leonhardt]

On 11/09/2025 11:19, Sad Clouds wrote:

On Solaris this setup is quite easy, just enable symmetric routing:

$ ipadm set-prop -p hostmodel=strong ipv4
$ ipadm set-prop -p hostmodel=strong ipv6

I was hoping for something similar on FreeBSD, but apart from
net.inet.ip.rfc1122_strong_es which doesn't quite achieve the same
effect, I could not find anything.

P.S. I followed this up as I was more than uncertain about this variable, and it seems seems no one knows what this actually does!

https://reviews.freebsd.org/D32912

[Dag-Erling Smørgrav]

Sad Clouds <cryintot...@gmail.com> writes:
> Hello, I have two network interfaces ix0 and ix1 on the same FreeBSD
> host. When transferring data between those interfaces, is there a way
> to force FreeBSD to send and receive packets across the physical
> network, instead of using loopback?

Assign one of them to a vnet jail:

# jail -c name=ix0 path=/ persist vnet vnet.interface=ix0
# jexec ix0 ifconfig ix0 inet 192.0.2.1/24 up
# ifconfig ix1 inet 192.0.2.2/24 up
# jexec ix0 ping -c 3 192.0.2.2
# ping -c 3 192.0.2.1

Alternatively, assign them both to separate jails:

# jail -c name=ix0 path=/ persist vnet vnet.interface=ix0
# jexec ix0 ifconfig ix0 inet 192.0.2.1/24 up
# jail -c name=ix1 path=/ persist vnet vnet.interface=ix1
# jexec ix1 ifconfig ix1 inet 192.0.2.2/24 up
# jexec ix0 ping -c 3 192.0.2.2
# jexec ix1 ping -c 3 192.0.2.1

DES
--
Dag-Erling Smørgrav - d...@FreeBSD.org

[Sad Clouds]

On Thu, 11 Sep 2025 11:57:18 +0100
Frank Leonhardt <freeb...@fjl.co.uk> wrote:

> P.S. I followed this up as I was more than uncertain about this
> variable, and it seems seems no one knows what this actually does!
>
> https://reviews.freebsd.org/D32912
>

The way I read it, if a packet arrives on a different interface, then
drop it, but the sysctl does not actually implement symmetric routing,
so seems to be somewhat useless.

[Sad Clouds]

On Thu, 11 Sep 2025 13:40:08 +0200
Dag-Erling Smørgrav <d...@FreeBSD.org> wrote:

> # jail -c name=ix0 path=/ persist vnet vnet.interface=ix0
> # jexec ix0 ifconfig ix0 inet 192.0.2.1/24 up
> # jail -c name=ix1 path=/ persist vnet vnet.interface=ix1
> # jexec ix1 ifconfig ix1 inet 192.0.2.2/24 up
> # jexec ix0 ping -c 3 192.0.2.2
> # jexec ix1 ping -c 3 192.0.2.1

Thanks for the examples, nice and short. I also arrived at a similar
conclusion, VNET jails are probably the quickest way to achieve it with
minimal overheads.