vmm: experimentation with networking on wifi interfaces

[Edd Barrett]

Hey,

As we saw earlier on misc@, getting a vmm host on the internet when the
host is using a wireless interface is not as straightforward as with
wired interfaces.

Specifically, a bridge won't work on a wireless interface, which in turn (I
think) means virtual switches don't work either (although I did not try
that).

Some mentioned that it's possible to use a nat with a vether bridge.

Striving for a simpler working setup, after some thinking, and a
discussion with mlarkin@, I decided to find out:

1) If you really need the vether interface in the equation.
2) If you could use dhcpd on the tap interface of a vm.

Mike asked me to write to tech@ reporting the outcome of 2.

Starting with 1, if all you want is to get a VM on the internet, you
don't need a vether.

On the host:
---8<---
# ifconfig tap0 192.168.10.1
# echo "pass out on iwn0 inet from tap0:network to any nat-to (iwn0)" >> /etc/pf.conf
# pfctl -f /etc/pf.conf
# sysctl net.inet.ip.forwarding=1
--->8---

On the guest:

---8<---
# ifconfig vio 192.168.10.2
# route add default 192.168.10.1
--->8---

(Or enter those parameters into the installer)

And you are good to go. I managed to install a guest via this method.

There are a couple of quirks though. First, you can't boot with that line in
pf.conf, as pf comes up before vmd, so the tap interface will not exist as pf
starts, causing pf to not parse its config file. Second, if you halt/reboot the
guest (I notice reboot actually halts), then the tap interface is deleted and
the IP is lost. If you want to bring the host back up, you need to set the IP
on the tap device again.

As for 2, DHCP over tap, this did not work for me.

On the host, in dhcpd.conf:

---8<---
subnet 192.168.10.0 netmask 255.255.255.0 {
option routers 192.168.1.1;
option domain-name-servers 192.168.1.1;
option domain-name "home";
range 192.168.10.2 192.168.10.10;
}
--->8---

Start with: doas dhcpd -df -c /etc/dhcpd.conf tap0

Then in the guest:

---8<---
Listening on tap0 (192.168.10.1).
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
already acking lease 192.168.10.2
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
already acking lease 192.168.10.2
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
already acking lease 192.168.10.2
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
already acking lease 192.168.10.2
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
already acking lease 192.168.10.2
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
already acking lease 192.168.10.2
..
already acking lease 192.168.10.2
DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
--->8---

The guest at this point is saying:

---8<---
# dhcpd -df -c /etc/dhcpd.conf tap0
DHCPDISCOVER on vio0 - interval 3
DHCPDISCOVER on vio0 - interval 8
DHCPDISCOVER on vio0 - interval 14
..
--->8---

An address is never acquired.

When I tried once more:

---8<---
# dhcpd -df -c /etc/dhcpd.conf tap0
Listening on tap0 (192.168.10.1).
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
..
--->8---

When I ran dhclient in the guest this time:

---8<---
# dhclient vio0
DHCPDISCOVER on vio0 - interval 3
panic: kernel diagnostic assertion "m != NULL" failed: file "../../../../dev/pci/if_vio.c", line 1008
Stopped at Debugger+0x9: leave
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*45447 45447 77 0x100013 0 0 dhclient
Debugger() at Debugger+0x9
panic() at panic+0xfe
__assert() at __assert+0x25
vio_rxeof() at vio_rxeof+0x1db
vio_rx_intr() at vio_rx_intr+0x28
virtio_check_vqs() at virtio_check_vqs+0x8c
virtio_pci_legacy_intr() at virtio_pci_legacy_intr+0x6b
intr_handler() at intr_handler+0x28
Xintr_legacy7() at Xintr_legacy7+0xdd
--- interrupt ---
Xspllower() at Xspllower+0xc
if_enqueue() at if_enqueue+0x69
ether_output() at ether_output+0x1b0
bpfwrite() at bpfwrite+0x153
spec_write() at spec_write+0xb5
end trace frame: 0xffff80000e3a8c60, count: 0
http://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
--->8---

Oops. Well, please don't take that as a bug report, as the guest is running
vanilla 6.0-release. If someone wants more info, I can mail it, but I should
try to reproduce that on 6.0-stable or -current.

Cheers

--
Best Regards
Edd Barrett

http://www.theunixzoo.co.uk

[Reyk Floeter]

> Am 14.10.2016 um 00:38 schrieb Edd Barrett <e...@theunixzoo.co.uk>:
>
> Hey,
>
> As we saw earlier on misc@, getting a vmm host on the internet when the
> host is using a wireless interface is not as straightforward as with
> wired interfaces.
>
> Specifically, a bridge won't work on a wireless interface, which in turn (I
> think) means virtual switches don't work either (although I did not try
> that).
>
> Some mentioned that it's possible to use a nat with a vether bridge.
>
> Striving for a simpler working setup, after some thinking, and a
> discussion with mlarkin@, I decided to find out:
>

What do you mean with simpler?

The vether approach is straightforward.

> 1) If you really need the vether interface in the equation.
> 2) If you could use dhcpd on the tap interface of a vm.
>

Of course you can.

But you will need -current to specify the name of the tap interface or rely on the fact that you only have one vm that coincidentally always picks tap0.

Using vether in a bridge with the tap allows you to preconfigure a fixed interface on the host that doesn't depend on the vm state. The tapX unit doesn't matter as it will be added to the bridge, your configuration is only done in vether0. Automatically, no scripting required when starting the vm - I think that's a lot easier :)

switch "foo" {
add vether0
}

vm "bar" {
interface { switch "foo" }
}

I will also add an option for setting the switch in vmctl start.

Now go ahead and configure vether0 as you configure tap0 below... I'm usually running dhcpd on vether0, but you can also use fixed IPs of course.

> ...

> already acking lease 192.168.10.2
> DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
> DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
> DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
> DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
> DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
> --->8---
>
> The guest at this point is saying:
>
> ---8<---
> # dhcpd -df -c /etc/dhcpd.conf tap0
> DHCPDISCOVER on vio0 - interval 3
> DHCPDISCOVER on vio0 - interval 8
> DHCPDISCOVER on vio0 - interval 14

> ...

> --->8---
>
> An address is never acquired.
>
> When I tried once more:
>
> ---8<---
> # dhcpd -df -c /etc/dhcpd.conf tap0
> Listening on tap0 (192.168.10.1).
> DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
> DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
> DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
> DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
> DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
> DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
> DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
> DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0
> DHCPDISCOVER from fe:e1:ba:d0:95:d1 via tap0
> DHCPOFFER on 192.168.10.2 to fe:e1:ba:d0:95:d1 via tap0

> ...

Get current or a snapshot for vmm, please.

Reyk

[Mike Belopuhov]

Just put a pair of parenthesis around "tap0:network" to indicate the dynamic
nature of your configuration.

[Mike Larkin]

On Thu, Oct 13, 2016 at 11:38:46PM +0100, Edd Barrett wrote:
> Hey,
>
> As we saw earlier on misc@, getting a vmm host on the internet when the
> host is using a wireless interface is not as straightforward as with
> wired interfaces.
>
> Specifically, a bridge won't work on a wireless interface, which in turn (I
> think) means virtual switches don't work either (although I did not try
> that).
>
> Some mentioned that it's possible to use a nat with a vether bridge.
>
> Striving for a simpler working setup, after some thinking, and a
> discussion with mlarkin@, I decided to find out:
>
> 1) If you really need the vether interface in the equation.
> 2) If you could use dhcpd on the tap interface of a vm.
>
> Mike asked me to write to tech@ reporting the outcome of 2.

.. snip ...

I know about this panic, it's on my list of things to fix, but I haven't
got around to investigating. My gut tells me this is a membar issue because
it's trying to reuse an already-used descriptor. But I could be wrong.

-ml

[Edd Barrett]

Hi,

On Fri, Oct 14, 2016 at 12:56:09AM +0200, Reyk Floeter wrote:
> Using vether in a bridge with the tap allows you to preconfigure a
> fixed interface on the host that doesn't depend on the vm state.

Ah. Yes, and this actually fixes the quirks I mentioned. Using tap
directly is actually *more* complicated.

> I will also add an option for setting the switch in vmctl start.

Thanks

> Now go ahead and configure vether0 as you configure tap0 below... I'm
> usually running dhcpd on vether0, but you can also use fixed IPs of
> course.

Yep, this just worked. Thanks for your help.

You just have to remember to change the DNS server addresses if you move
the host machine between different networks (in /etc/resolv.conf for a
static IP guest, or in the host DHCP server if the guest uses DHCP).

(At some point I'm going to start making an FAQ entry covering the
common vmm cases.)