Openbsd as a netvm

[ronpunz]

I'm attempting setup a netvm using openbsd.

I'm following Unman's guide
https://github.com/unman/notes/blob/master/openBSD_as_netvm but "fell at
the first hurdle" i.e. Line No1 states "Install OpenBSD as HVM
Template". Does this mean a standalone template? If so I've successfully
completed this stage, but, am unable to proceed to line No2 " Create
netvm "openFW" using OpenBSD as template" - as I understand it an appvm
cannot be created from a standalone template. That being the case, it
looks like I need to create an openbsd template - but how? I need either
source code to build it or a repository to download it? As far as I'm
aware, neither exists?

Any help would appreciated

[unman]

Those are notes, not really intended as a guide.

What you need is:
qvm-create --class TemplateVM openBSD --property virt_mode=HVM --property kernel='' -l purple
qvm-create -t openBSD --property virt_mode=HVM --property kernel='' -l purple open

[ronpunz]

Thanks Unman for getting me up and running.

I made it down to line 12 Set fw as netvm for openFW.
qvm-prefs openFW netvm fw. This command returns: qvm-prefs : error : the
fw qube does not provide network.

Is there a workaround for this?

[ronpunz]

On 5/31/19 12:51 AM, unman wrote:

Thanks Unman for getting me up and running.

I made it down to line 12 Set fw as netvm for openFW.
qvm-prefs openFW netvm fw. This command returns: qvm-prefs : error : the
fw qube does not provide network.

Is there a workaround for this?

I managed to get round this with "qvm-prefs openFW provides_network true".

This enabled me to proceed to the next step "start openFW". However, it
starts only in a transient state (i.e. qubes manager shows yellow led
not the usual green) As a consequence I can't start fw.

[unman]

On Fri, May 31, 2019 at 08:43:59AM +0000, ronpunz wrote:
>
> On 5/31/19 12:51 AM, unman wrote:
> > On Thu, May 30, 2019 at 09:56:18AM +0000, ronpunz wrote:
> > > I'm attempting setup a netvm using openbsd.
> > >
> > > I'm following Unman's guide
> > > https://github.com/unman/notes/blob/master/openBSD_as_netvm but "fell at the
> > > first hurdle" i.e. Line No1 states "Install OpenBSD as HVM Template". Does
> > > this mean a standalone template? If so I've successfully completed this
> > > stage, but, am unable to proceed to line No2 " Create netvm "openFW" using
> > > OpenBSD as template" - as I understand it an appvm cannot be created from a
> > > standalone template. That being the case, it looks like I need to create an
> > > openbsd template - but how? I need either source code to build it or a
> > > repository to download it? As far as I'm aware, neither exists?
> > >
> > > Any help would appreciated
> > Those are notes, not really intended as a guide.
> >
> > What you need is:
> > qvm-create --class TemplateVM openBSD --property virt_mode=HVM --property kernel='' -l purple
> > qvm-create -t openBSD --property virt_mode=HVM --property kernel='' -l purple open
>
> Thanks Unman for getting me up and running.
>
> I made it down to line 12 Set fw as netvm for openFW.
> qvm-prefs openFW netvm fw. This command returns: qvm-prefs : error : the fw
> qube does not provide network.
>
> Is there a workaround for this?
>
> I managed to get round this with "qvm-prefs openFW provides_network true".

I assume you meant:"qvm-prefs fw provides_network true".

>
> This enabled me to proceed to the next step "start openFW". However, it
> starts only in a transient state (i.e. qubes manager shows yellow led not
> the usual green) As a consequence I can't start fw.
>

Ignore this - it's because you dont have any qvm hooks in the HVM. Same
would apply for any HVM - windows, linux, BSDs
Start fw first. Then openFW.

[ronpunz]

Have now completed all the steps with the exception of line No 44; Bring
up em0 - dhclient em0 - which resulted in an error.

I now have a network applet associated with fw. But am unable to obtain
a connection to my router.

From openFW I'm able to ping 10.137.0.34 and the gateway to fw; 10.137.0.33

Not sure which direction to go next and to be honest, feel a bit out of
my depth. When I started this task I thought there was a simple
correlation between  openFW to sys-net and fw  to sys-firewall. In
reality it seems a fair bit more complicated than that. For example, fw
seems to have a dual firewall and network interface role?

[unman]

I dont understand what this means.
There is simple correlation as you describe, it's just that fw needs to
do a little more work to provide the internal interface to the HVM.

What error do you get when you bring up em0?
What's the output from ifconfig?

[ronpunz]

Hi I appreciate you're continuing patience and support.

I've started afresh on a development box.

I managed to get em0 up

Here's the results of ifconfig (on 2 screenshots - because I couldn't
expand the terminal dialogue box - I know that's sad)

> Have tried without success getting the network applet up and running - I chose vif26 as client and under ipv4 auto dhcp but recognise that probably wrong.
Incidently I note that the settings in OpenFW are non-persistent, as is
fw vif reference number. Once the system works properly, is there a way
to make things persistent across reboots?
>

[ronpunz]

I note the ifconfig screen shots were missed off my reply.

They should be here

[unman]

I'm sorry - can you cut and paste the contents rather than imaging?

[unman]

Dont bother with the network applet - all the work in fw is done with
the interplay between the vif+ interfaces. This is dealt with in the
scripts that you place in /rw/config.

OpenFW is indeed amnesiac - I like it that way.
If you want persistence, you can configure mounts to another disk, and
then put scripts on that disk to configure your setup as you want.

[ronpunz]

Copy/paste as requested

[unman]

??
I cant see the images - paste the contents in the mail.

[ronpunz]

Sorry. I'm a bit confused. I pasted them in the mail and they're
viewable on the qubes user forum at
https://groups.google.com/forum/#!topic/qubes-users/MpXLhz5COvM

Please let me know if there's more i can do

[unman]

I cant view them.
Please post the contents, not pictures.

[Sphere]

I think what unman means is for you to provide the logs in text and not just provide images to help diagnose this problem

[ronpunz]

Gotcha. However, that's easier said than done. After trying and failed
using various OCR software. To cut a long story short, I've ended up
typing the whole thing out as follows:

joo# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
        index 5 priortity 0 llprio 3
        groups: lo
        ininet6 :: 1 prefixlen 128
        inet6 fe80 ::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
xnf0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:16:3e:5e:6c:00
        index 1 priortity 0 llprio 3
        media: ethernet manual
        status: active
        inet: 10.137.0.10 netmask 0xff000000 broadcast 10.255.255.255
re0: flags =8802<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
        lladdr 1c:1b:0d:a4:1e:e4
        index 2 priortity 0 llprio 3
        media: ethernet autoselect (none)
        status: no carrier
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladr 68:05:ca:55:75:6f
        index 3 priortity 0 llprio 3
        groups: egress
        media: ethernet autoselect 1000baseT
(full-duplex,master.rxpause,txpause)
        status: active
enc0: flags=0<>
        index 4 priortity 0 llprio 3
        groups: enc
        status: active
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33136
        index 6 priortity 0 llprio 3
        groups: pflog

I'm now able to successfully ping 8.8.8.8 but not google.com. Indicating
a dns issue?

The dns setting in pf is iptables -t nat -I PR-QBS -p udp --dport 53 -j
DNAT --to 9.9.9.9

[unman]

I'm sorry for the pain in doing this - you could always have booted the
openBSD qube with a USB attached, and transferred the files that way.
Like a sneakernet but smaller scale - a fingernet?

You dont say *from where* you are able to ping. Yes, this looks like a
DNS issue.

If you want to get this working from the BSD qube, then check
/etc/resolv.conf
This isn't necessary - in fact you may prefer NOT to allow outgoing
traffic originating from the openBSD firewall.

You say that rule you have is "in pf" - do you mean "in fw"?? It's just
not a pf thing.
So if it *is* in fw, and you are able to ping from fw, then this is looking good.
Simplest way to proceed is to set /etc/resolv.conf in fw to use 9.9.9.9

Give just a little more detail on what's working and from where.

[ronpunz]

Yes, you're right I need to clarify some points.

1/ The pinging I referred to i.e. 8.8.8.8 & google.com was from openFW

2/ The rule I referred to in pf was a typo and as you guessed, should
read fwVM

3/ As suggested, I've input into /etc/resolv.conf "nameserver 9.9.9.9"

4/ Have tried to ping from fwVM 8.8.8.8 . It returned "network is
unreachable"

5/ Have tried to ping from fwVM google.com . It returned "temporary
failure in name resolution"

[ronpunz]

On 6/3/19 12:10 PM, unman wrote:

Update

I've noticed that the contents of fwVM's /etc/resolv.conf does not
survive a reboot. The file contents revert to default values:
"nameserver 10.137.0.1 nameserver 10.137.3.254"

I hope this helps debuging

[Sphere]

That's normal, apparently that happens every single time an AppVM starts up even if you personally changed the value in the TemplateVM itself. Trust me, that's part of my frustrations back then lol.

Could you try having another AppVM running and connected to your Openbsd netvm and see if you can access the internet just fine?

If it happens to be that your AppVM fails to make DNS queries then one other way for you to solve this DNS problem is to rely on DNSCrypt instead:
https://github.com/jedisct1/dnscrypt-proxy/releases

There are things that you need to setup before running dnscrypt-proxy
1. Creating a dnscrypt-proxy.toml file using the example-dnscrypt-proxy.toml file
2. Tailoring the dnscrypt-proxy.toml file settings according to your preferences
3. Changing contents of your /etc/resolv.conf to this:
nameserver 127.0.0.1
options edns0 single-request-reopen

Thankfully there is no need to compile it yourself so it shouldn't be too much of a hassle.

As for the problematic thing about /etc/resolv.conf resetting itself to 10.137. bla bla after starting, what you can do is copy the changed resolv.conf to your home directory and make a script in your dom0 terminal that will copy it and replace the newly refreshed /etc/resolv.conf. The general idea is to start the AppVM using qvm-run command that will do just that.

CAUTION: You would not want to do this on your Openbsd netvm but instead on your other AppVM connected to your netvm. But I recommend to first have a firewallvm connected before the AppVM then connect the AppVM to the firewallvm whilst having no firewall rules set to it just yet for testing purposes.

[unman]

On Tue, Jun 04, 2019 at 02:11:39AM -0700, Sphere wrote:
>
> As for the problematic thing about /etc/resolv.conf resetting itself to 10.137. bla bla after starting, what you can do is copy the changed resolv.conf to your home directory and make a script in your dom0 terminal that will copy it and replace the newly refreshed /etc/resolv.conf. The general idea is to start the AppVM using qvm-run command that will do just that.

Better way is to use the qubes config files:
https://www.qubes-os.org/doc/config-files

/rw/config/rc.local will do.
add "echo 'nameserver 9.9.9.9' > /etc/resolv.conf" to the file.

[unman]

This suggests to me that you havent yet set a default route on fw.
You need to set route on fw to use openFW as gateway.
I think you said you can ping openFW from fw.

You also need to configure openFW to forward packets and NAT trafic.
And alter raw table on fw to accept external traffic on vif interface

Testing to do:
From fw - ping openFW
ping 8.8.8.8 - use 'iptables -L -nv' to watch traffic in and out.
Look at '-t nat' and '-t raw' to see if traffic is being dropped.

on openFW:
tcpdump -i <iface>

[ronpunz]

Thanks

I'll make the suggested changes and testing you suggested.

In the meantime, I've noticed I made an omission from your guide/notes;
line  43 states "Run firewall script". Frankly, I've no idea how to do
this in openFW. Could you please be more explicit?

[ronpunz]

TEST RESULTS:

Test No1
From fw - ping openFW -- successful
Test No2
ping 8.8.8.8 - use 'iptables -L -nv' to watch traffic in and out. Ping result = "Network was unreachable"
Output from 'iptables -L -nv':
user@fw:~$ sudo iptables -L -nv
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- vif+ * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8082
0 0 DROP udp -- vif+ * 0.0.0.0/0 0.0.0.0/0 udp dpt:68
30 2344 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- vif+ * 0.0.0.0/0 0.0.0.0/0
4 208 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 REJECT all -- vif+ * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- vif+ vif+ 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 QBS-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- vif+ vif+ 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- vif+ * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 34 packets, 2552 bytes)
pkts bytes target prot opt in out source destination

Chain QBS-FORWARD (1 references)
pkts bytes target prot opt in out source destination

I cant see any reference to '-t nat' or '-t raw'. So I double checked '/rw/config/qubes-firewall-user-script'. Output as follows:
#!/bin/sh

# This script is called in AppVMs after every firewall update (configuration
# change, starting some VM etc). This is a good place to write own custom
# firewall rules, in addition to autogenerated ones. Remember that in most cases
# you'll need to insert the rules at the beginning (iptables -I) for it to be
# effective.

iptables -I FORWARD -i vif+ -o vif+ -j ACCEPT
iptables -t raw -I PREROUTING -i vif13.0 -j ACCEPT

iptables -t nat -I PR-QBS -p udp --dport 53 -j DNAT --to 9.9.9.9

Test No3

on openFW:
tcpdump -i <iface>

submitting to you the recorded test data from OPENBSD's xterm is a real problem for a BSD novice like me. I tried and failed with your earlier suggestion vis 'you could always have booted the

openBSD qube with a USB attached, and transferred the files that way.

Like a sneakernet but smaller scale - a fingernet?'
I could summarise the result of tcdump -i xnf0 as follows: (10.137.0.10 = ip of openFW, 10.137.0.11 = ip of fw, 10.64.0.1 is the primary DNS from a router)
10.137.0.10 > 10.137.0.11: icmp : echo request (DF)
arp who-has 10.137.0.11 tell 10.137.0.10
arp reply 10.137.0.11 is-at fe:ff blah blah
10.137.0.10 > 10.137.0.11: icmp: echo reply (DF)
arp who-has 10.64.0.1 tell 10.137.0.1

[unman]

On Wed, Jun 05, 2019 at 07:28:12AM +0000, ronpunz wrote:
>
> Thanks
>
> I'll make the suggested changes and testing you suggested.
>
> In the meantime, I've noticed I made an omission from your guide/notes;
> line  43 states "Run firewall script". Frankly, I've no idea how to do this
> in openFW. Could you please be more explicit?

And that's the difference between notes and a guide.

Ok - The user guide is here:
https://www.openbsd.org/faq/pf/filter.html

At a minimum, you need to do the following:
1. Configure openFW to allow forwarding.
Without this, no traffic will pass THROUGH openFW, which is what you
want.
# sysctl -w net.inet.ip.forwarding=1
# sysctl -w net.inet.ip.redirect=0

2. Configure the firewall.
In openBSD this is done in /etc/pf.conf:
Very briefly, the rules are read in order: the last matching rule
applies. You can circumvent this by putting "quick" in the rule. That
stops processing.
At a minimum, you want to allow traffic outbound, and you want
to apply NAT.
So, edit /etc/pf.conf and add rules like this:
match out on egress inet from 10.137.0.0/16 to any nat-to (egress:0)
pass out quick inet

Then reread the rules:
pfctl -f /etc/pf.conf

And check:
pfctl -sr

This is the bare minimum - you will want to configure more protection, but
that's enough to get started.

Don't forget to set the route correctly on fw, and edit the raw table to
allow external traffic to come in on the relevant vif interface.

unman