What's the best setup for accessing network?

[zhiting zhu]

Hey,

With the -n option on firecracker.py, client on the same machine can send packet to the VM. But the VM cannot access Internet. I setup NAT based network following: https://jamielinux.com/docs/libvirt-networking-handbook/custom-nat-based-network.html I can get Internet access but the boot time is very large(9.2 second for me).  

Here is the booting log:

bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
Detected virtio-mmio device: (2,0)
Detected virtio-mmio device: (1,0)
queue 0 size 256
queue 1 size 256
queue 2 size 0
eth0: ethernet address: 52:54:00:12:34:56
queue 0 size 256
queue 1 size 0
virtio-blk: Add blk device instances 0 as vblk0, devsize=536870912
Warning: No hardware source of entropy available to your platform,
	CSPRNG will rely on software source of entropy to provide high-quality randomness.
random: <Software, Yarrow> initialized
VFS: unmounting /dev
VFS: mounting rofs at /rofs
failed to mount /rofs, error = No error information
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys
random: device unblocked.
program zpool.so returned 1
BSD shrinker: event handler list found: 0xffffa00000989080
	BSD shrinker found: 1
BSD shrinker: unlocked, running
[I/26 dhcp]: Broadcasting DHCPDISCOVER message with xid: [1850312527]
[I/26 dhcp]: Waiting for IP...
[I/26 dhcp]: Broadcasting DHCPDISCOVER message with xid: [1771944366]
[I/26 dhcp]: Waiting for IP...
[I/26 dhcp]: Broadcasting DHCPDISCOVER message with xid: [1285738661]
[I/26 dhcp]: Waiting for IP...
[I/26 dhcp]: Broadcasting DHCPDISCOVER message with xid: [1533719522]
[I/26 dhcp]: Waiting for IP...
[I/195 dhcp]: Received DHCPOFFER message from DHCP server: 192.168.100.1 regarding offerred IP address: 192.168.100.173
[I/195 dhcp]: Broadcasting DHCPREQUEST message with xid: [1533719522] to SELECT offered IP: 192.168.100.173
[I/195 dhcp]: Received DHCPOFFER message from DHCP server: 192.168.100.1 regarding offerred IP address: 192.168.100.173
[W/195 dhcp]: Got packet with wrong transaction ID (1533719522, 1285738661)
[I/195 dhcp]: DHCP received hostname: osv

[I/195 dhcp]: Received DHCPACK message from DHCP server: 192.168.100.1 regarding offerred IP address: 192.168.100.173
[I/195 dhcp]: Server acknowledged IP 192.168.100.173 for interface eth0 with time to lease in seconds: 3600
eth0: 192.168.100.173
[I/195 dhcp]: Configuring eth0: ip 192.168.100.173 subnet mask 255.255.255.0 gateway 192.168.100.1 MTU 1500
[I/195 dhcp]: Set hostname to: osv
Booted up in 9237.45 ms

Is there any other recommended way to setup VM with internet? 

Best,
Zhiting

[Waldek Kozaczuk]

Hi,

I have realized that our firecracker script misses passing '--nameserver' argument that is used when DHCP is off.

diff --git a/scripts/firecracker.py b/scripts/firecracker.py

index 42a5e8ee..65345efc 100755

--- a/scripts/firecracker.py

+++ b/scripts/firecracker.py

@@ -229,7 +229,7 @@ def main(options):

         setup_tap_interface('fc_tap0', tap_ip, options.bridge)

         if not options.bridge:

             client_ip = '172.16.0.2'

-            cmdline = '--ip=eth0,%s,255.255.255.252 --defaultgw=%s %s' % (client_ip, tap_ip, cmdline)

+            cmdline = '--ip=eth0,%s,255.255.255.252 --defaultgw=%s --nameserver=%s %s' % (client_ip, tap_ip, tap_ip, cmdline)

 

     if options.verbose:

         cmdline = '--verbose ' + cmdline

I have also noticed that even though it sets up the tap device, it does not set up routing outside of VM. For now, I have manually followed the NAT setup examples (the iptables fragment) from firecracker readme - https://github.com/firecracker-microvm/firecracker/blob/master/docs/network-setup.md. In my case, after enabling NAT I was able to make outgoing traffic using an IP. However, I was not able to make DNS working so trying to fetch files with URLs with DNS names would fail. I think DNS relies on UDP and so maybe NAT setup is missing something.

I looked at the document you pointed to which does mention DNS setup so maybe it will all work for you.

Let us know if it works so I can update the script accordingly. If you feel like sending a patch and updating wiki - https://github.com/cloudius-systems/osv/wiki/Running-OSv-on-Firecracker - I would appreciate it. 

Waldek

[Waldek Kozaczuk]

Quick update - using Google public DNS IP - made DNS working for me (on top of NAT setup described by firecracker doc:

sudo iptables -t nat -A POSTROUTING -o ens9 -j MASQUERADE

sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

sudo iptables -A FORWARD -i fc_tap0 -o ens9 -j ACCEPT

diff --git a/scripts/firecracker.py b/scripts/firecracker.py

index 42a5e8ee..bf1aed10 100755

--- a/scripts/firecracker.py

+++ b/scripts/firecracker.py

@@ -229,7 +229,7 @@ def main(options):

         setup_tap_interface('fc_tap0', tap_ip, options.bridge)

         if not options.bridge:

             client_ip = '172.16.0.2'

-            cmdline = '--ip=eth0,%s,255.255.255.252 --defaultgw=%s %s' % (client_ip, tap_ip, cmdline)

+            cmdline = '--ip=eth0,%s,255.255.255.252 --defaultgw=%s --nameserver=%s %s' % (client_ip, tap_ip, "8.8.8.8", cmdline)

 

     if options.verbose:

         cmdline = '--verbose ' + cmdline

I think using public google DNS is not ideal and ideally we should figure out why DNS does not work if nameserver is equal to tap_ip.

Waldek

[zhiting zhu]

With dnsmasq as the dns server, I can use tap_ip as the nameserver. The boottime back to normal. I create two scripts to setup the network. setup_iptable.sh sets the ip table. setup_dnsmasq configures dnsmasq and start it. After running these two scripts, I can run firecracker.py with only -n. I'm not familiar with sending patch through email with git. I include the patch below.

From bbc12eb31878bdacaf20313be444ac71f9ff6578 Mon Sep 17 00:00:00 2001

From: Zhiting Zhu <zhit...@cs.utexas.edu>
Date: Wed, 9 Oct 2019 00:32:35 -0500
Subject: [PATCH] scripts to setup dnsmasq and NAT

Signed-off-by: Zhiting Zhu <zhit...@cs.utexas.edu>
---
 scripts/delete_dnsmasq_setting.sh |  6 ++++++
 scripts/restore_iptable.sh        |  5 +++++
 scripts/setup_dnsmasq.sh          | 20 ++++++++++++++++++++
 scripts/setup_iptable.sh          |  8 ++++++++
 4 files changed, 39 insertions(+)
 create mode 100755 scripts/delete_dnsmasq_setting.sh
 create mode 100755 scripts/restore_iptable.sh
 create mode 100755 scripts/setup_dnsmasq.sh
 create mode 100755 scripts/setup_iptable.sh

diff --git a/scripts/delete_dnsmasq_setting.sh b/scripts/delete_dnsmasq_setting.sh
new file mode 100755
index 00000000..aa9c3c15
--- /dev/null
+++ b/scripts/delete_dnsmasq_setting.sh
@@ -0,0 +1,6 @@
+if [ "$#" -ne 1 ]; then
+    echo "Need to specify interface name"
+fi
+DEV=$1
+sudo rm -rf /var/lib/dnsmasq/$DEV
+sudo rm -rf /etc/dnsmasq.d/$DEV.conf
diff --git a/scripts/restore_iptable.sh b/scripts/restore_iptable.sh
new file mode 100755
index 00000000..e882948a
--- /dev/null
+++ b/scripts/restore_iptable.sh
@@ -0,0 +1,5 @@
+if [ -f iptables.rules.old ]; then
+    sudo iptables-restore < iptables.rules.old
+    rm iptables.rules.old
+fi
+sudo sh -c "echo 0 > /proc/sys/net/ipv4/ip_forward"
diff --git a/scripts/setup_dnsmasq.sh b/scripts/setup_dnsmasq.sh
new file mode 100755
index 00000000..b93c88df
--- /dev/null
+++ b/scripts/setup_dnsmasq.sh
@@ -0,0 +1,20 @@
+if [ "$#" -ne 1 ]; then
+    echo "need to specify the interface name"
+fi
+DEV=$1
+sudo mkdir -p /var/lib/dnsmasq/$DEV
+sudo touch /var/lib/dnsmasq/$DEV/hostsfile
+sudo touch /var/lib/dnsmasq/$DEV/leases
+sudo touch /var/lib/dnsmasq/$DEV/dnsmasq.conf
+sudo sh -c "cat << 'EOF' >/var/lib/dnsmasq/$DEV/dnsmasq.conf
+except-interface=lo
+interface=$DEV
+bind-dynamic
+strict-order
+EOF"
+sudo mkdir -p /etc/dnsmasq.d/
+sudo touch /etc/dnsmasq.d/$DEV.conf
+sudo bash -c "echo "except-interface=$DEV" >> /etc/dnsmasq.d/$DEV.conf"
+sudo bash -c "echo "bind-interfaces" >> /etc/dnsmasq.d/$DEV.conf"
+sudo mkdir -p /var/run/dnsmasq/
+sudo dnsmasq --conf-file=/var/lib/dnsmasq/$DEV/dnsmasq.conf --pid-file=/var/run/dnsmasq/$DEV.pid
diff --git a/scripts/setup_iptable.sh b/scripts/setup_iptable.sh
new file mode 100755
index 00000000..ce638b7b
--- /dev/null
+++ b/scripts/setup_iptable.sh
@@ -0,0 +1,8 @@
+INTERFACE=enp7s3
+sudo iptables-save > iptables.rules.old
+sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
+sudo iptables -t nat -A POSTROUTING -o $INTERFACE -j MASQUERADE
+sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+sudo iptables -A FORWARD -i fc_tap0 -o $INTERFACE -j ACCEPT
+sudo iptables -A INPUT -i fc_tap0 -p udp -m udp -m multiport --dports 53 -j ACCEPT
+sudo iptables -A INPUT -i fc_tap0 -p tcp -m tcp -m multiport --dports 53 -j ACCEPT
--
2.17.1

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/7b0585d6-e7f2-49dc-afe6-c9a43d10b90f%40googlegroups.com.