Help needed: Allow OSv <-> OSv communication within 2 separate VMs

Hanaa Abbas

unread,

Oct 3, 2021, 8:26:11 AM10/3/21

to OSv Development

Hello,

Please, I need help to properly set up the network to allow OSv instances running on separate VirtualBox machines (Ubuntu 20.04) to communicate with each other.

Host is Windows 10.
Currently, I can ping Host <-> VB and VB <-> VB. For VB -> Host, I had to disable my firewall. Also I can ping VB -> OSv. However, I need OSv instances to communicate with each other.

I tried many things but without any use.

This is my current VB network settings. The ubuntu env also has a bridged network (see screenshots).

I will really appreciate any help. Thank you

Waldek Kozaczuk

unread,

Oct 6, 2021, 6:11:13 PM10/6/21

to OSv Development

Hi,

I see you have a nested virtualization setup - QEMU under VirtualBox. If that helps and desired you should be able to run OSv directly on VirtualBox - I think capstan supports it as well with the '-p' option. I think this might simplify things.

In general, for 2 OSv instances to communicate over TCP or UDP (which is what I am assuming) with each other, you need to enable network forwarding either through slirp or tap or bridge for each OSv instance : the capstan '-b' or '-p' options . The bottom line is that OSv instance 1 needs to know the "forwarded" IP of the OSv instance 2 and vice versa to communicate to each other.

Please note that this wiki - https://github.com/cloudius-systems/osv/wiki/Live-migration-of-OSv-guests-using-qemu - describes a setup when you run 2 OSv instances on QEMU directly on single Linux host. Your setup is quite different I think.

I am not sure if my answer is very helpful but if not let us keep this conversation going.

Waldek

Hanaa Abbas

unread,

Oct 9, 2021, 4:14:10 PM10/9/21

to OSv Development

Hello,

Thank you for your reply.

I'll give more general info about the project details before I explain my numerous attempts.

package.yaml contains:

require:

- osv.openjdk11-zulu

- osv.run-java

- osv.cli

- osv.httpserver-html5-gui-and-cli

run.yaml:

runtime: native

config_set:

default:

bootcmd: "/java.so -jar /Jar.jar"

config_set_default: default

For the osv.openjdk11-zulu package, I had generated it from OSv build scripts (./scripts/build-capstan-mpm-packages openjdk-zulu-9-and-above)

The run.yaml is not used, since I specify the cmdline in the --execute option.

Capstan Run commands:

capstan run -v -n "bridge" -b "virbr0" -c 4 -m 1200M --execute "--verbose --ip=eth0,192.168.123.10,255.255.255.0 --defaultgw=192.168.123.1 --nameserver=192.168.123.1 /java.so -Xmx800m -Xss228k -jar /Jar.jar -debug -tIp 192.168.123.10 -d 3600" image-name

capstan run -v -n "bridge" -b "virbr0" -c 2 -m 1200M --execute "--verbose --ip=eth0,192.168.123.11,255.255.255.0 --defaultgw=192.168.123.1 --nameserver=192.168.123.1 /java.so -Xmx800m -Xss228k -jar /Jar.jar -debug -tIp 192.168.123.11 -d 1800" image-name

ifconfig

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500

inet 192.168.123.1 netmask 255.255.255.0 broadcast 192.168.123.255

ether 52:54:00:2e:bd:ea txqueuelen 1000 (Ethernet)

RX packets 0 bytes 0 (0.0 B)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 0 bytes 0 (0.0 B)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

When running both OSv instances on the same VM, they are able to communicate fine.

Now the problem emerges when the instances are run separately on two different VirtualBox VMs.

VirtualBox Network Settings: (VirtualBox is running on Windows OS)

I was hoping the wiki https://github.com/cloudius-systems/osv/wiki/Live-migration-of-OSv-guests-using-qemu would help since it has an end section showing the drawing of 2 OSv instances on QEMU but on 2 separate Linux hosts. (which is somewhat similar to what I want to achieve, if we consider the VirtualBox images as our Linux hosts). However, the wiki may not be complete to explain exactly how the two bridges were set up.

Hanaa Abbas

unread,

Oct 9, 2021, 6:29:21 PM10/9/21

to OSv Development

Additional Info:

The communication protocol is UDP. In the Java application, we are providing a config file containing the IP addresses of all OSv instances. As per the example above, those are "192.168.123.10" and "192.168.123.11". That is, sender/receiver IP addresses are known.

Now with regards to my attempts:

For the nested virtualization case:

using tcpdump on virbr0 interface on VM-1, I can see a bunch of ARP requests for the second OSV's IP. However, the ARP requests do not reach VM-2.

I tried several guides and researched a lot, but it was always a dead end.

You said "In general, for 2 OSv instances to communicate over TCP or UDP (which is what I am assuming) with each other, you need to enable network forwarding either through slirp or tap or bridge for each OSv instance : the capstan '-b' or '-p' options . The bottom line is that OSv instance 1 needs to know the "forwarded" IP of the OSv instance 2 and vice versa to communicate to each other."

Can you please elaborate on OSv instance 1 needs to know the "forwarded" IP of the OSv instance 2 and vice versa to communicate to each other?

I am already using the -b and -n options for specifying the bridge interface. As for -p option, it is the default for "qemu".

============================================================================================================================

For the case of running OSv directly on VirtualBox:

Attempt 1) WINDOWS OS

I copied all of the previously generated packages and osv-loader image and placed them under .capstan directory.

Capstan Run Command:

capstan run -v -p "vbox" -c 4 -m 1200M --execute "--verbose --ip=eth0,192.168.123.10,255.255.255.0 --defaultgw=192.168.123.1 --nameserver=192.168.123.1 /java.so -Xmx800m -Xss228k -jar /Jar.jar -debug -tIp 192.168.123.10 -d 1800" image_name

Command line will be set based on --run parameter

image_name: no such image at: C:\Users\hanaa\.capstan\repository\image_name\image_name.vbox

After checking all of capstan help, the only other command that allows me to specify the hypervisor as "vbox" is capstan build

So I tried that on my Windows host:

I had to add Capstanfile (osv-loader is an existing image under ./capstan/repository)

base: osv.openjdk11-zulu

cmdline: /java.so -jar /Jar.jar

build: mvn package

files:

/Jar.jar: target/Jar-jar-with-dependencies.jar

/app.keystore: app.keystore

/config/config.prop.xml: config.prop.xml

/config/topology.prop.xml: topology.prop.xml

I tried changing the base image to cloudius-systems/osv-openjdk8 (and others available in the GitHub repo) or adding the --s3 option to fetch an image from the public repo.

I created my own GitHub repository with the structure indicated in the Capstan Repository.md, however all attempts are trying to access github repository and all of them are indicating fatal error repository not found.

When I used an existing image (osv-loader) under .capstan/repository: I get this error

fatal: not a git repository: 'C:\Users\hanaa\.capstan\repository\osv-loader\.git'

Attempt 2) following Wiki Guide: https://github.com/cloudius-systems/osv/wiki/Running-OSv-on-VirtualBox

hanaa@hanaa:~/osv$ ./scripts/gen-vbox-ova.sh

I was able to generate OVA and VDI files containing these two packages only:

- osv.openjdk11-zulu

- osv.httpserver-html5-gui-and-cli

Q: How can I include my java project (JAR and some config files) within this image?

Note: I was not able to access the REST API using curl (curl: command not found). curl http://osv:8000/version

Also, I couldn't figure out the IP of the OSV machine.

I also suspect that the networking might still fail (on Windows VirtualBox) even after I manage to build my project within the OSV image.

The reason is the wiki states it expects VirtualBox Host-only Networks with the name vboxnet0.

Please note that the name of the VB Host-only network on windows is not modifiable!! I was able to start up the VM using an Internal Host network, however I cannot specify the IP of the machine in that case.

Please advice.

Regards

Hanaa

Hanaa Abbas

unread,

Oct 12, 2021, 3:54:35 AM10/12/21

to OSv Development

Hello,

I can work with OSv scripts instead of capstan to build the VM image.

However, eventually I would like to be able to run that image using capstan run -p "vbox" -c 4 -m 1200M ...

This is because I want to use the --execute option to pass (dynamic) arguments to each app instance. Otherwise, I'd have to generate an image for each instance rather than using one. Is my understanding correct?

My project structure is as simple as:

-project

--config directory containing 2 XML files: config.prop.xml and topology.prop.xml

--keystore file

-- JAR

I understood I need to add module.py:

from osv.modules import api

api.require('java')

java_cmd = "-Xmx800m -Xss228k -jar /Unicure.jar"

default = api.run('/java.so ' + java_cmd)

native = api.run('/usr/bin/java ' + java_cmd)

I also added a usr.manifest containing the files

I built the image using ./scripts/build image=openjdk-zulu-9-and-above,httpserver-html5-gui-and-cli,myproject

then followed by: ./scripts/gen-vbox-ova.sh

I copied the the .vbox and .vdi files generated under '/home/hanaa/VirtualBox VMs/osv/osv.vbox' and placed them under .capstan/repository/myproject

When I tried

capstan run -v -p "vbox" -m 1200M -c 4 --execute "--verbose --ip=eth0,192.168.100.10,255.255.255.0 --defaultgw=192.168.100.1 --nameserver=192.168.100.1 /java.so -Xmx800m -Xss228k -jar /Jar.jar -debug -kp ./k.keystore -tf ./config/topology.prop.xml -tIp 192.168.100.10 -cfg ./config/config.prop.xml" myproject

This is the error I got:

vbox: image format of /home/hanaa/.capstan/repository/myproject/myproject.vbox is not supported, unable to run it.

============================================================

I tried using the OVA files on virtualbox and finally they launched. However, the same networking problem persists.

I think I need to modify each image to use a static ip address and try again. I'll update this thread once I have any results.

Hanaa Abbas

unread,

Oct 14, 2021, 1:24:26 AM10/14/21

to OSv Development

Please any help regarding the nested virtualization bridge network configuration scenario??

Or for the second scenario, using capstan run to run VBOX images for our project setup?

- I either get fatal: not a git repository

- Or, vbox image format is not correct (when i take vbox image generated by OSv scripts)

Thank you! I need your urgent support please!

Much appreciated,

Hanaa

Waldek Kozaczuk

unread,

Oct 14, 2021, 9:35:01 PM10/14/21

to Hanaa Abbas, OSv Development

I have been pretty busy this week but I should have some time this weekend and cobble together some example.

Waldek

--
You received this message because you are subscribed to a topic in the Google Groups "OSv Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osv-dev/UX2QcPzbEJM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/40be5929-d2fb-4ad6-9c33-9c49401132f2n%40googlegroups.com.

Waldek Kozaczuk

unread,

Oct 19, 2021, 10:56:04 PM10/19/21

to Hanaa Abbas, OSv Development

Looking at your diagram below that shows OSv running on QEMU within Ubuntu running as VBox VM on Windows host, can you ping OSv instance from the host? Your sentence above does not mention it and I think this is key to make the communication happen between two OSv instances. I am not that familiar with Windows but you have to somehow make traffic flow all the way through the Windows host network.

So for example, once you know which IP 1 to use to ping OSv 1 from the Windows host, you with then use the same IP 1 in the configuration of the OS2 instance to make it communicate with OSv 1 and vice versa. This would only work provided the network traffic would correctly get routed through all these nested virtualization layers.

I tried many things but without any use.
This is my current VB network settings. The ubuntu env also has a bridged network (see screenshots).

I will really appreciate any help. Thank you

--

Hanaa Abbas

unread,

Oct 20, 2021, 11:09:43 AM10/20/21

to OSv Development

No, I cannot reach the nested VM from the host... I have spent an incredible amount of hours trying to figure this out. Unfortunately, my networking knowledge is not great.

Ok lets forget about Windows. I'll get a host with Ubuntu installed as main OS.

Do you know how to get the bridge network to work for QEMU within VirtualBox on an Ubuntu host?

Waldek Kozaczuk

unread,

Oct 22, 2021, 9:34:48 PM10/22/21

to OSv Development

Hi,

I think I have some good news for you regarding your nested virtualization case! It might even work on Windows but I have not tested it.

My setup involved a hardware host with Ubuntu 21.04 and two VirtualBox 6.1 VM (whatever version Ubuntu 21.04 comes with) running Ubuntu Server 20.04.3. Each VBox VM was configured with a networking setup to use Bridged Adapter using the real ethernet NIC on the physical host and enabled Promiscuous Mode equal to "Allow All" which is key and I see you had it enabled.

Now the key thing inside of the two Ubuntu VMs was to set up a so-called external bridge so that each OSv instance could use an IP from the same subnet the physical host is attached to. Under the scripts directory, there is a script called setup-external-bridge.sh (please see the wiki https://github.com/cloudius-systems/osv/wiki/Running-OSv-on-Firecracker#networking which for some details), which takes the name of the VM NIC as a 1st argument and the name of the bridge (the default is virbr1) as the 2nd. In essence, this script creates another bridge within a VM that both Ubuntu guest and then OSv sub-guest can belong to and also allows for traffic to flow through all levels up and down to the physical host. Here is how the Ubuntu VM interfaces look like:

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

ether 08:00:27:1c:68:ac txqueuelen 1000 (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

inet 127.0.0.1 netmask 255.0.0.0

inet6 ::1 prefixlen 128 scopeid 0x10<host>

loop txqueuelen 1000 (Local Loopback)

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500

inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255

ether 52:54:00:42:32:17 txqueuelen 1000 (Ethernet)

virbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.1.118 netmask 255.255.255.0 broadcast 192.168.1.255

ether 08:00:27:1c:68:ac txqueuelen 1000 (Ethernet)

Please note the guest "lost" an IP on enp0s3 NIC and "switched" to an IP 192.168.1.118 attached to virb1. In my case, the other Ubuntu VM had an address 192.168.1.119 and both of those came from DHCP server on my local network.

Now on each Ubuntu VM I would run an OSv instance using a shell script runOSv_bridged.sh based on the output from "./scripts/run.py -k -n -b virbr1 --mac 52:54:00:12:34:01 --dry-run" (-k makes it easier to pass arguments):

COMMAND="$1"

qemu-system-x86_64 \

-m 256M \

-smp 1 \

-vnc :1 \

-gdb tcp::1234,server,nowait \

-kernel ./kernel.elf \

-append "--rootfs=rofs $COMMAND" \

-device virtio-blk-pci,id=blk0,drive=hd0,scsi=off \

-drive file=/home/wkozaczuk/usr.img,if=none,id=hd0,cache=none,aio=native \

-netdev bridge,id=hn0,br=virbr1,helper=/usr/lib/qemu/qemu-bridge-helper \

-device virtio-net-pci,netdev=hn0,id=nic0,mac=52:54:00:12:34:01 \

-device virtio-rng-pci \

-chardev stdio,mux=on,id=stdio,signal=off \

-mon chardev=stdio,mode=readline \

-device isa-serial,chardev=stdio

To test the communication between the OSv instances I used the cli app built like so:

./scripts/build image=cli fs=rofs

and then run it on VM1 like so:

sudo ./runOSv_bridged.sh "--ip=eth0,192.168.1.120,255.255.255.0 --defaultgw=192.168.1.254 --nameserver=192.168.1.254 /cli -a http://192.168.1.121:8000"

and then run it on VM2 like so:

sudo ./runOSv_bridged.sh "--ip=eth0,192.168.1.121,255.255.255.0 --defaultgw=192.168.1.254 --nameserver=192.168.1.254 /cli -a http://192.168.1.120:8000"

The full output here:

"

sudo ./runOSv_bridged.sh "--ip=eth0,192.168.1.120,255.255.255.0 --defaultgw=192.168.1.254 --nameserver=192.168.1.254 /cli -a http://192.168.1.121:8000"

[sudo] password for wkozaczuk:

OSv v0.56.0-5-g247d6837

eth0: 192.168.1.120

Booted up in 0.00 ms

Cmdline: /cli -a http://192.168.1.121:8000

Rest API server running on port 8000

/# dmesg

1 CPUs detected

Firmware vendor: SeaBIOS

bsd: initializing - done

VFS: mounting ramfs at /

VFS: mounting devfs at /dev

net: initializing - done

vga: Add VGA device instance

eth0: ethernet address: 52:54:00:12:34:02

virtio-blk: Add blk device instances 0 as vblk0, devsize=20144128

random: virtio-rng registered as a source.

random: <Software, Yarrow> initialized

VFS: unmounting /dev

VFS: mounting rofs at /rofs

VFS: mounting devfs at /dev

VFS: mounting procfs at /proc

VFS: mounting sysfs at /sys

VFS: mounting ramfs at /tmp

Running from /init/30-auto-03: /libhttpserver-api.so &!

random: device unblocked.

/usr/lib/libbsd.so.0: ignoring missing symbol fopencookie

/usr/lib/libbsd.so.0: ignoring missing symbol warnx

/usr/lib/libbsd.so.0: ignoring missing symbol sigpending

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_api.so

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_app.so

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_env.so

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_file.so

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_fs.so

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_hardware.so

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_network.so

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_os.so

httpserver: loaded plugin from path: /usr/mgmt/plugins/libhttpserver-api_trace.so

sysconf(): stubbed for parameter 43

/#

sudo ./runOSv_bridged.sh "--ip=eth0,192.168.1.121,255.255.255.0 --defaultgw=192.168.1.254 --nameserver=192.168.1.254 /cli -a http://192.168.1.120:8000"

OSv v0.56.0-5-g247d6837

eth0: 192.168.1.121

Booted up in 0.00 ms

Cmdline: /cli -a http://192.168.1.120:8000

Rest API server running on port 8000

/# dmesg

1 CPUs detected

Firmware vendor: SeaBIOS

bsd: initializing - done

VFS: mounting ramfs at /

VFS: mounting devfs at /dev

net: initializing - done

vga: Add VGA device instance

eth0: ethernet address: 52:54:00:12:34:01