[Q] Linux HVM & inter-VM networking
597 views
Skip to first unread message
Thomas Portmann
Feb 9, 2016, 6:37:01 AM2/9/16
to qubes-users
Hello,
According "qubes-firewall"
(https://www.qubes-os.org/doc/qubes-firewall/, part "Enabling networking
between two VMs"), I managed to have the networking work between two VMs
(both based on the debian-8 template). I had to add a rule on the
sys-firewall VM, as specified:
iptables -I FORWARD 2 -s 10.137.2.10 -d 10.137.2.13 -j ACCEPT
I also had to add a rule on the destination VM:
iptables -I INPUT 1 -s 10.137.2.10 -p icmp -j ACCEPT
Then I created an HVM and installed Debian. I had to configure the
network manually and followed the steps in "hvm-create"
(https://www.qubes-os.org/doc/hvm-create/, part "Setting up networking
for HVM domains"). According to the VM manager, the HVM has IP
10.137.2.12, netmask 255.255.255.0 and gateway 10.137.2.1. Configuring
this by hand gives:
ifconfig eth0 10.137.2.12/24 up
route add default gw 10.137.2.1
Now the hard part where I have a problem. I want the 10.137.2.10 VM to
be able to access the HVM (10.137.2.12), therefor I add a rule on the
sys-firewall VM:
iptables -I FORWARD 2 -s 10.137.2.10 -d 10.137.2.12 -j ACCEPT
Since on the HVM I have no firewall running, I don't have to open it.
When pinging from the source to the destination (HVM), I see the ICMP
echo requests arriving on the destination (10.137.2.12), however, in
order to answer it does an ARP request to obtain the MAC address of
10.137.2.10, which it thinks is in the same subnet (10.137.2.0/24).
Looking how the NIC is configured in an AppVM, I see (ifconfig eth0) an
IP 10.137.2.10 with an broadcast 10.255.255.255 and a netmask
255.255.255.255 which seems just plain wrong to me.
Any idea on how to configure the network in the HVM to make the inter-VM
networking work?
BTW, I did this all on Qubes 3.1RC2
Thanks in advance and best regards
tom
According "qubes-firewall"
(https://www.qubes-os.org/doc/qubes-firewall/, part "Enabling networking
between two VMs"), I managed to have the networking work between two VMs
(both based on the debian-8 template). I had to add a rule on the
sys-firewall VM, as specified:
iptables -I FORWARD 2 -s 10.137.2.10 -d 10.137.2.13 -j ACCEPT
I also had to add a rule on the destination VM:
iptables -I INPUT 1 -s 10.137.2.10 -p icmp -j ACCEPT
Then I created an HVM and installed Debian. I had to configure the
network manually and followed the steps in "hvm-create"
(https://www.qubes-os.org/doc/hvm-create/, part "Setting up networking
for HVM domains"). According to the VM manager, the HVM has IP
10.137.2.12, netmask 255.255.255.0 and gateway 10.137.2.1. Configuring
this by hand gives:
ifconfig eth0 10.137.2.12/24 up
route add default gw 10.137.2.1
Now the hard part where I have a problem. I want the 10.137.2.10 VM to
be able to access the HVM (10.137.2.12), therefor I add a rule on the
sys-firewall VM:
iptables -I FORWARD 2 -s 10.137.2.10 -d 10.137.2.12 -j ACCEPT
Since on the HVM I have no firewall running, I don't have to open it.
When pinging from the source to the destination (HVM), I see the ICMP
echo requests arriving on the destination (10.137.2.12), however, in
order to answer it does an ARP request to obtain the MAC address of
10.137.2.10, which it thinks is in the same subnet (10.137.2.0/24).
Looking how the NIC is configured in an AppVM, I see (ifconfig eth0) an
IP 10.137.2.10 with an broadcast 10.255.255.255 and a netmask
255.255.255.255 which seems just plain wrong to me.
Any idea on how to configure the network in the HVM to make the inter-VM
networking work?
BTW, I did this all on Qubes 3.1RC2
Thanks in advance and best regards
tom
Stefan
Feb 9, 2016, 6:50:49 AM2/9/16
to qubes-users, Thomas....@bger.ch
Am Dienstag, 9. Februar 2016 12:37:01 UTC+1 schrieb Thomas Portmann:
Since on the HVM I have no firewall running, I don't have to open it.
When pinging from the source to the destination (HVM), I see the ICMP
echo requests arriving on the destination (10.137.2.12), however, in
order to answer it does an ARP request to obtain the MAC address of
10.137.2.10, which it thinks is in the same subnet (10.137.2.0/24).
Looking how the NIC is configured in an AppVM, I see (ifconfig eth0) an
IP 10.137.2.10 with an broadcast 10.255.255.255 and a netmask
255.255.255.255 which seems just plain wrong to me.
Any idea on how to configure the network in the HVM to make the inter-VM
networking work?
BTW, I did this all on Qubes 3.1RC2
I think I had exactly the same problem yesterday I tried to access
the network shares of a Windows HVM from a Linux AppVM. This worked
using the "-I FORWARD" iptables rule on Qubes 3.0, but stopped working
on Qubes 3.1.
Solution: Qubes 3.1 disables ProxyARP by default; enable it on your
Firewall VM:
echo 1 > /proc/sys/net/ipv4/conf/$DEST_DEV/proxy_arp
where DEST_DEV is the device name of your target VM.
That did the trick in my case.
Stefan. P.S.: Sorry if this is a double post; I first tried to send this via my mail client, but the answer didn't come through…
Stefan Schlott
Feb 9, 2016, 8:13:48 AM2/9/16
to qubes...@googlegroups.com
Hi,
> Since on the HVM I have no firewall running, I don't have to open it.
> When pinging from the source to the destination (HVM), I see the ICMP
> echo requests arriving on the destination (10.137.2.12), however, in
> order to answer it does an ARP request to obtain the MAC address of
> 10.137.2.10, which it thinks is in the same subnet (10.137.2.0/24).
> Looking how the NIC is configured in an AppVM, I see (ifconfig eth0) an
> IP 10.137.2.10 with an broadcast 10.255.255.255 and a netmask
> 255.255.255.255 which seems just plain wrong to me.
>
> Any idea on how to configure the network in the HVM to make the inter-VM
> networking work?
>
> BTW, I did this all on Qubes 3.1RC2
I think I had exactly the same problem yesterday :-) I tried to access
> Since on the HVM I have no firewall running, I don't have to open it.
> When pinging from the source to the destination (HVM), I see the ICMP
> echo requests arriving on the destination (10.137.2.12), however, in
> order to answer it does an ARP request to obtain the MAC address of
> 10.137.2.10, which it thinks is in the same subnet (10.137.2.0/24).
> Looking how the NIC is configured in an AppVM, I see (ifconfig eth0) an
> IP 10.137.2.10 with an broadcast 10.255.255.255 and a netmask
> 255.255.255.255 which seems just plain wrong to me.
>
> Any idea on how to configure the network in the HVM to make the inter-VM
> networking work?
>
> BTW, I did this all on Qubes 3.1RC2
Marek Marczykowski-Górecki
Feb 9, 2016, 8:41:57 AM2/9/16
to Stefan Schlott, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Can you describe exactly your setup, including network configuration in
Windows? "route print" command output from there would be useful.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJWueydAAoJENuP0xzK19cs+ZsH/3hpmyTOfVP7zfU1n7DDrRlX
65EbS25Z0aiq62YFwzZtPrVjXFKI0/QKJtLkISQFq5EgXB0p0g4wZlmXZK9P4J9n
fOYDQ5Q5SR1lumLM9n7Qfgj9w9gB1ZPEZAFLdHkSVE+iz4p8ujk6eZ9I/wbauaOc
05+VV1WsFCCGvlKWqJNbbUgFrNN4moRE0e/VzKmfrBah7+o8H8UI5ooKy28XQ0GU
6pn1vyaedCGrrQcV32Q2BrLiiGBtX7Y/Wo6G2gSGlaBP4ezo/ASPEBttEyj/ejV4
MNq+mWPKgxeAYVn5+Q1Kwz0YvKSaQD3TCzrqtbe/VIZqtuCiBHUxTMGW995rPpE=
=c79o
-----END PGP SIGNATURE-----
Hash: SHA256
Windows? "route print" command output from there would be useful.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJWueydAAoJENuP0xzK19cs+ZsH/3hpmyTOfVP7zfU1n7DDrRlX
65EbS25Z0aiq62YFwzZtPrVjXFKI0/QKJtLkISQFq5EgXB0p0g4wZlmXZK9P4J9n
fOYDQ5Q5SR1lumLM9n7Qfgj9w9gB1ZPEZAFLdHkSVE+iz4p8ujk6eZ9I/wbauaOc
05+VV1WsFCCGvlKWqJNbbUgFrNN4moRE0e/VzKmfrBah7+o8H8UI5ooKy28XQ0GU
6pn1vyaedCGrrQcV32Q2BrLiiGBtX7Y/Wo6G2gSGlaBP4ezo/ASPEBttEyj/ejV4
MNq+mWPKgxeAYVn5+Q1Kwz0YvKSaQD3TCzrqtbe/VIZqtuCiBHUxTMGW995rPpE=
=c79o
-----END PGP SIGNATURE-----
Stefan
Feb 9, 2016, 9:06:47 AM2/9/16
to qubes-users, ste...@ploing.de
Am Dienstag, 9. Februar 2016 14:41:57 UTC+1 schrieb Marek Marczykowski-Górecki:
> Solution: Qubes 3.1 disables ProxyARP by default; enable it on your
> Firewall VM:
>
> echo 1 > /proc/sys/net/ipv4/conf/$DEST_DEV/proxy_arp
>
> where DEST_DEV is the device name of your target VM.
>
>
> That did the trick in my case.
Can you describe exactly your setup, including network configuration in
Windows? "route print" command output from there would be useful.
With pleasure :-)
Linux AppVM (10.137.3.15), based on the fedora-23 template, no modifications to the AppVM firewall
Windows HVM with Windows 7 (10.137.3.13), no Qubes Tools. Configured as home network, firewall disabled (though I don't think this is any longer necessary).
Both connected to the same FirewallVM (fedora-23 template).
Tried to establish a connection from Linux to Windows by following https://www.qubes-os.org/doc/qubes-firewall/:
sudo iptables -I FORWARD 2 -s 10.137.3.15 -d 10.137.3.13 -j ACCEPT
pinging didn't work, so I started digging… this is the routing of the FirewallVM:
$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.137.2.1 0.0.0.0 UG 0 0 0 eth0
10.137.2.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
10.137.3.13 0.0.0.0 255.255.255.255 UH 32744 0 0 vif8.0
10.137.3.13 0.0.0.0 255.255.255.255 UH 32745 0 0 vif7.0
10.137.3.15 0.0.0.0 255.255.255.255 UH 32747 0 0 vif5.0
Note that the Windows VM has two interfaces (for whatever reason).
Sniffing with Wireshark on vif7.0 showed no traffic. vif8.0 revealed that:
- the icmp requests arrive on the interface
- Windows sends ARP requests for 10.137.3.15
- ...but gets no ARP response
Manually adding an ARP entry didn't do the trick. Windows now sent ping replies (visible on vif8.0), but they didn't reach the Linux AppVM.
Enabling proxy_arp for vif8.0 solved the problem.
Enclosed is a screenshot of Windows' "route print" (sorry, no Qubes Tools, no copy-paste).
Hope this helps :-)
Stefan.
Marek Marczykowski-Górecki
Feb 9, 2016, 9:18:41 AM2/9/16
to Stefan, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
perhaps it would require additional route entry to 10.137.3.1 directly
through emulated ethernet. But in such a case, enabling proxy_arp is
just the simples solution.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
eb0IkOtU9JivFwOxi84jqmGedpcPOl+a9bcGWn5sXZYKuQQI+adah5+nGvV4veEi
9olrtXXuAbHeGYv7geLU4Yhtm0dkW/iSA7FkZoe8Gq9V/u9NR5FaCld6kmqBX60U
bv8mE+j4W86Rl0Za4FbDoACx0/VLfOiXVm/V37j2+59y0ocdTd1lH+W67Bo37J3n
Qnrutk8izL6QqkUntvUZOhVvqPmx6OewcY0i+kVEgDGRdG97WFAQ9rVYUf5FVLai
xyY+xauTj1vuIiypVF/HLv4uT8HVChCKjp7S12E61DI1YvnmhvyajOcBDe4gUYE=
=sltI
-----END PGP SIGNATURE-----
li...@mullvad.net
Apr 6, 2016, 4:39:40 AM4/6/16
to qubes-users, ste...@ploing.de
I have the same problem between an Ubuntu 14.04 HVM and a debian-8 AppVM. No arp responses sent to the HVM. I have the same setup as Thomas with a 255.255.255.0 mask in the HVM. Setting it to 255.255.255.255 didn't help (Because now my gw is outside my local network?)
I have tried basing my ProxyVM on both fedora-23-minimal and fedora-23, same results.
I could temporarily make it work by manually adding an arp entry in the HVM:
$ sudo arp -s <AppVM IP> <ProxyVM MAC>
But after finding this thread I solved it by activating proxy arp in /rw/config/qubes-firewall-user-script on the ProxyVM:
# Activate proxy arp
for i in /proc/sys/net/ipv4/conf/vif*/proxy_arp; do
echo 1 > $i
done
Are there any security implications of activating proxy arp? Why was it turned off (everything worked as expected in Qubes R3.0)
/Linus
Pablo Di Noto
Oct 5, 2016, 5:54:32 PM10/5/16
to qubes-users, ste...@ploing.de
El miércoles, 6 de abril de 2016, 5:39:40 (UTC-3),
li...@mullvad.net escribió:
Should we include this instructions on the https://www.qubes-os.org/doc/qubes-firewall/ page?
Somehow, it took me a whole day to get to this post. I just wanted to access a new HVM thru SSH.
I can add this info into it, if there is editing available.
Regards,
///Pablo
li...@mullvad.net escribió:
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
>
> Marek Marczykowski-Górecki
> Invisible Things Lab
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
>
> I have the same problem between an Ubuntu 14.04 HVM and a debian-8 AppVM. No arp responses sent to the HVM. I have the same setup as Thomas with a 255.255.255.0 mask in the HVM. Setting it to 255.255.255.255 didn't help (Because now my gw is outside my local network?)
>
> I have tried basing my ProxyVM on both fedora-23-minimal and fedora-23, same results.
>
> I could temporarily make it work by manually adding an arp entry in the HVM:
> $ sudo arp -s <AppVM IP> <ProxyVM MAC>
>
> But after finding this thread I solved it by activating proxy arp in /rw/config/qubes-firewall-user-script on the ProxyVM:
> # Activate proxy arp
> for i in /proc/sys/net/ipv4/conf/vif*/proxy_arp; do
> echo 1 > $i
> done
>
> Are there any security implications of activating proxy arp? Why was it turned off (everything worked as expected in Qubes R3.0)
>
> /Linus
I have just ran into this issue while using Qubes R3.2, and a Ubuntu HVM.
>
> I have tried basing my ProxyVM on both fedora-23-minimal and fedora-23, same results.
>
> I could temporarily make it work by manually adding an arp entry in the HVM:
> $ sudo arp -s <AppVM IP> <ProxyVM MAC>
>
> But after finding this thread I solved it by activating proxy arp in /rw/config/qubes-firewall-user-script on the ProxyVM:
> # Activate proxy arp
> for i in /proc/sys/net/ipv4/conf/vif*/proxy_arp; do
> echo 1 > $i
> done
>
> Are there any security implications of activating proxy arp? Why was it turned off (everything worked as expected in Qubes R3.0)
>
> /Linus
Should we include this instructions on the https://www.qubes-os.org/doc/qubes-firewall/ page?
Somehow, it took me a whole day to get to this post. I just wanted to access a new HVM thru SSH.
I can add this info into it, if there is editing available.
Regards,
///Pablo
Andrew David Wong
Oct 5, 2016, 7:03:14 PM10/5/16
to Pablo Di Noto, qubes-users, ste...@ploing.de
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Please feel free to submit a pull request against this page (click the "edit this page" link). Please also note our documentation guidelines, which contains a how-to guide for contributing:
https://www.qubes-os.org/doc/doc-guidelines/
- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJX9YaiAAoJENtN07w5UDAwtJAQAL5W5DejGUqeLl71blpmLbOj
iDQTrAGCVOAL389FA809ygutehwI1vIlamBSw2URrLy7IL5x7bL4ZbhYUfQPdHNp
AfK5Yqb2UZ6/cELmIYTQAOrz4lP+wRtz8uCvg0iDIBcojueXz1VxxIOOVBwljfo7
2RDasvNXHxS6XrmLfZaqIfUG65HKDsaSq93pr/Jje2ariVvQXg6ZDELFJpiejjLL
o1gszkyaAOC7hqTveaxRQ3eaTid3KO3PWIQ50YQf07bBf3SjlV6i/ThEAS0sqtTk
e4vm0cOz6WZ8A6z/nlv/237g4HOMD8x/HlMmjKVA74uxiB5joCcUo2YPnmRwyH7u
xVpr6O3E84xtP2Cwf8NnEPgpmbKZvbmFWVyMkk1QcXD89V05P7Ht/tL/aT9B56wd
GDj9YFO87yI7HoEcagWnFFq/dkW8So//MzqOMnfC5f3cARciGg+Pz9yCC7TswBhE
QzLxXB+SoiC5+Gfcv8jZFRurH8GUvDK2ZWCtT5SLW+b9/pR9Deuhd9T/ccXE/htJ
0tXOE02e9u4l/gFsIslJQszlHrQ5VDxANjcNDRfETF0FQsymYL9XURTUeCZZLB3u
F8d5w9f1obGbwVT+mLJca/2aGDM7s9Jer8DkimMeXyAozqYbE6lgsqPS5PKB6Ddp
FOMHiqud6duoObkrYO5U
=J6/u
-----END PGP SIGNATURE-----
Hash: SHA512
https://www.qubes-os.org/doc/doc-guidelines/
- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJX9YaiAAoJENtN07w5UDAwtJAQAL5W5DejGUqeLl71blpmLbOj
iDQTrAGCVOAL389FA809ygutehwI1vIlamBSw2URrLy7IL5x7bL4ZbhYUfQPdHNp
AfK5Yqb2UZ6/cELmIYTQAOrz4lP+wRtz8uCvg0iDIBcojueXz1VxxIOOVBwljfo7
2RDasvNXHxS6XrmLfZaqIfUG65HKDsaSq93pr/Jje2ariVvQXg6ZDELFJpiejjLL
o1gszkyaAOC7hqTveaxRQ3eaTid3KO3PWIQ50YQf07bBf3SjlV6i/ThEAS0sqtTk
e4vm0cOz6WZ8A6z/nlv/237g4HOMD8x/HlMmjKVA74uxiB5joCcUo2YPnmRwyH7u
xVpr6O3E84xtP2Cwf8NnEPgpmbKZvbmFWVyMkk1QcXD89V05P7Ht/tL/aT9B56wd
GDj9YFO87yI7HoEcagWnFFq/dkW8So//MzqOMnfC5f3cARciGg+Pz9yCC7TswBhE
QzLxXB+SoiC5+Gfcv8jZFRurH8GUvDK2ZWCtT5SLW+b9/pR9Deuhd9T/ccXE/htJ
0tXOE02e9u4l/gFsIslJQszlHrQ5VDxANjcNDRfETF0FQsymYL9XURTUeCZZLB3u
F8d5w9f1obGbwVT+mLJca/2aGDM7s9Jer8DkimMeXyAozqYbE6lgsqPS5PKB6Ddp
FOMHiqud6duoObkrYO5U
=J6/u
-----END PGP SIGNATURE-----
Drew White
Oct 5, 2016, 8:19:23 PM10/5/16
to qubes-users, pdi...@gmail.com, ste...@ploing.de
Easiest way to perform the action on an "InterVM" ProxyVM or NetVM
This is a part of my personal script that allows me to have an InerVM Network along with external networking available.
I'm a developer, so I use this sort of thing for testing and more.
This is a part of my personal script that allows me to have an InerVM Network along with external networking available.
I'm a developer, so I use this sort of thing for testing and more.
Replace X with the number for the network you have, whether it's a 1/2/3/4/5/6/7/8/9/10/11/12/13 ... etc...
My personal script auto detects the IPs upon boot. Then sets everything accordingly.
So as you can see, it works, and I have been using this for a long time.
#--------------------------------------------
intervm_internalnet='10.137.X.0';
# Sets iptables so that anything targeting local network can find itself. Only use for interVM machine.
iptables -I FORWARD 1 -i vif+ -o vif+ -s $intervm_internalnet/24 -d $intervm_internalnet/24 -m state --state NEW -p tcp -m tcp -j ACCEPT
iptables -I FORWARD 1 -i vif+ -o vif+ -s $intervm_internalnet/24 -d $intervm_internalnet/24 -p udp -m udp -j ACCEPT
#--------------------------------------------
Reply all
Reply to author
Forward
0 new messages