Trouble with enabling networking between two Vms
703 views
Skip to first unread message
Max
Oct 23, 2016, 5:11:48 AM10/23/16
to qubes-users
Hi,
I am a new user of Qubes OS so apologies in advance if the question here has been answered already in a separate topic (there are similar issues) and I haven’t discovered this or it is not one suited to this mailing list. I am running Qubes 3.2 and attempting to ping from one VM to another VM, specifically from a Standalone Windows 7 VM to a Qubes VM based on the Debian 8 template.
All my VM’s were initially connected in the default manner i.e. to a sys-firewall and through to the sys-net VM, both of which are Fedora 23. There are no firewall rules on these VMs restricting which IP addresses can be accessed.
Current status:
- I am able to ping from my Windows 7 VM (10.137.2.19) to the Firewall VM (10.137.1.8) using the IP address visible in the VM Manager
- I am unable to ping the Debian 8 VM (10.137.2.18) from my Windows VM.
Steps taken:
1) I followed the instructions here (https://www.qubes-os.org/doc/qubes-firewall/#enabling-networking-between-two-vms) and in the firewall VM’s terminal enter the following iptables rule...
sudo iptables -I FORWARD 2 -s <IP address of Windows 7 VM> -d <IP address of Debian 8 VM> -j ACCEPT
… In VM B’s terminal (Debian 8) I entered the following iptables rule...
sudo iptables -I INPUT -s <IP address of Windows 7 VM> -j ACCEPT
...but from here when using the ping function to my Debian 8 VM in the cmd prompt in Windows, all packets were lost.
2) As this was not successful I attempted to see if I could connect to VMs from an external machine and followed the instructions here https://www.qubes-os.org/doc/qubes-firewall/#port-forwarding-to-a-vm-from-the-outside-world.
The Eth0 IP address (192.168.1.6) appeared to be what I should expose the service to.
I put the below rule in the sys-net VM’s Terminal...
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -d 192.168.x.x -j DNAT --to-destination 10.137.1.x
...and this rule into the sys-firewall VM’s Terminal
iptables -I FORWARD 2 -i eth0 -d 10.137.1.x -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
But using ping or Telnet resulted in lost packets and failed to increase the counters when using the iptables -t nat -L -v -n command in the sys-firewall VM's terminal.
3) With this not being successful either I attempted to add a “sys-proxy” VM as described here https://groups.google.com/forum/#!searchin/qubes-users/intervm%7Csort:relevance/qubes-users/lA2SgPcV9fU/U969uapYAAAJ and entered the following in the new sys-proxy VM's terminal:
iptables -I FORWARD 1 -i vif+ -o vif+ -s $intervm_internalnet/24 -d $intervm_internalnet/24 -m state --state NEW -p tcp -m tcp -j ACCEPT
iptables -I FORWARD 1 -i vif+ -o vif+ -s $intervm_internalnet/24 -d $intervm_internalnet/24 -p udp -m udp -j ACCEPT
After this, I was still unable to ping the Debian 8 VM from my Windows VM.
Questions:
1) Are there any obvious errors in the steps I took and does anyone have any suggestions how I can resolve this issue?
2) There are a number of other incidences of what seemed to be a similar issue here: https://groups.google.com/forum/?nomobile=true#!msg/qubes-users/59kOjfQFBI4/bjS47-jJJgAJ, https://groups.google.com/forum/#!msg/qubes-users/vSyUaOSloYU/ONZNJlhrBAAJ. Are the enabling networking between VMs steps described here still correct and applicable for Qubes 3.2?
3) The IP address assignment suggests that the VMs are on the same network – the Subnet Mask is 255.255.255.0 so surely any devices with an IP address of 10.137.2.x would be able to communicate with each other? What is unique in Xen / Qubes that stops this?
4) Is there a way in which the current routing rules can be displayed and reset back to the default if required?
I am a new user of Qubes OS so apologies in advance if the question here has been answered already in a separate topic (there are similar issues) and I haven’t discovered this or it is not one suited to this mailing list. I am running Qubes 3.2 and attempting to ping from one VM to another VM, specifically from a Standalone Windows 7 VM to a Qubes VM based on the Debian 8 template.
All my VM’s were initially connected in the default manner i.e. to a sys-firewall and through to the sys-net VM, both of which are Fedora 23. There are no firewall rules on these VMs restricting which IP addresses can be accessed.
Current status:
- I am able to ping from my Windows 7 VM (10.137.2.19) to the Firewall VM (10.137.1.8) using the IP address visible in the VM Manager
- I am unable to ping the Debian 8 VM (10.137.2.18) from my Windows VM.
Steps taken:
1) I followed the instructions here (https://www.qubes-os.org/doc/qubes-firewall/#enabling-networking-between-two-vms) and in the firewall VM’s terminal enter the following iptables rule...
sudo iptables -I FORWARD 2 -s <IP address of Windows 7 VM> -d <IP address of Debian 8 VM> -j ACCEPT
… In VM B’s terminal (Debian 8) I entered the following iptables rule...
sudo iptables -I INPUT -s <IP address of Windows 7 VM> -j ACCEPT
...but from here when using the ping function to my Debian 8 VM in the cmd prompt in Windows, all packets were lost.
2) As this was not successful I attempted to see if I could connect to VMs from an external machine and followed the instructions here https://www.qubes-os.org/doc/qubes-firewall/#port-forwarding-to-a-vm-from-the-outside-world.
The Eth0 IP address (192.168.1.6) appeared to be what I should expose the service to.
I put the below rule in the sys-net VM’s Terminal...
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -d 192.168.x.x -j DNAT --to-destination 10.137.1.x
...and this rule into the sys-firewall VM’s Terminal
iptables -I FORWARD 2 -i eth0 -d 10.137.1.x -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
But using ping or Telnet resulted in lost packets and failed to increase the counters when using the iptables -t nat -L -v -n command in the sys-firewall VM's terminal.
3) With this not being successful either I attempted to add a “sys-proxy” VM as described here https://groups.google.com/forum/#!searchin/qubes-users/intervm%7Csort:relevance/qubes-users/lA2SgPcV9fU/U969uapYAAAJ and entered the following in the new sys-proxy VM's terminal:
iptables -I FORWARD 1 -i vif+ -o vif+ -s $intervm_internalnet/24 -d $intervm_internalnet/24 -m state --state NEW -p tcp -m tcp -j ACCEPT
iptables -I FORWARD 1 -i vif+ -o vif+ -s $intervm_internalnet/24 -d $intervm_internalnet/24 -p udp -m udp -j ACCEPT
After this, I was still unable to ping the Debian 8 VM from my Windows VM.
Questions:
1) Are there any obvious errors in the steps I took and does anyone have any suggestions how I can resolve this issue?
2) There are a number of other incidences of what seemed to be a similar issue here: https://groups.google.com/forum/?nomobile=true#!msg/qubes-users/59kOjfQFBI4/bjS47-jJJgAJ, https://groups.google.com/forum/#!msg/qubes-users/vSyUaOSloYU/ONZNJlhrBAAJ. Are the enabling networking between VMs steps described here still correct and applicable for Qubes 3.2?
3) The IP address assignment suggests that the VMs are on the same network – the Subnet Mask is 255.255.255.0 so surely any devices with an IP address of 10.137.2.x would be able to communicate with each other? What is unique in Xen / Qubes that stops this?
4) Is there a way in which the current routing rules can be displayed and reset back to the default if required?
Daniel Wilcox
Oct 23, 2016, 10:30:43 AM10/23/16
to Max, qubes-users
Hi Max, so it looks like you started getting complicated quick. I think your first attempt should be fine actually, with one modification.
--
You received this message because you are subscribed to the Google Groups "qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscribe@googlegroups.com.
To post to this group, send email to qubes...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/0514e15b-950e-4636-95f7-849fc5671fc1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Unman
Oct 23, 2016, 8:30:28 PM10/23/16
to Max, qubes-users
I would make sure the basics work before moving on.
1. You haven't allowed return traffic from the Debian qube.
Put in an ACCEPT FORWARD rule as you have with source and destination
reversed.
The rules you have entered to allow forwarding are for traffic to port
443. You don't seem to have either ping (icmp) or telnet(tcp port 23)
enabled.
These look like obvious mistakes.
2) Yes, I believe the instructions are still correct.
3) qubes are connected through a netvm - the default firewall rules
there prohibit traffic between qubes connected downstream: in the
FORWARD chain is -
DROP all vif+ vif+
4) The firewall and routing rules can be displayed using standard Linux
commands: iptables and ip route. (I'm not sure exactly what you are
looking for with this question.) Any changes that you make to a qube
will be reset on reboot UNLESS you explicitly make changes in /rw/config
using rc.local and the qubes-firewall script, or use some other
mechanism.
There isn't currently a way to see the full route from sys-net through
to a target qube, but it isnt difficult to get output from the
intermediates.
Hope this helps - try the simple window7 deb-8 connection first,and
then move on to the port forwarding.
Cheers
unman
Max
Oct 27, 2016, 11:48:28 AM10/27/16
to qubes-users, maxtan...@gmail.com, un...@thirdeyesecurity.org
Thank you for your detailed and quick responses. I have attempted to follow the instructions but have had not much luck yet. Some responses first:
> 4) The firewall and routing rules can be displayed using standard Linux
> commands: iptables and ip route. (I'm not sure exactly what you are
> looking for with this question.) Any changes that you make to a qube
> will be reset on reboot UNLESS you explicitly make changes in /rw/config
> using rc.local and the qubes-firewall script, or use some other
> mechanism.
> 3) qubes are connected through a netvm - the default firewall rules
> there prohibit traffic between qubes connected downstream: in the
> FORWARD chain is -
> DROP all vif+ vif+
> The rules you have entered to allow forwarding are for traffic to port
> 443. You don't seem to have either ping (icmp) or telnet(tcp port 23)
> enabled.
> 1. You haven't allowed return traffic from the Debian qube.
> Put in an ACCEPT FORWARD rule as you have with source and destination
> reversed.
Below is a copy of the iptables lists with the before and after results:
BEFORE - FIREWALL
[user@sys-firewall ~]$ sudo iptables -L -vx
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP udp -- vif+ any anywhere anywhere udp dpt:bootpc
0 0 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- any any anywhere anywhere
0 0 ACCEPT all -- lo any anywhere anywhere
0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- vif0.0 any anywhere anywhere
0 0 DROP all -- vif+ vif+ anywhere anywhere
0 0 ACCEPT udp -- any any 10.137.2.19 gateway udp dpt:domain
0 0 ACCEPT udp -- any any 10.137.2.19 10.137.1.254 udp dpt:domain
0 0 ACCEPT tcp -- any any 10.137.2.19 gateway tcp dpt:domain
0 0 ACCEPT tcp -- any any 10.137.2.19 10.137.1.254 tcp dpt:domain
0 0 ACCEPT icmp -- any any 10.137.2.19 anywhere
0 0 DROP tcp -- any any 10.137.2.19 10.137.255.254 tcp dpt:us-cli
0 0 ACCEPT all -- any any 10.137.2.19 anywhere
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
AFTER - FIREWALL
[user@sys-firewall ~]$ sudo iptables -L -vx
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP udp -- vif+ any anywhere anywhere udp dpt:bootpc
0 0 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- any any anywhere anywhere
0 0 ACCEPT all -- lo any anywhere anywhere
0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
116 9277 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- any any 10.137.2.19 10.137.2.18
0 0 ACCEPT all -- vif0.0 any anywhere anywhere
0 0 DROP all -- vif+ vif+ anywhere anywhere
0 0 ACCEPT udp -- any any 10.137.2.19 gateway udp dpt:domain
0 0 ACCEPT udp -- any any 10.137.2.19 10.137.1.254 udp dpt:domain
0 0 ACCEPT tcp -- any any 10.137.2.19 gateway tcp dpt:domain
0 0 ACCEPT tcp -- any any 10.137.2.19 10.137.1.254 tcp dpt:domain
0 0 ACCEPT icmp -- any any 10.137.2.19 anywhere
0 0 DROP tcp -- any any 10.137.2.19 10.137.255.254 tcp dpt:us-cli
0 0 ACCEPT all -- any any 10.137.2.19 anywhere
8 536 ACCEPT udp -- any any 10.137.2.18 gateway udp dpt:domain
0 0 ACCEPT udp -- any any 10.137.2.18 10.137.1.254 udp dpt:domain
0 0 ACCEPT tcp -- any any 10.137.2.18 gateway tcp dpt:domain
0 0 ACCEPT tcp -- any any 10.137.2.18 10.137.1.254 tcp dpt:domain
0 0 ACCEPT icmp -- any any 10.137.2.18 anywhere
0 0 DROP tcp -- any any 10.137.2.18 10.137.255.254 tcp dpt:us-cli
16 1216 ACCEPT all -- any any 10.137.2.18 anywhere
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
BEFORE - DEBIAN 8 AppVM
user@work-apps:~$ sudo iptables -L -vx
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP udp -- vif+ any anywhere anywhere udp dpt:bootpc
49 3688 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- vif+ any anywhere anywhere
1 52 ACCEPT all -- lo any anywhere anywhere
0 0 REJECT all -- vif+ any anywhere anywhere reject-with icmp-host-prohibited
16 2678 DROP all -- any any anywhere anywhere
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DROP all -- vif+ vif+ anywhere anywhere
0 0 ACCEPT all -- vif+ any anywhere anywhere
0 0 DROP all -- any any anywhere anywhere
Chain OUTPUT (policy ACCEPT 71 packets, 6762 bytes)
pkts bytes target prot opt in out source destination
AFTER - DEBIAN 8 AppVM
user@work-apps:~$ sudo iptables -L -vx
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- any any 10.137.2.19 anywhere
0 0 DROP udp -- vif+ any anywhere anywhere udp dpt:bootpc
165 567437 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- vif+ any anywhere anywhere
1 52 ACCEPT all -- lo any anywhere anywhere
0 0 REJECT all -- vif+ any anywhere anywhere reject-with icmp-host-prohibited
21 3043 DROP all -- any any anywhere anywhere
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- any any 10.137.2.18 10.137.2.19
0 0 DROP all -- vif+ vif+ anywhere anywhere
0 0 ACCEPT all -- vif+ any anywhere anywhere
0 0 DROP all -- any any anywhere anywhere
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
I have only started the relevant VMs to keep it as simple as possible and I have also provided the Kernel IP routing tables in case the virtual interface assignment is related to these rules:
[user@sys-firewall ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.137.1.1 0.0.0.0 UG 0 0 0 eth0
10.137.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
10.137.2.18 0.0.0.0 255.255.255.255 UH 32744 0 0 vif8.0
10.137.2.19 0.0.0.0 255.255.255.255 UH 32746 0 0 vif6.0
I used the above vif numbers for the instructions from Daniel which I did for a previous attempt to fix my inter-VM networking (sysctl -w net/ipv4/conf/vifX.0/proxy_arp=1 <= where vifX.0 are the interfaces to the VMs you want to network) and succeeded in identifying a broadcast mac but this didn't allow successful pinging from Windows 7 to the Debian 8 VM by itself. Was there a further step I was missing?
I have read up on iptables configuration to see if I can understand this further but I can't see what I am doing incorrectly. My firewall rules seem quite encompassing i.e. the input rule 2 allows any protocol and states it will accept data from the IP address of my Windows VM and send this onto the Debian AppVM. I presume that the order of rules should influence the routing and this would not base this off packets of data that match more criteria in the table?
Thanks again and apologies if I am requiring baby steps here.
Max
Nov 9, 2016, 12:36:18 PM11/9/16
to qubes-users, maxtan...@gmail.com, un...@thirdeyesecurity.org
I ran through the steps listed here again: https://www.qubes-os.org/doc/qubes-firewall/#enabling-networking-between-two-vms but instead of trying to ping my Debian 8 VM (10.137.2.18) from the Windows VM (10.137.2.19), I did this from a new Fedora VM (10.137.2.16) to test the results.
I simply did the following:
Firewall
sudo iptables -I FORWARD 2 -s 10.137.2.16 -d 10.137.2.18 -j ACCEPT
work-apps
iptables -I INPUT -s 10.137.2.16 -j ACCEPT
This enabled me to ping from Fedora to the Debian VM. No additional rules were required such as adding ports or adding an ACCEPT FORWARD rule in the Debian VM with the destination and source reversed.
Given the ease of achieving this, it seems that the issue here stopping me pinging my Debian VM from Windows is specific to Windows being an HVM. Pinging from an HVM to a PVM does not seem to work but PVM to PVM networking does. Please note that the HVM can ping the firewall and vice versa.
Does anyone have any suggestions given this information?
Many thanks.
Drew White
Nov 9, 2016, 6:34:06 PM11/9/16
to qubes-users, maxtan...@gmail.com, un...@thirdeyesecurity.org
On Thursday, 10 November 2016 04:36:18 UTC+11, Max wrote:
> Brief update on this. After attempting to use the Qubes Network Server from Manuel Amador (Rudd-O) to solve this issue with no luck I have gone back to looking at solving this by adjusting the iptables rules.
>
> I ran through the steps listed here again: https://www.qubes-os.org/doc/qubes-firewall/#enabling-networking-between-two-vms but instead of trying to ping my Debian 8 VM (10.137.2.18) from the Windows VM (10.137.2.19), I did this from a new Fedora VM (10.137.2.16) to test the results.
>
> I simply did the following:
>
> Firewall
> sudo iptables -I FORWARD 2 -s 10.137.2.16 -d 10.137.2.18 -j ACCEPT
>
> work-apps
> iptables -I INPUT -s 10.137.2.16 -j ACCEPT
>
> This enabled me to ping from Fedora to the Debian VM. No additional rules were required such as adding ports or adding an ACCEPT FORWARD rule in the Debian VM with the destination and source reversed.
>
> Given the ease of achieving this, it seems that the issue here stopping me pinging my Debian VM from Windows is specific to Windows being an HVM. Pinging from an HVM to a PVM does not seem to work but PVM to PVM networking does. Please note that the HVM can ping the firewall and vice versa.
>
> Does anyone have any suggestions given this information?
>
> Many thanks.
> Brief update on this. After attempting to use the Qubes Network Server from Manuel Amador (Rudd-O) to solve this issue with no luck I have gone back to looking at solving this by adjusting the iptables rules.
>
> I ran through the steps listed here again: https://www.qubes-os.org/doc/qubes-firewall/#enabling-networking-between-two-vms but instead of trying to ping my Debian 8 VM (10.137.2.18) from the Windows VM (10.137.2.19), I did this from a new Fedora VM (10.137.2.16) to test the results.
>
> I simply did the following:
>
> Firewall
> sudo iptables -I FORWARD 2 -s 10.137.2.16 -d 10.137.2.18 -j ACCEPT
>
> work-apps
> iptables -I INPUT -s 10.137.2.16 -j ACCEPT
>
> This enabled me to ping from Fedora to the Debian VM. No additional rules were required such as adding ports or adding an ACCEPT FORWARD rule in the Debian VM with the destination and source reversed.
>
> Given the ease of achieving this, it seems that the issue here stopping me pinging my Debian VM from Windows is specific to Windows being an HVM. Pinging from an HVM to a PVM does not seem to work but PVM to PVM networking does. Please note that the HVM can ping the firewall and vice versa.
>
> Does anyone have any suggestions given this information?
>
> Many thanks.
As I have said in other places, including his qubes network server post, I too use IPTables, because it's much simpler and cleaner.
I have a dedicated ProxyVM that is my inter-vm network.
These are the 2 rules...
$intervm_internalnet = '10.137.2.0';// this can be generated from the ifconfig if required. But conditions apply for success.
iptables -I FORWARD 1 -i vif+ -o vif+ -s $intervm_internalnet/24 -d $intervm_internalnet/24 -m state --state NEW -p tcp -m tcp -j ACCEPT
iptables -I FORWARD 1 -i vif+ -o vif+ -s $intervm_internalnet/24 -d $intervm_internalnet/24 -p udp -m udp -j ACCEPT
This has worked for me always. Never missed a beat. And it allows for inter-vm comms, as well as it communicating to the outside world.
Max
Nov 10, 2016, 12:09:30 PM11/10/16
to qubes-users, maxtan...@gmail.com, un...@thirdeyesecurity.org
Thanks Drew, unfortunately I tried this at the beginning (my step 3). It didn't work for me.
Have you tried pinging from a Windows HVM to another Debian or Fedora AppVM?
dba...@gmail.com
Feb 6, 2018, 8:21:31 AM2/6/18
to qubes-users
I am a newbie on Qubes, and i've the same issue on 3.2 version.
Did you finally succeeded in having interconnect between two HVM ?
Thanks for your feedback.
Regards
Mc
Alex Dubois
Feb 6, 2018, 9:49:31 AM2/6/18
to qubes-users
The documentation on how to open networking between 2 qubes is misleading as it probably open much more than required and incomplete.
Could you please specify what you want to do between these 2 VM (which port you want to open)? as I suppose you want more than pinging...
Max
Feb 6, 2018, 10:28:28 AM2/6/18
to qubes-users
Yes, I wanted to do a little more than pinging. For this particular issue, I wanted to be able to query a database connection between the two VMs.
Max
Feb 6, 2018, 10:30:16 AM2/6/18
to qubes-users
I was able to connect between Linux AppVMs only, not HVMs.
To solve my particular issue, I went with syncthing to transfer a text file between VMs which was very straightforward as the Windows and Linux clients are very easy to install.
Thanks
Reply all
Reply to author
Forward
0 new messages