XL VM connectivity to Qubes Network

[3mp...@gmail.com]

Hi,

I’m still working on my Qubes 4.0 migration and I’m trying to connect a Windows 10 XL created VM to Qubes Networking by adding this kind of line in the hvm file :

vif = [ 'mac=00:16:3e:5e:6d:11,model=e1000,
ip=10.137.0.50,script=vif-route-qubes,backend=sys-vpn’ ]

The backend is a functional proxy VM created by following the « Set up a ProxyVM as a VPN gateway using iptables and CLI scripts » tutorial

The guest VM is a Windows 10 1804 with Xen paravirtual drivers 8.2.1 installed. The e1000 is correctly detected as a Xen adapter with link up and configured with IP : 10.137.0.50 / 24, no gateway, I’m just trying to access some HTTP services in another qube connected to the same proxyVM. Firewall rules are OK because it’s working from a third qube connected to the proxyVM.

The Windows 10 VM is acting like it has no connectivy at all. Can’t connect to the HTTP server of my qube. I tried removing all firewall rules in proxyVM and qube HTTP server and setting ACCEPT policy for all chains, it doesn’t change anything so I think it’s not a firewall problem…

I’d be grateful if someone gives me a clue where the problem could be :(

[unman]

Some obvious questions.

You say the interface is correctly configured.
Do you have any routes set in the Windows box?
Do you see traffic outbound on the 10.137.0.50 iface?

If you sniff traffic inbound on the vif attached to the Windows HVM, do
you see anything there? (I mean sniff on the proxyVM)

[3mp...@gmail.com]

> Some obvious questions.
>
> You say the interface is correctly configured.
> Do you have any routes set in the Windows box?
> Do you see traffic outbound on the 10.137.0.50 iface?
>
> If you sniff traffic inbound on the vif attached to the Windows HVM, do
> you see anything there? (I mean sniff on the proxyVM)

Hi Unman, thanks for your help.

To eliminate some potential Windows issues I chose to boot on a fedora 28 live cd in this HVM.

I configured the eth0 interface with the following command to copy a normal qubes configuration :

ifconfig eth0 10.137.0.200 (HVM IP) netmask 255.255.255.255 broadcast 10.255.255.255
route add -host 10.137.0.10 (ProxyVM IP) dev eth0
route add default gw 10.137.0.10 eth0

I identified vif14 on ProxyVM, it corresponds to the HVM interace.

I launched tcpdump -n -i vif14.0 on ProxyVM, telnet from 10.137.0.200 (HVM) to 10.137.0.8 port 8080 (web server on qube I'm trying to reach, working great from a third qube)

telnet doesn't connect but here's the result of tcpdump :

https://pastebin.com/QXhyBx4Z

Any help appreciated

[3mp...@gmail.com]

Could someone help me on this issue ? Or explain the right network configuration in Qubes 4.0 ? The 255.255.255.255 netmask seems strange to me... Is the idea to block all kind of traffic except to the gateway with the route add -host commmand ?

[unman]

I believe that is indeed the aim.
You can either set to 255.255.255.0 or add specific route, as you have
done. (Did you set a return route on the destination also?)

The next step would be to examine the rules on the proxy to make sure
that you are allowing the traffic through the ProxyVM. You could listen
on the interface that's attached to 10.137.0.8 to see if traffic is
outbound from there.

unman

[3mp...@gmail.com]

> I believe that is indeed the aim.
> You can either set to 255.255.255.0 or add specific route, as you have
> done. (Did you set a return route on the destination also?)

The return route is automatically added to the Proxy with the vif-route-qubes, and the destination send all traffic to the proxy. The vif-route-qubes seems to work perfectly, I noticed no difference between a Qube and a HVM using the script...

> The next step would be to examine the rules on the proxy to make sure
> that you are allowing the traffic through the ProxyVM. You could listen
> on the interface that's attached to 10.137.0.8 to see if traffic is
> outbound from there.
>
> unman

I removed all firewall rules (from Proxy and Qube Server), here's the conf :

# iptables -L

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain QBS-FORWARD (0 references)
target prot opt source destination

# iptables -t raw --list

Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Always the same result, Qube client connects to Qube server perfectly, HVM doesn't connect to Qube server.

I captured traffic this afternoon and I think it's an ARP problem... Here's a working ARP session when I connect from Qube client to Qube Server :

QUBE CLIENT ARP (from its specific vif) :

18:32:09.483319 ARP, Request who-has 10.137.0.12 tell 10.137.0.10, length 28
18:32:09.483502 ARP, Reply 10.137.0.12 is-at 00:16:3e:5e:6c:00, length 28
18:32:09.746948 ARP, Request who-has 10.137.0.10 tell 10.137.0.12, length 28
18:32:09.746970 ARP, Reply 10.137.0.10 is-at fe:ff:ff:ff:ff:ff, length 28

QUBE SERVER ARP (from its specific vif) :

18:32:09.483343 ARP, Request who-has 10.137.0.8 tell 10.137.0.10, length 28
18:32:09.483536 ARP, Reply 10.137.0.8 is-at 00:16:3e:5e:6c:00, length 28
18:32:09.542795 ARP, Request who-has 10.137.0.10 tell 10.137.0.8, length 28
18:32:09.542817 ARP, Reply 10.137.0.10 is-at fe:ff:ff:ff:ff:ff, length 28

--

and here is the ARP session from HVM to Qube Server :

HVM ARP :

18:33:34.793593 ARP, Request who-has 10.137.0.10 tell 10.137.0.200, length 28
18:33:34.793631 ARP, Reply 10.137.0.10 is-at fe:ff:ff:ff:ff:ff, length 28

18:34:06.537570 ARP, Request who-has 10.137.0.10 tell 10.137.0.200, length 28
18:34:06.537609 ARP, Reply 10.137.0.10 is-at fe:ff:ff:ff:ff:ff, length 28

18:34:38.793679 ARP, Request who-has 10.137.0.10 tell 10.137.0.200, length 28
18:34:38.793699 ARP, Reply 10.137.0.10 is-at fe:ff:ff:ff:ff:ff, length 28

QUBE SERVER ARP :

Nothing :(

--

It's like the Proxy does not forward traffic to the destination vif, but there's no firewall rule blocking it, so I'm out of ideas...

Other observations : ping is working between HVM and Proxy. No luck between HVM and Qube Server.

I saw that every Qube had the same MAC address (I don't understand how it's possible, I'd really appreciate a document explaining how Qubes Networking work...) so I tried to set this same MAC address to the HVM, no more luck.

Any help appreciated ! I'm feeling desperate right now :(