Problem connecting via VPN ProxyVM (VPN works, but AppVM can't connect)

[PhR]

Hello,

I have successfully setup a fedora 25 bases ProxyVM, which has Cisco's
Anyconnect Secure Mobility Client installed.

I can successfully connect via VPN and can also ping/reach servers via VPN.

Unfortunately the App-VM which uses the VPN Proxy VM can't connect.

The Setup:

sys-net <-- sys-firewall <-- my-vpn (Proxy VM) <-- my-work (App VM)

As I can connect from the Proxy my-vpn VM, it seems the problem is
between the connection of my App-VM to the new Proxy VPN VM.

How can I troubleshoot and investigate the issues?

- PhR

[Chris Laprise]

You could ping a known IP address from the appVM. If it works the
problem is likely limited to DNS.

In the proxyVM, check the contents of /etc/resolv.conf after your Cisco
client connects. If its updated (not a 10.137.x.x number) you can run
/usr/lib/qubes/qubes-setup-dnat-to-ns to enable DNS forwarding over the VPN.

Another setting to check is /proc/sys/net/ipv4/ip_forward which should
contain a value of '1'. Also, the iptables 'POSTROUTING' chain should
have a masquerade target:

$ cat /proc/sys/net/ipv4/ip_forward
$ sudo iptables -L -t nat

-

Chris Laprise, tas...@posteo.net
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[PhR]

Hello Chris

On 08/21/2017 06:28 PM, Chris Laprise wrote:
> On 08/20/2017 05:38 PM, 'PhR' via qubes-users wrote:
>> Unfortunately the App-VM which uses the VPN Proxy VM can't connect.
>> The Setup:
>> sys-net <-- sys-firewall <-- my-vpn (Proxy VM) <-- my-work (App VM)

>> (...)

>
> You could ping a known IP address from the appVM. If it works the
> problem is likely limited to DNS.
>

Pinging a VPN-Adress from within my Proxy VPN (work-vpn) after
connecting via anyConnect VPN works.
But pinging from my work-AppVM doesn't work.

> In the proxyVM, check the contents of /etc/resolv.conf after your
> Cisco client connects. If its updated (not a 10.137.x.x number) you
> can run /usr/lib/qubes/qubes-setup-dnat-to-ns to enable DNS forwarding
> over the VPN.

Ihave checked /etc/resolv.conf:

[user@my-work-vpn ~]$ cat /etc/resolv.conf
domain intern.MYCOMPANY.de
nameserver 192.168.1.6
nameserver 192.168.1.11
nameserver 10.137.2.1
nameserver 10.137.2.254
search intern.MYCOMPANY.de

> Another setting to check is /proc/sys/net/ipv4/ip_forward which should
> contain a value of '1'. Also, the iptables 'POSTROUTING' chain should
> have a masquerade target:
>
> $ cat /proc/sys/net/ipv4/ip_forward

It is enabled (content: 1)

> $ sudo iptables -L -t nat

[user@my-work-vpn ~]$ sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
PR-QBS all -- anywhere anywhere
PR-QBS-SERVICES all -- anywhere anywhere

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
MASQUERADE all -- anywhere anywhere

Chain PR-QBS (1 references)
target prot opt source destination
DNAT udp -- anywhere 10.137.5.1 udp
dpt:domain to:10.137.2.1
DNAT tcp -- anywhere 10.137.5.1 tcp
dpt:domain to:10.137.2.1
DNAT udp -- anywhere 10.137.5.254 udp
dpt:domain to:10.137.2.254
DNAT tcp -- anywhere 10.137.5.254 tcp
dpt:domain to:10.137.2.254

Chain PR-QBS-SERVICES (1 references)
target prot opt source destination

Do I need to tweak any other rules or setting in the ProxyVM or AppVM?
As the ProxyVM can perfectly connect to corporate servers, VPN is working.

If I switch the Net-VM in my work AppVM to the normal sys-firewall I can
connect to the internet.
As such it seems that both proxyVM and AppVM seem to work normaly but
not if I put everything together.

Any more ideas?

- PhR

[Chris Laprise]

On 08/21/2017 05:19 PM, PhR wrote:
>
> Any more ideas?
>
> - PhR
>

Some more questions:

Is this Qubes 3.2?

What changes does the Cisco client make to the routing table ('route'
command)?

What changes (if any) to 'FORWARD' chain ('iptables -L')?

Does running '/usr/lib/qubes/qubes-setup-dnat-to-ns' update the PR-QBS
chain ('iptables -L -t nat)? Does that allow appVM to communicate?

What firewall rules are in the appVM's settings (Qubes Manager)? For
testing (and probably for use) it should be set to "Allow network access
except" and also allow DNS and ICMP with a blank list below.

Is the appVM based on a regular Linux template such as fedora-25 or
debian-8?

Further:

The 'vpnc' package may be a viable alternative to Anyconnect (the open
source counterpart is 'openconnect'). Also, Network Manager has an
openconnect plugin; you would need to install the plugin in the template
then enable NM for the proxyVM.

If you request help from the Cisco community, you can describe the
proxyVM as being like an external router, but my limited searching
suggests Cisco doesn't support this type of configuration.

Another option: Simply run the Anyconnect client in the appVM (no
proxyVM for the VPN client). This may be the simplest route.

--

[PhR]

Hello Chris,


On 08/22/2017 12:55 AM, Chris Laprise wrote:

> Is this Qubes 3.2?
Yes.

> What changes does the Cisco client make to the routing table ('route'
> command)?

Before starting AnyConnect:

[user@my-work-vpn ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.137.2.1 0.0.0.0 UG 0 0 0 eth0
10.137.2.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0

After starting AnyConnect:
[user@my-work-vpn ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.137.2.1 0.0.0.0 UG 0 0 0 eth0
10.5.48.0 0.0.0.0 255.255.255.0 U 0 0 0 cscotun0
10.137.2.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 cscotun0
vsrv-dc-3.xxxx 0.0.0.0 255.255.255.255 UH 0 0 0 cscotun0
vsrv-dc-2.xxxx 0.0.0.0 255.255.255.255 UH 0 0 0 cscotun0
213.xxx.xxx.xxx 10.137.2.1 255.255.255.255 UGH 0 0 0 eth0

> What changes (if any) to 'FORWARD' chain ('iptables -L')?

Before starting AnyConnect:

[user@my-work-vpn ~]$ sudo iptables -L

Chain INPUT (policy DROP)

target prot opt source destination

DROP udp -- anywhere anywhere udp dpt:bootpc
ACCEPT all -- anywhere anywhere ctstate
RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere

ACCEPT all -- anywhere anywhere

REJECT all -- anywhere anywhere reject-with
icmp-host-prohibited

Chain FORWARD (policy DROP)

target prot opt source destination

ACCEPT all -- anywhere anywhere ctstate
RELATED,ESTABLISHED

ACCEPT all -- anywhere anywhere

DROP all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

After starting AnyConnect:

[user@my-work-vpn ~]$ sudo iptables -L

Chain INPUT (policy DROP)

target prot opt source destination

ciscovpn all -- anywhere anywhere
ciscovpnfw all -- anywhere anywhere
DROP udp -- anywhere anywhere udp dpt:bootpc
ACCEPT all -- anywhere anywhere ctstate
RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere

ACCEPT all -- anywhere anywhere

REJECT all -- anywhere anywhere reject-with
icmp-host-prohibited

Chain FORWARD (policy DROP)

target prot opt source destination

ciscovpn all -- anywhere anywhere
ciscovpnfw all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate
RELATED,ESTABLISHED

ACCEPT all -- anywhere anywhere

DROP all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

ciscovpn all -- anywhere anywhere
ciscovpnfw all -- anywhere anywhere

Chain ciscovpn (3 references)

target prot opt source destination

ACCEPT all -- anywhere anywhere state
RELATED,ESTABLISHED

ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere

ACCEPT udp -- anywhere anywhere udp
spt:bootpc dpt:bootps
ACCEPT udp -- anywhere anywhere udp
spt:bootps dpt:bootpc
ACCEPT udp -- anywhere anywhere udp
spt:dhcpv6-client dpt:dhcpv6-server
ACCEPT udp -- anywhere anywhere udp
spt:dhcpv6-server dpt:dhcpv6-client
ACCEPT tcp -- 10.137.2.26 213.xxx.xxx.xxx tcp dpt:https
ACCEPT tcp -- 213.xxx.xxx.xxx 10.137.2.26 tcp spt:https
ACCEPT udp -- 10.137.2.26 213.xxx.xxx.xxx udp dpt:https
ACCEPT udp -- 213.xxx.xxx.xxx 10.137.2.26 udp spt:https
RETURN all -- 10.137.2.26 anywhere
RETURN all -- anywhere 10.137.2.26
RETURN all -- 10.137.2.26 10.137.2.26
RETURN all -- 10.137.2.26 10.137.2.26
RETURN udp -- 10.137.2.26 224.0.0.251 udp dpt:mdns
RETURN udp -- 10.137.2.26 after launching it I can
224.0.0.251 udp dpt:mdns
RETURN udp -- 10.137.2.26 239.255.255.250 udp dpt:ssdp
RETURN udp -- 10.137.2.26 239.255.255.250 udp dpt:ssdp
RETURN all -- anywhere base-address.mcast.net/4
RETURN all -- 10.137.2.26 base-address.mcast.net/4
RETURN all -- anywhere 255.255.255.255
RETURN all -- 10.137.2.26 255.255.255.255
RETURN all -- 172.21.2.13 aaaaa.de/24
RETURN all -- isys-team.de/24 172.21.2.13
RETURN all -- 172.21.2.13 192.168.3.0/24
RETURN all -- 192.168.3.0/24 172.21.2.13
RETURN all -- 172.21.2.13 10.5.48.0/24
RETURN all -- 10.5.48.0/24 172.21.2.13
RETURN all -- 172.21.2.13 192.168.5.0/24
RETURN all -- 192.168.5.0/24 172.21.2.13
RETURN all -- 172.21.2.13 192.168.100.0/24
RETURN all -- 192.168.100.0/24 172.21.2.13
RETURN all -- 172.21.2.13 vsrv-dc-3.xxx.yyy.de
RETURN all -- vsrv-dc-3.xxx.yyy.de 172.21.2.13
RETURN all -- 172.21.2.13 vsrv-dc-2.xxx.yyy.de
RETURN all -- vsrv-dc-2.xxx.yyy.de 172.21.2.13
RETURN udp -- 172.21.2.13 anywhere udp dpt:domain
RETURN udp -- anywhere 172.21.2.13 udp spt:domain
RETURN all -- anywhere 255.255.255.255
RETURN all -- 172.21.2.13 255.255.255.255
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere

Chain ciscovpnfw (3 references)

target prot opt source destination

> Does running '/usr/lib/qubes/qubes-setup-dnat-to-ns' update the PR-QBS
> chain ('iptables -L -t nat)? Does that allow appVM to communicate?
>
> What firewall rules are in the appVM's settings (Qubes Manager)? For
> testing (and probably for use) it should be set to "Allow network
> access except" and also allow DNS and ICMP with a blank list below.
>
> Is the appVM based on a regular Linux template such as fedora-25 or
> debian-8?

Both VMs are based on a Qubes 3.2 Templates:
VPN Proxy: Fedora 25
AppVM: Debian 8
(I have also tried to use a Fedora 25 AppVM, same problem)
No connection via Proxy

> Further:
> The 'vpnc' package may be a viable alternative to Anyconnect (the open
> source counterpart is 'openconnect'). Also, Network Manager has an
> openconnect plugin; you would need to install the plugin in the
> template then enable NM for the proxyVM.

I have already tried to use the openconnect plugin for network manager,
but when I click on Add in the network manager and choose VPN and then
"Cisco AnyConnect Compatible VPN (openconnect)" I get a new windows but
can't add any information here as every field looks disabled :-/ ?
Working with OpenConnect would be great.

> Another option: Simply run the Anyconnect client in the appVM (no
> proxyVM for the VPN client). This may be the simplest route.

Yes, but I'd like to connect two VMs (one Windows HVM and one Linux AppVM).
I also thought that is Qubes Best practise to use a dedicated VPN Proxy
VM vs. launching VPN from within an AppVM ?

regards

- PhR

[PhR]

Hello,

On 08/22/2017 12:55 AM, Chris Laprise wrote:

> Some more questions:
> [...]

some more information:

Strangely I can connect via OpenConnect from the command line/CLI:

root@my-work:~# openconnect -u MYUSERNAME VPNLINK.com
POST https://xxxx/
Attempting to connect to server 213.xxx.xxx.xxx:443
SSL negotiation with xxxx
Connected to HTTPS on xxxx
XML POST enabled
Please enter your username and password.
GROUP: [MYCOMPANY]:MYUSERNAME

POST https://xxxx/
XML POST enabled
Please enter your username and password.
Password:
POST https://xxxx/
Got CONNECT response: HTTP/1.1 200 OK
CSTP connected. DPD 30, Keepalive 20
Connected tun0 as 172.21.2.13, using SSL
Established DTLS connection (using GnuTLS). Ciphersuite AES256-SHA.

I can then connect to my corporate network.
As such it seems that the problem of greyed out fields in the VPN-Setup
of Network-Manager is not a OpenConnect issue, but more a Network
Manager problem.

- PhR

[Chris Laprise]

First, I surmise that 10.137.2.26 is the proxyVM address, not appVM.

Its hard to see the problem clearly because of the complexity of what
its adding, but I think your appVM packets are probably hitting the 2
DROP targets at the end of 'ciscovpn' chain. This may be because the
chain explicitly allows packets from 'source' address of your VPN
proxyVM, but there is nothing explicitly in that chain allowing 'source'
addresses for downstream appVMs. This is a problem when 'ciscovpn' chain
is called from the FORWARD chain.

As a test remedy you could try the following in the proxyVM (_after_
your appVM is already running):

$ sudo iptables -I FORWARD -i vif+ -j ACCEPT

...then try connections from the appVM.

This will cause anything coming from downstream VMs to be accepted for
forwarding (which is OK if you don't intend to use Qubes firewall
restrictions in appVM Settings). A full long-term solution would involve
reconciling the cisco firewall commands with the Qubes firewall service
(would be ideal if Anyconnect used a firewall script and it could be
located in the system, then you could have the right commands issued
from /rw/config/qubes-firewall-user-script).

>
> I have already tried to use the openconnect plugin for network
> manager, but when I click on Add in the network manager and choose VPN
> and then "Cisco AnyConnect Compatible VPN (openconnect)" I get a new
> windows but can't add any information here as every field looks
> disabled :-/ ?
> Working with OpenConnect would be great.

There is also a GUI part that needs to be installed:
NetworkManager-openconnect-gnome in Fedora.

> I also thought that is Qubes Best practise to use a dedicated VPN
> Proxy VM vs. launching VPN from within an AppVM ?
>

Should be OK for a work-only VPN where your appVM is also work-only; Its
a different threat model than an Internet-focused VPN.

-

Finally, I should mention leak prevention measures. If you are able to
get the VPN to function with proxyVM + appVMs, you can then add these
commands in proxyVM to prevent appVMs from having non-VPN access:

iptables -I FORWARD -o eth0 -j DROP
iptables -I FORWARD -i eth0 -j DROP

These need to show up at the _top_ of the FORWARD chain, which is why
'-I' insert is used; You can ensure they'll be at the top by executing
them last after a connection is made (probably from
/rw/config/qubes-firewall-user-script).

[PhR]

Hello Chris,

On 08/22/2017 05:37 PM, Chris Laprise wrote:
>> Working with OpenConnect would be great.
>
> There is also a GUI part that needs to be installed:
> NetworkManager-openconnect-gnome in Fedora.

I tried all hints you have given, but nothing seems to work.
At least I was able to get a fedora-25 based proxy VM up and running and
my work AppVm could connect through the proxy (without any VPN involved)

I've decided to try to setup AnyConnect from within my Work AppVM and
use openconnect-gnome to connect to our Cisco ASA.

I have therof created a new template based on a fedora 25 clone and made
sure that NetworkManager-openconnect-gnome is installed in the template.

But if I start the AppVM and start Network Manager I can open the Create
new VPN window but all options are still greyed out - can someone
reproduce this problem on Qubes 3.2

- Launch Network Connections from the App Menu
- Right Click > Edit Connections
- Add
- Connection Type = VPN
- Cisco AnyConnect Compatible VPN (openconnect)
- Create
- all options are greyed out in the next Screen

Any options how to make this work?

> Finally, I should mention leak prevention measures. If you are able to
> get the VPN to function with proxyVM + appVMs, you can then add these
> commands in proxyVM to prevent appVMs from having non-VPN access:
>
> iptables -I FORWARD -o eth0 -j DROP
> iptables -I FORWARD -i eth0 -j DROP
>
> These need to show up at the _top_ of the FORWARD chain, which is why
> '-I' insert is used; You can ensure they'll be at the top by executing
> them last after a connection is made (probably from
> /rw/config/qubes-firewall-user-script).

I'll try to get VPN up and running first, then I can harden it.

- PhR