VPN qubes preventing some websites from loading properly

[Christophe Pfeifer]

Hi,

I followed the tutorial "Set up a ProxyVM as a VPN gateway using iptables and CLI scripts" [1], then I subscribed to NordVPN and configured OpenVPN over UDP (since my ISP blocks OpenVPN over TCP).

---------------

My final architecture is the following:

AppVM ----> VPN (ProxyVM) ----> Firewall VM ----> Network VM

---------------

Firewall VM rules: Deny all but:

Address   |   Service   |   Protocol

*               | OpenVPN | UDP

*               | OpenVPN | TCP

*               | HTTPS     | TCP

---------------

Problem: this is working for mostly all websites I use, except some ones, like Protonmail, Facebook, etc. These latter sites are either showing first some contents just after logging in, or the logging in is impossible, and then loading endlessly.

It seems like a "Keep-alive connection issue".

---------------

Investigation:

1) I allowed full access on the firewall for 5 minutes

2) I launched Wireshark on the VPN VM

3) I tried to log in to Protonmail

Results: (excerpt)

- 10.137.0.14 -> 82.221.139.122 OpenVPN 110 MessageType: P_DATA_V2

- 192.168.43.1 -> 10.137.0.14 ICMP 592 Destination unreachable (Fragmentation needed)

- 185.70.40.151 -> 10.8.8.20 TCP 68 [TCP Dup ACK 711#1] 443 → 42938 [ACK] Seq=69096 Ack=1868 Win=66 Len=0 SLE=3193 SRE=3194

- 10.137.0.9 -> 185.70.40.151 TCP 1381 [TCP Retransmission] 42938 → 443 [ACK] Seq=1868 Ack=69096 Win=3261 Len=1325

[...]

- 10.137.0.9 -> 185.70.40.151 TCP 56 [TCP Keep-Alive] 42954 → 443 [ACK] Seq=977 Ack=1262 Win=32640 Len=0

---------------

Do you know any solution to prevent this from happening? Maybe a configuration trick of OpenVPN or of the VPN VM ?

[1] https://www.qubes-os.org/doc/vpn/

Thanks,

Christophe

[Chris Laprise]

Did you download the openvpn config from NordVPN or write it yourself?
Its preferable to download it. I see that NordVPN's config includes
'ping' and 'ping-restart' which is similar to using the 'keepalive' option.

The issue with only certain sites not working could indicate that a
third-party service like a CDN has blocked the IP addresses that your
VPN provider is using. I've also seen some services block VPN IPs on
certain servers but not others. I see this occasionally when connecting
through Private Internet Access. The solution rests with the VPN
operators to block abusive network patterns and switch to IPs that
haven't been blacklisted... its basically a VPN reputation thing.

BTW, you might find Qubes-vpn-support project better to use overall for
VPNs. You can control it as a system service and it uses connection
parameters that keep openvpn operating more smoothly (although for this
particular problem I don't think it would have an effect)...

https://github.com/tasket/Qubes-vpn-support

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Christophe Pfeifer]

On Sunday, November 25, 2018 7:43 PM, Chris Laprise <tas...@posteo.net> wrote:
> On 11/25/2018 10:32 AM, 'Christophe Pfeifer' via qubes-users wrote:
>
> > Hi,
> > I followed the tutorial "Set up a ProxyVM as a VPN gateway using
> > iptables and CLI scripts" [1], then I subscribed to NordVPN and
> > configured OpenVPN over UDP (since my ISP blocks OpenVPN over TCP).
> >

> > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

> >
> > My final architecture is the following:
> > AppVM ----> VPN (ProxyVM) ----> Firewall VM ----> Network VM
> >

> > -----------------------------------------------------------------------------------------------------

> >
> > Firewall VM rules: Deny all but:
> > Address   |   Service   |   Protocol
> > *               | OpenVPN | UDP
> > *               | OpenVPN | TCP
> > *               | HTTPS     | TCP
> >

> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------

> >
> > Problem: this is working for mostly all websites I use, except some
> > ones, like Protonmail, Facebook, etc. These latter sites are either
> > showing first some contents just after logging in, or the logging in is
> > impossible, and then loading endlessly.
> > It seems like a "Keep-alive connection issue".
> >

> > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > Investigation:
> >
> > 1. I allowed full access on the firewall for 5 minutes
> > 2. I launched Wireshark on the VPN VM
> > 3. I tried to log in to Protonmail

> > Results: (excerpt)
> >
> >
> > - 10.137.0.14 -> 82.221.139.122 OpenVPN 110 MessageType: P_DATA_V2
> > - 192.168.43.1 -> 10.137.0.14 ICMP 592 Destination unreachable
> > (Fragmentation needed)
> >
> > - 185.70.40.151 -> 10.8.8.20 TCP 68 [TCP Dup ACK 711#1] 443 → 42938
> > [ACK] Seq=69096 Ack=1868 Win=66 Len=0 SLE=3193 SRE=3194
> >
> > - 10.137.0.9 -> 185.70.40.151 TCP 1381 [TCP Retransmission] 42938 → 443
> > [ACK] Seq=1868 Ack=69096 Win=3261 Len=1325
> > [...]
> >
> > - 10.137.0.9 -> 185.70.40.151 TCP 56 [TCP Keep-Alive] 42954 → 443 [ACK]
> > Seq=977 Ack=1262 Win=32640 Len=0
> >
> >

> > Do you know any solution to prevent this from happening? Maybe a
> > configuration trick of OpenVPN or of the VPN VM ?
>
> Did you download the openvpn config from NordVPN or write it yourself?
> Its preferable to download it. I see that NordVPN's config includes
> 'ping' and 'ping-restart' which is similar to using the 'keepalive' option.
>
> The issue with only certain sites not working could indicate that a
> third-party service like a CDN has blocked the IP addresses that your
> VPN provider is using. I've also seen some services block VPN IPs on
> certain servers but not others. I see this occasionally when connecting
> through Private Internet Access. The solution rests with the VPN
> operators to block abusive network patterns and switch to IPs that
> haven't been blacklisted... its basically a VPN reputation thing.
>
> BTW, you might find Qubes-vpn-support project better to use overall for
> VPNs. You can control it as a system service and it uses connection
> parameters that keep openvpn operating more smoothly (although for this
> particular problem I don't think it would have an effect)...
>
> https://github.com/tasket/Qubes-vpn-support
>

> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Chris Laprise,tas...@posteo.net

> https://github.com/tasket
> https://twitter.com/ttaskett
> PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

I am using NordVPN's config files and now (thanks to your advice) Qubes-vpn-support.

I have compared this with NordVPN's autoconfig script [1] and with another ISP.
Here are some results (for the very same remote VPN server):
-With another ISP
----OpenVPN over UDP
--------Qubes-vpn-support: OK
--------NordVPN script: OK
----OpenVPN over TCP
--------Qubes-vpn-support: OK
--------NordVPN script: OK
-With my ISP over 4G network
----OpenVPN over UDP
--------Qubes-vpn-support: some websites are stalled
--------NordVPN script: OK
----OpenVPN over TCP
--------Qubes-vpn-support: ALL websites are stalled
--------NordVPN script: OK

Therefore, it is surprising that NordVPN's script works but not Qubes-vpn-support with NordVPN's config files!

I compared the Wireshark captures in both cases, when I try to log in to Protonmail, and here is something interesting.
* For NordVPN's script (at a certain moment in time):
- 192.168.43.1 -> 10.137.0.16 ICMP 592 Destination unreachable (Fragmentation needed)
- 10.137.0.16 -> 82.221.114.154 IPv4 1412 Fragmented IP protocol (proto=UDP 17, off=0, ID=f7e7) [Reassembled in #3066]

* For Qubes-vpn-support (at a certain moment in time):
- 192.168.43.1 -> 10.137.0.20 ICMP 592 Destination unreachable (Fragmentation needed)
- 192.168.43.1 -> 10.137.0.20 ICMP 592 Destination unreachable (Fragmentation needed)
[...]
- 192.168.43.1 -> 10.137.0.20 ICMP 592 Destination unreachable (Fragmentation needed)

Thus, I am convinced that it is a matter of fragmentation, which seems mandatory over my 4G network provider, but is not managed by Qubes-vpn-support and NordVPN's config file.
I tried to add a random value "fragmentation 1000" in the ovpn config file, but that just stalled all the traffic...
Does anyone know what i could append in the config file, in order to enable packet fragmentation (for both UDP/TCP)?

NordVPN'script is no good choice for me since it needs to be run in each AppVM. I really would like to have a proper separated VPN VM.

[1] https://nordvpn.com/download/linux/

Christophe

[Chris Laprise]

This appears to be a .deb with binary executables; I'm not seeing a
script so the decisions they're making aren't readily apparent.

Since their config is trying to force certain packet size parameters --
but the config is apparently not the preferred way to connect on Linux
-- you might assume these parameters are not as flexible as needed to
cope with fragmentation issues posed by your 4G provider. You might even
consider the config to be buggy.

I would try commenting-out the following lines from your config to see
if openvpn can handle the fragmentation on its own (this may actually be
what's happening when openvpn is started by the nordvpnd binary):

tun-mtu 1500
tun-mtu-extra 32
mssfix 1450

With Qubes-vpn-support, after changing the conf in /rw/config/vpn you'll
need to restart the VM (or copy the modified conf to /tmp before
restarting the service).

Another thing to try is start the connection with nordvpn's program and
then look at the log (or use 'ps aux | grep openvpn') to find the config
file its generating for openvpn. This should give you the specific
options that allow operation over your 4G ISP.

--

Chris Laprise, tas...@posteo.net

[Christophe Pfeifer]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Sunday, November 25, 2018 11:59 PM, Chris Laprise <tas...@posteo.net> wrote:

> On 11/25/2018 04:47 PM, Christophe Pfeifer wrote:
>
> > On Sunday, November 25, 2018 7:43 PM, Chris Laprise tas...@posteo.net wrote:
> >
> > > On 11/25/2018 10:32 AM, 'Christophe Pfeifer' via qubes-users wrote:
> > >
> > > > Hi,
> > > > I followed the tutorial "Set up a ProxyVM as a VPN gateway using
> > > > iptables and CLI scripts" [1], then I subscribed to NordVPN and
> > > > configured OpenVPN over UDP (since my ISP blocks OpenVPN over TCP).
> > > >

> > > > My final architecture is the following:
> > > > AppVM ----> VPN (ProxyVM) ----> Firewall VM ----> Network VM
> > > >

> > > > Firewall VM rules: Deny all but:
> > > > Address   |   Service   |   Protocol
> > > > *               | OpenVPN | UDP
> > > > *               | OpenVPN | TCP
> > > > *               | HTTPS     | TCP
> > > >

> > > > Problem: this is working for mostly all websites I use, except some
> > > > ones, like Protonmail, Facebook, etc. These latter sites are either
> > > > showing first some contents just after logging in, or the logging in is
> > > > impossible, and then loading endlessly.
> > > > It seems like a "Keep-alive connection issue".
> > > >

> > > Chris Laprise,tas...@posteo.net
> > > https://github.com/tasket
> > > https://twitter.com/ttaskett
> > > PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886
> >
> > I am using NordVPN's config files and now (thanks to your advice) Qubes-vpn-support.
> > I have compared this with NordVPN's autoconfig script [1] and with another ISP.
> > Here are some results (for the very same remote VPN server):
> > -With another ISP
> > ----OpenVPN over UDP
> > --------Qubes-vpn-support: OK
> > --------NordVPN script: OK
> > ----OpenVPN over TCP
> > --------Qubes-vpn-support: OK
> > --------NordVPN script: OK
> > -With my ISP over 4G network
> > ----OpenVPN over UDP
> > --------Qubes-vpn-support: some websites are stalled
> > --------NordVPN script: OK
> > ----OpenVPN over TCP
> > --------Qubes-vpn-support: ALL websites are stalled
> > --------NordVPN script: OK
> > Therefore, it is surprising that NordVPN's script works but not Qubes-vpn-support with NordVPN's config files!
> > I compared the Wireshark captures in both cases, when I try to log in to Protonmail, and here is something interesting.
> >

> > - For NordVPN's script (at a certain moment in time):

> >
> > - 192.168.43.1 -> 10.137.0.16 ICMP 592 Destination unreachable (Fragmentation needed)
> > - 10.137.0.16 -> 82.221.114.154 IPv4 1412 Fragmented IP protocol (proto=UDP 17, off=0, ID=f7e7) [Reassembled in #3066]
> >

> > - For Qubes-vpn-support (at a certain moment in time):

> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

>
> Chris Laprise,tas...@posteo.net
> https://github.com/tasket
> https://twitter.com/ttaskett
> PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

Yes, you are right, it is a binary.
I commented out these lines to no avail.

Then, I tried to locate the openvpn config file used by NordVPN's binary. Unfortunately, the file given in openvpn command line is removed just after it is launched...
Afterwards, I checked the NordVPN's logs. Indeed, there are hints that the tun-mtu has been negotiated.

Finally, I just reduced the tun-mtu to 1325, and it worked for UDP connections!
I didn't find any suitable values for TCP connections, but I'll go with this.

In my opinion, it is more a Qubes proxyVM-related issue, rather than bad NordVPN's config files. It seems that when the VPN is in a separate VM, it does not take into account ICMP packets, and therefore is unable to dynamically adapt some parameters.

Thanks for your help, I feel secure on the Internet when checking my encrypted mailbox now!

Christophe

[Chris Laprise]

On 11/26/2018 01:27 PM, Christophe Pfeifer wrote:
> Yes, you are right, it is a binary.
> I commented out these lines to no avail.
>
> Then, I tried to locate the openvpn config file used by NordVPN's binary. Unfortunately, the file given in openvpn command line is removed just after it is launched...
> Afterwards, I checked the NordVPN's logs. Indeed, there are hints that the tun-mtu has been negotiated.
>
> Finally, I just reduced the tun-mtu to 1325, and it worked for UDP connections!
> I didn't find any suitable values for TCP connections, but I'll go with this.
>
> In my opinion, it is more a Qubes proxyVM-related issue, rather than bad NordVPN's config files. It seems that when the VPN is in a separate VM, it does not take into account ICMP packets, and therefore is unable to dynamically adapt some parameters.
>
> Thanks for your help, I feel secure on the Internet when checking my encrypted mailbox now!
>
> Christophe
>
>

Interesting... Thanks for the valuable feedback!

--

Chris Laprise, tas...@posteo.net