Leak Problems with VPN ProxyVM + AirVPN & Network lock

[SEC Tester]

Im trying to setup a VPN ProxyVM on Qubes R3.2

==================
Here's what works:
==================
Ive got AirVPN GUI setup and working on Fedora-23-minimal
My AppVM can proxy through VPN ProxyVM
whatismyip.com shows the VPN IP

====================
Here's whats broken:
====================
When i leak test the browser on the AppVM, my real IP leaks.

The AirVPN GUI has a nice Network lock feature, that works well on the ProxyVM, stops leaks.

However, the network lock feature blocks the AppVM too, cutting off its internet.

In the AirVPN GUI, there are advanced settings that are suppose to allow lockal vpn traffic. And you can even specify specific IP's. Unfortunately this isnt working.
=====================

Im hoping someone with a higher understanding of IP tables, and networking can help me find a solution.

Here is a link to the airVPN GUI client https://airvpn.org/linux/

If you email them they will likely give your a 3 day trial account to test. but you probably dont even need an account to see what the network lock is doing to tables, and why the exception isn't working.

I have been trying to solve this on the AirVPN forum, but no fix yet. Here is the thread > https://airvpn.org/topic/20157-problem-with-network-lock-on-qube-os/

Thanks for your time :)

PS: Ive tried using fedora-23 standard template too, same problem.

[SEC Tester]

I've considered leaving network lock off, and building my own custom IP Tables, or firewall rules to stop the leaks.

But this is currently beyond my skill set, so would need some hand holding to learn what to do.

I have looked at the section here on the Qubes site on how to stop leaks using scripts, but its kinda confusing, and looks like its for a CLI approach, when i would prefer to have my AirVPN GUI for convince.

https://www.qubes-os.org/doc/vpn/#proxyvm

[Chris Laprise]

The VPN doc definitely uses scripts to block leaks. However, the editing
of files is kept to a minimum as its mostly copy and paste.

The two basic things the scripts do are block all direct forwarding
to/from eth0 (the Qubes uplink interface), and setup DNAT rules that
allow DNS requests to be tunneled. This is roughly analogous to running:

iptables -I FORWARD -o eth0 -j DROP
iptables -I FORWARD -i eth0 -j DROP
(during firewall setup)

/usr/lib/qubes/qubes-setup-dnat-to-ns
(triggered by the openvpn 'up' script command which runs after openvpn
puts new DNS values in /etc/resolv.conf)

To use the VPN doc scripts with the AirVPN GUI wrapper, you could try
running the GUI program under the 'qvpn' group, granting it access to
the net. Or you could change the policy of the OUTPUT chain to ACCEPT to
bypass the group restriction, which doesn't affect leak prevention for
forwarded traffic.

Chris

[David Hobach]

On 11/09/2016 01:51 PM, SEC Tester wrote:
> Im trying to setup a VPN ProxyVM on Qubes R3.2
>
> ==================
> Here's what works:
> ==================
> Ive got AirVPN GUI setup and working on Fedora-23-minimal
> My AppVM can proxy through VPN ProxyVM
> whatismyip.com shows the VPN IP
>
> ====================
> Here's whats broken:
> ====================
> When i leak test the browser on the AppVM, my real IP leaks.

What test do you use?

> The AirVPN GUI has a nice Network lock feature, that works well on the ProxyVM, stops leaks.
>
> However, the network lock feature blocks the AppVM too, cutting off its internet.
>
> In the AirVPN GUI, there are advanced settings that are suppose to allow lockal vpn traffic. And you can even specify specific IP's. Unfortunately this isnt working.
> =====================
>
> Im hoping someone with a higher understanding of IP tables, and networking can help me find a solution.

In general I'd recommend not to play with iptables in any Qubes proxy or
netvm unless you know for 100% what you're doing and are following Qubes
changes all of the time. I.e. I'd recommend to avoid any tools employing
iptables which were not written explicitly for Qubes as well.

Why: Qubes apparently manages its firewall settings for all VMs (Qubes
VM Manager --> Firewall tab) via qubes-firewall in dom0, which injects
all necessary firewall rules during runtime to the respective proxy VM.
This is done whenever a VM is started or stopped etc.

Your firewall settings are constantly being reset and manipulated by
Qubes. Your custom changes will disappear, if you don't use the
Qubes-method of persisting them. However even then your custom changes
might not work well with the Qubes changes and you might run into
unexpected issues such as your downstream appVMs suddenly having
internet access even though you configured it differently in Qubes (but
your custom rules somehow allow it).
Moreover this behaviour might change with newer Qubes versions...
Maybe the iptables lines mentioned at https://www.qubes-os.org/doc/vpn/
will continue to work in the future, maybe they won't. Will you check
that site every 3 months? Will you hope that no one forgot to update it
(is it currently up-to-date anyway)?

In short:
Avoid any iptables usage in proxy and netvms.

Use standard OpenVPN or the network manager GUI and implement firewall
rules using the Qubes Manager GUI to only allow access to your VPN
servers from your proxy VM.


P.S.:
I had once posted a script to do the VPN setup here, but I wouldn't
recommend that neither anymore as it did iptables changes in the proxy
VM as well.

[Chris Laprise]

On 11/10/2016 01:28 PM, David Hobach wrote:
> I'd recommend to avoid any tools employing iptables which were not
> written explicitly for Qubes as well.

This. Or at least don't use them without careful inspection.

>
> Your firewall settings are constantly being reset and manipulated by
> Qubes. Your custom changes will disappear, if you don't use the
> Qubes-method of persisting them. However even then your custom changes
> might not work well with the Qubes changes and you might run into
> unexpected issues such as your downstream appVMs suddenly having
> internet access even though you configured it differently in Qubes
> (but your custom rules somehow allow it).
> Moreover this behaviour might change with newer Qubes versions...
> Maybe the iptables lines mentioned at
> https://www.qubes-os.org/doc/vpn/ will continue to work in the future,
> maybe they won't. Will you check that site every 3 months? Will you
> hope that no one forgot to update it (is it currently up-to-date anyway)?

FYI, the VPN doc is up to date, and the way that Qubes firewall works
has not changed significantly in all the 3.x releases.

Note that qubes-firewall-user-script is designed to get the last say in
the iptables configuration process, and in this case place the two
anti-leak commands at the _start_ of the chain. This isn't going to just
stop working on its own, short of a Qubes bug that stops the script from
running.

Now that I see the airvpn client app changes iptables, I would say
/that/ is a cause for concern..... The user script might not have the
final say in the firewall configuration and we don't know what would
happen to the anti-leak commands then. But that's why clients such as
OpenVPN don't touch iptables in the first place-- leak prevention has to
happen outside the client.

So I think the takeaway here is that judicious use of
qubes-firewall-user-script is fine, but don't blindly use VPN clients
that try to alter iptables (or if you must, then at least turn all
firewall features off in the client). I would also advise users *not* to
rely on firewall settings in Qubes Manager/VM Settings as the options
are too limited to stop compromised VMs that are supposed to be confined
to the VPN tunnel from leaking data to clearnet (e.g. a hostile access
point or other upstream node) regardless of which address/port you specify.

If qubes-firewall-user-script goes away as a feature in Qubes 4.0, that
would certainly need to be addressed in the doc. But I'd expect all user
documentation to be reviewed for 4.0 in any case.

Chris

[Sec Tester]

Thank you Chris & David for the replies.

Unfortunately at this stage no one seems to know a solution. I will try out the Qubes VPN guide, as i really need to use my vpn. But will miss the AirVPN GUI features.

I hope in time i'll find a way to secure from leaks while still using the GUI.

Please post steps if anyone finds a way.

"What test do you use?"

I just googled "VPN leak test", ran a few on the first page.

[Sec Tester]

After further testing, more specifically its a DNS IP leak with the AirVPN GUI with network lock off.

I also leak DNS when running OpenVPN in the VPN-Proxy-VM,

Havent yet applied Qubes scripts to stop leaks.

[Sec Tester]

I have successfully applied the setup and scripting in https://www.qubes-os.org/doc/vpn/

No more DNS leaks.

This means i can atleast use my vpn, until i find a way to make things work with the AirVPN GUI.

[Chris Laprise]

A tip for stopping DNS leaks with the GUI: You have to run a script like
'qubes-setup-dnat-to-ns' (in Qubes) or 'qubes-vpn-handler.sh' (in the
VPN doc) after the client connects or else DNS packets won't get
forwarded through the tunnel. Looking at the airvpn program, you could
probably symlink its 'update-resolv-conf' to point to
'qubes-vpn-handler.sh' and it should work. Just don't click on the
'Activate Network Lock' as that will overwrite the firewall rules.

Chris

[David Hobach]

On 11/10/2016 10:07 PM, Chris Laprise wrote:
> On 11/10/2016 01:28 PM, David Hobach wrote:
>> I'd recommend to avoid any tools employing iptables which were not
>> written explicitly for Qubes as well.
>
> This. Or at least don't use them without careful inspection.

Might be worth to put some warning on the wiki page for less experienced
users...

> I would also advise users *not* to
> rely on firewall settings in Qubes Manager/VM Settings as the options
> are too limited to stop compromised VMs that are supposed to be confined
> to the VPN tunnel from leaking data to clearnet (e.g. a hostile access
> point or other upstream node) regardless of which address/port you
specify.

Can you please explain that in a more detailed way?

Currently I disagree as I don't see a way to leak anything if the user
employs the Qubes Firewall for the proxy VM to only allow access to his
VPN gateway IPs (preferably not hostnames) and disallows everything else
(incl. DNS); in particular nothing is leaked when the VPN is down.

This approach might not work for all VPN providers as some e.g. do load
balancing via DNS or other tricks, but for some it does. For the others,
yes, Qubes Firewall might be too limited.

Side note: I recently ran into
https://github.com/QubesOS/qubes-issues/issues/1183 - I'm still not 100%
sure whether it's absolutely impossible to get some unexpected DNS leaks
from that bug.

@Sec Tester:
I also checked for leaks using your "google method", but didn't observe
any except for the local IP which is a browser issue.
Glad to hear you got it done as well.

[Chris Laprise]

On 11/11/2016 01:24 PM, David Hobach wrote:
> On 11/10/2016 10:07 PM, Chris Laprise wrote:
> > On 11/10/2016 01:28 PM, David Hobach wrote:
> >> I'd recommend to avoid any tools employing iptables which were not
> >> written explicitly for Qubes as well.
> >
> > This. Or at least don't use them without careful inspection.
>
> Might be worth to put some warning on the wiki page for less
> experienced users...
>
> > I would also advise users *not* to
> > rely on firewall settings in Qubes Manager/VM Settings as the options
> > are too limited to stop compromised VMs that are supposed to be
> confined
> > to the VPN tunnel from leaking data to clearnet (e.g. a hostile access
> > point or other upstream node) regardless of which address/port you
> specify.
>
> Can you please explain that in a more detailed way?
>
> Currently I disagree as I don't see a way to leak anything if the user
> employs the Qubes Firewall for the proxy VM to only allow access to
> his VPN gateway IPs (preferably not hostnames) and disallows
> everything else (incl. DNS); in particular nothing is leaked when the
> VPN is down.
>
> This approach might not work for all VPN providers as some e.g. do
> load balancing via DNS or other tricks, but for some it does. For the
> others, yes, Qubes Firewall might be too limited.

People often use VPNs to safely access the Internet through Wifi access
points and routers and ISPs that are hostile. If the VPN-connected appVM
is compromised it could search for the VPN IP address in order to send
cleartext to the router. All of the known VPN addresses in the world
could very easily be programmed into the malware, as this search space
is tiny compared to the number of IPv4 addresses.

So we have a way of blocking anything at all that might be forwarded to
the upstream network interface. This is much better than filtering by
IP. Users can employ whatever addressing scheme, and we don't have to
instruct them to hard-code IP addresses into scripts, config files and
VM settings... the address preconfigured in a downloaded config file
will work automatically and be completely protected.

>
> Side note: I recently ran into
> https://github.com/QubesOS/qubes-issues/issues/1183 - I'm still not
> 100% sure whether it's absolutely impossible to get some unexpected
> DNS leaks from that bug.

That's causing a whitelist operation to fail, so the DNS packets would
be blocked instead of leaked. I believe that's why the issue was flagged
as minor. Also if Linux netfilter had decided to route them in a leaky
way (send to eth0) they would be blocked by the forward-blocking
commands from the VPN doc anyway.

Chris

[Sec Tester]

On Saturday, 12 November 2016 04:22:37 UTC+10, Chris Laprise wrote:
> >
>
> A tip for stopping DNS leaks with the GUI: You have to run a script like
> 'qubes-setup-dnat-to-ns' (in Qubes) or 'qubes-vpn-handler.sh' (in the
> VPN doc) after the client connects or else DNS packets won't get
> forwarded through the tunnel. Looking at the airvpn program, you could
> probably symlink its 'update-resolv-conf' to point to
> 'qubes-vpn-handler.sh' and it should work. Just don't click on the
> 'Activate Network Lock' as that will overwrite the firewall rules.
>
> Chris

Im interested in building a script to work around AirVPN GUI, as opposed to OpenVPN. I would really have to research and understand exactly what each line of the current script is doing to manipulated it to work with AirVPN.

This is currently out of my ability. I would welcome collaboration on this task. If i do eventually get something working, i will be sure to post it back here

Just for anyones future reference, https://ipleak.net/ was a nice tool for leak tests. others worked as well tho.

[entr0py]

Sec Tester:

> On Saturday, 12 November 2016 04:22:37 UTC+10, Chris Laprise wrote:
>>>
>>
>> A tip for stopping DNS leaks with the GUI: You have to run a script like
>> 'qubes-setup-dnat-to-ns' (in Qubes) or 'qubes-vpn-handler.sh' (in the
>> VPN doc) after the client connects or else DNS packets won't get
>> forwarded through the tunnel. Looking at the airvpn program, you could
>> probably symlink its 'update-resolv-conf' to point to
>> 'qubes-vpn-handler.sh' and it should work. Just don't click on the
>> 'Activate Network Lock' as that will overwrite the firewall rules.
>>
>> Chris
>
> Im interested in building a script to work around AirVPN GUI, as opposed to OpenVPN. I would really have to research and understand exactly what each line of the current script is doing to manipulated it to work with AirVPN.
>
> This is currently out of my ability. I would welcome collaboration on this task. If i do eventually get something working, i will be sure to post it back here
>

You might get more interest if you explained which features of the AirVPN GUI are worth having. The Github README is blank.

I think most openvpn users are content to use the official client since it's simpler and better audited. The current fail-close solution has also been reviewed by some intelligent (and paranoid) people. Once the VPN is up, the GUI is hidden behind your work so I'm not sure what advantage it has.

[Sec Tester]

> You might get more interest if you explained which features of the AirVPN GUI are worth having. The Github README is blank.
>
> I think most openvpn users are content to use the official client since it's simpler and better audited. The current fail-close solution has also been reviewed by some intelligent (and paranoid) people. Once the VPN is up, the GUI is hidden behind your work so I'm not sure what advantage it has.

Primary reason, the AirVPN GUI makes it very fast to change between the 172 servers AirVPN has https://airvpn.org/status/

GUI shows the stats for each server load, latency. Handy when picking which one to connect to.

Also handy to see current uplaod/download speeds. Shows current IP address.

[David Hobach]

>> > I would also advise users *not* to
>> > rely on firewall settings in Qubes Manager/VM Settings as the options
>> > are too limited to stop compromised VMs that are supposed to be
>> confined
>> > to the VPN tunnel from leaking data to clearnet (e.g. a hostile access
>> > point or other upstream node) regardless of which address/port you
>> specify.
>>
>> Can you please explain that in a more detailed way?
>>
>> Currently I disagree as I don't see a way to leak anything if the user
>> employs the Qubes Firewall for the proxy VM to only allow access to
>> his VPN gateway IPs (preferably not hostnames) and disallows
>> everything else (incl. DNS); in particular nothing is leaked when the
>> VPN is down.
>>
>> This approach might not work for all VPN providers as some e.g. do
>> load balancing via DNS or other tricks, but for some it does. For the
>> others, yes, Qubes Firewall might be too limited.
>
> People often use VPNs to safely access the Internet through Wifi access
> points and routers and ISPs that are hostile. If the VPN-connected appVM
> is compromised it could search for the VPN IP address in order to send
> cleartext to the router. All of the known VPN addresses in the world
> could very easily be programmed into the malware, as this search space
> is tiny compared to the number of IPv4 addresses.

Assuming your VPN setup was not compromised the infrastructure being
used does not matter - even if it's potentially hostile (most of it
should actually be assumed to be hostile). Due to the beauties of
cryptography there's a tunnel between your proxy VM and your VPN exit
node as if there was a fixed line. And yes, your appvm using that
proxyVM can obviously find your VPN exit node IP by checking its IP with
whatever public service. And then what?
It cannot communicate outside of that tunnel as Qubes makes sure it
cannot use anything else than your proxy VPN VM; so I don't see how it
should identify your ISP IP address (except for maybe state-level
attacks such as netflow pattern analysis on a global scale).

> So we have a way of blocking anything at all that might be forwarded to
> the upstream network interface. This is much better than filtering by
> IP. Users can employ whatever addressing scheme, and we don't have to
> instruct them to hard-code IP addresses into scripts, config files and
> VM settings... the address preconfigured in a downloaded config file
> will work automatically and be completely protected.

I fully agree that this more generic solution is better assuming it's
properly reviewed & maintained.

>> Side note: I recently ran into
>> https://github.com/QubesOS/qubes-issues/issues/1183 - I'm still not
>> 100% sure whether it's absolutely impossible to get some unexpected
>> DNS leaks from that bug.
>
> That's causing a whitelist operation to fail, so the DNS packets would
> be blocked instead of leaked. I believe that's why the issue was flagged
> as minor.

Not 100% correct. Another address is whitelisted instead, namely that of
the netvm. So your DNS requests might go plain text through the netvm
(!= VPN --> leak) in the worst case. The things that help here should be
the routing table installed by openvpn, the DNS server settings of your
proxy VM & maybe the other iptables commands from the VPN doc assuming a
user manually implemented them, yes. So probably only some further bugs
in combination would lead to serious issues.

@Sec Tester:
AirVPN enables you to download the openVPN config files per VPN exit
node, i.e. switching should be as easy as writing some small bash script
to get it done on the command-line.
Alternatively they provide some generic config files by country which
apparently do the selection for you, i.e. by using one of these you
should also have some variety of exit nodes.

[hed...@tutanota.com]

11. Nov 2016 12:20 by sectest...@gmail.com:

I have successfully applied the setup and scripting in https://www.qubes-os.org/doc/vpn

No more DNS leaks.

Quite some time ago I created a number of proxyVMs using the template supplied by (I think) Chris Laprise. The setup detailed in the Qubes docs, referred to above, is considerably more complex than the setup I'm running.

Can anyone explain the advantages of the "new" proxyVM setup (ie the one described in the Qubes docs) compared to Chris's original template, and do people think it would be worth my while to update the dozen+ proxyVMs I currently have to the new format?

[Chris Laprise]

On 11/12/2016 10:40 AM, hed...@tutanota.com wrote:
> 11. Nov 2016 12:20 by sectest...@gmail.com

> <mailto:sectest...@gmail.com>:

>
> I have successfully applied the setup and scripting in

> https://www.qubes-os.org/doc/vpn <https://www.qubes-os.org/doc/vpn/>

>
> No more DNS leaks.
>
>
> Quite some time ago I created a number of proxyVMs using the template
> supplied by (I think) Chris Laprise. The setup detailed in the Qubes
> docs, referred to above, is considerably more complex than the setup
> I'm running.
>
>
> Can anyone explain the advantages of the "new" proxyVM setup (ie the
> one described in the Qubes docs) compared to Chris's original
> template, and do people think it would be worth my while to update the
> dozen+ proxyVMs I currently have to the new format?
>

By 'template' you mean the setup at my github repo? If you look closely,
they are 90% the same except the doc version uses rc.local to start the
client and the one on github creates a systemd service for it. What
makes it look simpler is the github readme says 'download the file,
unzip in /rw/config and adjust the ovpn settings' and doesn't show
script code.

Chris

[Chris Laprise]

If the VPN VM isn't internally restricting all forwarding, and VM
firewall settings (upstream proxyVM) is filtering by IP address, then a
compromised downstream appVM can send packets to the server IP that are
crafted to go around the tunnel. The most famous example of this is DNS
packets. People complain about them leaking all the time. On Qubes, DNS
is dnat-ed to a set address, but having this set properly depends on a
trigger from the VPN client... which may have failed. The system should
also be setup to prevent leaks if the VPN client fails, which could mean
that the client fails to set the default route or has a failure mode
that un-sets it.

>
>> So we have a way of blocking anything at all that might be forwarded to
>> the upstream network interface. This is much better than filtering by
>> IP. Users can employ whatever addressing scheme, and we don't have to
>> instruct them to hard-code IP addresses into scripts, config files and
>> VM settings... the address preconfigured in a downloaded config file
>> will work automatically and be completely protected.
>
> I fully agree that this more generic solution is better assuming it's
> properly reviewed & maintained.
>
>>> Side note: I recently ran into
>>> https://github.com/QubesOS/qubes-issues/issues/1183 - I'm still not
>>> 100% sure whether it's absolutely impossible to get some unexpected
>>> DNS leaks from that bug.
>>
>> That's causing a whitelist operation to fail, so the DNS packets would
>> be blocked instead of leaked. I believe that's why the issue was flagged
>> as minor.
>
> Not 100% correct. Another address is whitelisted instead, namely that
> of the netvm. So your DNS requests might go plain text through the
> netvm (!= VPN --> leak) in the worst case. The things that help here
> should be the routing table installed by openvpn, the DNS server
> settings of your proxy VM & maybe the other iptables commands from the
> VPN doc assuming a user manually implemented them, yes. So probably
> only some further bugs in combination would lead to serious issues.

I see. So that is similar to the scenario I described above.

Chris

[hed...@tutanota.com]

12. Nov 2016 20:39 by tas...@openmailbox.org:

By 'template' you mean the setup at my github repo? If you look closely, they are 90% the same except the doc version uses rc.local to start the client and the one on github creates a systemd service for it. What makes it look simpler is the github readme says 'download the file, unzip in /rw/config and adjust the ovpn settings' and doesn't show script code.

Chris

No, it didn't come from github. After a brief search, I found the thread that was the source I used: https://groups.google.com/forum/#!msg/qubes-users/-9gR1Va3BnY/nQG6j-YOtZ4J;context-place=topic/qubes-users/T0wbCuIgISg  which dates from March 2015. The author was cprise, so I was wrong to attribute it to Chris Laprise, though the names do seem suspiciously similar. ;)

I guess the question still stands: is the latest version materially superior to the March 2015 version? (And enough to want to re-create over a dozen proxyVMs?)

[Chris Laprise]

On 11/12/2016 05:47 PM, hed...@tutanota.com wrote:
>
> I guess the question still stands: is the latest version materially
> superior to the March 2015 version? (And enough to want to re-create
> over a dozen proxyVMs?)

Yes, the VPN doc method is better in the sense that it separates packets
generated from the VPN VM from the packets going to/from appVMs. So
accidental net access generated while using the VPN CLI, for example,
will be blocked and stay out of the VPN tunnel. Its not critical but
Whonix people wanted it as a precaution.

Chris

[hed...@tutanota.com]

13. Nov 2016 02:54 by tas...@openmailbox.org:

On 11/12/2016 05:47 PM, hed...@tutanota.com wrote:

I guess the question still stands: is the latest version materially superior to the March 2015 version? (And enough to want to re-create over a dozen proxyVMs?)

Yes, the VPN doc method is better in the sense that it separates packets generated from the VPN VM from the packets going to/from appVMs. So accidental net access generated while using the VPN CLI, for example, will be blocked and stay out of the VPN tunnel. Its not critical but Whonix people wanted it as a precaution.

Chris

Thanks for that. "Not critical" sounds like a good reason to stay with what I have for now, though I'll ensure that any new VPN proxyVMs I create use the new code. I might even lazily migrate them over one by one if I feel motivated enough to do so.

And just to clarify, your github repo code is at https://github.com/tasket/Qubes-vpn-support . Correct?