Installing VPN in Qubes Versus VPN on a Router

[ama...@riseup.net]

We see much correspondence in these forums about installing a VPN within
Qubes. Surely, the most secure place for VPN is to install on a Router?
I say these things after reading the following paper [
https://cryptome.org/2013/12/Full-Disclosure.pdf ] in which a group of
hackers demonstrate that the majority of routers (in-particular those
provided by ISP's] have backdoors to government agencies. These
adversary's are able attack our LAN and its devices; including the
ability to intercept VPN and Tor traffic.
The solution they say is to isolate these rogue routers in the
Militarized Zone by creating a DMZ [demilitarized zone]. Achieved by
installing a 2nd router [flashed with open source firmware such as
OPenWRT]. It is here, on the router, that we should enable and run
OpenVPN.
Thoughts on this paper and it's conclusions are welcomed

[Sec Tester]

I guess the main benefit to having VPN on router is it takes that overhead off the PCs CPU & memory.

But the paper is right, a lot of network hardware is backdoored. Especially the cisco stuff. And im suspicious of the Chinese stuff too.

We should endeavor to run open source routers. But im not aware of any open source modems? Im actually surprised someone hasnt cracked the proprietary DSL code and leaked an open source modem.

I bet we would not like what we found in their proprietary code :/

Having a VPN-Proxy-VM offers the flexibility to chose what VMs directly connect to the internet, and which VMs are routed through the VPN which is nice.

I've set my VPN-Proxy-VM using a minimal template, to future reduce the attack surface.

You can also run the whonix-gw over the vpn, or vise versa.

I imagine since snowden said to the world he uses Qubes OS, the NSA have had their team looking for ways in. I think qubes can be hardened much more than it currently is.

[hed...@tutanota.com]

13. Nov 2016 08:48 by ama...@riseup.net:

An always-on VPN connection on the router works well but can be a bit slow since the processing power of router CPUs is generally quite limited. If choosing a router, I'd suggest a dual-core ARM-based device. Although openvpn is only single-threaded you can usually configure cpu-affinity to place it on one core and the other routing tasks on the other core.

For those who want to go beyond around 20-25 Mb/s, which is where an ARM router will start to reach its limits, a fine alternative is a small fanless PC, such as the Intel NUC or Gigabyte Brix, and run an open source firewall on it, instead of a router. I'm using IPFire. If the processor supports AES-NI, the limiting factor will be your network speed, not the firewall's CPU.

Finally, I've always felt that running a vpn on Qubes and having an always-on vpn running on a router/PC complement each other.

[Achim Patzner]

Am 13.11.2016 um 14:22 schrieb hed...@tutanota.com:

13. Nov 2016 08:48 by ama...@riseup.net:

We see much correspondence in these forums about installing a VPN within Qubes. Surely, the most secure place for VPN is to install on a Router?

You might continue proving that this is the case for a router running on its own VM compared to a router running on separate hardware but keep in mind counting the problem of keeping the router's os current and free of security-relevant problems.

The solution they say is to isolate these rogue routers in the Militarized Zone by creating a DMZ [demilitarized zone]. Achieved by installing a 2nd router [flashed with open source firmware such as OPenWRT]. It is here, on the router, that we should enable and run OpenVPN.

And of course another router/packet filter/firewall/whatever behind it as there could be something _inside_ the VPN that would not be agreaable to you.

Thoughts on this paper and it's conclusions are welcomed

There is a point where additional components won't give you defense-in-depth but only additional complexity that will in the end make you less secure.

An always-on VPN connection on the router works well but can be a bit slow since the processing power of router CPUs is generally quite limited. If choosing a router, I'd suggest a dual-core ARM-based device. Although openvpn is only single-threaded you can usually configure cpu-affinity to place it on one core and the other routing tasks on the other core.

One of the GL-Inet small arm(s 8-) ) routers is sufficient for 80 MBit/s (see https://www.gl-inet.com/). I'm using one of their "Mifi" devices (https://www.gl-inet.com/mifi/) to write this and right now it is holding up quite well with 150 MBit/s LTE plus an OpenVPN on top of it. The only problem is the about 1MBit/s I'm getting from their uplink.

For those who want to go beyond around 20-25 Mb/s, which is where an ARM router will start to reach its limits

Seriously? I doubt that. Right now I'm using an ASUS RT-AC5300 (ARM, dual core) router on a 400/20 MBit link (residential cable) and even if I'm sturating it using an OpenVPN process running on the router its cores seem quite unimpressed. But maybe DD-WRT is magical.

, a fine alternative is a small fanless PC, such as the Intel NUC or Gigabyte Brix, and run an open source firewall on it, instead of a router.

For security-sensitive applications I'm using a USBArmory-based "crypto-afterburner" that I can plug into other machines offering two "USB-NICs" and I don't have problems with reathing the USB bandwidth limit. If it wasn't impossible to get a single USB port into a VM I would have found a place to stick one inside my Thinkpad already. If there was a Qubes developer feeling bored I would have thrown one at him already to see if we could have a few interesting things introduced into Qubes (like boot media running on a separate volume that need to be unlocked first, external key storage, external crypto functions…)

Finally, I've always felt that running a vpn on Qubes and having an always-on vpn running on a router/PC complement each other.

And an independent packet filter in front of it. And one behind it. And no wireless networking in between any component. Again: Consider a USB Armory; write some interesting tools, add them to Qubes. That might really help.


Achom

[hed...@tutanota.com]

13. Nov 2016 16:01 by no...@noses.com:

Am 13.11.2016 um 14:22 schrieb hed...@tutanota.com:

13. Nov 2016 08:48 by ama...@riseup.net:

Thoughts on this paper and it's conclusions are welcomed

There is a point where additional components won't give you defense-in-depth but only additional complexity that will in the end make you less secure.

Allowing a backdoored router into your network will, complexity or no complexity, compromise your security. The only conclusion to reach is not to use them wherever possible, or isolate them if their use is mandatory.

An always-on VPN connection on the router works well but can be a bit slow since the processing power of router CPUs is generally quite limited. If choosing a router, I'd suggest a dual-core ARM-based device. Although openvpn is only single-threaded you can usually configure cpu-affinity to place it on one core and the other routing tasks on the other core.

One of the GL-Inet small arm(s 8-) ) routers is sufficient for 80 MBit/s (see https://www.gl-inet.com/). I'm using one of their "Mifi" devices (https://www.gl-inet.com/mifi/) to write this and right now it is holding up quite well with 150 MBit/s LTE plus an OpenVPN on top of it. The only problem is the about 1MBit/s I'm getting from their uplink.

I've never come across these devices. They look like good value for money.

For those who want to go beyond around 20-25 Mb/s, which is where an ARM router will start to reach its limits

Seriously? I doubt that. Right now I'm using an ASUS RT-AC5300 (ARM, dual core) router on a 400/20 MBit link (residential cable) and even if I'm sturating it using an OpenVPN process running on the router its cores seem quite unimpressed. But maybe DD-WRT is magical.

Yeah, maybe my 25 Mb/sec generalisation is a bit out-of date but it still depends on what you're prepared to spend. Let's see: ASUS RT-AC5300. It has 8 antennas and is a beast of a router that sells for 439 euros on amazon.de. At that price it really ought to be fast. Back in more reasonably-priced territory, I did some real-world tests 18 months ago on my ASUS RT-AC56U (97 euros on amazon.de, ARM x 2) and never exceeded 25 Mb/s with 80% cpu load. Even had it achieved 100% cpu, that would still only equate to 30 Mb/s. Flippant comments about magic aside, if you throw big mony at the hardware, you'll get more speed. I'm still betting that a small i3 with AES-NI would outperform it on openvpn, and for a fraction of the price.

[Tai...@gmx.com]

Ideally you would want a blob free coreboot system with no Intel ME or
AMD PSP type backdoors.
https://www.coreboot.org/Binary_situation
Intel is actively trying to nerf free software with Boot Guard/ME, if
you buy a computer with those features it isn't really your computer.

A backdoor in a modem is irrelevant, it is post WAN and should be
considered part of the "internet".

You need a computer with more than one server grade pci-e interfaced
nics if you want real LAN>WAN performance, 25Mbps is simply a pitiful
amount to settle for - the newer "server" grade ARM chipsets can do much
better than that.

[entr0py]

Tai...@gmx.com:

> Ideally you would want a blob free coreboot system with no Intel ME or AMD PSP type backdoors.
> https://www.coreboot.org/Binary_situation
> Intel is actively trying to nerf free software with Boot Guard/ME, if you buy a computer with those features it isn't really your computer.
>
> A backdoor in a modem is irrelevant, it is post WAN and should be considered part of the "internet".
>

Right, I've always followed the advice to secure each pc as if it were connected directly to the internet and not to rely on the router for any security.

But now I'm interested in actually building a secure router. One reason is what you mentioned regarding Intel ME. Since Qubes 4.0 will require VT-d (and unavoidably Intel ME) and the fact that it's cool to use new hardware, I'd like to place a physical barrier to block ME signals.

I had always imagined repurposing a Qubes PC to serve as a router, especially because of the flexibility it has with chaining and branching multiple transparent proxy VMs. But obviously now, it doesn't make any sense to use an ME equipped machine as a router.

So if I had a budget (for argument's sake) of $2000 to build a secure router for 10-15 clients in a small business environment where maximum throughput is not really an issue, what would you all advise? A libreboot machine? but then what kind of OS could it run that could meaningfully isolate sys-net and provide similar routing capabilities?

TIA.

[Grzesiek Chodzicki]

Have You considered running PfSense as Your main router OS on a dedicated box? You need a small PC with more than one network interface card. PfSense is open source, it's infinitely configurable and has an extensive plugin system to extend it beyond typical router capabilities.

[Tai...@gmx.com]

VT-d is intels marketing term for IOMMU, you can buy an AMD system that
has IOMMU (AMD-Vi) (but not FM2/AM4 as that has PSP). Qubes needs IOMMU
not "VT-d"

You can use a VMM with a pfsense VM and separate driver domains for the
network interfaces, qubes isn't a router operating system...

There is no getting around ME, on the coreboot list there is some talk
of nerfing the binary (thanks Trammel Hudson!) but other than that
you're still supporting a company that makes insecure technology if you
buy their products.

Things you may want to look in to (5K is a great deal for the level of
juice this has)
https://www.crowdsupply.com/raptorcs/talos

x86/wintel is only a small subsection of the computing world, you can
buy for instance an IBM OPOWER8 workstation or the Tyan Palmetto (also
OPOWER8) - they have open source firmware and no ME type stuff. - OPOWER
has an IOMMU equivalent.

The newish and readily available blob free x86 amd boards are high
performance level (kgpe-d16) I don't know what your connection is like
so if you want something lower power you could go with a coreboot board
with the features you want and simply not include the blobs (which could
mean no video, no fan control and no USB3 - but none of those are needed
on a passively cooled router anyways and you can install/control via serial)

There is the apu2 from pcengines, which has no blobs (AFIAK, ask them)
although it doesn't have an IOMMU.


I find it ironic that you apparently value your privacy but you are
using gmail - if you do not pay for a service YOU are the product.

[entr0py]

Tai...@gmx.com:

> VT-d is intels marketing term for IOMMU, you can buy an AMD system
> that has IOMMU (AMD-Vi) (but not FM2/AM4 as that has PSP). Qubes
> needs IOMMU not "VT-d"

Thanks for reply. I understood this previously but I'm not familiar with AMD's offerings and didn't realize they had a current lineup that fits this category. It also seems that Skylake i3's have IOMMU without vPro.

> You can use a VMM with a pfsense VM and separate driver domains for
> the network interfaces, qubes isn't a router operating system...

Is there an inherent reason that Qubes should not be used as a router?

> x86/wintel is only a small subsection of the computing world, you can
> buy for instance an IBM OPOWER8 workstation or the Tyan Palmetto
> (also OPOWER8) - they have open source firmware and no ME type stuff.
> - OPOWER has an IOMMU equivalent.
>
> The newish and readily available blob free x86 amd boards are high
> performance level (kgpe-d16) I don't know what your connection is
> like so if you want something lower power you could go with a
> coreboot board with the features you want and simply not include the
> blobs (which could mean no video, no fan control and no USB3 - but
> none of those are needed on a passively cooled router anyways and you
> can install/control via serial)
>
> There is the apu2 from pcengines, which has no blobs (AFIAK, ask
> them) although it doesn't have an IOMMU.

Small subsection? I guess I need to get out and see more of the computing world. Thanks for the suggestions. I'll do some reading!

> I find it ironic that you apparently value your privacy but you are
> using gmail - if you do not pay for a service YOU are the product.

Yes, and that maxim applies to every website you visit that doesn't cost you any money. Everyone uses Google. Just because there's no "g" in the url doesn't mean that you're free of Google's tentacles (and fingerprinting).

Yes, I use this gmail address to access groups.google.com and nothing else, in a dedicated vm, over Tor. But you are correct - a non-gmail address, in a dedicated vm, over Tor would be considerably better. But I fail to see the irony. This pseudonym has long-ago broadcast several hundred words onto the Internet so it would be naive to think that it's still an anonymous identity. The stylometry is out there for anyone that wants to look. The distinction is that I have other pseudonyms that aren't quite so vociferous. :) Of course, Google probably has them all linked already anyway...

[Chris Laprise]

Its not just backdoors... IIRC the NSA and probably other groups greatly
prefer to attack routers for some reason. I think the reason is they are
generally neglected and insecure.

Quite frankly, there is all too much insecurity to go around... and I
don't even think software is the worst culprit anymore. We're all using
souped-up ancient architectures that expose us to things like 'DRAMA'
and it seems there is little-to-no innovation with respect to more
secure hardware architecture. Qubes tries to propose new architecture in
software, but I worry even it may not be enough.

Router vs laptop: If we regard a well-maintained OpenWRT router as more
secure than Qubes on a laptop, then we've given up on link encryption in
our applications (HTTPS, ZRTP, etc.) by implication. Then the only way
to have reliable link encryption is to have everyone we communicate with
sitting at home connecting to a single VPN server... each from their
router-bound VPN clients... tethered by an ethernet cable between router
and PC. Egads.

Chris

[Tai...@gmx.com]

On 11/13/2016 07:39 PM, entr0py wrote:
> Tai...@gmx.com:
>> VT-d is intels marketing term for IOMMU, you can buy an AMD system
>> that has IOMMU (AMD-Vi) (but not FM2/AM4 as that has PSP). Qubes
>> needs IOMMU not "VT-d"
> Thanks for reply. I understood this previously but I'm not familiar with AMD's offerings and didn't realize they had a current lineup that fits this category. It also seems that Skylake i3's have IOMMU without vPro.

- All intel computers from around 2006+ have ME, not just the ones with
vPro (which again is just a marketing term for the business level remote
management services)
They are a shitty company and you shouldn't support them anyway. (ME,
outsoucing/h1b abuses, general anti-foss attitude)
https://hackaday.com/2016/01/22/the-trouble-with-intels-management-engine/

>
>> You can use a VMM with a pfsense VM and separate driver domains for
>> the network interfaces, qubes isn't a router operating system...
> Is there an inherent reason that Qubes should not be used as a router?

- I really don't know how to reply to this

>> x86/wintel is only a small subsection of the computing world, you can
>> buy for instance an IBM OPOWER8 workstation or the Tyan Palmetto
>> (also OPOWER8) - they have open source firmware and no ME type stuff.
>> - OPOWER has an IOMMU equivalent.
>>
>> The newish and readily available blob free x86 amd boards are high
>> performance level (kgpe-d16) I don't know what your connection is
>> like so if you want something lower power you could go with a
>> coreboot board with the features you want and simply not include the
>> blobs (which could mean no video, no fan control and no USB3 - but
>> none of those are needed on a passively cooled router anyways and you
>> can install/control via serial)
>>
>> There is the apu2 from pcengines, which has no blobs (AFIAK, ask
>> them) although it doesn't have an IOMMU.
> Small subsection? I guess I need to get out and see more of the computing world. Thanks for the suggestions. I'll do some reading!
>
>> I find it ironic that you apparently value your privacy but you are
>> using gmail - if you do not pay for a service YOU are the product.
> Yes, and that maxim applies to every website you visit that doesn't cost you any money. Everyone uses Google. Just because there's no "g" in the url doesn't mean that you're free of Google's tentacles (and fingerprinting).
>
> Yes, I use this gmail address to access groups.google.com and nothing else, in a dedicated vm, over Tor. But you are correct - a non-gmail address, in a dedicated vm, over Tor would be considerably better. But I fail to see the irony. This pseudonym has long-ago broadcast several hundred words onto the Internet so it would be naive to think that it's still an anonymous identity. The stylometry is out there for anyone that wants to look. The distinction is that I have other pseudonyms that aren't quite so vociferous. :) Of course, Google probably has them all linked already anyway...
>

- I use request policy and thus I don't load any of their services soooo.
I hear excuses - It is very lazy of you not to simply get another
service, either paid or free.
there are actually one or two unicorn email providers out there that
don't do gmail style abuses, but the storage limits are realistic (300MB
or so) and you exist to get their name out in to the world and thus
promote their *paid* business email offerings. It costs them next to
nothing to provide an account like that and then it results in people
singing their praises = more business.

[entr0py]

Tai...@gmx.com:

> On 11/13/2016 07:39 PM, entr0py wrote:
>> Tai...@gmx.com:

>>> You can use a VMM with a pfsense VM and separate driver domains for
>>> the network interfaces, qubes isn't a router operating system...
>>
>> Is there an inherent reason that Qubes should not be used as a router?
>
> - I really don't know how to reply to this

I can't tell if your reticence is indignance or if my question just can't be answered for some reason but it was meant to be a sincere question. Admittedly I know very little about this but AFAIK pfSense is just a front-end to manage filters with extensibility features. I don't know enough to discuss the relative merits of PF vs iptables, but I don't see any reason why a Qubes router wouldn't work since Debian based "router operating systems" do exist. Is it a question of reliability, complexity, ...? I just need a machine that can route and filter traffic and not get compromised in the process - or am I missing something? I wouldn't know the first thing about BSD or virtual driver domains, whereas I've become comfortable chaining Qubes proxyVMs and using iptables.

> - I use request policy and thus I don't load any of their services soooo.
> I hear excuses - It is very lazy of you not to simply get another service, either paid or free.
> there are actually one or two unicorn email providers out there that don't do gmail style abuses, but the storage limits are realistic (300MB or so) and you exist to get their name out in to the world and thus promote their *paid* business email offerings. It costs them next to nothing to provide an account like that and then it results in people singing their praises = more business.

I have a vfemail account that I've used to post to this list in the past. I weighed the risks to my privacy versus the convenience of using the list platform offered by google and made a decision I'm comfortable with. For the amount of "me" that google is getting, I think it's a good deal.

[entr0py]

entr0py:

> Tai...@gmx.com:
>> On 11/13/2016 07:39 PM, entr0py wrote:
>>> Tai...@gmx.com:
>>>> You can use a VMM with a pfsense VM and separate driver domains
>>>> for the network interfaces, qubes isn't a router operating
>>>> system...
>>>
>>> Is there an inherent reason that Qubes should not be used as a
>>> router?
>>
>> - I really don't know how to reply to this
>
> I can't tell if your reticence is indignance or if my question just
> can't be answered for some reason but it was meant to be a sincere
> question. Admittedly I know very little about this but AFAIK pfSense
> is just a front-end to manage filters with extensibility features. I
> don't know enough to discuss the relative merits of PF vs iptables,
> but I don't see any reason why a Qubes router wouldn't work since
> Debian based "router operating systems" do exist. Is it a question of
> reliability, complexity, ...? I just need a machine that can route
> and filter traffic and not get compromised in the process - or am I
> missing something? I wouldn't know the first thing about BSD or
> virtual driver domains, whereas I've become comfortable chaining
> Qubes proxyVMs and using iptables.
>

From advice I've received: the overhead introduced by Qubes (inter-vm operability, gui features) aren't necessary in a router that is largely non-interactive and headless.

My guess is that a cost-effective solution for now would be to use 2012 AMD hardware running Xen / KVM. Analogous to Qubes, it would have fat net VMs, minimal proxy VMs and a firewall VM (BSD or otherwise) in-between.

Both Xen & KVM support ARM so the forward-looking solution might be to combine Xen with something like MirageOS appliances (https://mirage.io/wiki/xen-on-cubieboard2) on an ARM device.

[amadaus]

ama...@riseup.net:

Thanks everyone for your contributions.
Implicit in most of your replies is a distinct distrust of the
modems/routers provided to us.
If anyone is interested, the solution we adopted to securing our LAN is
copied from this blog;
https://tokyobreeze.wordpress.com/2015/02/01/create-a-nsa-and-hacker-proof-home-network-that-you-control/
This guy uses a couple of cheap routers loaded with OpenWRT which sit
behind his infected Modem. His 2nd routed utilises OpenVPN Client and is
configured to protect "high value" devices.
We've successfully copied this configuration and it seems!! to work. -
unless you know better??

[3n7r...@gmail.com]

Sorry, I took your thread for a bit of a detour. Going back to your original post:

> Surely, the most secure place for VPN is to install on a Router?

Joanna might disagree with that for the same reason she posits that VMs connected via Qubes networking may be more secure than physical machines separated by a potentially vulnerable TCP/IP stack. (http://invisiblethingslab.com/resources/2014/Software_compartmentalization_vs_physical_separation.pdf)

Generally speaking, it seems to be a good idea to isolate your public-facing network adapter from your firewall and proxies (vpn). Whether it's best to use Qubes, other hypervisor, physical devices, or driver domains as taiidan suggested; I don't know.

As with all things security-related, the solution that works for you will depend on your threat model, which you haven't described. Certainly, I would question the credibility of a blog that claims to have a setup that is "NSA-proof". Most of the changes recommended in the blog are simply shifting trust from your ISP to other 3rd-parties: OpenDNS, VPN provider, etc. Make sure that's what you want since everyone involved is only guaranteeing "privacy by policy."

* Using OpenDNS does not protect your kids from inappropriate content. That's just bizarre.
* If you distrust your ISP enough to require a VPN, why allow the ISP to see any unencrypted traffic at all? Blogger only uses VPN for some "sensitive" traffic because he doesn't want the rest subjected to geographic blocking. Why not just use a VPN that exits in the country where it's needed? If your activity is so sensitive that you can't exit, for example, in a 5-Eyes country, then you should be using Tor - because again, a VPN is just "privacy by policy".
* You may want to confirm that the VPN is set to fail-closed (ie not allow traffic when VPN goes down.)

[Me]

amadaus:

The Blogger is correct, the best place to install OpenVPN is to use it
within OpenWRT on a Router. As well as helping protect incoming and
outgoing traffic to your Qubes device, it can help protect smart phones,
tablets & IoT devices from being attacked and employed for Denial of
Service purposes