Service VM Size, Memory, Security

[johny...@sigaint.org]

One of the banes of a Qubes addict's existence is memory.

Too many times I see that red stop sign and breathe a sigh of frustration,
that I need to shut down or mem-set other VM's to start up another AppVM.
I like my VM separation, dammit, which means lots of VMs.

In a perfect world, I'd have a kick-ass server with 64G. In the real
world, I have to make do with 4G.

And in the domain of laptops (which I personally just can't trust any
more, lol), memory constraints of 4G-8G are going to be present for
awhile.

A few questions and observations:

- Have the Qubes devs ever had any thoughts of making the service VM's of
sys-net and sys-firewall "headless," or generally more minimalistic?

For a netcard-hosting packet slinger and a packet filter firewall VM to
both have most of their memory used up by 30M X servers, pulseaudio, and
so on, is a bit ironic. And exposes a lot more attack surface, yadda,
yadda.

(More templates is a pain, for sure, just as Whonix-Cubes uses two more
templates of its own. But in the case of these important service VM's, I
think the benefits of memory saving and reduced attack surface would be
worth it.)

Those 300M VM's add up pretty quickly on a 4G system. I think the
evolution of Qubes is probably going towards having *more* service (and
user) VM's; e.g. a actual storage domain, a default USB domain, maybe even
a sound/keyboard/mouse domain, maybe bluetooth isolation, or whatever.

(The concept of Qubes is awesome vs. a traditional metal kernel and
multiple Qemu's on top, for example. But dom0 has so much in it right
now, that benefit is partly theoretical at the current time, IMHO.)

The more separation, and the more stuff moved out of dom0 the better,
which is going to mean more VM's. If these were headless 80M (or
whatever) VM's instead of honkin' 300M VM's, it becomes a *lot* more
practical for those without super-machines.

You'd need some way to access the VM, but the aforementioned X clients
talking X protocol to an outside X server, or, even leaner, sshd would
meet those needs.

Any need for an xterm or X apps from these headless VM's could easily be
done by the X clients connecting to another VM's (or dom0's) X server
directly, without needing their own. Just a bit of iptables and X
config'ing.

You'd be putting more trust in ssh, but doing certificate based auth
between VM's, could reduce the pain, and the ssh attack surface is
probably a lot smaller than a having whole damn X server in your packet
slinger/filter.

I started work last night on making a headless sys-net accessible via
ssh/X protocol, but got a big bogged down in iptables. I'm a bit rusty,
and I need to do a good re-education of myself on iptables and hit it
again (and not at 3am :P).

Also, I spun my wheels for a bit because I didn't realize at first that
everybody on 10.137.2.* isn't really on the same subnet in the traditional
sense, but has to send everything through 10.137.2.1 sys-firewall,
correct? Seems a bit odd, but I can understand the restriction from a
security standpoint, and with ipt forwarding, the desired effect can be
achieved--with some concentration, which I didn't have last night, lol.

Actually, part of me wonders if being forced to send everything through
sys-firewall for 10.137.2.*'s to talk to each other is *more* of a
security risk. (If sys-firewall is compromised, it gets to see all the
VM's talking to each other, which wouldn't be the case of 10.137.2.*/24
acted like a normal LAN. Plus, it'd probably be more efficient without
the additional iptables relaying. But that's a secondary concern.)

(- A bit of an oblique thought, might be the sharing of memory between the
common pages of X server (and other app) instances in different VM's. But
that strikes me as having a whole nightmare of complications to implement,
along with a host of security issues. Just trimming unnecessary stuff out
of the VM's seems a lot more straight-forward.)

- It might make sense to have the sys-net and sys-firewall service VM's
based upon a different, more minimal template than the
user-app-vm-intended fedora-23.

Having pulseaudio, alsa, modemmanager (what even is that??) and a bunch of
other stuff running in a VM that is dedicated to just slinging packets or
providing an iptables firewall seems terribly wasteful. (And I'm talking
using up real resident RSS memory, not just Virtual space.)

It's definitely a side-project of mine to take sys-net and sys-firewall
down to the most minimal state/size possible. Saves memory, and reduces
attack surface, allows for more work VM's.

Plus, for the truly paranoid (me, in case you haven't picked up on that
yet; although I prefer the term "cautious"), there has been talk about
sound cards being as covert channels (potentially between VMs in this
case). So it seems like an unnecessary security risk.

Yes, in order for sys-net or sys-firewall to talk to other VM's via the
sound card, you need both ends to be compromised, in which case you're
kinda screwed anyway, but still, why have any attack surface exposed, and
memory wasted, when you don't have to.

(Actually, that's not quite true, if there's any validity to Ruiu's
air-gap-jumping audio-based malware claims, then a compromised
sys-vm/sys-firewall could use their audio abilities to send stuff through
the Qubes systems dom0 speakers, to a listening smartphone or other
air-gapped compromised device. Some of that stuff seems a bit of a
stretch, but why run the risk, if you don't have a compelling need to be
playing MP3's or watching YouTubes from sys-net :) Even though I'm
slightly skeptical of some of his claims, I still only use headphones, not
speakers, and mute dom0's volume control when not in use, for the heck of
it.)

- With lean, mean service-based template, I'd probably be more willing to
run i2p, Freenet, a Tor relay, etc., in streamlined headless VM's. But as
it stands, I just can't spare the VM space, which is kind of sad. :(

- Another related question: if I have enough swap space in dom0, why a I
restricted to launching VM's due to memory restrictions? I realize you'd
get into swapping-hell pretty quickly if you weren't careful, and one can
crank down individual VM's max memory to make them do their own internal
swapping, but I'm still curious as to why VM's can't be a bit more
"swappy" from dom0's standpoint. Does their physical memory (from their
standpoint) have to be physical memory in dom0, period?

- And slightly off-topic, but somewhat related: I like Qubes-whonix
(despite my hesitation to add another trust actor on top of Redhat,
Invisible Labs [you guys are cool, tho' :)]).

It seems weird to me that Qubes-Whonix's whonix-gw VM talks to
sys-firewall and then sys-net. whonix-gw was kind of designed to *be* the
network facing VM, with it's own iptables firewall (more comprehensive
than Qubes'), protecting whonix-workstation from the net.

(I realize I could get rid of sys-net/sys-firewall, and add the net card
to sys-whonix(gw), to have that kind of configuration, but I'm a big
curious as to why Qubes-Whonix layered it this way. Requiring *four* 300M
VM's just to get the web browser fired up doesn't leave much room for
other VM's. I assume it was just to fit in with minimal disruption to the
standard Qubes configuration, but I was curious if there was other
motivations. I might be asking the wrong people on this list, but I
thought I'd throw it out there.)

- Finally, why does dom0 take up so much memory by default? Does it just
hold onto it to dole out to the other VM's? Most of my VM's are 300M'ish
(unless that pig Firefox is running with a few tabs open, in which case
they shoot up to 900M or more), but dom0 is almost always 1.5G or so.

mem-set'ing it down to 800M doesn't seem to harm things, and lets me start
more VM's. So what's up with that? :)

----
Apologies for the length of this note. If you guys weren't so responsive
and helpful, I wouldn't bother you wish so many questions! (And I'm only
trying to help improve things for everyone.)

Thanks!

JJ