Persistent /usr/local: Are there risks?

[Reg Tiangha]

According to the docs, both /home and /usr/local are persistent in an AppVM:

https://www.qubes-os.org/doc/software-update-vm/

The default PATH in a Qubes VM (Debian 8) looks like this:

user@Email:~$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/user/.local/bin:/home/user/bin

Unless I'm mistaken, it'll search /usr/local/bin first for a binary, and
if it doesn't find it there, it'll go on to the next directory which is
/usr/bin. If it does find it, then the search stops and the /usr/local
version gets executed, correct?

So a silly question: What's stopping an adversary from placing commonly
named but malicious binaries into /usr/local/bin like ls or
qrexec-related binaries? Unless the scripts and programs that call those
programs are hard-coded to use the absolute paths (ex. /bin/dmesg rather
than just dmesg), would there not be a danger that the /usr/local
versions could be invoked instead in *some* situations? Because
/usr/local is persistent, a user may not necessarily know that stuff may
have been placed in that directory, unless they're actively scanning it
for changes, so if they regularly do work in the default bash shell,
they may never know that their 'ls' command or similar may have been
compromised. I don't mind a persistent /home so much since that can be
useful, and I *can* see some use for a persistent /usr/local in some
circumstances, but is there a way to turn the feature off for those who
don't need it and make an AppVM use the TemplateVM's version instead?

[Chris Laprise]

Given the default Qubes security model, its not supposed to matter if
malware can persist. Even the read-only nature of root on template-based
VMs is supposed to be only a beneficial footnote.

OTOH, I'd say your inquiry implies that internal VM security matters to
you to at least some degree (no unguarded sudo, etc.) otherwise there is
nothing to distinguish the /rw/usrlocal persistence issue from
persistence via autostart scripts in /home (which can normally run sudo
without any resistence).

With sudo authentication enabled, the question of persistence becomes
similar to that for a standalone VM... if a malware process can escalate
privs (even via a temporary kernel bug) it can persist in the root-owned
/rw/usrlocal even in a template-based VM.

Maybe a better question is: Should we have the ability to turn off
execution of /rw contents in templates?

--

Chris Laprise, tas...@openmailbox.org
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Reg Tiangha]

On 04/10/2017 11:51 AM, Chris Laprise wrote:
> Given the default Qubes security model, its not supposed to matter if
> malware can persist. Even the read-only nature of root on
> template-based VMs is supposed to be only a beneficial footnote.
>
> OTOH, I'd say your inquiry implies that internal VM security matters
> to you to at least some degree (no unguarded sudo, etc.) otherwise
> there is nothing to distinguish the /rw/usrlocal persistence issue
> from persistence via autostart scripts in /home (which can normally
> run sudo without any resistence).
>
> With sudo authentication enabled, the question of persistence becomes
> similar to that for a standalone VM... if a malware process can
> escalate privs (even via a temporary kernel bug) it can persist in the
> root-owned /rw/usrlocal even in a template-based VM.
>
> Maybe a better question is: Should we have the ability to turn off
> execution of /rw contents in templates?
>

Well, here's a simple for-instance. Say you're a developer of a rival
firm and I want to know all of your secrets so I target you. I know
you're an emacs user, and I know you use Qubes. I also know you're
paranoid enough to segregate all of your development work into one Qubes
VM (yay!), but *not* paranoid enough to cut off that VM from the
Internet (boo!), so not only do you do dev work in it, but you use the
web browser (Chromium, in this case) to interact with websites related
to your work (ex. github) but not to do anything else.

So (somehow, let's say MITM) I find a way to access your file system and
put stuff there. I know that most things get destroyed as soon as the VM
is shutdown, so what I do is I find a way to infect your /usr/local/bin
with malicious versions of Chromium and emacs and ls and all the other
utilities that you'd probably use on a daily basis.

While my versions do exactly what you would expect them to do, the
difference is that each time you launch one of my versions, it starts up
a key logging service (no root required!) in the background that
persists even after you close the app that launched it. So for that
entire session (assuming that AppVM is connected to the Internet), I can
capture all of your keystrokes. And because /usr/local is persistent and
you probably don't constantly check /usr/local for changes (because
again, you're not paranoid), those programs will stick around and launch
the next time you access the VM.

That's just one idea. My imagination isn't as good as others though, so
I'm sure other people can think of nasty things that can be done that
doesn't require privilege escalation. Sure, you can mitigate a lot of
this by cutting off VM access to the network entirely, but that's
because you're smart. The guy next to you may not be as smart (or
paranoid). While there would be some merit to cutting off execution in
/rw areas, for the most part, the way Qubes is set up works well and
that might be going too far. I do think a persistent /usr/local should
be optional, however, and ideally, be disabled by default and only
turned on if needed. There are some cases where one may want to install
custom packages of software that they may have compiled themselves and
would rather put them in /usr/local rather than /usr to help keep the
file system clean. ~/bin isn't much of a problem since a) it's last in
the PATH and b) if you're putting stuff in there, you definitely do so
with the intent to use it only on that AppVM's user account.

[Chris Laprise]

Changing something in /usr/local/bin (or I assume /rw/usrlocal/bin)
requires privilege escalation. If sudo has no auth process, then there
is no challenge for the attacker... they can change /rw/usrlocal in any
case.

But also, they can change /rw/config including rc.local and firewall
scripts which run as root!

BTW, my "better question" above really referred to the non-home areas of
/rw.

>
> While my versions do exactly what you would expect them to do, the
> difference is that each time you launch one of my versions, it starts up
> a key logging service (no root required!) in the background that
> persists even after you close the app that launched it. So for that
> entire session (assuming that AppVM is connected to the Internet), I can
> capture all of your keystrokes. And because /usr/local is persistent and
> you probably don't constantly check /usr/local for changes (because
> again, you're not paranoid), those programs will stick around and launch
> the next time you access the VM.

But this is a problem for things like ~/.bashrc as well. Using PATH= or
alias, attacker could divert you to a phony `git` command that logs your
github password before executing the intended operation.

That's why I suggest people consider enabling sudo auth and securing
shell init scripts in /home (see my post "Protect AppVM init startup
scripts"). You could even have an in-template startup script that resets
most of /rw (root-owner bits) to defaults.... really shouldn't be hard.

In that case, your attack scenario hinges on having a Linux escalation
exploit, and even then it might not last long enough for you to collect
valuable info (e.g. resets or updates occur that patch the vulnerability).

[Reg Tiangha]

On 04/10/2017 12:41 PM, Chris Laprise wrote:
>
> Changing something in /usr/local/bin (or I assume /rw/usrlocal/bin)
> requires privilege escalation. If sudo has no auth process, then there
> is no challenge for the attacker... they can change /rw/usrlocal in
> any case.
>
> But also, they can change /rw/config including rc.local and firewall
> scripts which run as root!
>
> BTW, my "better question" above really referred to the non-home areas
> of /rw.
>

Ah, got it. Well, that's something, I suppose.

>>
>> While my versions do exactly what you would expect them to do, the
>> difference is that each time you launch one of my versions, it starts up
>> a key logging service (no root required!) in the background that
>> persists even after you close the app that launched it. So for that
>> entire session (assuming that AppVM is connected to the Internet), I can
>> capture all of your keystrokes. And because /usr/local is persistent and
>> you probably don't constantly check /usr/local for changes (because
>> again, you're not paranoid), those programs will stick around and launch
>> the next time you access the VM.
>
> But this is a problem for things like ~/.bashrc as well. Using PATH=
> or alias, attacker could divert you to a phony `git` command that logs
> your github password before executing the intended operation.
>
> That's why I suggest people consider enabling sudo auth and securing
> shell init scripts in /home (see my post "Protect AppVM init startup
> scripts"). You could even have an in-template startup script that
> resets most of /rw (root-owner bits) to defaults.... really shouldn't
> be hard.
>
> In that case, your attack scenario hinges on having a Linux escalation
> exploit, and even then it might not last long enough for you to
> collect valuable info (e.g. resets or updates occur that patch the
> vulnerability).
>
>

Still, though, let's say such Linux escalation exploit exists and a
malicious person can access the AppVM's (let's set aside TemplateVMs for
a moment) file system. They can't stick anything in most places in /
because they'll disappear when the VM shuts down. Changes in /home/user
could work, but because users interact with /home on a daily basis
through shells or file browsers, the likelihood of them noticing changes
there might be a bit higher so that might not be the smartest move. But
how many here on this list regularly check their app/sys VMs for
modifications to /usr/local? I'm doubting it's very many, and that I
guess is my main concern.

Some people may not even be aware that it *is* persistent, so people who
use something like Whonix who've been targeted might even have stuff
living in /usr/local right now and may not even know it (since privilege
escalation exploits in tor-browser seem to be found every so often).

I'm definitely going to apply your scripts to my TemplateVMs soon now
that I've been made aware, but I wish there were a way to turn off
persistent /usr/local and to make AppVMs use the TemplateVM's version
instead. I don't use the feature, so I would prefer that /usr/local gets
wiped every time like everything else in the root file system since
that's the behaviour I expected to happen when I first started using
Qubes (I only discovered for myself that it wasn't the case when I was
trying to figure out why a custom-compiled version of Wine that I had
made and installed in my TemplateVM wasn't showing up in my AppVM; its
default prefix is /usr/local, which is why). Is there a way to turn off
persistent /usr/local? Or is it something that's baked-in?

[Chris Laprise]

Some people, maybe. But also notice more than a couple of Qubes users
prefer to use dispVMs for most tasks. That is another way around the
risk, in addition to the way I suggested.

I think I'll try an /etc/rc.local script that deletes /rw/usrlocal and
re-creates just the top dir. Also /rw/config and /rw/bind-dirs. Pretty
much the only persistent thing left would be contents of /rw/home, which
is sort of a middle of the road between fully persistent /rw and using
dispVMs for everything.

>
> I'm definitely going to apply your scripts to my TemplateVMs soon now
> that I've been made aware, but I wish there were a way to turn off
> persistent /usr/local and to make AppVMs use the TemplateVM's version
> instead. I don't use the feature, so I would prefer that /usr/local gets
> wiped every time like everything else in the root file system since
> that's the behaviour I expected to happen when I first started using
> Qubes (I only discovered for myself that it wasn't the case when I was
> trying to figure out why a custom-compiled version of Wine that I had
> made and installed in my TemplateVM wasn't showing up in my AppVM; its
> default prefix is /usr/local, which is why). Is there a way to turn off
> persistent /usr/local? Or is it something that's baked-in?
>
>

[Chris Laprise]

BTW, /usr/local == /rw/usrlocal. Its a symlink.

[cooloutac]

absolutely! great discussion!

[Unman]

And it's set in the template - so if you don't want it open the template,
remove the symlink and move /usr/local.orig to /usr/local.
Then qubes based on that template wont have persistent /usr/local.

NB this will break torVMs and maybe other features of your Qubes.
An alternative approach would be to run tripwire against persistent
directories and monitor changes.

unman

[cooloutac]

Or just compartmentalize more and don't care so much if some of your vms get compromised. Assume some of them will and wipe them when in doubt. But being conscious of the potential issue is a good thing. Tiangha brings up a good point and why to me its silly for some people to care about vms being sudoless.

I stopped using dispvms for important tasks. I only use them for random or dangerous tasks now. And its because of that and other things I don't trust them for important tasks anymore.

I guess only monitoring a couple directories with tripwire on a few important vms would be ok. But on baremetal linux it used to drive me crazy and can be noisy and I would always miss something. Some extra Automatic hardening is always welcome and would be nice. But I'm sure Qubes devs are busy on other things.

[Chris Laprise]

On 04/10/2017 05:54 PM, Unman wrote:
> On Mon, Apr 10, 2017 at 03:39:26PM -0400, Chris Laprise wrote:
>> On 04/10/2017 03:17 PM, Chris Laprise wrote:
>>> On 04/10/2017 02:55 PM, Reg Tiangha wrote:
>>>
>>> I think I'll try an /etc/rc.local script that deletes /rw/usrlocal and
>>> re-creates just the top dir. Also /rw/config and /rw/bind-dirs. Pretty
>>> much the only persistent thing left would be contents of /rw/home, which
>>> is sort of a middle of the road between fully persistent /rw and using
>>> dispVMs for everything.
>>>

> And it's set in the template - so if you don't want it open the template,
> remove the symlink and move /usr/local.orig to /usr/local.
> Then qubes based on that template wont have persistent /usr/local.
>
> NB this will break torVMs and maybe other features of your Qubes.
> An alternative approach would be to run tripwire against persistent
> directories and monitor changes.
>
> unman
>

I think an exception can be carved out for Whonix/tor VMs without too
much trouble. Currently, the script can make exceptions based on
dir/files stored under /etc/default/vms/vm-name. But a more general
exception for this type of VM would be preferable in this case.