Scary Systemd Security Report

[ron...@riseup.net]

I've been reading a blog from the renowned Daniel Aleksandersen at
https://www.ctrl.blog/entry/systemd-service-hardening.html

The output from a Debian-10 based Appvm looks a little scary!! Should I
be concerned?

user@tmp3:~$ systemd-analyze security
UNIT EXPOSURE PREDICATE HAPPY
ModemManager.service 5.6 MEDIUM 😐
NetworkManager.service 7.6 EXPOSED 🙁
avahi-daemon.service 9.5 UNSAFE 😨
cron.service 9.5 UNSAFE 😨
cups-browsed.service 9.5 UNSAFE 😨
cups.service 9.5 UNSAFE 😨
dbus.service 9.5 UNSAFE 😨
dm-event.service 9.5 UNSAFE 😨
emergency.service 9.5 UNSAFE 😨
exim4.service 9.5 UNSAFE 😨
ge...@tty1.service 9.5 UNSAFE 😨
haveged.service 5.6 MEDIUM 😐
lvm2-lvmpolld.service 9.5 UNSAFE 😨
polkit.service 9.5 UNSAFE 😨
qubes-db.service 9.5 UNSAFE 😨
qubes-firewall.service 9.5 UNSAFE 😨
qubes-gui-agent.service 9.5 UNSAFE 😨
qubes-meminfo-writer.service 9.5 UNSAFE 😨
qubes-qrexec-agent.service 9.5 UNSAFE 😨
qubes-sync-time.service 9.5 UNSAFE 😨
qubes-updates-proxy.service 9.5 UNSAFE 😨
rc-local.service 9.5 UNSAFE 😨

rescue.service 9.5 UNSAFE 😨
rsyslog.service 9.5 UNSAFE 😨
rtkit-daemon.service 6.9 MEDIUM 😐
serial...@hvc0.service 9.5 UNSAFE 😨
systemd-ask-password-console.service 9.3 UNSAFE 😨
systemd-ask-password-wall.service 9.3 UNSAFE 😨
systemd-fsckd.service 9.5 UNSAFE 😨
systemd-initctl.service 9.3 UNSAFE 😨
systemd-journald.service 4.3 OK 🙂
systemd-logind.service 4.1 OK 🙂
systemd-networkd.service 2.8 OK 🙂
systemd-timesyncd.service 2.0 OK 🙂
systemd-udevd.service 8.3 EXPOSED 🙁
tinyproxy.service 8.7 EXPOSED 🙁
udisks2.service 9.5 UNSAFE 😨
us...@1000.service 9.1 UNSAFE 😨
wpa_supplicant.service 9.5 UNSAFE 😨
xendriverdomain.service 9.5 UNSAFE 😨

[unman]

On Tue, Feb 11, 2020 at 01:34:15AM -0800, ron...@riseup.net wrote:
> I've been reading a blog from the renowned Daniel Aleksandersen at
> https://www.ctrl.blog/entry/systemd-service-hardening.html
>
> The output from a Debian-10 based Appvm looks a little scary!! Should I
> be concerned?
>
> user@tmp3:~$ systemd-analyze security
> UNIT EXPOSURE PREDICATE HAPPY

> ModemManager.service 5.6 MEDIUM ????
> NetworkManager.service 7.6 EXPOSED ????
> avahi-daemon.service 9.5 UNSAFE ????
> cron.service 9.5 UNSAFE ????
> cups-browsed.service 9.5 UNSAFE ????
> cups.service 9.5 UNSAFE ????
> dbus.service 9.5 UNSAFE ????
> dm-event.service 9.5 UNSAFE ????
> emergency.service 9.5 UNSAFE ????
> exim4.service 9.5 UNSAFE ????
> ge...@tty1.service 9.5 UNSAFE ????
> haveged.service 5.6 MEDIUM ????
> lvm2-lvmpolld.service 9.5 UNSAFE ????
> polkit.service 9.5 UNSAFE ????
> qubes-db.service 9.5 UNSAFE ????
> qubes-firewall.service 9.5 UNSAFE ????
> qubes-gui-agent.service 9.5 UNSAFE ????
> qubes-meminfo-writer.service 9.5 UNSAFE ????
> qubes-qrexec-agent.service 9.5 UNSAFE ????
> qubes-sync-time.service 9.5 UNSAFE ????
> qubes-updates-proxy.service 9.5 UNSAFE ????
> rc-local.service 9.5 UNSAFE ????
>
> rescue.service 9.5 UNSAFE ????
> rsyslog.service 9.5 UNSAFE ????
> rtkit-daemon.service 6.9 MEDIUM ????
> serial...@hvc0.service 9.5 UNSAFE ????
> systemd-ask-password-console.service 9.3 UNSAFE ????
> systemd-ask-password-wall.service 9.3 UNSAFE ????
> systemd-fsckd.service 9.5 UNSAFE ????
> systemd-initctl.service 9.3 UNSAFE ????
> systemd-journald.service 4.3 OK ????
> systemd-logind.service 4.1 OK ????
> systemd-networkd.service 2.8 OK ????
> systemd-timesyncd.service 2.0 OK ????
> systemd-udevd.service 8.3 EXPOSED ????
> tinyproxy.service 8.7 EXPOSED ????
> udisks2.service 9.5 UNSAFE ????
> us...@1000.service 9.1 UNSAFE ????
> wpa_supplicant.service 9.5 UNSAFE ????
> xendriverdomain.service 9.5 UNSAFE ????
>

It does look scary.
The output from a Fedora based qube looks much the same..
You should run the analysis against each service and see where you think
they could be hardened. Post back your conclusions here.
Also, I see that you have many services that need not be there - some
of these will be disabled by Qubes- some you do not need in every qube
(cups-browsed, exim4, tinyproxy etc).
You need to review what services you are running, and disable those you
do not want. My list in an ordinary qube looks rather different from
yours. Those are steps you should be taking in any case.
Also, bear in mind that the analysis doesn't take in to account any
security features in the programs themselves, or other mitigations.
So you need to do a good deal more work before reaching any conclusions
about your system.
Look forward to hearing from you
unman

[ron...@riseup.net]

As I read it, your suggesting that the output is influence by User
preferences as opposed to default system settings? To test that theory,
I loaded a vanilla version of Qubes 4.0.3 onto a spare box and ran the
command systemd-analyze security against the virgin Debian-10 Template.
The output is identical to the one I originally posted. As you inferred,
the output from Fedora Template is similar.

I'm not sure if you'll agree, but my conclusion from this experiment is
that the Qubes Team have some work to do in hardening Qubes? Like you
say,"I see that you have many services that need not be there"; so my
question is, why are they present in a vanilla version of Qubes?

[unman]

On Tue, Feb 11, 2020 at 10:09:38PM -0800, ron...@riseup.net wrote:
> On 2020-02-11 11:39, unman wrote:
> > On Tue, Feb 11, 2020 at 01:34:15AM -0800, ron...@riseup.net wrote:
> >> I've been reading a blog from the renowned Daniel Aleksandersen at
> >> https://www.ctrl.blog/entry/systemd-service-hardening.html
> >>
> >> The output from a Debian-10 based Appvm looks a little scary!! Should I
> >> be concerned?
> >>
> >> user@tmp3:~$ systemd-analyze security
> >> UNIT EXPOSURE PREDICATE HAPPY
> >> ModemManager.service 5.6 MEDIUM ????

<snip>

The vanilla templates serve all sorts of purposes, and they are
(generally) configured so that you can just drop them in to act as
sys-net, sys-usb etc.
So there are a number of things that you probably don't want or need in
an ordinary qube.
This is a trade off to get maximum usability. There are many
alternatives - using minimal templates; building your own; customising
services on a per qube basis. But for ordinary users Qubes makes it as
simple as possible to use the default templates.

On the general issue, you're missing my point.
Before you can decide whether you should be worried I'm suggesting you
need to do some work. That's exactly what that blog post is about.
Take one of those services, run an analysis and look at the results.
Determine what's needed and what isn't.
Look at any mitigations in the programs themselves, or generally within
Qubes.
Post your conclusions here, or in qubes-issues.
You *are* a member of the Qubes Team.
We all have work to do in hardening Qubes

unman

[Claudia]

I'm curious how this compares to a vanilla (non-Qubes) Fedora or debian install. In general most packages' default service files use very few if any systemd security features on most distros. I think that's more of a DIY thing.

> I'm not sure if you'll agree, but my conclusion from this experiment is
> that the Qubes Team have some work to do in hardening Qubes? Like you
> say,"I see that you have many services that need not be there"; so my
> question is, why are they present in a vanilla version of Qubes?
>

My impression of the official Qubes developers' stance on this is "security by isolation," i.e. Xen is the only component they actually consider secure. This is the rationale for passwordless sudo for example. In practice, I can agree, it's difficult enough to develop and maintain an OS as sophisticated as Qubes in the first place, let alone if they had to also harden guest OSes at various levels. In principle, I say fair enough, I suppose it's not really Qubes' concern what goes on within VMs. Qubes just polices the border.

You might be interested in Chris's Qubes hardening tools, however I don't know it uses the systemd security features at all so it may not improve systemd's report.

[Bernhard]

<snip>

> Also, I see that you have many services that need not be there - some
> of these will be disabled by Qubes- some you do not need in every qube
> (cups-browsed, exim4, tinyproxy etc).

how do get rid of them? exim for example looks to me like a virus. I
found no way to uninstall it without destroying debian ... the trick is
maybe to keep them, but disabled? Cheers, Bernhard

[unman]

I could be wrong but I don't think that tool takes any notice of the
*state* of the service, which is why I suggested that op should actually
dig in to the results.
Many of these services can be disabled - I prefer to mask or remove them
all together.
A reasonable alternative would be to start with a micro template and
only add the services/tools that you need.

As far as exim is concerned you should be able to remove it quite
easily.
exim is "recommended" by cron, but can be removed without breaking the
system. If you use a tool like aptitude it becomes easy to see and
overcome dependencies when removing packages.

[Chris Laprise]

On 2/12/20 7:27 AM, Claudia wrote:
>> I'm not sure if you'll agree, but my conclusion from this experiment is
>> that the Qubes Team have some work to do in hardening Qubes? Like you
>> say,"I see that you have many services that need not be there"; so my
>> question is, why are they present in a vanilla version of Qubes?
>>
>
> My impression of the official Qubes developers' stance on this is "security by isolation," i.e. Xen is the only component they actually consider secure. This is the rationale for passwordless sudo for example. In practice, I can agree, it's difficult enough to develop and maintain an OS as sophisticated as Qubes in the first place, let alone if they had to also harden guest OSes at various levels. In principle, I say fair enough, I suppose it's not really Qubes' concern what goes on within VMs. Qubes just polices the border.

It does present an interesting angle for hardening (there *always* is
another one, isn't there?).

> You might be interested in Chris's Qubes hardening tools, however I don't know it uses the systemd security features at all so it may not improve systemd's report.

Qubes-VM-hardening probably wouldn't improve the report. The former is
mainly about restoring the guest's normal permissions-based security,
and helping ensure the startup state is uncompromised.

The analysis appears to be a measurement of a service's level of
sandboxing, according the the man page. It seems to look for
capabilities management of some kind(s). An example it gives is that a
service with the ability to mount/unmount volumes may be labeled UNSAFE.
This would imply that most of a system's services will never attain an
OK rating. So I think we're looking at another one of systemd's immature
pilots. It may even be a tool for scaring gratis CentOS/Fedora users
into purchasing RHEL (yes, my usual uncharitable assessment of Red Hat),
since systemd originates from Fedora/RHEL.

When I see stuff like this, I also ask whether the authors make any
distinctions about things like 'guardian' components... Does a
crypto-based verification tool or something doing little more than toss
data blocks from one port to another deserve the same steep (even
hyperbolic) grade scale that, say, CUPS or something even more complex
and less security-minded gets?

--
Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Chris Laprise]

That's odd. I use a regular debian-10 template for most things and
exim4* removal only takes out 2 other exim packages.

Bernhard should look into that; it would be great if this discussion
prompted the detection and removal of an actual malware.

[David Hobach]

On 2/14/20 4:01 AM, Chris Laprise wrote:
> That's odd. I use a regular debian-10 template for most things and
> exim4* removal only takes out 2 other exim packages.

Yes, they apparently put some effort into removing useless dependencies
between debian 9 and 10.

E.g. gnome-keyring can also be removed now. :-)

[Steve Coleman]

On 2020-02-12 01:09, ron...@riseup.net wrote:
> APL external email warning: Verify sender qubes-users+bncBCI3H2V5...@googlegroups.com before clicking links or attachments

I ran the report on my fedora-30, but then scripted up a test to see
what services listed in this report were actually running.

atd.service - Deferred execution scheduler
cups.service - CUPS Scheduler
ge...@tty1.service - Getty on tty1
haveged.service - Entropy Daemon based on the HAVEGE algorithm
polkit.service - Authorization Manager
qubes-db.service - Qubes DB agent
qubes-gui-agent.service - Qubes GUI Agent
qubes-meminfo-writer.service - Qubes memory information reporter
qubes-qrexec-agent.service - Qubes remote exec agent
serial...@hvc0.service - Serial Getty on hvc0
us...@1000.service - User Manager for UID 1000
xendriverdomain.service - Xen driver domain device daemon

This list does not look so scary to me, since these services actually
serve a purpose for which they were written. My question is why are the
required services being marked as "UNSAFE" and how to improve the
ratings on the required services. Without knowing why they received such
a bad rating under this utility seems to be the much more the important
question.

A service that is not running does not pose much of an attack surface
unless the adversary can launch that service. If the adversary can
launch that service you have much *bigger* problem than the service
running. At that point you are already p0ned.

What I would choose to do, if I were that worried about non-running
services, is to instrument them so that I get notified if one service
somehow gets launched. Kind of like a hackers honey pot with notification.

I will be running this utility on each individual running service to see
what I can do to improve upon their configuration, but I am not going to
loose much sleep over the ones that are not even configured to run.

[ron...@riseup.net]

Thanks all for taking time out to respond to this issue.

I have to say I'm still confused as to whether its a "scary" issue or
just a bug in the tool "systemd-analyze security".

I spotted this from Whonix
https://forums.whonix.org/t/using-apparmor-profile-everything-on-debian-buster/8650
- which if I'm not mistaken, claims to utilise a tool;
apparmor-profile-everything, to confine, amongst other things, the
systemd init process and and children it spawns. I thought I'd give it a
try and see if it gave less scary results! Here's the feedback:

Although Apparmor security feature is enabled by default in Debian
Buster (10), it is not, for some inexplicable reason, enabled by default
in the Qubes version of Debian-10. To enable it, issue the command in
Dom0; qvm-prefs -s <template name> "nopat apparmor=1 security=apparmor".
Then follow the link above to install apparmor-profile-everything. Also
install apparmor-utils.

Check if apparmor is running ok:
user@sysemd-test:~$ sudo aa-status
apparmor module is loaded.
18 profiles are loaded.
18 profiles are in enforce mode.
/**/*-browser/Browser/firefox
/usr/bin/apt-get
/usr/bin/evince
/usr/bin/evince-previewer
/usr/bin/evince-previewer//sanitized_helper
/usr/bin/evince-thumbnailer
/usr/bin/evince//sanitized_helper
/usr/bin/man
/usr/lib/cups/backend/cups-pdf
/usr/sbin/cups-browsed
/usr/sbin/cupsd
/usr/sbin/cupsd//third_party
/usr/sbin/haveged
init-systemd
man_filter
man_groff
nvidia_modprobe
nvidia_modprobe//kmod
0 profiles are in complain mode.
3 processes have profiles defined.
3 processes are in enforce mode.
/usr/sbin/cups-browsed (551)
/usr/sbin/cupsd (488)
/usr/sbin/haveged (481)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.


We see that the systemd init is loaded and in enforce mode!


However The output from the tool systemd-analyze security still gives
scary results. Is this what you professional developers would expect?

user@sysemd-test:~$ systemd-analyze security

[AJ Jordan]

tl;dr: if you don't care about the example and just want to know how
the heck to interpret this tool, read the first couple paragraphs, and
then skip to the last paragraph or two.

I think people have a deep misunderstanding of what `systemd-analyze
security` does, and in particular are expecting it to be much more
sophisticated than it is. Per systemd-analyze(1):

> Note that this only analyzes the per-service security features
> systemd itself implements. This means that any additional security
> mechanisms applied by the service code itself are not accounted
> for. The exposure level determined this way should not be
> misunderstood: a high exposure level neither means that there is no
> effective sandboxing applied by the service code itself, nor that
> the service is actually vulnerable to remote or local attacks.

The key takeaway from this is that the tool only considers
security-related systemd service file directives (see
systemd.exec(5)). It does NOT consider *anything* else like:

* AppArmor or SELinux confinement

* Service architecture (for example, whether it is composed of
several mutually-distrusting daemons)

* Service complexity

* Whether the service does anything to confine itself, like dropping
privileges after starting up as root

* Service vulnerability mitigations (for example, being written in a
memory-safe language, or being compiled with
-fstack-protector-strong or something)

and etc.

I case you're not familiar with the directives `systemd-analyze
security` looks at, basically what you need to know is that they can
be turned on in a service's definition and systemd will automatically
restrict the ability of the service's processes to do various things,
like reading other services' temporary files, or mucking with kernel
tunables. (These restrictions are imposed *on top of* any existing
restrictions. For example if the service is not run as root it already
can't muck with kernel tunables, so that systemd restriction does
nothing unless the service either runs as root or can escalate
privileges to root).

As an example of how these work, take the default systemd service file
for Apache httpd from my production Debian 10 machine:

```
% systemctl cat apache2.service
# /lib/systemd/system/apache2.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
Documentation=https://httpd.apache.org/docs/2.4/

[Service]
Type=forking
Environment=APACHE_STARTED_BY_SYSTEMD=true
ExecStart=/usr/sbin/apachectl start
ExecStop=/usr/sbin/apachectl stop
ExecReload=/usr/sbin/apachectl graceful
PrivateTmp=true
Restart=on-abort

[Install]
WantedBy=multi-user.target
```

If I run `systemd-analyze security apache2.service`, I get a long
detailed listing of things I could add to this unit file to improve
the sandboxing that systemd applies to the service (I'm not including
an example here because it's so long, but you can run this on any
service to get something similar). The score given is 9.2 UNSAFE

Now, Apache has absolutely no business writing to users' home
directories, so let's say I want systemd to enforce that. (Normally it
wouldn't have any business reading from home directories either except
that on my system I have it set up to serve `/home/$USER/public_html`
directories.) Apache also has no business writing to /usr, /boot,
/etc, /sys, /proc. Moreover it does not need most of the devices in
/dev. It would be nice, therefore, if it was not allowed to do these
things if it ever gets compromised.

So, what I can do is tell systemd to restrict these things. I create a
systemd drop-in file (maybe with `systemctl edit apache2.service`)
with the following content:

```
[Service]

ProtectSystem=full
ProtectHome=read-only
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
```

Now if I run `systemd-analyze security apache2.service` again, the
reported score is 7.9 EXPOSED. So it went down 1.3 points because I
added those sandboxing options, which as systemd-analyze(8) mentions,
are the *only* things that systemd considers.

Note in particular that systemd-analyze does *not* know or care that
Apache drops privileges when it starts up and thus can't write to /usr
et. al. anyway. So these options are useful for defense in depth
hardening, but the only time they'll matter is if Apache is
compromised, and *then* the attacker finds a way to escalate to root
privileges. At that point the attacker will not be able to write to
these directories despite having root.

It's probably a good idea to turn these options on for lots and lots
of services, because they're *so* easy to just put in the service file
and they provide defense in depth. But `systemd-analyze security`'s
warnings are much too scary. Instead of UNSAFE a better label honestly
would be NEEDS A LOT OF WORK or something like that. Think of the tool
as treating the service's code like a black box: it doesn't know
anything about what goes on inside the service (in fact in the general
case it is mathematically *impossible* for it to do so; see Rice's
theorem[1]) so it works with what it knows, which is declarative
systemd configuration. Another way to think about the scores are as an
upper bound on danger, enforced by systemd/the Linux kernel.

Hopefully this gives a sense of how better to interpret these
warnings.

-AJ

[1]: https://en.wikipedia.org/wiki/Rice%27s_theorem

[ron...@riseup.net]

Thank you for your comprehensive and really helpful reply.

Re: "It's probably a good idea to turn these options on for lots and

lots of services, because they're *so* easy to just put in the service
file and they provide defense in depth."

If as you say, it's so easy to provide strength in depth I wonder why
Qubes, Debian and the other distros don't turn them on by default?