Bug#1002738: redis-server: Default systemd unit file system protection settings prevent writing of logfiles, crashing redis

[Johannes Bülow]

Package: redis-server
Version: 5:6.0.15-1
Severity: normal

Dear Maintainer,

I installed openvas on my kali vm, but redis broke, so I tried to
install redis-server on a clean debian instance, with the same result.

If I disable System Protection and comment out all Read/Write stuff in
the systemd service unit file, redis
boots up just fine. I therefore suspect the issue being with the write
restrictions in the default Systemd unit file for redis-server.

The message in my syslog:

Dec 28 15:19:05 debian redis-server[521]: *** FATAL CONFIG FILE ERROR
(Redis 6.0.15) ***
Dec 28 15:19:05 debian redis-server[521]: Reading the configuration
file, at line 260
Dec 28 15:19:05 debian redis-server[521]: >>> 'logfile
/var/log/redis/redis-server.log'
Dec 28 15:19:05 debian redis-server[521]: Can't open the log file:
Read-only file system
Dec 28 15:19:05 debian systemd[1]: redis-server.service: Main process
exited, code=exited, status=1/FAILURE

-- System Information:
Debian Release: 11.2
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.15.8-1-default (SMP w/8 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=locale: Cannot
set LC_ALL to default locale: No such file or directory
UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages redis-server depends on:
ii init-system-helpers 1.60
ii lsb-base 11.1.0
ii redis-tools 5:6.0.15-1

redis-server recommends no packages.

redis-server suggests no packages.

-- Configuration Files:
/etc/redis/redis.conf [Errno 13] Permission denied:
'/etc/redis/redis.conf'

-- debconf-show failed
Report will be sent to Debian Bug Tracking System
<sub...@bugs.debian.org>

[Chris Lamb]

Hi Johannes,

> I therefore suspect the issue being with the write
> restrictions in the default Systemd unit file for redis-server.

It sounds like that to me too. However:

> Dec 28 15:19:05 debian redis-server[521]: >>> 'logfile/var/log/redis/redis-server.log'

> Dec 28 15:19:05 debian redis-server[521]: Can't open the log file: Read-only file system

That's interesting, because there is a specific allowance made so that
Redis can write to its own logfiles. Specifically:

ReadWriteDirectories=-/var/log/redis

Ah, perhaps your version of systemd is newer? Can you reset or
otherwise reinstate the "ReadWriteDirectories" lines along with the
rest of the file and replace the start with "ReadWritePaths=" and let
me know how you get on?


Regards,

--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org 🍥 chris-lamb.co.uk
`-

[Johannes Bülow]

On Tue, 2021-12-28 at 17:13 +0000, Chris Lamb wrote:
> That's interesting, because there is a specific allowance made so
> that
> Redis can write to its own logfiles. Specifically:
>
>   ReadWriteDirectories=-/var/log/redis

On my systems they are already ReadWritePaths, not ReadWriteDirectories
(except for /etc/redis and / )

> Ah, perhaps your version of systemd is newer? 

I am running systemd 247.3-6 on the affected systems, but Kernel
5.15.8-1-default. On Kernel 5.14 and older it seems to work fine.

> Can you reset or
> otherwise reinstate the "ReadWriteDirectories" lines along with the
> rest of the file and replace the start with "ReadWritePaths=" and let
> me know how you get on?

That sadly didn't help much, but setting ProtectSystem to "strict",
following the systemd.exec manpage, and removing
"ReadOnlyDirectories=/" solved the problem for me.
My only guess is that it's some issue with (Kernel) namespaces either
on my System specifically or with Kernel 5.15 in general.

ProtectSystem=
Takes a boolean argument or the special values "full" or "strict". If true, mounts the /usr/ and the boot loader directories (/boot and /efi) read-only for processes invoked by this unit. If set to "full", the /etc/
directory is mounted read-only, too. If set to "strict" the entire file system hierarchy is mounted read-only, except for the API file system subtrees /dev/, /proc/ and /sys/ (protect these directories using
PrivateDevices=, ProtectKernelTunables=, ProtectControlGroups=). This setting ensures that any modification of the vendor-supplied operating system (and optionally its configuration, and local mounts) is prohibited for
the service. It is recommended to enable this setting for all long-running services, unless they are involved with system updates or need to modify the operating system in other ways. If this option is used,
ReadWritePaths= may be used to exclude specific directories from being made read-only. This setting is implied if DynamicUser= is set. This setting cannot ensure protection in all cases. In general it has the same
limitations as ReadOnlyPaths=, see below. Defaults to off.




Regards,

Johannes Bülow

[Chris Lamb]

Hi,

>> Ah, perhaps your version of systemd is newer? 
> I am running systemd 247.3-6 on the affected systems, but Kernel
> 5.15.8-1-default. On Kernel 5.14 and older it seems to work fine.

[..]

> My only guess is that it's some issue with (Kernel) namespaces either
> on my System specifically or with Kernel 5.15 in general.

Hm, which makes me think the bug is elsewhere and not in the Redis
systemd unit. Unless I'm missing something, the configuration
directives used here should 'just work'. Or rather: you are likely to
encounter (or are already encountering!) other breakages on your system
if this is not working.

Thoughts?

Chris

ps. I assume you are using bullseye? (redis 6.0.15-1 is the version
available there)

[Johannes Bülow]

Hello,

These Problems could theoretically come from other settings, yes.

I did use Bullseye for testing this, I don't have any other Debian
systems currently (if you want, I can test it in bookworm or buster
through).

If you want, you can close this Bug Report if you feel this is a weird
edge-case, I can still reopen it if investigation downstream comes to a
different conclusion.

My reasoning for opening a Bug Report for this was because I had this
issue on multiple Debian LXC containers and VMs, but on my SuSE and
Rocky systems I didn't encounter this issue, and they do have different
unit files. When compairing the files for redis in Debian and SuSE, I
found those differences.

TLDR, if you want, feel free to close this ticket, I'll reopen it if
something changes downstream.

Greetings,

Johannes

[Chris Lamb]

Hi Johannes,

> TLDR, if you want, feel free to close this ticket, I'll reopen it if
> something changes downstream.

Thanks for your mail. I'm happy to keep this bug open in case
something comes up, but I'm not sure what I would do if we could
definitively demonstrate a bug in Redis' unit file.

That is to say, the only solution (whatever the cause might turn out
to be) would seem to be to remove all of the hardening features from
the unit file... hardly the right solution here. :)

[Johannes Bülow]

Hello Chris,

Downstream didn't really do anything so far on this topic.
I also can only reproduce this issue on Kernel 5.15 and higher.

I don't know whether setting / as ReadOnly directly or using
ProtectSystem=strict along with PrivateDevices=true,
ProtectKernelTunables=true and ProtectControlGroups=true is the more
secure configuration.

as for reproducing, I can reproduce it pretty consistently on LXC
containers on new kernels, but not on 5.10 or 5.12.

This is probably going to lead to a dead end

Best Regards,

Johannes