cannot remove '/etc/resolv.conf': Read-only file system

[Marc Villacorta Morera]

Hello there!

BACKGROUND:

I am using a confd container to dynamically update the contents of '/etc/resolv.conf'.

I do this because I have 3 containerized Mesos DNS services scheduled by Fleet and I don't know their IP address beforehand.

Whenever a Mesos DNS service starts, it registers itself to Etcd which triggers confd to regenerate /etc/resolv.conf in every CoreOS host in the fleet.

At least this is the plan (suggestion are very welcome).

THE PROBLEM:

In CoreOS the file /etc/resolv.conf is a symlink to a file generated by 'systemd-resolved' every time it starts.

Due to the way 'systemd-resolved' operates I have decided to create the following drop-in:

core@core-1 ~ $ cat /etc/systemd/system/systemd-resolved.service.d/50-link-resolv.conf

[Service]

ExecStartPre=-/usr/bin/ln -fs ../run/systemd/resolve/resolv.conf /etc/resolv.conf

ExecStartPost=-/usr/bin/ln -fs resolv.etcd /etc/resolv.conf

And this is the problem I see in the logs:

Dec 05 12:41:54 core-1.cell-1.dc-1.demo.lan systemd[1]: Stopping Network Name Resolution...

Dec 05 12:41:54 core-1.cell-1.dc-1.demo.lan systemd[1]: Stopped Network Name Resolution.

Dec 05 12:41:54 core-1.cell-1.dc-1.demo.lan systemd[1]: Starting Network Name Resolution...

Dec 05 12:41:54 core-1.cell-1.dc-1.demo.lan ln[21963]: /usr/bin/ln: cannot remove '/etc/resolv.conf': Read-only file system

Dec 05 12:41:54 core-1.cell-1.dc-1.demo.lan systemd-resolved[21966]: Using system hostname 'core-1'.

Dec 05 12:41:54 core-1.cell-1.dc-1.demo.lan ln[21967]: /usr/bin/ln: cannot remove '/etc/resolv.conf': Read-only file system

Dec 05 12:41:54 core-1.cell-1.dc-1.demo.lan systemd[1]: Started Network Name Resolution.

The weird thing is that I can do that in the shell:

core@core-1 ~ $ sudo /usr/bin/ln -fs ../run/systemd/resolve/resolv.conf /etc/resolv.conf; echo $?

0

core@core-1 ~ $ sudo /usr/bin/ln -fs resolv.etcd /etc/resolv.conf; echo $?

0

And / is mounted rw:

/dev/sda9 on / type ext4 (rw,relatime,seclabel,data=ordered)

Any idea? Thank you!

Marc

[Marc Villacorta Morera]

Ok. Thanks to @Ulexus who helped me to figure this out.

Here I post the solution for anyone who might run into the same problem.

The key parameter to modify is ProtectSystem=full.

The final systemd unit looks like this:

core@core-1 ~ $ systemctl cat systemd-resolved

# /usr/lib64/systemd/system/systemd-resolved.service

#  This file is part of systemd.

#

#  systemd is free software; you can redistribute it and/or modify it

#  under the terms of the GNU Lesser General Public License as published by

#  the Free Software Foundation; either version 2.1 of the License, or

#  (at your option) any later version.

[Unit]

Description=Network Name Resolution

Documentation=man:systemd-resolved.service(8)

After=systemd-networkd.service network.target

# On kdbus systems we pull in the busname explicitly, because it

# carries policy that allows the daemon to acquire its name.

Wants=org.freedesktop.resolve1.busname

After=org.freedesktop.resolve1.busname

[Service]

Type=notify

Restart=always

RestartSec=0

ExecStart=/usr/lib/systemd/systemd-resolved

CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_SETPCAP CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER

ProtectSystem=full

ProtectHome=yes

WatchdogSec=1min

[Install]

WantedBy=multi-user.target

# /etc/systemd/system/systemd-resolved.service.d/50-link-resolv.conf

[Service]

ProtectSystem=true

ExecStartPre=-/usr/bin/ln -fs ../run/systemd/resolve/resolv.conf /etc/resolv.conf

ExecStartPost=-/usr/bin/bash -c "[ -f /etc/resolv.etcd ] && ln -fs resolv.etcd /etc/resolv.conf"

[Paul Morgan]

On Sat, Dec 5, 2015 at 1:09 PM, Marc Villacorta Morera
<marc.vi...@gmail.com> wrote:
> Ok. Thanks to @Ulexus who helped me to figure this out.
> Here I post the solution for anyone who might run into the same problem.
> The key parameter to modify is ProtectSystem=full.

> -snip-

That's an interesting approach. Thanks for sharing!

We chose a different approach (not for mesos; we don't use it).
Our approach is to create /etc/systemd/resolved.conf with content according to
http://www.freedesktop.org/software/systemd/man/resolved.conf.html

This allowed us to achieve our objective while staying as close to
"out-of-box" as possible.

-paul