Bug#1009351: systemd: Static IP addressing (from config) in LXC containers not working anymore

[Michael Biebl]

Control: tags -1 + unreproducible

Am 12.04.22 um 11:26 schrieb Michael Biebl:
> Control: severity -1 important
>
> Am 12.04.22 um 09:15 schrieb Claudio Kuenzler:
>> Package: systemd
>> Version: 247.3-7
>> Severity: critical
>> Justification: breaks the whole system
>>
>> Dear Maintainer,
>>
>> After having created a new LXC container (using lxc-create) using the
>> lxc-download template and using Debian Bullseye, the static IP
>> addressing using the container's config file
>> (/var/lib/lxc/container/config) is not working anymore.
>>
>> This was the first Bullseye LXC I created since the 11.3 release. Older
>> Bullseye releases worked without a problem.
>>
>> This _could_ be caused by the following mentioned change in the systemd
>> package:
>>
>> "fix a regression when using systemd-networkd in an unprivileged LXD
>> container"
>
> Can you verify if reverting the patch helps?
>
>
>> A workaround is to start the containenr in foreground (-F), disable the
>> systemd-networkd service and then reboot the container. The static IP
>> from the container's config is now working again.
>>
>> To reproduce:
>>
>> 1. Create the container
>>
>> lxc-create -n bullseye -t download -- -d debian -r bullseye -a amd64
>>
>> 2. Adjust config, add static IP in networking
>>
>> root@host:~# cat /var/lib/lxc/bullseye/config | grep net
>> lxc.net.0.type = veth
>> lxc.net.0.flags = up
>> lxc.net.0.link = virbr1
>> lxc.net.0.veth.pair = bullseye
>> lxc.net.0.ipv4.address = 192.168.1.111/24
>> lxc.net.0.ipv4.gateway = 192.168.1.1
>>


Hm, works fine here, i.e. I can't reproduce your problem.
I'm running LXC on a Debian sid host though.
Maybe it's something related to your LXC configuration/installation?

[Michael Biebl]

> 3. Start container
>
> lxc-start -n bullseye
>
> For a very short time (1-2 seconds), the bullseye container is reachable
> on the configured IP but then loses the IP as soon as systemd-networkd
> is started.
>
> 4. Attach to container, disable systemd-networkd, reboot -> works again
>

If you are using a static IP it's probably a good idea to disable
networkd anyway.

[Claudio Kuenzler]

On Tue, Apr 12, 2022 at 11:33 AM Michael Biebl <bi...@debian.org> wrote:

Hm, works fine here, i.e. I can't reproduce your problem.
I'm running LXC on a Debian sid host though.
Maybe it's something related to your LXC configuration/installation?

Hi Michael

LXC host runs on Debian Bullseye.

ck@host:~$ dpkg -l|egrep "(systemd|lxc)" | awk '{print $2" "$3}'
dbus-user-session 1.12.20-2
liblxc1:amd64 1:4.0.6-2
libnss-systemd:amd64 247.3-6
libpam-systemd:amd64 247.3-6
libsystemd0:amd64 247.3-6
libvirt-daemon-system-systemd 7.0.0-3
lxc 1:4.0.6-2
lxcfs 4.0.7-1
systemd 247.3-6
systemd-container 247.3-6
systemd-sysv 247.3-6
systemd-timesyncd 247.3-6

 

I actually see a major difference when comparing a Debian 11.2 vs. a 11.3 container.

-----------------------------------------------------------

Inside 11.2 LXC:

root@112:~# cat /etc/debian_version
11.2

ck@112:~$ systemctl list-units|grep network
  networking.service                  loaded active exited    Raise network interfaces
  network-online.target               loaded active active    Network is Online
  network.target                      loaded active active    Network

Inside 11.3 LXC:

root@113test:~# cat /etc/debian_version
11.3

root@11test:~# systemctl list-units|grep network
  systemd-networkd.service            loaded active running   Network Service
  systemd-networkd.socket             loaded active running   Network Service Netlink Socket
  network.target                      loaded active active    Network

-----------------------------------------------------------

Note the different network units.

Both LXC containers have been setup with the same lxc-download template. Actually a locally modified lxc-download template for our infra so there was definitely no change inside the template.

Not sure where this change comes from though.

[Michael Biebl]

Am 12.04.22 um 12:01 schrieb Claudio Kuenzler:

Can you attach the output of
systemctl status systemd-networkd.service
journalctl -ulb systemd-networkd.service

for both containers.

If systemd-networkd wasn't running in your old, 11.2 based container, it
couldn't interfere with your custom network setup.

This is btw I recommended to disable systemd-networkd as the correct
solution for your issue.

I'm not actually sure if there is anything to fix here.
You simply have two conflicting networking setups.

[Claudio Kuenzler]

Can you attach the output of
systemctl status systemd-networkd.service
journalctl -ulb systemd-networkd.service

for both containers.

Older 11.2 container:

root@11-2:~# systemctl status systemd-networkd.service
Warning: The unit file, source configuration file or drop-ins of systemd-networkd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● systemd-networkd.service
     Loaded: masked (Reason: Unit systemd-networkd.service is masked.)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead)

root@11-2:~# journalctl -ulb systemd-networkd.service
Failed to add match 'systemd-networkd.service': Invalid argument

Newly created 11.3 container (11test2):

root@11test2:~#  systemctl status systemd-networkd.service
● systemd-networkd.service - Network Service
     Loaded: loaded (/lib/systemd/system/systemd-networkd.service; disabled; vendor preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (running) since Tue 2022-04-12 13:56:00 UTC; 34s ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
   Main PID: 85 (systemd-network)
     Status: "Processing requests..."
      Tasks: 1 (limit: 619000)
     Memory: 4.0M
        CPU: 38ms
     CGroup: /system.slice/systemd-networkd.service
             └─85 /lib/systemd/systemd-networkd

Apr 12 13:55:59 11test2 systemd[1]: Starting Network Service...
Apr 12 13:56:00 11test2 systemd-networkd[85]: eth0: Gained IPv6LL
Apr 12 13:56:00 11test2 systemd-networkd[85]: Enumeration completed
Apr 12 13:56:00 11test2 systemd[1]: Started Network Service.

root@11test2:~#  journalctl -ulb systemd-networkd.service
Failed to add match 'systemd-networkd.service': Invalid argument

There's the following diff in the zzz-lxc-service.conf:

root@11test2:~# diff /run/systemd/system/service.d/zzz-lxc-service.conf /tmp/zzz-lxc-service-11.2.conf
2d1
< ProcSubset=all

So the reason for the previously working case seems to be because the systemd-networkd service was actually masked before.

Where and why the mask did come from? No idea...

 

If systemd-networkd wasn't running in your old, 11.2 based container, it
couldn't interfere with your custom network setup.

This is btw I recommended to disable systemd-networkd as the correct
solution for your issue.

Yes, that's fine for me. I'll adjust our template for that case. In Ubuntu we need to do the same with Netplan.

[Claudio Kuenzler]

Possibly related: https://github.com/lxc/lxc-ci/pull/487