emulab-udev-settle-networkd.service failing after apt install

[JooYoung Park]

Dear cloudlab admins, 

I deployed a Kubernetes on experiment:

https://www.cloudlab.us/status.php?uuid=468ddefe-58f0-4041-9076-af1d1e2c0cfa#

and on node-002, I ran `sudo apt install some package` and it triggered restart of systemd-networkd.service.

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

Restarting services...
 /etc/needrestart/restart.d/systemd-manager
 systemctl restart multipathd.service packagekit.service polkit.service rdma-ndd.service rpcbind.service rsyslog.service smartmontools.service systemd-journald.service systemd-networkd.service systemd-resolved.service systemd-udevd.service udisks2.service
A dependency job for systemd-networkd.service failed. See 'journalctl -xe' for details.

Service restarts being deferred:
 /etc/needrestart/restart.d/dbus.service
 systemctl restart networkd-dispatcher.service
 systemctl restart systemd-logind.service
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

From journalctl and systemctl, I get

Dependency failed for systemd-networkd.service - Network Configuration.
systemd-networkd.service: Job systemd-networkd.service/start failed with result 'dependency'.

and it seems `emulab-udev-settle-networkd` failed to settle.

emulab-udev-settle-networkd.service: start operation timed out. Terminating.
emulab-udev-settle-networkd.service: Main process exited, code=killed, status=15/TERM
emulab-udev-settle-networkd.service: Failed with result 'timeout'.
Failed to start emulab-udev-settle-networkd.service - Force udev to settle before systemd-networkd starts.
emulab-udev-settle-networkd.service: Scheduled restart job, restart counter is at 18.
Starting emulab-udev-settle-networkd.service - Force udev to settle before systemd-networkd starts...
Waiting for udev to settle...
stat: cannot statx '/run/systemd/network/*.network': No such file or directory
stat: cannot statx '/run/systemd/network/*.network': No such file or directory
Still waiting for udev to settle...
stat: cannot statx '/run/systemd/network/*.network': No such file or directory
Still waiting for udev to settle...

And I cannot access the internet from the nodes since that file is missing.
As I checked, the file disappeared at the point when the systemctl restart was triggered.

Is this an issue with my setup? Or is there something wrong with the emulab udev setting?

Thank you, and have a wonderful day ahead!

[David M Johnson]

Hi, thanks for the report! Yes, something in the apt install must be
invoking `systemd-tmpfiles --clean` or similar. I will add an exemption
in the next round of package updates, but for now, if you run the below
command before your apt install command, I think that will keep
`emulab-udev-settle-networkd.service` happy on systemd-udevd restart.

echo 'X /run/systemd/network - - - - -' | sudo tee -a
/etc/tmpfiles.d/network-cleanup-exemption.conf

David