systemd user@###.service failure causing 90 sec delays during boot, login

[KCB Leigh]

I installed Debian 11 (Bullseye) with GNOME 3.38.5 (Wayland), LINUX kernel Linux version 5.10.0-11-amd64 (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld 2.35.2) #1 SMP Debian 5.10.92-1 (2022-01-18) on a USB stick, and am using it with an ACER Aspire 514 laptop.

This operating system has worked excellently for months, but for the last 2 days has suddenly been taking a very long time to boot.  The cause of the delay can be seen from the syslog:

Feb 28 10:09:30 cpe-67-241-65-193 systemd[1]: Started GNOME Display Manager.
    (The above is the last line on the verbose boot log printed on screen during boot process)
    (omitted next lines from network manager, & kernel, about setting up network & loading audio firmware, etc.)
Feb 28 10:09:31 cpe-67-241-65-193 systemd[1]: Created slice User Slice of UID 119.
Feb 28 10:09:31 cpe-67-241-65-193 systemd[1]: Starting User Runtime Directory /run/user/119...
Feb 28 10:09:31 cpe-67-241-65-193 systemd[1]: Finished User Runtime Directory /run/user/119.
Feb 28 10:09:31 cpe-67-241-65-193 systemd[1]: Starting User Manager for UID 119...
...
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Main process exited, code=exited, status=1/FAILURE
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1144 (gpgconf) with signal SIGKILL.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1145 (awk) with signal SIGKILL.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1174 (dirmngr) with signal SIGKILL.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1144 (gpgconf) with signal SIGKILL.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1145 (awk) with signal SIGKILL.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1174 (dirmngr) with signal SIGKILL.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Failed with result 'exit-code'.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Unit process 1174 (dirmngr) remains running after unit stopped.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: Failed to start User Manager for UID 119.
Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: Started Session c1 of user Debian-gdm.

    The login screen appeared at 10:11:09:

Feb 28 10:11:09 cpe-67-241-65-193 systemd[1]: Startup finished in 51.017s (kernel) + 1min 48.624s (userspace) = 2min 39.642s.

The same 90 sec delay then occurs again after any user enters his password (at 10:11:46):

Feb 28 10:11:46 cpe-67-241-65-193 systemd[1]: Created slice User Slice of UID 1003.
Feb 28 10:11:46 cpe-67-241-65-193 systemd[1]: Starting User Runtime Directory /run/user/1003...
Feb 28 10:11:46 cpe-67-241-65-193 systemd[1]: Finished User Runtime Directory /run/user/1003.
Feb 28 10:11:46 cpe-67-241-65-193 systemd[1]: Starting User Manager for UID 1003...
....
Feb 28 10:13:16 cpe-67-241-65-193 systemd[1]: us...@1003.service: Main process exited, code=exited, status=1/FAILURE
.... (as above)
Feb 28 10:13:16 cpe-67-241-65-193 systemd[1]: Failed to start User Manager for UID 1003.
Feb 28 10:13:16 cpe-67-241-65-193 systemd[1]: Started Session 2 of user kcl.

The first 90 sec. delay only occurs on initial startup, & the second only when any user logs in (the problem is not particular to UID1003, but occurs for all UIDs 100[0-5].  Once the machine has booted & the user is logged in, it functions normally with no observable problems.

I do not know what  caused this, but it occurred right after I:
   > allowed the installation of the latest software update (some lib files, the names of which I unfortunately did not record)
   > installed the ufw firewall package.

I would be very grateful for any information about how to resolve this (apart from re-installing the system) or even any reference that might give information about how I might resolve this problem.  (I have looked at the systemd.service man page, but although I have experience with the unix command line, I have only been using Debian since Nov. 2021 & am not familiar with its system administration, & I could find no information there about this problem.)

[Greg Wooledge]

On Mon, Feb 28, 2022 at 10:28:49PM +0000, KCB Leigh wrote:
> This operating system has worked excellently for months, but for the last 2 days has suddenly been taking a very long time to boot.  The cause of the delay can be seen from the syslog:

Obvious question 1: what changed 2 days ago?

> Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Main process exited, code=exited, status=1/FAILURE
> Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1144 (gpgconf) with signal SIGKILL.

That UID is extremely low. Do you intentionally use users with UIDs in
the 119 range, perhaps because of interoperability with some other system
on your network? Or because of a /home file system that has been migrated
from a different OS?

I would check to see which user(s) are defined with this UID. Maybe there's
a collision.

awk -F: '$3 == 119' /etc/passwd

That should show them all, if they're all defined in /etc/passwd. If you've
got NIS or LDAP or something like that, then you may want to check there
as well. I could easily see a new package being installed, which creates
a new user, which grabs the first unused UID from the users in /etc/passwd,
but doesn't realize that UID 119 is actually used in your network password
database.

> Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1145 (awk) with signal SIGKILL.
> Feb 28 10:11:01 cpe-67-241-65-193 systemd[1]: us...@119.service: Killing process 1174 (dirmngr) with signal SIGKILL.

Certainly looks like you're actually logging in with this UID.

Another thing I'd check, just based on past messages that I've seen here,
is the ownership/permissions of the / directory.

ls -ld /

It should be owned by root, group root, and have 755 perms. If that
ownership gets changed for some reason, it causes all kinds of mess.

[Nicholas Geovanis]

On Mon, Feb 28, 2022 at 4:45 PM KCB Leigh <kcbl...@yahoo.co.uk> wrote:

I installed Debian 11 (Bullseye) with GNOME 3.38.5 (Wayland), LINUX kernel Linux version 5.10.0-11-amd64 (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld 2.35.2) #1 SMP Debian 5.10.92-1 (2022-01-18) on a USB stick, and am using it with an ACER Aspire 514 laptop.

This operating system has worked excellently for months, but for the last 2 days has suddenly been taking a very long time to boot.  The cause of the delay can be seen from the syslog:

Hi -

A shot in the dark:

Make sure that name resolution is working the same now as it was before the problem.

That could include some or all of these: 

Order of name resolution in the /etc/resolv.conf file.

Sources of name-service information in the /etc/nsswitch.conf file.

Loss of network contact with a responsive DNS server which makes a difference. 

 

....

[Richard Hector]

On 1/03/22 12:05, Greg Wooledge wrote:
> On Mon, Feb 28, 2022 at 10:28:49PM +0000, KCB Leigh wrote:
>> This operating system has worked excellently for months, but for the last 2 days has suddenly been taking a very long time to boot.  The cause of the delay can be seen from the syslog:
>
> Obvious question 1: what changed 2 days ago?

Apologies for replying to the wrong message; I've already deleted the
older ones.

This reminded me of a problem I had a couple of months ago, where it
took a long time to log in. I notice also you'd just installed ufw for
firewalling.

My problem turned out to be that starting the us...@xxx.service requires
a network connection on the loopback interface - I was experimenting
with nftables and had neglected to allow that.

Check your firewall for loopback connections?

Cheers,
Richard

[KCB Leigh]

The problem occurred after I installed the ufw firewall package.  I finally figured out (as Mr Richard Hector wrote me) that the problem was caused by ufw blocking the network connection on the loopback interface.  Removing the ufw package resolved the problem.