Syslog/Rsyslog/Systemctl issue

[deb...@sioban.net]

Hi everyone,

I'm contacting you because I'm clueless on what's happening.
Basically my issue is I don't have logs anymore :/

I'm using rsyslog but since some times, I'm getting only this message
when I try to start it:

--
> systemctl start rsyslog.service
A dependency job for rsyslog.service failed. See 'journalctl -xe' for
details.
--

Also I have several services that do not want to start too:

--
❯systemctl --failed
  UNIT                            LOAD   ACTIVE SUB    DESCRIPTION
● fail2ban.service                loaded failed failed Fail2Ban Service
● dbus.socket                     loaded failed failed D-Bus System
Message Bus Socket
● syslog.socket                   loaded failed failed Syslog Socket
● systemd-fsckd.socket            loaded failed failed fsck to fsckd
communication Socket
● systemd-journald-audit.socket   loaded failed failed Journal Audit Socket
● systemd-journald-dev-log.socket loaded failed failed Journal Socket
(/dev/log)
● systemd-journald.socket         loaded failed failed Journal Socket
● systemd-udevd-control.socket    loaded failed failed udev Control Socket
● systemd-udevd-kernel.socket     loaded failed failed udev Kernel Socket
● uuidd.socket                    loaded failed failed UUID daemon
activation socket

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
10 loaded units listed.
--

A month ago I've managed to start manually and to start most of the
services (except systemd-udevd-control.socket and
systemd-udevd-kernel.socket) because I've seen some errors linked to a
selinux filesystem missing.

But this morning I've seen that I wasn't getting any emails logs so I
tried to restart my server. And now, despite the fact the selinux
filesystem is up, I can't start any of the services :/

I don't know where to look at as I don't have ANY logs, even in
journalctl :/

Please help :'(

[Nicolas George]

deb...@sioban.net (12023-01-31):

> A dependency job for rsyslog.service failed. See 'journalctl -xe' for
> details.

Have you considered seeing journalctl -xe for details?

--
Nicolas George

[deb...@sioban.net]

Le 31/01/2023 à 11:08, Nicolas George a écrit :

deb...@sioban.net (12023-01-31):

A dependency job for rsyslog.service failed. See 'journalctl -xe' for
details.

Have you considered seeing journalctl -xe for details?

Hi,

Yes, that was my first action.
The last line of log is from Oct 24 :/

[Greg Wooledge]

I don't know how you expect anyone to help you if you don't provide
any details. Try some or all of these commands:

df /var/log /var/log/journal
systemctl status rsyslog.service
journalctl -u rsyslog.service | tail

Also note that some of these commands give additional detail when run
as root.

Solving your problem may involve figuring out *which* "dependency job"
failed, and why. So, once you identify what's actually failing, you
may have to drill down into that one, the same way you started out
drilling down into rsyslog.service.

[deb...@sioban.net]

Hi Greg,

I'm sorry, I was unsure what kind of logs to give, thanks for giving me some
instructions.

❯ df /var/log /var/log/journal
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/md24       46096212 1620728  42101496   4% /var/log
/dev/md24       46096212 1620728  42101496   4% /var/log


❯ systemctl status rsyslog.service
● rsyslog.service - System Logging Service
     Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor
preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             man:rsyslog.conf(5)
https://www.rsyslog.com/doc/


❯ journalctl -u rsyslog.service | tail
-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24
14:14:33 CEST. --
Oct 24 14:14:02 shax systemd[1]: Stopping System Logging Service...
Oct 24 14:14:02 shax systemd[1]: rsyslog.service: Succeeded.
Oct 24 14:14:02 shax systemd[1]: Stopped System Logging Service.
Oct 24 14:14:02 shax systemd[1]: rsyslog.service: Consumed 47.066s CPU time.

--

Unfortunately the 24 oct is too far for me to remember exactly what happened there.
There is nothing in journalct -xe which might explain.

[Greg Wooledge]

On Tue, Jan 31, 2023 at 03:52:32PM +0100, deb...@sioban.net wrote:
> ❯ df /var/log /var/log/journal
> Filesystem     1K-blocks    Used Available Use% Mounted on
> /dev/md24       46096212 1620728  42101496   4% /var/log
> /dev/md24       46096212 1620728  42101496   4% /var/log

OK, not out of disk space. That's good. Also, interesting that you have
/var/log as a separate file system. That's uncommon (but nothing wrong
with it).

> ❯ systemctl status rsyslog.service
> ● rsyslog.service - System Logging Service
>      Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor
> preset: enabled)
>      Active: inactive (dead)
> TriggeredBy: ● syslog.socket
>        Docs: man:rsyslogd(8)
>              man:rsyslog.conf(5)
> https://www.rsyslog.com/doc/

Frustratingly devoid of details. It says "enabled" though, which is
normal.

> ❯ journalctl -u rsyslog.service | tail
> -- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24
> 14:14:33 CEST. --
> Oct 24 14:14:02 shax systemd[1]: Stopping System Logging Service...
> Oct 24 14:14:02 shax systemd[1]: rsyslog.service: Succeeded.
> Oct 24 14:14:02 shax systemd[1]: Stopped System Logging Service.
> Oct 24 14:14:02 shax systemd[1]: rsyslog.service: Consumed 47.066s CPU time.
>
> --
>
> Unfortunately the 24 oct is too far for me to remember exactly what happened there.
> There is nothing in journalct -xe which might explain.

How about:

systemctl cat rsyslog.service

Maybe you've customized or changed something that will be visible.
Another approach you can take is to Google your error message, and see
if any of the results are helpful. The idea that the unit file may have
been customized came from one of the Google results that I got.

[Sioban]

Here is the output.

❯ systemctl cat rsyslog.service
# /lib/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=man:rsyslog.conf(5)
Documentation=https://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n -iNONE
StandardOutput=null
Restart=on-failure

# Increase the default a bit in order to allow many simultaneous
# files to be monitored, we might need a lot of fds.
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
Alias=syslog.service

I'm pretty sure the issue is not with Rsyslog, but with another service that cannot start preventing rsyslog to start through systemctl (a dependency).
Starting Rsyslog by hand is just fine.

PS: thanks for trying to help me and apology for my previous mis-targeted email, I'm in a very bad day :/

[Greg Wooledge]

On Tue, Jan 31, 2023 at 05:22:23PM +0100, Sioban wrote:
> Here is the output.
>
> ❯ systemctl cat rsyslog.service

That looks exactly like mine. So that's probably fine.

> Requires=syslog.socket

This is the only part of it that looks like a "dependency". Maybe
this is the thing that's breaking? You can try
"systemctl status syslog.socket" and see if it gives anything useful.

> I'm pretty sure the issue is not with Rsyslog, but with another service that
> cannot start preventing rsyslog to start through systemctl (a dependency).
> Starting Rsyslog by hand is just fine.

How did you do that? "By hand" can mean a lot of different things.

[deb...@sioban.net]

Le 31/01/2023 à 19:29, Greg Wooledge a écrit :

Requires=syslog.socket

This is the only part of it that looks like a "dependency".  Maybe
this is the thing that's breaking?  You can try
"systemctl status syslog.socket" and see if it gives anything useful.

Here is the result:

❯ systemctl status syslog.socket
● syslog.socket - Syslog Socket
     Loaded: loaded (/lib/systemd/system/syslog.socket; static)
     Active: failed (Result: resources)
   Triggers: ● rsyslog.service
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/syslog
     Listen: /run/systemd/journal/syslog (Datagram)

And same for cat:

❯ systemctl cat syslog.socket
# /lib/systemd/system/syslog.socket
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Syslog Socket
Documentation=man:systemd.special(7)
Documentation=https://www.freedesktop.org/wiki/Software/systemd/syslog
DefaultDependencies=no
Before=sockets.target

# Don't allow logging until the very end
Conflicts=shutdown.target
Before=shutdown.target

# Don't try to activate syslog.service if sysinit.target has failed.
Conflicts=emergency.service
Before=emergency.service

[Socket]
ListenDatagram=/run/systemd/journal/syslog
SocketMode=0666
PassCredentials=yes
PassSecurity=yes
ReceiveBuffer=8M

# The default syslog implementation should make syslog.service a
# symlink to itself, so that this socket activates the right actual
# syslog service.
#
# Examples:
#
# /etc/systemd/system/syslog.service -> /lib/systemd/system/rsyslog.service
# /etc/systemd/system/syslog.service -> /lib/systemd/system/syslog-ng.service
#
# Best way to achieve that is by adding this to your unit file
# (i.e. to rsyslog.service or syslog-ng.service):
#
# [Install]
# Alias=syslog.service
#
# See https://www.freedesktop.org/wiki/Software/systemd/syslog for details.


And trying to start it:
❯ systemctl start syslog.socket
Job failed. See "journalctl -xe" for details.

Nothing in journalctl, and nothings in any logs related to this issue at this specific time

I'm pretty sure the issue is not with Rsyslog, but with another service that
cannot start preventing rsyslog to start through systemctl (a dependency).
Starting Rsyslog by hand is just fine.

How did you do that?  "By hand" can mean a lot of different things.

As root: "/usr/sbin/rsyslogd -i /run/rsyslogd.pid"

I know it's not the best but I was hoping to see things in the logs related to the issues :/

[Max Nikulin]

On 31/01/2023 16:57, deb...@sioban.net wrote:
> I'm contacting you because I'm clueless on what's happening.
> Basically my issue is I don't have logs anymore :/

...

> > systemctl start rsyslog.service
> A dependency job for rsyslog.service failed. See 'journalctl -xe' for
> details.

> ❯systemctl --failed
>   UNIT                            LOAD   ACTIVE SUB    DESCRIPTION

...

> ● systemd-journald.socket         loaded failed failed Journal Socket

This line looks suspicious. I would ensure at first that journald is
running correctly. Rsyslog likely receives logs from journald.

Are logs available through the following commands (last entries and for
current boot)?

journalctl -e
journalctl -b

Does

systemctl status systemd-journald.service

reports any errors?

[deb...@sioban.net]

Hi max,

"journalctl -e" ends the 24th of October.

--
Oct 24 14:14:33 shax systemd[1]: Stopped Update UTMP about System Boot/Shutdown.
Oct 24 14:14:33 shax systemd[1]: systemd-tmpfiles-setup.service: Succeeded.
Oct 24 14:14:33 shax systemd[1]: Stopped Create Volatile Files and Directories.
Oct 24 14:14:33 shax systemd[1]: Stopping Flush Journal to Persistent Storage...
--

--
❯ journalctl -b

-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24 14:14:33 CEST. --

-- No entries --
--

systemd-journald.service reports error but unclear :/

--
❯ systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
     Active: inactive (dead)
TriggeredBy: ● systemd-journald-dev-log.socket
             ● systemd-journald-audit.socket
             ● systemd-journald.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
--

--
❯ systemctl start systemd-journald.service
A dependency job for systemd-journald.service failed. See 'journalctl -xe' for details.
--

[Freyja]

Hi greg,

--
❯ df /run

Filesystem     1K-blocks  Used Available Use% Mounted on

tmpfs            1629236  3732   1625504   1% /run
--

For /run/systemd/journal/socket this is another story.
The folder journal does not exist at all.

--
❯ ls -la /run/systemd/
total 0
drwxr-xr-x 17 root            root             400 Jan 31 16:12 .
drwxr-xr-x 37 root            root            1100 Feb  1 06:25 ..
drwxr-xr-x  2 root            root              40 Jan 31 10:45 ask-password
drwx------  2 root            root              80 Jan 31 17:31 ask-password-block
drwxr-xr-x  8 root            root             680 Jan 31 16:08 generator
drwxr-xr-x  4 root            root             360 Jan 31 16:08 generator.late
drwxr-xr-x  3 root            root             160 Jan 31 10:45 inaccessible
srw-rw-rw-  1 root            root               0 Jan 31 10:45 io.system.ManagedOOM
drwxr-xr-x  2 root            root              40 Jan 31 10:45 machines
drwxr-xr-x  5 systemd-network systemd-network  100 Jan 31 10:45 netif
srwxrwxrwx  1 root            root               0 Jan 31 10:45 notify
srwxrwxrwx  1 root            root               0 Jan 31 10:45 private
drwxr-xr-x  2 root            root              40 Jan 31 10:45 seats
drwxr-xr-x  2 root            root              40 Jan 31 10:45 sessions
drwxr-xr-x  2 root            root              40 Jan 31 10:45 shutdown
drwxr-xr-x  2 root            root              40 Jan 31 10:45 system
drwx------  2 root            root              40 Jan 31 10:45 unit-root
drwxr-xr-x  2 root            root            2060 Feb  1 13:09 units
drwxr-xr-x  2 root            root              60 Jan 31 10:45 userdb
drwxr-xr-x  2 root            root              40 Jan 31 10:45 users
--

I don't know why :/

Le 01/02/2023 à 13:22, Greg Wooledge a écrit :

On Wed, Feb 01, 2023 at 11:15:58AM +0100, deb...@sioban.net wrote:

"journalctl -e" ends the 24th of October.

❯ journalctl -b
-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24
14:14:33 CEST. --
-- No entries --

❯ systemctl start systemd-journald.service
A dependency job for systemd-journald.service failed. See 'journalctl -xe'
for details.

On my system:

unicorn:~$ systemctl cat systemd-journald.service | grep -i require
Requires=systemd-journald.socket
# /proc/<pid>/exe requires this capability. Thus if this capability is missing

unicorn:~$ systemctl cat systemd-journald.socket | grep Listen
ListenDatagram=/run/systemd/journal/socket
ListenStream=/run/systemd/journal/stdout

For you, that's at least *two* socket services that appear to be failing.
I suspect you've got something broken that's preventing these sockets
from being created, or being accessed.

Man, what I wouldn't give for a nice, clear "/run/systemd/journal/socket:
Permission denied" or "... No such file or directory" error!

/run is supposed to be mounted as a separate file system, entirely in
memory:

unicorn:~$ df /run
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs            1215596  1924   1213672   1% /run

And the ownerships and permissions on this socket are:

unicorn:~$ namei -l /run/systemd/journal/socket
f: /run/systemd/journal/socket
drwxr-xr-x root root /
drwxr-xr-x root root run
drwxr-xr-x root root systemd
drwxr-xr-x root root journal
srw-rw-rw- root root socket

If you've got anything different, that would be important.

[Greg Wooledge]

On Wed, Feb 01, 2023 at 11:15:58AM +0100, deb...@sioban.net wrote:

> "journalctl -e" ends the 24th of October.

> ❯ journalctl -b
> -- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24
> 14:14:33 CEST. --
> -- No entries --

> ❯ systemctl start systemd-journald.service
> A dependency job for systemd-journald.service failed. See 'journalctl -xe'
> for details.

[Freyja]

There is an interesting part: I've already tried to solve this issue in December with some success except for 2 systemd services (udevd-control & udevd-kernel).
It stopped again working the 25th of december.

--
I've found a journal related to 24th December in /var/log/journal:

root in /var/log
❯ ls -la journal/
total 44
drwxr-sr-x+  3 root systemd-journal  4096 Aug 19  2021 .
drwxr-xr-x. 33 root root            12288 Feb  1 06:25 ..
drwxr-sr-x+  2 root systemd-journal 20480 Dec 24 11:35 ef72ee158c9d4ae68bd60ae8adbb20af

root in /var/log
❯ cd journal/ef72ee158c9d4ae68bd60ae8adbb20af/

root in log/journal/ef72ee158c9d4ae68bd60ae8adbb20af
❯ ls -la
total 98356
drwxr-sr-x+ 2 root systemd-journal    20480 Dec 24 11:35 .
drwxr-sr-x+ 3 root systemd-journal     4096 Aug 19  2021 ..
-rw-r-----+ 1 root systemd-journal 75497472 Oct 24 14:14 system.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 24 14:14 user-1000.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 23 19:56 user-1014.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 24 14:14 user-1015.journal
--

And here are our logs from the 24th October.

[David]

On Thu, 2 Feb 2023 at 00:07, Freyja <Sio...@sioban.net> wrote:

> I've already tried to solve this issue in December with some success

Hi, could you describe in detail what you did?

Can you also post the output of:
cat /etc/systemd/system.conf

Also, a small thing I happened to notice:

> root in /var/log
> ❯ ls -la journal/
> total 44
> drwxr-sr-x+ 3 root systemd-journal 4096 Aug 19 2021 .
> drwxr-xr-x. 33 root root 12288 Feb 1 06:25 ..

^
On my system I do not have the '.' character in column 11
which you show in your output line that I have quoted
immediately above. I don't know anything about that or
if is relevant, just mentioning it in case someone else
can comment on that.

Command
info -f coreutils -n 'What information is listed'
says:
Following the file mode bits is a single character that specifies
whether an alternate access method such as an access control list
applies to the file. When the character following the file mode
bits is a space, there is no alternate access method. When it is a
printing character, then there is such a method.

GNU ‘ls’ uses a ‘.’ character to indicate a file with a security
context, but no other alternate access method.

[Charles Curley]

On Wed, 1 Feb 2023 13:26:58 +0100
Freyja <deb...@sioban.net> wrote:

> For /run/systemd/journal/socket this is another story.
> The folder journal does not exist at all.

Well, that could be a problem. On one of my computers:

root@jhegaala:~# ll /run/systemd/journal/socket
srw-rw-rw- 1 root root 0 Jan 25 09:55 /run/systemd/journal/socket=
root@jhegaala:~# ll -d /run/systemd/journal/
drwxr-xr-x 3 root root 200 Jan 25 09:55 /run/systemd/journal//
root@jhegaala:~#

You might try rebuilding the directory /run/systemd/journal/ with
appropriate permissions and manually restarting the relevant services.
(The services should rebuild the sockets for you.)

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[Freyja]

I don't remember all the steps I've made.
I tried so many things.

But I've found that some processes were complaining that selinux filesystem was not mounted.
Mounted the filesystem with the command "cat /etc/systemd/system.conf" and it helped starting some process.

That time, even if the filesystem is mounted, the services do not want to start :/

I guess my systemd.conf is default
--
❯ cat /etc/systemd/system.conf

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See systemd-system.conf(5) for details.

[Manager]
#LogLevel=info
#LogTarget=journal-or-kmsg
#LogColor=yes
#LogLocation=no
#LogTime=no
#DumpCore=yes
#ShowStatus=yes
#CrashChangeVT=no
#CrashShell=no
#CrashReboot=no
#CtrlAltDelBurstAction=reboot-force
#CPUAffinity=1 2
#NUMAPolicy=default
#NUMAMask=
#RuntimeWatchdogSec=0
#RebootWatchdogSec=10min
#ShutdownWatchdogSec=10min
#KExecWatchdogSec=0
#WatchdogDevice=
#CapabilityBoundingSet=
#NoNewPrivileges=no
#SystemCallArchitectures=
#TimerSlackNSec=
#StatusUnitFormat=description
#DefaultTimerAccuracySec=1min
#DefaultStandardOutput=journal
#DefaultStandardError=inherit
#DefaultTimeoutStartSec=90s
#DefaultTimeoutStopSec=90s
#DefaultTimeoutAbortSec=
#DefaultRestartSec=100ms
#DefaultStartLimitIntervalSec=10s
#DefaultStartLimitBurst=5
#DefaultEnvironment=
#DefaultCPUAccounting=no
#DefaultIOAccounting=no
#DefaultIPAccounting=no
#DefaultBlockIOAccounting=no
#DefaultMemoryAccounting=yes
#DefaultTasksAccounting=yes
#DefaultTasksMax=15%
#DefaultLimitCPU=
#DefaultLimitFSIZE=
#DefaultLimitDATA=
#DefaultLimitSTACK=
#DefaultLimitCORE=
#DefaultLimitRSS=
#DefaultLimitNOFILE=1024:524288
#DefaultLimitAS=
#DefaultLimitNPROC=
#DefaultLimitMEMLOCK=
#DefaultLimitLOCKS=
#DefaultLimitSIGPENDING=
#DefaultLimitMSGQUEUE=
#DefaultLimitNICE=
#DefaultLimitRTPRIO=
#DefaultLimitRTTIME=
--

I have no idea what is this "."...
I know I've disabled apparmor and selinux in the past because too many processes were blocked by them

[Freyja]

Ok,

Instead of just creating the folders, I've reinstalled some packages:

--
❯ apt-get install --reinstall  systemd systemd-sysv libsystemd0 dbus-user-session
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 4 reinstalled, 0 to remove and 3 not upgraded.
Need to get 376 kB/5,090 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ftp.fr.debian.org//debian bullseye/main amd64 libsystemd0 amd64 247.3-7+deb11u1 [376 kB]
Fetched 376 kB in 1s (645 kB/s)
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../libsystemd0_247.3-7+deb11u1_amd64.deb ...
Unpacking libsystemd0:amd64 (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Setting up libsystemd0:amd64 (247.3-7+deb11u1) ...
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../systemd_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Setting up systemd (247.3-7+deb11u1) ...
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../systemd-sysv_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd-sysv (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Preparing to unpack .../dbus-user-session_1.12.24-0+deb11u1_amd64.deb ...
Unpacking dbus-user-session (1.12.24-0+deb11u1) over (1.12.24-0+deb11u1) ...
Setting up systemd-sysv (247.3-7+deb11u1) ...
Setting up dbus-user-session (1.12.24-0+deb11u1) ...
Processing triggers for libc-bin (2.31-13+deb11u5) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for dbus (1.12.24-0+deb11u1) ...
Scanning processes...
Scanning candidates...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

Restarting services...
 systemctl restart clamav-daemon.service ossec.service smartmontools.service squid.service ssh.service t...@default.service transmission-daemon.service
Service restarts being deferred:
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.
--

It helped!
The folders have been recreated:

--
❯ ls -la /run/systemd/journal/socket
srw-rw-rw- 1 root root 0 Feb  1 15:44 /run/systemd/journal/socket

❯ ls -lad /run/systemd/journal/
drwxr-xr-x 2 root root 120 Feb  1 15:44 /run/systemd/journal/

--

I was even able to start most of the failing process but two:

--
❯ systemctl --failed

  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION

● systemd-udevd-control.socket loaded failed failed udev Control Socket
● systemd-udevd-kernel.socket  loaded failed failed udev Kernel Socket

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed.
--

However journalctl still stucks on 24th Oct:
--

❯ journalctl -b
-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24 14:14:33 CEST. --
-- No entries --

--

[Greg Wooledge]

On Wed, Feb 01, 2023 at 03:39:05PM +0100, Freyja wrote:
> But I've found that some processes were complaining that selinux filesystem
> was not mounted.

> I know I've disabled apparmor and selinux in the past because too many
> processes were blocked by them

I don't know much about SELinux. If that's the underlying issue here,
you're going to need help from someone who knows SELinux.

Overall, the fact that you got it working by restarting various things
long after boot time makes it sound like things are not happening in
the right order *at* boot time.

Just as one *possible* example, a program that's supposed to create
the directory /run/systemd/journal won't be able to do that until the
/run file system is mounted. If the mounting of /run is happening
later than expected (due to SELinux or whatever), it might cause things
like systemd-journald to fail to start at boot time.

Tracking down the exact causes of these types of issues is *very*
difficult, especially due to the non-sequential nature of systemd's
boot process. It tries to do as many things as possible in parallel,
which is great for speed, but a massive headache when trying to
troubleshoot a problem.

But it sounds like you've at least got some inkling of where the
problem is. I don't know what an "selinux filesystem" is, but if you
need it to be mounted, or running, or whatever, before other things can
go, then you'll need to look at the dependency relationships of your
"selinux" unit with respect to other units. Make sure everything runs
at the correct time.

[songbird]

Freyja wrote:
...

> However journalctl still stucks on 24th Oct:
> --
> ❯ journalctl -b
> -- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon
> 2022-10-24 14:14:33 CEST. --
> -- No entries --

if you are not concerned about keeping the last journals for
any reason i would remove the entire directory and then reboot.
if you do want them back them up and then remove that directory
and try it again from a fresh reboot.


songbird

[Max Nikulin]

On 01/02/2023 21:48, Freyja wrote:
> Instead of just creating the folders, I've reinstalled some packages:

I would consider complete reinstall (install from scratch). It may be
faster even for a person having experience with systemd troubleshooting.
Something low level is affected.

However I find the problem interesting: how to get logs if journald can
not start.

[Freyja]

This is only a solution I'll look at as last resort.
There is too much work on that server.

But I agree it's something I was already thought of.

But we are not on Windows there, we should be able to do something.
The only time I had to do this is because dpkg was totally broken for an unknown reason (but that's a long time ago)

[Greg Wooledge]

On Thu, Feb 02, 2023 at 02:27:33PM +0100, Freyja wrote:
> This [reinstalling] is only a solution I'll look at as last resort.

> There is too much work on that server.
>
> But I agree it's something I was already thought of.
>
> But we are not on Windows there, we should be able to do something.

Another good reason to eschew reinstalling as a solution is that
it doesn't give you any understanding. Of course the main goal here
is to get the logs working, but a secondary goal has to be figuring
out WHY they're not working. Without that understanding, the problem
might happen again.

[Freyja]

Deleted the folder, restarted and back to square 1.

--
❯ systemctl --failed
  UNIT                            LOAD   ACTIVE SUB    DESCRIPTION

● dbus.socket                     loaded failed failed D-Bus System Message Bus Socket
● syslog.socket                   loaded failed failed Syslog Socket
● systemd-fsckd.socket            loaded failed failed fsck to fsckd communication Socket
● systemd-journald-audit.socket   loaded failed failed Journal Audit Socket

● systemd-journald-dev-log.socket loaded failed failed Journal Socket (/dev/log)

● systemd-journald.socket         loaded failed failed Journal Socket

● systemd-udevd-control.socket    loaded failed failed udev Control Socket
● systemd-udevd-kernel.socket     loaded failed failed udev Kernel Socket

● uuidd.socket                    loaded failed failed UUID daemon activation socket

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

9 loaded units listed.
--

--
❯ ls /run/systemd/journal
ls: cannot access '/run/systemd/journal': No such file or directory
--

To make it works again I had to:
- mount /sys/fs/selinux -o remount,ro
- reinstall again "systemd systemd-sysv libsystemd0 dbus-user-session"
- start all failed systemd services

Still can't these two:

--
❯ systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● systemd-udevd-control.socket loaded failed failed udev Control Socket
● systemd-udevd-kernel.socket  loaded failed failed udev Kernel Socket

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
2 loaded units listed.
--

But now I have journalctl giving me recent logs !
That's a starter.

So when I try to start them I'm getting this:
--
❯ systemctl start systemd-udevd-control.socket
Job failed. See "journalctl -xe" for details.

Feb 02 14:40:14 shax systemd[1]: systemd-udevd-control.socket: Socket service systemd-udevd.service already active, refusing.
Feb 02 14:40:14 shax systemd[1]: Failed to listen on udev Control Socket.
░░ Subject: A start job for unit systemd-udevd-control.socket has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-udevd-control.socket has finished with a failure.
░░
░░ The job identifier is 5423 and the job result is failed.
--

--
❯ systemctl start systemd-udevd-kernel.socket
Job failed. See "journalctl -xe" for details.

Feb 02 14:41:13 shax systemd[1]: systemd-udevd-kernel.socket: Socket service systemd-udevd.service already active, refusing.
Feb 02 14:41:13 shax systemd[1]: Failed to listen on udev Kernel Socket.
░░ Subject: A start job for unit systemd-udevd-kernel.socket has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-udevd-kernel.socket has finished with a failure.
░░
░░ The job identifier is 5431 and the job result is failed.
--

[Freyja]

This is only a solution I'll look at as last resort.

There is too much work on that server.

But I agree it's something I was already thought of.

But we are not on Windows there, we should be able to do something.

The only time I had to do this is because dpkg was totally broken for an unknown reason (but that's a long time ago)

Le 02/02/2023 à 13:15, Max Nikulin a écrit :

[songbird]

i always see something at least in other files because i
still use rsyslog.

restore the directory and file then, but if you have
/etc/syslog/journald.conf complicated somehow i can't
really say much about that. i use it very simply and
the only option in there i have is:

SplitMode=none

because i don't want stuff going in multuple places so
i don't have a lot of different files to deal with or
clean out.

my clean out script does not remove the directory itself.
i thought it did so i'm sorry for that, instead i just remove
the files in the bottom subdirectory by using:

rm journal/bl*/system@*
rm journal/bl*/system*

in my clean out script.


for a normal day this is what it looks like so you can see
that my systemd-journal directory hasn't been removed since
Apr 4 2018

/var/log:

drwxr-sr-x+ 3 root systemd-journal 4096 Apr 4 2018 journal


/var/log/journal/blahblah:

drwxr-sr-x+ 2 root systemd-journal 4096 Feb 2 06:46 blahblah

-rw-r-----+ 1 root systemd-journal 8388608 Feb 2 10:09 system.journal


songbird

[debia...@howorth.org.uk]

Maybe it's worth asking on the systemd-devel mailing list?

[songbird]

debia...@howorth.org.uk wrote:
...

> Maybe it's worth asking on the systemd-devel mailing list?

did you reinstall rsyslog? did you purge the configs for it
(back them up if needed)? when i remove a package i have been
having troubles with i often purge it instead of uninstalling
it.

you may need to reboot after reinstalling systemd and/or
rsyslog or any other parts.

see my other note about what your /var/log/journal directory
should look like if you have that set up right for permissions.

back up the files if you want to keep the old ones. i'm not
sure the subdirectory in /var/log/journal will be recreated or
not but i think it will if it isn't there, if it is there and
empty the journald service should set up a new one. note this
depends upon what setting you've got in your config files for
systemd. i'm not sure if you've altered systemd configuration
files or not.

note that reinstalling without purging will not necessarily
reset your configuration files.

rsyslog files should show up in /var/log or where ever you
put them.

these are things i can think of trying. sorry i've gotta
get going and won't be able to help more until later tonight.
hope you get it figured out before then. :)


songbird

[Michael Biebl]

Are you using SELinux or AppApparmor ?

Have you tried disabling it?

[Freyja]

Hi,

Both are installed but when I try to remove them it want to remove too many packages :/

--
❯ dpkg -l |grep selinux
ii  libselinux1:amd64                    3.1-3                                                  amd64        SELinux runtime shared libraries
--

--
❯ apt-get remove libselinux1

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done

The following packages were automatically installed and are no longer required:
  debugedit discover-data docutils-doc efibootmgr fonts-lyx fonts-urw-base35 geoip-database-extra gsasl-common i965-va-driver ienglish-common intel-media-va-driver jo jq liba52-0.7.4 libaacs0
  libaec0 libann0 libao-common libao4 libaom0 libaom3 libapr1 libarchive13 libaribb24-0 libarmadillo10 libarpack2 libaspell15 libassuan0 libasyncns0 libatk1.0-data libaudio2 libauparse0
  libavahi-client3 libavahi-common-data libavahi-common3 libavahi-core7 libavc1394-0 libavresample4 libavutil56 libbcg729-0 libbfio1 libblas3 libboost-thread1.74.0 libbpf0 libbs2b0 libbsd-dev
  libc-ares2 libc-dev-bin libcaca0 libcbor0 libcdio-cdda2 libcdio-paranoia2 libcdio19 libcdt5 libcgraph6 libcharls2 libcjson1 libcodec2-0.9 libconfig-inifiles-perl libcrypt-dev libdaemon0
  libdatrie1 libdav1d4 libdav1d5 libdavs2-16 libdbi1 libdc1394-25 libdca0 libde265-0 libdiscover2 libdjvulibre-text libdjvulibre21 libdouble-conversion3 libdrm-amdgpu1 libdrm-common libdrm-intel1
  libdrm-nouveau2 libdrm-radeon1 libdrm2 libdv4 libdvdnav4 libdw1 libebml5 libecap3 libedit2 libefiboot1 libefivar1 libegl-mesa0 libegl1 libenca0 libepoxy0 libepsilon1 libev4 libevent-2.1-7
  libevent-core-2.1-7 libevent-pthreads-2.1-7 libewf2 libexiv2-14 libexttextcat-2.0-0 libexttextcat-data libfaac0 libfcgi-bin libfcgi0ldbl libfdk-aac2 libffi-dev libfftw3-double3 libflite1 libfmt7
  libfontenc1 libfreexl1 libfsplib0 libfstrm0 libfuse2 libfyba0 libgbm1 libgc1 libgdbm-compat4 libgdbm6 libgdk-pixbuf2.0-common libgeos-3.9.0 libgeos-c1v5 libgfortran5 libgl1 libgl1-mesa-dri
  libgl1-mesa-glx libglapi-mesa libglib2.0-data libglvnd0 libglx-mesa0 libglx0 libgme0 libgraphite2-3 libgs9-common libgtk2.0-common libgvpr2 libhdf4-0-alt libhdf5-103-1 libhdf5-hl-100 libheif1
  libhiredis0.14 libhttp-parser2.9 libiec61883-0 libigdgmm11 libijs-0.35 libilbc3 libilmbase25 libimagequant0 libinih1 libjack-jackd2-0 libjargs-java libjbig2dec0 libjq1 libjs-bootstrap4
  libjs-codemirror libjs-jquery libjs-jquery-mousewheel libjs-jquery-timepicker libjs-jquery-ui libjs-openlayers libjs-popper.js libjs-sphinxdoc libjs-underscore libjson-c5 libjson-glib-1.0-common
  libjsoncpp24 libjxr-tools libjxr0 libk5crypto3 libkmlbase1 libkmldom1 libkmlengine1 libkrb5support0 libksba8 libkvazaar6 liblab-gamut1 liblapack3 liblavjpeg-2.2-0 liblbfgsb0 liblcms2-2
  libldap-2.4-2 libldap-common libldb2 liblensfun-data-v1 liblept5 liblilv-0-0 liblinear4 liblirc-client0 libllvm11 liblmdb0 liblua5.1-0 liblua5.2-0 liblua5.3-0 liblua5.4-0 libluajit-5.1-2
  libluajit-5.1-common liblz1 libmad0 libmarkdown2 libmatroska7 libmaxminddb0 libmbedcrypto3 libmbedtls12 libmbedx509-0 libmd-dev libmemcached11 libmemcachedutil2 libmfx1 libmilter1.0.1
  libminiupnpc17 libminizip1 libmjpegutils-2.2-0 libmms0 libmng2 libmono-btls-interface4.0-cil libmp3lame0 libmpdec2 libmpdec3 libmpeg2-4 libmpeg2encpp-2.2-0 libmpg123-0 libmplex2-2.2-0 libmysofa1
  libnatpmp1 libnftables1 libnghttp2-14 libnorm1 libnpth0 libnspr4 libnss3 libntlm0 libnuma1 libnutscan1 libobjc4 libodbc1 libogdi4.1 libonig5 libopenal-data libopenal1 libopendbx1
  libopendbx1-sqlite3 libopenexr25 libopenh264-6 libopenjp2-7 libopenmpt0 libopts25 libosp5 libossp-uuid16 libostyle1c2 libpathplan4 libpciaccess0 libpcre2-16-0 libpcre2-posix2 libpcsclite1
  libpgm-5.3-0 libpixman-1-0 libpkgconf3 libpostproc55 libprocps8 libprotobuf-c1 libproxy1v5 libpugixml1v5 libpython2.7-minimal libqhull8.0 libraw1394-11 librbl1 libre2-9 librhash0 librist4 librpm9
  librpmbuild9 librpmio9 librpmsign9 librtmp1 librttopo1 librubberband2 libsamplerate0 libsasl2-2 libsasl2-modules-db libsbc1 libsecret-common libsensors-config libsensors5 libserd-0-0 libshine3
  libsmi2ldbl libsndio7.0 libsnmp-base libsodium23 libsord-0-0 libsoxr0 libspandsp2 libspeex1 libspeexdsp1 libsqlite0 libsratom-0-0 libsrt1.4-gnutls libssh2-1 libstemmer0d libsuperlu5 libsvtav1enc1
  libswresample3 libswscale5 libsz2 libtalloc2 libtesseract4 libtevent0 libtfm1 libthai-data libthai0 libtinyxml2-8 libtirpc-common libtokyocabinet9 libtre5 libtwolame0 libudfread0 libunbound8
  libupsclient4 liburing1 liburiparser1 libusb-0.1-4 libutf8proc2 libuv1 libva-drm2 libva-x11-2 libva2 libvbr2 libvdpau-va-gl1 libvdpau1 libvhdi1 libvidstab1.1 libvmaf1 libvmdk1 libvo-amrwbenc0
  libvorbisidec1 libvpx6 libvte-2.91-common libvulkan1 libwayland-client0 libwayland-cursor0 libwayland-egl1 libwayland-server0 libwebpdemux2 libwebpmux3 libwireshark-data libwmf0.2-7 libx264-164
  libx265-192 libx265-199 libxavs2-13 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0 libxcb-render0 libxcb-shape0 libxcb-sync1 libxcb-xfixes0 libxcursor1 libxdamage1 libxi6
  libxinerama1 libxkbcommon0 libxkbfile1 libxmuu1 libxrandr2 libxrender1 libxshmfence1 libxss1 libxtst6 libxv1 libxvidcore4 libxvmc1 libxxf86dga1 libxxf86vm1 libyajl2 libyaml-0-2 libz3-4 libzen0v5
  libzimg2 libzvbi-common libzvbi0 linux-compiler-gcc-10-x86 linux-headers-5.10.0-19-common linux-headers-5.10.0-20-common lua-lpeg lzip mailutils-common media-types mesa-va-drivers
  mesa-vdpau-drivers mesa-vulkan-drivers ocl-icd-libopencl1 pbzip2 pinentry-curses pixz python-apt-common python-matplotlib-data python2.7-minimal qtchooser qttranslations5-l10n ruby2.7-doc
  runit-helper shim-helpers-amd64-signed shim-unsigned va-driver-all vdpau-driver-all xdelta3
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  acpi-support-base acpid adduser adwaita-icon-theme altermime amavisd-milter amavisd-new ant ant-optional apache2 apache2-bin apache2-utils apparmor apt apt-file apt-listchanges apt-rdepends
  apt-show-versions apt-transport-https apt-utils aspell aspell-en aspell-fr at at-spi2-core atop autoconf automake autoproject avahi-daemon bc bind9 bind9-dnsutils bind9-host bind9-libs
  bind9-utils bind9utils blt bsd-mailx build-essential ca-certificates ca-certificates-java ca-certificates-mono cabal-install cadaver certbot checksecurity chkrootkit cifs-utils clamav clamav-base
  clamav-daemon clamav-freshclam clamav-milter clamav-unofficial-sigs clamdscan cli-common cmake colordiff comerr-dev composer console-common console-data console-setup console-setup-linux
  coreutils cpanminus cpuid crda cron crowdsec curl cvs dash dbconfig-common dbconfig-mysql dbus dbus-user-session dbus-x11 dconf-gsettings-backend dconf-service ddupdate debconf debconf-i18n
  debhelper debsecan debsums default-jre default-jre-headless default-mysql-client devscripts dh-autoreconf dh-buildinfo dh-python dh-strip-nondeterminism dictionaries-common dirmngr discover dlint
  dmsetup dnsutils doc-base docbook docbook-dsssl docbook-xml docbook-xsl docutils-common dovecot-core dovecot-gssapi dovecot-imapd dovecot-ldap dovecot-mysql dovecot-pgsql dovecot-pop3d
  dovecot-sieve dovecot-sqlite dpatch dpkg dpkg-dev dput dsniff dupload eject elinks equivs expect fail2ban fancontrol fastjar fdisk fetchmail ffmpeg fig2dev findutils fontconfig fontconfig-config
  fonts-droid-fallback fonts-lato fortunes-fr fpdns ftp fuzzyocr g++ g++-10 galera-3 gawk gconf-service gconf2 gconf2-common geoip-bin gettext ghc ghostscript giflib-tools gir1.2-glib-2.0
  gir1.2-packagekitglib-1.0 git glib-networking glib-networking-services gnupg gnupg-utils gnupg2 gnuplot-nox gnustep-base-common gnustep-base-runtime gpg gpg-agent gpg-wks-client gpg-wks-server
  gpgconf gpgsm graphviz grep grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub2-common gsettings-desktop-schemas gtk-update-icon-cache guile-2.2-libs gzip hardinfo hddtemp
  hping3 hplip-data html2ps httping iamerican ibritish ibverbs-providers ifrench-gut ifupdown imagemagick imagemagick-6.q16 info init init-system-helpers initramfs-tools initramfs-tools-core
  install-info intltool-debian iotop iproute2 isc-dhcp-client java-package jnettop jsonlint keyboard-configuration kmod krb5-multidev latencytop lftp libafflib0v5 libalgorithm-c3-perl
  libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libaliased-perl libalpm13 libany-uri-escape-perl libapache2-mod-apreq2 libapache2-mod-gnutls libapache2-mod-perl2
  libapache2-mod-php7.4 libapache2-mod-security2 libapache2-reload-perl libapache2-request-perl libappstream4 libapreq2-3 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libapt-pkg-perl
  libarchive-cpio-perl libarchive-zip-perl libarray-intspan-perl libass9 libatk-bridge2.0-0 libatk-wrapper-java libatk-wrapper-java-jni libatk1.0-0 libatspi2.0-0 libauthen-sasl-perl libavcodec58
  libavdevice58 libavfilter7 libavformat58 libb-hooks-endofscope-perl libb-hooks-op-check-perl libberkeleydb-perl libbluray2 libbsd-resource-perl libbytes-random-secure-perl libc-client2007e
  libc-devtools libc6-dev libcairo-gobject2 libcairo2 libcapture-tiny-perl libcephfs2 libcfitsio9 libcgi-fast-perl libcgi-pm-perl libchromaprint-tools libchromaprint1 libclamav9 libclass-c3-perl
  libclass-c3-xs-perl libclass-data-inheritable-perl libclass-inspector-perl libclass-method-modifiers-perl libclass-xsaccessor-perl libclone-perl libcolord2 libcommon-sense-perl
  libconfig-tiny-perl libconst-fast-perl libcontextual-return-perl libconvert-asn1-perl libconvert-binhex-perl libconvert-tnef-perl libconvert-uulib-perl libcpan-changes-perl
  libcpan-distnameinfo-perl libcpan-meta-check-perl libcpanel-json-xs-perl libcrypt-openssl-bignum-perl libcrypt-openssl-random-perl libcrypt-openssl-rsa-perl libcrypt-passwdmd5-perl
  libcrypt-random-seed-perl libcryptsetup12 libcups2 libcurl3-gnutls libcurl3-nss libcurl4 libcurl4-openssl-dev libdap27 libdapclient6v5 libdata-dpath-perl libdata-dump-perl
  libdata-messagepack-perl libdata-optlist-perl libdata-perl-perl libdata-section-perl libdata-validate-domain-perl libdate-manip-perl libdbd-mysql-perl libdbi-perl libdbus-glib-1-2 libdconf1
  libdebhelper-perl libdevel-callchecker-perl libdevel-caller-perl libdevel-globaldestruction-perl libdevel-lexalias-perl libdevel-size-perl libdevel-stacktrace-perl libdevel-symdump-perl
  libdevmapper1.02.1 libdigest-bubblebabble-perl libdigest-hmac-perl libdigest-sha-perl libdist-checkconflicts-perl libdistro-info-perl libdns-export1104 libdns-export1110 libdpkg-perl
  libdynaloader-functions-perl libemail-address-xs-perl libemail-date-format-perl libencode-detect-perl libencode-locale-perl liberror-perl libeval-closure-perl libexception-class-perl
  libexpat1-dev libexporter-tiny-perl libfcgi-perl libfido2-1 libfile-basedir-perl libfile-chdir-perl libfile-copy-recursive-perl libfile-desktopentry-perl libfile-dirlist-perl
  libfile-fcntllock-perl libfile-find-rule-perl libfile-fnmatch-perl libfile-homedir-perl libfile-listing-perl libfile-pushd-perl libfile-slurp-perl libfile-stripnondeterminism-perl
  libfile-touch-perl libfile-which-perl libfont-afm-perl libfont-ttf-perl libfontconfig1 libfreezethaw-perl libgail-common libgail18 libgconf-2-4 libgd3 libgdal28 libgdk-pixbuf-2.0-0
  libgdk-pixbuf-xlib-2.0-0 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-bin libgeo-ip-perl libgeotiff5 libgetopt-long-descriptive-perl libgfapi0 libgfrpc0 libgfxdr0 libgirepository-1.0-1 libgit-wrapper-perl
  libgitlab-api-v4-perl libglib2.0-0 libglib2.0-bin libglusterfs0 libgnustep-base1.27 libgpgme11 libgs9 libgsasl7 libgssapi-krb5-2 libgssapi-perl libgssrpc4 libgstreamer1.0-0 libgtk-3-0
  libgtk-3-bin libgtk-3-common libgtk2.0-0 libgtk2.0-bin libgts-0.7-5 libgts-bin libgvc6 libharfbuzz0b libhash-fieldhash-perl libhtml-form-perl libhtml-format-perl libhtml-html5-entities-perl
  libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhtml-tree-perl libhttp-cookies-perl libhttp-daemon-perl libhttp-date-perl libhttp-message-perl libhttp-negotiate-perl
  libhttp-tiny-multipart-perl libibverbs1 libicu-dev libimage-magick-perl libimage-magick-q16-perl libimport-into-perl libintl-perl libintl-xs-perl libio-html-perl libio-multiplex-perl
  libio-prompter-perl libio-pty-perl libio-sessiondata-perl libio-socket-inet6-perl libio-socket-ssl-perl libio-string-perl libio-stringy-perl libipc-run-perl libipc-run3-perl libipc-shareable-perl
  libipc-signal-perl libipc-system-simple-perl libisc-export1100 libisc-export1105 libiterator-perl libiterator-util-perl libjson-glib-1.0-0 libjson-maybexs-perl libjson-perl libjson-xs-perl
  libkadm5clnt-mit12 libkadm5srv-mit12 libkdb5-10 libkmod2 libkrb5-3 liblavfile-2.2-0 liblavplay-2.2-0 liblensfun1 liblist-compare-perl liblist-moreutils-perl liblist-moreutils-xs-perl
  liblist-someutils-perl liblist-someutils-xs-perl liblist-utilsby-perl liblocal-lib-perl liblocale-gettext-perl liblog-any-adapter-screen-perl liblog-any-perl liblog-dispatch-perl
  liblog-log4perl-perl liblqr-1-0 libltdl-dev liblwp-mediatypes-perl liblwp-protocol-https-perl libmagickcore-6.q16-6 libmagickcore-6.q16-6-extra libmagickwand-6.q16-6
  libmail-authenticationresults-perl libmail-dkim-perl libmail-sendmail-perl libmail-spf-perl libmailtools-perl libmailutils7 libmariadb3 libmath-random-isaac-perl libmath-random-isaac-xs-perl
  libmediainfo0v5 libmime-lite-perl libmime-tools-perl libmime-types-perl libmldbm-perl libmldbm-sync-perl libmodule-build-perl libmodule-cpanfile-perl libmodule-find-perl
  libmodule-implementation-perl libmodule-runtime-perl libmodule-scandeps-perl libmodule-signature-perl libmono-corlib4.5-cil libmono-corlib4.5-dll libmono-i18n-west4.0-cil libmono-i18n4.0-cil
  libmono-security4.0-cil libmono-system-configuration4.0-cil libmono-system-core4.0-cil libmono-system-numerics4.0-cil libmono-system-security4.0-cil libmono-system-xml4.0-cil
  libmono-system4.0-cil libmoo-perl libmoox-aliases-perl libmoox-handlesvia-perl libmoox-struct-perl libmount1 libmouse-perl libmro-compat-perl libmsv1 libnamespace-autoclean-perl
  libnamespace-clean-perl libncurses-dev libncurses5-dev libneon27-gnutls libnet-cidr-lite-perl libnet-cidr-perl libnet-dns-fingerprint-perl libnet-dns-perl libnet-dns-sec-perl
  libnet-domain-tld-perl libnet-http-perl libnet-ip-perl libnet-ldap-perl libnet-libidn-perl libnet-patricia-perl libnet-server-perl libnet-smtp-ssl-perl libnet-snmp-perl libnet-ssleay-perl
  libnetaddr-ip-perl libnetcdf18 libnetsnmptrapd40 libnids1.21 libnode72 libnsl-dev libnsl2 libnss-mdns libnss-nis libnss-nisplus libnss-systemd libnumber-compare-perl libnumber-range-perl
  libobject-id-perl libopendkim11 libopenjfx-java libopenjfx-jni libossp-uuid-perl libpackage-stash-perl libpackage-stash-xs-perl libpackagekit-glib2-18 libpadwalker-perl libpam-modules
  libpam-modules-bin libpam-runtime libpam-systemd libpam0g libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpaper-utils libpaper1 libparams-classify-perl libparams-util-perl
  libparams-validate-perl libparams-validationcompiler-perl libparse-pmfile-perl libpath-iterator-rule-perl libpath-tiny-perl libpcre2-dev libperl4-corelibs-perl libperl5.32 libperlio-gzip-perl
  libpod-constants-perl libpod-markdown-perl libpod-parser-perl libpod-readme-perl libpolkit-agent-1-0 libpolkit-gobject-1-0 libpoppler102 libpq5 libproc-processtable-perl libproc-waitstat-perl
  libproj19 libpulse0 libpython2.7-stdlib libpython3-dev libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib libpython3.9 libpython3.9-dev libpython3.9-minimal libpython3.9-stdlib
  libqt5core5a libqt5dbus5 libqt5network5 libqt5xml5 libquicktime2 librabbitmq4 librados2 librdmacm1 libre-engine-re2-perl libreadline7 libreadline8 libreadonly-perl libref-util-perl
  libref-util-xs-perl libregexp-assemble-perl libregexp-pattern-license-perl libregexp-pattern-perl librest-0.7-0 librole-tiny-perl librrd8 librrds-perl librsvg2-2 librsvg2-common libruby2.7
  libsasl2-modules libsdl1.2debian libsdl2-2.0-0 libsecret-1-0 libselinux1 libsemanage1 libsereal-decoder-perl libsereal-encoder-perl libserf-1-1 libsmbclient libsnmp-perl libsnmp-session-perl
  libsnmp40 libsoap-lite-perl libsocket6-perl libsoftware-license-perl libsort-key-perl libsort-naturally-perl libsort-versions-perl libsoup-gnome2.4-1 libsoup2.4-1 libspatialite7 libspecio-perl
  libssh-gcrypt-4 libssl-dev libssl1.1 libstdc++-10-dev libstrictures-perl libstring-approx-perl libstring-copyright-perl libstring-escape-perl libstring-shellquote-perl libsub-exporter-perl
  libsub-exporter-progressive-perl libsub-identify-perl libsub-install-perl libsub-name-perl libsub-override-perl libsub-quote-perl libsvn1 libsys-cpuaffinity-perl libsys-hostname-long-perl
  libtask-weaken-perl libtcl8.6 libterm-readkey-perl libtest-fatal-perl libtext-charwidth-perl libtext-glob-perl libtext-iconv-perl libtext-levenshteinxs-perl libtext-markdown-discount-perl
  libtext-template-perl libtext-unidecode-perl libtext-wrapi18n-perl libtext-xslate-perl libtheora0 libtie-cache-perl libtime-duration-perl libtime-moment-perl libtimedate-perl libtirpc-dev
  libtirpc3 libtk8.6 libtool libtorrent-rasterbar10 libtry-tiny-perl libtsk19 libtype-tiny-perl libtype-tiny-xs-perl libtypes-serialiser-perl libunicode-utf8-perl libunix-syslog-perl
  liburi-escape-xs-perl liburi-perl libuuid-perl libvariable-magic-perl libvte-2.91-0 libwant-perl libwbclient0 libwireshark14 libwiretap11 libwrap0 libwsutil12 libwww-perl libwww-robotrules-perl
  libxdelta2 libxerces-c3.2 libxft2 libxml-libxml-perl libxml-namespacesupport-perl libxml-parser-perl libxml-sax-base-perl libxml-sax-expat-perl libxml-sax-perl libxml-simple-perl libxml2-dev
  libxmlrpc-lite-perl libxslt1-dev libxstring-perl libyaml-libyaml-perl libyaml-tiny-perl libzip4 libzmq5 licensecheck lilyterm links lintian linux-base linux-headers-5.10.0-19-amd64
  linux-headers-5.10.0-20-amd64 linux-headers-amd64 linux-image-5.10.0-19-amd64 linux-image-5.10.0-20-amd64 linux-image-amd64 linux-kbuild-5.10 lm-sensors locales logcheck login logrotate logtail
  lsb-release lsof ltrace mailcap mailutils makepasswd makepkg man-db mariadb-client-10.5 mariadb-client-core-10.5 maybe mc mdadm mediainfo mencoder mime-construct mime-support minissdpd mjpegtools
  mkvtoolnix mlocate mokutil monit mono-4.0-gac mono-gac mono-runtime mono-runtime-common mono-runtime-sgen mount mplayer mrtg mrtg-contrib munin munin-common munin-node munin-plugins-core
  munin-plugins-extra mutt myspell-fr-gut ndiff needrestart net-tools netselect netselect-apt nftables nmap nodejs ntp ntpdate nut nut-cgi nut-client nut-server opendkim opendkim-tools openjade
  openjdk-11-jre openjdk-11-jre-headless openjfx openssh-client openssh-server openssh-sftp-server openssl os-prober packagekit packagekit-tools pacman-package-manager passwd patchutils pciutils
  perl perl-base perl-doc perl-modules-5.28 perl-modules-5.32 perl-openssl-defaults perl-tk perlmagick php php-apcu php-apcu-bc php-auth-sasl php-bz2 php-cgi php-cli php-common
  php-composer-ca-bundle php-composer-semver php-composer-spdx-licenses php-composer-xdebug-handler php-curl php-db php-dev php-fpdf php-gd php-getid3 php-google-recaptcha php-http php-http-request
  php-imap php-intl php-json php-json-schema php-ldap php-log php-mail php-mail-mime php-mariadb-mysql-kbs php-mbstring php-mdb2 php-mdb2-driver-mysql php-mysql php-net-dime php-net-smtp
  php-net-socket php-net-url php-net-url2 php-pear php-phpdbg php-phpmyadmin-motranslator php-phpmyadmin-shapefile php-phpmyadmin-sql-parser php-phpseclib php-propro php-pspell php-psr-cache
  php-psr-container php-psr-log php-psr-simple-cache php-raphf php-react-promise php-readline php-sabre-dav php-sabre-vobject php-soap php-symfony-cache php-symfony-cache-contracts
  php-symfony-config php-symfony-console php-symfony-debug php-symfony-dependency-injection php-symfony-expression-language php-symfony-filesystem php-symfony-finder php-symfony-polyfill-php80
  php-symfony-process php-symfony-service-contracts php-symfony-var-exporter php-symfony-yaml php-tcpdf php-twig php-twig-i18n-extension php-xml php-zip php7.4 php7.4-bz2 php7.4-cgi php7.4-cli
  php7.4-common php7.4-curl php7.4-dev php7.4-gd php7.4-imap php7.4-intl php7.4-json php7.4-ldap php7.4-mbstring php7.4-mysql php7.4-opcache php7.4-phpdbg php7.4-pspell php7.4-readline php7.4-soap
  php7.4-xml php7.4-zip phpmyadmin pigz pinentry-gtk2 pkg-php-tools pkgconf plowshare po-debconf policykit-1 poppler-data poppler-utils pristine-tar privoxy procps proftpd-basic proftpd-core
  proftpd-mod-crypto proftpd-mod-wrap proj-bin psutils python-pip-whl python2.7 python3 python3-acme python3-appdirs python3-apt python3-asn1crypto python3-audit python3-blessings python3-blinker
  python3-bs4 python3-certbot python3-certifi python3-cffi-backend python3-chardet python3-configargparse python3-configobj python3-cryptography python3-cycler python3-dateutil python3-dbus
  python3-debconf python3-debian python3-debianbts python3-decorator python3-dev python3-distlib python3-distro python3-distro-info python3-distutils python3-dnspython python3-docutils
  python3-entrypoints python3-filelock python3-future python3-gdal python3-gi python3-gpg python3-html5lib python3-httplib2 python3-icu python3-idna python3-importlib-metadata python3-ipy
  python3-jeepney python3-josepy python3-jwt python3-keyring python3-keyrings.alt python3-kiwisolver python3-ldb python3-lib2to3 python3-lxml python3-magic python3-markdown python3-matplotlib
  python3-minimal python3-mock python3-more-itertools python3-networkx python3-numpy python3-oauthlib python3-olefile python3-openssl python3-parsedatetime python3-pbr python3-pexpect python3-pil
  python3-pip python3-pkg-resources python3-ply python3-ptrace python3-ptyprocess python3-pycryptodome python3-pycurl python3-pydot python3-pygments python3-pygraphviz python3-pyinotify
  python3-pyparsing python3-pysimplesoap python3-renderpm python3-reportbug python3-reportlab python3-reportlab-accel python3-requests python3-requests-oauthlib python3-requests-toolbelt
  python3-rfc3339 python3-roman python3-samba python3-scipy python3-secretstorage python3-semanage python3-setools python3-setuptools python3-simplejson python3-six python3-software-properties
  python3-soupsieve python3-systemd python3-talloc python3-tdb python3-tk python3-twitter python3-tz python3-unidiff python3-urllib3 python3-venv python3-virtualenv python3-webencodings
  python3-wheel python3-xdg python3-yaml python3-zipp python3-zope.component python3-zope.event python3-zope.hookable python3-zope.interface python3.7-minimal python3.9 python3.9-dev
  python3.9-minimal python3.9-venv pyzor qbittorrent-nox qdbus-qt5 rake razor readline-common reportbug rkhunter rpm rpm-common rpm2cpio rrdtool rsync rsyslog ruby ruby-dev ruby-dnsruby ruby-libxml
  ruby-rubygems ruby-simpleidn ruby-uconv ruby-unf ruby-unf-ext ruby-xmlparser ruby2.7 ruby2.7-dev rubygems-integration sa-compile samba samba-common samba-common-bin samba-dsdb-modules samba-libs
  samba-vfs-modules sasl2-bin sed sendmail sendmail-base sendmail-bin sensible-mda setools sgml-base sgml-data shared-mime-info shim-signed shim-signed-common shtool sleuthkit smartmontools
  smbclient snmp snmpd sntp socat software-properties-common spamassassin spamc sqlite sqlite3 squid ssh ssh-audit ssl-cert ssldump stunnel4 sublime-text subversion sudo swaks sysbench sysstat
  systemd systemd-sysv tar task-english tasksel tasksel-data tcl tcl-expect tcl8.6 tcpd tcpdump tcpflow tesseract-ocr tex-common texinfo tftpd tiger tix tk8.6-blt2.5 tor tor-geoipdb
  transmission-cli transmission-daemon tripwire tshark txt2pdbdoc tzdata ucf udev ufraw-batch unar unattended-upgrades unhide.rb update-inetd util-linux util-linux-locales uuid-runtime vim
  vim-addon-manager vim-tiny virtualenv vlan vnstat vorbis-tools wamerican wfrench winbind wireshark-common x11-utils xdelta xfsprogs xinetd xml-core yui-compressor zlib1g-dev
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  apt adduser (due to apt) coreutils libselinux1 (due to coreutils) dash dpkg (due to dash) debconf (due to dash) tar (due to dpkg) findutils grep install-info (due to grep) gzip init
  systemd-sysv (due to init) init-system-helpers (due to init) perl-base (due to init-system-helpers) login libpam0g (due to login) libpam-runtime (due to login) libpam-modules (due to login) sed
  util-linux libmount1 (due to util-linux)
0 upgraded, 0 newly installed, 1182 to remove and 0 not upgraded.
After this operation, 4,022 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
--

--
❯ dpkg -l |grep apparmor
ii  apparmor                             2.13.6-10                                              amd64        user-space parser utility for AppArmor
ii  libapparmor1:amd64                   2.13.6-10                                              amd64        changehat AppArmor library
--

--
❯ apt-get remove apparmor libapparmor1

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done

The following packages were automatically installed and are no longer required:
  libappstream4 libavahi-core7 libcolord2 libdaemon0 libdconf1 libepoxy0 libexttextcat-2.0-0 libexttextcat-data libglib2.0-bin libgstreamer1.0-0 libjson-glib-1.0-0 libjson-glib-1.0-common
  libpolkit-agent-1-0 libpolkit-gobject-1-0 libstemmer0d libvte-2.91-common python3-jeepney python3-pycryptodome
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  orphan-sysvinit-scripts sysvinit-core
Suggested packages:
  bootlogd
The following packages will be REMOVED:
  apparmor avahi-daemon dbus dbus-user-session dbus-x11 dconf-gsettings-backend dconf-service dovecot-core dovecot-gssapi dovecot-imapd dovecot-ldap dovecot-mysql dovecot-pgsql dovecot-pop3d
  dovecot-sieve dovecot-sqlite gconf2 glib-networking gsettings-desktop-schemas init libapparmor1 libgtk-3-0 libgtk-3-bin libgtk-3-common libnss-mdns libnss-systemd libopenjfx-java libopenjfx-jni
  libpam-systemd librest-0.7-0 libsoup-gnome2.4-1 libsoup2.4-1 libvte-2.91-0 lilyterm openjfx packagekit packagekit-tools policykit-1 python3-keyring python3-keyrings.alt python3-secretstorage
  sublime-text systemd systemd-sysv
The following NEW packages will be installed:
  orphan-sysvinit-scripts sysvinit-core
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  init systemd-sysv (due to init)
0 upgraded, 2 newly installed, 44 to remove and 3 not upgraded.
Need to get 168 kB of archives.
After this operation, 245 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?]
--

Selinux is in permissive mode:

--
root in /etc/selinux took 38s
❯ cat config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=permissive
#SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# default - equivalent to the old strict and targeted policies
# mls     - Multi-Level Security (for military and educational use)
# src     - Custom policy built from source
SELINUXTYPE=default

# SETLOCALDEFS= Check local definition changes
SETLOCALDEFS=0
--

[Freyja]

I've tried they sent me back here :/

[Freyja]

I need to look deeper in your recommendations, I will try them as soon as I can.

[debia...@howorth.org.uk]

You asked about something slightly different. Given what we know now it
might be worth asking again, specifically about this question of how to
diagnose journald not starting given there are no logs as a consequence.

[Freyja]

Hi everyone and for people dropping by having the same issue.

The problem is *solved* !!!! \o/

The issue was: missing /run/systemd/journal/socket
What solved the problem:
- apt-get install --reinstall  systemd systemd-sysv libsystemd0 dbus-user-session
- /run/systemd/journal/socket and /run/systemd/journal/ were re-created
- reboot

I'm not completely sure why it didn't work the first time, but now I don't have ANY failed services.

--
systemctl list-units --state failed

  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.
--

But it might also be linked with a kernel update

Many thanks to all people involved in this thread and sorry that it took me so long to answer, I was missing the energy to look deeper.
Today I just rebooted after a kernel update (5.10.0-20) and the problem was gone...