binfmt-support cannot start at boot time because /var is not mounted

Dmitry Katsubo

unread,

May 11, 2020, 3:40:04 AM5/11/20

to

Dear Debian users,

I am stuck with the following problem which I failed to debug on my Debian 10.3.

In my setup I have /var as mounted volume:

=== /etc/fatab ===
UUID=83a3cb60-3334-4d11-9fdf-70b8e8703167 /var btrfs noatime,nodev,nosuid,subvol=var
=== end ===

Unfortunately sometimes I notice that binfmt-support is tried to be started "before" /var is mounted. Once I login and restart the service (I don't mount and/or repair anything), it starts just fine.

=== console ===
root@debian:~ # systemctl status binfmt-support
* binfmt-support.service - Enable support for additional executable binary formats
Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2020-05-10 21:54:27 CEST; 10h ago
Docs: man:update-binfmts(8)
Process: 353 ExecStart=/usr/sbin/update-binfmts --enable (code=exited, status=2)
Main PID: 353 (code=exited, status=2)

May 10 21:54:27 debian update-binfmts[353]: update-binfmts: unable to open /var/lib/binfmts: No such file or directory

root@debian:~ # ll /var/lib/binfmts
total 16
drwxr-xr-x 1 root root 50 Mar 17 23:46 .
drwxr-xr-x 1 root root 988 Mar 2 01:57 ..
-rw-r--r-- 1 root root 49 Apr 14 2019 jar
-rw-r--r-- 1 root root 59 Mar 17 23:46 python3.7

root@debian:~ # systemctl start binfmt-support
root@debian:~ # systemctl status binfmt-support
* binfmt-support.service - Enable support for additional executable binary formats
Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2020-05-11 09:01:04 CEST; 4s ago
Docs: man:update-binfmts(8)
Process: 9683 ExecStart=/usr/sbin/update-binfmts --enable (code=exited, status=0/SUCCESS)
Main PID: 9683 (code=exited, status=0/SUCCESS)

May 11 09:01:04 debian systemd[1]: Starting Enable support for additional executable binary formats...
May 11 09:01:04 debian systemd[1]: Started Enable support for additional executable binary formats.

root@debian:~ # dpkg -l | grep -i systemd
ii systemd 241-7~deb10u3 amd64 system and service manager
...
=== end ===

One time the system was started with even more units failed because of inaccessible /var. There is nothing special in these units, and I haven't modified any *.service files. In other words the
problem is floating, as number of failed unit varies from boot to boot.

The difficulty is that I don't know how to debug system start / systemd at such early stage of system boot.

Any help is appreciated.

--
With best regards,
Dmitry

Reco

unread,

May 11, 2020, 3:50:04 AM5/11/20

to

Hi.

On Mon, May 11, 2020 at 09:33:59AM +0200, Dmitry Katsubo wrote:

> root@debian:~ # systemctl status binfmt-support
> * binfmt-support.service - Enable support for additional executable binary formats
> Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sun 2020-05-10 21:54:27 CEST; 10h ago
> Docs: man:update-binfmts(8)
> Process: 353 ExecStart=/usr/sbin/update-binfmts --enable (code=exited, status=2)
> Main PID: 353 (code=exited, status=2)
>
> May 10 21:54:27 debian update-binfmts[353]: update-binfmts: unable to open /var/lib/binfmts: No such file or directory
>

> Any help is appreciated.

This should help your problem:

mkdir /etc/systemd/system/binfmt-support.service.d

cat > /etc/systemd/system/binfmt-support.service.d/override.conf << EOF
[Unit]
RequiresMountsFor=/var
EOF

Reco

Dmitry Katsubo

unread,

May 17, 2020, 8:50:04 PM5/17/20

to

On 2020-05-11 20:11, Darac Marjal wrote:

> As another alternative, one can run "systemctl edit
> binfmt-support.service", which will create the intervening folders and
> files for you, and reload the daemon if the editor exits with success.

Thanks for suggestion! I have tried the advise and it actually worked
(I will keep an eye on that because one reboot may not be representative).
I wonder nevertheless what is the problem with this specific unit? It has
dependency on local-fs.target which in turn should mount /var. So what
exactly went wrong?

Sven Joachim

unread,

Apr 22, 2021, 12:30:05 PM4/22/21

to

On 2021-04-22 14:26 +0200, Dmitry Katsubo wrote:

> Dear Debian community,
>
> I have noticed that all failed services were missing some directories under /run directory. I checked the service which is supposed to create them:
>
> * systemd-tmpfiles-setup.service - Create Volatile Files and Directories
> Loaded: loaded (/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: enabled)
> Active: inactive (dead)
> Docs: man:tmpfiles.d(5)
> man:systemd-tmpfiles(8)
>
> systemd[1]: sysinit.target: Found ordering cycle on systemd-tmpfiles-setup.service/start
> systemd[1]: sysinit.target: Found dependency on local-fs.target/start
> systemd[1]: sysinit.target: Found dependency on zram-...@zram1.service/start
> systemd[1]: sysinit.target: Found dependency on sysinit.target/start
> systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start

This is obviously bad, since many files and directories under /run will
be missing.

> and it looks like there is problem in zram-...@zram1.service which I configured like that:
>
> [Unit]
> Requires=dev-%i.device
> After=dev-%i.device
> Before=local-fs.target
> [Install]
> WantedBy=local-fs.target
>
> # systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After local-fs.target
> Requires=home.mount -.mount var.mount
> Requisite=
> Wants=systemd-fsck-root.service zram-...@zram0.service zram-...@zram1.service systemd-remount-fs.service
> BindsTo=
> PartOf=
> Before=binfmt-support.service sysinit.target systemd-machine-id-commit.service systemd-tmpfiles-setup.service networking.service systemd-tmpfiles-clean.service console-setup.service
> After=run-user-1000.mount zram-...@zram0.service root.mount -.mount tmp.mount zram-...@zram1.service systemd-fsck-root.service systemd-remount-fs.service home.mount var.mount local-fs-pre.target
>
> Even though I don't see any conflict with dependencies, it is confusing systemd.
>
> Any idea concerning how to fix that is welcomed.

I think you need to add a line with

DefaultDependencies=no

to the [Unit] section of your .service file. That is because by
default, all services have an implicit dependency on sysinit.target (see
systemd.target(5)), and sysinit.target depends on local-fs.target:

,----
| $ systemctl show -p WantedBy local-fs.target
| WantedBy=sysinit.target
`----

HTH,
Sven

Kenneth Parker

unread,

Apr 22, 2021, 12:30:08 PM4/22/21

to

On Thu, Apr 22, 2021, 8:26 AM Dmitry Katsubo <dm...@mail.ru> wrote:

Dear Debian community,

I have noticed that all failed services were missing some directories under /run directory. I checked the service which is supposed to create them:

* systemd-tmpfiles-setup.service - Create Volatile Files and Directories
Loaded: loaded (/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: enabled)
Active: inactive (dead)
Docs: man:tmpfiles.d(5)
man:systemd-tmpfiles(8)

systemd[1]: sysinit.target: Found ordering cycle on systemd-tmpfiles-setup.service/start
systemd[1]: sysinit.target: Found dependency on local-fs.target/start
systemd[1]: sysinit.target: Found dependency on zram-...@zram1.service/start
systemd[1]: sysinit.target: Found dependency on sysinit.target/start
systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start

and it looks like there is problem in zram-...@zram1.service which I configured like that:

[Unit]
Requires=dev-%i.device
After=dev-%i.device
Before=local-fs.target
[Install]
WantedBy=local-fs.target

# systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After local-fs.target
Requires=home.mount -.mount var.mount
Requisite=
Wants=systemd-fsck-root.service zram-...@zram0.service zram-...@zram1.service systemd-remount-fs.service
BindsTo=
PartOf=
Before=binfmt-support.service sysinit.target systemd-machine-id-commit.service systemd-tmpfiles-setup.service networking.service systemd-tmpfiles-clean.service console-setup.service
After=run-user-1000.mount zram-...@zram0.service root.mount -.mount tmp.mount zram-...@zram1.service systemd-fsck-root.service systemd-remount-fs.service home.mount var.mount local-fs-pre.target

Even though I don't see any conflict with dependencies, it is confusing systemd.

Any idea concerning how to fix that is welcomed.

Is Debian using Zram now? (I will need to check my Bullseye system when I get home).

A couple of years (or so) ago, Ubuntu inflicted Zram on me without advanced notice, on a system with only 4G Ram, leaving me with nothing to run my system on. Needless to say, after I troubleshooted it, I banned Zram on any future Ubuntu systems.

So is Debian "sneaking" Zram on us, or do you have to select it yourself?

On 2020-06-29 01:37, Dmitry Katsubo wrote:
> Dear Debian community,
>

> I hit the similar problem but this time with /run folder. Few services have
> failed to start:
>
> # systemctl status php7.0-fpm.service
> Jun 24 23:09:48 debian php-fpm7.0[893]: [24-Jun-2020 23:09:48] ERROR: unable to bind listening socket for address '/run/php/php7.0-fpm.sock': No such file or directory (2)
> Jun 24 23:09:48 debian php-fpm7.0[893]: [24-Jun-2020 23:09:48] ERROR: FPM initialization failed
> Jun 24 23:09:48 debian systemd[1]: php7.0-fpm.service: Main process exited, code=exited, status=78/CONFIG
> Jun 24 23:09:48 debian systemd[1]: php7.0-fpm.service: Failed with result 'exit-code'.
> Jun 24 23:09:48 debian systemd[1]: Failed to start The PHP 7.0 FastCGI Process Manager.
>
> # systemctl status motioneye.service
> Jun 24 23:09:47 debian systemd[1]: Started motionEye Server.
> Jun 24 23:09:48 debian meyectl[895]: INFO: hello! this is motionEye server 0.41
> Jun 24 23:09:48 debian meyectl[895]: CRITICAL: pid directory "/run/motioneye" does not exist or is not writable
> Jun 24 23:09:48 debian systemd[1]: motioneye.service: Main process exited, code=exited, status=255/EXCEPTION
> Jun 24 23:09:48 debian systemd[1]: motioneye.service: Failed with result 'exit-code'.
>
> # cat /etc/tmpfiles.d/php.conf
> d /run/php/sessions 1733 root root
>
> # cat /etc/tmpfiles.d/motioneye.conf
> d /run/motioneye 0750 motion motion
>
> Just after the boot I have inspected /run folder. It was created/mounted
> correctly and there have been a lot of files/directories there. I suspect that
> all services that have created the necessary directory under /run were able to
> start normally. Few of them which relied on existence of specific directory,
> have failed to started. After I have replayed the corresponding instructions for
> tmpfiles.d, the services have started normally.
>
> I have a feeling that systemd-tmpfiles was executed before /run was mounted.
>
> Needless to note that the problem is not persistent: sometimes OS boots without
> a single failed service.
>
> How can I debug the problem?
>
> Thank you!

Kenneth Parker

Dan Ritter

unread,

Apr 22, 2021, 12:40:04 PM4/22/21

to

Kenneth Parker wrote:
> Is Debian using Zram now? (I will need to check my Bullseye system when I
> get home).
>

> So is Debian "sneaking" Zram on us, or do you have to select it yourself?

It's an option, not a default.

-dsr-

Kenneth Parker

unread,

Apr 22, 2021, 1:00:08 PM4/22/21

to

Awesome.

Kenneth Parker