Re: networking.service fails

[Gary Dale]

On 2021-02-16 19:44, Dmitry Katsubo wrote:
> Dear Debian community,
>
> I am puzzled with the following problem. When my Debian 10.8 starts, the unit "networking.service" is
> marked as failed with the following reason:
>
> root@debian:~ # systemctl status networking.service
> *— networking.service - Raise network interfaces
> Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Tue 2021-02-16 08:56:16 CET; 5h 27min ago
> Docs: man:interfaces(5)
> Process: 691 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
> Main PID: 691 (code=exited, status=1/FAILURE)
>
> however network is working just fine. I looked into /usr/lib/systemd/system/networking.service where
>
> TimeoutStartSec=5min
>
> and also set a big timeout for br0 in /etc/network/interfaces:
>
> auto lo
> auto eth0
> auto eth1
> iface lo inet loopback
> auto br0
> iface br0 inet static
> ...
> bridge_ports eth0 eth1
> bridge_maxwait 60
>
> but still the error occurs each time. Relative dmesg logs are:
>
> Feb 16 08:56:16 debian systemd[1]: Starting Raise network interfaces...
> Feb 16 08:56:16 debian ifup[691]: ifup: unknown interface eth0
> Feb 16 08:56:16 debian ifup[691]: ifup: unknown interface eth1
> Feb 16 08:56:16 debian ifup[691]: Waiting for br0 to get ready (MAXWAIT is 60 seconds).
> Feb 16 08:56:16 debian systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
> Feb 16 08:56:16 debian systemd[1]: networking.service: Failed with result 'exit-code'.
> Feb 16 08:56:16 debian systemd[1]: Failed to start Raise network interfaces.
> Feb 16 08:56:16.113716 debian systemd-udevd[387]: Using default interface naming scheme 'v240'.
> Feb 16 08:56:16.113796 debian systemd-udevd[387]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
> Feb 16 08:56:16.113851 debian systemd-udevd[387]: Could not generate persistent MAC address for br0: No such file or directory
> Feb 16 08:56:16.115750 debian kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this
> Feb 16 08:56:16.115828 debian kernel: br0: port 1(eth0) entered blocking state
> Feb 16 08:56:16.115875 debian kernel: br0: port 1(eth0) entered disabled state
> Feb 16 08:56:16.115929 debian kernel: device eth0 entered promiscuous mode
> Feb 16 08:56:16.119800 debian kernel: r8169 0000:02:00.0: firmware: direct-loading firmware rtl_nic/rtl8168g-2.fw
> Feb 16 08:56:16.120198 debian kernel: Generic PHY r8169-200:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-200:00, irq=IGNORE)
> Feb 16 08:56:16.251795 debian kernel: br0: port 2(eth1) entered blocking state
> Feb 16 08:56:16.251990 debian kernel: br0: port 2(eth1) entered disabled state
> Feb 16 08:56:16.391879 debian kernel: br0: port 1(eth0) entered blocking state
> Feb 16 08:56:16.391913 debian kernel: br0: port 1(eth0) entered forwarding state
> Feb 16 08:56:16.516862 debian systemd[1]: Starting Hostname Service...
> Feb 16 08:56:16.539520 debian systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
> Feb 16 08:56:16.539612 debian systemd[1]: networking.service: Failed with result 'exit-code'.
> Feb 16 08:56:16.539994 debian systemd[1]: Failed to start Raise network interfaces.
> Feb 16 08:56:16.671750 debian kernel: br0: port 3(wlan0) entered blocking state
> Feb 16 08:56:16.671808 debian kernel: br0: port 3(wlan0) entered disabled state
> Feb 16 08:56:16.671844 debian kernel: device wlan0 entered promiscuous mode
> Feb 16 08:56:16.671878 debian kernel: br0: port 3(wlan0) entered blocking state
> Feb 16 08:56:16.671912 debian kernel: br0: port 3(wlan0) entered forwarding state
> Feb 16 08:56:16.683579 debian hostapd[879]: wlan0: interface state UNINITIALIZED->ENABLED
> Feb 16 08:56:16.683579 debian hostapd[879]: wlan0: AP-ENABLED
>
> Any ideas where can I take a look? Thanks in advance!


Debian/Busteris still using Network Manager not systemd to control the
network so I think the network.service shouldn't be used.

I don't know how many interfaces you have but you seem to be using a
bridge to at least one of them. The network bridge would bring up the
physical interface. The physical interface should be listed in
interfaces as something like:

    iface eth0 inet manual

The listing of eth0 and eth1 (and even br0) may be obsolete. Try doing
ifconfig as root to see what interfaces are actually7 active (since you
say the network is running). These days they are frequently something
like enp4s0.

[Reco]

Hi.

On Wed, Feb 17, 2021 at 01:44:35AM +0100, Dmitry Katsubo wrote:
> root@debian:~ # systemctl status networking.service

...

> Process: 691 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
> Main PID: 691 (code=exited, status=1/FAILURE)

...

> Any ideas where can I take a look? Thanks in advance!

Try running this:

ifdown -a --force
ifup -a -v

Last command should show you the source of the trouble.

Reco

[Andrei POPESCU]

On Mi, 17 feb 21, 00:01:01, Gary Dale wrote:
> On 2021-02-16 19:44, Dmitry Katsubo wrote:
> > Dear Debian community,
> >
> > I am puzzled with the following problem. When my Debian 10.8 starts, the unit "networking.service" is
> > marked as failed with the following reason:
> >
> > root@debian:~ # systemctl status networking.service
> > *— networking.service - Raise network interfaces
> > Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
> > Active: failed (Result: exit-code) since Tue 2021-02-16 08:56:16 CET; 5h 27min ago
> > Docs: man:interfaces(5)
> > Process: 691 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
> > Main PID: 691 (code=exited, status=1/FAILURE)
>

> Debian/Busteris still using Network Manager not systemd to control the
> network so I think the network.service shouldn't be used.

Well, systemd as init is starting everything so that necessarily
includes starting "the network", which in practice means starting
whatever network management framework is in use[1].

The 'networking.service' service is part of ifupdown, Debian's default
network management framework (Priority: important).

Network Manager is Priority: optional and typically installed as a
Depends/Recommends of Desktop Environments.

[1] this is applicable even for systemd's own network management
framework systemd-networkd, which is included in the 'systemd' Debian
package, but not activated by default.

Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

[Henning Follmann]

On Wed, Feb 17, 2021 at 01:44:35AM +0100, Dmitry Katsubo wrote:
> Dear Debian community,
>
> I am puzzled with the following problem. When my Debian 10.8 starts, the unit "networking.service" is
> marked as failed with the following reason:
>

[...]

>
> and also set a big timeout for br0 in /etc/network/interfaces:
>
> auto lo
> auto eth0
> auto eth1
> iface lo inet loopback
> auto br0
> iface br0 inet static
> ...
> bridge_ports eth0 eth1
> bridge_maxwait 60
>
> but still the error occurs each time. Relative dmesg logs are:
>
> Feb 16 08:56:16 debian systemd[1]: Starting Raise network interfaces...
> Feb 16 08:56:16 debian ifup[691]: ifup: unknown interface eth0
> Feb 16 08:56:16 debian ifup[691]: ifup: unknown interface eth1

Are you using eth0, eth1?
Or are you using predictable network names?
https://wiki.debian.org/NetworkInterfaceNames

> Feb 16 08:56:16 debian ifup[691]: Waiting for br0 to get ready (MAXWAIT is 60 seconds).
> Feb 16 08:56:16 debian systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
> Feb 16 08:56:16 debian systemd[1]: networking.service: Failed with result 'exit-code'.
> Feb 16 08:56:16 debian systemd[1]: Failed to start Raise network interfaces.
> Feb 16 08:56:16.113716 debian systemd-udevd[387]: Using default interface naming scheme 'v240'.
> Feb 16 08:56:16.113796 debian systemd-udevd[387]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.

[...]

-H


--
Henning Follmann | hfol...@itcfollmann.com

[Gary Dale]

Sorry, it was midnight when I replied. However the failure is likely
still due to the interfaces misconfiguration - probably reporting a
failure to raise a non-existent interface.

[Jeremy Ardley]

On 4/4/22 8:38 am, Dmitry Katsubo wrote:
> Dear Debian community,
>

> I've exercised several directions to solve the issue, however I've failed. Would be great if somebody can share his idea. After upgrading from Debian buster to bullseye I still have the same issue:
>
> # systemctl status networking.service

> — networking.service - Raise network interfaces
>      Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)

>      Active: failed (Result: exit-code) since Mon 2022-04-04 00:37:29 CEST; 31min ago
>        Docs: man:interfaces(5)
>     Process: 1287 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
>    Main PID: 1287 (code=exited, status=1/FAILURE)
>         CPU: 83ms
>
>
I recently went through something similar where networking.service got
obscure errors I couldn't diagnose.

I went against my better judgement and changed to using
systemd-networkd.service

Rather surprisingly I got a working network with no errors. You could
try that approach and see if you have any luck.
When you switch, you need to backup /etc/network/interfaces and copy
/etc/network/interfaces.default over /etc/network/interfaces.

I remain perplexed however why there are at least three different mostly
incompatible network management systems :

* systemd.networkd.service
* networking.service
* NetworkManager.service


--
Jeremy

[Charles Curley]

On Mon, 4 Apr 2022 09:50:19 +0800
Jeremy Ardley <jer...@ardley.org> wrote:

> * systemd.networkd.service
> * networking.service
> * NetworkManager.service

Possibly because they do different things.

I have no experience what the first one does. The latter two are not
mutually exclusive. Use networking.service to manage interfaces that
NetworkManager should not manage, such as the loopback device, lo. Use
NetworkManager for devices that may come and go, e.g. wlan0 and eth0.

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[Jeremy Ardley]

On 4/4/22 10:15 am, Charles Curley wrote:
> On Mon, 4 Apr 2022 09:50:19 +0800
> Jeremy Ardley <jer...@ardley.org> wrote:
>

>> * systemd-networkd.service

>> * networking.service
>> * NetworkManager.service
> Possibly because they do different things.
>
> I have no experience what the first one does.

systemd.networkd.service does everything that networking.service does,
but then adds on features such as an inbuilt dhcp4 and dhcpv6 server
plus it does ipv6 route advertising. The added features are optional and
you can still use your existing ISC dhcp severs and radvd if you want.

In my case I am delegating ipv6 prefixes that I obtain from my ISP and
also advertising them, making radvd redundant. I have kept my ISC dhcp
servers as I am doing DDNS with my bind instance. Possibly
systemd-networkd.service does DDNS as well but I haven't explored that.

--
Jeremy

[Jeremy Ardley]

On 4/4/22 10:31 am, Jeremy Ardley wrote:
>
>
> In my case I am delegating ipv6 prefixes that I obtain from my ISP and
> also advertising them, making radvd redundant. I have kept my ISC dhcp
> servers as I am doing DDNS with my bind instance. Possibly
> systemd-networkd.service does DDNS as well but I haven't explored that.
>

I had a brief look at a wiki
https://wiki.archlinux.org/title/systemd-networkd to see if DDNS is
supported.

It seems systemd-networkd also supports dynamic creation and removal of
interfaces like NetworkManager does.

I did see the wiki says you have to use systemd-resolved with
systemd-networkd which is rubbish. resolved is a pain and better
disabled. It's certinaly not necessary for systemd-networkd.

The authoritive source is
https://www.freedesktop.org/software/systemd/man/systemd.network.html
and there are plenty of tutorials online to assist.

--
Jeremy

[Greg Wooledge]

On Mon, Apr 04, 2022 at 02:38:39AM +0200, Dmitry Katsubo wrote:
> I've exercised several directions to solve the issue, however I've failed. Would be great if somebody can share his idea. After upgrading from Debian buster to bullseye I still have the same issue:

Upgrading from one release to another can easily trigger changes in
interface names, especially if you were previously relying on a deprecated
scheme.

> Apr  4 00:37:28 debian ifup[1287]: ifup: unknown interface eth0
> Apr  4 00:37:28 debian ifup[1287]: ifup: unknown interface eth1
> Apr  4 00:37:28 debian kernel: [    1.049742] r8169 0000:02:00.0: can't disable ASPM; OS doesn't have ASPM control
> Apr  4 00:37:28 debian kernel: [    1.066552] r8169 0000:02:00.0 eth0: RTL8168g/8111g, 00:17:e8:92:b7:77, XID 4c0, IRQ 30
> Apr  4 00:37:28 debian kernel: [    1.066553] r8169 0000:02:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
> Apr  4 00:37:28 debian kernel: [    1.066581] r8169 0000:03:00.0: can't disable ASPM; OS doesn't have ASPM control
> Apr  4 00:37:28 debian kernel: [   10.804450] r8169 0000:02:00.0 eth0: renamed from enp2s0
> Apr  4 00:37:28 debian kernel: [   10.825386] r8169 0000:03:00.0 eth1: renamed from enp3s0

So... you've got some wild stuff going on here.

You have two interfaces. One of them was named enp2s0 and then got
renamed to eth0, which is usually the opposite direction from what one
expects.

The other was named enp3s0 and then got renamed to eth1. Which, again,
is the opposite of what one expects.

Normally, the kernel assigns the name eth0 temporarily to the first
interface that it finds, and then renames it to some "predictable"
name according to obtuse schemes. In your case, I'm imagining that
your interface started as eth0, then got renamed to enp2s0, then renamed
a second time back to eth0.

At the time when ifup tried to run, the interface was named enp2s0, but
your configuration tried to address it by the name eth0.

So, there are many questions here.

How many different interface configuration schemes are you using here?
Which ones are they?

a) Do you set the net.ifnames kernel parameter? If so, what value are
you setting it to?

b) Do you have any non-loopback interfaces configured in the
/etc/network/interfaces file? If so, which ones, and how are
they configured?

c) Do you have an /etc/udev/rules.d/70-persistent-net.rules file? Note
that this file was deprecated in buster. If it continued to work in
buster, that was just a happy accident. If it stopped working in
bullseye, that is not a surprise.

d) Do you have any /etc/systemd/network/*.link files? If so, what's in
them?

e) Do you have network-manager installed? If so, I don't know what
followup questions to ask about it, because I don't use it.

f) Do you have any /etc/systemd/network/*.network files? If so, what's
in those?

g) Do you have any *other* interface configuration schemes in play that
I don't know about?

Once you've identified all of the moving parts in this puzzle, then you
can figure out which ones you actually want to keep, and which ones
you should rip out.

For comparison, my system (Debian 11 bullseye) has one interface. I
configure its name using a file in /etc/systemd/network/ which sets
the name to lan0. Then I configure its in /etc/network/interfaces
using that name.

Some relevant log file lines:

Mar 26 08:01:54 unicorn kernel: [ 1.042579] r8169 0000:02:00.0 eth0: RTL8168gu/8111gu, 18:60:24:77:5c:ec, XID 509, IRQ 127
[...]
Mar 26 08:01:54 unicorn kernel: [ 1.057022] r8169 0000:02:00.0 lan0: renamed from eth0
[...]
Mar 26 08:01:59 unicorn kernel: [ 20.372606] r8169 0000:02:00.0 lan0: Link is Up - 100Mbps/Full - flow control rx/tx

That's just one of many possible ways to configure network interfaces.
You might be using a different scheme. The important thing is that
you pick *one* scheme and set it up correctly. If you've got two or
more competing schemes in play, and they're undoing each other's work,
that's not desirable.

[David Wright]

On Mon 04 Apr 2022 at 02:38:39 (+0200), Dmitry Katsubo wrote:

> On 2021-02-17 14:21, Henning Follmann wrote:
>
> > Are you using eth0, eth1?
> > Or are you using predictable network names?
> > https://wiki.debian.org/NetworkInterfaceNames

> Well, I use eth0/eth1 as I have renamed them from predictable network names via /etc/udev/rules.d/70-persistent-net.rules:
>
> ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="00:17:e8:92:b7:77", KERNEL=="eth*", NAME="eth0"
> ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="00:17:20:53:44:58", KERNEL=="eth*", NAME="eth1"

That is a really bad idea. If you're going to rename interfaces
yourself, then choose different names from anything that's already
used or could potentially be used.

Cheers,
David.

[Greg Wooledge]

It's also worth pointing out that /etc/udev/rules.d/70-persistent-net.rules
was deprecated in buster (Debian 10). According to the release notes,
it *may* work, or it may not. Users were instructed to migrate away
from it.

It would not surprise me one bit if there's a race condition which causes
the renaming done by 70-persistent-net.rules to occur at the wrong time,
if it even happens at all.

[Reco]

Hi.

It's pretty straightforward.

On Mon, Apr 04, 2022 at 02:38:39AM +0200, Dmitry Katsubo wrote:
> Dear Debian community,

First we have this - don't be confused by the "kernel's" timestamps,
systemd likes to add its own messages into the kernel's ringbuffer:

> Apr  4 00:37:28 debian kernel: [   10.804450] r8169 0000:02:00.0 eth0: renamed from enp2s0
> Apr  4 00:37:28 debian kernel: [   10.825386] r8169 0000:03:00.0 eth1: renamed from enp3s0

> Apr  4 00:37:28 debian kernel: [    1.082558] r8169 0000:03:00.0 eth1: RTL8168g/8111g, 00:17:20:53:44:58, XID 4c0, IRQ 32
> Apr  4 00:37:28 debian kernel: [    1.082560] r8169 0000:03:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]
> Apr  4 00:37:28 debian kernel: [    1.083615] r8169 0000:02:00.0 enp2s0: renamed from eth0
> Apr  4 00:37:28 debian kernel: [    1.108937] r8169 0000:03:00.0 enp3s0: renamed from eth1

And next we see this:

> # ifdown -a --force; ifup -a -v
...
> ifup: unknown interface eth0
> ifup: unknown interface eth1

> Well, I use eth0/eth1 as I have renamed them from predictable network names via /etc/udev/rules.d/70-persistent-net.rules:
>
> ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="00:17:e8:92:b7:77", KERNEL=="eth*", NAME="eth0"
> ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="00:17:20:53:44:58", KERNEL=="eth*", NAME="eth1"

As /var/log/messages helpfully show, your udev rules work.
The problem is, next thing udev does is renaming your network interfaces
back to (Un)Predictable Naming™ scheme.

Thus whatever stanzas you have in your interfaces(5) about eth0 and eth1
fail, thus the whole networking.service fails.


The conclusion is simple too:

1) Remove 70-persistent-net.rules, it's not doing what it should anyway.

2) Either use (Un)Predictable Network names in your interfaces, such as
enp2s0 and enp3s0.

3) Or use systemd network link files to rename network interfaces.

4) Or add "net.ifnames=0" to kernel's cmdline, as others suggested.

Reco

[Reco]

Hi.

On Mon, Apr 11, 2022 at 01:48:35AM +0200, Dmitry Katsubo wrote:
> The configuration is trivial: it adds both eth0 eth1 to the bridge
> br0.
>
> === cut /etc/network/interfaces ===

> auto lo
> auto eth0
> auto eth1
>
> iface lo inet loopback
>
> auto br0
> iface br0 inet static

> address 10.0.1.100
> gateway 10.0.1.1
> netmask 255.0.0.0

> bridge_ports eth0 eth1
> bridge_maxwait 60

> === cut ===

Good news - it explains "ifup: unknown interface eth0" messages.
Bad news - this /e/n/i is not valid.

The reason being - both eth0 and eth1 lack interface definitions, i.e.
have no "iface" stanzas.

If you absolutely need both eth0 and eth1 in the UP state by the time
you create and bring up br0 you should either:

1) Define both eth0 and eth1 in /e/n/i like this:

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto br0
iface br0 inet static

address 10.0.1.100
gateway 10.0.1.1
netmask 255.0.0.0

bridge_ports eth0 eth1
bridge_maxwait 60

2) Use pre-up and post-down hooks for br0, and remove both "auto eth0"
and "auto eth1":

auto br0
iface br0 inet static

address 10.0.1.100
gateway 10.0.1.1
netmask 255.0.0.0

bridge_ports eth0 eth1
bridge_maxwait 60

pre-up /sbin/ip link set eth0 up
pre-up /sbin/ip link set eth1 up
post-down /sbin/ip link set eth0 down
post-down /sbin/ip link set eth1 down

Reco

[Dmitry Katsubo]

Hooray, it worked! Thanks for pointing out the configuration error.
After fixing it everything worked like a charm. Would be very tricky
for me to locate the issue...

If the configuration is invalid, networking.service could probably
complain about that? I would be able then to resolve the issue myself.

Just for my record the reference to documentation:

https://wiki.debian.org/BridgeNetworkConnections#Configuring_bridging_in_.2Fetc.2Fnetwork.2Finterfaces

--
With best regards,
Dmitry