Bug#854475: postfix: systemd needs postfix@.service to have "After=network.target"

[Marcus Jodorf]

Package: postfix
Version: 3.1.4-4
Followup-For: Bug #854475

Dear Maintainer,

I found a bug that is probably related to #854475 because it seems
also to be caused by postfix starting too early.

I have several systems where mail gets stuck in the queue with
"(Host or domain name not found. Name service error for name=XXXXXXXX
type=AAAA: Host not found, try again)".

Turns out /var/spool/postfix/etc/resolv.conf is empty, only containing
the empty template
"
Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
"
from resolvconf.

But /etc/resolv.conf itself contains proper nameserver entries as
configured in /etc/network/interfaces.

So outgoing emails are stuck in the queue until I restart postfix and
/var/spool/postfix/etc/resolv.conf gets updated.

With "After=network.target" applied, postfix start is delayed and
the postfix copy of resolv.conf contains the necessary entries.


Best regards,

Marcus Jodorf



-- System Information:
Debian Release: 9.0
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-2-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

[Christian Schrötter]

Are you sure network.target is enough?

network-online.target sounds much better for me.

--
With kind regards,
Christian Schrötter

[Marcus Jodorf]

On 04/15/2017 09:14 PM, Christian Schrötter wrote:
> Are you sure network.target is enough?
>
> network-online.target sounds much better for me.

Hi Christian,

No, I'm not really sure. Just tried it after finding #854475 thinking
that it might actually catch two problems at once.

Especially after stumbling previously over #847440 with resolvconf where
I expected the problem to be in the first place.

Now, in combination with the proposed change there, my guess would be
network.target could be sufficient.

Setting network.target with postfix at least with my two machines works
but maybe network-online.target might be the safe bet considering there
are many machines out there that run slower or faster.

Since I'm currently hunting down some similar race-conditions with
systemd machines (e.g. start of inn2) I can tell you with systemd
nothing is really easy. Start a machine 20 times and at least 2 times of
these you get something quite different.


Best regards,

Marcus Jodorf

[Bernhard Schmidt]

On Sat, Apr 15, 2017 at 11:42:37PM +0200, Marcus Jodorf wrote:

Hi,

is there anything I can do to get this fixed for Stretch? I can
pretty reliably reproduce this issue with all my upgraded Stretch
systems, neither of them can send mails after a reboot until Postfix is
getting restarted

An After=network.target should be pretty low-risk even this late in the
release cycle and should fix this issue.

Best Regards,
Bernhard

[Scott Kitterman]

postfix.service has After=network.target. Why isn't that enough (I have not
had a lot of time to investigate this, but I have not given on up getting a
fix into stretch)?

Scott K

[Bernhard Schmidt]

AFAICT (Ccing systemd maintainers for input) postfix.service is mostly
used for compatibility with sysv (namely have "service postfix
something" do something reasonably by having the postfix@.service being
PartOf=postfix.service), but the actual decision what instance to start
is done by the generator
(/lib/systemd/system-generators/postfix-instance-generator) which
directly "wants" the instance.

IOW, I don't think postfix@.service being PartOf=postfix.service (which
does have After=network.target) influences its ordering.

The other quick fix for Stretch would be a "After=resolvconf.service", I
think most incarnations of this bug seen to far have been related to that.

Bernhard

[Michael Biebl]

I'm missing context here. Can you go into more detail please? What exactly is the question here?

Regards,
Michael

>_______________________________________________
>Pkg-systemd-maintainers mailing list
>Pkg-systemd...@lists.alioth.debian.org
>http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

[Bernhard Schmidt]

Am 08.05.2017 um 16:18 schrieb Michael Biebl:

Hi,

> I'm missing context here. Can you go into more detail please? What exactly is the question here?

Sorry.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854475 is about
several issues with postfix being started before the network is up,
failing to bind specific IPs or copying an empty resolvconf managed
/etc/resolv.conf into its chroot on startup. This is happening since
postfix ships native systemd units.

Postfix ships

postfix.service
https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix.service/
(oneshot, ExecStart=/bin/true, After=network.target

postfix@.service
https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix%40.service/)
(forking, PartOf=postfix.service, NO After=network.target)

Additionally a snipped in postinst drops an override in /etc in the
standard case of having systemd-resolved installed (not necessarily
used) and the configuration being a somehow managed standard
configuration. It adds additional After= dependencies for
postfix.service on network-online.target and systemd-resolved.service

https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix.postinst/#L650
https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix.postinst/#L185

It also ships a generator that directly links postfix@ instances into
the WANTDIR

postfix-instance-generator
https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix-instance-generator/


Scott's question was why the After=network.target (and the additional
ordering statements added by the override) of postfix.service isn't
enough to ensure proper ordering on startup. If I understand this
correctly the generator will independently start the postfix@.service
instances. The PartOf=postfix.service does not affect ordering at all.


So for a proper fix the additional After= tweaks need to be added to
postfix@.service, not just postfix.service

Bernhard

[Andreas Henriksson]

Hello Scott, Bernhard,

On Sun, May 07, 2017 at 11:59:10PM -0400, Scott Kitterman wrote:
> On Sunday, May 07, 2017 10:55:58 PM Bernhard Schmidt wrote:

>> [...]

> postfix.service has After=network.target. Why isn't that enough (I have not
> had a lot of time to investigate this, but I have not given on up getting a
> fix into stretch)?

I think the main issue here is that people are confused about what
network.target actually is/means.

For example this snippet is part of the original bug report:
The file /lib/systemd/system/postfix@.service needs to have the line
"After=network.target" to make sure that all the network interfaces are raised
[...]

The above description is completely false!
This is *not* what network.target does/means!

For further explanation and suggestions for solutions see:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

Regards,
Andreas Henriksson

[Scott Kitterman]

In the postinst, if the debconf configuration is something other than none or
local, we create an override file with After=network-online.target, which is
what I think is needed in most of these cases. I did not add it to the
service file because the documentation I read cautioned against using it when
not absolutely required.

The big question that's left unanswered for me is do I need it for
postfix@.service too? Do I need to depend on some other service?

Advice appreciated.

Scott K

[Bernhard Schmidt]

Hi Andreas,

I'm well aware of this, but in this particular case network.target is
probably what we want.

It is ordered after networking.service which raises the interfaces
configured by ifupdown. After network.target the majority of "server"
installations will have statically configured interfaces present and a
usable resolv.conf. This is also true for systemd-networkd.

This might still be broken for people using NetworkManager, but I doubt
those will have postfix explicitly bound to a static IP address anyway.
And it won't be worse for them than the current situation.

Bernhard

[Bernhard Schmidt]

Am 08.05.2017 um 20:54 schrieb Scott Kitterman:

Hi Scott,

> In the postinst, if the debconf configuration is something other than none or
> local, we create an override file with After=network-online.target, which is
> what I think is needed in most of these cases. I did not add it to the
> service file because the documentation I read cautioned against using it when
> not absolutely required.
>
> The big question that's left unanswered for me is do I need it for
> postfix@.service too? Do I need to depend on some other service?

My proposal for stretch (minimal intrusive) would be to add

After=postfix.service

to postfix@.service.

This would (transitively) order postfix@.service after network.target,
which means users of ifupdown/systemd-networkd will have networking
configured at this point. I don't think we'd need network-online.target,
there is nothing in most postfix configurations that actually needs
network access to work on startup.

If I understand this correctly an additional benefit of ordering
postfix@.service after postfix.service is that "service postfix reload"
in various hooks will actually work when any instance is running.

I have tested this on top of 3.1.4-4 and it works for me.



I don't fully grasp the service override anyway.

if [ -f /lib/systemd/systemd-resolved ] && \
[ ! -f /etc/systemd/system/postfix.service.d/override.conf ] ; then
if [ "$mailer" != "No configuration" ] && [ "$mailer" != "Local
only" ]; then
echo "Adding systemd overrides to ensure network and name
services available."
add_service_override
fi
fi

/lib/systemd/systemd-resolved is part of systemd, so the first part
always evaluates to "true" if systemd is installed. If it wasn't adding
the override file would have no consequences, so why bother with the
conditional here?

Second, the override adds

[Unit]
After=network-online.target
After=systemd-resolved.service

I think network-online.target is unnecessary, as I'm not aware of any
common postfix configurations that actually need network access to
startup. At least the common case of postfix or SQL the connection is
only made when a mail is getting processed.

systemd-resolved is debatable I think. But I don't see a reason not to
add it to the master postfix.service, it's just ordering and will not
start systemd-resolved by itself.

Bernhard

[Scott Kitterman]

If you specify an IP address in inet_interfaces, postfix will fail to start
withhout the override to depend on network-online.target. We can quibble
about if that qualifies a common or not, but we got bug reports [1], so it's
common enough in my opinion.

You may well be correct about the systemd-resolved.service. I tried to work
on this bug last night, but failed to make useful progress beyond learning
that NetworkManager and systemd will conspire to restart postfix once
resolv.conf is available, so the specific problem in this bug isn't an issue
for an NM based system. At least on the system with NM installed, nothing I
tried (your suggestion or others) seemed to make a difference. I need to rip
NM out of my test system and see what I can figure out next.

Scott K

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844277

[Bernhard Schmidt]

Am 10.05.2017 um 17:46 schrieb Scott Kitterman:

Hi Scott,

>> I think network-online.target is unnecessary, as I'm not aware of any
>> common postfix configurations that actually need network access to
>> startup. At least the common case of postfix or SQL the connection is
>
> If you specify an IP address in inet_interfaces, postfix will fail to start
> withhout the override to depend on network-online.target. We can quibble
> about if that qualifies a common or not, but we got bug reports [1], so it's
> common enough in my opinion.
>

> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844277

Has anyone ever confirmed that this setting on postfix.service actually
fixes this issue? I see no communication in the bug about it, and to my
understanding adding _anything_ on postfix.service won't make a
difference (since the only dependencies relevant for startup are on
postfix@.service).

With ifupdown and resolvconf, and your override in place, I can still
reproduce Bug#844277 (it is easier to trigger by adding

pre-up /bin/sleep 5

to /etc/network/interfaces)

May 10 18:26:22 voip20 postmulti[638]: fatal: parameter inet_interfaces:
no local interface found for xx.xx.xx.xx
May 10 18:26:23 voip20 systemd[1]: postfix@-.service: Control process
exited, code=exited status=1
May 10 18:26:23 voip20 systemd[1]: Failed to start Postfix Mail
Transport Agent (instance -).
May 10 18:26:23 voip20 systemd[1]: postfix@-.service: Unit entered
failed state.
May 10 18:26:23 voip20 systemd[1]: postfix@-.service: Failed with result
'exit-code'.
May 10 18:26:27 voip20 kernel: [ 7.787458] vmxnet3 0000:0b:00.0 eth0:
intr type 3, mode 0, 2 vectors allocated
May 10 18:26:27 voip20 kernel: [ 7.787725] vmxnet3 0000:0b:00.0 eth0:
NIC Link is Up 10000 Mbps
May 10 18:26:29 voip20 systemd[1]: Started Raise network interfaces.


Adding the After=postfix.service into postfix@.service fixes this issue
for good (because postfix@.service <- postfix.service <- network.target
<- networking.service)

NM is a completely different beast, I think you may need to be
After=network-online.target here. But static IP with NetworkManager, and
having postfix bind to it, should really be seldom enough.

Bernhard

[Scott Kitterman]

I recall doing tests. It's possible I got it wrong. I am still trying to figure out the 'simpler' world of systemd.

Scott K

[Michael Biebl]

Hi everyone,

a couple of comments inline

Am 08.05.2017 um 16:47 schrieb Bernhard Schmidt:
> Am 08.05.2017 um 16:18 schrieb Michael Biebl:
>
> Hi,
>
>> I'm missing context here. Can you go into more detail please? What exactly is the question here?
>
> Sorry.
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854475 is about
> several issues with postfix being started before the network is up,
> failing to bind specific IPs or copying an empty resolvconf managed
> /etc/resolv.conf into its chroot on startup. This is happening since
> postfix ships native systemd units.
>
> Postfix ships
>
> postfix.service
> https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix.service/
> (oneshot, ExecStart=/bin/true, After=network.target

So postfix.service is a dummy service. You actually would want a target
to group several services together. I notice though, that your
postfix@.service has ReloadPropagatedFrom=postfix.service.
This does indeed not work for .targets atm [1]. So such a dummy service
is the best you can do.
That said, what you usually want is that
systemctl start postfix.service is only marked as started once all
instanced postfix@.service units have started up. For that
postfix@.service should have
Before=postfix.service.

> postfix@.service
> https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix%40.service/)
> (forking, PartOf=postfix.service, NO After=network.target)
>
> Additionally a snipped in postinst drops an override in /etc in the
> standard case of having systemd-resolved installed (not necessarily
> used) and the configuration being a somehow managed standard
> configuration. It adds additional After= dependencies for
> postfix.service on network-online.target and systemd-resolved.service

> https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix.postinst/#L650
> https://sources.debian.net/src/postfix/3.1.4-4/debian/postfix.postinst/#L185

It's rather odd to have that After= ordering generated via a drop-in and
have that applied to the dummy service. You really would want to apply
that to the unit starting the real service, especially given my comment
above that you want postfix@.service have a Before=postfix.service.

I probably also wouldn't bother generating the network dependency
dynamically. After all, a user could modify the postfix config without
using dpkg-reconfigure. So you probably just want to use that
unconditionally.

As for network.target vs network-online.target, please read
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ and man
systemd.special

network.target is a so called passive target which is activated once
your network configuration has started. It doesn't actually mean that
your interfaces have been configured.
After=network.target is thus mostly interesting on shutdown, when you
want to ensure your service is stopped before the network configuration
is stopped.
What you probably want is
After=network-online.target
Wants=network-online.target

network-online.target is an active target, which needs to be pulled in
explicitly by the consumer, thus the Wants=.

For ifupdown network.target and network-online.target are mostly the
same (it basically just runs ifup -a) and that only includes interfaces
of type auto. allow-hotplug is currently not hooked up in
network-online.target/network.target in any way.

For other network configuration systems, the situation is different,
take NetworkManager.

network.target for NetworkManager simply means, that
NetworkManager.service, ie. the NetworkManager daemon has started. It
doesn't say anything about configured interfaces which are dynamically
actived by NetworkManager
network-online.target for NetworkManager means that
NetworkManager-wait-online.service has run. This is a service which
waits until nm-online reports an active connection (see man nm-online).

I suppose one of the bug reporters was using ifupdown with interfaces of
type auto. That's why After=network.target was sufficient for him.

As for systemd-resolved, I can't really comment on that why that's
necessary. At least that comment is wrong for Debian though:
> # If using systemd without systemd-resolved, you're on your own.

We do *not* enable systemd-resolved in Debian by default, so I don't
think this actually fixes something in Debian, the situation might be
different for Ubuntu.

The comment in postinst says, that you need DNS resolution. That's what
usually nss-lookup.target is for, which translates to $named in SysV. I
notice that the old SysV init script has Required-Start: $named, which
would translate into a After=nss-lookup.target ordering.

With that all said, this would be my recommendation:

- debian/postfix.postinst: drop the postfix.service.d/override.conf and
make sure to remove that conffile on upgrades

- postfix.service:
Drop After=network.target

- postfix@.service:
Add the following
PartOf=postfix.service
Before=postfix.service
ReloadPropagatedFrom=postfix.service
After=network-online.target nss-lookup.target
Wants=network-online.target


- debian/postfix.postinst: drop the postfix.service.d/override.conf and
make sure to remove that conffile on upgrades


If you have further questions, please ask.

Regards,
Michael


[1] https://github.com/systemd/systemd/issues/710

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Michael Biebl]

Am 14.05.2017 um 17:02 schrieb Michael Biebl:
> - debian/postfix.postinst: drop the postfix.service.d/override.conf and
> make sure to remove that conffile on upgrades

Seems due to some C&P failure I duplicated this point. I wanted to
clarify that postfix.service.d/override.conf is not actually a conffile,
as it's generated dynamically.

As for my reasons to not generate this configuration dynamically:
- It's less opaque. I was puzzled for a moment where this drop-in was
coming from

- It makes it easier to override postfix@.service. Atm you e.g. don't
respect if the admin removed that drop-in deliberately and you always
recreate it.

- It's more robust in case of local modifications which don't use
dpkg-reconfigure.

- The penalty of pulling in network-online.target is simply that for the
local case postfix is started a bit later then necessary during boot.

- The behaviour is more consistent with the old SysV init script which
had Required-Start: $network (The $network LSB facility translates to
network-online.target)

Michael

[Nuno Paquete]

Hi guys.

I'm trying to unsubscribe this newsletter but I'm not getting success.

Can someone Help?

Many thanks in advance.

[Scott Kitterman]

Thank you very much for the clear and complete input. I've just uploaded a
fixed package that puts your suggestions into operation (modulo I screwed it
up).

Scott K