Advanced master.cf query/update support

[Wietse Venema]

I recently picked up work on the postconf command that I suspended
in January this year. It's probably best to just give a few examples.

First, a word about notation. I wanted to describe master.cf
properties with a kind of pathname notation. The original idea was
to have things like servicename.servicetype.whatever but that turned
out to be problematic. Services can have '.' in their name, and
therefore, the '.' can also appear in service-defined parameters.

So I switched to servicename/servicetype/whatever. The result will
be released as a non-production snapshot because the code still
needs to be burned in (looking for feedback on the user interface).
Expected release is Wedneday.

The first example shows the smtp/inet service in the traditional
form:

$ postconf -M smtp/inet
smtp inet n - n - - smtpd

[note to self: add an option to replace '-' with default values]

Without the smtp/inet the above command would enumerate all services
and that would be too much output ("postconf -M smtp" would list
all services called "smtp", that's smtp/inet and smtp/unix).

The postconf command can now enumerate the fields as follows:

$ postconf -F smtp/inet
smtp/inet/service = smtp
smtp/inet/type = inet
smtp/inet/private = n
smtp/inet/unprivileged = -
smtp/inet/chroot = n
smtp/inet/wakeup = -
smtp/inet/process_limit = -
smtp/inet/command = smtpd

This form makes it very easy to change one field in master.cf.
For example to turn on chroot for the smtp/inet service you use:

$ postconf -F smtp/inet/chroot=y

Moreover, you can specify "*" for service name, service type or
field as a wild-card match. For example, to turn off chroot on
all Postfix daemons, use this:

$ postconf -F '*/*/chroot=n'

For a second example, let's look at the submission service. This
service typically has multple "-o parameter=value" overrides.

The postconf command can enumerate these parameters as follows:

$ postconf -P submission
submission/inet/milter_macro_daemon_name = ORIGINATING
submission/inet/smtpd_client_restrictions = $mua_client_restrictions
submission/inet/smtpd_helo_restrictions = $mua_helo_restrictions
submission/inet/smtpd_recipient_restrictions =
permit_sasl_authenticated,reject
submission/inet/smtpd_reject_unlisted_recipient = no
submission/inet/smtpd_sasl_auth_enable = yes
submission/inet/smtpd_sender_restrictions = $mua_sender_restrictions
submission/inet/smtpd_tls_security_level = encrypt
submission/inet/syslog_name = postfix/submission

Again, this form makes it very easy to modify one parameter
setting, for example to change the smtpd_tls_security_level setting for
the submission/inet service:

$ postconf -P 'submission/inet/smtpd_tls_security_level=may'

You can also a new parametername=parametervalue setting:

$ postconf -P 'submission/inet/parametername=parametervalue'

To get a quick report of all parameter overrides in master.cf:

$ postconf -P

This can produce a lot of output so no example is provided here.

Wietse

[Patrick Ben Koetter]

* Wietse Venema <wie...@porcupine.org>:

> I recently picked up work on the postconf command that I suspended
> in January this year. It's probably best to just give a few examples.
>
> First, a word about notation. I wanted to describe master.cf
> properties with a kind of pathname notation. The original idea was
> to have things like servicename.servicetype.whatever but that turned
> out to be problematic. Services can have '.' in their name, and
> therefore, the '.' can also appear in service-defined parameters.
>
> So I switched to servicename/servicetype/whatever. The result will
> be released as a non-production snapshot because the code still
> needs to be burned in (looking for feedback on the user interface).
> Expected release is Wedneday.

Out of curiosity: Will you aim for the ability to create services too?

$ postconf -E delay/unix/command=smtp

This would create a service 'delay' with 'reasonable defaults' (and get around
the tricky part to specify all options in one command).

Another way to create a new service would be to specify all options:

$ postconf -E delay/unix/private=yes,unpriv=yes,chroot=yes,wakeup=never,maxproc=100,command=smtp

[ What could be a good way to tell Postfix to use 'defaults'? Using '-' on
command line sounds like an invitation to produce errors ]

Once in place, one could use "postconf -F delay/unix/parameter=option" to
configure it as required before putting it into use.

> The first example shows the smtp/inet service in the traditional
> form:
>
> $ postconf -M smtp/inet
> smtp inet n - n - - smtpd
>
> [note to self: add an option to replace '-' with default values]

Replace in postconf -M ... output or in master.cf?

> Without the smtp/inet the above command would enumerate all services
> and that would be too much output ("postconf -M smtp" would list
> all services called "smtp", that's smtp/inet and smtp/unix).
>
> The postconf command can now enumerate the fields as follows:
>
> $ postconf -F smtp/inet
> smtp/inet/service = smtp
> smtp/inet/type = inet
> smtp/inet/private = n
> smtp/inet/unprivileged = -
> smtp/inet/chroot = n
> smtp/inet/wakeup = -
> smtp/inet/process_limit = -
> smtp/inet/command = smtpd
>
> This form makes it very easy to change one field in master.cf.
> For example to turn on chroot for the smtp/inet service you use:
>
> $ postconf -F smtp/inet/chroot=y

ACK

> Moreover, you can specify "*" for service name, service type or
> field as a wild-card match. For example, to turn off chroot on
> all Postfix daemons, use this:
>
> $ postconf -F '*/*/chroot=n'

ACK

> For a second example, let's look at the submission service. This
> service typically has multple "-o parameter=value" overrides.
>
> The postconf command can enumerate these parameters as follows:
>
> $ postconf -P submission
> submission/inet/milter_macro_daemon_name = ORIGINATING
> submission/inet/smtpd_client_restrictions = $mua_client_restrictions
> submission/inet/smtpd_helo_restrictions = $mua_helo_restrictions
> submission/inet/smtpd_recipient_restrictions =
> permit_sasl_authenticated,reject
> submission/inet/smtpd_reject_unlisted_recipient = no
> submission/inet/smtpd_sasl_auth_enable = yes
> submission/inet/smtpd_sender_restrictions = $mua_sender_restrictions
> submission/inet/smtpd_tls_security_level = encrypt
> submission/inet/syslog_name = postfix/submission
>
> Again, this form makes it very easy to modify one parameter
> setting, for example to change the smtpd_tls_security_level setting for
> the submission/inet service:
>
> $ postconf -P 'submission/inet/smtpd_tls_security_level=may'

That would use the option "-P" to read and to write (edit) service parameters
in master.cf. For main.cf you have dedicated command line options for reading
(-d, -n) and for editing (-e).

Personally I think it is a good idea to use different options for reading and
writing and I would strive to build that for postconf master.cf manipulation
too - as a measure to make accidental changes harder, but also for easy to
remember main.cf/master.cf command line option symmetry.

Editing service specific parameters in master.cf might then work like this:

$ postconf -F submission/inet -E 'smtpd_sasl_auth_enable = yes'

Which would also allow to identify more than one service and add/modify an
option in one run:

$ postconf -F '*/*' -E 'something = useful'

Option -F would create the context to which the editing should apply and -E
would specify what to edit.

To remove parameters -R might be introduced. (And -r for main.cf too, doubling
-X, but keeping the command line option names symmetrical for
main.cf/master.cf.


p@rick

P.S.
Hope this makes sense. I will go and get some coffee now. ;)


--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

[Wietse Venema]

Thanks for feedback.

Background information: the advanced query/update methods are
primarily for programmatic use by system management tools. This
means the "user" interface needs to be highly consistent.

Ideally, postconf enumerates Postfix main.cf and master.cf details
as a list of "name=value" pairs, with names organized into structured
name spaces. This allows other programs to reason about how Postfix
works, and to make precise updates without having to worry about
the exact layout of master.cf files.

The current implementation still lacks support to for managing
daemon options (-v, -D) and for daemon non-option command-line
arguments. That can be added later.

Patrick Ben Koetter:

> * Wietse Venema <wie...@porcupine.org>:
> > I recently picked up work on the postconf command that I suspended
> > in January this year. It's probably best to just give a few examples.
> >
> > First, a word about notation. I wanted to describe master.cf
> > properties with a kind of pathname notation. The original idea was
> > to have things like servicename.servicetype.whatever but that turned
> > out to be problematic. Services can have '.' in their name, and
> > therefore, the '.' can also appear in service-defined parameters.
> >
> > So I switched to servicename/servicetype/whatever. The result will
> > be released as a non-production snapshot because the code still
> > needs to be burned in (looking for feedback on the user interface).
> > Expected release is Wedneday.
>
> Out of curiosity: Will you aim for the ability to create services too?

Yes, that was implemented last January. Adding a master.cf entry
is similar to adding a main.cf one.

> This would create a service 'delay' with 'reasonable defaults' (and get around
> the tricky part to specify all options in one command).

Unfortunately services such as qmgr, bounce, local have mandatory
master.cf settings that differ from built-in defaults. I think it
is not reasonable to build (i.e. duplicate) all that logic into
the postconf command.

[Just like master is a stupid daemon that knows nothing about
Postfix services, postconf knows almost nothing. The main
exception is that postconf knows about a few parameter names
that depend on master.cf service names, and that was done only
because it allows postconf to report mistakes that are otherwise
dificult to find. Mistakes in mandatory master.cf settings are
already reported by daemons themselves.]

Instead of adding a service from scratch, perhaps a more reasonable
approach is to clone an existing service (take a service that works,
and then duplicate its settings under a different service name).

Currently, to clone the smtp/unix service settings you could do this:

$ postconf -M smtp/unix
smtp unix - - n - - smtp

Then copy those fields (except the first field) to the existing or
new delay/unix service:

$ postconf -M delay/unix="delay unix - - n - - smtp"

This notation is similar to how "postconf parameter=value" replaces
or adds a main.cf entry.

To combine the above, and clone smtp/unix -> delay/unix in one command:

$ postconf -M delay/unix="`postconf -M smtp/unix|sed 's/smtp/delay/'`"

This is perhaps not super-convenient for manual cloning, but it
should be sufficient for programmatic configuration management.

> $ postconf -M delay/unix/command=smtp

This notation is already used to change a single service field, so
I think it is not suitable to manipulate (create or update) multiple
service fields.

> > Again, this form makes it very easy to modify one parameter
> > setting, for example to change the smtpd_tls_security_level setting for
> > the submission/inet service:
> >
> > $ postconf -P 'submission/inet/smtpd_tls_security_level=may'
>
> That would use the option "-P" to read and to write (edit) service parameters
> in master.cf. For main.cf you have dedicated command line options for reading
> (-d, -n) and for editing (-e).

FYI, The "-e" option has been optional since 20100715.

"postconf parametername" reads, and "postconf parametername=value"
updates. I'm using the same to read or update master.cf. When all
operations are similar there are fewer things to remember.

main.cf entry:
$ postconf smtpd_tls_security_level
$ postconf smtpd_tls_security_level=may
$ postconf -e smtpd_tls_security_level=may

master.cf entry:
$ postconf -M smtp/inet
$ postconf -M smtp/inet="smtp inet .... smtpd"
$ postconf -Me smtp/inet="smtp inet .... smtpd"

master.cf field:
$ postconf -F smtp/inet/chroot
$ postconf -F smtp/inet/chroot=n
$ postconf -Fe smtp/inet/chroot=n

master.cf service parameter:
$ postconf -P submission/inet/smtpd_tls_security_level
$ postconf -P submission/inet/smtpd_tls_security_level=may
$ postconf -Pe submission/inet/smtpd_tls_security_level=may

I also use the same options for the same operations on main.cf
entries, master.cf entries (-M), or on fields (-F) or parameters
(-P) in master.cf entries:

-x Expand $name in main.cf entry, or master.cf entry (-M) or parameter (-P).
-X Delete main.cf entry, or master.cf entry (-M) or parameter (-P).
-# Comment out main.cf entry, or master.cf entry (-M).

If I did not use the same option for the same operation on different
things, then I would run out of option letters, and you would not
be able to remember them.

Wietse

[Wietse Venema]

I have uploaded postfix-2.11-20131126-nonprod to the master FTP
site. This has revised DANE support as well as "advanced" master.cf
query/update support. The postconf code has been tested, but the
documentation is still rough.

Examples are at http://www.porcupine.org/postfix-mirror/wip.html
and at the top of the RELEASE_NOTES file. Documentation is also in
the postconf(1) manpage (nroff -man man/man1/postconf.1 | less).

Wietse

[LuKreme]

On 27 Nov 2013, at 17:13 , Wietse Venema <wie...@porcupine.org> wrote:
> Examples are at http://www.porcupine.org/postfix-mirror/wip.html
> and at the top of the RELEASE_NOTES file. Documentation is also in
> the postconf(1) manpage (nroff -man man/man1/postconf.1 | less).

My only comment is that the delineation between -F and -P is going to cause confusion, misconfigurations, and much wailing and gnashing of teeth.

since the targets of -F are known, it seems these switches can be combined.

$ postconf -P smtp/inet/chroot=y
$ postconf -P 'submission/inet/smtpd_tls_security_level=may'

There's no chance the first entry could be mistaken for a parameter, right?

--
"Yessir, Captain Tight Pants."

[Wietse Venema]

LuKreme:

> On 27 Nov 2013, at 17:13 , Wietse Venema <wie...@porcupine.org> wrote:
> > Examples are at http://www.porcupine.org/postfix-mirror/wip.html
> > and at the top of the RELEASE_NOTES file. Documentation is also in
> > the postconf(1) manpage (nroff -man man/man1/postconf.1 | less).
>
> My only comment is that the delineation between -F and -P is going
> to cause confusion, misconfigurations, and much wailing and gnashing
> of teeth.

Specify -F if you want to change a column (chroot) and specify -P
if you want to change a parameter (smtpd_recipient_restrictions).

The alternative is to encode it in the name:

field/smtp/inet/chroot=n

parameter/smtp/inet/mynetworks=127.0.0.1

> since the targets of -F are known, it seems these switches can be combined.

You are assuming that -F and -P will be the only name spaces, and
that one name cannot appear in more than one place. I think that's
two mistakes.

Wietse

[Wietse Venema]

Wietse Venema:

> LuKreme:
> > On 27 Nov 2013, at 17:13 , Wietse Venema <wie...@porcupine.org> wrote:
> > > Examples are at http://www.porcupine.org/postfix-mirror/wip.html
> > > and at the top of the RELEASE_NOTES file. Documentation is also in
> > > the postconf(1) manpage (nroff -man man/man1/postconf.1 | less).
> >
> > My only comment is that the delineation between -F and -P is going
> > to cause confusion, misconfigurations, and much wailing and gnashing
> > of teeth.

My initial reaction was that fields and parameters support
different operations.

- Update (x/x/x=y) works for both master.cf parameters and fields.

- Create (x/x/x=y) works for master.cf parameters but not fields.

- Delete (-X) works for master.cf parameters but not fields.

- Comment out (-#) works for neither.

But the biggest differences are that

$ postconf -F

knows that you are mis-typing a field name (where as you proposal
would think that you specify a parametername) and that

$ postconf -P
$ postconf -P service
$ postconf -P service/type

lists *only* master.cf (-o name=value) parameters while your
proposal would also list all the service/name/field settings.

What would you do to get only the parameters listed?

Wietse

[LuKreme]

On 27 Nov 2013, at 19:26 , Wietse Venema <wie...@porcupine.org> wrote:

> LuKreme:
>> On 27 Nov 2013, at 17:13 , Wietse Venema <wie...@porcupine.org> wrote:
>>> Examples are at http://www.porcupine.org/postfix-mirror/wip.html
>>> and at the top of the RELEASE_NOTES file. Documentation is also in
>>> the postconf(1) manpage (nroff -man man/man1/postconf.1 | less).
>>
>> My only comment is that the delineation between -F and -P is going
>> to cause confusion, misconfigurations, and much wailing and gnashing
>> of teeth.
>

> Specify -F if you want to change a column (chroot) and specify -P
> if you want to change a parameter (smtpd_recipient_restrictions).

Yes, I understand the difference, my point is that other people will be confused and that I suspect it is going to be a constant source of misconfiguration.

> The alternative is to encode it in the name:

Why? chroot is a field, can't it be parsed correctly?

> You are assuming that -F and -P will be the only name spaces, and
> that one name cannot appear in more than one place. I think that's
> two mistakes.

OK. I couldn't think of any field that collided with a parameter or what other name spaces there might be, but I predict it is going to be a issue.

Maybe field/smtp/inet/chroot=n and smtp/inet/mynetworks=127.0.0.1 (because I suspect changing the fields is going to be done much less often that parameters?)

> But the biggest differences are that
>
> $ postconf -F
>
> knows that you are mis-typing a field name (where as you proposal
> would think that you specify a parameter name)

That’s' certainly an issue. How many parameters are there that are going to take a simple y/n setting though? There's not a single one in my main.cf, though there are a couple of single digits.

Also, the Fields are all single words, and almost all the parameters have multiple_words_with_underscores, so there might be something there.

myhostname mydomain myorigin mydestination and mynetworks seem to be the only parameter labels without an _ on my system, for example. They all conveniently start with "my".

> and that
>
> $ postconf -P
> $ postconf -P service
> $ postconf -P service/type
>
> lists *only* master.cf (-o name=value) parameters while your
> proposal would also list all the service/name/field settings.

That does seem like a good feature to have.

> What would you do to get only the parameters listed?

Off the top of my head I'd say that -P lists only the parameters by default and you'd need to specify a second flag to also list the fields --full or --fields or something, perhaps. Again, I think mucking with fields is something that is done rather less than mucking with parameters.

So, maybe it doesn't matter. People will learn to use -P and use that most of the time and may never use -F at all and it will be fine. Maybe.


--
Real magic is the hand around the bandsaw, the thrown spark in the
powder keg, the dimension-warp linking you straight into the heart of a
star, the flaming sword that burns all the way to the pommel.

[Viktor Dukhovni]

On Thu, Nov 28, 2013 at 09:09:32AM -0700, LuKreme wrote:

> Also, the Fields are all single words, and almost all the parameters
> have multiple_words_with_underscores, so there might be something
> there.
>
> myhostname mydomain myorigin mydestination and mynetworks seem
> to be the only parameter labels without an _ on my system, for
> example. They all conveniently start with "my".

Counter-examples exist:

$ postconf -d | awk '{print $1}' | egrep -v _ | egrep -v '^my'
biff
relayhost
stress

users are free to create their own parameters. You may, for example,
have seen some of my posts that introduce ${indexed}...

At Morgan Stanley I implemented a separate postmast(1) utility,
that could be used to show all, or just non-default, master.cf
entries, and to insert or delete master.cf entries. It did not
support granular access to the components of an entry.

In postmast(1) there was no ambiguity between "." in a service name
and "." between the service name and the service type, because
these were specified separately "postmast [-s <service>] [-t <type>]
...". While integrating a super-set of postmast(1) functionality into
postconf(1) Wietse has switched from "." to "/", but this may not
solve the problem because some service names have "/" in them
(paths of sockets outside /var/spool/postfix/{private,public}).

To safely eliminate the ambiguity we can either use white-space to
separate the name and type:

# Single type
$ postconf -F "smtp inet"

# Any type
$ postconf -F smtp

or make the type mandatory with the empty string or "any" as a
wildcard:

# Single type
$ postconf -F smtp.inet

# Any type
$ postconf -F smtp.
OR
$ postconf -F smtp.any

I think "." is less visually intrusive than "/", and since "/" does
not eliminate the ambiguity, I would revert the separator to "."
with the next snapshot, with "any" for a wildcard:

# inet type
$ postconf -F smtp.inet

# any type
$ postconf -F smtp.any

With ".<type>" required, the ambiguity goes away.

--
Viktor.

[Wietse Venema]

Viktor Dukhovni:

> In postmast(1) there was no ambiguity between "." in a service name
> and "." between the service name and the service type, because
> these were specified separately "postmast [-s <service>] [-t <type>]
> ...". While integrating a super-set of postmast(1) functionality into
> postconf(1) Wietse has switched from "." to "/", but this may not
> solve the problem because some service names have "/" in them
> (paths of sockets outside /var/spool/postfix/{private,public}).

First, there is no ambiguity with service/type/parametername, because
there is no '/' in master.cf service names (on the other hand there
IS ambiguity with service.type.parametername, because '.' can appear
in service and parameter names).

There is no '/' in master.cf service names because master.cf services
must have sockets or FIFOs under the postfix-owned "private" or
"public" directory. It's unsafe for the Postfix master daemon to
open sockets or FIFOs in directories that aren't under control by
Postfix.

Second, the idea is to uniformly represent all configuration features
as "name = value" pairs, and to have the same uniform representation
in query results and in update requests.

The purpose of this is not to make the postconf command easier to
implement. Instead, the purpose is to make Postfix easier to manage
with third-party automated system management tools (whether open
source or closed). For that it is essential that all configuration
features can be represented in a uniform manner and that we avoid
the need to represent feature names as a combination of command-line
options and other command arguments (this could be an argument to
eliminate the -F/-P and to encode that into the feature name iself).

If all feature of Postfix can be represented as a list of name=value
pairs, for example:

feature context | name = value
====================|=====================
main.cf | parameter = value
master.cf field | service/type/field = value
master.cf -o override | service/type/parameter = value

and if this form can be used for both query resault and update
requests, then other systems can reason about Postfix and manage
settings in a less ad-hoc and hopefully more intelligent manner.

Wietse

[Viktor Dukhovni]

On Fri, Nov 29, 2013 at 05:41:35PM -0500, Wietse Venema wrote:

> There is no '/' in master.cf service names because master.cf services
> must have sockets or FIFOs under the postfix-owned "private" or
> "public" directory. It's unsafe for the Postfix master daemon to
> open sockets or FIFOs in directories that aren't under control by
> Postfix.

I was thinking of master.cf type "unix" entries, where the system
administrator could be using spawn(8) to create sockets outside
the Postfix queue directory. Perhaps such configurations are both
unwise and unsupported.

> Second, the idea is to uniformly represent all configuration features
> as "name = value" pairs, and to have the same uniform representation
> in query results and in update requests.

This does however mean that updates to multiple fields of the same
service entry are rather verbose:

postconf ... \
"service/type/name1 = value1" \
"service/type/name2 = value2"

where it might arguably be more "economical" for the user to use:

postconf ... \
-<SERVICE_NAME_FLAG> "service/type" \
"name1 = value1" \
"name2 = value2"

However, if the interface is intended primarily for scripts, not
interactive users, that is not a major issue.

> If all feature of Postfix can be represented as a list of name=value
> pairs, for example:
>
> feature context | name = value
> ====================|=====================
> main.cf | parameter = value
> master.cf field | service/type/field = value
> master.cf -o override | service/type/parameter = value

The down-side here is that software parsing the output may need to
implement a list of "reserved names" that distinguish parameter
names from "private", "unpriv", "chroot", "wakeup" and "maxproc".

One can even imagine perverse cases in which "chroot" or "maxproc"
are user-defined variables in main.cf parameters, with potential
master.cf overrides.

smtp unix ... smtp
-o maxproc=mumble

(maxproc is then used indirectly in some actual parameter processed by
smtp(8)).

I am not arguing against a uniform naming scheme, just suggesting
caution to disambiguate names of different types. It could, for
an off the cuff example, be as subtle as:

service/type.chroot (master.cf field)

vs.

service/type/chroot (-o override)

--
Viktor.

[Wietse Venema]

Viktor Dukhovni:

> > If all feature of Postfix can be represented as a list of name=value
> > pairs, for example:
> >
> > feature context | name = value
> > ====================|=====================
> > main.cf | parameter = value
> > master.cf field | service/type/field = value
> > master.cf -o override | service/type/parameter = value
>
> The down-side here is that software parsing the output may need to
> implement a list of "reserved names" that distinguish parameter
> names from "private", "unpriv", "chroot", "wakeup" and "maxproc".

Please do not ignore the left side of the table! Currently, feature
context is indicated with -M, -F and -P, respectively. Instead we
Could make names fully-qualified (within the same Postfix instance)
by adding an extra field to the pathname,

But before I go there I want to find the article that inspired this
development.

Wietse