how to make postfix split messages into one copy per recipient
Ralf Hildebrandt
> On Fri, Oct 11, 2002 at 10:41:42AM +0200, Ralf Hildebrandt wrote:
>
> > use
> > sendmail_destination_recipient_limit = 1
>
> Thanks, but the above comment is too cryptic to be helpful.
>
> More context, please.
This splits all mail that goes down the transport called "sendmail"
into mails with exactly one recipient.
--
Ralf Hildebrandt Ralf.Hil...@charite.de
Postfix Tips: http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
Why you can't find your system administrators:
(S)he's out buying some caffeine.
-
To unsubscribe, send mail to majo...@postfix.org with content
(not subject): unsubscribe postfix-users
Rahul Dhesi
filter. This lets me run just one instance of Postfix. Into master.cf
I put these lines:
smtp inet n - y - - smtpd
-o content_filter=sendmail:
sendmail unix - n n - - pipe
flags=Rq user=someuser argv=/usr/sbin/sendmail -f ${sender} -- ${recipient}
And in main.cf I have sendmail_destination_recipient_limit = 1.
But this still causes a new process to be invoked for each recipient,
which is still not satisfactory to me.
Rahul
Rahul Dhesi
> What problem are you trying to solve?
My original goal was to simply preserve the envelope recipient in mail
arriving via SMTP, so as to allow a domain namespace to reside in a
single mailbox.
But I am taking a slightly broader view than just doing that.
Once we receive mail via SMTP with the intent of delivering it to a
local user, we are going to split it into one copy per mailbox before
delivery occurs. I simply want to be able to do this split early, as
soon as the mail is received via SMTP. So there is no loss of
efficiency -- the split must be done anyway. (I do realize that this
means we don't get to cast out duplicates -- and sometimes we don't.)
This will then allow policy routing of mail based on the combination
envelope sender + envelope recipient. It provides a framework in which
many spam-filtering solutions could be implemented.
Obviously, we don't want to do the split if we are receiving mail via
SMTP for the purpose of relaying it to remote hosts. This is easily
achieved by keeping our relay SMTP servers distinct from our MX SMTP
servers.
Wietse Venema
> On Sat, Oct 12, 2002 at 07:21:17PM -0400, Wietse Venema wrote:
>
> > What problem are you trying to solve?
>
> My original goal was to simply preserve the envelope recipient in mail
> arriving via SMTP, so as to allow a domain namespace to reside in a
> single mailbox.
OK, that is a popular application, and I am aware that Postfix
envelope address rewriting can get in the way, at least until
Postfix consistently stores and reports the original envelope
recipient.
> But I am taking a slightly broader view than just doing that.
>
> Once we receive mail via SMTP with the intent of delivering it to a
> local user, we are going to split it into one copy per mailbox before
> delivery occurs. I simply want to be able to do this split early, as
> soon as the mail is received via SMTP. So there is no loss of
> efficiency -- the split must be done anyway. (I do realize that this
> means we don't get to cast out duplicates -- and sometimes we don't.)
It is not as simple as you make it appear to be.
If the split must happen before the SMTP server sends the "OK you
can delete your copy of the message now" to the SMTP client, then
Postfix needs the ability to atomically create and delete an
arbitrary number of queue files, and it must be able to clean up
and recover when something crashes in the middle of doing all this,
without siginificant mail duplication and without mail loss.
None of this complexity exists when Postfix first stores only ONE
multi-recipient copy of the message, after which it deletes recipients
from that single multi-recipient copy as it delivers to them.
> This will then allow policy routing of mail based on the combination
> envelope sender + envelope recipient. It provides a framework in which
> many spam-filtering solutions could be implemented.
The same can be done by making the content etc. filtering a property
of the recipient, as opposed to what it is now (a property of the
queue file that applies to all recipients).
This is something for Postfix version 2, though. It's unlikely to
be compatible with the present architecture.
> Obviously, we don't want to do the split if we are receiving mail via
> SMTP for the purpose of relaying it to remote hosts. This is easily
> achieved by keeping our relay SMTP servers distinct from our MX SMTP
> servers.
Indeed.
Wietse
Rahul Dhesi
smtp inet n - y - - smtpd
-o content_filter=sendmail: sendmail_destination_recipient_limit=1
did not result in sendmail_destination_recipient_limit being set to 1
from the perspective of the smtpd process. I had to set
sendmail_destination_recipient_limit to 1 in main.cf.
Rahul
> I found another way of achieving the same goal: By using a content
> filter. This lets me run just one instance of Postfix. Into master.cf
> I put these lines:
>
> smtp inet n - y - - smtpd
> -o content_filter=sendmail:
> sendmail unix - n n - - pipe
> flags=Rq user=someuser argv=/usr/sbin/sendmail -f ${sender} -- ${recipient}
>
> And in main.cf I have sendmail_destination_recipient_limit = 1.
Wietse Venema
manager. Therefore, the setting has to be specified on the qmgr
command line (in master.cf) or in the main.cf file.
Postfix daemons do not pass main.cf settings from one process to
the next one. It's especially impossible with the single queue
manager and with privileged processes such as the local delivery
agent.
Wietse
Rahul Dhesi:
Wietse Venema
Rahul Dhesi
[ about splitting incoming mail into one copy per recipient ]
> If the split must happen before the SMTP server sends the "OK you
> can delete your copy of the message now" to the SMTP client, then
> Postfix needs the ability to atomically create and delete an
> arbitrary number of queue files, and it must be able to clean up
> and recover when something crashes in the middle of doing all this,
> without siginificant mail duplication and without mail loss.
>
> None of this complexity exists when Postfix first stores only ONE
> multi-recipient copy of the message, after which it deletes recipients
> from that single multi-recipient copy as it delivers to them.
So it appears that using a content filter, and invoking an external
process, is about as simple a solution as we can reasonably get, without
adding a great deal of complexity to Postfix. The only problem that
still remains is that we are invoking the filter even for those messages
that have only one recipient.
So, what do you think of the following, which does not add too much
complexity?
- There should be some way of specifying that a content filter should
be used only if there are multiple recipients.
Rahul
Victor....@morganstanley.com
> - There should be some way of specifying that a content filter should
> be used only if there are multiple recipients.
>
Set up a null content filter transport (smtp right into the backend smtpd)
and set the recipient limit for the corresponding transport to 1. The cost
of doing this for all recipients is quite low.
The snapshots process smtpd_data_restrictions, a custom (new C code)
restriction at that stage could in principle trigger on multiple
recipients and set the content filter. This is too specialized to be in
the main release IMHO. A private custom patch would not be too hard.
--
Viktor.
Wietse Venema
> On Mon, Oct 14, 2002 at 03:54:15PM -0400, Wietse Venema wrote:
>
> [ about splitting incoming mail into one copy per recipient ]
>
> > If the split must happen before the SMTP server sends the "OK you
> > can delete your copy of the message now" to the SMTP client, then
> > Postfix needs the ability to atomically create and delete an
> > arbitrary number of queue files, and it must be able to clean up
> > and recover when something crashes in the middle of doing all this,
> > without siginificant mail duplication and without mail loss.
> >
> > None of this complexity exists when Postfix first stores only ONE
> > multi-recipient copy of the message, after which it deletes recipients
> > from that single multi-recipient copy as it delivers to them.
>
> So it appears that using a content filter, and invoking an external
> process, is about as simple a solution as we can reasonably get, without
> adding a great deal of complexity to Postfix. The only problem that
> still remains is that we are invoking the filter even for those messages
> that have only one recipient.
Indeed. The content filter is just another way to "delivering"
mail. Postfix is built as a queue with multiple interfaces
to the external world (for receiving and delivering mail).
However, if all you want is to split mail, then why not work with
the system and do your special processing upon final delivery? That
delivery can be configured as one recipient per transaction.
> So, what do you think of the following, which does not add too much
> complexity?
>
> - There should be some way of specifying that a content filter should
> be used only if there are multiple recipients.
Maybe, but not in the mainstream Postfix release. Weird hacks
like this are hard to maintain in an evolving program.
Wietse
Rahul Dhesi
> > - There should be some way of specifying that a content filter should
> > be used only if there are multiple recipients.
>
> Maybe, but not in the mainstream Postfix release. Weird hacks
> like this are hard to maintain in an evolving program.
And I can certainly see why you think so. Postfix is a complicated (and
wonderful) package already, so you certainly don't want to trivially add
non-essential features.
So let me provide you with a possible argument for considering this as a
useful feature worth making permanent in the long run, should somebody
provide the code to do so.
- We know that preserving the SMTP envelope recipient is a very common
request from users and from site administrators.
- The only solution currently provided in the Postfix FAQ is not very
efficient (linear regex scan), somewhat inelegant (does not literally
preserve envelope recipient), and error-prone (must edit regex entry
for each domain in a hard-to-read syntax).
- So if somebody could find a way of allowing the envelope recipient
to be added when needed, which would have overhead of O(1) rather than
O(n) in the number of domains, and which would exactly preserve the
SMTP envelope recipient, and which would not require any per-domain
entry in any table; and if the changes in Postfix were not very
complicated; then the benefit/cost ratio for this would be quite high.
- Here is that method, which actually requires no changes within Postfix.
We simply use the following content filter, feeding it the same
arguments expected by Postfix's sendmail, i.e.: -f sender -- recipient.
And we configure Postfix to set a recipient limit of 1 for this
content filter.
#! /bin/sh
( echo "X-SMTP-To: $4"; cat) | /usr/sbin/sendmail "$@"
# if error, return temp failure
case $? in
0) exit 0 ;;
*) exit 75 ;;
esac
- So why should Postifx need any changes? Only for efficiency.
The changes would be:
1. The smtp server adds X-SMTP-To: to each message that has
only one recipient. (Could be a permanent feature or could be
optional based on Postfix configuration.) If this is a permanent
feature, the code changes would be tiny, I think.
2. A content filter can be conditionally invoked for each message that
has multiple recipients. The content filter used in this case
would be equivalent to the shell script above.
Rahul
P.S. I have a production machine here with three IP addresses, and it
runs all three of these MTAs: sendmail, postfix, and qmail. Had any two
of these done what I needed, I would not have gone to the trouble of
running all three. :-) I am now planning to replace the other two with
Postfix, and as I do so, various questions are coming up, including the
above.
Victor....@morganstanley.com
> 1. The smtp server adds X-SMTP-To: to each message that has
> only one recipient. (Could be a permanent feature or could be
> optional based on Postfix configuration.) If this is a permanent
> feature, the code changes would be tiny, I think.
>
If all you want to do is add X-SMTP-To: to messages destined to particular
recipients, the simplest approach is to.
- Use a nul content filter. Postfix talking directly to Postfix on another
port per FILTER_README but without a filter.
- Avoid all envelope rewrites in the first hop cleanup daemon (do them in
the second cleanup daemon). This requires some magic for Postfix-style
virtual domains. The $virtual_maps of the second Postfix needs to the
$virtual_mailbox_maps of the first instance.
- Set the recipient limit of filter "smtp" delivery agent to 1.
- Patch either the sending "smtp" or the receiving "smtpd" to add the
"X-RCPT-To:" header.
> 2. A content filter can be conditionally invoked for each message that
> has multiple recipients. The content filter used in this case
> would be equivalent to the shell script above.
>
It is IMHO simpler to add the header in either the "smtp" client or the
"smtpd" server. Invoking a content filter to insert a single header is too
much work.
It appears likely that Postfix will eventually be able to propage the
original envelope to delivery agents. When this happens, a much simpler
solution to your problem will emerge.
--
Viktor.
Wietse Venema
What you request appears to be a workaround for something that can
be fixed more structurally by adding to Postfix the ability to
accurately record and reproduce the original envelope sender address
upon final delivery.
Given a choice, I will do the structural solution any time. It's
the same amount of work and it solves more problems.
Propagating the original envelope recipient involves writing an
REC_TYPE_ORCP queue file record for every recipient, having the
queue manager copy that info in requests to delivery agents, and
having delivery agents record the address in Original-Rcpt: message
headers, and updating the bounce/defer protocols.
OK, so it is a little more work than making content filtering
dependent on the initial recipient count, but the result is a lot
more useful.
Tell you what, I will try to implement this before Postfix 1.2,
that is, this calendar year.
Wietse
Rahul Dhesi:
> 1. The smtp server adds X-SMTP-To: to each message that has
> only one recipient. (Could be a permanent feature or could be
> optional based on Postfix configuration.) If this is a permanent
> feature, the code changes would be tiny, I think.
>
> 2. A content filter can be conditionally invoked for each message that
> has multiple recipients. The content filter used in this case
> would be equivalent to the shell script above.
>
> Rahul
>
> P.S. I have a production machine here with three IP addresses, and it
> runs all three of these MTAs: sendmail, postfix, and qmail. Had any two
> of these done what I needed, I would not have gone to the trouble of
> running all three. :-) I am now planning to replace the other two with
> Postfix, and as I do so, various questions are coming up, including the
> above.
Bennett Todd
--dc+cDN39EJAMEtIO
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
2002-10-15-15:05:00 Rahul Dhesi:
> - Here is that method, which actually requires no changes within Postfix.
> We simply use the following content filter, feeding it the same
> arguments expected by Postfix's sendmail, i.e.: -f sender -- recipient.
> And we configure Postfix to set a recipient limit of 1 for this
> content filter.
>=20
> #! /bin/sh
> ( echo "X-SMTP-To: $4"; cat) | /usr/sbin/sendmail "$@"
> # if error, return temp failure
> case $? in
> 0) exit 0 ;;
> *) exit 75 ;;
> esac
Please take a look at smtpprox[1].
It's a minimalist, passthrough SMTP proxy, open-coded in perl. It's
preforking, and each forked child does a [configurable] number of
messages then exits. Thus the fork cost can be amortized over a lot
of work and so made to disappear. As contrasted, doing content
filtering using pipe requires a fork-n-exec per message, not
breathtakingly fast even on the best of systems, and really
irritatingly slow on the rest.
The above msg tweak should be, I believe, just applying this change
to smtpprox-1.0:
--- smtpprox Thu Mar 1 21:37:46 2001
+++ smtpprox-customized Tue Oct 15 16:59:27 2002
@@ -146,10 +146,13 @@
my $banner =3D $client->hear;
$banner =3D "$debugtrace.$$" if defined $debugtrace;
$server->ok($banner);
+ my $sender;
while (my $what =3D $server->chat) {
if ($what eq '.') {
+ $server->{data} =3D~ s/^/X-SMTP-To: $sender/;
$client->yammer($server->{data});
} else {
+ $sender =3D $1 if $what =3D~ /^rcpt to:\s+(.*)/i;
$client->say($what);
}
$server->ok($client->hear);
There Is More Than One Way To Do It, of course, but I believe the
above is little enough work to be worth trying. This should give
really pleasing performance; pick a concurrency that seems well
suited to the platform you are running this on --- i.e. how many of
these perl message-whacking smtp proxies do you want active at once,
peak. Set the parent (cmdline arg) to fork that many, and for good
measure tell postfix to use no more than that many at once.
Actually, I always chicken and set say 15 children if I intend to
tell postfix to use a maximum concurrency of 10.
-Bennett
[1] <URL:http://bent.latency.net/smtpprox/>
--dc+cDN39EJAMEtIO
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE9rIL0HZWg9mCTffwRAmEBAJ4jXCm+gHHkMFJRVdZirtscSZaAhwCfaB74
F1uCyXQ7aioICRSwLHLvL+k=
=2aMD
-----END PGP SIGNATURE-----
--dc+cDN39EJAMEtIO--
Rahul Dhesi
> Given a choice, I will do the structural solution any time. It's
> the same amount of work and it solves more problems.
...
> Tell you what, I will try to implement this before Postfix 1.2,
> that is, this calendar year.
Great!
I think we have here the best of both worlds. As Kernighan and Plauger
said in "The Elements of Programming Style":
Make it right before you make it faster.
Keep it right when you make it faster.
Make it clear before you make it faster.
My little shell script is close enough to doing it right, for now, and
it's certainly quite clear. And have I no doubt whatsoever that you
will keep it right when you make it faster.
Thanks!
Rahul
Rahul Dhesi
> Please take a look at smtpprox[1].
>
> It's a minimalist, passthrough SMTP proxy, open-coded in perl. It's
> preforking, and each forked child does a [configurable] number of
> messages then exits....
...
> The above msg tweak should be, I believe, just applying this change
> to smtpprox-1.0:
I took a look -- it looks very promising! I can see all sorts of uses
for an efficient SMTP proxy of this type, where one can add custom code
at any point.
Thanks.
Ralf Hildebrandt
> I think we have here the best of both worlds. As Kernighan and Plauger
> said in "The Elements of Programming Style":
>
> Make it right before you make it faster.
>
> Keep it right when you make it faster.
>
> Make it clear before you make it faster.
"Premature optimization is the root of all evil".
Dunno who said that, though.
--
Ralf Hildebrandt Ralf.Hil...@charite.de
Postfix Tips: http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
Security-wise, NT is a server with a "Kick me" sign taped to it.