Bounce TTL versus non-bounce TTL
Wietse Venema
regular mail and for bounces?
I can see an argument for having shorter TTLs for bounces. Bounces
for legitimate messages are likely to be delivered soon. It is
unlikely for a legitimate site to send mail and then to disappear
from the face of the earth.
Wietse
Michael Breton
I would agree with that.
The only problem is that many sites have separate outgoing and incoming
systems, and if their incoming is down, there may still be outgoing mail.
The temptation for some will be to lower the bounce_maximal_queue_lifetime
too low for these bounces to ever be sent. I can see some lowering this to
just a few hours.
Michael Breton
Commtel
Victor....@morganstanley.com
> Would it make sense to have different maximal queue life times for
> regular mail and for bounces?
>
> I can see an argument for having shorter TTLs for bounces. Bounces
> for legitimate messages are likely to be delivered soon. It is
> unlikely for a legitimate site to send mail and then to disappear
> from the face of the earth.
>
If the goal is to reduce the deferred queue size, perhaps this is
worthwhile, but I don't see anything less than 2 days as practical in most
cases. If the goal is to reduce active queue congestion via undeliverable
bounces, I would suggest separate backoff timers for bounces rather than
or in addition to a separate maximal_queue_lifetime.
So on the table would be some combination of:
bounce_maximal_queue_lifetime = 2d
bounce_minimal_backoff_time = 1h
bounce_maximal_backoff_time = 8h
So bounces would be tried at
0, 1, 2, 4, 8, 16, 24, 32, 40 and 48 hours after injection into
the queue, for a total of at most 10 attempts in a 48 hour window.
--
Viktor.
Clifton Royston
> Would it make sense to have different maximal queue life times for
> regular mail and for bounces?
>
> I can see an argument for having shorter TTLs for bounces. Bounces
> for legitimate messages are likely to be delivered soon. It is
> unlikely for a legitimate site to send mail and then to disappear
> from the face of the earth.
"Yes, but..."
Yes, this would have some benefit to the vast majority of sites, and
in particular would avoid their consternation at always seeing many
spam bounces which will never be deliverable in their queues.
But, due to Postfix's efficiency in handling deferred mail, I do not
see this bounce backlog creating an actual problem in practice. IMHO
it is only a perceived problem. The few people who might have deferred
queues so big that it creates a real problem for processing of their
active queue either have b0rken their configuration by tweaking
parameters poorly, or are running huge volumes of outgoing mail to
poorly reachable addresses. In either case this solution will not help
them.
As already noted, it also gives the naive admin a easy way to end up
further damaging end-to-end reliability of email.
If you are considering the effort to support multiple queue lifetimes
at all, and to create any kind of knobs to control them, I would hope
it could be done in a way which might lead *eventually* to support for
configuring different queue lifetimes for different messages in a more
general way. In particular, it would be nice eventually to be able to
preset message lifetimes based on recipient domains, without having to
run multiple Postfix instances. This I would find truly useful. If
adding the short bounce lifetime would further this goal happening some
year, then I'm all for it; otherwise I don't care.
My opinion only; boil vigorously and add 2 Tbsp salt after reading..
-- Clifton
--
Clifton Royston -- LavaNet Systems Architect -- clif...@lava.net
"If you ride fast enough, the Specialist can't catch you."
"What's the Specialist?" Samantha says.
"The Specialist wears a hat," says the babysitter. "The hat makes noises."
She doesn't say anything else.
Kelly Link, _The Specialist's Hat_
Hamish Marson
>Would it make sense to have different maximal queue life times for
>regular mail and for bounces?
>
>I can see an argument for having shorter TTLs for bounces. Bounces
>for legitimate messages are likely to be delivered soon. It is
>unlikely for a legitimate site to send mail and then to disappear
>from the face of the earth.
>
> Wietse
>
>
Given that the nbulk of the mail in my deferred queue (At home) is for
stuff from high speed mailers that'd be a godsend...
--
I don't suffer from Insanity... | Linux User #16396
I enjoy every minute of it... |
|
http://www.travellingkiwi.com/ |
Wietse Venema
> >>>>> "WV" == Wietse Venema <wie...@porcupine.org> writes:
>
> WV> Would it make sense to have different maximal queue life times for
> WV> regular mail and for bounces?
>
> It would eliminate my primary reason for making content filters run at
> SMTP receive time, rather than at a later time.
Since the decision is made by the queue manager, this can be added
without breaking compatibility.
> I'd also love to be able to define different queue life times for
> messages of different Precedence: values.
If I recall, that requires adding an expiration record to queue
files (can be done without breaking things) plus some configurable
mapping from Precedence: value to some time limit and a default
for unrecognized precedence values.
As far as I know, Precedence: is not standardized.
Wietse
Rahul Dhesi
> Would it make sense to have different maximal queue life times for
> regular mail and for bounces?
Yes, please.
Rahul
Victor....@morganstanley.com
What lifetime would you propose? This last weekend many nameservers and
mail relays on the net were unreachable for at least two days. Many sites
don't fix friday night problems until Monday morning.
It seems impractical to use queue lifetimes much less than two days on
general purpose email servers.
I think that deferred queue size has little consequence for Postfix
performance. What matters is the active queue size, and for that I would
look to retransmit deferred mail less frequently, rather than timing it
out faster. The disk space requirements for a few thousand bounces on
disk are not generally prohibitive.
I am concerned that email will get even less reliable (reliability means
either delivering mail to its destination or delivering a bounce to the
sender). If anything bounces should have a *longer* lifetime than regular
mail. The sender will know faster that their message did not get through
if the non-bounce timeout is short, but more effort is made to not lose
email outright.
Decreasing the reliability of ordinary email further in order to combat
spam seems questionable...
--
Viktor.
Rahul Dhesi
Victor....@morganstanley.com wrote:
> What lifetime would you propose? This last weekend many nameservers and
> mail relays on the net were unreachable for at least two days. Many sites
> don't fix friday night problems until Monday morning.
There is an old maxim:
Every critical message must be acknowledged.
Those who send essential email and assume it reached, without getting
any confirmation, will never be satisfied with the reliability of any
network.
So long as your bounced-mail queue has less than an infinite lifetime,
and so long as your RAID array has a greater-than-zero probabilty of
multiple disks failing within the length of your RAID rebuild time, and
so long as operating systems have a finite rate of filesystem bugs,
bounced messages will be lost with some frequency.
Even in the days of the Pony Express (an American concept -- when they
delivered mail coast-to-coast in 48 hours via a relay of horseeback
riders) their mail was 100% reliable. The sender simply waited until he
got a reply. If he did not get one, he assumed his letter had not
reached and sent another. The life expectancy of the Pony Express
riders was not high, because they would often be attacked and killed on
the way. Only orphans were hired for this job. In effect, their
bounced-mail queue had a very short timeout. So indeed individual
letters did get lost. But messages, as opposed to letters, still got
through reliably.
Once again:
Every critical message must be acknowledged.
Rahul
Wietse Venema
> On Thu, 30 Jan 2003, Rahul Dhesi wrote:
>
> > On Thu, Jan 30, 2003 at 01:11:35PM -0500, Wietse Venema wrote:
> >
> > > Would it make sense to have different maximal queue life times for
> > > regular mail and for bounces?
> >
> > Yes, please.
>
> mail relays on the net were unreachable for at least two days. Many sites
> don't fix friday night problems until Monday morning.
It depends on how long it takes for mail to bounce. On a machine
that does final delivery only, a bounce will be almost instantaneous.
If the mail was legitimate, the bounce is likely to be delivered soon.
With other systems it can take a week or more before mail is returned
as undeliverable after it has percolated through forwarding labyrinths.
Example: a week or so ago some spammer advertised a product using
non-existent sender addresses in one of my own domains. Since this
was a domain where I never receive mail, I updated the MX record.
After the spammer stopped sending mail, bounces kept trickling in
for more than a week. Meanwhile I set up a REDIRECT body_checks
pattern to keep the product owner informed of all delivery problems
with their advertising campaign.
> It seems impractical to use queue lifetimes much less than two days on
> general purpose email servers.
>
> I think that deferred queue size has little consequence for Postfix
> performance. What matters is the active queue size, and for that I would
> look to retransmit deferred mail less frequently, rather than timing it
> out faster. The disk space requirements for a few thousand bounces on
> disk are not generally prohibitive.
I think it just annoys a lot of sysadmins to see mail in the queue
that is never going anywhere, even when the amount does not hurt
Postfix.
Wietse
lst...@kwsoft.de
> Would it make sense to have different maximal queue life times for
> regular mail and for bounces?
>
> I can see an argument for having shorter TTLs for bounces. Bounces
> for legitimate messages are likely to be delivered soon. It is
> unlikely for a legitimate site to send mail and then to disappear
> from the face of the earth.
>
> Wietse
>
I guess for secondary MXs which have no userbase to check at hand this will be
really usefull to cut down Spam bounces.
Regards
Andreas
Hamish Marson
Not just that. ALL of the spam (And virus warnings) I bounce has already
been checked for the destination user actually existing (My secondaries
have the userlists as well. It's not hard, in fact it's easier because
the configs are then the same).
Craig Sanders
> I think it just annoys a lot of sysadmins to see mail in the queue
> that is never going anywhere, even when the amount does not hurt
> Postfix.
what would be useful would be a queue-manager (possibly with a curses or
similar interface) that would:
1. scan the postfix queues
this is possibly the most difficult part to get right. items can enter
or leave the queue at any time, so the scan needs to be performed
regularly. a brute force scan of the mailq would be prohibitively
expensive on systems with large amounts of queued mail (i.e. the more
useful the manager is, the slower it is to use)...so perhaps there
should be some way for it to query master for a list of queue items.
OTOH, if this is used mainly to view deferred bounces to decide whether
to delete them or not then one scan every 5 or 10 minutes is
enough...it's enough when i run "mailq | less" to do it manually.
alternatively, only do a re-scan at the user's request - ^R or
something.
2. display a list of queue items, sortable by various attributes (age,
sender, recipient, size, etc). the list would be navigable with
arrow keys or vi-keys.
it should be possible to limit the display to tagged messages only (see
below).
3. allow individual queue items to be viewed (e.g. forking postcat and
less), re-queued, bounced, deleted or tagged with a single keystroke.
e.g.
<enter> - view
r - re-queue
B - bounce (capital B)
D - deletd (capital D)
t - tag
4. allow queue items to be tagged by number, age, and regexp - similar
to mutt's tagging capabilities. it would be useful to be able to tag by
applying a regexp to either the queue information OR a complete scan of
the message bodies.
e.g. tag all messages matching /MAILER-DAEMON/ or all messages
containing the URL /http:\/\/spam\.domain/
5. allow all tagged messages to be re-queued, bounced, or deleted in one
operation.
e.g. like mutt, ";" followed by r, B, or D.
6. a hold queue would also be useful, with H to hold and U to unhold.
i do this manually now, with "mailq | less" in one xterm window and
cutting and pasting queue-ids into another to view, delete, or requeue
them. i also use a tool posted here a few months ago called mqview by
Jon Bensen which makes it easy to list and/or delete queue items by
sender or recipient regexp (no message body search, though).
the only thing that bothers me is that it may not be a good idea to make
it too easy to view queue items. sometimes you have to, e.g. to
determine whether a stuck bounce is spam or not, or to help track down
problems. if it's too easy to view queue items, however, then novice or
unethical mail admins may use the tool to indulge idle curiousity. of
course, whether the slight skill barrier makes any difference is
debatable.
craig
--
craig sanders <c...@taz.net.au>
Fabricati Diem, PVNC.
-- motto of the Ankh-Morpork City Watch
Faber Fedor
> On Thu, Jan 30, 2003 at 04:26:38PM -0500, Wietse Venema wrote:
> > I think it just annoys a lot of sysadmins to see mail in the queue
> > that is never going anywhere, even when the amount does not hurt
> > Postfix.
>
> what would be useful would be a queue-manager (possibly with a curses or
> similar interface) that would:
Hey! This is exactly what I Was asking about in another thread ("Queue
GUIs")! So I take it I *do* have to write this, eh?
Thanks for the ideas about how it should/will work.
--
Regards,
Faber
Linux New Jersey: Open Source Solutions for New Jersey
http://www.linuxnj.com
Wietse Venema
that people can build their own tools on so I don't have to write
GUIs.
Wietse
Craig Sanders:
> On Thu, Jan 30, 2003 at 04:26:38PM -0500, Wietse Venema wrote:
> > I think it just annoys a lot of sysadmins to see mail in the queue
> > that is never going anywhere, even when the amount does not hurt
> > Postfix.
>
> what would be useful would be a queue-manager (possibly with a curses or
> similar interface) that would:
>
Faber Fedor
> Sounds like another reason to have a mail queue reporting protocol
> that people can build their own tools on so I don't have to write
> GUIs.
>
> Wietse
Forgive a newbie's impertinence, but what is involved in writing such a
mail queue reporting protocol?
Simon White
> Sounds like another reason to have a mail queue reporting protocol
> that people can build their own tools on so I don't have to write
> GUIs.
Good point. However, just keeping mailq output consistent means a tool
could be hacked together, and then if it shows enough real interest,
then a protocol could be cobbled together.
After all, mailq output is more or less a pseudo-protocol over STDOUT...
no?
--
[Simon White. vim/mutt. si...@mtds.com. Folding@home no log script yet...]
If you don't like what is going on in Palestine, or are curious, look:
http://www.inminds.com/boycott-israel.html
I am not anti-jewish. I am against the Israeli régime headed by Sharon.
Wietse Venema
[ Charset ISO-8859-1 unsupported, converting... ]
> 31-Jan-03 at 23:35, Wietse Venema (wie...@porcupine.org) wrote :
> > Sounds like another reason to have a mail queue reporting protocol
> > that people can build their own tools on so I don't have to write
> > GUIs.
>
> Good point. However, just keeping mailq output consistent means a tool
> could be hacked together, and then if it shows enough real interest,
> then a protocol could be cobbled together.
>
> After all, mailq output is more or less a pseudo-protocol over STDOUT...
> no?
mailq is a sendmail compatibility crutch. It's not a final
queue reporting solution.
Wietse
Wietse Venema
> On Fri, Jan 31, 2003 at 11:35:00PM -0500, Wietse Venema wrote:
> > Sounds like another reason to have a mail queue reporting protocol
> > that people can build their own tools on so I don't have to write
> > GUIs.
> >
>
> Forgive a newbie's impertinence, but what is involved in writing such a
> mail queue reporting protocol?
That depends on how fancy you want to get. The present mailq/showq
client/server do not really implement a protocol at all.
Wietse
Simon White
> Simon White:
> [ Charset ISO-8859-1 unsupported, converting... ]
> > > Sounds like another reason to have a mail queue reporting protocol
> > > that people can build their own tools on so I don't have to write
> > > GUIs.
> >
> > could be hacked together, and then if it shows enough real interest,
> > then a protocol could be cobbled together.
> >
> > After all, mailq output is more or less a pseudo-protocol over STDOUT...
> > no?
>
> mailq is a sendmail compatibility crutch. It's not a final
> queue reporting solution.
I know. But most of us are familiar with it, and human parse it or
already have awk scripts to deal with it.
Since I would consider that a mailq protocol is pretty low on your list
of priorities, it might not be a bad idea to have a queue manager that
takes output of mailq as input and parses it. There's no point for this
tool to get too clever though, because
- messages don't stick around too long in some queues
- duplicate queue IDS is always a possibility
- it might encourage newbies to commit atrocities with the queues
For the deferred / bounce queues a simple PERL based parser might be
useful though.
--
[Simon White. vim/mutt. si...@mtds.com. Folding@home no log script yet...]
Tis better to be silent and thought a fool, than to open your mouth and
remove all doubt. -- Abraham Lincoln
[Linux user #170823 http://counter.li.org. Home cooked signature rotator.]
Victor....@morganstanley.com
> For the deferred / bounce queues a simple PERL based parser might be
> useful though.
>
Since operating on the resulting queued messages requires privileges, it
is enough IMHO to forgo the protocol entirely and directly (in Perl) parse
the underlying queue files. I have some tools along these lines, and
have recently posted the core queue file parsing code to this list.
Since it is so short here it is again:
sub rec_get {
my ($h) = @_;
my $r = getc($h) || return;
my $l = 0;
my $shift = 0;
while (defined(my $lb = getc($h))) {
my $o = ord($lb);
$l |= ($o & 0x7f) << $shift ;
last if (($o & 0x80) == 0);
$shift += 7;
return if ($shift > 7); # XXX: max rec len of 4096
}
my $d;
return unless ($l == 0 || read($h,$d,$l) == $l);
($r, $l, $d);
}
sub qenv {
my ($qfile) = @_;
my $h = new IO::File($qfile, "r") || return;
my ($t, $s, @r);
while (my ($r, $l, $d) = rec_get($h)) {
push(@r, $d) if ($r eq "R");
$s = $d if ($r eq "S");
$t = $d if ($r eq "T");
last if ($r eq "M");
}
close($h);
($t, $s, @r);
}
The qenv() routine does not correctly process messages submitted via "sendmail
-t", since these have recipients after the message body, such messages are
not present on the gateway system where I use the code. Extending the code
to deal with "extracted" recipients is left as an exercise for the
reader (of this message and the Postfix source code)...
--
Viktor.
Ben Rosengart
wrote:
>
> If the goal is to reduce active queue congestion via undeliverable
> bounces, I would suggest separate backoff timers for bounces rather than
> or in addition to a separate maximal_queue_lifetime.
Agreed wholeheartedly. I might be able to reduce Panix's
maximal_backoff_time back to the default if I could set a
separate, higher maximal_backoff_time for bounces.
--
Ben Rosengart (212) 741-4400 x215
"From start to finish, getting data to do the right things and go to the
right places can seem complicated." --Sun Cobalt Qube 3 Developer's Guide