Unorthodox (?) idea to improve Sendmail+Milter performance
David F. Skoll
and it may provide extreme improvement under extreme situations.
I propose a new option to Sendmail: IncomingQueueDirectory
If this option is set, then when mail comes in via SMTP, it is
collected and queued in the IncomingQueueDirectory. Assuming all goes
well, then after the client has transmitted its "." at the end of
DATA, then Sendmail copies the queue files out of
IncomingQueueDirectory into the normal QueueDirectory. Once that
succeeds (and *only* once that succeeds), it returns a 2xx reply code.
It then delivers (in background mode, for example) out of the normal
queue.
What? Are you crazy? Double the queue I/O?
The idea here is to mount IncomingQueueDirectory on a ramdisk. If a
milter does content-filtering and decides to reject the message, you
*never* have to commit it to stable storage. When one of the regular
Mickey$oft virus outbreaks happens and your mail volume spikes, none
of the viruses use up queue I/O because they're all rejected while
they still sit in the RAM-based IncomingQueueDirectory. Similarly, if
20% of your mail is spam that scores so high that it can be rejected
out-of-hand, you save that queue I/O.
Since queue bandwidth is usually the first thing you run out of, you
can add lots of RAM and lots of processing power to the box fairly
cheaply, and it can handle virus spikes quite nicely.
If IncomingQueueDirectory is not set, of course, Sendmail should behave as
it does now so people who don't use content-filtering see no impact.
Comments? Thoughts? Flames?
Regards,
David.
Chris Adams
>Since queue bandwidth is usually the first thing you run out of, you
>can add lots of RAM and lots of processing power to the box fairly
>cheaply, and it can handle virus spikes quite nicely.
Instead of adding lots of RAM (that you can't "take with you" when you
upgrade the server) and complexity to sendmail, get a solid state disk
for your mail queue. They aren't cheap, but they'll make a huge
difference in mail overhead.
In any case, what you propose shouldn't make that much difference.
IIRC, sendmail doesn't force the queue files to disk until after it has
determined that it will deliver the message (i.e. after milter has said
"ok"). In many cases, messages deleted by milter will in the kernel
disk buffers and never hit the physical storage, unless memory pressure
forces the kernel to flush buffers (in which case you probably need to
add RAM anyway).
--
Chris Adams <cma...@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
Andrzej Adam Filip
> [...]
> The idea here is to mount IncomingQueueDirectory on a ramdisk. If a
> milter does content-filtering and decides to reject the message, you
> *never* have to commit it to stable storage. When one of the regular
> Mickey$oft virus outbreaks happens and your mail volume spikes, none
> of the viruses use up queue I/O because they're all rejected while
> they still sit in the RAM-based IncomingQueueDirectory. Similarly, if
> 20% of your mail is spam that scores so high that it can be rejected
> out-of-hand, you save that queue I/O.
>
> Since queue bandwidth is usually the first thing you run out of, you
> can add lots of RAM and lots of processing power to the box fairly
> cheaply, and it can handle virus spikes quite nicely.
>
> If IncomingQueueDirectory is not set, of course, Sendmail should behave as
> it does now so people who don't use content-filtering see no impact.
>
> Comments? Thoughts? Flames?
Sendmail 8.12 offer something similar: "double interactive" delivery
O DeliveryMode=interactive
O SuperSafe=interactive
In this mode sendmail delays sending back final 250 ACK to sending host to
allow operating system keep the files in its caches without sendmail *forcing*
disk commits as in other modes. Writing to OS file system caches is fast,
committing to non solid state disk is much slower. Double interactive mode may
produce delays unaceptable to sending host.
IMHO one way of improving performance may be "opportunistic double interactive
mode" (double interactive with timeout) - sendmail tries to deliver received
message in double interactive mode (avoids committing to disk) but if the full
transaction is not finished in 5s (some configurable period) then it switches
to standard deliver mode (e.g. background) and forces disk commits.
My assumption is that on most system most messages are fully processes in less
than 2-3s, so most messages in normal situations will be processed using OS
file system caches without "true" disk I/O operations.
--
Andrzej [en:Andrew] Adam Filip an...@priv.onet.pl an...@xl.wp.pl
http://anfi.homeunix.net/ http://slashdot.org/~anfi
*Random Epigram* :
I didn't fight with honor. I fought to win.
-- Ender in "Ender's Game" by Orson S. Card
David F. Skoll
> Instead of adding lots of RAM (that you can't "take with you" when you
> upgrade the server) and complexity to sendmail, get a solid state disk
> for your mail queue. They aren't cheap, but they'll make a huge
> difference in mail overhead.
Yes, obviously; a solid-state disk is the expensive way to go. But
on a cluster of a few MX hosts, the costs can mount up.
> In any case, what you propose shouldn't make that much difference.
> IIRC, sendmail doesn't force the queue files to disk until after it has
> determined that it will deliver the message (i.e. after milter has said
> "ok").
It also looks like code in collect.c fsync's the df file, even before
Milter has had a chance to reject the message. I have customers whose
servers routinely have hundreds of Sendmail processes in a disk-wait
state waiting to commit messages into the queue. A lot of these
messages are subsequently rejected.
Removing the fsync from collect() and putting it after a milter
reply code has been received would also help a lot, and would be
a much smaller change.
Regards,
David.
David F. Skoll
> IMHO one way of improving performance may be "opportunistic double
> interactive mode" (double interactive with timeout) - sendmail tries to
> deliver received message in double interactive mode (avoids committing to
> disk) but if the full transaction is not finished in 5s (some configurable
> period) then it switches to standard deliver mode (e.g. background) and
> forces disk commits.
My idea was to avoid committing if milter is going to reject the mail
anyway.
I've submitted a patch to Claus that adds a new possibility for
SuperSafe called "PostFilter". It defers the commit until after
milter says it's OK to deliver the message.
We'll see if it gets accepted into Sendmail; I hope so.
Regards,
David.
Andrzej Adam Filip
I wish you luck with making sendmail.org accept your donation.
How many percent of messages is rejected at your site by milter ?
[spam/e-mail ratio]
Could you publish the patch somwhere at www.mimedefang.org ?
--
Andrzej [en:Andrew] Adam Filip an...@priv.onet.pl an...@xl.wp.pl
http://anfi.homeunix.net/ http://slashdot.org/~anfi
*Random Epigram* :
Living in LA is like not having a date on Saturday night.
-- Candice Bergen
David F. Skoll
>> I've submitted a patch to Claus that adds a new possibility for
>> SuperSafe called "PostFilter". It defers the commit until after
>> milter says it's OK to deliver the message.
> I wish you luck with making sendmail.org accept your donation.
I will need it... they've rejected all my previous ideas... :-(
> How many percent of messages is rejected at your site by milter ?
> [spam/e-mail ratio]
Our stats are at http://www.roaringpenguin.com/canit/stats.php
(Log in as demo/demo).
It looks like in the last 30 days, we accepted
9760 messages and rejected 14081, for a spam/total ratio of
0.59 (Note that "rejected" messages include viruses.)
Those statistics do not include "User unknown" bounces from joe-jobs,
which would make our spam/total ratio closer to 0.8.
> Could you publish the patch somwhere at www.mimedefang.org ?
I'm not happy with it yet; I need to understand the Sendmail source
a little better, and then if Claus doesn't reply, I will post it
publicly.
Regards,
David.
Andrzej Adam Filip
> Andrzej Adam Filip wrote:
>>>I've submitted a patch to Claus that adds a new possibility for
>>>SuperSafe called "PostFilter". It defers the commit until after
>>>milter says it's OK to deliver the message.
>
>>I wish you luck with making sendmail.org accept your donation.
>
> I will need it... they've rejected all my previous ideas... :-(
They seem to be "a little bit" too conservative in accepting donations
[but they will have to maintain all the "donations" later].
>>How many percent of messages is rejected at your site by milter ?
>>[spam/e-mail ratio]
>
> Our stats are at http://www.roaringpenguin.com/canit/stats.php
> (Log in as demo/demo).
>
> It looks like in the last 30 days, we accepted
> 9760 messages and rejected 14081, for a spam/total ratio of
> 0.59 (Note that "rejected" messages include viruses.)
>
> Those statistics do not include "User unknown" bounces from joe-jobs,
> which would make our spam/total ratio closer to 0.8.
60%+ milter reject ratio makes your idea *very interesting*.
>>Could you publish the patch somewhere at www.mimedefang.org ?
>
> I'm not happy with it yet; I need to understand the Sendmail source
> a little better, and then if Claus doesn't reply, I will post it
> publicly.
Let us know as soon as it is made public.
--
Andrzej [en:Andrew] Adam Filip an...@priv.onet.pl an...@xl.wp.pl
http://anfi.homeunix.net/ http://slashdot.org/~anfi
*Random Epigram* :
Even if you can't control fate, you can control yourself.
-- Anon.
Claus Aßmann
> I've submitted a patch to Claus that adds a new possibility for
> SuperSafe called "PostFilter". It defers the commit until after
> milter says it's OK to deliver the message.
After some changes it's in the CVS tree now.
--
A: Maybe because some people are too annoyed by top-posting.
Q: Why do I not get an answer to my question(s)?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?