Limitations of smtpd_proxy_filter
Stefan Foerster
defer transmission of received messages to a SMTP proxy until the
message receiption completes (smtpd_proxy_options = speed_adjust).
However, compared to the content_filter setting, it stil lacks some
functionality: A content filter supports all (well, the ones I'm aware
of) the settings which are valid in a transport(5) map. It thereby
provides a very easy way to realize DNS based kiad balancing by
specifying a content filter setting like
"filter:mx-filter.example.com:10024", where the zone
"mx-filter.example.com" holds several MX entries pointing to the
content filter machines. As far as my understanding goes that is
possible because the content_filter setting merely overrides the
nexthop setting, making all standard functionality (MX lookups,
queuing etc.) available.
So, my question is: Are there any plans to further enhance Postfix's
functionality as far as SMTP proxy filtering is concerned, and make
that kind of load balancing available to pre-queue setups using
smtpd_proxy_filter?
Of course, I realize that a proxy setup is much more time critical
than a content_filter setup. One would have to set a low timeout for
trying any of the proxy filters, and one would perhaps need a way to
specifiy different timeouts for each try (e.g., specify a short
timeout for the first proxy filter that is contacted and a long
timeout for the last one - better get a message filtered at all than
to discard it).
Still, getting load balancers out of a mail setup would be a nice
thing.
Stefan
Wietse Venema
> So, my question is: Are there any plans to further enhance Postfix's
> functionality as far as SMTP proxy filtering is concerned, and make
> that kind of load balancing available to pre-queue setups using
> smtpd_proxy_filter?
For TRUE load balancing you need a stateful system that remembers
which connections are going where. You could try the nginx reverse
proxy; it speaks SMTP among many protocols, and is often used for
software-based load balancing.
For STATISTICAL load balancing you can get by with multiple IP
addresses per proxy filter "host" name. However this requires new
code in Postfix to randomize the address order. You may know that
the Postfix smtpd_proxy client does not make its own SMTP connections.
Instead it uses the same Postfix library functions for making a
connection over TCP, UNIX-domain sockets, System V streams, etc.
as all other Postfix programs (except the SMTP client, which does
its own name-to-address translation). Changing this library means
changing applications that call into this library (Postfix invokes
the _connect() functions via function pointers, so a simple "#define
old(x,y,z) new(x,y,z,0)" won't do the job.
I have a bunch of deliverables in the second and third quarters
of 2010, so I must get Postfix 2.7 out in the first quarter. This
means I have few cycles left for Postfix 2.7. I would like to spend
those on postscreen.
Wietse
Stefan Foerster
> For STATISTICAL load balancing you can get by with multiple IP
> addresses per proxy filter "host" name. However this requires new
The main difference I see here is that delivery to a content_filter
will try more than one server, giving us not only (statistical) load
balancing, but also some kind of redundancy.
> code in Postfix to randomize the address order. You may know that
> the Postfix smtpd_proxy client does not make its own SMTP connections.
> Instead it uses the same Postfix library functions for making a
> connection over TCP, UNIX-domain sockets, System V streams, etc.
> as all other Postfix programs (except the SMTP client, which does
> its own name-to-address translation). Changing this library means
> changing applications that call into this library (Postfix invokes
> the _connect() functions via function pointers, so a simple "#define
> old(x,y,z) new(x,y,z,0)" won't do the job.
>
> I have a bunch of deliverables in the second and third quarters
> of 2010, so I must get Postfix 2.7 out in the first quarter. This
> means I have few cycles left for Postfix 2.7. I would like to spend
> those on postscreen.
Understood. I can live with deploying load balancers. Thanks anyways!
Stefan
Wietse Venema
> * Wietse Venema <wie...@porcupine.org>:
> > For STATISTICAL load balancing you can get by with multiple IP
> > addresses per proxy filter "host" name. However this requires new
>
> The main difference I see here is that delivery to a content_filter
> will try more than one server, giving us not only (statistical) load
> balancing, but also some kind of redundancy.
That would work only when the smtpd_proxy client is modified to
use the SMTP connection cache. Otherwise, a connection still has
to time out before the next host is tried. (*)
There is no such performance hit with true load balancers, which
maintain state about which hosts are busy/dead/other.
Wietse
(*) It takes major surgery to make the smtpd_proxy client as smart
as the Postfix SMTP client (address randomization, connection
caching, bug workarounds, and more) Namely, to make the Postfix
SMTP client usable as an smtpd_proxy client.
Wietse Venema
> One of the greatest improvements in the 2.7 tree is the ability to
> defer transmission of received messages to a SMTP proxy until the
> message receiption completes (smtpd_proxy_options = speed_adjust).
Can you be more specific about the benefits?
Wietse
Stefan Foerster
Number of filters is down by more than 40%. Comparing a typical day
before and after we deployed that feature:
http://www.incertum.net/~cite/speed_adjust.png
Other sites might not notice such a large impact: Our setup includes a
large number of clients connected via lines with little bandwidth
(2MBit/s) which is typically used for other purposes besides sending
mail. If the time a mail transmission takes is negligible, I expect
the results to be vastly different.
Furthermore (I'm not 100% sure about this, would be glad if you could
confirm this) it's much harder to DoS a Postfix server with that
patch: Previously, all you had to do was opening a number of
connections equal to the number of pre-queue content filter processes,
go through the initial (E)SMTP steps and then send a mail very slowly.
With "smtpd_proxy_options=speed_adjust", you'd have to DoS the Postfix
SMTP server itself, and between postscreen guarding it, high process
limits, access control and the stress adaptive server personality,
this one is well able to take care of itself.
Stefan
Wietse Venema
> * Wietse Venema <wie...@porcupine.org>:
> > Stefan Foerster:
> > > One of the greatest improvements in the 2.7 tree is the ability to
> > > defer transmission of received messages to a SMTP proxy until the
> > > message receiption completes (smtpd_proxy_options = speed_adjust).
> >
> > Can you be more specific about the benefits?
>
> Number of filters is down by more than 40%. Comparing a typical day
> before and after we deployed that feature:
>
> http://www.incertum.net/~cite/speed_adjust.png
As with the first public release release, Postfix avoids the need
for a forklist upgrade (that was geek talk for having to install
more powerful hardware). Now I can claim it as a "green" option.
> Other sites might not notice such a large impact: Our setup includes a
> large number of clients connected via lines with little bandwidth
> (2MBit/s) which is typically used for other purposes besides sending
> mail. If the time a mail transmission takes is negligible, I expect
> the results to be vastly different.
It also depends on how far away the clients are: each round-trip time
adds to the duration of a session (from this location near New York
I measure some 90ms RTT to europe and 175ms to Japan).
> Furthermore (I'm not 100% sure about this, would be glad if you could
> confirm this) it's much harder to DoS a Postfix server with that
> patch: Previously, all you had to do was opening a number of
> connections equal to the number of pre-queue content filter processes,
> go through the initial (E)SMTP steps and then send a mail very slowly.
As long as the number of SMTP server slots is finite, there will
always be saturation effects. The main trick of speed-adjust is
that you can run more SMTP server processes than filters, and
thereby reduce the impact of slow or non-cooperative clients.
Of course the system still saturates immediately when the PEAK
email volume exceeds the filter capacity. With after-queue filters,
saturation happens when the AVERAGE volume exceeds filter capacity.
> With "smtpd_proxy_options=speed_adjust", you'd have to DoS the Postfix
> SMTP server itself, and between postscreen guarding it, high process
> limits, access control and the stress adaptive server personality,
> this one is well able to take care of itself.
I have been looking for incremental steps to improve Postfix's
handling of non-cooperating clients, and speed-adjust is part of
those steps. Postscreen greylisting support should also help a
lot, but that will probably not be in Postfix 2.7.
Of course all these things do is increase the threshold. No system
except a wire cutter can deal with unlimited amounts of traffic.
Wietse