Re: postfix, ldap and performance
Patrick Ben Koetter
> we run a free email service for between 500,000 and 1,000,000 users,
> processing about 2,000,000 emails a day. The mail is currently handled by 4
> load balanced SMTP servers with authentication handled by 4 load balanced
> LDAP servers running Novell's eDirectory. We have had complaints from the
> admins of these LDAP servers that postfix is making too many queries to the
> LDAP servers. I think the figure is between 4 and 6 lookups per email, this
> possibly varies depending on whether the mail is destined for one of our own
> mailboxes or going external. We are now faced with a proposal to develop and
> implement a caching server which will run on our SMTP servers and
> concentrate and cache queries to the LDAP. The intention being that by
> concentrating the connections we will reduce concurrent connections to the
> LDAP servers and that by caching we will reduce hits on the LDAP. (The real
> benefit of this would probably be that a single query could obtain all
> attributes that postfix may require for a single email transaction and thus
> eliminate future queries against the same account for that transaction
> rather than for future emails).
>
> I have some queries and concerns relating to this. First my queries:
>
> 1. can somebody point me to a detailed document on how postfix used LDAP?
man 5 ldap_table would be a good start. It contains some performance tuning
tips as well e.g. using the "domain" parameter in LDAP queries to reduce the
query load.
> 2. would proxymap be a viable candidate to handle this situation?
Yes, proxymap is what you want to use to bundle database connections:
..._maps = proxy:ldap:$config_directory/ldapQuery.cf
> 3. how do other high(er) volume sites handle authentication?
We use the Cyrus SASL ldapdb plugin on a company site that has 15.000 users
who do about 100.000 messages a day. The ldapdb plugin is stable and works
well. It requires proxy authentication i.e. the plugin needs to authenticate
with the openLDAP server before it is allowed to do the user authentication.
We configured proxy authentication using DIGEST-MD5 and tested with EXTERNAL
i.e. using a TLS client certificate. However the time we lost because using
TLS was significant (I don't recall the exact figures) and so we stayed with
DIGEST-MD5.
> And my concerns, on which any comments would be appreciated:
Sorry, running out of time right now.
p@rick
>
> 1. this seems to be offloading work that should be done by the LDAP to the
> mail server
> 2. caching systems often become bottlenecks as the number of cached entries
> grows
> 3. this is adding complexity to a functioning system
> 4. since this is what LDAP is designed to do, why re-invent the wheel?
>
> It is probably apparent from the above what my views on this proposal are,
> but I will welcome all and any input, particularly if it serves to change my
> mind.
>
> Thanks,
>
> mike
--
The Book of Postfix
<http://www.postfix-book.com>
saslfinger (debugging SMTP AUTH):
<http://postfix.state-of-mind.de/patrick.koetter/saslfinger/>
Patrick Ben Koetter
> Thanks Patrick,
>
> nice to know that proxymap is probably the right tool (I found that
> information in your book, thanks for that too)
>
> I had read the ldap_table man page and though this provided useful
> information it does not (to me anyway) explain the sequence of and reason
> for the numerous hits against the LDAP. I am sure these are justified and
> necessary, I am just looking to understand it better.
Postfix splits the search string into substrings and does searches for them as
well. The specific sequence in which these search strings are used is
determined by the mumble_maps parameter you use. Alias-maps are searched in a
different order than transport maps, for example. You will find the search
order documented in the corresponding man page.
> As regards proxymap, I came across a reference on google that stated this
> would not work for aliases. Is this true? And also how does it perform under
Not to my knowledge.
> load? As mentioned we host somewhere approaching 1,00,000 mail users and
> this is increasing by about 1,000 every day.
proxymap was specifically designed to perform under load. Before proxymap
people would complain that their DB would report "too many open connections"
and that latency was too high. The job of proxymap is to eliminate both
problems - open only one connection and keep it open so reconnects aren't
required and queries can start immediately.
I don't recall anybody doing load tests though. You might need to find our
yourself what performance increase you gain and if that is enough for your
requirements. But I remember that others with a userbase as large as yours use
proxymap and that they didn't have any problems.
HTH,
p@rick