sendmail 8.13, access_db and ldap

[Michael Osten]

I need some advice on moving sendmail's access_db to ldap. I have it
working, and have gone the overkill route on indexing with ldap.

index sendmailMTAAliasGrouping eq
index sendmailMTAHost eq
index sendmailMTACluster eq
index sendmailMTAClassName eq
index sendmailMTAMapName eq
index sendmailMTAKey eq
index sendmailMTAMapValue eq
index sendmailMTAClassValue eq
index sendmailMTAAliasValue eq

ldap is a replica on localhost, and the ldap search is only on the
specific branch where the record lives,

i.e. ou=inbound,ou=sendmail,dn=foo,dn=net,o=ldap

We recieve approx. 4-5 messages a second, and even with the indexing,
the server is getting killed with iowait. I'm getting the impression
that ldap just can't handle the amount of requests from sendmail.

Any thoughts?

[David H.]

Could you give a bit more deatil on the setup? Server Specs, how it is
handled, what else is running and so on? I have seen LDAP servers do
more than 100 request a second, so 4-5 a second should nto be an issue :)

-d

[M. Osten]

Single dual xeon, 6 gigs of ram and 6 15k drives RAID 1+0. Machine
acts simply as a SMTP gateway (one of several MX's) and does
ldap-routing (that all it does).

Its not that its only doing 4-5 LDAP requests a second. A conservitive
estimate is that it does 60 LDAP lookup's per email (when access_db is
in ldap).

[Claus Aßmann]

Michael Osten wrote:
> I need some advice on moving sendmail's access_db to ldap. I have it
> working, and have gone the overkill route on indexing with ldap.

> We recieve approx. 4-5 messages a second, and even with the indexing,

> the server is getting killed with iowait. I'm getting the impression
> that ldap just can't handle the amount of requests from sendmail.

The anti-spam rules do a lot of queries, for example, they "search"
through the address parts.

> Any thoughts?

Some MTA author suggested to use a different map type, e.g., Berkeley
DB, when some user asked a similar question. That allows several
thousands queries per second without breaking a sweat (thanks to
caching).
--
A: Maybe because some people are too annoyed by top-posting.
Q: Why do I not get an answer to my question(s)?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[M. Osten]

As it is now we use Berkeley. The problem being that it is time
consuming to have to go to each machine to change an entry in the
access_db. It would be much nicer to control it all via ldap. Alas,
I'm thinking that it doesn't work. It *may* work for low traffic
sites, but not for us. (unless some kind soul can point me in the right
direction).

The line of thinking (for google archive) is that I will write a script
to run from cron to pull the access data from ldap and rehash the file
every few minutes.

[Howard Chu]

Yes, anybody who's looked at Sendmail's access map implementation would
see this is the case.

The sendmail access db basically has these fields:
[<access>]<address pattern> TAB <result>

The access mapper looks up every possible permutation of
<access>:<address pattern> which can lead to literally hundreds of
queries per email. This seems OK when using an embedded database, but
it's horrendous when querying an external data source.

For a given email sender and recipient, the domain name is searched:
user@
us...@some.sub.domain
some.sub.domain
sub.domain
domain

Also the IP address is searched:
ww.xx.yy.zz
ww.xx.yy
ww.xx
ww

<access> can be Connect:, From:, To:, or (absent). (I'm leaving out the
other types of accesses, just for clarity. Suffice to say, there are
other types, and they only add to the lookup overhead...)

For an email with a recipient us...@domain.com and sender
us...@domain.com the access db will be queried up to 64 times to verify
the sender and 12 times to verify the recipient. If the sender email
address has a three part domain that goes to 80 queries for the sender.
If there are multiple recipients, each recipient is queried
additionally. The worst case is always reached for addresses that aren't
in your map at all.

FYI, Symas Corporation has a Sendmail LDAP Accelerator product that will
store your Sendmail access map in LDAP and perform lookups using a
dedicated milter. The milter can verify an email address in a maximum of
two LDAP queries. For sites that use LDAP-based access maps, the
difference in performance is phenomenal. This product was developed for
Sendmail 8.12, the version for 8.13 is still in beta test but you can
email sa...@symas.com for more details. (Don't reply to the above email
address, it doesn't exist.)

--
-- Howard Chu
Chief Architect, Symas Corp. Director, Highland Sun
http://www.symas.com http://highlandsun.com/hyc
Symas: Premier OpenSource Development and Support