Solution for using MAPS, ORBS and adding headers like X-RBL-Warning or X-SPAM-Warning in Sendmail 8.11.3
Glen
add a warning header such as X-RBL-Warning or something similar when
an incoming mail message is from a sending machine that is listed in
MAPS or ORBS. I saw examples from 8.9 and 8.10 but they didn't quite
work for me. After finally printing out doc/op/op.ps and going through
all kinds of email messages, I finally found a working solution, and
wanted to send it back to everyone from whom I had previously asked for
help.
A significant parameter here was that I did NOT want to use any third-
party software patches/hacks/etc. I wanted something that would work
on sendmail out of the box. The third-party things I was referred to,
like the digitalanswers site, are much more powerful, but they are also
more work. I wanted just this one function, and wanted to be able to
apply it to future sendmail versions as cleanly as possible.
Please note that I am NOT a sendmail "expert" but I *was* able to
figure out rewriting rulesets. (Now I just need help adding my changes
back into config.mc! :-)
SOLUTION FOR SENDMAIL 8.11.x:
First, generate your sendmail.cf file, using the DNSBL features... in
this example I've truncated the error messages for brevity, since we
won't be returning errors anyway. This just generates the rulesets.
------------------------------
FEATURE(`relay_hosts_only')
FEATURE(`access_db')
FEATURE(`blacklist_recipients')
FEATURE(dnsbl, `blackholes.mail-abuse.org', `Rejected - please contact')
FEATURE(dnsbl, `dialups.mail-abuse.org', `Rejected - please contact')
FEATURE(dnsbl, `relays.mail-abuse.org', `Rejected - please contact')
FEATURE(dnsbl, `inputs.orbs.org', `Rejected - please contact')
FEATURE(dnsbl, `outputs.orbs.org', `Rejected - please contact')
FEATURE(dnsbl, `manual.orbs.org', `Rejected - please contact')
FEATURE(dnsbl, `spamsources.orbs.org', `Rejected - please contact')
FEATURE(dnsbl, `untestable-netblocks.orbs.org', `Rejected - ')
FEATURE(dnsbl, `spamsource-netblocks.orbs.org', `Rejected - ')
FEATURE(`delay_checks',`friend')
------------------------------
Now, you have to modify the sendmail.cf file itself. There are three
parts to this:
Part 1 - The local configuration area - Add these lines at the bottom
of the local area, which is just above the Header definitions...
------------------------------
Kstorage macro
C{persistentMacros}{mm}
------------------------------
Part 2 - The header area - You next need to define the header. As you
might note, we are using macro {mm} as our macro. Add this to the
middle of the Headers area - I added mine just below the Received
headers and above the Resent-Date header.
------------------------------
H?${mm}?X-SPAM-Warning: Sending machine is listed in ${mm}
------------------------------
Part 3 - The ruleset modifications. In my config.mc example above, I
referenced all of the public MAPS and ORBS databases, including the
experimental ones. This is, no doubt, excessive for most people, but
since we're warning instead of rejecting, I figured, what the heck?
For most implementations, you'll want to choose only the databases
that make sense to you. See the MAPS and ORBS websites for help in
picking them.
At any rate, for each DNSBL feature entry you use, the system will
generate a six-line (well, five lines plus a leading comment) ruleset
group for the lookup. The LAST line of each ruleset group can be
identified by the presence of the $#error mailer. It looks like:
R<?>$+ $#error $@ 5.7.1 $: Rejected - please contact
The message will vary based on what message you use. What you need
to do is comment that out, and add a new line. Or, you could just
remove that line and replace it with the new line. The new line will
store the name of the database that matched in the ${mm} macro.
The new line is this:
R<?>$* $: $(storage {mm} $@ dialups.mail-abuse.org $) OKSOFAR
When you're done, it might look like:
# R<?>$+ $#error $@ 5.7.1 $: Rejected - please contact
R<?>$* $: $(storage {mm} $@ dialups.mail-abuse.org $) OKSOFAR
This change needs to be made for EACH ruleset group for which you
want to do warnings instead of rejections.
------------------------------
As I said above, I'm no expert. I just found that this works. Many
of you are doubtless much more of sendmail experts than I am. So please
don't abuse me and tell me what a terrible job I did with these rules.
I asked everyone, and nobody knew an answer, so I'm just offering this
as is, with no warranty, your mileage may vary. But I really love
sendmail, and I love MAPS and ORBS, and appreciated the fact that the
people at sendmail responded to my email, so I'm just trying to "give
back" to the sendmail community, etc., etc.
Also, I'm no expert at debugging! So please don't ask me to debug your
environment or setup. But one thing I found that was helpful in
debugging this was the "syslog" map. To use this, you add:
Ksyslog syslog
to the top of your sendmail.cf file (I added it just above the
Kstorage macro line), and then you can log things in the syslog. For
example, when I was trying to see if my macros got set, I added the
following line just below each of the changed ruleset lines in the
DNSBL lookup ruleset groups:
R<?>$* $: $(storage {mm} $@ dialups.mail-abuse.org $) OKSOFAR
R$* $: $(syslog dnsbl-test: CA: $&{client_addr} MM: $&{mm} $)
This printed a syslog entry in the system syslog for each time that
the rulesets ran. It would log the fact that the test ran, the client
address, and the resultant value of the {mm} macro. I could then watch
the syslog and see what was happening with live incoming mail. But take
this back out when you're done - it makes your syslog grow VERY fast!
------------------------------
As a postscript, it's beyond me to figure out how to incorporate these
changes in my config.mc file. But if anyone else is adventurous, and
would like to tell us how, I'd be grateful!
Glen
Glen Barney
SENDMAIL/MAPS/ORBS fan (NOT guru!)
gl...@ldscn.com
Per Hedeland
>Please note that I am NOT a sendmail "expert" but I *was* able to
>figure out rewriting rulesets.
Good work.:-)
> (Now I just need help adding my changes
>back into config.mc! :-)
See below.
>Part 1 - The local configuration area - Add these lines at the bottom
>of the local area, which is just above the Header definitions...
>
>------------------------------
>
>Kstorage macro
>C{persistentMacros}{mm}
You can just put these in a LOCAL_CONFIG section of your .mc file. It's
a good idea to use a macro that starts with an uppercase character
though, to avoid conflict with sendmail-internal ones (which always
start with lowercase). Using more than two characters helps too.:-)
>Part 2 - The header area - You next need to define the header. As you
>might note, we are using macro {mm} as our macro. Add this to the
>middle of the Headers area - I added mine just below the Received
>headers and above the Resent-Date header.
>
>------------------------------
>
>H?${mm}?X-SPAM-Warning: Sending machine is listed in ${mm}
This would also go in the LOCAL_CONFIG section.
>Part 3 - The ruleset modifications. In my config.mc example above, I
>referenced all of the public MAPS and ORBS databases, including the
>experimental ones. This is, no doubt, excessive for most people, but
>since we're warning instead of rejecting, I figured, what the heck?
Well, you will incur a lot of DNS lookups for each message...
>The message will vary based on what message you use. What you need
>to do is comment that out, and add a new line. Or, you could just
>remove that line and replace it with the new line. The new line will
>store the name of the database that matched in the ${mm} macro.
>
>The new line is this:
>
>R<?>$* $: $(storage {mm} $@ dialups.mail-abuse.org $) OKSOFAR
>
>When you're done, it might look like:
>
># R<?>$+ $#error $@ 5.7.1 $: Rejected - please contact
>R<?>$* $: $(storage {mm} $@ dialups.mail-abuse.org $) OKSOFAR
This is what you can't *directly* do from the .mc file, of course.
However if you instead of using FEATURE(dnsbl), do the "right" thing
first off, there are several possibilities. The simplest is perhaps to
copy cf/feature/dnsbl.m4 to (say) cf/hack/dnsblwarn.m4, and change the
last rule in the latter per above (you might want to simplify the file a
bit too - you never use the second argument, and you may not care for a
default). Now you can invoke this from your .mc with
HACK(`dnsblwarn', `blackholes.mail-abuse.org')
HACK(`dnsblwarn', `dialups.mail-abuse.org')
and so on. You will of course have to carry this file along when you
upgrade etc, and you are relying on the fact that the divert(8) puts the
rules in the right place - the latter *is* documented, at the very end
of cf/README, but perhaps not really with the intent that it should be
used directly.
You can alternatively put the rules in a Local_check_relay ruleset
(which would go in a LOCAL_RULESETS section of your .mc of course). That
way you both get everything in one file, and avoid dependancy on the
divert(8) - but of course it's really boring and error-prone to manually
repeat all the rules, with minor edits for each DNS-blocker.
Perhaps the cleanest way is to define a little ruleset of your own that
has just the check lines, and then invoke that from your
Local_check_relay for each blocker. Perhaps something like this (totally
untested!):
LOCAL_RULESETS
SDnsblWarn
R$* $: $&{client_addr} $| $1
R::ffff:$-.$-.$-.$- $| $* $: <?> $(host $4.$3.$2.$1.$5. $: OK $) $| $5
R$-.$-.$-.$- $| $* $: <?> $(host $4.$3.$2.$1.$5. $: OK $) $| $5
R<?>OK $* $: OKSOFAR
R<?>$* $| $* $: $(storage {mm} $@ $2 $) OKSOFAR
SLocal_check_relay
R$* $: $>DnsblWarn blackholes.mail-abuse.org
R$* $: $>DnsblWarn dialups.mail-abuse.org
and so on.
There is a problem with all of the suggestions so far, including your
original one - you will always check *all* the blockers, which is a bit
wasteful since you only put one of them in the header (though that could
be changed, of course) - it would be better to stop when you got a
hit. That can be rather easily fixed in the last case by having the
DnsblWarn ruleset return different things depending on whether there was
a hit or not, and check for that in the Local_check_relay - left as an
exercise for the reader.:-) (You can do it with the same number of
rules...)
--Per Hedeland
p...@bluetail.com
Glen
> Hi everyone! For a while now I've been trying to figure out how to
> add a warning header such as X-RBL-Warning or something similar when
> an incoming mail message is from a sending machine that is listed in
> MAPS or ORBS.
> I finally found a working solution, and
> wanted to send it back to everyone from whom I had
> previously asked for help.
After posting the solution, I received the following comments from
the staff at the Dialup User List (DUL) project at MAPS. (Their website
is http://www.mail-abuse.org/dul/). I wanted to pass them on to the
list here, and am doing so with their permission.
The DUL Maintainers wrote:
------------------------------
The DUL Maintainers have a problem with this scheme for the following
reasons:
* It doesn't do anything to save bandwidth caused by spam - you still
have to receive it to attach a new header to it
* Only one mailer for Win32 / MacOS (Eudora) can filter on arbitraty
headers
* You still have to receive it to filter it - you still have to pay for
it
* It causes complaints to US from people who think we're spamming them -
Eudora shows all x-headers wether you tell it to show all headers or not
* It doesn't warn of possible incorrect DUL entries to listees or to
deter spammers - they never see the warning
Over all, this is a Bad Idea[tm].
If you want to introduce user-controlled filtering, find a way to still
refuse mail at the SMTP incoming level and allow users to enable or
disable DNS based lists. We have one such solution in the stop.mail-
abuse.org forwarding system.
------------------------------
In the conversation that ensued following their email to me, I asked
them how they would implement such a user-controlled filtering system.
They pointed me back in the direction of a behavior in Sendmail that I
had noted before - namely, the SPAMFRIEND feature of `delay_checks'.
MAPS suggests that each user be allowed to choose whether to receive
blacklisted mail - with those users who elect NOT to receive it still
getting the benefits of the rejection of their mail at the SMTP server.
This is quite easy to do. Instead of putting in my modifications, just
use the out-of-the-box sendmail.cf that you generated. The people who
want their mail filtered will get the filtering with no further action.
And the people who do NOT want filtering should be added to the access
database using the SPAMFRIEND option.
MAPS has been quite helpful, and I appreciate their taking the time to
email me on this matter.
Just wanted to pass that recommendation on!
Glen
Glen
> Good work.:-)
> You can just put these in a LOCAL_CONFIG section of your .mc file...
> This would also go in the LOCAL_CONFIG section...
> Perhaps the cleanest way is to define a little ruleset of your own...
> and check for that in the Local_check_relay - left as an
> exercise for the reader.:-) (You can do it with the same number of
> rules...)
> --Per Hedeland
> p...@bluetail.com
Per -
That was an AWESOME posting, thanks! I won't be free to figure it all
out for some time because of project deadlines, but I'm saving this in
my archive and looking forward to doing everything you suggested! If
I get it all cleaned up, I'll post results here next month or so!
Thanks again for the AWESOME posting!!!
Excited,
Glen