auto-whitelisting without the backscatter?
Vic
we use DNSBLs to reduce the amount of spam and it works quite well.
Unfortunately we occasionally block legitimate mail and we eventually
end up whitelisting the server or domain in question. Understandably
this can be a little frustrating for the sender and also for the mail
server admins. We would like to automate this if possible.
Googling around I've come across many SAV solutions but all of them
suffer from backscatter. What I'd like to do instead is setup a
verification method that would involved the sender visiting a website
that explains in more detail what happened and using some random/
unique key, allow them to whitelist themselves.
I've searched for examples of such a setup for sendmail and though I
have found mentions of them I've not come across any examples or
code.
Does anyone have any suggestions on how to go about it?
Thanks.
Vic.
Vic
Just to expand on how i see this implemented.
Part of the solution is in the rejection of the SMTP connection. When
we reject a message, the user gets back something like:
554 5.7.1 Rejected message from xx.xx.xx.xx - see http://somednsbl.org
The idea is to replace the above message with something like
554 5.7.1 Message rejected as it may be spam. If this is in error
please visit http://someplace.com/whitelist and quote ID2342342232 to
allow your messages to be delivered.
This way the user can whitelist themselves. The website would of
course be designed to prevent bots from autowhitelisting themselves.
Once the send is whitelisted the user would have to resend the
message. I've read the arguments about putting the burden on the
senders and frankly I have no problem with this.
Any known implementations of this?
Vic.
David F. Skoll
> The idea is to replace the above message with something like
> 554 5.7.1 Message rejected as it may be spam. If this is in error
> please visit http://someplace.com/whitelist and quote ID2342342232 to
> allow your messages to be delivered.
Bad idea. Bad, bad idea. The SMTP reply message is typically buried in
a confusing DSN that perhaps 0.5% of Internet users will understand.
Furthermore, some MTAs replace the text of the SMTP reply message
with a "friendly" error so your sender won't even see the web site
or ID.
No, the only sure-fire way to implement Challenge-Response is by crafting
and sending out challenge e-mails, and that concept is in a league of
evil unto itself.
Regards,
David.
Vic
> Vic wrote:
> > The idea is to replace the above message with something like
> > 554 5.7.1 Message rejected as it may be spam. If this is in error
> > allow your messages to be delivered.
>
> Bad idea. Bad, bad idea. The SMTP reply message is typically buried in
> a confusing DSN that perhaps 0.5% of Internet users will understand.
> Furthermore, some MTAs replace the text of the SMTP reply message
> with a "friendly" error so your sender won't even see the web site
> or ID.
>
David, while I agree SOME MTAs replace the text, not all do. In fact,
in my experience, most leave the message intact and simply quote it or
add to it.
Its true as well that not all understand what the reply message means.
BUT for those that do understand and receive the message intact why
not offer the choice? If it doesnt break any standards why not?
Vic.
Grant Taylor
> David, while I agree SOME MTAs replace the text, not all do. In fact,
> in my experience, most leave the message intact and simply quote it
> or add to it.
It's been my experience that MTAs are not the problem, rather MUAs are.
Namely Outlook (better known as LookOUT) hides the real DSN with a
supposedly ""Friendly error message. If you take said DSN and save it
as a .msg file and look at it with a text editor you will find the
original DSN text is still there. You just have to know how to find it.
(I've witnessed this behavior with LookOUT 2k / XP (2k2) / 2k3 with
and with out Exchange in the mix.)
> Its true as well that not all understand what the reply message
> means. BUT for those that do understand and receive the message
> intact why not offer the choice? If it doesnt break any standards why
> not?
Aside from MUAs hiding things from end users I don't see a problem with
this. Besides if the end user has an email provider worth their salt
they will be able to go back through logs and see "Oh, the receiving
server rejected the message with this '<bla><URL><bla>' to say. Let's
take a look at <URL>... ... ...Oh..." and then help the end user.
I'd suggest that you not check your DNSBL directly from Sendmail, but
rather have a milter like SpamAssassin or MIMEdefang check it for you.
That way you can fairly easily check a DB (via SA) of white listed
senders and / or recipients and bypass the DNSBL check. With the
entries stored in an SQL DB it would be trivial to create a web page to
allow end users to alter their own settings.
Grant. . . .
David F. Skoll
> David, while I agree SOME MTAs replace the text, not all do.
AFAIK, Microsft Exchange does, and that's widely-used (alas) by corporations.
> In fact, in my experience, most leave the message intact and simply
> quote it or add to it. Its true as well that not all understand
> what the reply message means.
Right. I'd guess about 1% of users would know what to do when given
such a DSN, especially given that the subject line is likely to be
"Undeliverable" or "Message failed - returned to sender" or something
like that.
> BUT for those that do understand and
> receive the message intact why not offer the choice? If it doesnt
> break any standards why not?
There are no "standards" for this. In fact, you are probably breaking
the standards; a 5xx reply code means "permanent failure; do not try again."
You're abusing it to mean "try again once you prove you're human."
Regards,
David.
David F. Skoll
> Aside from MUAs hiding things from end users I don't see a problem with
> this. Besides if the end user has an email provider worth their salt
> they will be able to go back through logs and see "Oh, the receiving
> server rejected the message with this '<bla><URL><bla>' to say. Let's
> take a look at <URL>... ... ...Oh..." and then help the end user.
Aiieee.... you've never had to support end-users, obviously. That's
all overworked sysadmins need: Requests from hundreds or thousands of
confused users for help interpreting DSNs....
Furthermore, some MTAs (including *cough* Sendmail) do not always log
the actual text of the 5xx failure message from a remote SMTP server,
but replace it with generic text. Try it yourself.
It boils down to this: If you're misanthropic, challenge/response is a
great solution. If you have any shred of consideration for other
users, you won't use challenge/response.
Regards,
David.
David F. Skoll
> Right. I'd guess about 1% of users would know what to do when given
> such a DSN, especially given that the subject line is likely to be
> "Undeliverable" or "Message failed - returned to sender" or something
> like that.
Just some anecdotal evidence: We run anti-spam software that rejects
unwanted mail with a 5xx error code. We include in the error code
a phone number for people to contact us in case we've rejected something
in error.
Of the over 200,000 times we've issued such a 5xx error code with our
phone number in it, guess how many times someone called?
Once.
Regards,
David.
Grant Taylor
> AFAIK, Microsft Exchange does, and that's widely-used (alas) by
> corporations.
Eh. The data is there somehow. See my other post about the fact that
it can be extracted out of .msg files. I guess it is possible that it
is encoded in some way that it is later recreated, though things are exact.
> Right. I'd guess about 1% of users would know what to do when given
> such a DSN, especially given that the subject line is likely to be
> "Undeliverable" or "Message failed - returned to sender" or something
> like that.
I believe this to be how the error messages are translated to be ""friendly.
> There are no "standards" for this. In fact, you are probably
> breaking the standards; a 5xx reply code means "permanent failure; do
> not try again." You're abusing it to mean "try again once you prove
> you're human."
Um... I think you are a bit out side of the scope of what I understand
RFC (2)821 error codes mean. Rather that is to say that RFC (2)821
error codes are used for a given message / envelope, not across multiple
messages / envelopes. If a deliver fails because of security reason
(wrong encryption (5.7.x) and / or encoding (5.6.x)) it is perfectly
likely (and arguably encouraged) for the problem to be corrected and for
the sender to try re-sending the message with corrections.
Grant. . . .
Grant Taylor
> Aiieee.... you've never had to support end-users, obviously. That's
> all overworked sysadmins need: Requests from hundreds or thousands of
> confused users for help interpreting DSNs....
Um, actually that's my day job. Supporting circa 1,500 end users. The
organizations that I've worked for believed that our (IT staff) job was
to help the end user(s). (Note: I'm not saying that you don't believe
that, just making a statement.) If that means we hold lots of hands,
then guess what we are doing. Now if we spend too much time (read:
more than expected) holding hands, we will evaluate if things need to
change.
All said and done, most of the end users were able to be helped by a
small (level 1) help desk staff with few problems being passed on to
level 2. The only thing that ever made it beyond level 2 was something
that was a mis-configuration or a remote server that was doing something
stupid like not honoring ESMTP even though it said that it would.
(Thank you Cisco SMTP F&#k Up.)
> Furthermore, some MTAs (including *cough* Sendmail) do not always log
> the actual text of the 5xx failure message from a remote SMTP server,
> but replace it with generic text. Try it yourself.
Hum. I can't speak to the contrary with any firm evidence. However in
my short time of administering Sendmail systems (8+ years) I've never
had this happen to me. The closest that I've come is with a multi-line
SMTP response not being properly logged. However even at that I had
most of the information in the first line of the reply.
> It boils down to this: If you're misanthropic, challenge/response is
> a great solution. If you have any shred of consideration for other
> users, you won't use challenge/response.
I think (my opinion is) that C/R systems are an interesting idea and
might be ok for some very specific situations, but not for mass
consumption on production systems.
Grant. . . .