PIN Alan Connor: C-R in corporate mail schemes
N1POP
Your anti-spam system, as you say, uses a three-step process to
examine and possibly challenge email. Can you tell me if that system
has been used in a corporate (high-volume) mail environment? The
reason I ask is because my company has over 21,000 users and we
receive an average of 50,000 legitemate emails per day. Like any
other company, however, we are also receiving more than 100,000 spams
per day and it's not only a burden on our users who have to clean
their inboxes but on the servers that have to receive and process the
spam. We are always looking at and reviewing solutions to the spam
problem, and something like yours seems worth reviewing.
Do you think your system would be useful or capable in such an
environment?
Nancy McGough
recommend that you read this:
<http://www.templetons.com/brad/spam/challengeresponse.html>
and then, if you still want to use it, try it out at one of the
mail hosting providers that offer this. I list a number here:
<http://www.ii.com/internet/messaging/imap/isps/>
Some of the ones that offer C/R filtering are:
bluebottle.com
mailsnare.com
geekmail.com
mailblocks.com
Some of these are free or have free trial periods so it won't
cost you much (in time or money) to try out challenge/response
filtering and see if it's something that will work for your
company.
Hope this helps,
Nancy
--
Nancy McGough
Infinite Ink ~ <http://www.ii.com>
Deflexion & Reflexion ~ <http://deflexion.com>
Malcolm Dew-Jones
: Alan,
Gee, sounds pretty risky to me.
The pre-filter required to "prevent" false challenges might remove 80% of
the spam (if you're lucky).
That leaves 20,000 messages to challenge. My experience in testing spam
for replyable addresses would make me suggest that, as a ball park figure,
10% of that mail will have spoofed, but real, addresses.
That means you send about 2,000 challenge messages per day to people who
have not sent you mail - in other words your organization is likely to be
sending thousands of spam messages every day.
( Coincidently, the mail that gets through a spam filter will typically be
the most legitimate looking, which will likely depend to some extend on
the mail addresses possibly being valid. So after filtering, it would not
be surprising if the percentage of replyable addresses goes up, so that
the percentage of false challenges also go up. )
If you do challenge, it has to occur as a rejection during the initial
smtp connection.
Frostbite
If you really want to hassle clients, why not make them dance for
you, too?
--
parsimonious withdrew degassing sal bloomington aeneas belate outrageous
marriage tapeworm akron acute geopolitic pyle giuliano premature hiroshima
colgate bromfield shulman centenary repartee orthodontic bellboy manage
Jem Berkes
Since you posted this in a public group, I gather that you want responses
from everyone.
> Your anti-spam system, as you say, uses a three-step process to
> examine and possibly challenge email. Can you tell me if that system
> has been used in a corporate (high-volume) mail environment? The
> reason I ask is because my company has over 21,000 users and we
> receive an average of 50,000 legitemate emails per day. Like any
> other company, however, we are also receiving more than 100,000 spams
If bandwidth is free for your company, then sending out challenges to all
the incoming spam might not cost you anything (immediately). However, the
problem with spam is generally that it eats up bandwidth, not that it is
all that annoying. Installing anything like SpamAssassin will kill spam.
Using C-R on corporate mail servers will eat up tons of bandwidth. On the
other hand, using one of many freely available DNSBLs can instantly save
you bandwidth and kill a signficant % of spam without any complex
processing. Cautious selection of a DNSBL will also avoid false-positives.
Another thing to note is that those 100,000 spams almost all have forged
return paths. By sending out 100,000 challenges a day you may be spamming
other people. They might not like that.
--
Jem Berkes
http://www.sysdesign.ca/
Thor Kottelin
Jem Berkes wrote:
> Cautious selection of a DNSBL will also avoid false-positives.
Which DNS-based blacklist doesn't cause false positives?
Thor
Jem Berkes
>
>> Cautious selection of a DNSBL will also avoid false-positives.
>
> Which DNS-based blacklist doesn't cause false positives?
Sorry, more accurately: minimize false-positives. Nothing, not even humans,
can entirely avoid false-positives (if you disagree try manually sorting
200 spams a day).
For those who are members of the project, wpbl.pc9.org avoids false-
positives by design since past legitimate mail effectively whitelists
relays.
sbl.spamhaus.org is quite careful in their netblock selection. They list
servers engaged in professional spamming operation (spamhauses).
relays.ordb.org has, in my experience, only lists hosts that can be tested
and determined to be open relays of some sort (e.g. open proxy).
Jonathan de Boyne Pollard
TK> Which DNS-based blacklist doesn't cause false positives?
Any empty one will do.
N1POP
> > Alan,
>
> Since you posted this in a public group, I gather that you want responses
> from everyone.
Well, if I could spell PING, maybe Alan would have responded. :)
I was more interested in how his system would do when confronted with
a volume of 150,000 messages a day, two-thirds of which are spam.
Email would not have to be noticeably slowed, and there must be
absolutely no false positives.
I know the issue about sending a challenge to a forged address, and
any CR system we adopt (if we even go that route) would need to send a
challenge to the envelope sender (we see the envelope), assuming that
is a valid address.
Malcolm Dew-Jones
: Jem Berkes <j...@users.pc9.org> wrote in message news:<Xns94A6BA97BD4...@130.179.16.24>...
The envelope sender is no more dependable than any other bit of
information in the message. It is still just a bit of text specified by
the sender.
Telnet to port 25 on your own mail server and try it out.
If you challenge then it has to occur as a rejection during the smtp
conversation.
(That might cause an open relay to then send a bogus bounce, but there's
nothing you can do about that anyway. E.g. even if you refused to open
the tcp connection at all then the open relay is going to do whatever it
is configured to do - such as sending a bounce message.)
Alan Connor
The design of an effective mail filter using Challenge-Responses
----------------------------------------------------------------
pass-list ---------> block-list (content-filter) ---------> Challenge-Response
Along with many others, I use a filter designed like the above:
I never see any spam.
I send out very few C-Rs.
I don't have to mess with my filter at all.
Spam is just gone from my life, no muss and no fuss.
The only complaint I have ever had was from a domain that turned
out to be on the dnsbl (dns blacklist) for spamming.
Anyone can reach me with no problem if they use their real return
address. And read their mail...
(In some ways, Challenge-Responses are like Caller-ID for phones: Someone
who wants to communicate with you must use their real return address, and
actually read the mail there.)
On the rare occassion that I send a C-R to an innocent party (I have to
assume that his has happenned at least once.) whose address has been forged,
then I have done them the favor of letting them know that it is being forged,
which can cause a person one hell of a lot of trouble.
They have my real address and are told in the C-R that I have a copy
of the mail if they need it.
Like I said: ONE complaint, ever, and from spammers.
C-Rs are only one stage, and the final stage, of an effective mail filter
that is designed like so:
PASSLIST
Where every friend, associate, commercial mailing list, discussion
mailing list, business, and non-profit org is given free access
to my inbox with my consent.
Anyone you send mail to must be passlisted.
This is where you make SURE you get all the mail you REALLY need
to get, or just want.
BLOCKLIST (Content-Filter)
General, to kill most of the spam. You can use SpamAssasin here
called by procmail. Or just use good procmail recipes.
And specific, where domains and addresses are killed for various
offenses, generally repeated C-Rs that aren't returned.
(Do remember to blocklist your own address, or some clown will
send mail to you with your own address on the From: line and
set up a mail loop.)
You just can't catch all the spam with a content-filter like
SpamAssassin. If you even try, you will end up losing mail
that isn't spam, and getting some spam anyway. And you have
to update your filters all the time to keep abreast of the
spammers.
(As well as save the spam to look through to make sure there
isn't any mail you want in there, which rather defeats the
whole purpose of a spam filter, in my opinion. I don't want
to ever see that garbage.)
Better to just kill the mail you KNOW is spam, and send a
C-R to the few that your content filter isn't sure about.
Most people that use these filters block any mail from
non-passlisted addresses that doesn't have a valid local
address alone on the To: line.
This kills, of itself, the vast majority of spam, so that
no un-necessary C-Rs are sent out to whatever address
they have chosen to present as their return address.
CHALLENGE-RESPONSE
A little note that asks the receiver to paste a password on the
Subject: line and return it.
The Subject line on the C-R should have Re: original subject
on it.
This forces people to use their actual email address if they want
me to see their posts.
(And to read their mail...)
This is only required once. From then on, it's as if there is no
filter on my mail from their perspective.
AC
--
Pass-List -----> Block-List ----> Challenge-Response
The key to taking control of your mailbox.
http://www.uwasa.fi/~ts/info/spamfoil.html
http://tinyurl.com/3c3ag
N1POP
> The design of an effective mail filter using Challenge-Responses
I've seen all this before, but it doesn't answer my questions: could
your system keep up with 150,000+ messages per day on a corporate system?
What kind of initial effort did you have when putting your system in
production? For example, our whitelist would have to support 21,000
separate clients, each with a possible 1 to 500 users. The time and
effort to populate such a whitelist can be daunting, to say the least.
The spam filters would have to handle 150,000 messages with zero false
positives and no appreciable slow-down in delivery. One or more false
positives and any delays in mail delivery are just not acceptable by
corporate.
And the CR portion would have to be extremely configurable, sending
challenges that are closely related to the particular business unit to
which the mail was sent (it's not always part of the address) so our
correspondents aren't too confused by the challenge.
Mark Crispin
> I've seen all this before, but it doesn't answer my questions: could
> your system keep up with 150,000+ messages per day on a corporate
> system?
> What kind of initial effort did you have when putting your system in
> production?
Why don't you bring it up on your system in production and see for
yourself? Then you will know the answer.
> The spam filters would have to handle 150,000 messages with zero false
> positives and no appreciable slow-down in delivery. One or more false
> positives and any delays in mail delivery are just not acceptable by
> corporate.
If these are your criteria then I respectfully suggest that you review the
extensive traffic about this subject which has taken place on this
newsgroup.
-- Mark --
http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
Jonathan de Boyne Pollard
Of course you have. He stopped The Herculean Effort for a while, but he seems
to be back after a short break with renewed zest and vigour and still has a
good chance of making the magic 20 within the permitted time.
Alan Connor
Isn't that nice?
Spammers REALLY hate these sorts of mail filters.
Because they can't beat them.
What's my Breidbart Index now, sockpuppet?
What are you going to do about it?
Other than sit on it and rotate, I mean....
The design of an effective mail filter using Challenge-Responses
Jem Berkes
> ----------------------------------------------------------------
>
> pass-list ---------> block-list (content-filter) --------->
> Challenge-Response
I think that's a nice approach, actually. Not suitable for thousands of
users, but still pretty good.
N1POP
> On Wed, 10 Mar 2004, N1POP wrote:
>
>> I've seen all this before, but it doesn't answer my questions: could
>> your system keep up with 150,000+ messages per day on a corporate system?
>> What kind of initial effort did you have when putting your system in
>> production?
>
> Why don't you bring it up on your system in production and see for
> yourself? Then you will know the answer.
Because putting up a system - any system - in production without proper
testing and preparation is just asking for trouble.
>> The spam filters would have to handle 150,000 messages with zero false
>> positives and no appreciable slow-down in delivery. One or more false
>> positives and any delays in mail delivery are just not acceptable by
>> corporate.
>
> If these are your criteria then I respectfully suggest that you review
> the extensive traffic about this subject which has taken place on this
> newsgroup.
Thanks, but as you know, the people who do the R&D are often not the
suits who make the demands. I am in the not-so-unique position of
reviewing and possibly testing options that are given to me. I have
read the various discussions here, but I wanted to discuss it with Alan
since he is so adamant that a system like his works.
I will then make my recommendations to the corporate, who will ignore my
input and select what they want anyway (as long as I'm well paid, they
can ignore me all they want).
Mark Crispin
>>> could your system keep up with 150,000+ messages per day on a
>>> corporate system? What kind of initial effort did you have when
>>> putting your system in production?
>> Why don't you bring it up on your system in production and see for
>> yourself? Then you will know the answer.
> Because putting up a system - any system - in production without proper
> testing and preparation is just asking for trouble.
You have a choice in determining your answer.
You can bring it up on your system in production and obtain empirical
results.
You can rely upon the judgement of one or more of the individuals who have
weighed in on this topic.
Pick your poison.
> I have read the various discussions here, but I wanted to discuss it
> with Alan since he is so adamant that a system like his works.
I do not believe that you have read the discussions thoroughly. I suggest
that you do so.
I suggest also that you do a little bit of research into the backgrounds
and qualifications of the various individuals who have commented on this
topic. Certain of these individuals have years or even decades of
experience in building scalable email systems. Certain names appear
frequently in the email RFCs.
Others do not.
I believe that when you go through this exercise, the answers to your
questions will become obvious.
D. Stussy
> Thanks, but as you know, the people who do the R&D are often not the
> suits who make the demands. I am in the not-so-unique position of
> reviewing and possibly testing options that are given to me. I have
> read the various discussions here, but I wanted to discuss it with Alan
> since he is so adamant that a system like his works.
Apparently, you haven't figured out that AC is "full of it" yet.
N1POP
> Do you think your system would be useful or capable in such an
> environment?
Apparently not, since Alan has refused to directly respond to support
his system. I guess everyone was right; any system that uses CR is not
a viable solution.