Commentary: Its time to rethink spam control
Jon R. Kibler
Let’s face it, we have made a valiant effort to control spam within our current email model, but even our best efforts have been a dismal failure. We have implemented mail filters, the access database, various blackhole lists, and dozens of other schemes to stop spam. Yeah, we probably stop half of spam on a regular basis, and on a good day we may block the majority of it. But the bottom line is, many, if not most, of our users are still getting more spam than they are getting legitimate business email. Clearly, we are doing a lousy job of stopping spam.
But why are we failing? We are expending tens of thousands of dollars to block spam, but we are still wasting thousands of more dollars in lost productivity (not to mention wasted network capacity, computing power, and storage resources) dealing with the spam that manages to get past our road blocks. Our reasons for failing are not for lack of effort. In fact, I question whether our current efforts are even cost effective.
I believe that the reason we are failing to block spam is that we are trying to fix a broken, un-fixable, email architecture. The only way we will ever be able to control spam is to radically rethink the very premises upon which we have built our email systems. Re-architecting our email systems is exactly what I would like to propose. In my humble opinion, it is the only way we will ever solve our ever-growing spam problem.
(Note: I am not knocking sendmail or any other mail program. I am not bashing anything historical about the way email works. Email was, and still is, a great concept. It was built upon proven means of delivering messages. The problem is that what was great up until a few years ago has now become broken. This is not the fault of any designer, or programmer, or anyone else -- except for spammers. If it wasn't for spammers, the email systems we have today would be fantastic!)
Our current email architecture is based upon the centuries proven means of delivering messages: snail mail, as we call it today. It is the system that every country in the world uses to deliver their mail. It is a simple system -- you can send mail to anyone whose address you can obtain. Just like email today.
If snail mail and email both use the same architecture, then why do we not have a snail mail spam problem? If you think about it, the answer is simple: It costs to send snail mail. It doesn’t cost to send email (relatively speaking).
Well, is it time to start charging for sending email? Obviously, this is neither desirable nor doable. We have to look elsewhere for the spam solution.
The only place left to look is to examine our email architecture. And exactly what is that architecture? You can send email to anyone whose email address you can obtain. Ah-ha, the source of the problem: Anyone can send email to anyone else, you just need an email address.
To fix the spam problem, we really must change this basic email architecture. That is, change the basic email architecture from “I will accept email from anyone (except those on my blacklist)” to “I will accept email only from pre-approved sources.” We have already demonstrated that it is impossible to maintain an effective blacklist, so it is time to try something radically different: Filter email through a “whitelist.”
New idea? No, the idea of using whitelists is not new. However, the idea of changing our standard email architecture to one where email is accepted only from pre-approved senders does not appear to have been heavily debated. (Yes, I know there have been several discussion threads on various ways to implement custom whitelists, but I really think that whitelists need to become standard and we need to change our entire email approach.)
Radical approach? Yes. But it is the only way I believe that we will ever be able to effectively stop spam. And let’s face it, the spam problem is only going to get worse, not better. And no amount of regulation will ever even slow it down. We must have a technological fix, and I think this should be our next universal approach to the problem.
Yeah, yeah, yeah, I know that this raises all sorts of issues. You can probably think of ten million reasons why not to try this approach, you can probably think of thousands of problems that this approach may cause, and raise all sorts of other objections to this idea, but I think a careful examination of the issues raised by this approach will show that the problems are solvable and this spam solution has a chance of working if properly implemented.
First, let’s consider the type of email changes required. Then, let’s examine a couple of the operational issues involved.
What type of changes to sendmail would be required to implement this type of architecture? I would think the only change required would be the implementation of a standard whitelist feature. The whitelist would have to be at two levels: the entire email system, and the individual email user.
A system-wide whitelist would function like the access database, only in reverse. That is, unless you are in the whitelist database (system or user), then we will not accept email from you. (Yes, I know that you can make the access database behave like a whitelist, but in its current state, the access database itself does not have all the functionality required of a full whitelist feature.)
The system-wide database would allow the mail administrator to set up the list of addresses (domains, users, email address, etc.) that can send mail to any user. When mail comes in, the system-wide database would be checked to see if the sender (it would be nice to check both envelope and header, but let’s not go there for now) is allowed to send mail to users in this system. If the sender is not in the system-wide database, then the recipient’s whitelist would be checked.
Each email user should have their own whitelist (just like .forward). This would be the list of senders that the recipient has indicated they will accept mail from. (It may be desirable for each user to have the ability to reject senders as well.) The system would also have to have a mechanism that would automatically add to a user's whitelist all email addresses to which they send email.
With this conceptually simplistic change, email would now be received only from pre-approved users. Thus, no more spam.
Now, let’s examine some of the issues raised by implementing such a change.
1) I need people to be able to send me email without my pre-approving each sender.
I have several thoughts on this problem.
a) Just like the access database, the whitelist would be an optional feature. You don’t want the control, then don’t enable it.
b) Individual users could allow all senders through their individual whitelist.
c) The “Mail accepted only from preapproved senders.” bounce message could include a URL to a company web form which can be used by nonapproved senders to submit email messages. (This is my preferred solution.)
2) The whitelist won’t work, because all a spammer needs is to guess a preapproved address and forge the headers and/or envelope.
True. And this is a problem for which I do not have a complete solution. Clearly, there needs to be a tighter way of validating senders. (Perhaps the whitelist should have rules that require the sending MTA’s hostname to belong to the same domain as the sender. Or, maybe map sender domain to MTA host domain.) However, even if it is not perfect, forcing spammers to determine a preapproved sender address for each spamming destination should go a long way to eliminating the vast majority of spam.
3) If you block spammers through SMTP access, it wouldn’t be but a few days before someone starts spamming through your web form.
Using a simple web form is definitely not the solution. What is needed is a java applet to build and process the web form. The applet must use encryption and unique keys for each connection, and accept data only from the keyboard. It should not be too difficult to build a standard secure web mail interface that unapproved users can use to submit mail, but whose interface cannot be programatically spammed. I would think this is something that could become a part of the standard sendmail distribution.
4) You expect my stupid users who can barely use email to be able to edit some file that contains their whitelist?
Well, no... manually editing a file is clearly not a solution for the average user. Obviously some sort of front end program is needed if you want users to be able to maintain their whitelists.
5)... 10,000 other objections...
Okay. That concludes my basic thoughts. Perfect solution? Probably not. Have I thought of all the problems? Definitely not. Intelligent or workable solution? I wish I knew. Regardless of whether this is the right solution, or the wrong solution to the spam problem, clearly our current approach is broken.
The bottom line: It is time that we start discussing alternative approaches to spam control. I have now submitted by two-cents worth of ideas. I will now shut up and hope the discussion begins.
--
Jon R. Kibler
Systems Architect
Advanced Systems Engineering Technology, Inc.
Charleston, SC USA
Neil W Rickert
>I believe that the reason we are failing to block spam is that we are try=
>ing to fix a broken, un-fixable, email architecture. The only way we will=
> ever be able to control spam is to radically rethink the very premises u=
>pon which we have built our email systems. Re-architecting our email syst=
>ems is exactly what I would like to propose. In my humble opinion, it is =
>the only way we will ever solve our ever-growing spam problem.
Hey, don't preach about rearchitecting. Just re-architect it
yourself. Write the RFCs and some preliminary software to
demonstrate its feasability. They start persuading people to use
it.
>To fix the spam problem, we really must change this basic email architect=
>ure. That is, change the basic email architecture from =93I will accept e=
>mail from anyone (except those on my blacklist)=94 to =93I will accept em=
>ail only from pre-approved sources.=94 We have already demonstrated that =
>it is impossible to maintain an effective blacklist, so it is time to try=
> something radically different: Filter email through a =93whitelist.=94
That would not work for me. And probably not for most email users
either.
Claus Aßmann
[See the original post, basically a white-list approach]
1. I run this on the host which I use for the posting address,
it's basically adding a default case to check_rcpt
(using delay_checks):
R$* $#error $@ 4.7.0 $: "451 not on my white list."
I have some more hacks that allow me to check the mail itself without
accepting it, and to specify a "score" for addresses (sender X can
send me mail but only if it's not HTML etc).
BTW: almost all direct spam is rejected by this:
the spammer tries only once...
spam via open relays get caught by other means
(most of the time).
2. Look at
<a href="http://software.libertine.org/tmda/">
Tagged Message Delivery Agent (TMDA) Homepage</a>
--
If you feel the urgent wish to send me a courtesy copy of a Usenet
posting, then make sure it's recognizable as such!
The FAQ: http://www.sendmail.org/faq/ Before you ask.
bol...@hotmail.com
>f delivering messages: snail mail, as we call it today. It is the system =
>that every country in the world uses to deliver their mail. It is a simpl=
>e system -- you can send mail to anyone whose address you can obtain. Jus=
>t like email today.
Which planet are you on that has no snail mail spam problem? Your
initial premise is demonstrably broken.
>If snail mail and email both use the same architecture, then why do we no=
>t have a snail mail spam problem? If you think about it, the answer is si=
>mple: It costs to send snail mail. It doesn=92t cost to send email (relat=
>ively speaking).
>
>Well, is it time to start charging for sending email? Obviously, this is =
>neither desirable nor doable. We have to look elsewhere for the spam solu=
>tion.
Whitelisting is not the only way to go. It is not really scalable,
will need bucket loads of different applications to support it for
users, mainly because there are bucket loads of different MTA's, and
it does not work for the 'send a mail to in...@blah.blah' scenario. Try
telling your marketing department that they are no longer allowed to
receive enquiries from anywhere, and they have to first provide you
with the source addresses. The very next thing they do will be to
purchase the "get 100 million addresses for $150" CD and give it to
you to add to the whitelist!
PS. Adding your real e-mail address to a usenet posting is a real,
real cool way to attract lots of spam!
B