Re: Greylisting Experiences, Customer Support Perspective
Robert L Mathews
>Why do we believe that spammers will not try twice? If greylisting becomes
>sufficiently pervasive they will, indeed it is much cheaper for them to
>try twice, since the mail queue is just a small number of messages and
>a large recipient list, so with the right tools, the retry cost can be
>very low...
Actually, the retry cost can be quite high: failed deliveries naturally
take more time than successful ones because of unreachable destinations,
etc., and are likely more expensive to retry than just trying new
addresses.
Even if spammers were able to track exactly which deliveries had been
rejected due to greylisting, and adapted their systems to retry only
those, you can still make the retry cost too high by adding a small
tarpit delay on the RCPT TO response for any address checked during
greylisting (even if the message would be allowed because the triplet has
been seen before).
Imagine you're a spammer and 60% of your deliveries succeed, 20% fail
with a 5XX code, and 20% fail with a 4XX code. We'll give the spammer the
best of all possible worlds and assume that failed deliveries take no
longer than successes: say, 2 seconds on average. We'll also assume that
the spammer is lucky and all of those 4XX responses were due to
greylisting (and not due to mailbox full, etc.).
Also imagine you're very clever and you only wait for two seconds before
a timeout on the first delivery attempt, but you're willing to wait five
seconds on the second delivery attempt because you know that will defeat
a greylisting-plus-five-second-tarpit system the recipients are using.
The first run through 1000 addresses takes you 2000 seconds for 600
successes. You then have 200 addresses to retry, which takes another 1000
seconds due to the five-second tarpit. You've spent 3000 seconds
delivering to 800 addresses.
But if you instead tried 1500 addresses once each, you could have
delivered to 900 addresses in that same 3000 seconds.
We can make it worse for the spammer: if the tarpit was ten seconds
instead of five, it takes 4000 seconds to deliver to 800 addresses. If
the spammer had sent to new addresses instead, he could have had 1200
successful deliveries -- a 50% better delivery rate by not retrying.
Given that, it doesn't make sense to retry against a
greylisting-plus-tarpit scheme, even if the spammer encounters no other
problems at all in the retry attempts.
And there are ways to intentionally introduce other problems for spammers
who retry 4XX codes, such as making wpoison type addresses always return
4XX errors with tarpit delays instead of 5XX errors. (The extra
processing power you gain back from not having to deliver spam when using
greylisting frees up plenty of CPU time and connections to do this kind
of stuff if you're bored.)
So even if spammers start retrying 4XX failures (which they have not
shown the slightest sign of doing yet), there are ways to adjust the
larger concept of greylisting to make it much more expensive than just
trying new addresses, at least until almost everyone on the Internet uses
greylisting... and we all know spammers will never choose "expensive" if
they can avoid it.
>Bottom line, I don't think greylisting is a good idea, but I admit that
>it likely works moderately well at this point in time.
This makes me think perhaps you haven't actually tried it, because you
probably wouldn't use the word "moderately" if you had. :-) It actually
works *fantastically* well in terms of stopping spam.
More than 90% of our spam simply disappeared when we turned it on. It's
blocked literally millions of pieces of spam for us over the last few
months. Although it has issues (you can't avoid them with any system that
intentionally interferes with mail delivery), those issues are far less
of a concern than have been suggested -- much less than with, say,
blacklisting based on RBLs, which is routinely accepted.
And we've found that what issues exist can be almost completely
eliminated by using only selective greylisting. For example, only
greylisting clients with no reverse DNS, or with reverse DNS that appears
to be dynamic, or which are listed on bl.spamcop.net -- mail from servers
like "russian-caravan.cloud9.net", "pivsbh2.ms.com", and commonly
whitelisted servers is automatically whitelisted with no effort or
configuration needed on our part under this scheme, and it still catches
almost all the spam.
In fact, the issues are almost nonexistent. We block about a million
messages a month with greylisting, and it's been at least six weeks since
I've touched it or looked at anything beyond the daily summaries. No
complaints, no problems, no whitelist adjustments needed, no manual
tweaks of any kind.... nothing. I wish I could say that about our other
spam filtering systems.
On balance the effects are *overwhelmingly* positive.
--
Robert L Mathews, Tiger Technologies http://www.tigertech.net/
"Ignorance more frequently begets confidence than does knowledge."
-- Darwin
Peter Kiem
> Popfile + spamassassin combo working with 99.94% efficiency.
Yes I too have been running a (mostly) spam free Postfix installation but
my point is with Greylisting I can turn OFF most of my anti-spam controls
that I was using before.
It is a technique that kills a LOT of the spam BEFORE it his the filters
and other Postfix rules and has the benefit of far less false positives
than any other method I have seen. Unless the sending mail server is
seriously borked the mail WILL get through so it is safe :)
Nowadays I use greylisting and Maia Mailguard and VERY satisfied with that
solution.
> IMHO greylisting is a temporary solution, and spammers will evolve to work
> around it.
I agree that it will probably lose some effectiveness over time as the
spammers adapt to it but I think it is a great method for now with real
tangible benefits right NOW
> A well written bayesian filter will evolve with the spam, and keep the high
> efficiency, as ours did for over a year now. The amount of spam has been
> steadily increasing for ages, but POPFile was able to keep up with it, and it
> keeps improving with time.
The idea is to use greylisting to kill 90%+ of the spam BEFORE it hits
your bayesian filters thereby saving bandwidth and server resources.
> A good solution does not have to hurt anybody (...but the spammers :-).
And I believe greylisting does exactly that.
> Ps: No, I'm not affiliated with PF, I'm just a satisfied user.
Go Maia Mailguard! Hehehe
--
Regards,
+-----------------------------+---------------------------------+
| Peter Kiem .^. | E-Mail : <zor...@zordah.net> |
| Zordah IT /V\ | Mobile : +61 0414 724 766 |
| IT Consultancy & /( )\ | WWW : www.zordah.net |
| Internet Services ^^-^^ | ICQ : "Zordah" 866661 |
+-----------------------------+---------------------------------+
Cami
>> server ONCE, total, or maybe once per six months. The server caches
>> the names of servers that have "passed" and doesn't delay delivery a
>> second time.
>>
>> Is that not correct?
>
> This is correct, but how much legitimate mail represents new triples?
Cant answer this one yet, but i will be implementing greylisting
soon so we're gonna see how well it works on a large scale..
Our company has just bought over a very large ISP in ZA which
will push our mailboxes close to the 1/2 million mark.. Postfix
ofcourse is steady as she goes ;)
> What happens when an ISP adds a new outbound MTA?
Depends on the implementation of Greylisting you use.
> What happens to VERP mail
> where the bounce address contains a message specific extension?
Very true.. Whitelisting is your friend..
> Why do we believe that spammers will not try twice?
There is plenty evidence/proof to support this.
Currently, they do not try twice.
> If greylisting becomes
> sufficiently pervasive they will, indeed it is much cheaper for them to
> try twice, since the mail queue is just a small number of messages and
> a large recipient list, so with the right tools, the retry cost can be
> very low...
Indeed.. Just like certain spammers are adding 200k attachments to
their spam so SpamAssassin doesnt scan the message. Bandwidth is
becoming so cheap these days..
> Bottom line, I don't think greylisting is a good idea, but I admit that
> it likely works moderately well at this point in time.
Agreed.. While more and more people are implementing greylisting
solutions, they only do so untill 'the next best thing' comes
along (hopefully)..
Cami
Cami
> corruption, I have been using the same greylist database without
> ever deleting old entries. That's since October last year.
>
> I admit that greylist could be a little smarter about things, and
> whitelist any MTA that properly retries delivery.
Now THAT's a good idea, and if done properly it would be quite
effective.. *adds to TODO list*
Cami
Cami
> concerned about it affecting business operations in the following ways:
>
> 1) sales: a prospective client runs greylisting and your sales rep is
> having a conversation with the decision-maker. The decision-maker says
> 'sure, send me some information (the contract, etc.) and we'll talk' -
> instead of that coming through instantly your sales rep will have to
> follow-up later which could cause loss of business
> 2) operations: an operational unit in the company is having critical
> problems with a vendor that you haven't already whitelisted or that they
> don't normally (or haven't ever) email(ed). The solution is provided by
> the vendor via email and the problem is prolonged
> 3) customer service: a) you run a public/retail service and a client
> who can have any email address is having a problem that needs to fixed
> right now. He's hosed waiting for MTA retries, or he has a weak ISP who
> doesn't do retries. Also your b) people pay your company for email
> support and clients get less service for their dollar because email is
> delayed.
> 4) misc: you have new/existing vendors with business (ERP, CRM, etc.)
> systems which don't do retries. Those tickets are all lost until a
> whitelist.
> 5) IT: a) one of the above happens and the person responsible for the
> whitelist gets hit by a bus. b) It's also bad PR for the IT department
> when people are used to their once-instantaneous email system which is
> now time-delayed. Finally, c) at some point in the future everybody's
> MTAs all work perfectly including spammer's MTAs. Time is lost and time
> = money.
>
> I can think of a lot more business reasons which make it not worthy.
Good points. Yet as its stated before, peoples implementation of
greylisting stinks.. For example, we split up all corperate mail
from dialup mail.. Secondly, each user can opt-in and opt-out of
greylisting so if they do not want, they dont have to have it.
Cami
Viktor Fougstedt
On 20 sep 2004, at 22:48, Covington, Chris wrote:
> Well I think the 'technique' leaves something to be desired. I'm
> concerned about it affecting business operations in the following ways:
I am also concerned about the false positives (apart from the workload,
mostly because I'm not fond of the idea of having to manually detect
legitimate but misbehaving MTA's and whitelist them).
My idea is to start greylisting only after having received X messages
from the same IP but with different sender address domains within some
period of time.
As long as a particular client IP uses the same sender domain, there's
no greylisting done. So ticket agencies, Amazon, and similar places
will never be greylisted. However, most stupid spammer-MTA:s use unique
sender addresses for each mail, and will quickly pass the limit and
start being greylisted.
I am also considering not greylisting on the (sender, recipient, ip)
tuple, but instead on (recipient, ip, helo-name). It would seem to be
effective (according to my logs, but further study and afterthough will
be needed), and would automatically avoid the "unique sender
address"-problem.
I am additionally considering doing greylisting _only_ for IP:s which
do not resolve to any hostname, or where the hostname contains certain
patterns, e.g. "dsl", "[0-9]-[0-9]", "dialup" etc.
There will still be false positives, but these techniques should
minimize them.
/Viktor...
--
Of course I'm being paranoid, everybody's trying to kill me!
Andras Kristof
Well put answers in your email, can't argue much. But let me comment on the
last two items:
> > A well written bayesian filter will evolve with the spam, and keep the
> > high efficiency, as ours did for over a year now. The amount of spam has
> > been steadily increasing for ages, but POPFile was able to keep up with
> > it, and it keeps improving with time.
>
> The idea is to use greylisting to kill 90%+ of the spam BEFORE it hits
> your bayesian filters thereby saving bandwidth and server resources.
Yes, this is one of the reasons I don't like greylisting: it saves your
resources by potentionally wasting everyone else's. Because of the retries,
more bandwidth will be used, and mail servers will have to work more.
I prefer a solution --if one exists-- which does not potentionally harm
others :-)
>
> > A good solution does not have to hurt anybody (...but the spammers :-).
>
> And I believe greylisting does exactly that.
Please refer to the previous point.
One more thought:
It is being said in this thread that even if spammers adapt to re-try the
sends, it will increase their operational costs, potentially driving them out
of business.
This is true.
But then all other legitimate businesses sending out large amount of emails
will have to increase their operational costs as well. The damage should not
be that big as in the case of spammers, but it will be there, and adding
together the damages could amount to a nice sum.
I am probably sensitive to this, because our company also sends out quite an
amount of emails, so in some way it affects us, too.
The other thing, that the spammers may decide to make up the loss of money by
increasing the value of junk they send out...
Third thing, I can't accept that a solution which make servers send out *more*
email is the way to fight spam...
All in all, I agree that greylisting can be an effective tactic in the
desperate war against spammers, but I just can't shrug off that uneasy
feeling that it's not the right thing to do. But that's just me... :-)
Andras
Robin Lynn Frank
Content-Type: text/plain; charset=US-ASCII
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
On Mon, 20 Sep 2004 15:08:48 -0500
Dan D Niles <d...@more.net> wrote:
> Gld also has two features I believe are not in the original greylist
> discussion. Lightgrey masks the last octet of the ip, so in the
> example above the ip entry would be 12.34.56. The other is
> Lightgreydomain, which only uses domains for the sender and recipient.
I've been testing lightgrey here, and while it appears to be both
effective and to eliminate the problem of whitelisting multiple
server/IPs, it totally breaks something else we are doing which is take
those one-shot IPs and export them to a permanent blacklist.
I think lightgreydomain would significantly reduce the effectiveness of
greylisting.
--
Robin Lynn Frank
Director of Operations
Paradigm-Omega, LLC
http://www.paradigm-omega.com
==============================
Sed quis custodiet ipsos custodes?
--Signature=_Tue__21_Sep_2004_09_09_13_-0700_w9X9ix/6IJZWESBk
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
iD8DBQFBUFIvo0pgX8xyW4YRA6EhAKDdaaC+3reAbvFdsgttQx7fOjjbrwCgkACB
1lyM/xQcPMOULqLYuI72pjQ=
=JdQ4
-----END PGP SIGNATURE-----
--Signature=_Tue__21_Sep_2004_09_09_13_-0700_w9X9ix/6IJZWESBk--
Camis
>> discussion. Lightgrey masks the last octet of the ip, so in the
>> example above the ip entry would be 12.34.56. The other is
>> Lightgreydomain, which only uses domains for the sender and recipient.
>
>
> I've been testing lightgrey here, and while it appears to be both
> effective and to eliminate the problem of whitelisting multiple
> server/IPs, it totally breaks something else we are doing which is take
> those one-shot IPs and export them to a permanent blacklist.
>
> I think lightgreydomain would significantly reduce the effectiveness of
> greylisting.
In my implementation of greylisting, this is the default.
However, when rejection logging happens, the connecting
host's ip is sent to syslog so the type of functionality
you're requiring would be really simple to get out of a
bit of data mining.. I have no idea what information Gld
sends to syslog..
Cami
Robin Lynn Frank
Content-Type: text/plain; charset=US-ASCII
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
On Tue, 21 Sep 2004 19:17:29 +0200
Camis <ca...@mweb.co.za> wrote:
> > I've been testing lightgrey here, and while it appears to be both
> > effective and to eliminate the problem of whitelisting multiple
> > server/IPs, it totally breaks something else we are doing which is
> > take those one-shot IPs and export them to a permanent blacklist.
> >
> > I think lightgreydomain would significantly reduce the effectiveness
> > of greylisting.
>
> In my implementation of greylisting, this is the default.
> However, when rejection logging happens, the connecting
> host's ip is sent to syslog so the type of functionality
> you're requiring would be really simple to get out of a
> bit of data mining.. I have no idea what information Gld
> sends to syslog..
It is simpler to pull the data out of one MySQL table and put it in
another.
--
Robin Lynn Frank
Director of Operations
Paradigm-Omega, LLC
http://www.paradigm-omega.com
==============================
Sed quis custodiet ipsos custodes?
--Signature=_Tue__21_Sep_2004_11_09_15_-0700_PLKhKB4S8jpvmblr
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
iD8DBQFBUG5do0pgX8xyW4YRAwVfAKCF1XDOm7kwA0FkdBMc4eFq19kzOwCg0UDt
rm42tj9qJT7LarOCOBYy78Q=
=x6tg
-----END PGP SIGNATURE-----
--Signature=_Tue__21_Sep_2004_11_09_15_-0700_PLKhKB4S8jpvmblr--
Csillag Tamas
Search the archives.
It's Wietse's code with a bit improovement.
You can get it here: http://digitus.itk.ppke.hu/~cstamas/greylist.pl
It's not the best but it work here (almost) perfectly.
Any suggestion, comment Is Welcomed.
--
cstamas
Cami
>>> effective and to eliminate the problem of whitelisting multiple
>>> server/IPs, it totally breaks something else we are doing which is
>>> take those one-shot IPs and export them to a permanent blacklist.
>>>
>>> I think lightgreydomain would significantly reduce the effectiveness
>>> of greylisting.
>>
>> In my implementation of greylisting, this is the default.
>> However, when rejection logging happens, the connecting
>> host's ip is sent to syslog so the type of functionality
>> you're requiring would be really simple to get out of a
>> bit of data mining.. I have no idea what information Gld
>> sends to syslog..
>
> It is simpler to pull the data out of one MySQL table and put it in
> another.
Valid point. Any suggestions/preposals on how that type
of functionality can stay in place yet also cater for
what you are after?
Cami
Len Conrad
>Yes, this is one of the reasons I don't like greylisting: it saves your
>resources by potentionally wasting everyone else's.
BS. if a legit MTA doesn't have the resources to do a single, one-time
retry to get past greylist rejection, then it's got huge problems which I
really don't carabout
>Because of the retries, more bandwidth will be used
how many seconds and bytes consumed by a 450 reject?
>and mail servers will have to work more.
that's what they are there for, working. the immeasurably, undetectably
small increment of greylisting is simply not a serious concern.
>I prefer a solution --if one exists-- which does not potentionally harm
>others :-)
Greylisting harms no one. and such "golden rule/good will" sentiments in
the disastrous world of SMTP are worthless.
Len
_____________________________________________________________________
http://IMGate.MEIway.com : free anti-spam gateway, runs on 1000's of sites
lst_...@kwsoft.de
Integration in postgrey ???
If my perl wasn't that bad i could do it myself :-(
It's time to buy some good perl book and get a silent corner to read.
Regards
Andreas
Csillag Tamas
On 09/20, Victor Duchovni wrote:
...
> What happens when an ISP adds a new outbound MTA?
You can demand the sender to send from the same /24 block only you do not
need to require to be a single host (that is still very efficient).
a.b.c/sender/recipt can be stored instead of a.b.c.d/sender/recipt
> What happens to VERP mail where the bounce address contains a message
> specific extension?
You can replace numbers with # -s
Look: sender=sentto-11448959-397-1095789631-user=host....@returns.groups.yahoo.com
will be: sentto-#-#-#-user=host....@returns.groups.yahoo.com
>
> Why do we believe that spammers will not try twice?
It is only experience.
> If greylisting becomes
> sufficiently pervasive they will, indeed it is much cheaper for them to
> try twice, since the mail queue is just a small number of messages and
> a large recipient list, so with the right tools, the retry cost can be
> very low...
That's true, but right now they are not re-trying.
It they can deliver (let's say) 60% of their spam at the first try they
won't mess with the remaining 40%
Greylisting is only for identifying _not real MTAs_.
It's worth to implement a way to remember hosts that prooved that they are
real MTAs and whitelist them (in the greylist daemon itself) for a
month.
--
cstamas
Csillag Tamas
> > It's Wietse's code with a bit improovement.
> > You can get it here: http://digitus.itk.ppke.hu/~cstamas/greylist.pl
> > It's not the best but it work here (almost) perfectly.
> >
> > Any suggestion, comment Is Welcomed.
>
> Integration in postgrey ???
> If my perl wasn't that bad i could do it myself :-(
> It's time to buy some good perl book and get a silent corner to read.
postgrey itself.
Hmm, maybe that can be the next step. (Note: I am not a perl guru either)
It's multi-threaded modell is nice, but it is much more complex.
(I did not look into postgrey internals (enought) yet)
--
cstamas
Robin Lynn Frank
Content-Type: text/plain; charset=US-ASCII
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
I've been thinking of looking at the space the domain owns, and if they
have a /24, whitelisting it all. But that will be on a case by case
basis.
--
Robin Lynn Frank
Director of Operations
Paradigm-Omega, LLC
http://www.paradigm-omega.com
==============================
Sed quis custodiet ipsos custodes?
--Signature=_Tue__21_Sep_2004_16_26_50_-0700_k7JV49VC6gA5BN8.
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
iD8DBQFBULjBo0pgX8xyW4YRA51dAJ4krh5B3jyDx6vMRkAsEtQtwS3+XwCbBTDY
oZe1rdIhyYb3h1dyWY9ElIM=
=7IW9
-----END PGP SIGNATURE-----
--Signature=_Tue__21_Sep_2004_16_26_50_-0700_k7JV49VC6gA5BN8.--
lst_...@kwsoft.de
> On 09/21, lst_...@kwsoft.de wrote:
> > Zitat von Csillag Tamas <cst...@digitus.itk.ppke.hu>:
> ...
> > > It's Wietse's code with a bit improovement.
> > > You can get it here: http://digitus.itk.ppke.hu/~cstamas/greylist.p=
l
> > > It's not the best but it work here (almost) perfectly.
> > >
> > > Any suggestion, comment Is Welcomed.
> >
> > Integration in postgrey ???
> > If my perl wasn't that bad i could do it myself :-(
> > It's time to buy some good perl book and get a silent corner to read.
> In fact (if I remember correctly) the verp replace sub come from
> postgrey itself.
>
> Hmm, maybe that can be the next step. (Note: I am not a perl guru eithe=
r)
> It's multi-threaded modell is nice, but it is much more complex.
> (I did not look into postgrey internals (enought) yet)
It wasn't that hard :-)
I spent some spare time yesterday and made a patch against postgrey-1.16 =
and
sent it to the maintainer. We will see if he integrate it in postgrey 1.1=
7.
If you are interested in testing it (works fine on my machine :-) i can s=
end you
the patch.
Regards
Andreas
lst_...@kwsoft.de
>
> For a site like mine that is constantly being assaulted with junk
> mail, it helps. It is not nearly as problematic with legitimate
> mail as sender address verification.
>
> Ever since I fixed greylist.pl to stop crashing due to db file
> corruption, I have been using the same greylist database without
> ever deleting old entries. That's since October last year.
>
> I admit that greylist could be a little smarter about things, and
> whitelist any MTA that properly retries delivery.
Interested in testing postgrey with automatic whitelist of sender-hosts a=
lready
delivered valid mail?
Because i found this really useful i have modified postgrey to do this. I=
have
submitted the patch to the maintainer but he seams to be on vacation unti=
l next
week.
Regards
Andreas
Dan D Niles
lst_...@kwsoft.de writes:
> >
> > I admit that greylist could be a little smarter about things, and
> > whitelist any MTA that properly retries delivery.
>
> Interested in testing postgrey with automatic whitelist of
> really useful i have modified postgrey to do this. I have submitted
> the patch to the maintainer but he seams to be on vacation until
> next week.
I do not believe automatic whitelisting is a good idea.
1) If a spammer accidently gets a message through to 1 user, such as
might happen if an email addr was on their list twice, they would
be whitelisted and suddenly everyone would start getting their
spam. I have seen instances where this would be the case.
2) The whitelist would get very large becuase of adding individual IPs
instead of ranges. This could lead to permormance problems. Since
whitelisting is the first check on every email, it should be as
fast as possible.
3) It gives spammers a reason to retry. If they retry just 1 address
to each destination, they will be whitelisted and all their spam
will get through. It make the economics of retrying beneficial to
the spammers. Spammers might eventually start retrying anyway, but
this would make it happen for certain, and happen very quickly.
I think adding known good hosts or ranges to the whitelist is a good
idea. I think it needs to be done with some discression though.
Dan
lst_...@kwsoft.de
>
> lst_...@kwsoft.de writes:
> > >
> > > I admit that greylist could be a little smarter about things, and
> > > whitelist any MTA that properly retries delivery.
> >
> > Interested in testing postgrey with automatic whitelist of
> > sender-hosts already delivered valid mail? Because i found this
> > really useful i have modified postgrey to do this. I have submitted
> > the patch to the maintainer but he seams to be on vacation until
> > next week.
>
> I do not believe automatic whitelisting is a good idea.
>
> 1) If a spammer accidently gets a message through to 1 user, such as
> might happen if an email addr was on their list twice, they would
> be whitelisted and suddenly everyone would start getting their
> spam. I have seen instances where this would be the case.
Only from the one IP which get the message through. But if the spammer do=
n't
shuffle their sending IP all the time, they get caught by the RBLs. Greyl=
isting
doesn't work at all (and shouldn't of course) against systems doing
mail-queueing.
> 2) The whitelist would get very large becuase of adding individual IPs
> instead of ranges. This could lead to permormance problems. Since
> whitelisting is the first check on every email, it should be as
> fast as possible.
I have not tested this yet but someone reported on the postfix-list that =
even
100 of thousends of records can be searched with no real performance impa=
ct.
The list of sending hosts will even be smaller then the conventional list=
.
> 3) It gives spammers a reason to retry. If they retry just 1 address
> to each destination, they will be whitelisted and all their spam
> will get through. It make the economics of retrying beneficial to
> the spammers. Spammers might eventually start retrying anyway, but
> this would make it happen for certain, and happen very quickly.
Spammers already retry. I always see 4-5 attempts to the same destination
address from different IPs, sometimes with the same sender address someti=
mes
not. And as said only one IP get whitelisted so the spammer have to press=
all
the spam to you through this host. If a host is able to do mailqueueing i=
t is a
job for RBLs not for greylisting. Greylisting is only useful against movi=
ng
targets which try to avoid RBLs, but for this it is really useful.
> I think adding known good hosts or ranges to the whitelist is a good
> idea. I think it needs to be done with some discression though.
We have customers for projectbased work. So often a group of customer add=
resses
have to communicate with a group of internal addresses. Whitelist all of =
the
company mailservers by hand isn't really an option.
I prefer the other way around : If i detect a spammer not listed on any o=
f our
RBLs which does mail-queueing he gets in the personal blocklist.
But as always YMMV
Regards
Andreas
Cami
> lst_...@kwsoft.de writes:
> > >
> > > I admit that greylist could be a little smarter about things, and
> > > whitelist any MTA that properly retries delivery.
> >
> > Interested in testing postgrey with automatic whitelist of
> > sender-hosts already delivered valid mail? Because i found this
> > really useful i have modified postgrey to do this. I have submitted
> > the patch to the maintainer but he seams to be on vacation until
> > next week.
>
> I do not believe automatic whitelisting is a good idea.
>
> 1) If a spammer accidently gets a message through to 1 user, such as
> might happen if an email addr was on their list twice, they would
> be whitelisted and suddenly everyone would start getting their
> spam. I have seen instances where this would be the case.
>
> instead of ranges. This could lead to permormance problems. Since
> whitelisting is the first check on every email, it should be as
> fast as possible.
>
> to each destination, they will be whitelisted and all their spam
> will get through. It make the economics of retrying beneficial to
> the spammers. Spammers might eventually start retrying anyway, but
> this would make it happen for certain, and happen very quickly.
>
> idea. I think it needs to be done with some discression though.
Indeed.. My approach will be to do some daily datamining of the
MySQL data.. whitelisting aswell as blacklisting automatically
happens on a nightly basis, but there are thresholds in place
for certain categories so that you dont whitelist everyone on the planet..
Eg:
Whitelist:
If i find more than 1000 verified triplets for hosts spread across
an entire C-class (or just from 1 host), then whitelist that host..
Whitelisted ranges get dropped if there has been no verified triplets
in the last 7 days..
Blacklist:
If i find more than 1000 unverified triplets host hosts spread across
en entire C-class (or just from 1 host), then blacklist that host.
Blacklisted ranges automatically get dropped from MySQL if there has
been no unverified triplets in the last 7 days.
Thats the rough idea..
Cami
Len Conrad
>1) If a spammer accidently gets a message through to 1 user, such as
> might happen if an email addr was on their list twice, they would
> be whitelisted and suddenly everyone would start getting their
> spam.
wrong. postgrey does a DUNNO, not a whitelisting, and DUNNOs only the
postgrey function, while all other restrictions remains effective.
>2) The whitelist would get very large becuase of adding individual IPs
> instead of ranges. This could lead to permormance problems.
but it doesn't. and postgrey purges triplets unseen for 30, or whatever, days.
> Since
> whitelisting is the first check on every email, it should be as
> fast as possible.
no, the first check is whether the recipient is ours.
postgrey has no apparent delay accessing dbm file.
>3) It gives spammers a reason to retry.
> If they retry just 1 address
> to each destination, they will be whitelisted and all their spam
> will get through.
total BS, from postgrey POV. see above.
> It make the economics of retrying beneficial to
> the spammers.
100's of 1000's of infected PCs on unpoliced subscriber access networks do
not retry when attacking direct to MXs and are effectively blocked by
postgrey. There are some spam farms that do not retry.
> Spammers might eventually start retrying anyway, but
> this would make it happen for certain, and happen very quickly.
it hasn't happened so far, and "so far, so very excellent" with every MX
where I've installed postgrey. an overwhelming, unquestionable, continuing
success.
>I think adding known good hosts or ranges to the whitelist is a good
>idea.
I don't know what greylist implementation you are talking about, but it
really sucks. try postgrey.
Victor Duchovni
> Indeed.. My approach will be to do some daily datamining of the
> MySQL data.. whitelisting aswell as blacklisting automatically
> happens on a nightly basis, but there are thresholds in place
> for certain categories so that you dont whitelist everyone on the planet..
>
> Eg:
>
> Whitelist:
> If i find more than 1000 verified triplets for hosts spread across
> an entire C-class (or just from 1 host), then whitelist that host..
> Whitelisted ranges get dropped if there has been no verified triplets
> in the last 7 days..
>
> Blacklist:
> If i find more than 1000 unverified triplets host hosts spread across
> en entire C-class (or just from 1 host), then blacklist that host.
> Blacklisted ranges automatically get dropped from MySQL if there has
> been no unverified triplets in the last 7 days.
>
Please if possiblel share the code when you are done...
--
Viktor.
Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.
To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majo...@postfix.org?body=unsubscribe%20postfix-users>
Cami
>> MySQL data.. whitelisting aswell as blacklisting automatically
>> happens on a nightly basis, but there are thresholds in place
>> for certain categories so that you dont whitelist everyone on the planet..
>>
>> Eg:
>>
>> Whitelist:
>> If i find more than 1000 verified triplets for hosts spread across
>> an entire C-class (or just from 1 host), then whitelist that host..
>> Whitelisted ranges get dropped if there has been no verified triplets
>> in the last 7 days..
>>
>> Blacklist:
>> If i find more than 1000 unverified triplets host hosts spread across
>> en entire C-class (or just from 1 host), then blacklist that host.
>> Blacklisted ranges automatically get dropped from MySQL if there has
>> been no unverified triplets in the last 7 days.
>
I'm busy working on this code and realized something.. Once a c-class
has been automatically whitelisted or blacklisted, the hosts inside
that c-class wont be able to generate anymore triplets as the Policyd
will reject/accept there mails explicitly. I could make it automatically
expire whitelisted/blacklisted c-classes every 7 days from the start of
their white/black listing..
i guess that even if a host is whitelisted or blacklisted, i could
still insert the triplet information into the database .. Question is,
like in the case of whitelisting, would the point of whitelisting them
in the first place to be to cut down the amount of triplets in the
database (this increasing the lookup performance somewhat) or solely
to prevent those whitelisted hosts from getting any rejections at all?
Comments / Ideas?
Cami
Victor Duchovni
> I'm busy working on this code and realized something.. Once a c-class
> has been automatically whitelisted or blacklisted, the hosts inside
> that c-class wont be able to generate anymore triplets as the Policyd
> will reject/accept there mails explicitly. I could make it automatically
> expire whitelisted/blacklisted c-classes every 7 days from the start of
> their white/black listing..
>
Continue to greylist all mail to postmaster and any spamtrap addresses...
That should provide some ongoing traffic, if no evidence emerges from
always greylisted recipients, instead of dropping the whitelist/blacklist
entry entirely, greylist a small number of randomly selected messages and
keep or drop the entry based on the results, if an address is repeatedly
placed in the same category extend the time between "tests" from 1-week
to 1-month.
Dan D Niles
Cami writes:
>
> Indeed.. My approach will be to do some daily datamining of the
> MySQL data.. whitelisting aswell as blacklisting automatically
> happens on a nightly basis, but there are thresholds in place
> for certain categories so that you dont whitelist everyone on the planet..
>
> Eg:
>
> Whitelist:
> If i find more than 1000 verified triplets for hosts spread across
> an entire C-class (or just from 1 host), then whitelist that host..
> Whitelisted ranges get dropped if there has been no verified triplets
> in the last 7 days..
>
> Blacklist:
> If i find more than 1000 unverified triplets host hosts spread across
> en entire C-class (or just from 1 host), then blacklist that host.
> Blacklisted ranges automatically get dropped from MySQL if there has
> been no unverified triplets in the last 7 days.
>
>
> Cami
I've been thinking about this. You could end up with the same entry
on the blacklist and the whitelist. It also does not scale if you use
a hard number such as 1000 verifed or unverified entries.
How about using a ratio of verified to unverified? If you have 1000
verified and 1000 unverified, do nothing. If you have say 100 times
verified than unverified, whitelist them. It would then scale since
100:1, 1000:10, 100000:1000 would all result in whitelisting while
1:100, 10:1000, 1000:100000 would all result in blacklisting. I'm not
sure what ratio would be most accurate. I plan on digging through our
greylist database (2,697,928 entries and growing) to see if a ratio
suggests itself.
New unverified entries should also be ignored, but you probably
already plan on doing that. You need to give servers at least an hour
to retry before subjecting them to blacklisting.
There is still the issue of what size block to add. It depends on the
implementation, but most implementations should be able to hanble
adding individual IPs (as someone mentioned). I have 156,375
individual IPs in my greylist at the moment. It would be nice to add
class C (or class B) blocks when appropriate. Perhaps if the ratio is
over a certain amount for all IPs in a class C (that are in the
greylist) add the class C to the whitelist or blacklist. It would be
nice if the program could use whois (or something else) to find the
correct size of block to add.
Dan
Dan D Niles
Cami writes:
>
> I'm busy working on this code and realized something.. Once a c-class
> has been automatically whitelisted or blacklisted, the hosts inside
> that c-class wont be able to generate anymore triplets as the Policyd
> will reject/accept there mails explicitly. I could make it automatically
> expire whitelisted/blacklisted c-classes every 7 days from the start of
> their white/black listing..
>
> still insert the triplet information into the database .. Question is,
> like in the case of whitelisting, would the point of whitelisting them
> in the first place to be to cut down the amount of triplets in the
> database (this increasing the lookup performance somewhat) or solely
> to prevent those whitelisted hosts from getting any rejections at all?
>
> Comments / Ideas?
You would not be able to put accurate triplet information into the
database. Whitelisted and blacklisted entries would never retry since
it was either accepted the first time or permanantly rejected the
first time. On the other hand, it would identify repeated triplets,
which most normal email would have. It would require modification of
the greylist software as opposed to just manipulating the database
with an external program.
It would probably work reasonably well to expire the whitelist and
blacklist entries after a period of time. The server would then have
the opportunity to reestablish itself on either the whitelist or the
blacklist. You would need to add a date field to the whitelist and
blacklist, and it would probably be better to use IPs instead of
blocks. You would need to differentiate between broken hosts that
need permanent whitelisting and automatic entries that need to be
expired.
I like the expiration method because it does not require any
modification of the greylist software itself. I think I'll write a perl
script to parse the gld mysql database and see what turns up.
Dan