DNS Whitelisting
Wietse Venema
[ Charset UTF-8 unsupported, converting... ]
> On 8/23/2010 8:48 AM, Ralf Hildebrandt wrote:
> > * p...@alt-ctrl-del.org<p...@alt-ctrl-del.org>:
> >> I find that a lot of spam comes from recently registered, throw away
> >> domains. The new domain may be used as the sender, hostname, or name
> >> server.
> >>
> >> Are there any rbl type lists that block fresh domains, for the first
> >> 10-15 days of their existence?
> >
> > http://www.mail-archive.com/us...@spamassassin.apache.org/msg57008.html
> > Dunno if Marc is still active
> >
>
> Yes, the "hostkarma" lists are active, IMO best used in SA
> because they mix whitelist with blacklist using different
> return codes. (Might be time to revisit DNS whitelists in
> postfix.)
Maybe someone can draft a strawman user interface:
- what is the configuration syntax
- what does that syntax mean
- how to make it safe ( we don't want "open relay" problems)
I'm currently doing this for postscreen, and won't have time for
other Postfix features.
Wietse
> http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists
>
>
>
> I've also been playing with these:
> http://spameatingmonkey.com/lists.html
> The FRESH lists are what you're looking for.
>
>
>
> -- Noel Jones
>
>
p...@alt-ctrl-del.org
>
> I've also been playing with these:
> http://spameatingmonkey.com/lists.html
> The FRESH lists are what you're looking for.
>
Very nice.
I'm now using their geobl.spameatingmonkey.net, right before I accept a
delivery. But not for blocking. Just for statistics at this point.
Ralf Hildebrandt
> >Dunno if Marc is still active
> >
>
> Yes, the "hostkarma" lists are active, IMO best used in SA because
> they mix whitelist with blacklist using different return codes.
reject_dnsbl_client hostkarma.junkemailfilter.com=127.0.0.6
should work for that particular purpose.
--
Ralf Hildebrandt
Geschäftsbereich IT | Abteilung Netzwerk
Charité - Universitätsmedizin Berlin
Campus Benjamin Franklin
Hindenburgdamm 30 | D-12203 Berlin
Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
ralf.hil...@charite.de | http://www.charite.de
Stan Hoeppner
> Noel Jones:
> (Might be time to revisit DNS whitelists in
>> postfix.)
>
> Maybe someone can draft a strawman user interface:
>
> - what is the configuration syntax
>
> - what does that syntax mean
>
> - how to make it safe ( we don't want "open relay" problems)
>
> I'm currently doing this for postscreen, and won't have time for
> other Postfix features.
accept_dnswl_client (default: 0)
0 - accept all messages
1 - accept messages with trust level 1-3
2 - accept messages with trust level 2-3
3 - accept messages with trust level 3
Trust level is a numerical 0-3 value returned as the last octet of the
127.0.x.x address, see below. The third octet is more informational
than actionable, and IMO would simply add unnecessary complexity if used
to determine deliverability. Decisions should be based solely on the
last octet data, the "trust" level.
I assume postscreen processes or passes this data to smtpd in a way that
smtpd will automatically bypass all checks normally performed during the
CONNECT phase.
Below is how dnswl.org does dns based whitelisting. Other dns whitelist
operators may use a different standard, or no standard--I've not
investigated the few others at this point. AFAIK dnswl.org is the most
popular, and was the original dns whitelist operator.
=======================================================================
How to query DNSWL
The query must always go to the zone "list.dnswl.org" in standard DNSBL
format, ie with a reversed dotted quad IP address. To query whether the
IP address "1.2.3.4" is listed, the query would thus be
The list contains the standard test entry of 127.0.0.2, which you can
also test manually
matthias:~ > host 2.0.0.127.list.dnswl.org
2.0.0.127.list.dnswl.org has address 127.0.10.0
Return codes
The return codes are structured as 127.0.x.y, with "x" indicating the
category of an entry and "y" indicating how trustworthy an entry has
been judged.
Categories (127.0.X.y):
* 2 - Financial services
* 3 - Email Service Providers
* 4 - Organisations (both for-profit [ie companies] and non-profit)
* 5 - Service/network providers
* 6 - Personal/private servers
* 7 - Travel/leisure industry
* 8 - Public sector/governments
* 9 - Media and Tech companies
* 10 - some special cases
* 11 - Education, academic
* 12 - Healthcare
* 13 - Manufacturing/Industrial
* 14 - Retail/Wholesale/Services
* 15 - Email Marketing Providers
Trustworthiness / Score (127.0.x.Y):
* 0 = none - only avoid outright blocking (eg Hotmail, Yahoo
mailservers, -0.1)
* 1 = low - reduce chance of false positives (-1.0)
* 2 = medium - make sure to avoid false positives but allow override
for clear cases (-10.0)
* 3 = high - avoid override (-100.0).
The scores in parantheses are typical SpamAssassin scores
--
Stan
Noel Jones
- This is specific for dnswl.org. Postfix needs a general
mechanism. Other whitelists are not required to follow
dnswl.org's 127.0.x.y mechanism.
- what do you mean by "accept the message"; OK? suppress
further rbl lookups?
- If "accept" means OK, how will you protect postfix from open
relay if the dns whitelist accidentally or intentionally lists
the whole internet?
- what would the user interface look like? Is it possible to
document it clearly?
-- Noel
Wietse Venema
> Wietse Venema put forth on 8/23/2010 10:11 AM:
> > Noel Jones:
>
> > (Might be time to revisit DNS whitelists in
> >> postfix.)
> >
> > Maybe someone can draft a strawman user interface:
> >
> > - what is the configuration syntax
> >
> > - what does that syntax mean
> >
> > - how to make it safe ( we don't want "open relay" problems)
> >
> > I'm currently doing this for postscreen, and won't have time for
> > other Postfix features.
>
> accept_dnswl_client (default: 0)
>
> 0 - accept all messages
> 1 - accept messages with trust level 1-3
> 2 - accept messages with trust level 2-3
> 3 - accept messages with trust level 3
This looks somewhat like RFC 5782, with reputation scores and
confidence values encoded in the lower octets as numbers in the
range 0-255.
With reject_rbl_client etc. Postfix can use different DNSXLs names
in different access lists, and filter the result. For example, to
select responses from some.example.com with value 127.0.0.4:
smtpd_mumble_restrictions =
...
reject_rbl_client some.example.com=127.0.0.4
...
I suppose that similar selection would be help with whitelists.
> I assume postscreen processes or passes this data to smtpd in a way that
> smtpd will automatically bypass all checks normally performed during the
> CONNECT phase.
Postscreen passes no session information to the SMTP server. The
file handle is all the SMTP server gets. We're talking about a
really tight development budget here.
Wietse
Stan Hoeppner
> - This is specific for dnswl.org. Postfix needs a general mechanism.
> Other whitelists are not required to follow dnswl.org's 127.0.x.y
> mechanism.
Yeah, I used this example as dnswl is, afaik, the most "established" of
the dns whitelists. I haven't yet looked at the return codes the others
use.
> - what do you mean by "accept the message"; OK? suppress further rbl
> lookups?
This is something that needs discussion due to various expectations of
what a dns "whitelist" entry actually means and how it can best be
implemented. I think messages with a "positive" dnswl return code
should obviously bypass certain restrictions, mainly dnsbl hits, but not
all restrictions--we shouldn't bypass virus checks and content filters.
And if an admin wants to locally ban an IP listed in a dnswl for any
reason, we shouldn't be bypassing user defined access tables.
I think bypassing reject_r*bl_* and most of the inbuilt/standard client,
sender, helo, etc sanity checks would be a good idea. Obviously we
don't want to bypass something like reject_unlisted_recipient.
I think what we really need here is consensus on the use of local
whitelists. Should dns whitelists carry the same weight? If so, we may
want to bypass all restrictions but virus/content filters and recipient
address verification.
> - If "accept" means OK, how will you protect postfix from open relay if
> the dns whitelist accidentally or intentionally lists the whole internet?
We currently run similar risk, but in the opposite direction, using
dnsbls. I don't believe there is a technical solution to this potential
issue. Every dnsbl comes with a "use at your own risk... no
warranty..." disclaimer for this reason and others. Though I think the
dnsbl case is worse as it can potentially stop all email flow. In the
dnswl case, you'd simply see much more spam. Although, at the inbox
level, the "denial of service" result may be equivalent, much like
yesteryear when folks had to sift through 100 spams to find 3 legit
emails they received on a given day. But at least they still received
those 3.
Probably difficult to implement, but maybe some kind of analysis on mail
flow would help. Keep statistics counters and establish an "average
volume" baseline. If volume increases 10x over baseline, throttle all
connections. The programming effort may not be worth the payoff here
given the odds of "list world" occurrences.
> - what would the user interface look like? Is it possible to document
> it clearly?
I think something like I mentioned before would be good. Simple single
line configuration parameter populating one global program variable.
How the variable would be used is the tricky part, and up to others.
I'm not much of a programmer these days. :( I think it would be wise to
allow flexible use of accept_dnswl_client much like reject_r*bl_foo
which can today be used within access tables in addition to directly
within smtpd_foo_restrictions. If we make its functionality very
permissive by default, I probably wouldn't make it globally effective by
default.
WRT documentation, that will depend on the values we allow for the
variable. If we do something like the 0-3 thing similar to dnswl.org,
it's not as clear due to complexity. If we make it boolean, it's very
clear. In this latter case, lots of bold starred italic documentation
needs to inform the OP of the risks of turning it on.
--
Stan
Wietse Venema
> Noel Jones put forth on 8/24/2010 2:18 PM:
>
> > - This is specific for dnswl.org. Postfix needs a general mechanism.
> > Other whitelists are not required to follow dnswl.org's 127.0.x.y
> > mechanism.
>
> Yeah, I used this example as dnswl is, afaik, the most "established" of
> the dns whitelists. I haven't yet looked at the return codes the others
> use.
>
> > - what do you mean by "accept the message"; OK? suppress further rbl
> > lookups?
Postfix has smtpd_mumble_restrictions with a large number of
reject-like features and a smaller number of permit-like features.
A reject terminates evaluation for all smtpd_mumble_restrictions;
a permit terminates evaluation only within one smtpd_mumble_restriction.
If whitelisting were to be used as a permit-like feature (which
has dangerous failure modes as discussed next) then it will have
to behave like all other permit-like features without exception.
DNSWL as a permit-like feature increases the risk of becoming an
open relay. I don't think we want Postfix to massively fail wide
open and become an open relay just because some DNSWL operator made
a bad decision. Besides, I am not convinced that DNSWL is best used
as an unconditional "permit" operation.
Alternatively, DNSWLs would be safe to use when scores from different
lists are added up, and mail is rejected when the total score
exceeds some threshold. With DNSXL lookup implemented as a
reject-like feature, there is no danger of Postfix massively failing
wide open when the DNSWQL operator screws up.
Currently the smtpd configuration language does not yet have weighted
DNSXL lookups. It would be easy enough to configure a global fixed
list with domains and weights. Just copy the code for the deprecated
maps_rbl_domains configuration parameter and the no longer documented
reject_maps_rbl restriction, and add some syntax to the maps_rbl_domains
parser.
Wietse
Stan Hoeppner
> With reject_rbl_client etc. Postfix can use different DNSXLs names
> in different access lists, and filter the result. For example, to
> select responses from some.example.com with value 127.0.0.4:
>
> smtpd_mumble_restrictions =
> ...
> reject_rbl_client some.example.com=127.0.0.4
> ...
>
> I suppose that similar selection would be help with whitelists.
Yes, I believe the dnswl.org implementation is based on RFC5782.
Regarding "safety" against the "list world" and other scenarios, see
section 5, and paragraph 2 of section 7. The tests are the same for
dnsbls and dnswls. You simply check for responses to make sure a dnsXl
is running on the host you're querying before sending real queries. I'm
guessing you already have code for this in the reject_rbl_client code.
>> I assume postscreen processes or passes this data to smtpd in a way that
>> smtpd will automatically bypass all checks normally performed during the
>> CONNECT phase.
>
> Postscreen passes no session information to the SMTP server. The
> file handle is all the SMTP server gets. We're talking about a
> really tight development budget here.
Darn. With all candor and humility Wietse, I don't think postscreen is
the right place to implement dnswl whitelisting. Or, I should say, it's
not a complete dns whitelisting solution, but only a small first step.
If I understand correctly, all this will do is shoot such a whitelisted
client past all the postscreen checks. Virtually all MTAs at IPs listed
by dnswl servers are going to pass the postscreen checks anyway since
postscreen is looking for bots and other ill behaved SMTP clients. Thus
there is little gain in dns whitelisting here here but maybe some extra
performance on loaded MX'en.
IMHO, for dnswl whitelisting to be implemented "properly", or "the way
most people will expect it to behave", it should be implemented almost
exactly like
reject_rbl_client abc.dnsbl.tld n.n.n.n
but with the opposite action--immediately queuing the message for
mailbox delivery--but still executing recipient address verification,
virus checks, and possibly content filter checks etc.
For a full up dnswl implementation, if we can assume some but not all
dnswl servers have multiple responses, then I'm thinking we should do
something just like your reject_rbl_client example, so an OP can
configure the level of "trust/accept" behavior based on his/her
knowledge of a particular configured dnswl server's responses:
accept_dnswl_client abc.dnsbl.tld [n.n.n.n]
As with reject_rbl_client the n.n.n.n would be optional and the default
behavior would be to accept or "OK" a message with any positive response
code from the dnswl server, in the same manner as we would process a
local access table whitelist OK action. We obviously still want
recipient verification, virus scanning, maybe content filtering. So if
I understand Postfix design correctly, we'd implement this in smtpd and
skip all smtpd checks on a positive dnswl hit, assuming the OP inserts
this parameter appropriately close to the top of the restrictions list.
As with reject_rbl_client an OP would be free to insert this dnswl
check in the most appropriate place in the OP's processing order.
--
Stan
Steve Linford
> Stan Hoeppner:
>> Wietse Venema put forth on 8/23/2010 10:11 AM:
>>> Noel Jones:
>>
>>> (Might be time to revisit DNS whitelists in
>>>> postfix.)
>>>
>>> Maybe someone can draft a strawman user interface:
>>>
>>> - what is the configuration syntax
>>>
>>> - what does that syntax mean
>>>
>>> - how to make it safe ( we don't want "open relay" problems)
>>>
>>> I'm currently doing this for postscreen, and won't have time for
>>> other Postfix features.
>>
>> accept_dnswl_client (default: 0)
>>
>> 0 - accept all messages
>> 1 - accept messages with trust level 1-3
>> 2 - accept messages with trust level 2-3
>> 3 - accept messages with trust level 3
>
> This looks somewhat like RFC 5782, with reputation scores and
> confidence values encoded in the lower octets as numbers in the
> range 0-255.
>
> With reject_rbl_client etc. Postfix can use different DNSXLs names
> in different access lists, and filter the result. For example, to
> select responses from some.example.com with value 127.0.0.4:
>
> smtpd_mumble_restrictions =
> ...
> reject_rbl_client some.example.com=127.0.0.4
> ...
>
> I suppose that similar selection would be help with whitelists.
Just to add to the mix if Postfix is working on whitelist implementation... Spamhaus has assigned 127.0.2.0/24 for whitelist return codes. The new Spamhaus Whitelist ("SWL") due out very shortly will return 127.0.2.2 and 127.0.2.3 and Spamhaus' new Domain Whitelist ("DWL") will return 127.0.2.12 and 127.0.2.13. We will explain what these codes mean closer to the release date (sometime in September).
In the case of the Spamhaus Whitelist ("SWL"), the implementation we want people to use is that a message from an IP on the SWL should be allowed immediately past all spam filtering and all content filtering and passed straight to the virus filter if any.
Steve Linford
The Spamhaus Project
http://www.spamhaus.org
Stan Hoeppner
> Just to add to the mix if Postfix is working on whitelist implementation... Spamhaus has assigned 127.0.2.0/24 for whitelist return codes. The new Spamhaus Whitelist ("SWL") due out very shortly will return 127.0.2.2 and 127.0.2.3 and Spamhaus' new Domain Whitelist ("DWL") will return 127.0.2.12 and 127.0.2.13. We will explain what these codes mean closer to the release date (sometime in September).
>
> In the case of the Spamhaus Whitelist ("SWL"), the implementation we want people to use is that a message from an IP on the SWL should be allowed immediately past all spam filtering and all content filtering and passed straight to the virus filter if any.
Thanks for the heads up Steve. :) Great timing.
--
Stan
Noel Jones
with DNS whitelists, each with a slightly different purpose.
While these are both useful, neither depends on the other, so
postfix can implement either or both.
My proposals:
A) scoring in postscreen
A dns whitelist/blacklist scoring system in postscreen,
whereby a dnswl hit can overrule dnsbl hits and other
postscreen tests.
Since this is in postscreen, the only data available would be
the client IP and hopefully the hostname. This would be
automatically safe from open-relay abuse since the client gets
no more rights than other "screened" clients. The transaction
could still be rejected by other tests done in the real SMTP
server. The user interface would consist of
- postscreen_dnsl_client_sites (default empty); a list of
dnsbl and dnswl domains for client IP lookup.
Specify a list of dns blacklist and whitelist sites with an
optional A result: dnsl_site=d.d.d.d, dnsl_site=d.d.d.d, ...
Count one hit when the A record =d.d.d.d is listed. If no
=d.d.d.d is specified, count one hit if one or more A records
is listed.
- postscreen_dnsl_hostname_sites (default empty); a list of
dnsbl and dnswl sites for hostname lookup using the verified
client hostname. This is recommended for domain-based whitelists.
Specify a list of dns blacklist and whitelist sites with an
optional A result, dnsl_site=d.d.d.d, dnsl_site=d.d.d.d, ...
Count one hit when the A record =d.d.d.d is listed. If no
=d.d.d.d is specified, count one hit if one or more A records
is listed.
- postscreen_dnsl_reverse_hostname_sites (default empty); a
list of dnsbl and dnswl sites for hostname lookup using the
unverified client hostname. This is recommended for
domain-based blacklists.
Specify a list of dns blacklist and whitelist sites with an
optional A result, dnsl_site=d.d.d.d, dnsl_site=d.d.d.d, ...
Count one hit when the A record =d.d.d.d is listed. If no
=d.d.d.d is specified, count one hit if one or more A records
is listed.
- postscreen_dnsl_site_score_default (default 1); default
score for any dns list result. I suppose this could be a
non-user-visible default, since it doesn't seem to make much
sense to adjust this; any adjustments would be made in the
score map.
- postscreen_dnsl_site_score_maps (no default); an optional
table that overrides postscreen_dnsl_site_score_default,
indexed by the list name. The key is the exact dnsl site
name as specified in the dnsl sites maps, including any
optional =d.d.d.d result filter. The result is a positive
number for blacklists, negative number for whitelist.
This table is required when using dns whitelists.
- postscreen_dnsl_whitelist_score (default=-1); a "pass"
threshold score. clients scoring at or BELOW this value
trigger the postscreen_dnsl_whitelist_action.
- postscreen_dnsl_blacklist_score (default=1) a "drop"
threshold score. Clients scoring at or ABOVE this value
trigger the postscreen_dnsl_blacklist_action.
- postscreen_dnsl_whitelist_action (default continue); the
action postscreen takes when a client matches the
postscreen_dnsl_whitelist_score. Specify one of:
continue; perform additional postscreen tests to determine
disposition.
pass; exempt the client from further postscreen tests and pass
it to a real SMTP server process
- postscreen_dnsl_blacklist_action (default continue); the
action postscreen takes when a client exceeds the
postscreen_dnsl_blacklist_score. Specify one of:
continue; perform additional postscreen tests to determine
disposition.
drop; drop the connection with a 521 SMTP reply
There are a lot of parameters, but sane defaults would allow a
simple "drop everything using this one blacklist" config,
while still allowing much more complex multi-list scoring. A
minimal config would be simply:
postscreen_dnsl_client_sites = zen.spamhaus.org
postscreen_dnsl_blacklist_action = drop
The downside of this method is it will require a lot of new code.
B) a "permit" based system, a mirror of reject_rbl_client.
This would have a user interface similar to the existing
reject_rbl_client with expected usage similar to access(5)
based whitelists.
Seems to me that checks using sender-supplied info such as
{helo, sender domain, recipient domain} are unsafe -- give
whitelist control to unverified information -- and probably
shouldn't be implemented.
To prevent open-relay accidents, this would need to return
permit_auth_destination rather than a blanket permit. Maybe
the result action will need to be configurable? Nah.
The user interface would be familiar to anyone using rbl
checks. Sample documentation under the appropriate
smtpd_mumble_restrictions section:
- permit_dnswl_client dnswl_domain=d.d.d.d
Accept the request when the reversed client IP network
address is listed with an A record of d.d.d.d under
dnswl_domain. If no =d.d.d.d is given, accept the request
with any A record under dnswl_domain. For safety, only
authorized destinations are accepted, see permit_auth_destination.
- permit_rhswl_client rhswl_domain=d.d.d.d
Accept the request when the client hostname is listed with
an A record of d.d.d.d under rhswl_domain. If no =d.d.d.d is
given, accept the request with any A record under
rhswl_domain. For safety, only authorized destinations are
accepted, see permit_auth_destination.
Seems like this one would be very easy to use, and fairly easy
to implement.
-- Noel Jones
Wietse Venema
> As I see it, there are two complementary paths we can take
> with DNS whitelists, each with a slightly different purpose.
> While these are both useful, neither depends on the other, so
> postfix can implement either or both.
I'll read the entire proposal later.
Would this notation work:
dnswl1.example.com=127.0.0.2*weight1, dnswl2.example.com=127.0.0.1*weight2
dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
It would reduce the number of config parameters.
Do we want to allow mixing DNSWLs and DNSBLs in one list?
Currently, postscreen does not look up the client hostname, that
is something that can be added later when there is time.
Wietse
Noel Jones
> Noel Jones:
>> As I see it, there are two complementary paths we can take
>> with DNS whitelists, each with a slightly different purpose.
>> While these are both useful, neither depends on the other, so
>> postfix can implement either or both.
>
> I'll read the entire proposal later.
>
> Would this notation work:
>
> dnswl1.example.com=127.0.0.2*weight1, dnswl2.example.com=127.0.0.1*weight2
> dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
>
> It would reduce the number of config parameters.
Yes. Both the A filter and the weight would need to be
optional. ie, if you want any result to have a custom weight,
it must accept
dnsbl1.example.com=*weight
This would eliminate one lookup table and make list usage more
obvious. Good.
>
> Do we want to allow mixing DNSWLs and DNSBLs in one list?
I see them as being the same thing; just different weights.
Default to blacklist weight of 1; the user must specify a
negative weight for a whitelist.
I think it would unnecessarily clutter the user interface to
have separate lists. But maybe that's just me.
> Currently, postscreen does not look up the client hostname, that
> is something that can be added later when there is time.
I know. I was just hoping the hostname would be available in
the new-and-improved postscreen discussed elsewhere.
>
> Wietse
-- Noel Jones
Wietse Venema
> On 8/25/2010 4:27 PM, Wietse Venema wrote:
> > Noel Jones:
> >> As I see it, there are two complementary paths we can take
> >> with DNS whitelists, each with a slightly different purpose.
> >> While these are both useful, neither depends on the other, so
> >> postfix can implement either or both.
> >
> > I'll read the entire proposal later.
> >
> > Would this notation work:
> >
> > dnswl1.example.com=127.0.0.2*weight1, dnswl2.example.com=127.0.0.1*weight2
> > dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
> >
> > It would reduce the number of config parameters.
>
> Yes. Both the A filter and the weight would need to be
> optional. ie, if you want any result to have a custom weight,
> it must accept
>
> dnsbl1.example.com=*weight
I assume the syntax would be like this:
domain[=addr][*weight]
where content inside [] is optional.
> > Currently, postscreen does not look up the client hostname, that
> > is something that can be added later when there is time.
>
> I know. I was just hoping the hostname would be available in
> the new-and-improved postscreen discussed elsewhere.
The primary goal is to keep spambots away from Postfix, so I'll
focus on features that do for the short term. The only luxury will
be the dummy SMTP engine that logs the rejected client, helo,
sender, recipient. Right now I don't miss that at all.
Wietse
Rob Foehl
> The user interface would be familiar to anyone using rbl checks. Sample
> documentation under the appropriate smtpd_mumble_restrictions section:
>
> - permit_dnswl_client dnswl_domain=d.d.d.d
> Accept the request when the reversed client IP network address is listed
> with an A record of d.d.d.d under dnswl_domain. If no =d.d.d.d is given,
> accept the request with any A record under dnswl_domain. For safety, only
> authorized destinations are accepted, see permit_auth_destination.
>
> - permit_rhswl_client rhswl_domain=d.d.d.d
> Accept the request when the client hostname is listed with an A record of
> d.d.d.d under rhswl_domain. If no =d.d.d.d is given, accept the request with
> any A record under rhswl_domain. For safety, only authorized destinations
> are accepted, see permit_auth_destination.
>
> Seems like this one would be very easy to use, and fairly easy to implement.
This sounds like a reasonable proposal, and I would argue that maintaining
parity with existing smtpd features is important, whether or not
postscreen ever grows an analogous mechanism. Unconditionally returning
permit_auth_destination should make this suitably safe, despite the simple
interface.
As it happens, I have a partial implementation of such a feature that I
was playing with a few years ago, and could probably be coerced into
updating it for current releases and posting a patch, if there is further
consensus that this is the desired interface and/or mechanism.
-Rob
Noel Jones
> Noel Jones:
>> On 8/25/2010 4:27 PM, Wietse Venema wrote:
>>> Noel Jones:
>>>> As I see it, there are two complementary paths we can take
>>>> with DNS whitelists, each with a slightly different purpose.
>>>> While these are both useful, neither depends on the other, so
>>>> postfix can implement either or both.
>>>
>>> I'll read the entire proposal later.
>>>
>>> Would this notation work:
>>>
>>> dnswl1.example.com=127.0.0.2*weight1, dnswl2.example.com=127.0.0.1*weight2
>>> dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
>>>
>>> It would reduce the number of config parameters.
>>
>> Yes. Both the A filter and the weight would need to be
>> optional. ie, if you want any result to have a custom weight,
>> it must accept
>>
>> dnsbl1.example.com=*weight
>
> I assume the syntax would be like this:
>
> domain[=addr][*weight]
>
> where content inside [] is optional.
Excellent.
>
>>> Currently, postscreen does not look up the client hostname, that
>>> is something that can be added later when there is time.
>>
>> I know. I was just hoping the hostname would be available in
>> the new-and-improved postscreen discussed elsewhere.
>
> The primary goal is to keep spambots away from Postfix, so I'll
> focus on features that do for the short term. The only luxury will
> be the dummy SMTP engine that logs the rejected client, helo,
> sender, recipient. Right now I don't miss that at all.
I've been reluctant to turn on postscreen dns blacklists on
the main MX because it's so much harder for me to find the
once-in-a-blue-moon FPs. But it runs gangbusters on the
secondary.
I won't miss the dnsbl hostname lookups in postscreen (it also
eliminates a few more parameters), but it would still be nice
to have simple hostname rhswl support in smtpd as described
later in my outline. I see these as complementary features.
-- Noel Jones
Ralf Hildebrandt
> Noel Jones:
> > As I see it, there are two complementary paths we can take
> > with DNS whitelists, each with a slightly different purpose.
> > While these are both useful, neither depends on the other, so
> > postfix can implement either or both.
>
> I'll read the entire proposal later.
>
> Would this notation work:
>
> dnswl1.example.com=127.0.0.2*weight1, dnswl2.example.com=127.0.0.1*weight2
> dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
weightn can be negative?
> Do we want to allow mixing DNSWLs and DNSBLs in one list?
Probably, with positiv and negative weights?
Matthias Leisi
> dnswl1.example.com=127.0.0.2*weight1, dnswl2.example.com=127.0.0.1*weight2
> dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
What about wildcarding? dnswl.org currently returns 127.0.n.[0-3],
with "n" being numerical for the category (eg banks etc). People may
want to have something like "whitelist on 127.0.*.2 and 127.0.*.3".
// Yes, the use and ordering of the octets was possibly not the best
design choice back when.
-- Matthias
Wietse Venema
> > ?dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
>
> What about wildcarding? dnswl.org currently returns 127.0.n.[0-3],
> with "n" being numerical for the category (eg banks etc). People may
> want to have something like "whitelist on 127.0.*.2 and 127.0.*.3".
The "*" operator is already used for the weight factor. I'm not
sure that it is a good idea to use the same "*" operator for
wildcarding.
Wietse
Stan Hoeppner
> Noel Jones:
>> As I see it, there are two complementary paths we can take
>> with DNS whitelists, each with a slightly different purpose.
>> While these are both useful, neither depends on the other, so
>> postfix can implement either or both.
>
> I'll read the entire proposal later.
>
> Would this notation work:
>
> dnswl1.example.com=127.0.0.2*weight1, dnswl2.example.com=127.0.0.1*weight2
> dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
IMO this really depends on the "role" you expect postscreen to play. If
it is expected that all Postfix implementations are going to be using
postscreen as in integral important feature, not just the big operations
with hundreds of thousands or millions of connections per day, then I
would suggest this user configuration interface is too complicated.
There will be OPs who simply want to use a single dns whitelist without
any kind of scoring. They will want an "OK" on any 127.0.x.x response.
> It would reduce the number of config parameters.
Always a plus, but in this case it may sacrifice (ease of) usability for
some OPs.
> Do we want to allow mixing DNSWLs and DNSBLs in one list?
If you're implementing a pure scoring system this might work. I don't
know what the math should be though for calculating an "OK" threshold.
> Currently, postscreen does not look up the client hostname, that
> is something that can be added later when there is time.
Won't all of these dns lookups slow postscreen down? I thought it was
supposed to be a really high throughput low latency check/allow/dump
filter mainly to shield smtpd processes from the bot load. What's the
average dnsxl response time? 50ms? 100ms? Won't this terribly
decrease the throughput of postscreen?
If postscreen is doing dnsxl lookups merely to decide if a connection
gets past postscreen, and then we do the same dnsbl lookup in smtpd to
decide accept/reject because postscreen doesn't pass the result to
smtpd, it makes me wonder why we're doing two such lookups for each
connect that is OK'd by postscreen.
Again, I'm not so sure that doing dnswl lookups with postscreen is
productive. Most hosts listed on dnswls are going to pass postscreen's
normal checks anyway. By doing this it seems we're merely adding
processing latency to a latency sensitive code path, and we're not
really gaining anything by doing so. These dnswl lookups will have to
be duplicated in smtpd anyway to get the desired whitelisting result OPs
expect from a dnswl positive return.
--
Stan
Wietse Venema
> Wietse Venema put forth on 8/25/2010 4:27 PM:
> > Noel Jones:
> >> As I see it, there are two complementary paths we can take
> >> with DNS whitelists, each with a slightly different purpose.
> >> While these are both useful, neither depends on the other, so
> >> postfix can implement either or both.
> >
> > I'll read the entire proposal later.
> >
> > Would this notation work:
> >
> > dnswl1.example.com=127.0.0.2*weight1, dnswl2.example.com=127.0.0.1*weight2
> > dnsbl3.example.com=127.0.0.3*weight3, dnsbl4.example.com=127.0.0.1*weight4
>
> IMO this really depends on the "role" you expect postscreen to play. If
The 127.x.x.x filters are OPTIONAL, as are the weight factors.
Sites that don't need them don't specify them.
This is a basic Postfix principle. If you don't need some feature
then you don't need to know how to configure it.
> > Currently, postscreen does not look up the client hostname, that
> > is something that can be added later when there is time.
>
> Won't all of these dns lookups slow postscreen down?
Hostname DNS lookup is no worse than DNSBL or DNSWL lookup, or
talking with an SMTP client for pregreet or greylist tests.
Postscreen does these things in parallel: one postscreen process
handles many clients, instead of wasting one smtpd process per
spambot. It's a scalability feature.
> If postscreen is doing dnsxl lookups merely to decide if a connection
> gets past postscreen, and then we do the same dnsbl lookup in smtpd to
In that case Postscreen's DNSWL lookup would work as a DNS cache
prefetch, so it would still improve Postfix over-all performance.
Wietse
Noel Jones
> On 8/25/2010 4:27 PM, Wietse Venema wrote:
>> Noel Jones:
>
> Default to blacklist weight of 1; the user must specify a
> negative weight for a whitelist.
I've changed my mind on this. While might make sense to
combine dnsbl & dnswl in one list, the problem comes up if we
ever implement rhsXl because the hostname requirements are
different. With an rhsbl, you want the unverified reverse
name; with an rhswl, you need the FCrDNS standard postfix
hostname.
Separating the lists also allows us to use whitelist and
blacklist default weight values, reducing configuration burden.
Matthias Leisi wrote:
> What about wildcarding? dnswl.org currently returns 127.0.n.[0-3],
> with "n" being numerical for the category (eg banks etc). People may
> want to have something like "whitelist on 127.0.*.2 and 127.0.*.3".
>
This looks like a useful concept. If we use "*" as an octet
wildcard, we'll need to use something else as the weight
modifier. dnsbl_site=127.0.*.3w1 seems reasonable.
So, the modified proposal for postscreen would be:
- postscreen_dnsbl_sites (default empty); A comma separated
list of dnsbl IP blacklist sites with optional result filter
and optional weight. Specify one or more dnsbl sites as:
dnsbl_site[=d.d.d.d][wN]
where dnsbl_site is the site name, d.d.d.d is the optional
result filter, and N is the optional weight value. Wildcard
octets in the result filter can be indicated with "*". If no
result filter is given, any result is considered a match. The
weight is given as the letter "w" followed by a value in the
range [-99~+99] inclusive. If "+" or "-" is not specified,
"+" is assumed. If no weight is given, the default weight
value is +1.
Examples:
postscreen_dnsbl_sites =
dnsbl_site1
dnsbl_site2=127.0.*.2w5
dnsbl_site3=w+6
- postscreen_dnswl_sites (default empty); A comma separated
list of dnswl IP whitelist sites with optional result filter
and optional weight. Specify one or more dnswl sites as:
dnswl_site[=d.d.d.d][wN]
where dnswl_site is the site name, d.d.d.d is the optional
result filter, and N is the optional weight value. Wildcard
octets in result filter can be indicated with "*". If no
result filter is given, any result is considered a match. The
weight is given as the letter "w" followed by a value in the
range [-99~+99] inclusive. If "+" or "-" is not specified,
"-" is assumed. If no weight is given, the default weight
value is -1.
Examples:
postscreen_dnswl_sites =
dnswl_site1
dnswl_site2=127.0.*.2w5
dnswl_site3=w-6
(next two items are for future expansion if hostnames are
available)
(name tweaks? postscreen_dnsbl_hostname_sites and
postscreen_dnswl_hostname_sites so one can grep all related
parameters with postscreen_dns*?)
- postscreen_rhsbl_sites (default empty); A comma separated
list of rhsbl hostname blacklist sites using the unverified
client hostname with optional result filter and optional
weight. Specify one or more rhsbl sites as:
rhsbl_site[=d.d.d.d][wN]
where rhsbl_site is the site name, d.d.d.d is the optional
result filter, and N is the optional weight value. Wildcard
octets in the result filter can be indicated with "*". If no
result filter is given, any result is considered a match. The
weight is given as the letter "w" followed by a value in the
range [-99~+99] inclusive. If "+" or "-" is not specified,
"+" is assumed. If no weight is given, the default weight
value is +1.
Examples:
postscreen_rhsbl_sites =
rhsbl_site1
rhsbl_site2=127.0.*.2w5
rhsbl_site3=w+6
- postscreen_rhswl_sites (default empty); A comma separated
list of rhswl hostname whitelist sites using the client
hostname with optional result filter and optional weight.
Specify one or more rhswl sites as:
rhswl_site[=d.d.d.d][wN]
where rhswl_site is the site name, d.d.d.d is the optional
result filter, and N is the optional weight value. Wildcard
octets in the result filter can be indicated with "*". If no
result filter is given, any result is considered a match. The
weight is specified as the letter "w" followed by a value in
the range [-99~+99] inclusive. If "+" or "-" is not
specified, "-" is assumed. If no weight is given, the default
weight value is -1.
Examples:
postscreen_rhsbl_sites =
rhswl_site1
rhswl_site2=127.0.*.2w5
rhswl_site3=w-6
(below is unchanged except for the name change to *_dnsxl_*)
- postscreen_dnsxl_whitelist_score (default=-1); a "pass"
threshold score. clients scoring at or BELOW this value
trigger the postscreen_dnsxl_whitelist_action.
- postscreen_dnsxl_blacklist_score (default=1) a "drop"
threshold score. Clients scoring at or ABOVE this value
trigger the postscreen_dnsxl_blacklist_action.
- postscreen_dnsxl_whitelist_action (default continue); the
action postscreen takes when a client matches the
postscreen_dnsxl_whitelist_score. Specify one of:
continue; perform additional postscreen tests to determine
disposition.
pass; exempt the client from further postscreen tests and pass
it to a real SMTP server process
- postscreen_dnsxl_blacklist_action (default continue); the
action postscreen takes when a client exceeds the
postscreen_dnsxl_blacklist_score. Specify one of:
continue; perform additional postscreen tests to determine
disposition.
drop; drop the connection with a 521 SMTP reply
-- Noel Jones
Wietse Venema
> This looks like a useful concept. If we use "*" as an octet
> wildcard, we'll need to use something else as the weight
> modifier. dnsbl_site=127.0.*.3w1 seems reasonable.
You can't use an alphanumerical operator such as "w", because the
"=127.0.*.3" portion is optional.
Wietse
Noel Jones
Rats. =127.0.*.3^1 maybe? Your suggestion? * is such a
natural wildcard I would hate to use something else for the octet.
-- Noel Jones
Wietse Venema
> This looks like a useful concept. If we use "*" as an octet
> wildcard, we'll need to use something else as the weight modifier.
> dnsbl_site=127.0.*.3w1 seems reasonable.
On 8/26/2010 2:28 PM, Wietse Venema wrote:
> You can't use an alphanumerical operator such as "w", because the
> "=127.0.*.3" portion is optional.
Noel Jones:
> Rats. =127.0.*.3^1 maybe? Your suggestion? * is such a
> natural wildcard I would hate to use something else for the octet.
Use "*" for the multiplier and "?" for the wildcard?
Another idea is to implement wildcards with net/mask patterns:
example.com=127.0.0.3/255.255.0.255*1
The more precise solution is to implement wildcards with ranges:
example.com=127.0.[0-128].3*1
example.com=127.0.[0-5,6-9].3*1
Wietse
Noel Jones
> On 8/26/2010 2:28 PM, Wietse Venema wrote:
>> You can't use an alphanumerical operator such as "w", because the
>> "=127.0.*.3" portion is optional.
> The more precise solution is to implement wildcards with ranges:
>
> example.com=127.0.[0-128].3*1
> example.com=127.0.[0-5,6-9].3*1
>
> Wietse
I like the range idea. You want proto docs reflecting that
syntax?
-- Noel Jones
Wietse Venema
> The more precise solution is to implement wildcards with ranges:
>
> example.com=127.0.[0-128].3*1
> example.com=127.0.[0-5,6-9].3*1
Noel Jones:
> I like the range idea. You want proto docs reflecting that
> syntax?
Yes, that would help everyone to understand how it will work.
Wietse
Noel Jones
Updated Proposal for weighted dnsXl support in postscreen.
(Change parameter names to all start with postscreen_dns* for
easy reading in postconf. Get rid of negative site weight
values [the client dnsxl score total may still be negative].
Add filter octet range docs.)
(The weight ranges documented are arbitrary.)
- postscreen_dnsbl_sites (default empty); A comma separated
list of dnsbl IP blacklist sites with optional result filter
and optional weight. When the reversed client network address
is listed with an A record matching the result filter, add
{weight} points (default 1) to the client dnsxl score. If no
result filter is specified, add {weight} points (default 1) to
the client dnsxl score if any A record is found. If multiple
A records are found, {weight} will only be added once.
Specify one or more dnsbl sites as:
dnsbl_site[=d.d.d.d][*N]
where dnsbl_site is the site name, d.d.d.d is the optional
result filter, and N is the optional weight value. A range in
the result filter may be specified within brackets in place of
an octet. Multiple ranges or single values may be separated
by commas within the brackets. The weight may be specified as
the character "*" followed by a value in the range [0~99]
inclusive.
Examples:
postscreen_dnsbl_sites =
dnsbl_site1,
dnsbl_site2=127.0.[0-5,22,128-255].2*5,
dnsbl_site3=*6
- postscreen_dnswl_sites (default empty); A comma separated
list of dnswl IP whitelist sites with optional result filter
and optional weight. When the reversed client network address
is listed with an A record matching the result filter,
subtract {weight} points (default 1) from the client dnsxl
score. If no result filter is specified, subtract {weight}
points (default 1) from the client dnsxl score if any A record
is listed. If multiple A records are found, {weight} will
only be subtracted once.
Specify one or more dnswl sites as:
dnswl_site[=d.d.d.d][*N]
where dnswl_site is the site name, d.d.d.d is the optional
result filter, and N is the optional weight value. A range in
the result filter may be specified within brackets in place of
an octet. Multiple ranges or single values may be separated
by commas within the brackets. The weight may be specified as
the character "*" followed by a value in the range [0~99]
inclusive.
Examples:
postscreen_dnswl_sites =
dnswl_site1,
dnswl_site2=127.0.[0-5,22,128-255].2*5,
dnswl_site3=*6
(these next parameters behavior is unchanged, but the docs
have been updated)
(Require a "+" or "-" sign for the score thresholds to prevent
ambiguity. The alternatives are to assume "-" for the
whitelist and "+" for the blacklist, or always assume "+". I
think it's least confusing to just require the sign.)
(The score threshold range is arbitrary.)
- postscreen_dnsxl_whitelist_score (default -1); a "pass"
threshold for the total of the client's dnsxl points. Specify
a value in the range [-999~+999] inclusive. The sign must be
specified. Clients scoring at or BELOW this value trigger the
postscreen_dnsxl_whitelist_action. Clients scoring greater
than postscreen_dnsxl_whitelist_score, but less than
postscreen_dnsxl_blacklist_score continue with postscreen
analysis for disposition.
Example:
postscreen_dnsxl_whitelist_score = -5
- postscreen_dnsxl_blacklist_score (default=1) a "drop"
threshold for the total of the client's dnsxl points. Specify
a value in the range [-999~+999] inclusive. The sign must be
specified. Clients scoring at or ABOVE this value trigger the
postscreen_dnsxl_blacklist_action. Clients scoring greater
than postscreen_dnsxl_whitelist_score, but less than
postscreen_dnsxl_blacklist_score continue with postscreen
analysis for disposition.
Example:
postscreen_dnsxl_blacklist_score = +5
- postscreen_dnsxl_whitelist_action (default continue); the
action postscreen takes when a client matches the
postscreen_dnsxl_whitelist_score.
Specify one of:
continue; perform additional postscreen tests to determine
disposition.
pass; exempt the client from further postscreen tests and pass
it to a real SMTP server process
- postscreen_dnsxl_blacklist_action (default continue); the
action postscreen takes when a client exceeds the
postscreen_dnsxl_blacklist_score.
Specify one of:
continue; perform additional postscreen tests to determine
disposition.
drop; drop the connection with a 521 SMTP reply
(next two items are for future expansion if hostnames are
available)
- postscreen_dnsbl_hostname_sites (default empty); A comma
separated list of rhsbl hostname blacklist sites using the
unverified client hostname with optional result filter and
optional weight. When the unverified reverse client hostname
is listed with an A record matching the result filter, add
{weight} points (default 1) to the client dnsxl score. If no
result filter is specified, add {weight} points (default 1) to
the client dnsxl score if any A record is listed. If multiple
A records are found, {weight} will only be added once.
Specify one or more rhsbl sites as:
rhsbl_site[=d.d.d.d][*N]
where rhsbl_site is the site name, d.d.d.d is the optional
result filter, and N is the optional weight value. A range in
the result filter may be specified within brackets in place of
an octet. Multiple ranges or single values may be separated
by commas within the brackets. The weight may be specified as
the character "*" followed by a value in the range [0~99]
inclusive.
Examples:
postscreen_dnsbl_hostname_sites =
dnsbl_site1,
dnsbl_site2=127.0.[0-5,22,128-255].2*5,
dnsbl_site3=*6
- postscreen_dnswl_hostname_sites (default empty); A comma
separated list of rhswl hostname whitelist sites using the
FCrDNS verified client hostname with optional result filter
and optional weight. When the client hostname is listed with
an A record matching the result filter, subtract {weight}
points (default 1) from the client dnsxl score. If no result
filter is specified, subtract {weight} points (default 1) from
the client dnsxl score if any A record is listed. If multiple
A records are found, {weight} will only be subtracted once.
Specify one or more rhswl sites as:
rhswl_site[=d.d.d.d][*N]
where rhswl_site is the site name, d.d.d.d is the optional
result filter, and N is the optional weight value. A range in
the result filter may be specified within brackets in place of
an octet. Multiple ranges or single values may be separated
by commas within the brackets. The weight may be specified as
the character "*" followed by a value in the range [0~99]
inclusive.
Examples:
postscreen_dnswl_hostname_sites =
dnswl_site1,
dnswl_site2=127.0.[0-5,22,128-255].2*5,
dnswl_site3=*6
-- Noel Jones
Noel Jones
> On Wed, 25 Aug 2010, Noel Jones wrote:
>
>> The user interface would be familiar to anyone using rbl
>> checks. Sample documentation under the appropriate
>> smtpd_mumble_restrictions section:
>>
>> - permit_dnswl_client dnswl_domain=d.d.d.d
>> address is listed with an A record of d.d.d.d under
>> dnswl_domain. If no =d.d.d.d is given, accept the request
>> with any A record under dnswl_domain. For safety, only
>> authorized destinations are accepted, see
>> permit_auth_destination.
>>
>> - permit_rhswl_client rhswl_domain=d.d.d.d
>> an A record of d.d.d.d under rhswl_domain. If no =d.d.d.d is
>> given, accept the request with any A record under
>> rhswl_domain. For safety, only authorized destinations are
>> accepted, see permit_auth_destination.
>>
>> Seems like this one would be very easy to use, and fairly
>> easy to implement.
>
> This sounds like a reasonable proposal, and I would argue that
> maintaining parity with existing smtpd features is important,
> whether or not postscreen ever grows an analogous mechanism.
> Unconditionally returning permit_auth_destination should make
> this suitably safe, despite the simple interface.
Although most discussion has been about postscreen, I'm still
very interested in dns whitelisting in smtpd.
Once we (collectively) get the postscreen dnsxl scoring user
interface sorted out, it should be possible to adapt the
framework for smtpd without reinventing the wheel.
Then there would be a unified interface for dnsxl scoring.
> As it happens, I have a partial implementation of such a
> feature that I was playing with a few years ago, and could
> probably be coerced into updating it for current releases and
> posting a patch, if there is further consensus that this is
> the desired interface and/or mechanism.
>
> -Rob
The simple interface proposed above should be much easier to
implement (I'll bet a lot of existing rbl code could be
reused), but let's shoot for Mars right now. If we have to
settle for the Moon later, that's not so bad.
-- Noel Jones
pf at alt-ctrl-del.org
Noel Jones August 23, 2010
>> * p...@alt-ctrl-del.org<p...@alt-ctrl-del.org>:
>>> I find that a lot of spam comes from recently registered, throw away
>>> domains. The new domain may be used as the sender, hostname, or name
>>> server.
>>>
>>> Are there any rbl type lists that block fresh domains, for the first
>>> 10-15 days of their existence?
>
> I've also been playing with these:
> http://spameatingmonkey.com/lists.html
> The FRESH lists are what you're looking for.
>
>
Again, thanks.
The fresh15 list + log monitoring really worked out well. It's been a good early warning system.
I have placed the fresh15 test, after all other tests. A few weeks of monitoring show that most of the positive hits
come from a few specific networks. The senders from these networks generally have proper fcrdns, and the helo and "from"
domain matches the fcrdns.
Blacklisting mail from these networks has made a significant dent.
Prior to blocking, >1 fresh15 hit per minute.
After blocking: as low as 1 fresh15 hit per 2-3 hours, up to 15 hits per hour.