How important is using an SMTP server that matches the email address?

[Yeechang Lee]

I have been a Pobox customer for many years. Everything addressed to
my pobox.com address is forwarded to wherever and however I read my
mail. I run a Postfix server at home and use Pobox's SMTP server as a
relay for outgoing messages.

I may switch to another forwarding address, which might not have an
associated SMTP server. Am I correct in believing that SPF (which
Pobox pioneered) and other anti-spam technologies rely, in part, on
matching the address the message is putatively from with the SMTP
server that sent it? In other words, if I switch away from my current
setup,[*] how much more likely is my mail going to be mistakenly
flagged as spam by some filter along the way because my From: line
doesn't match the originating SMTP server?

[*] If I were to change I wouldn't send out messages directly from my
home Postfix server; I know many spam filters zap all SMTP traffic
from ISP address blocks for good reason. I would still use a reputable
SMTP relay; it just wouldn't match my From: line.

[Martin Gregorie]

On Thu, 28 Jun 2012 10:27:19 -0700, Yeechang Lee wrote:

> I may switch to another forwarding address, which might not have an
> associated SMTP server. Am I correct in believing that SPF (which Pobox
> pioneered) and other anti-spam technologies rely, in part, on matching
> the address the message is putatively from with the SMTP server that
> sent it?
>

Yes. The SPF record needs to give the IP of your ISP's network-facing
mailserver - probably one or more of their MX addresses.

> [*] If I were to change I wouldn't send out messages directly from my
> home Postfix server; I know many spam filters zap all SMTP traffic from
> ISP address blocks for good reason.
>

Correct. Sending SMTP direct from an IP that is in an ISP's block of
dynamically assigned IPs is a sure-fire way to get labeled as a spammer.

I send via my ISPs mail host despite having a static IP. It works for me.


--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |

[Yeechang Lee]

Martin Gregorie wrote:
> Yes. The SPF record needs to give the IP of your ISP's
> network-facing mailserver - probably one or more of their MX
> addresses.

But how much difference do such tests make? Let's say I switch to
yl...@invalid.com as my public address, but still use smtp.pobox.com as
the relay. Is there a meaningfully higher chance that a non-spam
message I send out will now get caught by some filter somewhere
because of the mismatch, even though I authenticate properly with the
pobox smtp server? Or am I worrying over nothing?

[Martin Gregorie]

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Thu, 28 Jun 2012 19:00:33 -0700, Yeechang Lee wrote:

Do some testing using actual spam you've received vs legit mail with a
non-forged sender. Everybody's spam stream is different.

I have a validated SPF record out there which does a good job of flagging
up backscatter. I don't use it for anything else but then again I don't
need to.

My incoming mail stream goes:
ISP's mail server---pop3-->getmail--->SA--->sendmail---smtp--->my Postfix

while the outgoing stream does this:
my Postfix ---smtp--->ISP's mail server

So, as you can see, I have complete separation between incoming and
outgoing mail streams and, by definition, any mail Spamassassin sees that
claims to be sent by me is spam.

[Martin Gregorie]

On Fri, 29 Jun 2012 03:31:42 +0000, Martin Gregorie wrote:

Arrrgh!

Blasted Pan and/or newtwok conspired to cause that multi-post.

[Nomen Nescio]

Yeechang Lee <yl...@pobox.com> wrote:
>
> [*] If I were to change I wouldn't send out messages directly from
> my home Postfix server; I know many spam filters zap all SMTP
> traffic from ISP address blocks for good reason. I would still use a
> reputable SMTP relay; it just wouldn't match my From: line.

Actually no, there is no good reason to block on the basis of IP
address. Misguided configurations like this are a reckless and
unsophisticated means to fight spam - (increasing the *real* spam
damage: false positives). It violates EFF principles, but the large
corps that do it have little sense of ethics. The bottom line is that
careless IP-based validation saves them money. It's purely about
saving a buck.

Anyway, a good approach is to send mail direct by default. If an
unwitting recipient is a Time Warner customer, the way to give Time
Warner special treatment is to use the postfix transport file.
E.g. you can have the transport file exceptionally send *.rr.com mail
via whatever relay you specify. When a message is refused, add the
MX domain to the transport file. This file can be considered a wall
of shame listing unethical or naively configured mail servers.

[Spam Guy]

Nomen Nescio wrote:

> Actually no, there is no good reason to block on the basis of IP
> address.

I manage the smtp server at a small company, have been since about 1998.

When-ever we get a spam, I check the sending IP against an inventory of
all the e-mail we've ever received going back about 6 years.

More specifically, say the spam came from 1.2.3.4. I check and see if
we've ever received a valid or "good" e-mail from 1.2.0.0/16. If not,
then I block 1.2.0.0/16.

We get maybe a dozen spams a week to addresses like "sales@" and
"support@".

Back in 2005 these accounts were getting 50 spams a day - and I would
block 1.2.3.0/24 - but that was useless.

Then in 2006 we changed ISP's and I forgot to enter our MX record - but
I didn't realize it for a few months. We were still getting mail - but
the spam seemed to have stopped. I didn't realize that when an MX
lookup fails, that the sending server will resort to the A record, and
that's what was happening. But spam zombies don't impliment the full
smtp specification, so they didn't do anything when the mx lookup
failed. That worked well for about 2 years and then some spam started
trickling back in.

I then looked up some IP allocation lists for various countries.

We block all of Russia and Ukraine, all of South and Central America,
most of China (with some exceptions) and most of Africa (except South
Africa).

Our "Contact Us" web-page lists a gmail account (which forwards to our
"sales@" account) that can be used if someone has tried our regular
contact address and gets a "delivery failed" response. When someone
tells us they've gotten a "delivery failed" or "rejected" response, I'll
go in and adjust the IP-address blocking setting to prevent it from
happening in the future. I might have to do that once or twice a year.

If I find that a particular domain is used frequently in the "Return
Path" then I can block based on that as well. I block many "yahoo.x"
domains (where x is .co.uk, .hk, .mx, etc, but not .com or .ca).

Our SMTP software (post.office, made by the defunct software.com) runs
on an NT-4 server and dates to about 1999 or 2001. It's the most
rock-solid piece of software running on a rock-solid server I've ever
seen. I basically do nothing to manage it other than adjust the
blocking list and create new accounts, adjust aliases and forwarding,
etc. And I do all that from a simple web-based management interface.

I only wish it could do more in terms of blocking based on subject line,
user-agent, etc. It doesn't do any message heuristics (I think blocking
based on that is a crock and a mine-field of trouble). It can only
block based on IP, and the full or partial address in the Return Path.

It has a limit of 10,000 lines in the IP block-list. Each line can
contain an individual IP or a subnet. I've got it filled with about
8,000 entries at the moment. It doesn't connect to any DNSRBL service
(I probably wouldn't use it even if it did).

[Spam Guy]

Spam Guy wrote:

> More specifically, say the spam came from 1.2.3.4. I check and see
> if we've ever received a valid or "good" e-mail from 1.2.0.0/16.
> If not, then I block 1.2.0.0/16.
>

> It has a limit of 10,000 lines in the IP block-list. I've got it

> filled with about 8,000 entries at the moment.

Back in early September last year I did an analysis on one day of the
server's log files in terms of how many messages were received and
rejected.

In summary:

==============
SMTP-Deliver (sent) 42
SMTP-Accept (connect) 85 (non-local machines)
SMTP-Accept (received) 86
SMTP-Accept (closed) 83
SMTP-Accept (timeout) 2
SMTP-Accept (sender blocked) 2
SMTP-Accept (connection refused) 477

So what we have here is this: 42 messages were sent from local users to
a combination of other local users and external users. 85 external
machines made connections which resulted in messages received and 83
closed properly, 2 resulted in a time-out. 2 connections were made but
the mail was rejected due to blocking rules based on envelope sender
(Return-Path).

There were 477 times when a machine tried to make a connection but was
refused based on the machine's IP address. Lets take a more detailed
look at those.

Sorting those based on IP address, we find that those 477 attempts were
made by a total of 67 unique IP addresses. If we break those 67 down in
terms of how many times they tried to make an SMTP connection, we have
this list:

Number of unique IP's / Number of attempts per unique IP

38 / 1
6 / 2
3 / 3
1 / 4
2 / 5
1 / 6
1 / 7
1 / 8
1 / 9
1 / 10
2 / 13
1 / 14
1 / 15
2 / 16
1 / 20 (91.121.14.131)
1 / 25 (173.224.223.172)
1 / 35 (98.129.53.22)
1 / 45 (161.58.219.184)
1 / 136 (216.127.132.240)

So we have 38 IP's (38 out of 67 or 57%) that made only 1 attempt.
Apparently they got the message that their connection was rejected by
our server, and they did not try again. 6 made 2 attempts, 3 made 3
attempts, and so on.

We see that one or two made up to 15 or 16 attempts, then none made 17,
18, or 19 attempts. But one made an even 20 attempts, and another one
made an even 25 attempts, and another one made 35 attempts, and another
one made 45 attempts (I've put the IP address beside those that made
those nice even or round-number attempts).

I would guess that these attempts (20, 25, 35, 45) are hard-coded based
on their divisibility by 5, and might indicate a spam-bot engine that is
either unaware of SMTP rejection rules or ignores them. An analysis of
more of my mail logs would be needed to confirm this hypothesis of the
"rule of 5".

Now there is one IP that really sticks out. It was rejected a total of
136 times. I believe I've seen this sort of phenomena during casual
over-looking of my log files in the past - where one (or more) machines
just don't get the message and keep trying to connect to my server
dozens or hundreds of times on a given day.

So I would have to say that how-ever my server is performing the
rejection, that the majority (57% on this day) of rejected machines did
fully understand the rejection and behaved accordingly (they did not
make a second attempt).

One final aspect is to sort the list based on IP and look for patterns.
Such as here:

IP address / number of rejected attempts

173.208.81.171 / 10
173.208.81.173 / 13

69.156.243.37 / 16
69.156.243.43 / 15
69.156.243.44 / 13
69.156.243.45 / 14

74.62.154.217 / 1
74.62.154.221 / 6

So what we have here are closely related machines (sharing the same
C-class subnet) trying to connect to my server, but each or all of them
being rejected. Sort of like a tag-team effort.

Between Aug 5 / 1999 and March 17 / 2010, my server rejected a total of
858,574 SMTP connection attempts from external machines on port 25 on
the basis of the external machine's IP address.

These 858,574 attempts came from a total of 14,100 unique /24 IP
subnets.

# of rejected # of percentage of
connection subnets total subnets
attempts

1 7453 52.9%
2 2256 16.0%
3 1384 9.8%
4 452 3.2%
5 298 2.1%
7 482 3.4%
8 157 1.1%
9 138 1.0%
10 93 0.7%

So of the 14,100 subnets that have experienced at least 1 rejected
connection attempt, 7453 of them (52.9%) have experienced exactly 1
rejection, and 93 of them have experienced exactly 10 rejections.

So which subnets are responsible for most of these rejection attempts?

First, I will separate out a few of the /24 subnet's that are
responsible for 15% of the 858k connection attempts:

208.85.55.0 100,927
208.85.53.0 20,674
208.85.48.0 4,578

Those IP address are owned by Silverpop Systems (Atlanta). I don't know
if Silverpop's servers know enough to abort a delivery attempt when they
get an SMTP-rejection error from my server, but the 101k attempts from
subnet 208.85.55.0 happened over the course of 70 days between Sept
11/2008 and Nov 30/2008. The attempts from the other two subnets
happened roughly over the same time period.

The removal of Silverpop's subnets leaves 732k rejected connection
attempts. 25% of those come from 10 subnets, 50% come from 36 subnets,
60% come from 54 subnets, 70% come from 85 subnets, and 80% come from
158 subnets. The top 36 subnets are (in order of rejection attempts):

# of year year
/24 IP rejected and and
subnet connection month month
attempts of first of last
rejection rejection

216.251.36 23218 2002 6 2006 6
209.167.79 20644 2001 4 2002 10
198.212.10 18898 2002 3 2003 2
209.119.0 18707 2001 10 2003 10
209.119.1 18611 2001 6 2003 10
216.109.73 16948 2002 11 2003 8
64.164.98 16464 2002 7 2003 10
151.193.165 15352 2001 8 2003 10
200.207.163 15226 2003 6 2006 2
63.240.103 13663 2006 10 2007 4
205.150.6 12012 2001 7 2002 8
209.5.112 10827 2001 7 2003 1
216.191.74 10747 2004 3 2006 4
87.225.129 9269 2008 2 2009 3
129.33.162 8648 2004 2 2006 2
207.139.216 8616 2004 2 2005 2
65.214.61 8056 2002 12 2008 12
210.243.211 7781 2002 8 2002 8
211.191.202 7465 2004 3 2005 11
204.138.239 7334 2002 4 2002 4
204.225.163 7092 2003 6 2007 4
209.226.175 6966 2000 2 2003 3
65.218.108 6954 2003 3 2003 6
206.47.169 6602 2002 4 2003 8
209.18.70 6581 2006 4 2008 6
208.240.165 6210 2001 4 2001 10
216.24.225 5952 2008 12 2010 3
216.70.8 5919 2002 4 2002 10
66.59.143 5853 2001 4 2003 10
209.5.115 5831 2001 3 2002 11
216.130.10 5811 2002 10 2003 10
151.164.30 5756 2002 1 2003 10
217.98.11 5713 2002 5 2002 5
64.211.230 5281 2001 6 2002 10
195.4.92 5115 2008 1 2010 3
61.248.24 5082 2004 4 2005 11