Delay before initial 220 greeting

[Alex]

Hi,

I have a postfix-2.8.7 system with fedora15 and amavisd-new-2.6.6.
Lately I have been experiencing significant delays before receiving
the initial postfix 220 greeting from all remote hosts. I've verified
DNS is working properly and can resolve hosts properly. It seems to be
related to system load and doesn't occur as frequently when the server
is less busy.

I've experimented with various parameters but am really unsure where
to start. I was unsure if it was somehow related to
smtpd_client_connection_count_limit or some other limit parameter. My
guess is it's related to not spawning enough smtp processes to accept
the incoming connections?

This issue occurs even when amavisd is otherwise idle, with a least a
few processes idle. This server is a dual quad-core Xeon with 16GB.

I've also enabled debug_peer_list and it really doesn't reveal
anything helpful. I can see the initial postscreen connection, then a
large delay, then immediately the match_hostname, match_hostaddr
statements, and the 220 greeting:

Feb 22 13:14:35 mail01 postfix/postscreen[25319]: CONNECT from
[64.XXX.YYY.2]:48144
Feb 22 13:14:35 mail01 postfix/postscreen[25319]: WHITELISTED
[64.XXX.YYY.2]:48144
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: connect from
rogue.example.com[64.XXX.YYY.2]
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostname:
rogue.example.com ~? 127.0.0.0/8
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostaddr:
64.XXX.YYY.2 ~? 127.0.0.0/8
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostname:
rogue.example.com ~? 192.168.1.0/24
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostaddr:
64.XXX.YYY.2 ~? 192.168.1.0/24
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostname:
rogue.example.com ~? 192.168.6.0/24
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostaddr:
64.XXX.YYY.2 ~? 192.168.6.0/24
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostname:
rogue.example.com ~? 68.XXX.YYY.40/29
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostaddr:
64.XXX.YYY.2 ~? 68.XXX.YYY.40/29
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostname:
rogue.example.com ~? 64.XXX.YYY.0/27
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: match_hostaddr:
64.XXX.YYY.2 ~? 64.XXX.YYY.0/27
Feb 22 13:16:31 mail01 postfix/smtpd[3184]: >
rogue.example.com[64.XXX.YYY.2]: 220 mail01.example.com ESMTP Postfix

I've included my postfinger output below. I also have another issue
that I'm unsure is related. I've had to create a client_checks_special
and sender_checks_special maps, in addition to my normal sender and
client maps, because something in my smtpd_recipient_restrictions was
rejecting my exceptions in those maps before reaching my other
sender_checks and client_checks maps.

postfinger - postfix configuration on Wed Feb 22 13:03:09 EST 2012
version: 1.30

--System Parameters--
mail_version = 2.8.7
hostname = mail01.example.com
uname = Linux mail01.example.com 2.6.42.3-2.fc15.x86_64 #1 SMP Thu Feb
9 01:42:06 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

--Packaging information--
looks like this postfix comes from RPM package: postfix-2.8.7-1.fc15.x86_64

--main.cf non-default parameters--
alias_database = hash:/etc/postfix/aliases
alias_maps = hash:/etc/postfix/aliases
allow_mail_to_files = alias,forward
always_bcc = bcc-user
biff = no
body_checks = regexp:/etc/postfix/body_checks.pcre
content_filter = smtp-amavis:[127.0.0.1]:10024
debug_peer_list = 64.XXX.YYY.0/27
default_process_limit = 200
delay_warning_time = 4h
disable_vrfy_command = yes
header_checks =
pcre:/etc/postfix/header_checks.pcre pcre:/etc/postfix/header_checks-jimsun.pcre
initial_destination_concurrency = 20
mailbox_command = /usr/bin/procmail
mailbox_size_limit = 2000000000
manpage_directory = /usr/share/man
maximal_queue_lifetime = 2d
message_size_limit = 13312000
mime_header_checks = pcre:/etc/postfix/mime_header_checks
mydestination = $myhostname, localhost.$mydomain
mynetworks = 127.0.0.0/8, 192.168.1.0/24, 192.168.6.0/24,
68.XXX.YYY.40/29, 64.XXX.YYY.0/27, 206.XXX.YYY.45/32,
167.XXX.YYY.192/26, 74.XXX.YYY.160/27, 67.XXX.YYY.224/28,
160.XXX.YYY.3
postscreen_access_list = permit_mynetworks,
cidr:/etc/postfix/postscreen_access.cidr
postscreen_blacklist_action = enforce
postscreen_dnsbl_action = enforce
postscreen_dnsbl_sites = zen.spamhaus.org*2 bl.spamcop.net*1
b.barracudacentral.org*1
postscreen_dnsbl_threshold = 2
postscreen_greet_action = enforce
rbl_reply_maps = ${stress?hash:/etc/postfix/rbl_reply_maps}
readme_directory = /usr/share/doc/postfix-2.8.7/README_FILES
relay_domains = $mydestination, $transport_maps, example.com,
cs.example.com, hotel.example.com, example.com
sample_directory = /usr/share/doc/postfix-2.8.7/samples
smtpd_recipient_restrictions =
reject_non_fqdn_recipient, check_client_access
hash:/etc/postfix/client_checks_special, check_sender_access
hash:/etc/postfix/sender_checks_special, reject_non_fqdn_sender, reject_unlisted_recipient, permit_mynetworks, reject_unauth_destination, reject_unknown_sender_domain, reject_unknown_recipient_domain, check_helo_access
pcre:/etc/postfix/helo_checks.pcre, reject_invalid_helo_hostname, check_client_access
hash:/etc/postfix/client_checks, check_sender_access
hash:/etc/postfix/sender_checks, check_recipient_access
pcre:/etc/postfix/relay_recips_segtravel, check_recipient_access
pcre:/etc/postfix/relay_recips_access, check_recipient_access
pcre:/etc/postfix/property_recip_map, check_recipient_access
pcre:/etc/postfix/recipient_checks, check_recipient_access
pcre:/etc/postfix/bwi_relay_recip_checks, check_recipient_access
pcre:/etc/postfix/relay_recips_ecartis, reject_rbl_client
zen.spamhaus.org, reject_rbl_client psbl.surriel.com, permit
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = $myhostname, mail01.example.com
smtpd_sasl_path = private/auth
smtpd_sasl_type = dovecot
smtpd_sender_restrictions = reject_sender_login_mismatch
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /etc/pki/dovecot/certs/dovecot.pem
smtpd_tls_key_file = /etc/pki/dovecot/private/dovecot.pem
smtpd_tls_received_header = yes
smtpd_tls_security_level = may
smtpd_tls_session_cache_database =
btree:/var/lib/postfix/smtpd_tls_session_cache
smtp_tls_CAfile = /etc/pki/tls/cacert.pem
smtp_use_tls = yes
transport_maps = hash:/etc/postfix/transport
virtual_alias_maps = hash:/etc/postfix/virtual,
hash:/etc/postfix/virtual-segtravel

--master.cf--
submission inet n - n - - smtpd
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o milter_macro_daemon_name=ORIGINATING
pickup fifo n - n 60 1 pickup
-o content_filter=
cleanup unix n - n - 0 cleanup
qmgr fifo n - n 300 1 qmgr
tlsmgr unix - - n 1000? 1 tlsmgr
rewrite unix - - n - - trivial-rewrite
bounce unix - - n - 0 bounce
defer unix - - n - 0 bounce
trace unix - - n - 0 bounce
verify unix - - n - 1 verify
flush unix n - n 1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - n - - smtp
relay unix - - n - - smtp
-o smtp_fallback_relay=
showq unix n - n - - showq
error unix - - n - - error
retry unix - - n - - error
discard unix - - n - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - n - - lmtp
anvil unix - - n - 1 anvil
scache unix - - n - 1 scache
smtp-amavis unix - - n - - smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=40
127.0.0.1:10025 inet n - n - 18 smtpd
-o content_filter=
-o smtpd_delay_reject=no
-o smtpd_client_restrictions=permit_mynetworks,reject
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=reject_unauth_pipelining
-o smtpd_end_of_data_restrictions=
-o smtpd_restriction_classes=
-o mynetworks=127.0.0.0/8
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
-o local_header_rewrite_clients=
bwitmp unix - - n - - smtp
-o fallback_relay=[206.XXX.YYY.20]
cstmp unix - - n - - smtp
-o fallback_relay=[206.XXX.YYY.20]
smtp inet n - n - 1 postscreen
smtpd pass - - n - - smtpd
-o receive_override_options=no_address_mappings
dnsblog unix - - n - 0 dnsblog
tlsproxy unix - - n - 0 tlsproxy

-- end of postfinger output --

Thanks,
Alex

[Wietse Venema]

Alex:

> Hi,
>
> I have a postfix-2.8.7 system with fedora15 and amavisd-new-2.6.6.
> Lately I have been experiencing significant delays before receiving
> the initial postfix 220 greeting from all remote hosts. I've verified

What is the output from:

grep warning: /var/log/maillog

Then you may want to read this webpage:

http://www.postfix.org/STRESS_README.html

Wietse

[Stan Hoeppner]

You've already got a default_process_limit of 200 which should be more
than plenty for a very large inbound stream, assuming everything else is
in order.

In addition to the recommendations in the STRESS_README WRT decreasing
time spent per connection by each smtpd, I'd add that if you're not
already running a local caching DNS resolver on your Postfix host, you
should implement such. It could shave up to a second off the smtpd
processing time for each connection, which is significant for a loaded
server with 200 smtpds experiencing this problem.

On 2/22/2012 12:23 PM, Alex wrote:

> postscreen_blacklist_action = enforce
> postscreen_dnsbl_action = enforce
> postscreen_dnsbl_sites = zen.spamhaus.org*2 bl.spamcop.net*1
> b.barracudacentral.org*1
> postscreen_dnsbl_threshold = 2

> reject_rbl_client zen.spamhaus.org, reject_rbl_client psbl.surriel.com

You're rejecting based on zen twice, once in Postscreen and once in
smtpd. May as well remove the smtpd entry. And you may as well move
psbl into postscreen. If it's going to kill a connection you may as
well do it before tying up an smtpd process, which is currently a problem.

You're currently doing a combined 5 dnsbl lookups per connection plus
two for the forward and reverse names, for a total of 7 per connection.
This may likely be part of your current problem, especially if using a
high latency external resolver (ISP for instance).

Four external dnsbl queries per connect may be a bit excessive for a
host under this kind of load. In addition to a local caching resolver,
consider using a local rbldnsd instance for serving the spamcop and psbl
zones, and zen if you're a paying customer. This will cut total
external dns queries down to 4 or 3 from your current 7. Along with a
local resolver, this should pretty much eliminate dns latency as a
factor in tying up smtpd processes.

If your current average dns latency is 10ms you're looking at 0.7
seconds of time in smtpd just for dns lookups. If it's 30ms it's 2.1
seconds. That's peanuts on a lightly loaded MX, but more than
significant on a host with 200 smtpds that can't seem to keep up with
the load currently.

--
Stan

[Alex]

Hi,

>>> I have a postfix-2.8.7 system with fedora15 and amavisd-new-2.6.6.
>>> Lately I have been experiencing significant delays before receiving
>>> the initial postfix 220 greeting from all remote hosts. I've verified
>>
>> What is the output from:
>>
>>     grep warning: /var/log/maillog
>>
>> Then you may want to read this webpage:
>>
>>     http://www.postfix.org/STRESS_README.html
>
> You've already got a default_process_limit of 200 which should be more
> than plenty for a very large inbound stream, assuming everything else is
> in order.

It turned out increasing default_process_limit to 300 solved the
problem, although after reading your other comments, I understand
there are other areas to be improved.

I do already have a local bind resolver and will investigate rbldnsd.
I'm using munin to provide reports for bind, and it shows an average
of 60 to 80 queries per second, with peaks above 120, but doesn't show
latency. How can I report this information?

I've removed the zen and psbl queries from smtpd and moved psbl to postscreen.

How can I monitor the number of smtpd processes currently being
utilized in real-time? Even though it's set to 300, "ps ax" shows
significantly fewer than that. It would be nice to have a log of that
information over time.

Thanks so much for your help,
Alex

[Wietse Venema]

Alex:

> I've removed the zen and psbl queries from smtpd and moved psbl to postscreen.
>
> How can I monitor the number of smtpd processes currently being
> utilized in real-time? Even though it's set to 300, "ps ax" shows

Now that postscreen is taking most of the bullets, the number of
smtpd processes will be much reduced. That was the primary goal of
the implementing postscreen.

Wietse

[Stan Hoeppner]

On 2/23/2012 1:50 PM, Alex wrote:
> Hi,
>
>>>> I have a postfix-2.8.7 system with fedora15 and amavisd-new-2.6.6.
>>>> Lately I have been experiencing significant delays before receiving
>>>> the initial postfix 220 greeting from all remote hosts. I've verified
>>>
>>> What is the output from:
>>>
>>> grep warning: /var/log/maillog
>>>
>>> Then you may want to read this webpage:
>>>
>>> http://www.postfix.org/STRESS_README.html
>>
>> You've already got a default_process_limit of 200 which should be more
>> than plenty for a very large inbound stream, assuming everything else is
>> in order.
>
> It turned out increasing default_process_limit to 300 solved the
> problem, although after reading your other comments, I understand
> there are other areas to be improved.

Increasing the smtpd count is the quick/dirty way to fix such a problem,
but doing thsi can create other problems. Which is why I had you look
at other optimizations first.

> I do already have a local bind resolver and will investigate rbldnsd.
> I'm using munin to provide reports for bind, and it shows an average
> of 60 to 80 queries per second, with peaks above 120, but doesn't show
> latency. How can I report this information?

I don't know that you can get this in real time, but you certainly don't
need to. The important thing is that you have a local resolver. This
alone guarantees you will decrease cached lookup latency to 1ms or so.
Simply having a good local resolver in working order eliminates any
latency issues. There is no need to measure it.

> I've removed the zen and psbl queries from smtpd and moved psbl to postscreen.

As Wietse confirmed, this alone will decrease the number of smtpd
processes used, and it's exactly why he created postscreen. Read the docs.

> How can I monitor the number of smtpd processes currently being
> utilized in real-time?

I don't know of any off the shelf method for this, nor any API method
for querying Postfix for such info. smtpd processes will sit for
max_idle seconds before terminating, waiting on additional incoming
connections. So you'll always have a combination of active and idle
smtpds. However, knowing which is which in real time doesn't really
benefit you. There is plenty of other data available to let you know if
you've got the system configured and running properly.

> Even though it's set to 300, "ps ax" shows

> significantly fewer than that. It would be nice to have a log of that
> information over time.

It's called default_process_limit. Note the last word. 300 is the peak
you allowed. The default peak is 100. When everything else is tuned
properly, you should be able to average 100 msgs/sec or more with the
defaults. If you're seeing 60-80 dns queries/sec that would seem to
indicate your connect rate is around 10-14 msgs/sec, given 6 dns lookups
per connection. 10-14 connects/sec is very low connect rate. You
shouldn't even need 100 smtpds for that load. Maybe my math is off
here. If not, you've got more optimizing to do. Now, if every remote
host connecting to you is slow, say you're in Singapore and all of your
connections are in the US, it's going to take more smtpd time for each
connection simply due to RTT delays, especially if satellite links are
involved (unlikely today due to oceanic fiber, but possible).

Now that you done some optimization, kick default_process_limit back
down to 200 and see it that works. If so, keep backing it off by 25
each time until you start to see the slowdown again. Then bump it back
up by 10 and leave it.

Next I'd look at smtp-amavis and see if it's keeping up with demand. If
it can't service requests fast enough postfix will spawn more smtpds to
handle the incoming connections and then they'll sit and wait on amavis.
Optimizing a complex setup as yours is a balancing act because you have
so many layers depending on each other.

> Thanks so much for your help,

Sure thing. If you know what your peak daily connection rate is, and
can share that, it would be helpful.

--
Stan

[Alex]

Hi,

> Now that you done some optimization, kick default_process_limit back
> down to 200 and see it that works.  If so, keep backing it off by 25
> each time until you start to see the slowdown again.  Then bump it back
> up by 10 and leave it.
>
> Next I'd look at smtp-amavis and see if it's keeping up with demand.  If
> it can't service requests fast enough postfix will spawn more smtpds to
> handle the incoming connections and then they'll sit and wait on amavis.
>  Optimizing a complex setup as yours is a balancing act because you have
> so many layers depending on each other.
>
>> Thanks so much for your help,
>
> Sure thing.  If you know what your peak daily connection rate is, and
> can share that, it would be helpful.

I'm still adjusting it a little bit at a time, and will follow up next
week with that info.

I had thought it was related, but another issue I'm trying to figure
out is how to prevent a single remote server from sending thousands of
messages at a time, filling the queue, and causing significant
delivery delays for all mail.

I thought either greylisting or a few iptables rules to throttle the
connection rate, but I haven't been able to figure this out.

Thanks again for your help,
Alex

[Noel Jones]

On 2/28/2012 8:37 PM, Alex wrote:
> I had thought it was related, but another issue I'm trying to figure
> out is how to prevent a single remote server from sending thousands of
> messages at a time, filling the queue, and causing significant
> delivery delays for all mail.
>
> I thought either greylisting or a few iptables rules to throttle the
> connection rate, but I haven't been able to figure this out.

Greylisting won't help with connections from a legit mail server.

The proper solution is firewall rules to limit connections per
client, and/or QOS rules to limit the bandwidth per client.

As a last choice, you can use the various postfix client limits
provided by anvil(8).
http://www.postfix.org/anvil.8.html
Warning: anvil is intended to limit abusive clients only. Using
anvil limits to slow down legit mail may cause extreme delivery delays.




-- Noel Jones

[Stan Hoeppner]

On 2/28/2012 8:37 PM, Alex wrote:

> Hi,
>
>> Now that you done some optimization, kick default_process_limit back
>> down to 200 and see it that works. If so, keep backing it off by 25
>> each time until you start to see the slowdown again. Then bump it back
>> up by 10 and leave it.
>>
>> Next I'd look at smtp-amavis and see if it's keeping up with demand. If
>> it can't service requests fast enough postfix will spawn more smtpds to
>> handle the incoming connections and then they'll sit and wait on amavis.
>> Optimizing a complex setup as yours is a balancing act because you have
>> so many layers depending on each other.
>>
>>> Thanks so much for your help,
>>
>> Sure thing. If you know what your peak daily connection rate is, and
>> can share that, it would be helpful.
>
> I'm still adjusting it a little bit at a time, and will follow up next
> week with that info.
>

> I had thought it was related, but another issue I'm trying to figure
> out is how to prevent a single remote server from sending thousands of
> messages at a time, filling the queue, and causing significant
> delivery delays for all mail.

Is it safe to assume these thousands of messages are spam, and not legit
mail? If so, simply block the IP address(es) in a cidr table:

smtpd_recipient_restrictions
permit_mynetworks
reject_unauth_destination
check_client_access cidr:/etc/postfix/blacklist.cidr
...

/etc/postfix/blacklist.cidr
#single IP
10.10.10.10/32 REJECT high rate spammer
#class C network
10.10.10.0/24 REJECT snowshoe spammer

If it's legit mail, anvil typically takes care of rate throttling, IIRC.
Need more info. What version of Postfix are you using again?

> I thought either greylisting or a few iptables rules to throttle the
> connection rate, but I haven't been able to figure this out.

Use the above method to block known spammer IPs and other SMTP abusers.
This 'punishes' the abusers and doesn't bother other senders as
greylisting delays can.

> Thanks again for your help,

Sure thing.

--
Stan

[Alex]

Hi,

>> I had thought it was related, but another issue I'm trying to figure
>> out is how to prevent a single remote server from sending thousands of
>> messages at a time, filling the queue, and causing significant
>> delivery delays for all mail.
>
> Is it safe to assume these thousands of messages are spam, and not legit
> mail?  If so, simply block the IP address(es) in a cidr table:
>
> smtpd_recipient_restrictions
>        permit_mynetworks
>        reject_unauth_destination
>        check_client_access cidr:/etc/postfix/blacklist.cidr
>        ...
>
> /etc/postfix/blacklist.cidr
> #single IP
> 10.10.10.10/32          REJECT high rate spammer
> #class C network
> 10.10.10.0/24           REJECT snowshoe spammer
>
> If it's legit mail, anvil typically takes care of rate throttling, IIRC.
>  Need more info.  What version of Postfix are you using again?

Yes, it is for legit mail, such as that from constantcontact and other
bulk mailers that overwhelm my servers and upset my users who want
their more important mail.

I'm using postfix-2.8.7 on fedora15.

Perhaps someone hows a proper iptables QoS or other throttling ruleset
that I could use?

I also appreciate other input on using anvil, but that also appears to
have at least some unwanted side-effects that may just upset my users
in other ways.

Thanks again,
Alex

[Brian Evans - Postfix List]

On 3/1/2012 11:49 AM, Alex wrote:
> Hi,
>
>>> I had thought it was related, but another issue I'm trying to figure
>>> out is how to prevent a single remote server from sending thousands of
>>> messages at a time, filling the queue, and causing significant
>>> delivery delays for all mail.
>> Is it safe to assume these thousands of messages are spam, and not legit
>> mail? If so, simply block the IP address(es) in a cidr table:
>>
>> smtpd_recipient_restrictions
>> permit_mynetworks
>> reject_unauth_destination
>> check_client_access cidr:/etc/postfix/blacklist.cidr
>> ...
>>
>> /etc/postfix/blacklist.cidr
>> #single IP
>> 10.10.10.10/32 REJECT high rate spammer
>> #class C network
>> 10.10.10.0/24 REJECT snowshoe spammer
>>
>> If it's legit mail, anvil typically takes care of rate throttling, IIRC.
>> Need more info. What version of Postfix are you using again?
> Yes, it is for legit mail, such as that from constantcontact and other
> bulk mailers that overwhelm my servers and upset my users who want
> their more important mail.

Perhaps you would like to rate limit certain IPs?
If so, you could apply the above map and, instead of rejecting, perform
a check_policy_service action along with something like Postfwd.

A policy server could also be global depending on your needs.

Brian

[Stan Hoeppner]

I just fired an email to another list where a few Constant Contact folks
have been known to participate. A member (non CC) already responded
with their outbound range. Active hosts list within the range after my sig.

208.75.123.0/24

I also offered to put CC in contact with Alex if this rate issue might
be better addressed on their end.

--
Stan



208.75.123.1 coi001.confirmedcc.com
208.75.123.2 coi002.confirmedcc.com
208.75.123.3 coi003.confirmedcc.com
208.75.123.103 coi103.confirmedcc.com
208.75.123.130 ccm22.constantcontact.com
208.75.123.131 ccm23.constantcontact.com
208.75.123.132 ccm24.constantcontact.com
208.75.123.133 ccm25.constantcontact.com
208.75.123.134 ccm134.constantcontact.com
208.75.123.135 ccm135.constantcontact.com
208.75.123.161 ccm26.constantcontact.com
208.75.123.162 ccm27.constantcontact.com
208.75.123.163 ccm38.constantcontact.com
208.75.123.164 ccm39.constantcontact.com
208.75.123.165 ccm165.constantcontact.com
208.75.123.166 ccm166.constantcontact.com
208.75.123.167 ccm167.constantcontact.com
208.75.123.168 ccm168.constantcontact.com
208.75.123.169 ccm169.constantcontact.com
208.75.123.170 ccm170.constantcontact.com
208.75.123.193 ccm33.constantcontact.com
208.75.123.194 ccm34.constantcontact.com
208.75.123.195 ccm35.constantcontact.com
208.75.123.196 ccm36.constantcontact.com
208.75.123.197 ccm197.constantcontact.com
208.75.123.198 ccm198.constantcontact.com
208.75.123.200 ccm200.constantcontact.com
208.75.123.201 ccm201.constantcontact.com
208.75.123.202 ccm202.constantcontact.com
208.75.123.225 ccm29.constantcontact.com
208.75.123.226 ccm30.constantcontact.com
208.75.123.227 ccm31.constantcontact.com
208.75.123.228 ccm32.constantcontact.com
208.75.123.245 mail245.nutshellmail.com
208.75.123.250 ccm37.constantcontact.com