421 service not available (connection refused, too many connections): ALL servers

[Stanisław Findeisen]

Hi!

I am running a small Postfix server, and for a couple of hours I've been
getting: "host ... refused to talk to me: 421 service not available
(connection refused, too many connections)" for all the outgoing mail,
all destination servers.

What's wrong?

I wasn't even able to subscribe to this mailing list:

Mar 4 00:41:38 k8ux postfix/smtp[2987]: 1462B1F2505:
to=<majo...@postfix.org>, relay=mail.cloud9.net[168.100.1.7]:25,
delay=417, delays=417/0.02/0.06/0, dsn=4.0.0, status=deferred (host
mail.cloud9.net[168.100.1.7] refused to talk to me: 421 service not
available (connection refused, too many connections))

(so I had to use a different account).

--
http://people.eisenbits.com/~stf/
http://www.eisenbits.com/

OpenPGP: E3D9 C030 88F5 D254 434C 6683 17DD 22A0 8A3B 5CC0

[Stanisław Findeisen]

On 2012-03-04 09:20, Stanisław Findeisen wrote:
> Hi!
>
> I am running a small Postfix server, and for a couple of hours I've been
> getting: "host ... refused to talk to me: 421 service not available
> (connection refused, too many connections)" for all the outgoing mail,
> all destination servers.
>
> What's wrong?
>
> I wasn't even able to subscribe to this mailing list:
>
> Mar 4 00:41:38 k8ux postfix/smtp[2987]: 1462B1F2505:
> to=<majo...@postfix.org>, relay=mail.cloud9.net[168.100.1.7]:25,
> delay=417, delays=417/0.02/0.06/0, dsn=4.0.0, status=deferred (host
> mail.cloud9.net[168.100.1.7] refused to talk to me: 421 service not
> available (connection refused, too many connections))
>
> (so I had to use a different account).

This is an up-to-date Debian GNU/Linux (6.0.4 squeeze) with Postfix
version 2.7.1.

[Michael Tokarev]

On 04.03.2012 13:30, Stanisław Findeisen wrote:
> On 2012-03-04 09:20, Stanisław Findeisen wrote:
>> Hi!
>>
>> I am running a small Postfix server, and for a couple of hours I've been
>> getting: "host ... refused to talk to me: 421 service not available
>> (connection refused, too many connections)" for all the outgoing mail,
>> all destination servers.
>>
>> What's wrong?
>>
>> I wasn't even able to subscribe to this mailing list:
>>
>> Mar 4 00:41:38 k8ux postfix/smtp[2987]: 1462B1F2505:
>> to=<majo...@postfix.org>, relay=mail.cloud9.net[168.100.1.7]:25,
>> delay=417, delays=417/0.02/0.06/0, dsn=4.0.0, status=deferred (host
>> mail.cloud9.net[168.100.1.7] refused to talk to me: 421 service not
>> available (connection refused, too many connections))

This smells very much like your outgoing SMTP connections are being
trapped by your ISP and redirected to _their_ SMTP server.

/mjt

[Stanisław Findeisen]

Wha... what a... ??! 8-O

You say that mail.cloud9.net[168.100.1.7] was in reality my ISP's
network node? I.e., they are doing some kind of man in the middle attack
/ IP address spoofing?

Why do you think they should be doing crap like that??!

It just started to work after some 15 hours or so. ALL destination
servers (the whole queue has been sent out).

[/dev/rob0]

On Sun, Mar 04, 2012 at 04:24:44PM +0100, Stanisław Findeisen wrote:
> On 2012-03-04 11:26, Michael Tokarev wrote:
> > On 04.03.2012 13:30, Stanisław Findeisen wrote:
> >> On 2012-03-04 09:20, Stanisław Findeisen wrote:
> >>> I am running a small Postfix server, and for a couple of hours
> >>> I've been getting: "host ... refused to talk to me: 421 service
> >>> not available (connection refused, too many connections)" for
> >>> all the outgoing mail, all destination servers.
> >>>
> >>> What's wrong?
> >>>
> >>> I wasn't even able to subscribe to this mailing list:
> >>>
> >>> Mar 4 00:41:38 k8ux postfix/smtp[2987]: 1462B1F2505:
> >>> to=<majo...@postfix.org>,
> >>> relay=mail.cloud9.net[168.100.1.7]:25, delay=417,
> >>> delays=417/0.02/0.06/0, dsn=4.0.0, status=deferred (host
> >>> mail.cloud9.net[168.100.1.7] refused to talk to me: 421 service
> >>> not available (connection refused, too many connections))
> >
> > This smells very much like your outgoing SMTP connections are
> > being trapped by your ISP and redirected to _their_ SMTP server.
>
> Wha... what a... ??! 8-O
>
> You say that mail.cloud9.net[168.100.1.7] was in reality my ISP's
> network node? I.e., they are doing some kind of man in the middle
> attack / IP address spoofing?

Respectively: no, sort of, and no. mail.cloud9.net is still on its
own IP address, as are the other hosts you tried. It looks like
transparent redirection.

> Why do you think they should be doing crap like that??!

Controlling/limiting outbound abuse in case of spammers on their
networks ... this is my guess. But I don't work for your ISP.

> It just started to work after some 15 hours or so. ALL
> destination servers (the whole queue has been sent out).

Given this additional information, it looks like you triggered an
automated rate limiting system in the ISP firewall.

Review your terms of service and acceptable use policy. Ensure that
you're in compliance. Then, talk to the ISP and ask them about it.
--
http://rob0.nodns4.us/ -- system administration and consulting
Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

[Michael Tokarev]

On 04.03.2012 19:24, Stanisław Findeisen wrote:
> On 2012-03-04 11:26, Michael Tokarev wrote:
>> On 04.03.2012 13:30, Stanisław Findeisen wrote:
>>> On 2012-03-04 09:20, Stanisław Findeisen wrote:

>>>> Hi!

>>>>
>>>> I am running a small Postfix server, and for a couple of hours I've been
>>>> getting: "host ... refused to talk to me: 421 service not available
>>>> (connection refused, too many connections)" for all the outgoing mail,
>>>> all destination servers.
>>>>
>>>> What's wrong?
>>>>
>>>> I wasn't even able to subscribe to this mailing list:
>>>>
>>>> Mar 4 00:41:38 k8ux postfix/smtp[2987]: 1462B1F2505:
>>>> to=<majo...@postfix.org>, relay=mail.cloud9.net[168.100.1.7]:25,
>>>> delay=417, delays=417/0.02/0.06/0, dsn=4.0.0, status=deferred (host
>>>> mail.cloud9.net[168.100.1.7] refused to talk to me: 421 service not
>>>> available (connection refused, too many connections))
>>
>> This smells very much like your outgoing SMTP connections are being
>> trapped by your ISP and redirected to _their_ SMTP server.
>
> Wha... what a... ??! 8-O
>
> You say that mail.cloud9.net[168.100.1.7] was in reality my ISP's
> network node? I.e., they are doing some kind of man in the middle attack
> / IP address spoofing?
>

> Why do you think they should be doing crap like that??!

It is very common practice among consumer ISPs worldwide.
Either redirecting outgoing SMTP (port 25) to their own
mailserver, or merely disabling (dropping) outgoing SMTP
completely. This is in order to stop mass spamming by
bots/trojans installed in huge quantities on client machines.

> It just started to work after some 15 hours or so. ALL destination
> servers (the whole queue has been sent out).

That's probably good.

/mjt

[Stanisław Findeisen]

On 2012-03-04 17:14, /dev/rob0 wrote:

> On Sun, Mar 04, 2012 at 04:24:44PM +0100, Stanisław Findeisen wrote:
>> On 2012-03-04 11:26, Michael Tokarev wrote:
>>> On 04.03.2012 13:30, Stanisław Findeisen wrote:
>>>> On 2012-03-04 09:20, Stanisław Findeisen wrote:
>>>>> I am running a small Postfix server, and for a couple of hours
>>>>> I've been getting: "host ... refused to talk to me: 421 service
>>>>> not available (connection refused, too many connections)" for
>>>>> all the outgoing mail, all destination servers.
>>>>>
>>>>> What's wrong?
>>>>>
>>>>> I wasn't even able to subscribe to this mailing list:
>>>>>
>>>>> Mar 4 00:41:38 k8ux postfix/smtp[2987]: 1462B1F2505:
>>>>> to=<majo...@postfix.org>,
>>>>> relay=mail.cloud9.net[168.100.1.7]:25, delay=417,
>>>>> delays=417/0.02/0.06/0, dsn=4.0.0, status=deferred (host
>>>>> mail.cloud9.net[168.100.1.7] refused to talk to me: 421 service
>>>>> not available (connection refused, too many connections))
>>>
>>> This smells very much like your outgoing SMTP connections are
>>> being trapped by your ISP and redirected to _their_ SMTP server.
>>
>> Wha... what a... ??! 8-O
>>
>> You say that mail.cloud9.net[168.100.1.7] was in reality my ISP's
>> network node? I.e., they are doing some kind of man in the middle
>> attack / IP address spoofing?
>

> Respectively: no, sort of, and no. mail.cloud9.net is still on its
> own IP address, as are the other hosts you tried. It looks like
> transparent redirection.
>

>> Why do you think they should be doing crap like that??!
>

> Controlling/limiting outbound abuse in case of spammers on their
> networks ... this is my guess. But I don't work for your ISP.
>

>> It just started to work after some 15 hours or so. ALL
>> destination servers (the whole queue has been sent out).
>

> Given this additional information, it looks like you triggered an
> automated rate limiting system in the ISP firewall.
>
> Review your terms of service and acceptable use policy. Ensure that
> you're in compliance. Then, talk to the ISP and ask them about it.

It stopped to work again. :-(

My ISP say there are no limits, and that this is a failure of theirs.
They were unable (or not willing) to explain why outgoing TCP traffic to
ports 25 and 587 (they say this one is a problem too) is handled in a
different way than other traffic (say TCP 80).

I know what TCP/IP is, but I don't know too much about routing practice.
What do you think they could be doing with this traffic and why?

/dev/rob0: what is transparent redirection? Did you mean that my Postfix
was tricked and talking to my ISP's SMTP server, instead of
mail.cloud9.net[168.100.1.7] in this case?

[/dev/rob0]

They say this is their failure? Or that is is NOT their failure?

> They were unable (or not willing) to explain why outgoing TCP
> traffic to ports 25 and 587 (they say this one is a problem too)
> is handled in a different way than other traffic (say TCP 80).

They told you the same thing happens with 587?

*boggles*

Maybe the person you talked to does not know, or lied. Whatever.
Doesn't matter. It looks like it's time to find a different ISP.

Another WAG: maybe your ISP's upstream provider got tired of
complaints and implemented this redirection upstream. This would
explain why the ISP would not know.

> I know what TCP/IP is, but I don't know too much about routing
> practice. What do you think they could be doing with this
> traffic and why?
>
> /dev/rob0: what is transparent redirection? Did you mean that my
> Postfix was tricked and talking to my ISP's SMTP server, instead
> of mail.cloud9.net[168.100.1.7] in this case?

"Tricked" is not the right word. "Hijacked" fits better.

In railroad terms, think of a switch in the track. The locomotive
driver cannot help but go where the track takes him, and if someone
sets that switch to send him down the wrong track, there he goes!

In Linux/Netfilter terms, it is the REDIRECT target combined with
some sort of detection:

iptables -N SmtpRedirect
iptables -A SmtpRedirect -p tcp -m multiport --dports 25,587 \
-j REDIRECT --to-ports 2525
iptables -A FORWARD <something to detect abuse> -j SmtpRedirect

And they'd have a simple daemon listening on 2525/tcp which gives a
"421 service not available ..." banner, then drops the connection,
such as you are seeing.

I think tcptraceroute is the tool to check this. While blocked you
could compare port 25 to port 80 on a known host. But the right
answer, for running a mail server, is to use a better ISP. Check on
cheap VPS services for an affordable alternative.

[Rod Dorman]

On Monday, March 5, 2012, 09:53:31, /dev/rob0 wrote:
> ...

> Another WAG: maybe your ISP's upstream provider got tired of
> complaints and implemented this redirection upstream. This would
> explain why the ISP would not know.

I would be horrified is this turned out to be the cause.

Without deep packet inspection there would be no way to distinguish
between SMTP packets originating from the ISP's MTA vs. his MTA.

--
ro...@polylogics.com "The avalanche has already started, it is too
Rod Dorman late for the pebbles to vote." - Ambassador Kosh

[Wietse Venema]

Rod Dorman:

> On Monday, March 5, 2012, 09:53:31, /dev/rob0 wrote:
> > ...
> > Another WAG: maybe your ISP's upstream provider got tired of
> > complaints and implemented this redirection upstream. This would
> > explain why the ISP would not know.
>
> I would be horrified is this turned out to be the cause.
>
> Without deep packet inspection there would be no way to distinguish
> between SMTP packets originating from the ISP's MTA vs. his MTA.

Are you thinking of an ISP with the entire network behind a NAT router?

Wietse

[/dev/rob0]

On Mon, Mar 05, 2012 at 11:21:30AM -0500, Rod Dorman wrote:
> On Monday, March 5, 2012, 09:53:31, /dev/rob0 wrote:
> > ...
> > Another WAG: maybe your ISP's upstream provider got tired of
> > complaints and implemented this redirection upstream. This would
> > explain why the ISP would not know.
>
> I would be horrified is this turned out to be the cause.
>
> Without deep packet inspection there would be no way to
> distinguish between SMTP packets originating from the ISP's
> MTA vs. his MTA.

Sure there is: IP address. To expand on the previous example:

iptables -N SmtpRedirect
iptables -A SmtpRedirect -p tcp -m multiport --dports 25,587 \
-j REDIRECT --to-ports 2525

iptables -A FORWARD -s IPS.MTA.IP.addr -j ACCEPT

iptables -A FORWARD <something to detect abuse> -j SmtpRedirect

Packets from that address would never enter the SmtpRedirect chain.

That said, there seems to be cause for horror in any case. One such
case which I have not yet addressed: the OP could indeed be an
abuser. But even that case is ISP fail, because limiting it is not
the solution; cutting it off entirely would be.

[/dev/rob0]

On Mon, Mar 05, 2012 at 11:06:26AM -0600, I wrote:
> On Mon, Mar 05, 2012 at 11:21:30AM -0500, Rod Dorman wrote:
> > On Monday, March 5, 2012, 09:53:31, /dev/rob0 wrote:
> > > ...
> > > Another WAG: maybe your ISP's upstream provider got tired of
> > > complaints and implemented this redirection upstream. This would
> > > explain why the ISP would not know.
> >
> > I would be horrified is this turned out to be the cause.
> >
> > Without deep packet inspection there would be no way to
> > distinguish between SMTP packets originating from the ISP's
> > MTA vs. his MTA.
>
> Sure there is: IP address. To expand on the previous example:

Oh, but I think I get Rod's point: this requires the upstream
provider to know and make exception for the MTA's IP address; the
aforementioned deep packet inspection being the only sure way to
automate this.

And yes, surely someone at the ISP should have been aware of this.

[Rod Dorman]

On Monday, March 5, 2012, 12:06:09, Wietse Venema wrote:
> Rod Dorman:

>> On Monday, March 5, 2012, 09:53:31, /dev/rob0 wrote:
>> > ...
>> > Another WAG: maybe your ISP's upstream provider got tired of
>> > complaints and implemented this redirection upstream. This would
>> > explain why the ISP would not know.
>>
>> I would be horrified is this turned out to be the cause.
>>
>> Without deep packet inspection there would be no way to distinguish
>> between SMTP packets originating from the ISP's MTA vs. his MTA.
>

> Are you thinking of an ISP with the entire network behind a NAT router?

No (that would be weird :-)

rob0 got my point (which I guess I shudda been clearer in making) being
anything upstream would have to somehow know the set of IP addresses to
allow to pass.

In retrospect it could be maintained manually by the downstream ISP
telling the upstream ISP whenever it changes which servers to allow but
it would be an awfully wimpy ISP to accept those conditions from their
upstream provider.

[Stanisław Findeisen]

They said this *is* their failure, and that they are trying to resolve it.

>> They were unable (or not willing) to explain why outgoing TCP
>> traffic to ports 25 and 587 (they say this one is a problem too)
>> is handled in a different way than other traffic (say TCP 80).
>
> They told you the same thing happens with 587?

Yes.

> *boggles*
>
> Maybe the person you talked to does not know, or lied. Whatever.
> Doesn't matter. It looks like it's time to find a different ISP.
>

> Another WAG: maybe your ISP's upstream provider got tired of
> complaints and implemented this redirection upstream. This would
> explain why the ISP would not know.
>

>> I know what TCP/IP is, but I don't know too much about routing
>> practice. What do you think they could be doing with this
>> traffic and why?
>>
>> /dev/rob0: what is transparent redirection? Did you mean that my
>> Postfix was tricked and talking to my ISP's SMTP server, instead
>> of mail.cloud9.net[168.100.1.7] in this case?
>
> "Tricked" is not the right word. "Hijacked" fits better.
>
> In railroad terms, think of a switch in the track. The locomotive
> driver cannot help but go where the track takes him, and if someone
> sets that switch to send him down the wrong track, there he goes!
>
> In Linux/Netfilter terms, it is the REDIRECT target combined with
> some sort of detection:
>

> iptables -N SmtpRedirect
> iptables -A SmtpRedirect -p tcp -m multiport --dports 25,587 \
> -j REDIRECT --to-ports 2525

> iptables -A FORWARD <something to detect abuse> -j SmtpRedirect
>

> And they'd have a simple daemon listening on 2525/tcp which gives a
> "421 service not available ..." banner, then drops the connection,
> such as you are seeing.

Ok, thanks.

> I think tcptraceroute is the tool to check this. While blocked you
> could compare port 25 to port 80 on a known host. But the right
> answer, for running a mail server, is to use a better ISP. Check on
> cheap VPS services for an affordable alternative.

Ok, thanks. Or a friend SMTP server to see what gets through.

Now it is working again...

My bad suspicion is that they are in the process of installing some
(more or less crappy) mail intercepting facility (i.e. to spy on users)
and that this is probably the government who ordered that. This is
Europe (Poland) but do you think such things are uncommon elsewhere? I
think they are common.

I wonder if you have yet any other guess.

[Peter Blair]

2012/3/5 Stanisław Findeisen <stf.list.po...@eisenbits.com>:

> My bad suspicion is that they are in the process of installing some
> (more or less crappy) mail intercepting facility (i.e. to spy on users)
> and that this is probably the government who ordered that. This is
> Europe (Poland) but do you think such things are uncommon elsewhere? I
> think they are common.

I think that "spy on users" is a bit harsh. Companies have been
selling solutions like this for years:
http://www.mailchannels.com/product/transparent-antispam.html

[Jerry]

On Mon, 05 Mar 2012 20:44:57 +0100
Stanisław Findeisen articulated:

> My bad suspicion is that they are in the process of installing some
> (more or less crappy) mail intercepting facility (i.e. to spy on
> users) and that this is probably the government who ordered that.
> This is Europe (Poland) but do you think such things are uncommon
> elsewhere? I think they are common.

Well, there is always "Echelon"; although its true capabilities has
never been disclosed.

--
Jerry ✌
postfi...@seibercom.net
_____________________________________________________________________
TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail
TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html

[Stan Hoeppner]

On 3/5/2012 1:44 PM, Stanisław Findeisen wrote:

> My bad suspicion is that they are in the process of installing some
> (more or less crappy) mail intercepting facility (i.e. to spy on users)
> and that this is probably the government who ordered that. This is
> Europe (Poland) but do you think such things are uncommon elsewhere? I
> think they are common.

This is paranoia. Unless you are a legitimate target of govt
eavesdropping. But if that were the case, why would they screw with
your smtp traffic in a way that would tip you off? If the govt were
going to do such a thing, surely they'd simply jack a tap into your
provider's routers/switches and have the provider send a duplicate
stream of ALL or your traffic to the tap interface. This is absolutely
trivial to do with most decent switches/routers, technically anyway.
From a legal standpoint it requires a warrant in the US, in Poland I
have no idea. In the US this is done with the infamous 'Carnivore' box.

I would guess the cause of this problem is due to reasons far less
sinister. Have you bothered to inquire with your ISP about the problem?
Or are you afraid they are 'in league' with the govt powers that are
spying on you? Sigh. I'm about to use the word nutbar. Doh, I just
did. :)

--
Stan