destination_rate_delay and deferred queue

[sashk]

Hello,

 

I have CentOS 6 system running 2.6.6 version of postfix. This is low volume mail server with roughly 400

messages being sent through it during the day to my google apps domain. I've ran it for awhile without any issues,

but recently  I've started seeing issues with sending emails to my google apps. I've configured spf records to allow

mail from that server, as well as specifically added server's IP to white list and skip spam check lists, which worked

flawlessly for more than two years. Recently, they started refusing  to accept my messages with following message. 

They would ban access by IP only, as emails from other sources would continue to be delivered. 

 

 status=deferred (host ALT1.ASPMX.L.GOOGLE.com[64.233.186.26] said:

  450-4.2.1 The user you are trying to contact is receiving mail at a rate that

  450-4.2.1 prevents additional messages from being delivered. Please resend your

  450-4.2.1 message at a later time. If the user is able to receive mail at that

  450-4.2.1 time, your message will be delivered. For more information, please

  450-4.2.1 visit 450 4.2.1 https://support.google.com/mail/answer/6592 emailidhere.87 - gsmtp (in reply to RCPT TO command))

 

 I have following configuration lines in main.cf in order not to go over receiving limits of google.  

 

     transport_maps = hash:/etc/postfix/transport

     default_destination_rate_delay = 12

     default_destination_concurrency_limit = 1

     default_destination_recipient_limit = 5

 

     slow_initial_destination_concurrency = 1

     slow_destination_concurrency_limit = 1

     slow_destination_recipient_limit = 5

     slow_destination_rate_delay = 12s

 

and content of transport file:

 

     gmail.com slow:

     xxxxx.com slow:

     xxxxx.im slow:

     xxxxxx.com slow:

 

 

According to google, they temporary ban anyone who attempts to connect to their servers more frequently than once a second,

aside from their time sync issues, making five seconds on my end look like less than a second on their,  I've noticed that all fresh

emails received by the server will be throttled to be sent every 12 seconds, as I expect; but all deferred emails would attempt

delivery on it's own schedule, without paying attention to the slow delivery rule. 

 

I am wondering if someone could help me to identify what I've missed and how to re-configure my mail server to either:

 

 - make sure that deferred queue is not attempting to deliver messages more frequently than I specified in slow configuration; or 

 - deliver more than one email to google per connection;

 

 

Thank you in advance for your help.

 

-sashk

[Wietse Venema]

There is only one scheduler, and it applies the same rule for new
mail and delayed mail.

However, rate delays have no effect if you restart the queue manager
with "postfix reload".

Wietse

[sashk]

Wietse,

There is only one scheduler, and it applies the same rule for new
mail and delayed mail.

 

 

Thank you for the quick response. But I'm lost here with my postfix logs. Despite, I said not to make outbound connections more frequently than 12 seconds, postfix attempted two connections within 3 seconds from each other. As you can see, there are two different IDs from google (replaced with id0.78 and id1.84). These two log messages have different format, but I couldn't figure out what each means. 

 

 

Jun 22 17:30:27 stmp1 smtp-slow/smtp[4389]: 23F891F008F: host ASPMX.L.GOOGLE.com[173.194.207.27] said: 450-4.2.1 The user you are trying to contact is receiving mail at a rate that 450-4.2.1 prevents additional messages from being delivered. Please resend your 450-4.2.1 message at a later time. If the user is able to receive mail at that 450-4.2.1 time, your message will be delivered. For more information, please 450-4.2.1 visit 450 4.2.1 https://support.google.com/mail/answer/6592 id0.78 - gsmtp (in reply to RCPT TO command)

Jun 21 17:30:30 smtp1 smtp-slow/smtp[4389]: 23F891F008F: to=<t...@domain.com>, relay=ALT1.ASPMX.L.GOOGLE.com[64.233.186.27]:25, delay=4.2, delays=1.1/0.03/2.7/0.36, dsn=4.2.1, status=deferred (host ALT1.ASPMX.L.GOOGLE.com[64.233.186.27] said: 450-4.2.1 The user you are trying to contact is receiving mail at a rate that 450-4.2.1 prevents additional messages from being delivered. Please resend your 450-4.2.1 message at a later time. If the user is able to receive mail at that 450-4.2.1 time, your message will be delivered. For more information, please 450-4.2.1 visit 450 4.2.1 https://support.google.com/mail/answer/6592 id1.84 - gsmtp (in reply to RCPT TO command))

Somehow Google sees those connections within one second and blocks me for some time. This happens on first attempt to deliver messages, right after it received them from the smtp connection.

Could you please point me into right direction?

 

 

Thank you.

 

-sashk

[Wietse Venema]

Show actual evidence: two complete logfile records that are too close
together.

Wietse

[Wietse Venema]

Wietse Venema:

>
> Show actual evidence: two complete logfile records that are too close
> together.

And also, show evidence that you haven't made the mistake of setting
xxx_destination_recipient_limit=1 (where xxx is the mail delivery
transport that talks to google etc.).

As documented:

o With a corresponding per-destination recipient limit equal to 1,
the rate delay specifies the time between deliveries to the same
recipient. Different recipients are delivered in parallel, sub-
ject to the process limits specified in master.cf.

Wietse

[sashk]

Wietse,

 

 

 Show actual evidence: two complete logfile records that are too close
 together.

 

My original email showed those two records.  Aren't they what you're asking for?

 

And also, show evidence that you haven't made the mistake of setting
xxx_destination_recipient_limit=1 (where xxx is the mail delivery
transport that talks to google etc.).

master.cf:

slow unix - - n - 1 smtp
-o syslog_name=smtp-slow

main.cf:

transport_maps = hash:/etc/postfix/transport
default_destination_rate_delay = 12
default_destination_concurrency_limit = 1
default_destination_recipient_limit = 5
slow_initial_destination_concurrency = 1
slow_destination_concurrency_limit = 1
slow_destination_recipient_limit = 5
slow_destination_rate_delay = 12s

transport:

gmail.com slow:
domain.com slow:

Thank you.

 

-sashk

 

[Viktor Dukhovni]

On Wed, Jun 24, 2015 at 10:10:28PM -0400, sashk wrote:

> Wietse,
>
> Show actual evidence: two complete logfile records that are too
> close
> together.
>
> My original email showed those two records. Aren't they what you're
> asking for?

It did not show back-to-back delivery of two separate messages.
Rather it showed a single message that tried two MX hosts in
succession, because the first MX host returned a 4XX response.

The reported error was an excess arrival rate of mail into a single
user's mailbox, (joe-job, too many bounces for mail from that user,
...). It was not a violation of a rate limit from your IP address
to Gmail.

Gmail's rate limit on mail to individual users is a major PITA for
real MTAs. Ultimately all the mail arrives anyway, but giant queues
build up on the sending MTA, needlessly delaying mail for other
users.

--
Viktor.

[sashk]

Viktor,

It did not show back-to-back delivery of two separate messages.
Rather it showed a single message that tried two MX hosts in
succession, because the first MX host returned a 4XX response.

Is there a way for me to instruct postfix not to attempt delivery to second MX host, if first attempt failed?  

According to the Google Apps support, it's exactly what causes the temporarily ban. 

 

The reported error was an excess arrival rate of mail into a single
user's mailbox, (joe-job, too many bounces for mail from that user,
...). It was not a violation of a rate limit from your IP address
to Gmail.

Well, google says otherwise. They say, there was two attempts to connect within the second, and they're banning me for that. 

 

Thank you,

-sashk

 

 

[Viktor Dukhovni]

On Thu, Jun 25, 2015 at 08:11:45AM -0400, sashk wrote:

> Is there a way for me to instruct postfix not to attempt delivery to
> second MX host, if first attempt failed?

That would be wrong.

> According to the Google Apps support, it's exactly what causes the
> temporarily ban.

Nonsense. Trying the next MX host is what every MTA should do when
the first MX host returns a 4XX code:

https://tools.ietf.org/html/rfc5321#section-5.1
https://tools.ietf.org/html/rfc5321#page-70

When the lookup succeeds, the mapping can result in a list
of alternative delivery addresses rather than a single
address, because of multiple MX records, multihoming, or
both. To provide reliable mail transmission, the SMTP client
MUST be able to try (and retry) each of the relevant addresses
in this list in order, until a delivery attempt succeeds.
However, there MAY also be a configurable limit on the number
of alternate addresses that can be tried. In any case, the
SMTP client SHOULD try at least two addresses.

> Well, google says otherwise. They say, there was two attempts to
> connect within the second, and they're banning me for that.

What evidence do you have for that? The error messages are about
message arrival rate for the same user. I am quite familiar with
these, and they are not caused by a small number of closely spaced
messages. Rather this symptom happens when a user quickly receives
a large number of messages due to a flood of mail coming in.

--
Viktor.

[sashk]

Viktor,

What evidence do you have for that? The error messages are about
message arrival rate for the same user. I am quite familiar with
these, and they are not caused by a small number of closely spaced
messages. Rather this symptom happens when a user quickly receives
a large number of messages due to a flood of mail coming in.

Here is what I've got so far:

 

1. They will block messages only from my server. Other servers will able to deliver messages to my mailbox while my server reports an error - so this is not related to the message flood, something else is happening. 

2. They block not one recipient, but everyone on my domain. 

3. Error message is and for last couple weeks was the same: "450-4.2.1  The user you are trying to contact is receiving mail at a rate that prevents additional messages from being delivered..."

4. They can lock on first message they receiving from my server after hours of no communications between. 

5. Their response is:

As per the logs so far It your server is making more than one connection in less than a second and this is strange as It does not seem to be configured this way.

In this scenario even tho the queue is empty, making a few connections at once will cause a lock out, even if you send 3 messages for the day if they are sent in less than a second we will lock the connection.

 

Thank you, 

-sashk

[Wietse Venema]

sashk:
> between.?</div><div>5. Their response is:</div><blockquote><div>As

> per the logs so far It your server is making more than one connection
> in less than a second and this is strange as It does not seem to

> be configured this way.</div><div>In this scenario even tho the

If they can't handle RFC 5321 which says try an alternate MX after
a 4xx reply (see below), then you can turn off RFC 5321 retries:

/etc/postfix/main.cf:
smtp_mx_address_limit=1
smtp_mx_session_limit=1

(in addition to setting the _rate_delay so that multiple messages
won't trigger their threshold).

I work for Google now, but I have nothing to with their mail service
(which also has problems with their handling of IPv6 connections).

Here is what RFC 5321 has to say about this:

When the [MX] lookup succeeds, the mapping can result in a list of

alternative delivery addresses rather than a single address, because
of multiple MX records, multihoming, or both. To provide reliable
mail transmission, the SMTP client MUST be able to try (and retry)
each of the relevant addresses in this list in order, until a
delivery attempt succeeds. However, there MAY also be a configurable
limit on the number of alternate addresses that can be tried. In any
case, the SMTP client SHOULD try at least two addresses.

Wietse