rate limiting

[Casey Connor]

Hello -- we're sending from a postfix instance as part of our stress-testing infrastructure.

We need to limit the rate of outbound messages per second: sometimes to a particular slow rate, sometimes to a medium speed, sometimes "as fast as possible".

I found no easy way to do that; have I missed something?

Our current method is to use a mail gun to send to postfix over localhost which tops up the queues periodically, and then use default_destination_concurrency_limit as a way to limit the number of concurrent outgoing clients (we're sending to only one domain for these tests).

This works well enough, but it's coarse, and I believe it's sensitive to changing network latencies.

We've considered an intermediate process between the mail gun and the local postfix instance to limit based on bandwidth, but I'm wondering if there is a smarter way we can manage this via postfix configs?

Thanks for any ideas!
-c

[Wietse Venema]

See:
http://www.postfix.org/postconf.5.html#default_transport_rate_delay
http://www.postfix.org/postconf.5.html#default_destination_rate_delay

The names are similar but things work differently.

Wietse

[Casey Connor]

Thank you -- will it accept decimal seconds?

We are sending on the order of 50-200+ messages per second in this
stress test, so the delay between messages could be smaller than .005
seconds.

[Viktor Dukhovni]

> On Apr 7, 2016, at 11:32 PM, Casey Connor <gmanep...@caseyconnor.org> wrote:
>
> Thank you -- will it accept decimal seconds?

No.

> We are sending on the order of 50-200+ messages per second in this stress test, so the delay between messages could be smaller than .005 seconds.

Inject 50-200 messages per second into Postfix, and mail will not
be delivered faster than it comes in (unless a backlog develops
and clears when downstream conditions improve).

--
Viktor.

[Casey Connor]

>> We are sending on the order of 50-200+ messages per second in this stress test, so the delay between messages could be smaller than .005 seconds.
> Inject 50-200 messages per second into Postfix, and mail will not
> be delivered faster than it comes in (unless a backlog develops
> and clears when downstream conditions improve).

We don't have a tool to inject at a particular rate, but maybe we can
find one or make one. I was hoping we could handle it in postfix, but it
sounds like a no.

Please consider decimal-second rate limiting as a feature request. :-)

Thanks,
-c

[Phil Stracchino]

It's a pretty niche use case. In a traffic-test environment such as
yours, it would seem the better approach is to simply have your test
tool control the rate at which it injects messages into Postfix. The
message stream coming *out* of Postfix may occasionally not be 100%
uniform, but hey ... *neither is the real world*.


--
Phil Stracchino
Babylon Communications
ph...@caerllewys.net
ph...@co.ordinate.org
Landline: 603.293.8485

[Viktor Dukhovni]

> On Apr 7, 2016, at 11:57 PM, Casey Connor <gmanep...@caseyconnor.org> wrote:
>
> Please consider decimal-second rate limiting as a feature request. :-)

It is very unlikely to happen. To deliver much more than one message per
second, given typical message transaction latencies, delivery concurrency
is required, and simple message spacing strategies no longer work.

Achieving target rates is much more difficult than enforcing minimum
spacing. Injecting at a target rate is easier, just create requests
at that rate and have some reasonable number of threads (10 to 20 or
so) submit messages in parallel via the Postfix server.

--
Viktor.

[Curtis Villamizar]

In message <5707263D...@caseyconnor.org>

Casey Connor writes:
>
> Thank you -- will it accept decimal seconds?
>

> We are sending on the order of 50-200+ messages per second in this
> stress test, so the delay between messages could be smaller than .005
> seconds.

If you are trying to simulate a very busy mailserver, then you should
be concerned about connections to it from multiple hosts per second
most sending one or a few messages and a realistic test would have to
have a nominal TCP RTT in the range of a few to a few 10s of msec
since the speed of light is limited and geographic delays come into
play. I've been involved in testing and some simulation of this type
but on routers and various switchy-thingies rather than mailservers.

Curtis

[Casey Connor]

Thanks, Curtis. We have taken all that in to consideration. I'll spare
you the long story, but we are testing somewhat specific things. :-) -c