Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Increasing Active Queue Requeue Rates
100 views
Skip to first unread message
Travis Dolan
Feb 16, 2016, 11:35:33 AM2/16/16
to
I have 1000's of messages in my active queue destined for a single host. This is noticeably slowing down the delivery of messages to other destinations/users.
I have created a custom smtp transport for the host responsible for all the messages in my queue. I have then tweaked the following for the custom transport
(dc is the name of the custom transport in master.cf)
dc_destination_recipient_limit = 100
dc_destination_concurrency_limit = 100
Obviously this will flood my destination server (I manage this destination) with messages, exactly what I want.
However the only way I can currently get the messages out my active queue fast enough is with the following HACK via CRON (makes me sad)
postsuper -r ALL; postifx flush.
How can I tweak Postfix to essentially do what I have CRONed?
Thanks in advance.
Wietse Venema
Feb 16, 2016, 11:55:08 AM2/16/16
to
Travis Dolan:
deferred, which is exactly the wrong thing to do.
> However the only way I can currently get the messages out my active queue
> fast enough is with the following HACK via CRON (makes me sad)
>
> postsuper -r ALL; postifx flush.
>
> How can I tweak Postfix to essentially do what I have CRONed?
Don't flood the destination server. It causes deliveries to be
deferred, which is exactly the wrong thing to do.
Wietse
> I have 1000's of messages in my active queue destined for a single host.
> This is noticeably slowing down the delivery of messages to other
> destinations/users.
>
> I have created a custom smtp transport for the host responsible for all the
> messages in my queue. I have then tweaked the following for the custom
> transport
>
> (dc is the name of the custom transport in master.cf)
>
> dc_destination_recipient_limit = 100
> dc_destination_concurrency_limit = 100
>
> Obviously this will flood my destination server (I manage this destination)
> with messages, exactly what I want.
Don't flood the destination server. It causes deliveries to be
> This is noticeably slowing down the delivery of messages to other
> destinations/users.
>
> I have created a custom smtp transport for the host responsible for all the
> messages in my queue. I have then tweaked the following for the custom
> transport
>
> (dc is the name of the custom transport in master.cf)
>
> dc_destination_recipient_limit = 100
> dc_destination_concurrency_limit = 100
>
> Obviously this will flood my destination server (I manage this destination)
> with messages, exactly what I want.
deferred, which is exactly the wrong thing to do.
> However the only way I can currently get the messages out my active queue
> fast enough is with the following HACK via CRON (makes me sad)
>
> postsuper -r ALL; postifx flush.
>
> How can I tweak Postfix to essentially do what I have CRONed?
deferred, which is exactly the wrong thing to do.
Wietse
Wietse Venema
Feb 16, 2016, 1:10:13 PM2/16/16
to
Travis Dolan:
> The destination server has no issue with the amount of messages we send it.
> We have custom a python script accepting port 25, and simply takes
> everything and places in RabbitMQ. The incoming queue for this destination
> is highly active, which is really the root issue. I need to move those
> messages out as fast as they come in.
Have you considered slowing down the injection rate, so that it
better matches the output rate and does not congets the queue.
/etc/postfix/main.cf:
check_client_access hash:/etc/postfix/access
/etc/postfix/access:
1.2.3.4 sleep 2
Requires Postfix 2.3 and later.
Wietse
> The destination server has no issue with the amount of messages we send it.
> We have custom a python script accepting port 25, and simply takes
> everything and places in RabbitMQ. The incoming queue for this destination
> is highly active, which is really the root issue. I need to move those
> messages out as fast as they come in.
Have you considered slowing down the injection rate, so that it
better matches the output rate and does not congets the queue.
/etc/postfix/main.cf:
check_client_access hash:/etc/postfix/access
/etc/postfix/access:
1.2.3.4 sleep 2
Requires Postfix 2.3 and later.
Wietse
Wietse Venema
Feb 16, 2016, 1:14:20 PM2/16/16
to
Wietse Venema:
/etc/postfix/main.cf:
smtpd_client_restrictions = check_client_access hash:/etc/postfix/access
/etc/postfix/access:
1.2.3.4 sleep 2
This assumes the default setting of smtpd_delay_reject=no.
Wietse
> Travis Dolan:
> > The destination server has no issue with the amount of messages we send it.
> > We have custom a python script accepting port 25, and simply takes
> > everything and places in RabbitMQ. The incoming queue for this destination
> > is highly active, which is really the root issue. I need to move those
> > messages out as fast as they come in.
>
> Have you considered slowing down the injection rate, so that it
> better matches the output rate and does not congets the queue.
Corrected example:
> > The destination server has no issue with the amount of messages we send it.
> > We have custom a python script accepting port 25, and simply takes
> > everything and places in RabbitMQ. The incoming queue for this destination
> > is highly active, which is really the root issue. I need to move those
> > messages out as fast as they come in.
>
> Have you considered slowing down the injection rate, so that it
> better matches the output rate and does not congets the queue.
/etc/postfix/main.cf:
smtpd_client_restrictions = check_client_access hash:/etc/postfix/access
/etc/postfix/access:
1.2.3.4 sleep 2
This assumes the default setting of smtpd_delay_reject=no.
Wietse
Noel Jones
Feb 16, 2016, 1:41:34 PM2/16/16
to
On 2/16/2016 10:35 AM, Travis Dolan wrote:
> I have 1000's of messages in my active queue destined for a single
> host. This is noticeably slowing down the delivery of messages to
> other destinations/users.
>
> I have created a custom smtp transport for the host responsible for
> all the messages in my queue. I have then tweaked the following for
> the custom transport
>
> (dc is the name of the custom transport in master.cf <http://master.cf>)
> I have 1000's of messages in my active queue destined for a single
> host. This is noticeably slowing down the delivery of messages to
> other destinations/users.
>
> I have created a custom smtp transport for the host responsible for
> all the messages in my queue. I have then tweaked the following for
> the custom transport
>
>
> dc_destination_recipient_limit = 100
> dc_destination_concurrency_limit = 100
>
> Obviously this will flood my destination server (I manage this
> destination) with messages, exactly what I want.
>
> dc_destination_recipient_limit = 100
> dc_destination_concurrency_limit = 100
>
> Obviously this will flood my destination server (I manage this
> destination) with messages, exactly what I want.
>
> However the only way I can currently get the messages out my active
> queue fast enough is with the following HACK via CRON (makes me sad)
>
> postsuper -r ALL; postifx flush.
>
> How can I tweak Postfix to essentially do what I have CRONed?
>
> Thanks in advance.
> queue fast enough is with the following HACK via CRON (makes me sad)
>
> postsuper -r ALL; postifx flush.
>
> How can I tweak Postfix to essentially do what I have CRONed?
>
You may need to slow down the input if the receiving server can't
keep up.
Flushing while clogged will likely make things worse. Clear the
obstruction, then flush.
http://www.postfix.org/TUNING_README.html
http://www.postfix.org/QSHAPE_README.html
and seems particularly relevant:
http://www.postfix.org/QSHAPE_README.html#backlog
-- Noel Jones
Travis Dolan
Feb 16, 2016, 1:44:44 PM2/16/16
to
Unfortunately there are many MTAs/MUAs that send to my host, which then sends to the custom smtp transport destination. Building a hash map for all potential hosts would be problematic.
I need to allow as many messages as possible IN, and then relay those messages as fast as possible to my custom smtp transport destination.
There must be a way to requeue messages in my active queue faster.
Thanks in advance.
Travis Dolan
Feb 16, 2016, 1:48:31 PM2/16/16
to
That is the issue, the receiving server can receive more than what I am sending to it. When I run "postsuper -r ALL; postix flush" I can watch all the queued messages get delivered within a couple minutes no problem. Netstat shows the allotted amount of connections defined in the custom smtp transport open, and messages fly off my server. The bottle neck does not seem to be on the receiver, it seems to be the active queue on the sending host.
Wietse Venema
Feb 16, 2016, 1:56:09 PM2/16/16
to
Travis Dolan:
deferred (and you don't know what to look for) or your mail queue
sits on a file server, and the client and server clocks aren't
properly maintained.
Wietse
> That is the issue, the receiving server can receive more than what I am
> sending to it. When I run "postsuper -r ALL; postix flush" I can watch all
> the queued messages get delivered within a couple minutes no problem.
> Netstat shows the allotted amount of connections defined in the custom smtp
> transport open, and messages fly off my server. The bottle neck does not
> seem to be on the receiver, it seems to be the active queue on the sending
> host.
Postfix will try to deliver all mail immediately, unless mail is
> sending to it. When I run "postsuper -r ALL; postix flush" I can watch all
> the queued messages get delivered within a couple minutes no problem.
> Netstat shows the allotted amount of connections defined in the custom smtp
> transport open, and messages fly off my server. The bottle neck does not
> seem to be on the receiver, it seems to be the active queue on the sending
> host.
deferred (and you don't know what to look for) or your mail queue
sits on a file server, and the client and server clocks aren't
properly maintained.
Wietse
Travis Dolan
Feb 16, 2016, 2:13:30 PM2/16/16
to
I am working from the following data
qshape deferred = TOTAL 10
qshape -s deferred = TOTAL 9
qshape active = TOTAL 2819
qshape -s active = TOTAL 1469
If I am reading these results incorrectly please let me know.
No mail sits on any form of a file server, all queues are local to sending server.
You mentioned clocks, that could be a potential issue. The sending server in this case is UTC, and the receiving server is PST. How would this manifest itself?
Thanks in advance.
Viktor Dukhovni
Feb 16, 2016, 2:33:49 PM2/16/16
to
> On Feb 16, 2016, at 2:13 PM, Travis Dolan <travis...@gmail.com> wrote:
>
> I am working from the following data
>
> qshape deferred = TOTAL 10
> qshape -s deferred = TOTAL 9
>
> qshape active = TOTAL 2819
> qshape -s active = TOTAL 1469
>
> If I am reading these results incorrectly please let me know.
>
> No mail sits on any form of a file server, all queues are local to sending server.
>
> You mentioned clocks, that could be a potential issue. The sending server in this case is UTC, and the receiving server is PST. How would this manifest itself?
Throughput = concurrency/latency. If the active queue
occupancy for this destination is high, either the latency
is high or the concurrency is low.
Running "postsuper -r", and setting excessive concurrency
limits is likely to make things worse.
--
Viktor.
Travis Dolan
Feb 16, 2016, 2:59:15 PM2/16/16
to
Requested config dump
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
always_bcc = smtp@custom_smtp_transport@example.com
append_dot_mydomain = no
biff = no
broken_sasl_auth_clients = yes
config_directory = /etc/postfix
content_filter = smtp-amavis:[127.0.0.1]:10024
dc_destination_concurrency_limit = 100
dc_destination_recipient_limit = 100
dovecot_destination_recipient_limit = 1
inet_interfaces = all
inet_protocols = all
mailbox_size_limit = 0
message_size_limit = 26214400
mydestination = mail.example.com localhost.example.com, localhost, ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal
mydomain = example.com
myhostname = mail.example.com
myorigin = example.com
readme_directory = no
recipient_delimiter = +
relayhost =
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
smtpd_client_connection_count_limit = 100
smtpd_end_of_data_restrictions = check_policy_service inet:127.0.0.1:10031
smtpd_recipient_restrictions = check_client_access hash:/etc/postfix/policyd_override, check_policy_service inet:127.0.0.1:10031, permit_mynetworks, permit_sasl_authenticated,reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unlisted_recipient,reject_unauth_destination, reject_non_fqdn_helo_hostname, reject_invalid_helo_hostname, reject_rbl_client b.barracudacentral.org, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net, reject_rbl_client dnsbl.sorbs.net=127.0.0.2, reject_rbl_client dnsbl.sorbs.net=127.0.0.3, reject_rbl_client dnsbl.sorbs.net=127.0.0.4, reject_rbl_client dnsbl.sorbs.net=127.0.0.5, reject_rbl_client dnsbl.sorbs.net=127.0.0.7, reject_rbl_client dnsbl.sorbs.net=127.0.0.9,reject_rbl_client dnsbl.sorbs.net=127.0.0.11, reject_rbl_client dnsbl.sorbs.net=127.0.0.12, warn_if_reject reject_rhsbl_sender dsn.rfc-ignorant.org, warn_if_reject reject_rhsbl_sender abuse.rfc-ignorant.org, warn_if_reject reject_rhsbl_sender whois.rfc-ignorant.org, warn_if_reject reject_rhsbl_sender bogusmx.rfc-ignorant.org, warn_if_reject reject_rhsbl_sender postmaster.rfc-ignorant.org, permit
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
smtpd_sasl_auth_enable = yes
smtpd_sasl_exceptions_networks = $mynetworks
smtpd_sasl_path = private/auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_tls_CAfile = /etc/ssl/certs/example.com-gd_bundle-g2-g1.crt
smtpd_tls_cert_file = /etc/ssl/certs/example.com.crt.pem
smtpd_tls_key_file = /etc/ssl/private/example.com.key.pem
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes
transport_maps = hash:/etc/postfix/transport
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_gid_maps = static:8
virtual_mailbox_base = /vmstore/vmail/
virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql_virtual_domains_maps.cf
virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_minimum_uid = 7777
virtual_transport = dovecot
virtual_uid_maps = static:7777
Thanks in advance.
Wietse Venema
Feb 16, 2016, 3:25:05 PM2/16/16
to
Travis Dolan:
command "postconf -P" (Postfix 2.11 and later), otherwise use eye
balls...
Perhaps all you're doing with "postsuper -r" is pushing mail around
the content filter.
> dc_destination_concurrency_limit = 100
> dc_destination_recipient_limit = 100
Good, you are running a modern Postfix that shows per-transport
settings.
Wietse
> Requested config dump
>
> alias_database = hash:/etc/aliases
> alias_maps = hash:/etc/aliases
> always_bcc = smtp@custom_smtp_transport@example.com
> append_dot_mydomain = no
> biff = no
> broken_sasl_auth_clients = yes
> config_directory = /etc/postfix
> content_filter = smtp-amavis:[127.0.0.1]:10024
Do you have any content_filter= settings in master.cf? Try the
>
> alias_database = hash:/etc/aliases
> alias_maps = hash:/etc/aliases
> always_bcc = smtp@custom_smtp_transport@example.com
> append_dot_mydomain = no
> biff = no
> broken_sasl_auth_clients = yes
> config_directory = /etc/postfix
> content_filter = smtp-amavis:[127.0.0.1]:10024
command "postconf -P" (Postfix 2.11 and later), otherwise use eye
balls...
Perhaps all you're doing with "postsuper -r" is pushing mail around
the content filter.
> dc_destination_concurrency_limit = 100
> dc_destination_recipient_limit = 100
settings.
Wietse
Travis Dolan
Feb 16, 2016, 4:14:42 PM2/16/16
to
I do have content filtering in my master.cf details below
pickup/fifo/content_filter =
pickup/fifo/receive_override_options = no_header_body_checks
relay/unix/fallback_relay =
smtp-amavis/unix/disable_dns_lookups = yes
smtp-amavis/unix/max_use = 20
smtp-amavis/unix/smtp_data_done_timeout = 1200
smtp-amavis/unix/smtp_send_xforward_command = yes
127.0.0.1:10025/inet/receive_override_options = no_unknown_recipient_checks,no_header_body_checks,no_address_mappings
127.0.0.1:10025/inet/smtpd_client_restrictions = permit_mynetworks,reject
127.0.0.1:10025/inet/smtpd_data_restrictions = reject_unauth_pipelining
127.0.0.1:10025/inet/smtpd_recipient_restrictions = permit_mynetworks,reject
Thanks in advance.
Viktor Dukhovni
Feb 16, 2016, 4:17:40 PM2/16/16
to
On Tue, Feb 16, 2016 at 07:58:45PM +0000, Travis Dolan wrote:
> content_filter = smtp-amavis:[127.0.0.1]:10024
There's your problem. Your content filter is too slow, it is
> content_filter = smtp-amavis:[127.0.0.1]:10024
probably doing remote DNS lookups.
The mail in the active queue is pre-filter, not post-filter.
--
Viktor.
Travis Dolan
Feb 16, 2016, 4:50:51 PM2/16/16
to
Indeed it is the problem! I have increased the amount of amavis daemons on this host however I did not add the corresponding lines to master.cf.
I am now tweaking this as needed, and the active queues are not longer an issue.
Thanks for the all the help Postfix Jedi's.
Wietse Venema
Feb 16, 2016, 6:51:53 PM2/16/16
to
Wietse Venema:
Wietse
> Perhaps all you're doing with "postsuper -r" is pushing mail around
> the content filter.
Yup. that was exactly what was happening.
> the content filter.
Wietse
0 new messages