postfix queue tuning
Yaoxing
I'm looking for some help of postfix server configuration. hope this is
the right place to ask.
I have a mail server running iRedMail (which is based on postfix). It
sends mails to our subscribers every 4s. I think this doesn't seem to be
a very heavy load. however, there're likely 140,000 mails congesting
after several days' running. So I tried qshape to analyse the queue, and
found that almost all mails are congesting in incoming queue, while
active queue reaches it's limit of 20,000 mails. Mails rarely went to
deferred queue. I think this means something reaches its limitation. So
I checked the following features, and this is the result:
1. CPU is not busy at all
2. Almost 3GB memory left
3. 3.2MB/s disk IO write, 0.01MB/s read.
4. Less than 20 postfix process (while limitation is explicitly set to 100)
There seems to be nothing wrong (or did I miss anything?). Can anybody
provide some more information on locating the problem?
Any help is appreciated.
--
Regards,
Yaoxing
John Adams
Does your postfix check whether the recipients' exist on your side?
Stan Hoeppner
> 3. 3.2MB/s disk IO write, 0.01MB/s read.
MB/s throughput isn't usually a factor, but IOPS definitely can be.
What's in the iostat tps column for the device your mail queues reside on?
If your mail queue resides on a single mechanical disk spindle you may
be running into an IOPS limit. A single 7.2k rpm SATA disk can only
sink approximately 150 seeks/sec (IOPS) under optimal conditions. You
didn't mention the size of your subscriber base, but you mentioned that
you queue mail to those subscriuber addresses every 4 seconds. If using
a single SATA disk as described above, you will run out of IOPS with
~600 subscribers. If all of your queues and your log files, and your
entire *nix system, resides on such a single disk, you'll run out of
IOPS well below 600 subscribers.
After investigating, if this is indeed the cause of your queue
performance problem, the solution is to put the Postfix queues on a
separate storage device, optimally for performance and cost reasons, and
SSD.
Using a dedicated device for the queue files, a single 15k SAS disk can
sink approximately 300 seeks/sec (IOPS) which would allow for
approximately 1200 subscribers with a 4 sec interval between mailings.
A RAID 10 set of 4 such drives will allow double that amount, or 2400
subscribers.
The cost of a good quality high performance SSD for Postfix queues is
much less than 15k rpm SAS drives and far less than RAID 10, and will
give you tens of thousands of IOPS, allowing for many thousands of
subscribers, limited by SSD capacity. If your problem is limited IOPS,
I suggest putting your queues on one of these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16820227610
If your total queue size is exceeding 40GB before mail can be flushed to
the destinations, OCZ offers both 60GB and 90GB models for less than
$200 USD. IOPS performance for all 3 sizes of OCZ devices is similar,
around 4k Random Write (Aligned): 45,000-50,000 IOPS.
An SSD such as this is by far the best price/performance solution to
such a queue IOPS problem, if that is indeed your queue performance issue.
--
Stan
Ramprasad
> Hi all,
> I'm looking for some help of postfix server configuration. hope this is
> the right place to ask.
> I have a mail server running iRedMail (which is based on postfix). It
> sends mails to our subscribers every 4s. I think this doesn't seem to be
> a very heavy load. however, there're likely 140,000 mails congesting
> after several days' running. So I tried qshape to analyse the queue, and
> found that almost all mails are congesting in incoming queue, while
> active queue reaches it's limit of 20,000 mails. Mails rarely went to
> deferred queue. I think this means something reaches its limitation. So
> I checked the following features, and this is the result:
> 1. CPU is not busy at all
> 2. Almost 3GB memory left
> There seems to be nothing wrong (or did I miss anything?). Can anybody
> provide some more information on locating the problem?
> Any help is appreciated.
>
If you have a lot of RAM you could increase the active queue size.
Where are these mails getting delivered to ? Is they are going to the
same destination then you may require to increase the
default_destination_concurrency_limit
If you are choking on I/O iostat can indicate that
or else go to your spool directory and do a
du -s * .. see if some directory is taking really too long to du
Thanks
Ram
Yaoxing
Thank you very much for your adequate explanation. I made some comments
below.
Regards,
Yaoxing
2010/12/23 16:06, Stan Hoeppner:
> Yaoxing put forth on 12/22/2010 9:59 PM:
>
>> 3. 3.2MB/s disk IO write, 0.01MB/s read.
> MB/s throughput isn't usually a factor, but IOPS definitely can be.
> What's in the iostat tps column for the device your mail queues reside on?
Is this what you're talking about?
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 148.63 27.55 6550.60 523033469 124353201092
sda1 0.00 0.00 0.00 2524 116
sda2 148.63 27.55 6550.59 523027626 124353101816
sda3 0.00 0.00 0.01 2895 99160
> If your mail queue resides on a single mechanical disk spindle you may
> be running into an IOPS limit. A single 7.2k rpm SATA disk can only
> sink approximately 150 seeks/sec (IOPS) under optimal conditions. You
> didn't mention the size of your subscriber base, but you mentioned that
> you queue mail to those subscriuber addresses every 4 seconds. If using
> a single SATA disk as described above, you will run out of IOPS with
> ~600 subscribers. If all of your queues and your log files, and your
> entire *nix system, resides on such a single disk, you'll run out of
> IOPS well below 600 subscribers.
I don't fully understand how this 600 subscribers is calculated. maybe I
should describe my situation like this:
It's a weekly promotion mail. no matter how many subscribers are there,
I send 1 email every 4sec.
Despite all I said above, do you mean if I have 600 subscribers, and I
send to all of them 1 mail every 4 sec (which is 600 mails/sec), I'll
run out of IOPS?
> After investigating, if this is indeed the cause of your queue
> performance problem, the solution is to put the Postfix queues on a
> separate storage device, optimally for performance and cost reasons, and
> SSD.
>
> Using a dedicated device for the queue files, a single 15k SAS disk can
> sink approximately 300 seeks/sec (IOPS) which would allow for
> approximately 1200 subscribers with a 4 sec interval between mailings.
> A RAID 10 set of 4 such drives will allow double that amount, or 2400
> subscribers.
>
> The cost of a good quality high performance SSD for Postfix queues is
> much less than 15k rpm SAS drives and far less than RAID 10, and will
> give you tens of thousands of IOPS, allowing for many thousands of
> subscribers, limited by SSD capacity. If your problem is limited IOPS,
> I suggest putting your queues on one of these:
>
> http://www.newegg.com/Product/Product.aspx?Item=N82E16820227610
>
> If your total queue size is exceeding 40GB before mail can be flushed to
> the destinations, OCZ offers both 60GB and 90GB models for less than
> $200 USD. IOPS performance for all 3 sizes of OCZ devices is similar,
> around 4k Random Write (Aligned): 45,000-50,000 IOPS.
>
> An SSD such as this is by far the best price/performance solution to
> such a queue IOPS problem, if that is indeed your queue performance issue.
I read more documents of postfix today, and according to my
understanding the active queue is in memory. Thus if I'm running out of
IOPS, I shouldn't have got a full active queue (tell me if I'm wrong).
But actually I do. So it's more likely a limitation of network or maybe
quantity of process. but either seems to be all right. That's why I'm
confused.
Yaoxing
I do have some more spare memory, but I'm afraid it doesn't resolve my
problem.
Let's say, my active queue is filled with 20,000 mails, but mails are
not going out but remains in memory. In this case if I increase active
queue size, I just put more mails in memory, they still don't go out.
What do you think?
I checked the directories, it's the incoming directory which is very
big. if I use
find incomfing/ | wc -l
it tells me there are likely 130,000 mails in this directory.
Regards,
Yaoxing
Ralf Hildebrandt
> Hi all,
> I'm looking for some help of postfix server configuration. hope this is
> the right place to ask.
> I have a mail server running iRedMail (which is based on postfix). It
> sends mails to our subscribers every 4s. I think this doesn't seem to be
> a very heavy load. however, there're likely 140,000 mails congesting
> after several days' running. So I tried qshape to analyse the queue, and
> found that almost all mails are congesting in incoming queue, while
> active queue reaches it's limit of 20,000 mails.
Please show the qshape output
--
Ralf Hildebrandt
Geschäftsbereich IT | Abteilung Netzwerk
Charité - Universitätsmedizin Berlin
Campus Benjamin Franklin
Hindenburgdamm 30 | D-12203 Berlin
Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
ralf.hil...@charite.de | http://www.charite.de
Yaoxing
screen of
qshape active
T 5 10 20 40 80 160 320 640
1280 1280+
TOTAL 1000 0 0 0 0 0 0 0
0 0 1000
gmail.com 254 0 0 0 0 0 0 0
0 0 254
hotmail.com 168 0 0 0 0 0 0 0
0 0 168
yahoo.com 70 0 0 0 0 0 0 0
0 0 70
comcast.net 14 0 0 0 0 0 0 0
0 0 14
msn.com 12 0 0 0 0 0 0 0
0 0 12
aol.com 11 0 0 0 0 0 0 0
0 0 11
sbcglobal.net 10 0 0 0 0 0 0 0
0 0 10
yahoo.com.br 9 0 0 0 0 0 0 0
0 0 9
hotmail.co.uk 9 0 0 0 0 0 0 0
0 0 9
web.de 8 0 0 0 0 0 0 0
0 0 8
mail.ru 8 0 0 0 0 0 0 0
0 0 8
free.fr 8 0 0 0 0 0 0 0
0 0 8
yahoo.es 7 0 0 0 0 0 0 0
0 0 7
hotmail.fr 7 0 0 0 0 0 0 0
0 0 7
gmx.de 6 0 0 0 0 0 0 0
0 0 6
juno.com 6 0 0 0 0 0 0 0
0 0 6
cox.net 5 0 0 0 0 0 0 0
0 0 5
live.com 5 0 0 0 0 0 0 0
0 0 5
sympatico.ca 5 0 0 0 0 0 0 0
0 0 5
and for qshape incoming:
T 5 10 20 40 80 160 320 640
1280 1280+
TOTAL 1000 9 7 33 47 79 181 393
251 0 0
gmail.com 221 1 2 6 9 18 45 100
40 0 0
hotmail.com 153 1 1 4 9 15 32 63
28 0 0
e.dealextreme.com 152 1 0 4 4 2 17 35
89 0 0
yahoo.com 88 0 0 3 7 3 14 37
24 0 0
msn.com 12 0 0 0 0 0 1 6
5 0 0
yahoo.com.br 12 0 0 1 2 1 1 6
1 0 0
aol.com 9 0 0 0 0 1 1 5
2 0 0
mail.ru 8 0 0 0 0 1 2 3
2 0 0
free.fr 7 0 0 0 0 2 1 4
0 0 0
comcast.net 5 0 0 0 0 0 2 2
1 0 0
web.de 4 0 0 0 0 1 0 2
1 0 0
yahoo.es 4 0 0 0 1 0 1 2
0 0 0
hotmail.fr 4 0 0 0 0 0 0 2
2 0 0
terra.com.br 4 0 0 0 0 1 2 1
0 0 0
sympatico.ca 4 0 0 0 0 1 0 2
1 0 0
ntlworld.com 4 0 0 0 0 1 0 3
0 0 0
bellsouth.net 4 0 0 0 0 0 2 1
1 0 0
gmx.de 3 0 0 0 0 1 0 2
0 0 0
ono.com 3 0 0 0 0 0 0 3
0 0 0
Regards,
Yaoxing
Jeroen Geilman
Post the output from postconf -n, and a relevant section of the mail logs.
--
J.
Wietse Venema
Then, you are sending all mail through the same relay host. Why
are you sending mass mail through a relay host?
Wietse
Yaoxing Zhang
No special reasons, just trying to find a simple way to program. If this is the reason why it's so slow, how about using smtp instead? does it resolve the problem?
Wietse Venema
> No special reasons, just trying to find a simple way to program. If this is
> the reason why it's so slow, how about using smtp instead? does it resolve
> the problem?
Please go back to the default settings, and report if the
system is still behaving slower than expected.
It is silly to make configuration changes, and then complain that
Postfix is not working well without sharing your changes (the
"postconf -n" output that you have been asked for).
Wietse
Stan Hoeppner
> Is this what you're talking about?
Yes.
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 148.63 27.55 6550.60 523033469 124353201092
> sda1 0.00 0.00 0.00 2524 116
> sda2 148.63 27.55 6550.59 523027626 124353101816
> sda3 0.00 0.00 0.01 2895 99160
This shows 148 transactions per second to device sda. The average max
IOPS for any vendor's 7.2k rpm drive is ~150, a 15k drive hits ~300.
Your spool appears to be on partition sda2. You are hitting the maximum
IOPS capability of that drive if it is a 7.2k drive. If it is a 7.2k
drive, it would seem you need more spindles or an SSD.
I arrived at the 600 subscriber limit by multiplying 150 IOPS by 4
seconds, which is the interval you stated for successive mailings. The
result is 600 subscribers maximum before exceeding the IOPS capability
of that drive.
--
Stan
Stan Hoeppner
It would appear my recommendation may merely mask (for a short time) a
symptom of the root problem, but not solve he problem itself. Wietse's
insight seems to show that your IOPS issue is a result of the mail not
going out fast enough, and thus the disk is getting hammered by the
queue piling up.
Wieste is right. You don't need faster storage, but more mails
delivered per second so they aren't piling up in the queue. You need a
faster outbound path, or multiple of the same.
Apologies if my lesser insight/experience clogged this thread.
--
Stan
Yaoxing
It's OK, you helped a lot already. And your figure is a very good
reference for me. Thank you very much.
@Wietse
I'll try not to use relay host to check the speed.
And sorry if I didn't express myself clearly, I don't mean to complain
anything. Just don't know what's wrong and what to do because I'm a
green hand. I pasted the changed parameters below. removed some lines
because I think it's not related and don't want to make the list too
long. (it's still very long I know ;)
debug_peer_level = 2
default_process_limit = 100
enable_original_recipient = no
inet_interfaces = all
inet_protocols = all
maximal_backoff_time = 4000s
maximal_queue_lifetime = 1d
message_size_limit = 15728640
minimal_backoff_time = 300s
mydestination = $myhostname, localhost, localhost.localdomain,
localhost.$myhostname
mydomain = abc.com
myhostname = e.abc.com
mynetworks = 127.0.0.0/8, 10.12.157.130/32, 10.12.157.132/32,
127.0.0.1/32, 10.9.76.200/32, 10.9.76.196/32, 10.9.76.194/32,
10.16.117.68/32
mynetworks_style = subnet
myorigin = e.abc.com
newaliases_path = /usr/bin/newaliases.postfix
proxy_read_maps = $canonical_maps $lmtp_generic_maps
$local_recipient_maps $mydestination $mynetworks $recipient_bcc_maps
$recipient_canonical_maps $relay_domains $relay_recipient_maps
$relocated_maps $sender_bcc_maps $sender_canonical_maps
$smtp_generic_maps $smtpd_sender_login_maps $transport_maps
$virtual_alias_domains $virtual_alias_maps $virtual_mailbox_domains
$virtual_mailbox_maps
queue_run_delay = 300s
recipient_bcc_maps =
proxy:mysql:/etc/postfix/mysql_recipient_bcc_maps_domain.cf,
proxy:mysql:/etc/postfix/mysql_recipient_bcc_maps_user.cf
relay_domains = $mydestination,
proxy:mysql:/etc/postfix/mysql_relay_domains.cf
relay_recipient_maps =
proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
sender_bcc_maps =
proxy:mysql:/etc/postfix/mysql_sender_bcc_maps_domain.cf,
proxy:mysql:/etc/postfix/mysql_sender_bcc_maps_user.cf
smtp_connect_timeout = 15
smtp_helo_timeout = 15
smtpd_data_restrictions = reject_unauth_pipelining
smtpd_enforce_tls = no
smtpd_error_sleep_time = 0
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks,permit_sasl_authenticated,
check_helo_access pcre:/etc/postfix/helo_access.pcre
smtpd_recipient_restrictions = reject_unknown_sender_domain,
reject_unknown_recipient_domain, reject_non_fqdn_sender,
reject_non_fqdn_recipient, reject_unlisted_recipient, permit_mynetworks,
permit_sasl_authenticated, reject_unauth_destination,
reject_non_fqdn_helo_hostname, reject_invalid_helo_hostname,
check_policy_service inet:127.0.0.1:10031
smtpd_reject_unlisted_recipient = yes
smtpd_reject_unlisted_sender = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = no
smtpd_sasl_local_domain =
smtpd_sasl_path = dovecot-auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_login_maps =
proxy:mysql:/etc/postfix/mysql_sender_login_maps.cf
smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated
smtpd_tls_loglevel = 0
smtpd_tls_security_level = may
tls_random_source = dev:/dev/urandom
transport_maps = proxy:mysql:/etc/postfix/mysql_transport_maps_user.cf,
proxy:mysql:/etc/postfix/mysql_transport_maps_domain.cf
unknown_local_recipient_reject_code = 550
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_gid_maps = static:500
virtual_mailbox_base = /var/vmail/vmail01
virtual_mailbox_domains =
proxy:mysql:/etc/postfix/mysql_virtual_mailbox_domains.cf
virtual_mailbox_maps =
proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_minimum_uid = 500
virtual_transport = dovecot
virtual_uid_maps = static:500
Regards,
Yaoxing
Wietse Venema
> @Stan
> It's OK, you helped a lot already. And your figure is a very good
> reference for me. Thank you very much.
>
> @Wietse
> I'll try not to use relay host to check the speed.
> And sorry if I didn't express myself clearly, I don't mean to complain
> anything. Just don't know what's wrong and what to do because I'm a
> green hand. I pasted the changed parameters below. removed some lines
> because I think it's not related and don't want to make the list too
> long. (it's still very long I know ;)
>
> debug_peer_level = 2
If you are concerned about performance, then don't turn on verbose
logging. This alone can easily double the load on your disk, and
as we know the disk is often the slowest component in the system.
If your performance still is poor, keep removing main.cf changes
that not absolutely necessary (such as maximal/minimal backoff time
and so on).
If the performance is still poor, report "postconf -n" and qshape
results.
Wietse
Ramprasad A Padmanabhan
-----Original Message-----
From: Yaoxing
Sent: 23/12/2010 4:23:51 pm
To: Ramprasad
Cc: postfi...@postfix.org
Subject: Re: postfix queue tuning
Hi Ram,
I do have some more spare memory, but I'm afraid it doesn't resolve my
problem.
Let's say, my active queue is filled with 20,000 mails, but mails are
not going out but remains in memory. In this case if I increase active
queue size, I just put more mails in memory, they still don't go out.
What do you think
You must either have a network choke , or you must be reaching the concurrency limit
See your bandwidth graph , is there a flat line?
There is no point biting more than what you can chew. I would suggest do not let your active queue fill up, postfix clears slower with a full queue. Send the mails slower if possible.
You would probably end up clearing quicker.
Victor Duchovni
> qshape active
>
> T 5 10 20 40 80 160 320 640 1280 1280+
> TOTAL 1000 0 0 0 0 0 0 0 0 0 1000
> gmail.com 254 0 0 0 0 0 0 0 0 0 254
> hotmail.com 168 0 0 0 0 0 0 0 0 0 168
> yahoo.com 70 0 0 0 0 0 0 0 0 0 70
> comcast.net 14 0 0 0 0 0 0 0 0 0 14
> msn.com 12 0 0 0 0 0 0 0 0 0 12
> aol.com 11 0 0 0 0 0 0 0 0 0 11
> sbcglobal.net 10 0 0 0 0 0 0 0 0 0 10
> yahoo.com.br 9 0 0 0 0 0 0 0 0 0 9
> hotmail.co.uk 9 0 0 0 0 0 0 0 0 0 9
> web.de 8 0 0 0 0 0 0 0 0 0 8
> mail.ru 8 0 0 0 0 0 0 0 0 0 8
> free.fr 8 0 0 0 0 0 0 0 0 0 8
> yahoo.es 7 0 0 0 0 0 0 0 0 0 7
> hotmail.fr 7 0 0 0 0 0 0 0 0 0 7
> gmx.de 6 0 0 0 0 0 0 0 0 0 6
> juno.com 6 0 0 0 0 0 0 0 0 0 6
> cox.net 5 0 0 0 0 0 0 0 0 0 5
> live.com 5 0 0 0 0 0 0 0 0 0 5
> sympatico.ca 5 0 0 0 0 0 0 0 0 0 5
So it seems that the active queue contains *only* mail that is 1 day
or so old, NO mail (for the destinations above) has entered the active
queue in 1280 minutes.
> and for qshape incoming:
>
No mail in the incoming queue that is older than 640 minutes, so it
would seem that the active queue is not being drained at all. Is the
queue-manager even running? Are any attempted deliveries logged?
Some logging of qmgr(8) and smtp(8) activity is essential at this
juncture. It is implausible that no mail has entered the active queue
at all, while the incoming queue is full, unless your queue manager is
not running at all, or consistently crashes.
Are any of the messages in the queue addressed to thousands of recipients?
Can you report the file counts in the active and incoming queues?
cd /var/spool/postfix/active
find . ! -name '?' -print | wc -l
cd ../incoming
find . ! -name '?' -print | wc -l
> T 5 10 20 40 80 160 320 640 1280 1280+
> TOTAL 1000 9 7 33 47 79 181 393 251 0 0
> gmail.com 221 1 2 6 9 18 45 100 40 0 0
> hotmail.com 153 1 1 4 9 15 32 63 28 0 0
> e.dealextreme.com 152 1 0 4 4 2 17 35 89 0 0
> yahoo.com 88 0 0 3 7 3 14 37 24 0 0
> msn.com 12 0 0 0 0 0 1 6 5 0 0
> yahoo.com.br 12 0 0 1 2 1 1 6 1 0 0
> aol.com 9 0 0 0 0 1 1 5 2 0 0
> mail.ru 8 0 0 0 0 1 2 3 2 0 0
> free.fr 7 0 0 0 0 2 1 4 0 0 0
> comcast.net 5 0 0 0 0 0 2 2 1 0 0
> web.de 4 0 0 0 0 1 0 2 1 0 0
> yahoo.es 4 0 0 0 1 0 1 2 0 0 0
> hotmail.fr 4 0 0 0 0 0 0 2 2 0 0
> terra.com.br 4 0 0 0 0 1 2 1 0 0 0
> sympatico.ca 4 0 0 0 0 1 0 2 1 0 0
> ntlworld.com 4 0 0 0 0 1 0 3 0 0 0
> bellsouth.net 4 0 0 0 0 0 2 1 1 0 0
> gmx.de 3 0 0 0 0 1 0 2 0 0 0
> ono.com 3 0 0 0 0 0 0 3 0 0 0
--
Viktor.
Yaoxing
like15Mb/s according to
iftop -i eth1
For the concurrency issue, what parameter would you suggest to change? I
found some parameters from the documents but do not fully understand
them yet.
Regards,
Yaoxing
Victor Duchovni
> I think the bandwidths is OK. I have a 100Mb ethernet but until now it's
> like15Mb/s according to
> iftop -i eth1
> For the concurrency issue, what parameter would you suggest to change? I
> found some parameters from the documents but do not fully understand them
> yet.
Waste of time. Post NON-VERBOSE LOGGING by smtp(8) and qmgr(8).
logfiles=/some/where
egrep 'postfix/(qmgr|smtp)\[' $logfiles | tail -100
Also post any signs of trouble:
egrep ': (error|fatal|panic): ' $logfiles | tail -100
And potential trouble:
egrep ': warning: ' $logfiles | tail -20
--
Viktor.
Yaoxing
new mails. I think that's why all mails are from 1280+ min ago.
I use
find active/ | wc -l
which gives me 20,002, while
find incoming/ | wc -l
gives 130,000+
and the incoming queue is slowly decreasing.
Regards,
Yaoxing
Yaoxing
2010/12/24 1:16, Victor Duchovni Wrote:
> On Fri, Dec 24, 2010 at 01:07:58AM +0800, Yaoxing wrote:
> Waste of time. Post NON-VERBOSE LOGGING by smtp(8) and qmgr(8).
>
> logfiles=/some/where
> egrep 'postfix/(qmgr|smtp)\[' $logfiles | tail -100
Dec 23 11:23:25 e postfix/qmgr[29972]: 3C15BFB9143: removed
Dec 23 11:23:25 e postfix/qmgr[29972]: EB0E829E9CBC:
from=<ne...@e.xxx.com>, size=18376, nrcpt=1 (queue active)
Dec 23 11:23:27 e postfix/smtp[30439]: 88B5B2F385D0:
to=<fun...@freenet.de>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=7,
delay=410207, delays=364344/45849/0/13, dsn=2.0.0, status=sent (250
2.0.0 Ok, id=30461-01-7, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok:
queued as 722BA2988573)
Dec 23 11:23:27 e postfix/qmgr[29972]: 88B5B2F385D0: removed
Dec 23 11:23:27 e postfix/qmgr[29972]: CCB172D690D7:
from=<ne...@e.xxx.com>, size=45424, nrcpt=1 (queue active)
> Also post any signs of trouble:
>
> egrep ': (error|fatal|panic): ' $logfiles | tail -100
nothing
> And potential trouble:
>
> egrep ': warning: ' $logfiles | tail -20
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: mail for
[127.0.0.1]:10024 is using up 20000 of 20000 active queue entries
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: you may need to reduce
smtp-amavis connect and helo timeouts
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: so that Postfix quickly
skips unavailable hosts
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: you may need to increase
the main.cf minimal_backoff_time and maximal_backoff_time
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: so that Postfix wastes
less time on undeliverable mail
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: you may need to increase
the master.cf smtp-amavis process limit
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: please avoid flushing
the whole queue when you have
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: lots of deferred mail,
that is bad for performance
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: to turn off these
warnings specify: qmgr_clog_warn_time = 0
Victor Duchovni
> It's a newsletter group. because it's congesting so I stopped posting new
> mails. I think that's why all mails are from 1280+ min ago.
No. This is wrong, the incoming queue contains fairly fresh mail.
> I use
> find active/ | wc -l
> which gives me 20,002, while
OK, so the active queue has 20,000 single-recipient messages.
> find incoming/ | wc -l
> gives 130,000+
> and the incoming queue is slowly decreasing.
Then fresh mail should be showing up in the active queue, but your
qshape output reports none. you mentioned the ever-broken MailScanner,
which manipulates the Postfix queue in unsupported ways, how many files
are in your "hold" queue? Disable MainScanner and see whether you still
have problems. MailScanner is not supported here...
Post NON-VERBOSE LOGGING by smtp(8) and qmgr(8).
logfiles=/some/where
egrep 'postfix/(qmgr|smtp)\[' $logfiles | tail -100
Also post any signs of trouble:
egrep ': (error|fatal|panic): ' $logfiles | tail -100
And potential trouble:
egrep ': warning: ' $logfiles | tail -20
--
Viktor.
Victor Duchovni
>> Waste of time. Post NON-VERBOSE LOGGING by smtp(8) and qmgr(8).
>>
>> logfiles=/some/where
>> egrep 'postfix/(qmgr|smtp)\[' $logfiles | tail -100
>
> Dec 23 11:23:25 e postfix/qmgr[29972]: 3C15BFB9143: removed
> Dec 23 11:23:25 e postfix/qmgr[29972]: EB0E829E9CBC: from=<ne...@e.xxx.com>,
> size=18376, nrcpt=1 (queue active)
> Dec 23 11:23:27 e postfix/smtp[30439]: 88B5B2F385D0:
> to=<fun...@freenet.de>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=7,
> delay=410207, delays=364344/45849/0/13, dsn=2.0.0, status=sent (250 2.0.0
It takes mail many days to get through the content filter. Fix your content
filter.
> Ok, id=30461-01-7, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as
> 722BA2988573)
> Dec 23 11:23:27 e postfix/qmgr[29972]: 88B5B2F385D0: removed
> Dec 23 11:23:27 e postfix/qmgr[29972]: CCB172D690D7: from=<ne...@e.xxx.com>,
> size=45424, nrcpt=1 (queue active)
This is not 100 log entries, how quickly do you rotate your logs? This
shows mail entering a disasterously slow content filter. Do you have a
filter loop? What happens to queue-id "722BA2988573"? Are you using both
amavisd-new and MailScanner?
>> egrep ': warning: ' $logfiles | tail -20
> Dec 23 11:24:28 e postfix/qmgr[29972]: warning: mail for [127.0.0.1]:10024
> is using up 20000 of 20000 active queue entries
Well, your content-filter is completely saturated to the point of being
non-functional..
--
Viktor.
Yaoxing
stopped that 10 hours ago. but it doesn't seem to make the situation better.
If I restart postfix, does it make the figure like that? because I
noticed mails in active queues went back to incoming queue while
restarting. So maybe those mails acts new mails?
Regards,
Yaoxing
Victor Duchovni
> There's nothing in my hold queue. MailScanner do you mean amavis? I stopped
> that 10 hours ago. but it doesn't seem to make the situation better.
You can't just "stop" the content filter, existing messages have the
content_filter transport recorded in the queue file, as evidenced by
your logs. To drop this record from existing messages you need to
run "postsuper -r ALL", which will move all mail from active, incoming
deferred (and I think also "hold") to "maildrop". From there, mail
will be re-processes as new mail in the queue with the new setting
of "content_filter" (as seen by pickup from main.cf or master.cf
overrides).
Restarting Postfix generally makes queue congestion worse.
> If I restart postfix, does it make the figure like that? because I noticed
> mails in active queues went back to incoming queue while restarting.
Yes.
> So maybe those mails acts new mails?
No, they do not.
Do you have ANY logging of email going somewhere other than the content
filter?
So far all the evidence is that your content_filter is slow and/or busted,
there is nothing showing what happens post-filter. What happened to that
post-filter queue-id???
--
Viktor.
Yaoxing
Dec 23 11:38:35 e postfix/qmgr[29972]: 5E55BFC749F: removed
Dec 23 11:38:35 e postfix/qmgr[29972]: 6FC51297C081:
from=<ne...@e.xxx.com>, size=18380, nrcpt=1 (queue active)
Dec 23 11:38:38 e postfix/smtp[34263]: 4E4C7297BE10:
to=<Cryog...@yahoo.com>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=6,
delay=383792, delays=337025/46756/0/11, dsn=2.0.0, status=sent (250
2.0.0 Ok, id=33658-01-6, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok:
queued as 50E79297B632)
Dec 23 11:38:38 e postfix/qmgr[29972]: 4E4C7297BE10: removed
Dec 23 11:38:38 e postfix/qmgr[29972]: 057DE2978A4D:
from=<ne...@e.xxx.com>, size=45412, nrcpt=1 (queue active)
Dec 23 11:38:40 e postfix/smtp[29728]: 057DE2978A4D:
to=<alor...@gmail.com>,
relay=gmail-smtp-in.l.google.com[74.125.65.27]:25, delay=466500,
delays=466499/0.02/0.12/1.3, dsn=2.0.0, status=sent (250 2.0.0 OK
1293125920 p26si24523792ybk.80)
Dec 23 11:38:40 e postfix/qmgr[29972]: 057DE2978A4D: removed
Dec 23 11:38:40 e postfix/qmgr[29972]: 0809EFC9676:
from=<ne...@e.xxx.com>, size=45409, nrcpt=1 (queue active)
Dec 23 11:38:40 e postfix/smtp[29486]: 0809EFC9676:
to=<bch...@hotmail.com>, relay=mx1.hotmail.com[65.54.188.126]:25,
delay=318488, delays=318487/0.02/0.14/0.45, dsn=2.0.0, status=sent (250
<2010121808082...@e.xxx.com> Queued mail for delivery)
Dec 23 11:38:40 e postfix/qmgr[29972]: 0809EFC9676: removed
Dec 23 11:38:40 e postfix/qmgr[29972]: BABE3FB5C82:
from=<ne...@e.xxx.com>, size=45423, nrcpt=1 (queue active)
Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host
g.mx.mail.yahoo.com[98.137.54.238] refused to talk to me: 421 4.7.1
[TS03] All messages from 67.228.151.195 will be permanently deferred;
Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host
f.mx.mail.yahoo.com[98.137.54.237] refused to talk to me: 421 4.7.1
[TS03] All messages from 67.228.151.195 will be permanently deferred;
Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host
i.mx.mail.yahoo.com[74.6.140.64] refused to talk to me: 421 4.7.1 [TS03]
All messages from 67.228.151.195 will be permanently deferred; Retrying
will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host
j.mx.mail.yahoo.com[66.94.237.64] refused to talk to me: 421 4.7.1
[TS03] All messages from 67.228.151.195 will be permanently deferred;
Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Dec 23 11:38:42 e postfix/smtp[32893]: 9FE65FB01E1:
to=<bil...@moonwalk-rental.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=7, delay=454234, delays=407464/46764/0/6.2, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33594-01-7, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 70B572978A4D)
Dec 23 11:38:42 e postfix/qmgr[29972]: 9FE65FB01E1: removed
Dec 23 11:38:42 e postfix/qmgr[29972]: E0934FB405C:
from=<ne...@e.xxx.com>, size=19318, nrcpt=1 (queue active)
Dec 23 11:38:42 e postfix/smtp[29766]: BABE3FB5C82:
to=<email....@yahoo.com>,
relay=d.mx.mail.yahoo.com[209.191.88.254]:25, delay=177367,
delays=177366/0.02/0.71/0, dsn=4.7.1, status=deferred (host
d.mx.mail.yahoo.com[209.191.88.254] refused to talk to me: 421 4.7.1
[TS03] All messages from 67.228.151.195 will be permanently deferred;
Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html)
Dec 23 11:38:42 e postfix/qmgr[29972]: BABE3FB5C82:
from=<ne...@e.xxx.com>, status=expired, returned to sender
Dec 23 11:38:42 e postfix/qmgr[29972]: BABE3FB5C82: removed
Dec 23 11:38:42 e postfix/qmgr[29972]: 9E83FF3844C:
from=<ne...@e.xxx.com>, size=26887, nrcpt=1 (queue active)
Dec 23 11:38:45 e postfix/smtp[29728]: E0934FB405C:
to=<Diederik...@gmail.com>,
relay=gmail-smtp-in.l.google.com[74.125.157.27]:25, delay=95227,
delays=95223/0.06/0.1/3.3, dsn=2.0.0, status=sent (250 2.0.0 OK
1293125925 72si16705236yhl.114)
Dec 23 11:38:45 e postfix/qmgr[29972]: E0934FB405C: removed
Dec 23 11:38:45 e postfix/qmgr[29972]: 64E45FBCD9D:
from=<ne...@e.xxx.com>, size=44493, nrcpt=1 (queue active)
Dec 23 11:38:51 e postfix/smtp[34263]: 7D28DFB50A0:
to=<efeca...@gmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=7, delay=383758, delays=336978/46767/0/13, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33658-01-7, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 596FAFB405C)
Dec 23 11:38:51 e postfix/qmgr[29972]: 7D28DFB50A0: removed
Dec 23 11:38:51 e postfix/qmgr[29972]: CAFFA29EAC1A:
from=<ne...@e.xxx.com>, size=45398, nrcpt=1 (queue active)
Dec 23 11:38:54 e postfix/smtp[29486]: CAFFA29EAC1A:
to=<djb...@mail.ru>, relay=mxs.mail.ru[94.100.176.20]:25, delay=214118,
delays=214116/0.02/0.51/1.8, dsn=2.0.0, status=sent (250 OK
id=1PVp7t-0001oN-00)
Dec 23 11:38:54 e postfix/qmgr[29972]: CAFFA29EAC1A: removed
Dec 23 11:38:54 e postfix/qmgr[29972]: 87FB5F388D5:
from=<ne...@e.xxx.com>, size=26879, nrcpt=1 (queue active)
Dec 23 11:38:54 e postfix/smtp[32893]: 91EA6297F898:
to=<organiza...@gmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=8, delay=320559, delays=273776/46770/0/13, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33594-01-8, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as E4823FB50A0)
Dec 23 11:38:54 e postfix/qmgr[29972]: 91EA6297F898: removed
Dec 23 11:38:54 e postfix/qmgr[29972]: 040C5297A74C: from=<>,
size=47396, nrcpt=1 (queue active)
Dec 23 11:38:55 e postfix/qmgr[29972]: 040C5297A74C: removed
Dec 23 11:38:55 e postfix/qmgr[29972]: 9EA9AF3872C:
from=<ne...@e.xxx.com>, size=26879, nrcpt=1 (queue active)
Dec 23 11:39:04 e postfix/smtp[34263]: 2F529FBB8EC:
to=<crack...@hotmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=8, delay=388149, delays=341356/46780/0/13, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33658-01-8, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 5D235FB5C82)
Dec 23 11:39:04 e postfix/qmgr[29972]: 2F529FBB8EC: removed
Dec 23 11:39:04 e postfix/qmgr[29972]: 5DA2B297FD72:
from=<ne...@e.xxx.com>, size=19313, nrcpt=1 (queue active)
Dec 23 11:39:06 e postfix/smtp[29766]: 5DA2B297FD72:
to=<alexeyko...@gmail.com>,
relay=gmail-smtp-in.l.google.com[74.125.47.27]:25, delay=317342,
delays=317341/0.02/0.09/1.6, dsn=2.0.0, status=sent (250 2.0.0 OK
1293125946 v32si36100328yba.88)
Dec 23 11:39:06 e postfix/qmgr[29972]: 5DA2B297FD72: removed
Dec 23 11:39:06 e postfix/qmgr[29972]: A23B02A68553:
from=<ne...@e.xxx.com>, size=44501, nrcpt=1 (queue active)
Dec 23 11:39:06 e postfix/smtp[32893]: AF544FB57EE:
to=<dave.t...@baesystems.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=9, delay=369810, delays=323015/46783/0/12, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33594-01-9, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 89F8E297A74C)
Dec 23 11:39:06 e postfix/qmgr[29972]: AF544FB57EE: removed
Dec 23 11:39:06 e postfix/qmgr[29972]: E86C8297BE96:
from=<ne...@e.xxx.com>, size=26875, nrcpt=1 (queue active)
Dec 23 11:39:10 e postfix/smtp[34263]: EDAFA297E99C:
to=<highfive...@hotmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=9, delay=338289, delays=291490/46793/0/5.8, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33658-01-9, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 83CC1FB57EE)
Dec 23 11:39:10 e postfix/qmgr[29972]: EDAFA297E99C: removed
Dec 23 11:39:10 e postfix/qmgr[29972]: 6F9F12988487:
from=<ne...@e.xxx.com>, size=44493, nrcpt=1 (queue active)
Dec 23 11:39:15 e postfix/smtp[32893]: 74640FBE36A:
to=<david....@gmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=10, delay=363735, delays=316931/46795/0/8.9, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33594-01-10, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 8EF7C297BE10)
Dec 23 11:39:15 e postfix/qmgr[29972]: 74640FBE36A: removed
Dec 23 11:39:15 e postfix/qmgr[29972]: D5DC7297F5CE: from=<>,
size=47643, nrcpt=1 (queue active)
Dec 23 11:39:16 e postfix/qmgr[29972]: D5DC7297F5CE: removed
Dec 23 11:39:16 e postfix/qmgr[29972]: D96D2FB9F59:
from=<ne...@e.xxx.com>, size=26885, nrcpt=1 (queue active)
Dec 23 11:39:22 e postfix/smtp[34263]: 47A3D3968FB4:
to=<david...@ntlworld.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=10, delay=407720, delays=360909/46799/0/12, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33658-01-10, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 86341297D6DA)
Dec 23 11:39:22 e postfix/qmgr[29972]: 47A3D3968FB4: removed
Dec 23 11:39:22 e postfix/qmgr[29972]: 41E47FB2C3B:
from=<ne...@e.xxx.com>, size=26893, nrcpt=1 (queue active)
Dec 23 11:39:32 e postfix/smtp[32893]: A99202F385F0:
to=<frederi...@gmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=11, delay=411140, delays=364319/46804/0/17, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33594-01-11, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 0AFFD297E99C)
Dec 23 11:39:32 e postfix/qmgr[29972]: A99202F385F0: removed
Dec 23 11:39:32 e postfix/qmgr[29972]: F1D3C2B68DD4:
from=<ne...@e.xxx.com>, size=18378, nrcpt=1 (queue active)
Dec 23 11:39:33 e postfix/smtp[34263]: D5F59FC68CC:
to=<gustav...@hotmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=11, delay=346000, delays=299178/46811/0/11, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33658-01-11, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 8E3BB297F5CE)
Dec 23 11:39:33 e postfix/qmgr[29972]: D5F59FC68CC: removed
Dec 23 11:39:34 e postfix/qmgr[29972]: A565EFB1891: from=<>, size=47457,
nrcpt=1 (queue active)
Dec 23 11:39:34 e postfix/qmgr[29972]: A565EFB1891: removed
Dec 23 11:39:34 e postfix/qmgr[29972]: D5B323168C31:
from=<ne...@e.xxx.com>, size=44509, nrcpt=1 (queue active)
Dec 23 11:39:40 e postfix/smtp[34263]: 9CCC9298AD82:
to=<phil...@shaw.ca>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=12,
delay=311093, delays=264264/46822/0/6.8, dsn=2.0.0, status=sent (250
2.0.0 Ok, id=33658-01-12, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok:
queued as BD9F1FB1891)
Dec 23 11:39:40 e postfix/qmgr[29972]: 9CCC9298AD82: removed
Dec 23 11:39:40 e postfix/qmgr[29972]: 460D32979315:
from=<ne...@e.xxx.com>, size=26887, nrcpt=1 (queue active)
Dec 23 11:39:40 e postfix/qmgr[29972]: warning: mail for
skips unavailable hosts
Dec 23 11:39:40 e postfix/qmgr[29972]: warning: you may need to increase
to=<erna...@yahoo.com.br>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=12, delay=378227, delays=331395/46820/0/12, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33594-01-12, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 2DD13297F898)
Dec 23 11:39:44 e postfix/qmgr[29972]: 88058297C9E4: removed
Dec 23 11:39:44 e postfix/qmgr[29972]: 15FF22D693F3:
from=<ne...@e.xxx.com>, size=26871, nrcpt=1 (queue active)
Dec 23 11:39:52 e postfix/smtp[34263]: 594BD2D69805:
to=<invi...@invisual.es>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=13, delay=325019, delays=278178/46829/0/12, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33658-01-13, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as A0EA7297C9E4)
Dec 23 11:39:52 e postfix/qmgr[29972]: 594BD2D69805: removed
Dec 23 11:39:52 e postfix/qmgr[29972]: 7043F2979CE2:
from=<ne...@e.xxx.com>, size=45428, nrcpt=1 (queue active)
Dec 23 11:39:55 e postfix/smtp[29728]: 7043F2979CE2:
to=<dario...@comcast.net>, relay=mx2.comcast.net[76.96.30.116]:25,
delay=36569, delays=36567/0.03/0.87/1.8, dsn=5.1.1, status=bounced (host
mx2.comcast.net[76.96.30.116] said: 550 5.1.1 Not our Customer (in reply
to RCPT TO command))
Dec 23 11:39:55 e postfix/qmgr[29972]: 7043F2979CE2: removed
Dec 23 11:39:55 e postfix/smtp[32893]: 134E8297B4CD:
to=<clou...@hotmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=13, delay=402412, delays=355568/46833/0/11, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33594-01-13, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as A4211297FD72)
Dec 23 11:39:55 e postfix/qmgr[29972]: 54C3C29EAF40:
from=<ne...@e.xxx.com>, size=19287, nrcpt=1 (queue active)
Dec 23 11:39:55 e postfix/qmgr[29972]: 134E8297B4CD: removed
Dec 23 11:39:55 e postfix/qmgr[29972]: 683BC29A89C0:
from=<ne...@e.xxx.com>, size=26871, nrcpt=1 (queue active)
Dec 23 11:39:56 e postfix/smtp[29486]: 54C3C29EAF40:
to=<bil...@siteria.com>, relay=mail.siteria.com[206.123.72.210]:25,
delay=192289, delays=192288/0.02/0.26/0.19, dsn=2.0.0, status=sent (250
ok 1293126005 qp 7832)
Dec 23 11:39:56 e postfix/qmgr[29972]: 54C3C29EAF40: removed
Dec 23 11:39:56 e postfix/qmgr[29972]: A96872F38248:
from=<ne...@e.xxx.com>, size=26881, nrcpt=1 (queue active)
Dec 23 11:40:00 e postfix/smtp[34263]: 7365B2978DFB:
to=<bd...@rogers.com>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=14,
delay=464257, delays=417409/46841/0/7.6, dsn=2.0.0, status=sent (250
2.0.0 Ok, id=33658-01-14, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok:
queued as ABE6B2979CE2)
Dec 23 11:40:00 e postfix/qmgr[29972]: 7365B2978DFB: removed
Dec 23 11:40:00 e postfix/qmgr[29972]: 25263FBAB9F:
from=<ne...@e.xxx.com>, size=26881, nrcpt=1 (queue active)
Dec 23 11:40:02 e postfix/smtp[32893]: 55D54FBA28E:
to=<Discor...@hotmail.com>, relay=127.0.0.1[127.0.0.1]:10024,
conn_use=14, delay=394867, delays=348017/46844/0/6.9, dsn=2.0.0,
status=sent (250 2.0.0 Ok, id=33594-01-14, from MTA([127.0.0.1]:10025):
250 2.0.0 Ok: queued as 713E22978DFB)
Dec 23 11:40:02 e postfix/qmgr[29972]: 55D54FBA28E: removed
Dec 23 11:40:02 e postfix/qmgr[29972]: 28E6A29EA884:
from=<ne...@e.xxx.com>, size=45430, nrcpt=1 (queue active)
Dec 23 11:40:09 e postfix/smtp[29766]: 28E6A29EA884:
to=<breivi...@gmail.com>,
relay=gmail-smtp-in.l.google.com[74.125.67.27]:25, delay=270614,
delays=270608/0.02/5.1/1.3, dsn=2.0.0, status=sent (250 2.0.0 OK
1293126009 g6si24321991anh.62)
Dec 23 11:40:09 e postfix/qmgr[29972]: 28E6A29EA884: removed
Dec 23 11:40:09 e postfix/qmgr[29972]: AAF1E29E90FC:
from=<ne...@e.xxx.com>, size=44507, nrcpt=1 (queue active)
Stan Hoeppner
> relay=127.0.0.1[127.0.0.1]:10024
Why are you sending outbound newsletters through a content filter? You
should already know that the content is not spam, and virus free, yes?
And if they are newsletters, why are you sending them every 4 seconds to
the entire subscriber base?
Speaking of that 4 second interval for each newsletter blast out, might
your application/content delivery be better served by an instant
messaging system or IRC channel?
One newsletter every 4 seconds = 21,600 newsletters per subscriber per
day. No person would want that. At the rate of one every 4 seconds,
this isn't a newsletter at all.
What, exactly, are you sending these subscribers every 4 seconds? It is
probably a good time to explain what you're trying to accomplish with
your content delivery system. Or am I completely misunderstanding the
"every 4 seconds" thing?
--
Stan
Yaoxing
I have like 400,000 subscribers. every week I send to all of them a news
letter. Every 4 sec, I send out 1 mail to 1 person. I know it's very
slow, but still it congests. That's why I'm wondering what's wrong. also
the same server serves some images. So if it's because of the image
server which takes most of the disk IO, I'll just remove the image
server to another host.
And about the filter, at the very beginning I don't know it causes so
many troubles. that's what green hands do, right :)
Regards,
Yaoxing
Stan Hoeppner
> Then I think I didn't express it clearly. sorry for my bad English.
> I have like 400,000 subscribers. every week I send to all of them a news
> letter. Every 4 sec, I send out 1 mail to 1 person. I know it's very
> slow, but still it congests. That's why I'm wondering what's wrong. also
> the same server serves some images. So if it's because of the image
> server which takes most of the disk IO, I'll just remove the image
> server to another host.
> And about the filter, at the very beginning I don't know it causes so
> many troubles. that's what green hands do, right :)
After doing some checking, it appears you are blasting out porn spam on
behalf of xxx.com whose domain registration is privacy protected by
contactprivacy.com in Toronto. Thankfully you're sending less than what
you're attempting to send due to these technical problems.
According to your log snippet Yahoo has permanently deferred all your
mail from 67.228.151.195 (Softlayer IP space, a known spam haven ISP due
to a horribly vetted reseller program--actually no vetting at all) due
to rate violation.
The Barracuda BRBL dnsbl has your sending IP blacklisted:
http://www.mxtoolbox.com/SuperTool.aspx?action=blacklist%3a67.228.151.195
The IP resolved to your hostname in your sender address, e.xxx.com, is
listed with dnsbl Tiopan:
http://www.mxtoolbox.com/SuperTool.aspx?action=blacklist%3ae.xxx.com
and xxx.com has wildcard dns. any.thing.i.type.xxx.com resolves to
66.114.124.140, which explains the above sender address resolved host IP
being on the Tiopan list.
All of these things are 100% spammer sign. You appear to be affiliate
spamming (attempting to anyway) on behalf of xxx.com.
I'm sure other dnsbl listings are forthcoming, especially if we help you
get your queue problem resolved. I'm thinking at this point it would be
best if we simply stopped assisting you, as we're not in the business of
helping spammers.
As you are a "green" spammer, demonstrated in spades here in this help
thread, I implore you to stop spamming now, and apply your (developing)
technical skill set to other, more socially positive, endeavors.
--
Stan
Victor Duchovni
> Dec 23 11:38:35 e postfix/qmgr[29972]: 6FC51297C081: from=<ne...@e.xxx.com>,
> size=18380, nrcpt=1 (queue active)
Some new mail is entering the active queue either from "incoming" or
"deferred" queue.
Do you really want the hostname "e" in the envelope sender address?
> Dec 23 11:38:38 e postfix/smtp[34263]: 4E4C7297BE10:
> to=<Cryog...@yahoo.com>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=6,
> delay=383792, delays=337025/46756/0/11, dsn=2.0.0, status=sent (250 2.0.0
> Ok, id=33658-01-6, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as
> 50E79297B632)
Your content filter is FUBAR, processing 4-day old mail, it takes 11
seconds to process one message! This is absurdly high filter latency.
> Dec 23 11:38:38 e postfix/qmgr[29972]: 057DE2978A4D: from=<ne...@e.xxx.com>,
> size=45412, nrcpt=1 (queue active)
> Dec 23 11:38:40 e postfix/smtp[29728]: 057DE2978A4D:
> to=<alor...@gmail.com>,
> relay=gmail-smtp-in.l.google.com[74.125.65.27]:25, delay=466500,
> delays=466499/0.02/0.12/1.3, dsn=2.0.0, status=sent (250 2.0.0 OK
> 1293125920 p26si24523792ybk.80)
Google accepts your mail in a timely manner, but it had been delayed
in "incoming" for a very long time...
> Dec 23 11:38:40 e postfix/smtp[29486]: 0809EFC9676:
> to=<bch...@hotmail.com>, relay=mx1.hotmail.com[65.54.188.126]:25,
> delay=318488, delays=318487/0.02/0.14/0.45, dsn=2.0.0, status=sent (250
> <2010121808082...@e.xxx.com> Queued mail for delivery)
Hotmail also accepts your mail, be it with a bit more latency. Of
course given how congested your incoming queue is with pre-filter
mail, it takes forever to enter the active queue...
> Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host
> g.mx.mail.yahoo.com[98.137.54.238] refused to talk to me: 421 4.7.1 [TS03]
> All messages from 67.228.151.195 will be permanently deferred; Retrying
> will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Yahoo wants nothing to do with you. And will NOT accept your mail,
no point in sending it unless you can resolve this with them. See
the friendly URL.
> Dec 23 11:38:42 e postfix/smtp[32893]: 9FE65FB01E1:
> to=<bil...@moonwalk-rental.com>, relay=127.0.0.1[127.0.0.1]:10024,
> conn_use=7, delay=454234, delays=407464/46764/0/6.2, dsn=2.0.0, status=sent
> (250 2.0.0 Ok, id=33594-01-7, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok:
> queued as 70B572978A4D)
It is somewhat unlikely that <bil...@moonwalk-rental.com> subscribed
to your newsletter. Such a role address is typically used for
"billing", not newsletter subscriptions. How do you manage subscriptions
to your list?
> Dec 23 11:38:51 e postfix/smtp[34263]: 7D28DFB50A0:
> to=<efeca...@gmail.com>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=7,
> delay=383758, delays=336978/46767/0/13, dsn=2.0.0, status=sent (250 2.0.0
> Ok, id=33658-01-7, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as
> 596FAFB405C)
More filter trouble. Your filter is FUBAR.
> Dec 23 11:39:55 e postfix/smtp[29728]: 7043F2979CE2:
> to=<dario...@comcast.net>, relay=mx2.comcast.net[76.96.30.116]:25,
> delay=36569, delays=36567/0.03/0.87/1.8, dsn=5.1.1, status=bounced (host
> mx2.comcast.net[76.96.30.116] said: 550 5.1.1 Not our Customer (in reply to
> RCPT TO command))
Some of your list members don't exist. Do you remove non-working
addresses from your list?
--
Viktor.
Yaoxing
with xxx.com due to my policies. I used to mask it with abc.com but it
seems to be a TV chanel. I don't want to confuse people so I changed to
xxx.com, just randomly.
My company is a ecommerce company which send newsletters to our
subscribed clients weekly. we have nothing to do with spammers.
Regards,
Yaoxing
2010/12/24 2:43, Stan Hoeppner Wrote:
> Yaoxing put forth on 12/23/2010 12:05 PM:
Wietse Venema
> No this is a misunderstanding. I just masked my company's domain name
> with xxx.com due to my policies. I used to mask it with abc.com but it
> seems to be a TV chanel. I don't want to confuse people so I changed to
> xxx.com, just randomly.
> My company is a ecommerce company which send newsletters to our
> subscribed clients weekly. we have nothing to do with spammers.
I was getting suspicious because Yahoo is permanently refusing your
mail, but this is bad:
% host 195.151.228.67.b.barracudacentral.org
195.151.228.67.b.barracudacentral.org has address 127.0.0.2
BTW your SMTP server banner says e.dealextreme.com.
Wietse
Victor Duchovni
> My company is a ecommerce company which send newsletters to our subscribed
> clients weekly. we have nothing to do with spammers.
Sufficiently poor list management and/or privacy policies are
indistinguishable from spam. If you want to have your mail accepted widely
ensure that all email addresses are opted-in, never given away and never
purchased. Non-working addresses need to be reasonably promptly scrubbed
from the list.
network:Class-Name:network
network:ID:NETBLK-SOFTLAYER.67.228.128.0/19
network:Auth-Area:67.228.128.0/19
network:Network-Name:SOFTLAYER-67.228.128.0
network:IP-Network:67.228.151.192/27
network:IP-Network-Block:67.228.151.192-67.228.151.223
network:Organization;I:DealExtreme
network:Street-Address:PO BOX 779 TAI PO POST OFFICE
network:City:TAI PO
network:State:ot
network:Postal-Code:Hong Kong
network:Country-Code:HK
network:Tech-Contact;I:sysa...@softlayer.com
network:Abuse-Contact;I:deale...@gmail.com
network:Admin-Contact;I:IPADM258-ARIN
network:Created:20100329
network:Updated:20100329
network:Updated-By:ipa...@softlayer.com
Also:
195.151.228.67.in-addr.arpa. IN PTR 67.228.151.195-static.reverse.softlayer.com.
is not a particularly wise PTR for a bulk-mail sending MTA.
--
Viktor.
Yaoxing
as you found out already, try Alexa to get more about dealextreme.com
which would prove to you we are not spammers.
Regards,
Yaoxing
2010/12/24 3:05, Wietse Venema Wrote:
> Yaoxing:
Yaoxing
send newsletters until recently. That's why I suddenly get so many
subscribers, valid and invalid, all at a time, and get so many troubles
then. Otherwise I won't work until 3:00AM.
Anyway, we're too far away from the topic. So far, I find that the issue
is mainly caused by the content filter which is completely unnecessary.
I removed the filter and tried
postsuper -r ALL
service postfix reload
It works great, now no congesting at all. Thank you so much. I'll follow
your other suggestions to adjust our servers.
And for Stan, sorry to have made you misunderstand. Again, your figure
150 tps disk IO helps to decide the image server should be removed from
the mail server host.
Merry Christmas & Happy new year to all of you!
Regards,
Yaoxing
Ralf Hildebrandt
> It takes mail many days to get through the content filter. Fix your content
> filter.
Or circumvent it for this type of mail! If your KNOW what you're
sending out, why scan for viruses?
--
Ralf Hildebrandt
Geschäftsbereich IT | Abteilung Netzwerk
Charité - Universitätsmedizin Berlin
Campus Benjamin Franklin
Hindenburgdamm 30 | D-12203 Berlin
Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
ralf.hil...@charite.de | http://www.charite.de
Ralf Hildebrandt
> I was getting suspicious because Yahoo is permanently refusing your
> mail, but this is bad:
>
> % host 195.151.228.67.b.barracudacentral.org
> 195.151.228.67.b.barracudacentral.org has address 127.0.0.2
>
> BTW your SMTP server banner says e.dealextreme.com.
It's also listed on SORBS as dealextreme.com
Stan Hoeppner
> * Wietse Venema <wie...@porcupine.org>:
>
>> I was getting suspicious because Yahoo is permanently refusing your
>> mail, but this is bad:
>>
>> % host 195.151.228.67.b.barracudacentral.org
>> 195.151.228.67.b.barracudacentral.org has address 127.0.0.2
>>
>> BTW your SMTP server banner says e.dealextreme.com.
>
> It's also listed on SORBS as dealextreme.com
Whether it's porn or gadgets, the abundant evidence gathered, starting
with his own logs, makes it pretty clear that he's spamming. If you buy
his explanation of obfuscating his sender address, note that he didn't
give a rip about obfuscating the recipient addresses, which will all
undoubtedly now get scraped by other spammers from websites that archive
this list.
His hasty exit from the thread after being called out is simply the
exclamation point.
--
Stan
张耀星
I didn't obfusecate the clients email that's my fault. But why do u
think I post only a few lines at first?
Besides, I did not exit from the thread, removing amavis does resolve
my problem. And it's really too late for me if u ever noticed the
timestamp of my mail, did u? What else? I'm still here reading ur
posts but not hiding away after 5 hours sleep.
If it's still not enough, what else would u suggest I can prove myself?
Sent from my iPhone
Victor Duchovni
> I cannot really understand based on what do u insist on I'm spamer? Ok
> I didn't obfusecate the clients email that's my fault. But why do u
> think I post only a few lines at first?
> Besides, I did not exit from the thread, removing amavis does resolve
> my problem. And it's really too late for me if u ever noticed the
> timestamp of my mail, did u? What else? I'm still here reading ur
> posts but not hiding away after 5 hours sleep.
> If it's still not enough, what else would u suggest I can prove myself?
- Make sure your lists really contain users who want to receive the
newsletter, not just users whose email address you happened to collect
at some point in the past. Opt-in, not opt-out.
- Fix your DNS to have non-generic PTR records.
- Work with Yahoo, Barracuda, ... to be removed from blacklists, added
to whitelists, ...
- Remove non-working addresses promptly from your lists.
- ...
Large scale bulk-email is 99% list management and 1% software/hardware
delivery infrastructure.
If you err on the side of not mailing users whose voluntary subscription
status is uncertain, you should be able to improve your sender
"reputation" over time.
--
Viktor.
Ramprasad A Padmanabhan
-----Original Message-----
From: Yaoxing
Sent: 23/12/2010 10:37:58 pm
To: r...@netcore.co.in
Cc: postfi...@postfix.org
Subject: Re: postfix queue tuning
> I think the bandwidths is OK. I have a 100Mb > ethernet but until now it's
> like15Mb/s according to
> iftop -i eth1
> For the concurrency issue, what parameter
> you suggest to change? I
default_destination_concurrency_limit
See how many smtp processes are already running and change.
Use with care if you are delivering to other servers not in your control. Not everyone would appreciate too many simultaneous connections. You can also set different limits for different destination with trivial changes. Postfix is extremely configurable
Ralf Hildebrandt
> - Remove non-working addresses promptly from your lists.
This step alon considerably improves reputation AND delivery time.
Yaoxing
I'll have to delete the dead mails from our database manually from time
to time. Any ideas how I can get everything work fluently? I mean, for
example, every several days I get all dead mail addresses from postfix
by maybe a windows console program, then delete them from our
subscription database. But where can I get the list firstly? dead mails
seems to have been deleted directly.
Regards,
Yaoxing
Yaoxing
Regards,
Yaoxing
2010/12/24 9:17, Victor Duchovni Wrote:
> On Fri, Dec 24, 2010 at 09:11:19AM +0800, ????????? wrote:
> - Make sure your lists really contain users who want to receive the
> newsletter, not just users whose email address you happened to collect
> at some point in the past. Opt-in, not opt-out.
The list comes from our clients' subscriptions. However, we didn't
verify the ownership of the emails before which maybe lead to invalid
email addresses. This is what we can improve in future.
> - Fix your DNS to have non-generic PTR records.
Does it matter if I use the same sub domain, which is "e", or should I
use some other domain maybe "email" instead?
> - Work with Yahoo, Barracuda, ... to be removed from blacklists, added
> to whitelists, ...
>
> - Remove non-working addresses promptly from your lists.
>
Wietse Venema
> True but there got to be some easy way to export that list, otherwise
> I'll have to delete the dead mails from our database manually from time
> to time. Any ideas how I can get everything work fluently? I mean, for
> example, every several days I get all dead mail addresses from postfix
> by maybe a windows console program, then delete them from our
> subscription database. But where can I get the list firstly? dead mails
> seems to have been deleted directly.
Postfix returns undeliverable mail to the envelope sender address.
Postfix logs the envelope sender address as:
Dec 24 08:36:59 spike postfix/qmgr[31619]: 9E33D1F3EA7:
from=<wie...@porcupine.org>, size=X, nrcpt=Y (queue active)
The idea is to to feed the returned mail through a program that
extracts the failed recipient address and unsubscribes the recipient.
Wietse
> Regards,
> Yaoxing
>
>
> 2010/12/24 20:21, Ralf Hildebrandt Wrote:
> > * Victor Duchovni<Victor....@morganstanley.com>:
> >
> >> - Remove non-working addresses promptly from your lists.
Yaoxing
there any more specific way to do that? because scanning spends a lot of
time, and you don't know where you stopped last time (or not easy to
find out). especially our front end platform is based on .NET which does
not work with Linux logs easily. It would be great if there's a way for
postfix to output such a list directly, so that other programs just read
it directly.
Regards,
Yaoxing
pf at alt-ctrl-del.org
> Yaoxing:
>> True but there got to be some easy way to export that list, otherwise
>> I'll have to delete the dead mails from our database manually from time
>> to time. Any ideas how I can get everything work fluently? I mean, for
>> example, every several days I get all dead mail addresses from postfix
>> by maybe a windows console program, then delete them from our
>> subscription database. But where can I get the list firstly? dead mails
>> seems to have been deleted directly.
>
> Postfix logs the envelope sender address as:
>
> Dec 24 08:36:59 spike postfix/qmgr[31619]: 9E33D1F3EA7:
> from=<wie...@porcupine.org>, size=X, nrcpt=Y (queue active)
>
> The idea is to to feed the returned mail through a program that
> extracts the failed recipient address and unsubscribes the recipient.
>
> Wietse
If searching for 'status=bounced' is a valid way to find the addresses, then see the attached example.
Victor Duchovni
> So I must scan the log for the list, isn't it? It works of course but is
> there any more specific way to do that? because scanning spends a lot of
> time, and you don't know where you stopped last time (or not easy to find
> out). especially our front end platform is based on .NET which does not
> work with Linux logs easily. It would be great if there's a way for postfix
> to output such a list directly, so that other programs just read it
> directly.
For bulk mail in high volume VERP is essentially a must, and software
needs to be developed and deployed to parse the bounces to determine which
are non-deliveries, and which are out-of-office or other auto-replies.
http://www.postfix.org/VERP_README.html
Just parsing logs is not enough, you need to parse returning bounce messages.
Reading the source code of bounce processors in Majordomo, Mailman, ...
will set you on the right path.
--
Viktor.
Stan Hoeppner
> The list comes from our clients' subscriptions. However, we didn't
> verify the ownership of the emails before which maybe lead to invalid
> email addresses. This is what we can improve in future.
You should have already had a process in place for "list vetting" before
you decided to send bulk email on behalf of a "client". You are
functioning as an Email Service Provider or "ESP" in your current
described capacity, or you are simply a spammer. If you are bulk
emailing on behalf of a client _without_ already having a list vetting
process in place then you _ARE_ a spammer, by definition.
If you are a trying to be a "legit" bulk mailer, you need to implement
an automated COI process. COI - Confirmed Opt In. This means you have
a closed loop system for web based subscriptions where people can sign
up and remove themselves at will. Typically this is done with a mailing
list manager program such as GNU Mailman or majordomo on *nix platforms.
If you need to integrate this with a Microsoft database that's clue you
will have to create yourself or acquire software that's already been
written for this purpose. There surely already is as you can't be the
first person to have such a need.
DONT'T USE THIRD PARTY LISTS!! If your email subscribers actually want
your mailings, setup a subscription URL that they can visit to sign up
for your newsletters.
The fact that you have done none of these things already shows that you
are a spammer, whether you consider yourself one or not. There are
masses of online resources that teach one to properly do legit bulk
mailing. Google.
If you _are_ a legit bulk mailer, I'd suggest you sign up with an ESP
like Constant Contact and have them do your mailings for you. They have
all of the vetting processes in place, and will teach you how to
properly do legit bulk mailing, COI, etc. There obviously is a fee for
their services, but it will be more than cost effective if you have
legitimate subscribers who want your newsletters/product fliers/whatever.
Your current "shoot from the hip" method of bulk mailing is never going
to fly in today's world. This is why you're already listed on dns black
lists and being wholesale rejected by Yahoo.
Given your current methods, you need to immediately stop all of your
bulk mailings until get a proper closed loop subscription service up and
running, whether you do that in house or with an ESP like Constant Contact.
If you are a legit bulk mailer, please have a Merry Christmas. If you
are a spammer, choke on your eggnog.
--
Stan
Yaoxing
themselves. And each newsletter contains such a link in case users want
to uns*bscribe.
We already have a very big list which is filled with users who purchased
and s*bscribed in our web site. So there's no need to buy that from a
3rd party company. It's also the reason why we want to manage the list
ourselves. Handing such a list to a 3rd party ESP is always a potential
risk.
So far thank you for all your messages and merry christmas!
Regards,
Yaoxing