Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
transport_maps "loops back to myself"
265 views
Skip to first unread message
MV
Mar 26, 2014, 3:42:45 PM3/26/14
to
I'm trying to use multiple smtp IPs without having multiple postfix instances.
So I tried to split smtp using transport_maps and that works ok for
local mail destined to foreign destinations.
But it breaks the delivery of foreign mail destined to local
destinations with the error "mail for example.com loops back to
myself".
Where's the place place to "plug" the custom service
(postfix-smtp-roundrobin.pl)?
# main.cf
transport_maps = tcp:127.0.0.1:9999
# -------%<----------------------------------
# master.cf
smtp unix - - n - - smtp
relay unix - - n - - smtp
127.0.0.1:smtp inet n - n - - smtpd
1.1.1.1:smtp inet n - n - - smtpd
-o myhostname=mx1.example.com
-o smtp_helo_name=mx1.example.com
-o smtp_bind_address=1.1.1.1
-o syslog_name=smtpd-mx1
-o smtpd_tls_cert_file=/etc/pki/postfix/mx1.example.com.cert
-o smtpd_tls_key_file=/etc/pki/postfix/mx1.example.com.key
-o smtp_tls_cert_file=/etc/pki/postfix/mx1.example.com.cert
-o smtp_tls_key_file=/etc/pki/postfix/mx1.example.com.key
-o smtpd_sasl_local_domain=mx1.example.com
2.2.2.2:smtp inet n - n - - smtpd
-o myhostname=mx2.example.com
-o smtp_helo_name=mx2.example.com
-o smtp_bind_address=2.2.2.2
-o syslog_name=smtpd-mx2
-o smtpd_tls_cert_file=/etc/pki/postfix/mx2.example.com.cert
-o smtpd_tls_key_file=/etc/pki/postfix/mx2.example.com.key
-o smtp_tls_cert_file=/etc/pki/postfix/mx2.example.com.cert
-o smtp_tls_key_file=/etc/pki/postfix/mx2.example.com.key
-o smtpd_sasl_local_domain=mx2.example.com
# Round-robin outgoing smtp
# And the contents of /usr/local/sbin/postfix-smtp-roundrobin.pl
# can be found at https://gist.github.com/mvsantos/9786697
#
127.0.0.1:9999 inet n n n - 0 spawn
user=nobody argv=/usr/local/sbin/postfix-smtp-roundrobin.pl
smtp1 unix - - n - - smtp
-o myhostname=mx1.example.com
-o smtp_helo_name=mx1.example.com
-o smtp_bind_address=1.1.1.1
-o syslog_name=smtp-mx1
smtp2 unix - - n - - smtp
-o myhostname=mx2.example.com
-o smtp_helo_name=mx2.example.com
-o smtp_bind_address=2.2.2.2
-o syslog_name=smtp-mx2
# ------->%----------------------------------
--version--
CentOS release 6.5
Postfix 2.11.0
--Mailbox locking methods--
flock fcntl dotlock
--Supported Lookup tables--
btree cidr environ fail hash internal ldap memcache mysql nis pcre
proxy regexp socketmap static tcp texthash unix
Thanks
Marcus
So I tried to split smtp using transport_maps and that works ok for
local mail destined to foreign destinations.
But it breaks the delivery of foreign mail destined to local
destinations with the error "mail for example.com loops back to
myself".
Where's the place place to "plug" the custom service
(postfix-smtp-roundrobin.pl)?
# main.cf
transport_maps = tcp:127.0.0.1:9999
# -------%<----------------------------------
# master.cf
smtp unix - - n - - smtp
relay unix - - n - - smtp
127.0.0.1:smtp inet n - n - - smtpd
1.1.1.1:smtp inet n - n - - smtpd
-o myhostname=mx1.example.com
-o smtp_helo_name=mx1.example.com
-o smtp_bind_address=1.1.1.1
-o syslog_name=smtpd-mx1
-o smtpd_tls_cert_file=/etc/pki/postfix/mx1.example.com.cert
-o smtpd_tls_key_file=/etc/pki/postfix/mx1.example.com.key
-o smtp_tls_cert_file=/etc/pki/postfix/mx1.example.com.cert
-o smtp_tls_key_file=/etc/pki/postfix/mx1.example.com.key
-o smtpd_sasl_local_domain=mx1.example.com
2.2.2.2:smtp inet n - n - - smtpd
-o myhostname=mx2.example.com
-o smtp_helo_name=mx2.example.com
-o smtp_bind_address=2.2.2.2
-o syslog_name=smtpd-mx2
-o smtpd_tls_cert_file=/etc/pki/postfix/mx2.example.com.cert
-o smtpd_tls_key_file=/etc/pki/postfix/mx2.example.com.key
-o smtp_tls_cert_file=/etc/pki/postfix/mx2.example.com.cert
-o smtp_tls_key_file=/etc/pki/postfix/mx2.example.com.key
-o smtpd_sasl_local_domain=mx2.example.com
# Round-robin outgoing smtp
# And the contents of /usr/local/sbin/postfix-smtp-roundrobin.pl
# can be found at https://gist.github.com/mvsantos/9786697
#
127.0.0.1:9999 inet n n n - 0 spawn
user=nobody argv=/usr/local/sbin/postfix-smtp-roundrobin.pl
smtp1 unix - - n - - smtp
-o myhostname=mx1.example.com
-o smtp_helo_name=mx1.example.com
-o smtp_bind_address=1.1.1.1
-o syslog_name=smtp-mx1
smtp2 unix - - n - - smtp
-o myhostname=mx2.example.com
-o smtp_helo_name=mx2.example.com
-o smtp_bind_address=2.2.2.2
-o syslog_name=smtp-mx2
# ------->%----------------------------------
--version--
CentOS release 6.5
Postfix 2.11.0
--Mailbox locking methods--
flock fcntl dotlock
--Supported Lookup tables--
btree cidr environ fail hash internal ldap memcache mysql nis pcre
proxy regexp socketmap static tcp texthash unix
Thanks
Marcus
Noel Jones
Mar 26, 2014, 4:10:49 PM3/26/14
to
On 3/26/2014 2:42 PM, MV wrote:
> I'm trying to use multiple smtp IPs without having multiple postfix instances.
>
> So I tried to split smtp using transport_maps and that works ok for
> local mail destined to foreign destinations.
> But it breaks the delivery of foreign mail destined to local
> destinations with the error "mail for example.com loops back to
> myself".
>
> Where's the place place to "plug" the custom service
> (postfix-smtp-roundrobin.pl)?
>
> # main.cf
> transport_maps = tcp:127.0.0.1:9999
>
> # -------%<----------------------------------
> # master.cf
>
> smtp unix - - n - - smtp
> relay unix - - n - - smtp
>
> 127.0.0.1:smtp inet n - n - - smtpd
>
> 1.1.1.1:smtp inet n - n - - smtpd
> -o myhostname=mx1.example.com
> -o smtp_helo_name=mx1.example.com
> -o smtp_bind_address=1.1.1.1
You can't set "smtp" options within the "smtpd" service.
> I'm trying to use multiple smtp IPs without having multiple postfix instances.
>
> So I tried to split smtp using transport_maps and that works ok for
> local mail destined to foreign destinations.
> But it breaks the delivery of foreign mail destined to local
> destinations with the error "mail for example.com loops back to
> myself".
>
> Where's the place place to "plug" the custom service
> (postfix-smtp-roundrobin.pl)?
>
> # main.cf
> transport_maps = tcp:127.0.0.1:9999
>
> # -------%<----------------------------------
> # master.cf
>
> smtp unix - - n - - smtp
> relay unix - - n - - smtp
>
> 127.0.0.1:smtp inet n - n - - smtpd
>
> 1.1.1.1:smtp inet n - n - - smtpd
> -o myhostname=mx1.example.com
> -o smtp_helo_name=mx1.example.com
> -o smtp_bind_address=1.1.1.1
Use separate postfix instances.
-- Noel Jones
Wietse Venema
Mar 26, 2014, 4:11:54 PM3/26/14
to
MV:
The SMTP client is for REMOTE deliveries.
Wietse
> I'm trying to use multiple smtp IPs without having multiple postfix instances.
>
> So I tried to split smtp using transport_maps and that works ok for
> local mail destined to foreign destinations.
> But it breaks the delivery of foreign mail destined to local
> destinations with the error "mail for example.com loops back to
> myself".
Don't route LOCAL destinations to the "smtp" delivery agent.
>
> So I tried to split smtp using transport_maps and that works ok for
> local mail destined to foreign destinations.
> But it breaks the delivery of foreign mail destined to local
> destinations with the error "mail for example.com loops back to
> myself".
The SMTP client is for REMOTE deliveries.
Wietse
MV
Mar 26, 2014, 4:42:16 PM3/26/14
to
Noel Jones wrote:
fair point.. they shouldn't be there.
But that doesn't solve the initial problem.
> Use separate postfix instances.
That's not an option.
> MV wrote:
>>
>> 1.1.1.1:smtp inet n - n - - smtpd
>> -o myhostname=mx1.example.com
>> -o smtp_helo_name=mx1.example.com
>> -o smtp_bind_address=1.1.1.1
>
> You can't set "smtp" options within the "smtpd" service.
smtp options won't affect the smtpd service (they are harmless), but
>>
>> 1.1.1.1:smtp inet n - n - - smtpd
>> -o myhostname=mx1.example.com
>> -o smtp_helo_name=mx1.example.com
>> -o smtp_bind_address=1.1.1.1
>
> You can't set "smtp" options within the "smtpd" service.
fair point.. they shouldn't be there.
But that doesn't solve the initial problem.
> Use separate postfix instances.
That's not an option.
Noel Jones
Mar 26, 2014, 4:58:16 PM3/26/14
to
On 3/26/2014 3:42 PM, MV wrote:
> Noel Jones wrote:
>> MV wrote:
>>>
>>> 1.1.1.1:smtp inet n - n - - smtpd
>>> -o myhostname=mx1.example.com
>>> -o smtp_helo_name=mx1.example.com
>>> -o smtp_bind_address=1.1.1.1
>>
>> You can't set "smtp" options within the "smtpd" service.
> smtp options won't affect the smtpd service (they are harmless), but
> fair point.. they shouldn't be there.
> But that doesn't solve the initial problem.
Don't use the smtp transport for local addresses.
> Noel Jones wrote:
>> MV wrote:
>>>
>>> 1.1.1.1:smtp inet n - n - - smtpd
>>> -o myhostname=mx1.example.com
>>> -o smtp_helo_name=mx1.example.com
>>> -o smtp_bind_address=1.1.1.1
>>
>> You can't set "smtp" options within the "smtpd" service.
> smtp options won't affect the smtpd service (they are harmless), but
> fair point.. they shouldn't be there.
> But that doesn't solve the initial problem.
-- Noel Jones
MV
Mar 26, 2014, 4:59:48 PM3/26/14
to
>Wietse wrote:
I guess this should've been my initial question:
how can I add a transport to REMOTE deliveries only?
Marcus
how can I add a transport to REMOTE deliveries only?
Marcus
MV
Mar 26, 2014, 5:02:39 PM3/26/14
to
> Noel Jones wrote:
> Don't use the smtp transport for local addresses.
And how would I do that? I mean how can I have custom smtp for REMOTE
> Don't use the smtp transport for local addresses.
deliveries and not breaking the LOCAL deliveries,
Marcus
Wietse Venema
Mar 26, 2014, 5:03:53 PM3/26/14
to
MV:
> > Wietse:
By having the transport map reply "not found" for local destinations.
Wietse
> > Wietse:
Wietse
Noel Jones
Mar 26, 2014, 5:09:36 PM3/26/14
to
On 3/26/2014 3:59 PM, MV wrote:
>> Wietse wrote:
>
I see. You're using a TCP based table for your transport_maps and
it's incorrectly answering for local addresses too.
Don't do that. Configure your TCP table ignore local addresses. In
the context of a TCP table, the response should be something like
500 not found
-- Noel Jones
>> Wietse wrote:
>>> MV wrote:
>>> I'm trying to use multiple smtp IPs without having multiple postfix instances.
>>>
>>> So I tried to split smtp using transport_maps and that works ok for
>>> local mail destined to foreign destinations.
>>> But it breaks the delivery of foreign mail destined to local
>>> destinations with the error "mail for example.com loops back to
>>> myself".
>>
>> Don't route LOCAL destinations to the "smtp" delivery agent.
>> The SMTP client is for REMOTE deliveries.
>
> I guess this should've been my initial question:
> how can I add a transport to REMOTE deliveries only?
>
> Marcus
>>> I'm trying to use multiple smtp IPs without having multiple postfix instances.
>>>
>>> So I tried to split smtp using transport_maps and that works ok for
>>> local mail destined to foreign destinations.
>>> But it breaks the delivery of foreign mail destined to local
>>> destinations with the error "mail for example.com loops back to
>>> myself".
>>
>> Don't route LOCAL destinations to the "smtp" delivery agent.
>> The SMTP client is for REMOTE deliveries.
>
> I guess this should've been my initial question:
> how can I add a transport to REMOTE deliveries only?
>
>
I see. You're using a TCP based table for your transport_maps and
it's incorrectly answering for local addresses too.
Don't do that. Configure your TCP table ignore local addresses. In
the context of a TCP table, the response should be something like
500 not found
-- Noel Jones
Viktor Dukhovni
Mar 26, 2014, 5:10:42 PM3/26/14
to
On Wed, Mar 26, 2014 at 03:58:16PM -0500, Noel Jones wrote:
> Don't use the smtp transport for local addresses.
To attempt to evade receiving system client controls by spreading
> Don't use the smtp transport for local addresses.
load over multiple local IPs use sender_dependent_default_transport_maps.
If all you want is round-robin of the IPs, without regard to the
actual sender, you can use the socketmap table support to implement
the round-robin policy.
http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps
http://www.postfix.org/socketmap_table.5.html
--
Viktor.
MV
Mar 26, 2014, 8:00:58 PM3/26/14
to
> Wietse wrote:
> By having the transport map reply "not found" for local destinations.
> Noel Jones wrote:
> By having the transport map reply "not found" for local destinations.
> ... Configure your TCP table ignore local addresses. In
> the context of a TCP table, the response should be something like
> 500 not found
So, am I right to assume that by replying to smtp with "500 not found"
> 500 not found
it will fallback to local delivery?
Also I've noticed that different queries are sent to the transport
map. Looking at the logs I see that early on in the request the
transport map is queried as follow
get *
get *
get b...@foreign.tld
get f...@mydomain.tld
and I can't figure out what those two initial "get *" queries are and
where they come from and why are they there. Any ideas?
Here's the complete log crop of an inbound email (from gmail to
mydomain) and the subsequent bounce mail back to gmail because of the
"look back to myself".
For clarity, the queries sent to the transport map are logged under
"smtp-roundrobin" and can be found after the string "(...)transport
service. Query: "
Mar 26 18:58:09 mailer2 smtpd-mx1/smtpd[22438]: connect from
mail-ob0-f178.google.com[209.85.214.178]:53853
Mar 26 18:58:09 mailer2 smtpd-mx1/smtpd[22438]: Anonymous TLS
connection established from
mail-ob0-f178.google.com[209.85.214.178]:53853: TLSv1 with cipher
ECDHE-RSA-RC4-SHA (128/128 bits)
Mar 26 18:58:09 mailer2 smtp-roundrobin[22444]: Using: 'smtp2:'
transport service. Query: get *
Mar 26 18:58:09 mailer2 smtp-roundrobin[22444]: Using: 'smtp1:'
transport service. Query: get *
Mar 26 18:58:09 mailer2 smtp-roundrobin[22444]: Using: 'smtp2:'
transport service. Query: get XXXX...@gmail.com
Mar 26 18:58:10 mailer2 smtp-roundrobin[22444]: Using: 'smtp1:'
transport service. Query: get f...@MYDOMAIN.com
Mar 26 18:58:10 mailer2 smtpd-mx1/smtpd[22438]: 3fvJW60bZtzyg2:
client=mail-ob0-f178.google.com[209.85.214.178]:53853
Mar 26 18:58:10 mailer2 postfix/cleanup[22445]: 3fvJW60bZtzyg2:
message-id=<CAAv8khP0+Ak91B=4=WvwE-vRPv6Bawz9D...@mail.gmail.com>
Mar 26 18:58:10 mailer2 postfix/qmgr[22435]: 3fvJW60bZtzyg2:
from=<XXXX...@gmail.com>, size=1728, nrcpt=1 (queue active)
Mar 26 18:58:10 mailer2 postfix/qmgr[22435]: amavisfeed:
default_destination_concurrency_positive_feedback feedback type 0
value at 5: 1
Mar 26 18:58:10 mailer2 postfix/qmgr[22435]: amavisfeed:
default_destination_concurrency_negative_feedback feedback type 0
value at 5: 1
Mar 26 18:58:10 mailer2 smtpd-mx1/smtpd[22438]: disconnect from
mail-ob0-f178.google.com[209.85.214.178]:53853
Mar 26 18:58:11 mailer2 postfix/smtpd[22449]: connect from
localhost.localdomain[127.0.0.1]:49877
Mar 26 18:58:11 mailer2 smtp-roundrobin[22444]: Using: 'smtp2:'
transport service. Query: get XXXX...@gmail.com
Mar 26 18:58:11 mailer2 smtp-roundrobin[22444]: Using: 'smtp1:'
transport service. Query: get f...@MYDOMAIN.com
Mar 26 18:58:11 mailer2 postfix/smtpd[22449]: 3fvJW7374jzyg3:
client=localhost.localdomain[127.0.0.1]:49877,
orig_queue_id=3fvJW60bZtzyg2,
orig_client=mail-ob0-f178.google.com[209.85.214.178]:53853
Mar 26 18:58:11 mailer2 postfix/cleanup[22445]: 3fvJW7374jzyg3:
message-id=<CAAv8khP0+Ak91B=4=WvwE-vRPv6Bawz9D...@mail.gmail.com>
Mar 26 18:58:11 mailer2 postfix/smtpd[22449]: disconnect from
localhost.localdomain[127.0.0.1]:49877
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: 3fvJW7374jzyg3:
from=<XXXX...@gmail.com>, size=2347, nrcpt=1 (queue active)
Mar 26 18:58:11 mailer2 smtp-roundrobin[22444]: Using: 'smtp2:'
transport service. Query: get f...@MYDOMAIN.com
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: smtp2:
default_destination_concurrency_positive_feedback feedback type 0
value at 5: 1
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: smtp2:
default_destination_concurrency_negative_feedback feedback type 0
value at 5: 1
Mar 26 18:58:11 mailer2 amavis[13883]: (13883-13) Passed CLEAN
{RelayedInbound}, [209.85.214.178]:53853 [209.85.214.178]
<XXXX...@gmail.com> -> <f...@MYDOMAIN.com>, Queue-ID: 3fvJW60bZtzyg2,
Message-ID: <CAAv8khP0+Ak91B=4=WvwE-vRPv6Bawz9D...@mail.gmail.com>,
mail_id: 1UI9pBamXvhy, Hits: 0.012, size: 1728, queued_as:
3fvJW7374jzyg3, dkim_sd=20120113:gmail.com, 1058 ms
Mar 26 18:58:11 mailer2 postfix/lmtp[22446]: 3fvJW60bZtzyg2:
to=<f...@MYDOMAIN.com>, relay=127.0.0.1[127.0.0.1]:10111, delay=1.6,
delays=0.52/0/0/1.1, dsn=2.0.0, status=sent (250 2.0.0 from
MTA(smtp:[127.0.0.1]:10211): 250 2.0.0 Ok: queued as 3fvJW7374jzyg3)
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle: feedback 1
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle:
queue [127.0.0.1]:10111: limit 20 window 6 success 0 failure 0
fail_cohorts 0
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: 3fvJW60bZtzyg2: removed
Mar 26 18:58:11 mailer2 smtp-mx2/smtp[22450]: 3fvJW7374jzyg3:
to=<f...@MYDOMAIN.com>, relay=none, delay=0.14, delays=0.1/0.01/0.03/0,
dsn=5.4.6, status=bounced (mail for MYDOMAIN.com loops back to myself)
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle: feedback 1
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle:
queue MYDOMAIN.com: limit 20 window 6 success 0 failure 0 fail_cohorts
0
Mar 26 18:58:11 mailer2 postfix/cleanup[22445]: 3fvJW74mxHzyg5:
message-id=<3fvJW74...@mailer2.MYDOMAIN.com>
Mar 26 18:58:11 mailer2 postfix/bounce[22451]: 3fvJW7374jzyg3: sender
non-delivery notification: 3fvJW74mxHzyg5
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: 3fvJW74mxHzyg5: from=<>,
size=4382, nrcpt=1 (queue active)
Mar 26 18:58:11 mailer2 smtp-roundrobin[22444]: Using: 'smtp1:'
transport service. Query: get XXXX...@gmail.com
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: smtp1:
default_destination_concurrency_positive_feedback feedback type 0
value at 5: 1
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: smtp1:
default_destination_concurrency_negative_feedback feedback type 0
value at 5: 1
Mar 26 18:58:11 mailer2 postfix/qmgr[22435]: 3fvJW7374jzyg3: removed
Mar 26 18:58:12 mailer2 smtp-mx1/smtp[22452]: Trusted TLS connection
established to gmail-smtp-in.l.google.com[173.194.78.26]:25: TLSv1.2
with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
Mar 26 18:58:12 mailer2 smtp-mx1/smtp[22452]: 3fvJW74mxHzyg5:
to=<XXXX...@gmail.com>,
relay=gmail-smtp-in.l.google.com[173.194.78.26]:25, delay=1.2,
delays=0.07/0.01/0.57/0.57, dsn=2.0.0, status=sent (250 2.0.0 OK
1395860292 ek7si2054896wib.117 - gsmtp)
Mar 26 18:58:12 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle: feedback 1
Mar 26 18:58:12 mailer2 postfix/qmgr[22435]: qmgr_queue_unthrottle:
queue gmail.com: limit 20 window 6 success 0 failure 0 fail_cohorts 0
Mar 26 18:58:12 mailer2 postfix/qmgr[22435]: 3fvJW74mxHzyg5: removed
Wietse Venema
Mar 26, 2014, 8:04:28 PM3/26/14
to
MV:
In other words, RTFM.
Wietse
> Also I've noticed that different queries are sent to the transport
> map. Looking at the logs I see that early on in the request the
> transport map is queried as follow
> get *
> get *
> get b...@foreign.tld
> get f...@mydomain.tld
>
> and I can't figure out what those two initial "get *" queries are and
> where they come from and why are they there. Any ideas?
These secrets are revealed in http://www.postfix.org/transport.5.html
> map. Looking at the logs I see that early on in the request the
> transport map is queried as follow
> get *
> get *
> get b...@foreign.tld
> get f...@mydomain.tld
>
> and I can't figure out what those two initial "get *" queries are and
> where they come from and why are they there. Any ideas?
In other words, RTFM.
Wietse
MV
Mar 27, 2014, 8:04:34 AM3/27/14
to
> Wietse Venema wrote:
> In other words, RTFM.
I'd love to say I haven't read the manual and thank you for pointing
it out to me, but my OCD is too damn high, so I always read manuals.
Unfortunately this time I can't quite get my head around it to figure
it out on my own how to correctly and sanely configure postfix to use
a different smtp delivery transports for mail destined to foreign
destinations.
I believe the below is irrelevant to issue I'm trying to solve, but
I'd to understand postfix better instead of just copy+paste or get
ready-made crafted answer.
The documentation says:
...
With lookups from indexed files such as DB or DBM, or from
networked tables such as NIS, LDAP or SQL, patterns are tried *in the
order as listed below*:
user+extension@domain transport:nexthop
user@domain transport:nexthop
domain transport:nexthop
.domain transport:nexthop
* transport:nexthop
...
TCP-BASED TABLES
Each lookup operation uses the entire recipient address once.
Thus, some.domain.hierarchy is not looked up via its parent domains,
nor is user+foo@domain looked up as user@domain.
...
As far as I can tell, in my case since I'm using sing tcp-base tables
some look ups are not performed and that's fine. But there are no
mentions to the change in the order which patterns are checked.
So am I wrong to expect to see the logs showing "get bar <at>
foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
As oppose to what I see now:
get *
get *
get bar <at> foreign.tld
get foo <at> mydomain.tld
Regards, Marcus
> In other words, RTFM.
I'd love to say I haven't read the manual and thank you for pointing
it out to me, but my OCD is too damn high, so I always read manuals.
Unfortunately this time I can't quite get my head around it to figure
it out on my own how to correctly and sanely configure postfix to use
a different smtp delivery transports for mail destined to foreign
destinations.
I believe the below is irrelevant to issue I'm trying to solve, but
I'd to understand postfix better instead of just copy+paste or get
ready-made crafted answer.
The documentation says:
...
With lookups from indexed files such as DB or DBM, or from
networked tables such as NIS, LDAP or SQL, patterns are tried *in the
order as listed below*:
user+extension@domain transport:nexthop
user@domain transport:nexthop
domain transport:nexthop
.domain transport:nexthop
* transport:nexthop
...
TCP-BASED TABLES
Each lookup operation uses the entire recipient address once.
Thus, some.domain.hierarchy is not looked up via its parent domains,
nor is user+foo@domain looked up as user@domain.
...
As far as I can tell, in my case since I'm using sing tcp-base tables
some look ups are not performed and that's fine. But there are no
mentions to the change in the order which patterns are checked.
So am I wrong to expect to see the logs showing "get bar <at>
foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
As oppose to what I see now:
get *
get *
get bar <at> foreign.tld
get foo <at> mydomain.tld
Regards, Marcus
Wietse Venema
Mar 27, 2014, 9:11:09 AM3/27/14
to
MV:
not expected to change, therefore it is cached once during the
initialization of a trivial-rewrite process instance.
Wietse
> As far as I can tell, in my case since I'm using sing tcp-base tables
> some look ups are not performed and that's fine. But there are no
> mentions to the change in the order which patterns are checked.
> So am I wrong to expect to see the logs showing "get bar <at>
> foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
> As oppose to what I see now:
> get *
> get *
The "*" is the documented wild-card pattern. The query result is
> some look ups are not performed and that's fine. But there are no
> mentions to the change in the order which patterns are checked.
> So am I wrong to expect to see the logs showing "get bar <at>
> foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
> As oppose to what I see now:
> get *
> get *
not expected to change, therefore it is cached once during the
initialization of a trivial-rewrite process instance.
Wietse
Viktor Dukhovni
Mar 27, 2014, 10:48:07 AM3/27/14
to
On Thu, Mar 27, 2014 at 12:04:34PM +0000, MV wrote:
> As far as I can tell, in my case since I'm using sing tcp-base tables
> some look ups are not performed and that's fine. But there are no
> mentions to the change in the order which patterns are checked.
> So am I wrong to expect to see the logs showing "get bar <at>
> foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
> As oppose to what I see now:
> get *
> get *
> As far as I can tell, in my case since I'm using sing tcp-base tables
> some look ups are not performed and that's fine. But there are no
> mentions to the change in the order which patterns are checked.
> So am I wrong to expect to see the logs showing "get bar <at>
> foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
> As oppose to what I see now:
> get *
> get *
> get bar <at> foreign.tld
> get foo <at> mydomain.tld
The documentation describes semantic precedence order in which
> get foo <at> mydomain.tld
results are used. With cached results, order of use is not
necessarily the same as order of lookup. When you're looking at
low level activity logs you encounter implementation details
which are not fixed and hence not documented.
--
Viktor.
MV
Mar 27, 2014, 12:50:25 PM3/27/14
to
>Wietse:
>> MV:
Got a much better picture now (and the of the diagrams at
http://www.postfix.org/OVERVIEW.html also helped)
Going back to my original question...
Lets say I have "transport_maps = hash:/etc/postfix/transport", and
the contents of /etc/postfix/transport are as follows:
# Begin /etc/postfix/transport
mydomain.ltd :
.mydomain.ltd :
* smtpX
# End /etc/postfix/transport
And in the master.cf file I have
# -------%<------------------------------------
smtpX unix - - n - - smtp
-o myhostname=X.mydomain.tld
-o smtp_helo_name=mx1.mydomain.tld
-o smtp_bind_address=1.1.1.1
-o syslog_name=smtpX
smtpY unix - - n - - smtp
-o myhostname=Y.mydomain.tld
-o smtp_helo_name=mx2.mydomain.tld
-o smtp_bind_address=2.2.2.2
-o syslog_name=smtpY
# ------->%------------------------------------
The config above should work as expected.
Now going a step further, how can I split the "*" (all non-local)
between smtpX and smtpY (without running multiple postfix instances) ?
Something like ...
mydomain.ltd :
.mydomain.ltd :
* smtpX
* smtpY
>> MV:
>> As far as I can tell, in my case since I'm using sing tcp-base tables
>> some look ups are not performed and that's fine. But there are no
>> mentions to the change in the order which patterns are checked.
>> So am I wrong to expect to see the logs showing "get bar <at>
>> foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
>> As oppose to what I see now:
>> get *
>> get *
>
>> some look ups are not performed and that's fine. But there are no
>> mentions to the change in the order which patterns are checked.
>> So am I wrong to expect to see the logs showing "get bar <at>
>> foreign.tld" and "get foo <at> mydomain.tld" before "get *"?
>> As oppose to what I see now:
>> get *
>> get *
>
> The "*" is the documented wild-card pattern. The query result is
> not expected to change, therefore it is cached once during the
> initialization of a trivial-rewrite process instance.
Thanks for explaining that.
> not expected to change, therefore it is cached once during the
> initialization of a trivial-rewrite process instance.
Got a much better picture now (and the of the diagrams at
http://www.postfix.org/OVERVIEW.html also helped)
Going back to my original question...
Lets say I have "transport_maps = hash:/etc/postfix/transport", and
the contents of /etc/postfix/transport are as follows:
# Begin /etc/postfix/transport
mydomain.ltd :
.mydomain.ltd :
* smtpX
# End /etc/postfix/transport
And in the master.cf file I have
# -------%<------------------------------------
smtpX unix - - n - - smtp
-o myhostname=X.mydomain.tld
-o smtp_helo_name=mx1.mydomain.tld
-o smtp_bind_address=1.1.1.1
-o syslog_name=smtpX
smtpY unix - - n - - smtp
-o myhostname=Y.mydomain.tld
-o smtp_helo_name=mx2.mydomain.tld
-o smtp_bind_address=2.2.2.2
-o syslog_name=smtpY
# ------->%------------------------------------
The config above should work as expected.
Now going a step further, how can I split the "*" (all non-local)
between smtpX and smtpY (without running multiple postfix instances) ?
Something like ...
mydomain.ltd :
.mydomain.ltd :
* smtpX
* smtpY
Viktor Dukhovni
Mar 27, 2014, 1:03:31 PM3/27/14
to
On Thu, Mar 27, 2014 at 04:50:25PM +0000, MV wrote:
> Now going a step further, how can I split the "*" (all non-local)
> between smtpX and smtpY (without running multiple postfix instances) ?
> Something like ...
>
> mydomain.ltd :
> .mydomain.ltd :
> * smtpX
> * smtpY
No. To get round-robit results, you need a dynamic transport reply.
> Now going a step further, how can I split the "*" (all non-local)
> between smtpX and smtpY (without running multiple postfix instances) ?
> Something like ...
>
> mydomain.ltd :
> .mydomain.ltd :
> * smtpX
> * smtpY
Thus socketmap or similar, with the replies provided by a program, not
a fixed key->value table.
Furthermore, because "*" is cached, you really don't want to use
"*" at all for dynamic transport resolution.
I answered your question upthread, use:
sender_dependent_default_transport_maps
For some reason you're still looking elsewhere...
--
Viktor.
MV
Mar 27, 2014, 2:23:39 PM3/27/14
to
>Viktor Dukhovni wrote:
> Furthermore, because "*" is cached, you really don't want to use
> "*" at all for dynamic transport resolution.
Thanks for your input RE the caching of the special pattern "*" results.
> Furthermore, because "*" is cached, you really don't want to use
> "*" at all for dynamic transport resolution.
> I answered your question upthread, use:
> sender_dependent_default_transport_maps
> For some reason you're still looking elsewhere...
want to use sender-based static "routes". I'm looking for a "random"
or round-robin-ish split of smtp that provides consistent "helo ..
hostname .. ip .. reverse-dns-lookup"
Anyways, I've got the round-robin working now, using transport_maps
and this script https://gist.github.com/mvsantos/9813415
But I'm finding the solution very ugly. There must be a more elegant
way to do dynamic smtp deliveries...
$ postconf mail_version
mail_version = 2.11.0
# master.cf
smtp1 unix - - n - - smtp
-o myhostname=X.mydomain.tld
-o smtp_helo_name=X.mydomain.tld
-o smtp_bind_address=1.1.1.1
-o syslog_name=smtp-1
smtp2 unix - - n - - smtp
-o myhostname=Y.mydomain.tld
-o smtp_helo_name=Y.mydomain.tld
-o smtp_bind_address=2.2.2.2
-o syslog_name=smtp-2
# End of master.cf
And for every mail sent or received I have 8 extra lines in my logs
(below, not in the correct order) and the time of processing has
increased a bit - not because of the extra logging, but because of the
transport_maps.
Let's see how it performs under load tomorrow, when I make it live...
smtp-roundrobin[1234]: Using: 'smtp2:' transport service. Query: get *
smtp-roundrobin[1234]: Using: 'smtp1:' transport service. Query: get *
smtp-roundrobin[1234]: Using: 'smtp2:' transport service. Query: get
b...@foreign.tld
smtp-roundrobin[1234]: Using: 'smtp1:' transport service. Query: get
f...@mydomain.tld
postfix/qmgr[4567]: smtp1:
default_destination_concurrency_positive_feedback feedback type 0
value at 5: 1
postfix/qmgr[4567]: smtp1:
value at 5: 1
default_destination_concurrency_negative_feedback feedback type 0
value at 5: 1
postfix/qmgr[4567]: smtp2:
value at 5: 1
default_destination_concurrency_positive_feedback feedback type 0
value at 5: 1
postfix/qmgr[4567]: smtp2:
value at 5: 1
Wietse Venema
Mar 27, 2014, 2:52:58 PM3/27/14
to
MV:
Wietse
> I don't want to use sender-based static "routes". I'm looking for
> a "random" or round-robin-ish split of smtp that provides consistent
> "helo .. hostname .. ip .. reverse-dns-lookup"
What is the legitimate use case for this kind of policy evasion?
> a "random" or round-robin-ish split of smtp that provides consistent
> "helo .. hostname .. ip .. reverse-dns-lookup"
Wietse
Viktor Dukhovni
Mar 27, 2014, 3:07:18 PM3/27/14
to
On Thu, Mar 27, 2014 at 06:23:39PM +0000, MV wrote:
> >Viktor Dukhovni wrote:
> > Furthermore, because "*" is cached, you really don't want to use
> > "*" at all for dynamic transport resolution.
> Thanks for your input RE the caching of the special pattern "*" results.
>
> > I answered your question upthread, use:
> > sender_dependent_default_transport_maps
> > For some reason you're still looking elsewhere...
>
> I can't use sender_dependent_default_transport_maps because I don't
> >Viktor Dukhovni wrote:
> > Furthermore, because "*" is cached, you really don't want to use
> > "*" at all for dynamic transport resolution.
> Thanks for your input RE the caching of the special pattern "*" results.
>
> > I answered your question upthread, use:
> > sender_dependent_default_transport_maps
> > For some reason you're still looking elsewhere...
>
> want to use sender-based static "routes". I'm looking for a "random"
> or round-robin-ish split of smtp that provides consistent "helo ..
> hostname .. ip .. reverse-dns-lookup"
Of course you can. You're just not listening carefully. Your
> or round-robin-ish split of smtp that provides consistent "helo ..
> hostname .. ip .. reverse-dns-lookup"
sender dependent maps would actually largely ignore the sender,
and just provide a round-robin response. The important part is
that this mechanism returns a "default_transport" which never
overrides local or other more specific transport information.
This mechanism (being a lookup on the sender, which is a message,
not recipient property) can help avoid unnecessarily splitting of
the envelope for multi-recipient mail. You should try to send all
the recipients of a message to the same default transport, therefore
you should have an modestly sized LRU cache within the round-robin
process that returns a cached answer for a cached sender, but
returns and caches a random answer for a new sender.
If many messages come predominantly from a single sender and (since
your traffic is likely single-recipient bulk mail, solicited or
otherwise) multi-recipient messages are rare, you may not need the
cache, since envelope splitting will not be useful, but the cache
might defeat the load-balancing you want.
> Anyways, I've got the round-robin working now, using transport_maps
> and this script https://gist.github.com/mvsantos/9813415
> But I'm finding the solution very ugly. There must be a more elegant
> way to do dynamic smtp deliveries...
sender_dependent_default_transport_maps
Don't bother logging the queries except briefly if you're unsure
it is working.
--
Viktor.
MV
Mar 27, 2014, 3:28:20 PM3/27/14
to
> Wietse:
far from it.
I'm in the business of (strictly subscription-only) "monitoring
stuff". I mean, as soon as an event happens the subscribers who signed
up to that kind of event must be alerted immediately (99.9999% of the
times subscribers get a single email per day). But because we have
grown considerably recently, we needed extra outbound IPs because we
started to hit some ESP's limit of mail sent per hour from a single
source IP.
So I added an extra IP to the existing stack and used iptables to
split the outbound connections using the nat table.
But that created a problem because postfix would picks up the hostname
used in the helo (and other stuff???) from IP 1 and iptables would
route the connection through IP 2, so the client would see the message
headers like this one
Received: from AAAA.mydomain.tld (BBBB.mydomain.tld. [1.1.1.1])
by mta.foreign.tld with ESMTPS id abcdf....
for <f...@foreign.tld>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Thu, 27 Mar 2014 ....
See how postfix helo was "AAAA.mydomain.tld" but the message actually
came from "BBBB.mydomain.tld".
And that simple header inconsistency bothers me (OCD)
So instead of splitting smtp using iptables I now use that script I
posted upthread, and the headers now look like this:
Received: from AAAA.mydomain.tld (AAAA.mydomain.tld. [1.1.1.1])
by mta.foreign.tld with ESMTPS id abcdf....
for <f...@foreign.tld>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Thu, 27 Mar 2014 ....
Received: from BBBB.mydomain.tld (BBBB.mydomain.tld. [2.2.2.2])
by mta.foreign.tld with ESMTPS id abcdf....
for <f...@foreign.tld>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Thu, 27 Mar 2014 ....
Is there a more elegant way achieve this?
> What is the legitimate use case for this kind of policy evasion?
Just to be clear, I'm not a spammer, if anything, I couldn't be more
far from it.
I'm in the business of (strictly subscription-only) "monitoring
stuff". I mean, as soon as an event happens the subscribers who signed
up to that kind of event must be alerted immediately (99.9999% of the
times subscribers get a single email per day). But because we have
grown considerably recently, we needed extra outbound IPs because we
started to hit some ESP's limit of mail sent per hour from a single
source IP.
So I added an extra IP to the existing stack and used iptables to
split the outbound connections using the nat table.
But that created a problem because postfix would picks up the hostname
used in the helo (and other stuff???) from IP 1 and iptables would
route the connection through IP 2, so the client would see the message
headers like this one
Received: from AAAA.mydomain.tld (BBBB.mydomain.tld. [1.1.1.1])
by mta.foreign.tld with ESMTPS id abcdf....
for <f...@foreign.tld>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Thu, 27 Mar 2014 ....
See how postfix helo was "AAAA.mydomain.tld" but the message actually
came from "BBBB.mydomain.tld".
And that simple header inconsistency bothers me (OCD)
So instead of splitting smtp using iptables I now use that script I
posted upthread, and the headers now look like this:
Received: from AAAA.mydomain.tld (AAAA.mydomain.tld. [1.1.1.1])
by mta.foreign.tld with ESMTPS id abcdf....
for <f...@foreign.tld>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Thu, 27 Mar 2014 ....
Received: from BBBB.mydomain.tld (BBBB.mydomain.tld. [2.2.2.2])
by mta.foreign.tld with ESMTPS id abcdf....
for <f...@foreign.tld>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Thu, 27 Mar 2014 ....
Is there a more elegant way achieve this?
MV
Mar 27, 2014, 3:44:46 PM3/27/14
to
Viktor Dukhovni:
Finding this thread
http://thread.gmane.org/gmane.mail.postfix.user/203958 has helped a
bit..
> sender dependent maps would actually largely ignore the sender,
But how to I define the sender (map), if that's not asking too much,
could you please provide me with an example file?
> and just provide a round-robin response. The important part is
> that this mechanism returns a "default_transport" which never
> overrides local or other more specific transport information.
> ...
> ... can help avoid unnecessarily splitting of
> the cache might defeat the load-balancing you want.
yep!
> Use socketmap (or the obsolete tcp table) with:
> sender_dependent_default_transport_maps
(mental note: this surely is the right way to do it, not the way I've done it.)
> Don't bother logging the queries except briefly if you're unsure
> it is working.
I'm just logging while debugging..
Marcus
> Of course you can. You're just not listening carefully. Your
I'm failing to grasp the concept and can't find any working examples online...
Finding this thread
http://thread.gmane.org/gmane.mail.postfix.user/203958 has helped a
bit..
> sender dependent maps would actually largely ignore the sender,
could you please provide me with an example file?
> and just provide a round-robin response. The important part is
> that this mechanism returns a "default_transport" which never
> overrides local or other more specific transport information.
> ... can help avoid unnecessarily splitting of
> the envelope for multi-recipient mail. You should try to send all
And therefore would cut the overhead I currently have, right?
> the cache might defeat the load-balancing you want.
> Use socketmap (or the obsolete tcp table) with:
> sender_dependent_default_transport_maps
> Don't bother logging the queries except briefly if you're unsure
> it is working.
Marcus
Viktor Dukhovni
Mar 27, 2014, 4:28:36 PM3/27/14
to
On Thu, Mar 27, 2014 at 07:44:46PM +0000, MV wrote:
> > sender dependent maps would actually largely ignore the sender,
>
> But how to I define the sender (map), if that's not asking too much,
> could you please provide me with an example file?
It is a program! Not a fixed mapping. It receives a sender address,
> > sender dependent maps would actually largely ignore the sender,
>
> But how to I define the sender (map), if that's not asking too much,
> could you please provide me with an example file?
and replies (with a possibly cached per-sender) answer which is
computed on a mostly-round-robin basis.
> > and just provide a round-robin response. The important part is
> > that this mechanism returns a "default_transport" which never
> > overrides local or other more specific transport information.
> > ...
> > ... can help avoid unnecessarily splitting of
> > the envelope for multi-recipient mail. You should try to send all
>
> And therefore would cut the overhead I currently have, right?
in many places in Postfix. Your transport switch needs to be fast,
(no verbose logging, ...) the rest is irrelevant.
> > the cache might defeat the load-balancing you want.
>
> yep!
from the same sender? If not the cache will help with multi-recipient
mail. If all mail is single-recipient, you don't need a cache.
--
Viktor.
MV
Mar 27, 2014, 4:48:17 PM3/27/14
to
Viktor Dukhovni wrote:
> It is a program! Not a fixed mapping. It receives a sender address,
> and replies (with a possibly cached per-sender) answer which is
> computed on a mostly-round-robin basis.
If that's not asking too much, could you please provide me with a
> It is a program! Not a fixed mapping. It receives a sender address,
> and replies (with a possibly cached per-sender) answer which is
> computed on a mostly-round-robin basis.
practical example or point me to where I could to find one?
> Don't know what overhead you have in mind.
this is irrelevant right now.
> Do you send a flood of separate messages each from the same sender?
If you mean sender=f...@mydomain.tld, then yes, most of the mail sent
comes from a single email account and is sent to a great number of
indivual recipients.
Marcus
Wietse Venema
Mar 27, 2014, 5:12:11 PM3/27/14
to
MV:
Wietse
> > Wietse:
> > What is the legitimate use case for this kind of policy evasion?
>
> Just to be clear, I'm not a spammer, if anything, I couldn't be more
> far from it.
> I'm in the business of (strictly subscription-only) "monitoring
> stuff". I mean, as soon as an event happens the subscribers who signed
> up to that kind of event must be alerted immediately (99.9999% of the
> times subscribers get a single email per day). But because we have
> grown considerably recently, we needed extra outbound IPs because we
> started to hit some ESP's limit of mail sent per hour from a single
> source IP.
In that case, arrange for whitelisting like ever legitimate sender does.
> > What is the legitimate use case for this kind of policy evasion?
>
> Just to be clear, I'm not a spammer, if anything, I couldn't be more
> far from it.
> I'm in the business of (strictly subscription-only) "monitoring
> stuff". I mean, as soon as an event happens the subscribers who signed
> up to that kind of event must be alerted immediately (99.9999% of the
> times subscribers get a single email per day). But because we have
> grown considerably recently, we needed extra outbound IPs because we
> started to hit some ESP's limit of mail sent per hour from a single
> source IP.
Wietse
Viktor Dukhovni
Mar 27, 2014, 5:18:53 PM3/27/14
to
On Thu, Mar 27, 2014 at 08:48:17PM +0000, MV wrote:
> > It is a program! Not a fixed mapping. It receives a sender address,
> > and replies (with a possibly cached per-sender) answer which is
> > computed on a mostly-round-robin basis.
>
> If that's not asking too much, could you please provide me with a
> practical example or point me to where I could to find one?
A small socketmap or tcp table service that ignores the lookup key
> > It is a program! Not a fixed mapping. It receives a sender address,
> > and replies (with a possibly cached per-sender) answer which is
> > computed on a mostly-round-robin basis.
>
> If that's not asking too much, could you please provide me with a
> practical example or point me to where I could to find one?
and returns the next element of a circular list of values with each
lookup is surely just a few lines of custom code. As for how to
write a socketmap or tcp table driver in general, your Google skills
are likely not too dissimilar from mine.
In perl the core lookup function boils down to:
@choices = ( ... );
my $count = @choices;
my $next = 0;
sub lookup { return $choices[$next++ % $count]; }
which ignores the lookup key and returns the next choice. You have
many choices of languages to choose from, and ways to run this as
a network or unix-domain socket service.
> > Don't know what overhead you have in mind.
>
> The overhead caused by my crappy perl script and its IO ops... but
> this is irrelevant right now.
quite fast.
> > Do you send a flood of separate messages each from the same sender?
>
> By sender, do you mean email address or just $mydomain ?
> If you mean sender=f...@mydomain.tld, then yes, most of the mail sent
> comes from a single email account and is sent to a great number of
If you have the same sender for many concurrent messages (not just
multiple recipients of a single message), then you would not use
the proposed cache, making the code simpler.
--
Viktor.
MV
Mar 27, 2014, 5:32:52 PM3/27/14
to
Wietse wrote:
> In that case, arrange for whitelisting like ever legitimate sender does.
I do that for Gmail, Yahoo, Microsoft, AOL .. and it works, so much
> In that case, arrange for whitelisting like ever legitimate sender does.
so that we have never been graylisted by any of those folks despite
the tens of thousands emails we send daily.
But I can't afford do that for the thousands of other ESPs that my clients use.
We actually stop emailing users after the second failure and let them
know about it upon login to the web dashboard.
Users normally contact their ESPs on their own and once sorted they
normally ask us to restore their alerts. And sometimes they just
create a new email account in a big free ESP .
0 new messages