How to log/archive full outgoing mails including BCC info?
Yang Zhang
complete form (all headers + payload) as received from clients? The
closest param I've found so far are always_bcc & friends, but these
lose some information (at least the BCC field). Thanks in advance for
any hints.
--
Yang Zhang
http://yz.mit.edu/
Adam Tauno Williams
> How do you get Postfix to keep a log of all outgoing mails, in their
> complete form (all headers + payload) as received from clients?
You can't. [Typically someone on this list will chime in that doing so
is trivial if you just read the docs; and pointedly provide no pointers
on how to accomplish the aforementioned item].
> The
> closest param I've found so far are always_bcc & friends, but these
> lose some information (at least the BCC field). Thanks in advance for
> any hints.
Yep, I know. Someone needs to write some kind of milter to accomplish
this. It is on my to-do list but I don't know when I'll get to it.
Jeroen Geilman
> How do you get Postfix to keep a log of all outgoing mails, in their
> closest param I've found so far are always_bcc& friends, but these
> lose some information (at least the BCC field). Thanks in advance for
> any hints.
Email does not have a "BCC field".
The mail client generates the messages that are caused by specifying BCC
recipients, adding each BCC recipient in turn.
If the message DID contain a "BCC" field, it would still be there when
the recipient received it.
This more or less defeats the entire purpose of the "BCC" functionality....
You need to look to your email client for logging that.
--
J.
Jeroen Geilman
> On Tue, 2010-09-21 at 11:56 -0700, Yang Zhang wrote:
>
>> How do you get Postfix to keep a log of all outgoing mails, in their
>> complete form (all headers + payload) as received from clients?
>>
Of course you can.
That's what always_bcc does.
> [Typically someone on this list will chime in that doing so
> is trivial if you just read the docs; and pointedly provide no pointers
> on how to accomplish the aforementioned item].
>
I just told you how.
>> The
>> closest param I've found so far are always_bcc& friends, but these
>> lose some information (at least the BCC field). Thanks in advance for
>> any hints.
>>
> Yep, I know. Someone needs to write some kind of milter to accomplish
> this. It is on my to-do list but I don't know when I'll get to it.
>
>
Perhaps a $read_RFC_first milter would help.
--
J.
Wietse Venema
> How do you get Postfix to keep a log of all outgoing mails, in their
> closest param I've found so far are always_bcc & friends, but these
> lose some information (at least the BCC field). Thanks in advance for
> any hints.
This function is not built into Postfix.
It is usually achieved with a small SMTP-based "null filter" that
forks the incoming mail stream into two output streams.
- The first output stream goes back to Postfix (as is normal with
an SMTP-based content filter).
- The second output stream is a new mail transaction with the
original MAIL FROM/RCPT TO information prepended to the message
content.
This "null filter" program sends "." on the first stream only after
the "." on the second stream produces a 250 reply. Otherwise it
just closes the first stream without sending "." and reports a
temporary error to the Postfix SMTP client.
Writing an SMTP-based "null filter" is easy in scripting languages
such as Perl.
Wietse
Adam Tauno Williams
> On 09/21/2010 09:01 PM, Adam Tauno Williams wrote:
> > On Tue, 2010-09-21 at 11:56 -0700, Yang Zhang wrote:
> >> complete form (all headers + payload) as received from clients?
> Of course you can.
> That's what always_bcc does.
No, that does *NOT* do this. Reread the original post.
Jeroen Geilman
The original post is confused, since he talks about "as received from
clients" in the same sentence as "headers and payload", as if these
refer to the same things.
Wietse provided a very simple and graciously corrected solution, but
don't for a minute think the OPs question was sane. it wasn't.
--
J.
Victor Duchovni
> How do you get Postfix to keep a log of all outgoing mails, in their
> complete form (all headers + payload) as received from clients? The
> closest param I've found so far are always_bcc & friends, but these
> lose some information (at least the BCC field). Thanks in advance for
> any hints.
The archives of this list contain suggestions of how to achieve this
with a regexp or pcre recipient_bcc_maps which generates copies of
the message without losing envelope information.
The alternative, as suggested by Wietse, is an SMTP "Y-adaptor" that forks
an SMTP transaction into two streams. I've been using this approach for
5+ years. Messages are transmitted in parallel to both the archive and
the real destination, but "." is sent to the archive stream first, and
if that fails, the transmission to the real destination is aborted.
This has proved quite reliable.
The main risk is that mail can be archived, but the main delivery could
tempfail, leading to multiple archive copies. It is useful to have
de-duplication logic in the archive-stream. And of course keep that
parallel to the archive next-hop as reliable as possible, in my case,
a local Postfix instance that does the real message routing.
--
Viktor.
Yang Zhang
<Victor....@morganstanley.com> wrote:
> On Tue, Sep 21, 2010 at 11:56:41AM -0700, Yang Zhang wrote:
>
>> How do you get Postfix to keep a log of all outgoing mails, in their
>> complete form (all headers + payload) as received from clients? The
>> closest param I've found so far are always_bcc & friends, but these
>> lose some information (at least the BCC field). Thanks in advance for
>> any hints.
>
> The archives of this list contain suggestions of how to achieve this
> with a regexp or pcre recipient_bcc_maps which generates copies of
> the message without losing envelope information.
Was this the technique?
http://www.arschkrebs.de/postfix/postfix_archive.shtml
I tried this, but it doesn't preserve envelope information (BCCs are
still lost - which makes sense to me and is what I expected).
>
> The alternative, as suggested by Wietse, is an SMTP "Y-adaptor" that forks
> an SMTP transaction into two streams. I've been using this approach for
> 5+ years. Messages are transmitted in parallel to both the archive and
> the real destination, but "." is sent to the archive stream first, and
> if that fails, the transmission to the real destination is aborted.
> This has proved quite reliable.
>
> The main risk is that mail can be archived, but the main delivery could
> tempfail, leading to multiple archive copies. It is useful to have
> de-duplication logic in the archive-stream. And of course keep that
> parallel to the archive next-hop as reliable as possible, in my case,
> a local Postfix instance that does the real message routing.
We were planning to handle things at our application level, but we
wanted to first see if there was a simple way to accomplish this
already with Postfix.
Thanks to the helpful responses.
Victor Duchovni
> > The archives of this list contain suggestions of how to achieve this
> > with a regexp or pcre recipient_bcc_maps which generates copies of
> > the message without losing envelope information.
>
> Was this the technique?
>
> http://www.arschkrebs.de/postfix/postfix_archive.shtml
Does the method described there use "recipient_bcc_maps" to preserve
the envelope recipient list (via "user@domain"@archive-domain encoding)?
If not, then this is not the right document. The right method is
in the list archives.
> I tried this, but it doesn't preserve envelope information (BCCs are
> still lost - which makes sense to me and is what I expected).
The suggestion in the list archives does not lose the message envelope,
but you need a delivery agent that processes the encoded envelope
recipients.
--
Viktor.
Yang Zhang
<Victor....@morganstanley.com> wrote:
> On Tue, Sep 21, 2010 at 01:19:00PM -0700, Yang Zhang wrote:
>
>> > The archives of this list contain suggestions of how to achieve this
>> > with a regexp or pcre recipient_bcc_maps which generates copies of
>> > the message without losing envelope information.
>>
>> Was this the technique?
>>
>> http://www.arschkrebs.de/postfix/postfix_archive.shtml
>
> Does the method described there use "recipient_bcc_maps" to preserve
> the envelope recipient list (via "user@domain"@archive-domain encoding)?
>
> If not, then this is not the right document. The right method is
> in the list archives.
That's what it does - it generates user@domain@bcc.invalid - but I was
looking for something that preserves the BCC headers.
>
>> I tried this, but it doesn't preserve envelope information (BCCs are
>> still lost - which makes sense to me and is what I expected).
>
> The suggestion in the list archives does not lose the message envelope,
> but you need a delivery agent that processes the encoded envelope
> recipients.
Not entirely sure what that last part means, but I haven't had much
luck digging up this email. Anybody know what this might be referring
to?
Thanks.
Victor Duchovni
> On Tue, Sep 21, 2010 at 3:05 PM, Victor Duchovni
> <Victor....@morganstanley.com> wrote:
> > On Tue, Sep 21, 2010 at 01:19:00PM -0700, Yang Zhang wrote:
> >
> >> > The archives of this list contain suggestions of how to achieve this
> >> > with a regexp or pcre recipient_bcc_maps which generates copies of
> >> > the message without losing envelope information.
> >>
> >> Was this the technique?
> >>
> >> http://www.arschkrebs.de/postfix/postfix_archive.shtml
> >
> > Does the method described there use "recipient_bcc_maps" to preserve
> > the envelope recipient list (via "user@domain"@archive-domain encoding)?
> >
> > If not, then this is not the right document. The right method is
> > in the list archives.
>
> That's what it does - it generates user@domain@bcc.invalid - but I was
> looking for something that preserves the BCC headers.
You are mightily confused. Bcc headers are never sent by the MUA,
so there is nothing to preserve. All that needs to be preserved is the
message content and envelope. The "Bcc" header is a user-interface
element, it is not a part of the message in-transit.
--
Viktor.
Yang Zhang
Good grief, I was mightily confused indeed. Thank you.
Victor Duchovni
> >> That's what it does - it generates user@domain@bcc.invalid - but I was
> >> looking for something that preserves the BCC headers.
> >
> > You are mightily confused. ?Bcc headers are never sent by the MUA,
> > so there is nothing to preserve. All that needs to be preserved is the
> > message content and envelope. The "Bcc" header is a user-interface
> > element, it is not a part of the message in-transit.
>
> Good grief, I was mightily confused indeed. Thank you.
It would not be a "Bcc" if the list of Bcc'd recipients travelled along
with the message...
--
Viktor.
Yang Zhang
Actually my model of the world was that the first SMTP server was the
one to strip them out, instead of the MUA!
Jeroen Geilman
Let me add the Duh to my original reply (which already explained about
BCC): how does it know it is the first SMTP server ?
Only the MUA knows it is the MUA.