Semantically valid "Received:" Headers.
D. Stussy
Received: <0 or more name-value pairs> ; <date>
However, for SMTP transmissions, RFC 2821 overrides:
Received: FROM <domain> BY <domain> <options> ; <date>
Where <options> includes the fields: VIA, WITH, ID, and FOR.
---
I'm having problems with a certain other mailer (MTA) that claims itself to be
an SMTP transport adding a header which is effectively (comment-stripped):
Received: ; <date>
The issue I have with this is that if that mail server receives its mail via
SMTP and passes it on via SMTP, the shouldn't the trace header format of RFC
2821 apply - thus making this [latter] "Received:" header invalid because it
lacks the required fields "FROM" and "BY"?
The problem isn't in software per se, but with my spam filters, because I don't
get email that has travelled by other than SMTP, so all received headers for
messages I get should have the "FROM" and "BY" fields. However, there is this
mailer which apparently inserts the above type of header (with an embedded
comment indicating such stuff as mail queue status and such) which my anti-spam
rule is killing its email. The older RFC 822 implies that received headers are
for host-to-host transfer tracing, not for what happens internally to a host
where the mail just passes through without delivery.
I want your opinions on this: Am I justified in denying email that routes
through a server that uses this MTA because it is inserting an EXTRANEOUS header
that is not RFC for the SMTP environment? I am aware that this may lower my
interoperability, but isn't RFC compliance the other person's problem?
PS: Since I don't generally get anything other than spam-mail from people
employing these servers, I don't care about denying the mail. What I'm worried
about is whether or not I've gone too far with RFC compliance.
Mail Guy
> RFC 2822 defines a SYNTACTICALLY valid, comment-stripped received header as:
>
> Received: <0 or more name-value pairs> ; <date>
>
> However, for SMTP transmissions, RFC 2821 overrides:
>
> Received: FROM <domain> BY <domain> <options> ; <date>
>
> Where <options> includes the fields: VIA, WITH, ID, and FOR.
>
> ---
> I'm having problems with a certain other mailer (MTA) that claims itself to be
> an SMTP transport adding a header which is effectively (comment-stripped):
>
> Received: ; <date>
>
> The issue I have with this is that if that mail server receives its mail via
> SMTP and passes it on via SMTP, the shouldn't the trace header format of RFC
> 2821 apply - thus making this [latter] "Received:" header invalid because it
> lacks the required fields "FROM" and "BY"?
rfc 2821 only applies 'When an SMTP server receives a message for
delivery or further processing, it MUST insert trace ("time stamp" or
"Received") information at the beginning of the message content'. Is
the MTA not putting in an RFC 2821 received header when it receives the
message?
>
> The problem isn't in software per se, but with my spam filters, because I don't
> get email that has travelled by other than SMTP, so all received headers for
> messages I get should have the "FROM" and "BY" fields. However, there is this
> mailer which apparently inserts the above type of header (with an embedded
> comment indicating such stuff as mail queue status and such) which my anti-spam
> rule is killing its email. The older RFC 822 implies that received headers are
> for host-to-host transfer tracing, not for what happens internally to a host
> where the mail just passes through without delivery.
>
> I want your opinions on this: Am I justified in denying email that routes
> through a server that uses this MTA because it is inserting an EXTRANEOUS header
> that is not RFC for the SMTP environment? I am aware that this may lower my
> interoperability, but isn't RFC compliance the other person's problem?
>
> PS: Since I don't generally get anything other than spam-mail from people
> employing these servers, I don't care about denying the mail. What I'm worried
> about is whether or not I've gone too far with RFC compliance.
>
Sounds like you are in violation of rfc 2821 sec 3.8.2
D. Stussy
>D. Stussy wrote:
>> RFC 2822 defines a SYNTACTICALLY valid, comment-stripped received header as:
>>
>> Received: <0 or more name-value pairs> ; <date>
>>
>> However, for SMTP transmissions, RFC 2821 overrides:
>>
>> Received: FROM <domain> BY <domain> <options> ; <date>
>>
>> Where <options> includes the fields: VIA, WITH, ID, and FOR.
>>
>> ---
>> I'm having problems with a certain other mailer (MTA) that claims itself to be
>> an SMTP transport adding a header which is effectively (comment-stripped):
>>
>> Received: ; <date>
>>
>> The issue I have with this is that if that mail server receives its mail via
>> SMTP and passes it on via SMTP, the shouldn't the trace header format of RFC
>> 2821 apply - thus making this [latter] "Received:" header invalid because it
>> lacks the required fields "FROM" and "BY"?
>
>rfc 2821 only applies 'When an SMTP server receives a message for
>delivery or further processing, it MUST insert trace ("time stamp" or
>"Received") information at the beginning of the message content'. Is
>the MTA not putting in an RFC 2821 received header when it receives the
>message?
That's not the issue. Yes, it is putting in the header when it receives the
message into its host, but between that and the header inserted by the next
host, it is inserting another received header. It is this second, extraneous
header that my spam filter rule is hitting and killing the email as a result.
>> The problem isn't in software per se, but with my spam filters, because I don't
>> get email that has travelled by other than SMTP, so all received headers for
>> messages I get should have the "FROM" and "BY" fields. However, there is this
>> mailer which apparently inserts the above type of header (with an embedded
>> comment indicating such stuff as mail queue status and such) which my anti-spam
>> rule is killing its email. The older RFC 822 implies that received headers are
>> for host-to-host transfer tracing, not for what happens internally to a host
>> where the mail just passes through without delivery.
>>
>> I want your opinions on this: Am I justified in denying email that routes
>> through a server that uses this MTA because it is inserting an EXTRANEOUS header
>> that is not RFC for the SMTP environment? I am aware that this may lower my
>> interoperability, but isn't RFC compliance the other person's problem?
>>
>> PS: Since I don't generally get anything other than spam-mail from people
>> employing these servers, I don't care about denying the mail. What I'm worried
>> about is whether or not I've gone too far with RFC compliance.
>
>Sounds like you are in violation of rfc 2821 sec 3.8.2
Perhaps, but on the surface, I disagree. The header that I'm matching and thus
killing the message because of SHOULDN'T BE THERE because it does not represent
a host-to-host transport of the message. Per the RFC's (2822, as supported by
822 for the purpose of the header), it is my opinion that this instance of the
header shouldn't even have been inserted into the message - and doing so in
itself is non-RFC. Under that position, I am killing a message for being
non-RFC and thus invalid.
Regardless of my purpose or method of detecting the message, what I am asking is
whether or not anyone feels that a received header that does NOT represent a
host-to-host transport of a message is RFC under any circumstance. I say it is
not, but seek opinions either confirming or disproving that.
Mail Guy
> That's not the issue. Yes, it is putting in the header when it receives the
> message into its host, but between that and the header inserted by the next
> host, it is inserting another received header. It is this second, extraneous
> header that my spam filter rule is hitting and killing the email as a result.
Extraneous headers can be put in for any reason. You shouldn't reject
emails because of that. Is there an rfc part I'm overlooking that says
you can't insert extraneous headers?
>>Sounds like you are in violation of rfc 2821 sec 3.8.2
>
>
> Perhaps, but on the surface, I disagree. The header that I'm matching and thus
> killing the message because of SHOULDN'T BE THERE because it does not represent
> a host-to-host transport of the message. Per the RFC's (2822, as supported by
> 822 for the purpose of the header), it is my opinion that this instance of the
> header shouldn't even have been inserted into the message - and doing so in
> itself is non-RFC. Under that position, I am killing a message for being
> non-RFC and thus invalid.
Just cause you aren't expecting it, doesn't mean its non-rfc. is there
some part that says you can't insert received headers willy-nilly?
otherwise, you are standing on shaky ground.
>
> Regardless of my purpose or method of detecting the message, what I am asking is
> whether or not anyone feels that a received header that does NOT represent a
> host-to-host transport of a message is RFC under any circumstance. I say it is
> not, but seek opinions either confirming or disproving that.
if it's not disallowed, then it must be allowed, i think.
Mail Guy
Neil W Rickert
> Received: ; <date>
No, you are not justified. Your spam filter is badly designed. It
should not even look at that header.
You are justified in rejecting the email because you happen to feel
like it. You are not justified in the claim that there is an RFC
violation involved.
Neil W Rickert
>Perhaps, but on the surface, I disagree. The header that I'm matching and thus
>killing the message because of SHOULDN'T BE THERE because it does not represent
>a host-to-host transport of the message.
Which RFC asserts that headers cannot be added when they represent
something other than host-to-host transport?
Where is it stated that a "Received:" header may not be added
for a local pipeline transport between a receiving MTA and
a virus scanner?
> Per the RFC's (2822, as supported by
>822 for the purpose of the header), it is my opinion that this instance of the
>header shouldn't even have been inserted into the message - and doing so in
>itself is non-RFC. Under that position, I am killing a message for being
>non-RFC and thus invalid.
Your are simply wrong in this case. By all means reject the mail on
your personal whim. But don't make the bogus claim that you are
rejecting for RFC non-compliance.
D. Stussy
In that respect, I look to the PURPOSE of the trace headers. The purpose of the
received header is to show which hosts (and in which order) a message has
traversed the network from submission to delivery. However, this use of the
header does NOT represent a transmission of the message between hosts (virtual
or real) but the INTERNAL workings of some MTA. That is outside of the purpose
as I read it from the RFC's and therefore, it shouldn't be there at all. At
most (and only defined for the SMTP environment), the RFC's do indicate the "ID"
field for the purpose of marking where a message may have been stored internally
while in transit (i.e. a mail-cache [or other] identifier), but that is NOT a
stand-alone piece of information (i.e. must be accompanied by a "FROM" and "BY"
pair of fields).
What happens inside a host where delivery or [re]submission hasn't occurred is
not relevant. If the message is transported as SMTP into and SMTP out from (and
without delivery to) the same [virtual] host, RFC 2821's format applies. That
means that any header without "FROM" and "BY" is invalid since the message
NEVER LEAVES the SMTP environment.
I do note that some message lists (especially those that work as programs
external to the mail server) will insert a header. However, in those cases, the
message was delivered and resubmitted - and as having left (and reentered) the
SMTP environment, a non-RFC-2821 header otherwise complying with RFC 2822 can be
inserted. However, such is not the case where a mail server runs separate
processes as a mail receiver and as a transmitter/queue-runner - it may be
separate processes, but it's the same server on the same host keeping the SMTP
environment intact.
Neil W Rickert
>On Sun, 6 Oct 2002, Mail Guy wrote:
>In that respect, I look to the PURPOSE of the trace headers. The purpose of the
>received header is to show which hosts (and in which order) a message has
>traversed the network from submission to delivery.
The purpose is to trace the processing of the mail.
This may often include:
Receive from network add a trace header
Pipe to procmail
Use a filter program
pipe to sendmail add a trace header
or
Receive from network add a trace header
pipe to majordomo
pipe to sendmail add a trace header
You seem to be asserting that these long established ways of
handling mail violate some RFC.
Provide a detailed citation (RFC, and section number) where these
long standing and well established practices are stated to be
contrary to the standards.
D. Stussy
>"D. Stussy" <kd6...@bde-arc.ampr.org> writes:
>>Perhaps, but on the surface, I disagree. The header that I'm matching and thus
>>killing the message because of SHOULDN'T BE THERE because it does not represent
>>a host-to-host transport of the message.
>
>Which RFC asserts that headers cannot be added when they represent
>something other than host-to-host transport?
Trace through the history of the purpose of the "Received:" header from RFC 2822
to 822 and earlier. It is clearly for identifying the path a message takes from
source host to destination host (as each transmission occurs).
Also think about the choice of keyword. If all a mail server is doing is
running through its queue of undelivered messages destined for other hosts, it
hasn't actually received anything. It had already received it (and may have
inserted a received header) when it initially queued it - not when it's trying
to export it. Common sense should be clear with this one.
>Where is it stated that a "Received:" header may not be added
>for a local pipeline transport between a receiving MTA and
>a virus scanner?
Wouldn't that entail if not imply delivery OUT of the SMTP environment? That
would already mean that the stricter format of RFC 2821 doesn't apply. [A
system shouldn't be scanning mail that is just passing through (not being
delivered or not having been submitted locally).
>>822 for the purpose of the header), it is my opinion that this instance of the
>>header shouldn't even have been inserted into the message - and doing so in
>>itself is non-RFC. Under that position, I am killing a message for being
>>non-RFC and thus invalid.
>
>Your are simply wrong in this case. By all means reject the mail on
>your personal whim. But don't make the bogus claim that you are
>rejecting for RFC non-compliance.
Why not? If a header were to be inserted at that point, then RFC 2821 dictates
that since the email is still in an SMTP environment, certain fields that are
optional per RFC 2822 are required. I see no reason why I cannot reject a
message for lacking a required field. The semantic reason for rejection would
be: Noncompliance with the standard.
Simplified situation:
1) Mail come into a host via SMTP.
2) Mail is queued for export to the next host (NOT delivered locally).
3) Mail leave the host via SMTP.
Action 1 causes a received header which must be RFC 2821 formatted to be
prepended. Same with action 3.
However, a received header is being inserted for action 2 as well. How can it
be considered that the message has ever left the SMTP environment if it weren't
delivered (and resubmitted) or transmitted in or out via some other protocol.
Therefore, IF a received header were to be inserted at this point, it must be
RFC 2821 compliant. I argue two points:
1) It is not proper to insert ANY header in that case, because nothing is
considered received, and
2) If a Received header were to be inserted at this point, then it MUST comply
with RFC 2821, which requires "FROM" and "BY" fields in addition to the
semicolon followed by a date.
What I am seeing is a mailer that is doing all three of these steps and
inserting a header for step 2 which is NOT RFC-2821. Since RFC 2821 indicates
"MUST" for these fields, why is rejecting a message based on such a header
considered wrong?
Is this a requirement that is unenforcable, and therefore a nullity?
Neil W Rickert
>On 6 Oct 2002, Neil W Rickert wrote:
>>"D. Stussy" <kd6...@bde-arc.ampr.org> writes:
>>>Perhaps, but on the surface, I disagree. The header that I'm matching and thus
>>>killing the message because of SHOULDN'T BE THERE because it does not represent
>>>a host-to-host transport of the message.
>>Which RFC asserts that headers cannot be added when they represent
>>something other than host-to-host transport?
>Trace through the history of the purpose of the "Received:" header from RFC 2822
>to 822 and earlier. It is clearly for identifying the path a message takes from
>source host to destination host (as each transmission occurs).
As far as I know, it does not exclude the path between software
components on a single system.
>Also think about the choice of keyword. If all a mail server is doing is
>running through its queue of undelivered messages destined for other hosts, it
>hasn't actually received anything. It had already received it (and may have
>inserted a received header) when it initially queued it - not when it's trying
>to export it. Common sense should be clear with this one.
Just as a point of fact, sendmail inserts its "Received:" header
when sending the mail. It determines the content of the data
fields when receiving, but the decision to add a "Received:" is
made when sending.
>>Where is it stated that a "Received:" header may not be added
>>for a local pipeline transport between a receiving MTA and
>>a virus scanner?
>Wouldn't that entail if not imply delivery OUT of the SMTP environment?
In that case, storing the mail on disk in the local queue is taking
it out of the SMTP environment. Sending from the queue is injecting
it back into the SMTP environment.
How far do you want to go in dividing up to whether in or out of the
SMTP environment? Surely it is a bogus categorization. "SMTP" is
the name of a transmission protocol, not of an environment in which
the message resides.
> [A
>system shouldn't be scanning mail that is just passing through (not being
>delivered or not having been submitted locally).
Why not?
Many organization choose to do virus filtering at their external mail
gateway, for both incoming and outgoing mail. In either direction
the mail may pass through other hosts both before and after the
filtering. If the mail is being forwarded via a ".forward", it might
pass through the virus filter twice, once in each direction.
>>>822 for the purpose of the header), it is my opinion that this instance of the
>>>header shouldn't even have been inserted into the message - and doing so in
>>>itself is non-RFC. Under that position, I am killing a message for being
>>>non-RFC and thus invalid.
>>Your are simply wrong in this case. By all means reject the mail on
>>your personal whim. But don't make the bogus claim that you are
>>rejecting for RFC non-compliance.
>Why not? If a header were to be inserted at that point, then RFC 2821 dictates
>that since the email is still in an SMTP environment, certain fields that are
>optional per RFC 2822 are required.
This is nonsense. The mail might be in transit between two software
components on the same system. I am not aware of any RFC that
prohibits inserting a header at that point.
> I see no reason why I cannot reject a
>message for lacking a required field. The semantic reason for rejection would
>be: Noncompliance with the standard.
You can reject the mail on a whim if you like. But don't claim
bogus justification.
>Simplified situation:
> 1) Mail come into a host via SMTP.
> 2) Mail is queued for export to the next host (NOT delivered locally).
> 3) Mail leave the host via SMTP.
Do you actually know what happens between steps 1 and 3? There might
be processing going on that warrants adding a header. Do you know
for a fact that there is no such processing?
Here is my assessment -- which I expect you don't want to hear.
1: You are writing a spam filter.
2: The spam filter is horribly misdesigned.
3: Because of this misdesign, the filter is looking at
"Received:" headers that it should not be examining.
4: The software is stumbling over the format of one of
these headers that it should not be examining.
5: You want to solve this problem by rejecting the mail on your
personal whim. (Well go ahead, you have that right). But
you also want to erroneously claim that the RFCs are somehow
giving you license to do this.
Mail Guy
>
> Trace through the history of the purpose of the "Received:" header from RFC 2822
> to 822 and earlier. It is clearly for identifying the path a message takes from
> source host to destination host (as each transmission occurs).
>
> Also think about the choice of keyword. If all a mail server is doing is
> running through its queue of undelivered messages destined for other hosts, it
> hasn't actually received anything. It had already received it (and may have
> inserted a received header) when it initially queued it - not when it's trying
> to export it. Common sense should be clear with this one.
>
>
>>Where is it stated that a "Received:" header may not be added
>>for a local pipeline transport between a receiving MTA and
>>a virus scanner?
>
>
> Wouldn't that entail if not imply delivery OUT of the SMTP environment? That
> would already mean that the stricter format of RFC 2821 doesn't apply. [A
> system shouldn't be scanning mail that is just passing through (not being
> delivered or not having been submitted locally).
>
>
Nope. Most virus scanners are set up as an intermediary that get
checked as they are transported through the server. They normally add
headers too, and don't have to have 'from' or 'by'.
Wouldn't someone else have seen a problem as big as this one right after
the new updated RFCs came out?
>>Your are simply wrong in this case. By all means reject the mail on
>>your personal whim. But don't make the bogus claim that you are
>>rejecting for RFC non-compliance.
This I believe :)
> Why not? If a header were to be inserted at that point, then RFC 2821 dictates
> that since the email is still in an SMTP environment, certain fields that are
> optional per RFC 2822 are required. I see no reason why I cannot reject a
> message for lacking a required field. The semantic reason for rejection would
> be: Noncompliance with the standard.
And the sender can claim noncompliance on your part for rejecting based
on a trace header. Pot calling kettle black i'd say.
>
> Simplified situation:
> 1) Mail come into a host via SMTP.
> 2) Mail is queued for export to the next host (NOT delivered locally).
> 3) Mail leave the host via SMTP.
>
> Action 1 causes a received header which must be RFC 2821 formatted to be
> prepended. Same with action 3.
>
Actually, under the rfc you don't have to add an RFC 2821 received
header in step 3. only when mail is received is it required. of
course, it CAN insert a received header. it doesn't have to though.
> However, a received header is being inserted for action 2 as well. How can it
> be considered that the message has ever left the SMTP environment if it weren't
> delivered (and resubmitted) or transmitted in or out via some other protocol.
> Therefore, IF a received header were to be inserted at this point, it must be
> RFC 2821 compliant. I argue two points:
>
> 1) It is not proper to insert ANY header in that case, because nothing is
> considered received, and
You may not think it proper, but i don't see where it is forbidden.
>
> 2) If a Received header were to be inserted at this point, then it MUST comply
> with RFC 2821, which requires "FROM" and "BY" fields in addition to the
> semicolon followed by a date.
not if you are talking about a pipeline or something like that. all
sorts of virus scanners, procmail pipes, and filter programs would all
be noncompliant if you said this. and you havn't provided concrete rfc
evidence to the contrary.
>
> What I am seeing is a mailer that is doing all three of these steps and
> inserting a header for step 2 which is NOT RFC-2821. Since RFC 2821 indicates
> "MUST" for these fields, why is rejecting a message based on such a header
> considered wrong?
rfc 2821 section 3.8.2
>
> Is this a requirement that is unenforcable, and therefore a nullity?
>
you cannot enforce this w/o violating 2821, that would be against sec
3.8.2... of course, you can do it, but if you don't receive mail, you
can only blame yourself.
Mail Guy
D. Stussy
>"D. Stussy" <kd6...@bde-arc.ampr.org> writes:
>>On 6 Oct 2002, Neil W Rickert wrote:
>>>"D. Stussy" <kd6...@bde-arc.ampr.org> writes:
>
>>>>Perhaps, but on the surface, I disagree. The header that I'm matching and thus
>>>>killing the message because of SHOULDN'T BE THERE because it does not represent
>>>>a host-to-host transport of the message.
>
>>>Which RFC asserts that headers cannot be added when they represent
>>>something other than host-to-host transport?
>
>>Trace through the history of the purpose of the "Received:" header from RFC 2822
>>to 822 and earlier. It is clearly for identifying the path a message takes from
>>source host to destination host (as each transmission occurs).
>
>As far as I know, it does not exclude the path between software
>components on a single system.
>
>>Also think about the choice of keyword. If all a mail server is doing is
>>running through its queue of undelivered messages destined for other hosts, it
>>hasn't actually received anything. It had already received it (and may have
>>inserted a received header) when it initially queued it - not when it's trying
>>to export it. Common sense should be clear with this one.
>
>Just as a point of fact, sendmail inserts its "Received:" header
>when sending the mail. It determines the content of the data
>fields when receiving, but the decision to add a "Received:" is
>made when sending.
However, it's still marking the event of having received the email, not some
internal manipulation of it.
>>>Where is it stated that a "Received:" header may not be added
>>>for a local pipeline transport between a receiving MTA and
>>>a virus scanner?
>
>>Wouldn't that entail if not imply delivery OUT of the SMTP environment?
>
>In that case, storing the mail on disk in the local queue is taking
>it out of the SMTP environment. Sending from the queue is injecting
>it back into the SMTP environment.
>
>How far do you want to go in dividing up to whether in or out of the
>SMTP environment? Surely it is a bogus categorization. "SMTP" is
>the name of a transmission protocol, not of an environment in which
>the message resides.
Then, just what do the RFC's mean when they say "SMTP environment?" This is
clearly more than just the SMTP transmission itself....
I am of the opinion that the act of queueing by itself does NOT remove a message
from the "SMTP environment" as long as the "envelope" information remains
intact (i.e. not discarded).
>>system shouldn't be scanning mail that is just passing through (not being
>>delivered or not having been submitted locally).
>
>Why not?
>
>Many organization choose to do virus filtering at their external mail
>gateway, for both incoming and outgoing mail. In either direction
>the mail may pass through other hosts both before and after the
>filtering. If the mail is being forwarded via a ".forward", it might
>pass through the virus filter twice, once in each direction.
But that is for mail that is either sourced or destined for its own domain, not
other people's domains (ignoring exceptions for "as requested").
...And just what processing could that be (please give an example). The host
which has received that mail and will send it on to the next hop is NOT the host
the mail is destined for (but a relay - e.g. MX'ed host).
>Here is my assessment -- which I expect you don't want to hear.
>
> 1: You are writing a spam filter.
> 2: The spam filter is horribly misdesigned.
> 3: Because of this misdesign, the filter is looking at
> "Received:" headers that it should not be examining.
> 4: The software is stumbling over the format of one of
> these headers that it should not be examining.
> 5: You want to solve this problem by rejecting the mail on your
> personal whim. (Well go ahead, you have that right). But
> you also want to erroneously claim that the RFCs are somehow
> giving you license to do this.
1) True.
2) False.
3) True - but only to the extent that all of my legitimate mail has proper RFC
2821 headers, and 25%+ of my spam-mail does not. As a characteristic of spam,
it became a valid reason to consider.
4) Nonsensical.
5) Spam is not my "personal whim."
Historically, received headers log the fact of communication BETWEEN SEPARATE
HOSTS (export from one with import by another), and I can find NO LEGITIMATE
REASON for logging an intra-host transfer in this context. Unlike mailing list
expansion, where a significant operation occurs (to the recipient list in the
envelope - or discarding and generating a new envelope), the simple act of
queueing a message for export is NOT a significant operation that has been
historically logged by its own header.
My question comes down to: Why is this now considered significant? We already
have the queue identifier from the header when the message was first received by
that host (the optional "ID" field). What purpose does it serve an EXTERNAL
person to know things such as a task/process number or user ID the mail server
ran as? That information will already be available in the system's log and is
of use ONLY to that system's administrator - and the queue ID is his index into
his system logs for the other info. The header does not serve to aid the
tracing of the message in any way.
Let me ask you another question: If you were to get a message that had a
received header without a semicolon in it (a required element per the loosest
format in RFC 2822), would you consider that message legitimate?
D. Stussy
Wrong - you misunderstood. Although the exporting host would not be adding it,
the importing host (for which this would be equivalent to step 1) would be.
>> However, a received header is being inserted for action 2 as well. How can it
>> be considered that the message has ever left the SMTP environment if it weren't
>> delivered (and resubmitted) or transmitted in or out via some other protocol.
>> Therefore, IF a received header were to be inserted at this point, it must be
>> RFC 2821 compliant. I argue two points:
>>
>> 1) It is not proper to insert ANY header in that case, because nothing is
>> considered received, and
>
>You may not think it proper, but i don't see where it is forbidden.
It fails the PURPOSE of the header - to mark host-to-host transfers of the
message. When passing through without delivery, hosts (as relays) are
BLACK-BOXES - i.e. what happens inside is irrelevant.
>> 2) If a Received header were to be inserted at this point, then it MUST comply
>> with RFC 2821, which requires "FROM" and "BY" fields in addition to the
>> semicolon followed by a date.
>
>not if you are talking about a pipeline or something like that. all
>sorts of virus scanners, procmail pipes, and filter programs would all
>be noncompliant if you said this. and you havn't provided concrete rfc
>evidence to the contrary.
Procmail only acts upon DELIVERY. Why do you keep bringing these things up when
I have already indicated that a condition (i.e. delivery) has NOT occurred?
Virus scanners: Since when it is valid to scan and kill mail that is not either
sourced from or destined to your own host?
>> What I am seeing is a mailer that is doing all three of these steps and
>> inserting a header for step 2 which is NOT RFC-2821. Since RFC 2821 indicates
>> "MUST" for these fields, why is rejecting a message based on such a header
>> considered wrong?
>
>rfc 2821 section 3.8.2
>
>>
>> Is this a requirement that is unenforcable, and therefore a nullity?
>>
>
>you cannot enforce this w/o violating 2821, that would be against sec
>3.8.2... of course, you can do it, but if you don't receive mail, you
>can only blame yourself.
So, you have effectively said that I cannot enforce RFC 2821 section 4.4 without
violating RFC 2821 section 3.8.2. Then, what is the point of the standard
defined in 4.4? If it's unenforcable, then why even bother with the standard?
Neil W Rickert
>So, you have effectively said that I cannot enforce RFC 2821 section 4.4 without
>violating RFC 2821 section 3.8.2. Then, what is the point of the standard
>defined in 4.4? If it's unenforcable, then why even bother with the standard?
No. I believe he is saying that you cannot enforce STUSSY 2821
Neil W Rickert
>On 6 Oct 2002, Neil W Rickert wrote:
>>Just as a point of fact, sendmail inserts its "Received:" header
>>when sending the mail. It determines the content of the data
>>fields when receiving, but the decision to add a "Received:" is
>>made when sending.
>However, it's still marking the event of having received the email, not some
>internal manipulation of it.
When sendmail pipes the message to majordomo, and then majordomo
remails it, and extra "Received:" header is added corresponding to
internal manipution of the message on that host.
Before the milter API existed, there used to be a whole range
of applications which worked along the line of the MTA piping
to a processor which did something (made a copy, virus scanned,
added a disclaimer, etc) and then resent to the MTA. Before now, nobody
has ever suggested that this in any way violates RFC 2821.
But now your interpretation seems to be that it was a violation.
I say that your interpretation if RFC 2821 is seriously mistaken on
this point.
Mail Guy
> Wrong - you misunderstood. Although the exporting host would not be adding it,
> the importing host (for which this would be equivalent to step 1) would be.
>
>
>>>However, a received header is being inserted for action 2 as well. How can it
>>>be considered that the message has ever left the SMTP environment if it weren't
>>>delivered (and resubmitted) or transmitted in or out via some other protocol.
>>>Therefore, IF a received header were to be inserted at this point, it must be
>>>RFC 2821 compliant. I argue two points:
so you are saying that the MTA's subcomponents are talking to each other
via SMTP between themselves? Normally it would be done via shared
memory or pipes or files. If the subcomponents do talk SMTP between
themselves, then ya, they should put in rfc2821 received lines.
Otherwise, no, they don't since that rfc applies to SMTP transactions only.
> It fails the PURPOSE of the header - to mark host-to-host transfers of the
> message. When passing through without delivery, hosts (as relays) are
> BLACK-BOXES - i.e. what happens inside is irrelevant.
Ok, maybe you don't wanna know that an internal manipulation occurred...
so strip it out then. Question: where is it forbidden to add the
header? That what an rfc is all about... is there a 'should not' or a
'must not' anywhere with regard to this?
Also, what would convince you that you are wrong besides the fact that
hundreds if not thousands of filters/virus scanners/spam filters/mtas
put received lines into the headers on their whim? No one (from my
searches so far) has ever brought this point up except you. A direct
'must not' with regard to your rejection of mail didn't seem to faze
you. would a call from the rfc writer telling you were wrong convince
you?
> Virus scanners: Since when it is valid to scan and kill mail that is not either
> sourced from or destined to your own host?
All the time. Your organization probably does this. Have a central
virus scanning computer receive all email, then pass non-virused email
through to get delivered to the real destination. It's not sourced, or
destined, and if you find somewhere in the rfc that forbids this, a lot
of people (mcaffee, command software, others) will be quite pissed off.
I would think at least someone would have said something if this was
that bad.
> So, you have effectively said that I cannot enforce RFC 2821 section 4.4 without
> violating RFC 2821 section 3.8.2. Then, what is the point of the standard
> defined in 4.4? If it's unenforcable, then why even bother with the standard?
Maybe in a future rfc section 3.8.2 might be taken out, then you would
be able to do what you want to do (kill the mail) without violating rfc.
sec 4.4 is put in so that there is a consistent looking received line
when an smtp server receives something. but 3.8.2 will probably never
be taken out because of legacy intermediary systems or things like virus
scanners/mtas.
Mark Crispin
> No. I believe he is saying that you cannot enforce STUSSY 2821
> section 4.4 without violating RFC 2821 section 3.8.2.
I'm afraid that I have to agree with Neil's conclusion. Mr. Stussy seems
to be reading much more into RFC 2821 section 4.4 than what it actually
says.
I'm sympathetic to Mr. Stussy's intention to write an effective spam
filter. His techniques may be effective in catching spam. However, these
techniques will also falsely label legitimate mail as being spam; and when
(not if) that happens the blame is in the spam filter that uses those
techniques.
-- Mark --
http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
D. Stussy
>"D. Stussy" <kd6...@bde-arc.ampr.org> writes:
>>On 6 Oct 2002, Neil W Rickert wrote:
>
>>>Just as a point of fact, sendmail inserts its "Received:" header
>>>when sending the mail. It determines the content of the data
>>>fields when receiving, but the decision to add a "Received:" is
>>>made when sending.
>
>>However, it's still marking the event of having received the email, not some
>>internal manipulation of it.
>
>When sendmail pipes the message to majordomo, and then majordomo
>remails it, and extra "Received:" header is added corresponding to
>internal manipution of the message on that host.
That involves a DELIVERY of the message. That's already excluded.
What part of "delivery of a message" don't you all understand?
>Before the milter API existed, there used to be a whole range
>of applications which worked along the line of the MTA piping
>to a processor which did something (made a copy, virus scanned,
>added a disclaimer, etc) and then resent to the MTA. Before now, nobody
>has ever suggested that this in any way violates RFC 2821.
>
>But now your interpretation seems to be that it was a violation.
>
>I say that your interpretation if RFC 2821 is seriously mistaken on
>this point.
Wrong. Those "interceptions" take the mail OUT of the SMTP environment by
delivering the message to a local process OUTSIDE of the mail server system.
However, if they did not remove the message from that environment AND they also
insert a received header, then if that header doesn't comply with the RFC 2821
format, then yes, there would be a violation.
D. Stussy
Wrong again.
RFC 2821 section 4.4 indicates that certain fields (FROM and BY) MUST be
present in a Received Header.
RFC 2821 section 3.8.2 says that we can't reject a message SOLELY based of the
format of a received header.
This effectively means that the format can be anything, so how do you reconcile
this against the format that MUST be inserted and used defined in section 4.4?
If a certain format MUST be used, but we're not allowed to check for that
format, then why have the requirement? It's not enforcable. What's the point?
D. Stussy
>D. Stussy wrote:
>> Wrong - you misunderstood. Although the exporting host would not be adding it,
>> the importing host (for which this would be equivalent to step 1) would be.
>>
>>>>However, a received header is being inserted for action 2 as well. How can it
>>>>be considered that the message has ever left the SMTP environment if it weren't
>>>>delivered (and resubmitted) or transmitted in or out via some other protocol.
>>>>Therefore, IF a received header were to be inserted at this point, it must be
>>>>RFC 2821 compliant. I argue two points:
>
>so you are saying that the MTA's subcomponents are talking to each other
>via SMTP between themselves? Normally it would be done via shared
>memory or pipes or files. If the subcomponents do talk SMTP between
>themselves, then ya, they should put in rfc2821 received lines.
>Otherwise, no, they don't since that rfc applies to SMTP transactions only.
I didn't say that. However, what I said is that as long as the message comes in
by use of SMTP, leaves by use of SMTP, and isn't delivered (i.e. where the SMTP
envelope information is tossed away in the bit bucket and NOT kept), then the
message NEVER LEAVES THE SMTP ENVIRONMENT, and thus any received header that is
inserted must comply with the stricter RFC 2821 format.
Piping a message from a mail server's listener to the queue is not an operation
where the message leaves that environment, especially since the SMTP envelope
information is kept and passed on as is. That is an INTERNAL operation which
does not merit a received header, but if one were to be inserted anyway, then it
must insert a properly formatted header (see above).
>> It fails the PURPOSE of the header - to mark host-to-host transfers of the
>> message. When passing through without delivery, hosts (as relays) are
>> BLACK-BOXES - i.e. what happens inside is irrelevant.
>
>Ok, maybe you don't wanna know that an internal manipulation occurred...
>so strip it out then. Question: where is it forbidden to add the
>header? That what an rfc is all about... is there a 'should not' or a
>'must not' anywhere with regard to this?
Try using common sense to fill in what isn't stated. Just what would be the
purpose of a header such as:
Received: from x.net by x.net ; <some_random_date>
It doesn't tell you anything new because there was already a header indicating
"... by x.net" when the message came in from a remote host, and there will be a
header "from x.net ..." added by the next host when the message is transmitted
to it. Both of the received headers bracketing the above one would already
contain all the information. Not even the date field produces anything useful -
because we already know from the other headers that it was there then.
>Also, what would convince you that you are wrong besides the fact that
>hundreds if not thousands of filters/virus scanners/spam filters/mtas
>put received lines into the headers on their whim? No one (from my
>searches so far) has ever brought this point up except you. A direct
>'must not' with regard to your rejection of mail didn't seem to faze
>you. would a call from the rfc writer telling you were wrong convince
>you?
Just because they are doing it doesn't make it correct usage.
If I must not reject a message based on the format of a received header, then
just WHY is one required to insert a received header as a result of an SMTP
transmission formatted in any particular way (let alone RFC 2821 section 4.4)?
If I cannot reject the message when this format applies but isn't followed, then
why define the format?
>> Virus scanners: Since when it is valid to scan and kill mail that is not either
>> sourced from or destined to your own host?
>
>All the time. Your organization probably does this. Have a central
>virus scanning computer receive all email, then pass non-virused email
>through to get delivered to the real destination. It's not sourced, or
>destined, and if you find somewhere in the rfc that forbids this, a lot
>of people (mcaffee, command software, others) will be quite pissed off.
> I would think at least someone would have said something if this was
>that bad.
So, if "aol.com" (or any large domain with millions of mailboxes) were to start
using my mail server as a relay, I could KILL any message of theirs at a whim?
>> So, you have effectively said that I cannot enforce RFC 2821 section 4.4 without
>> violating RFC 2821 section 3.8.2. Then, what is the point of the standard
>> defined in 4.4? If it's unenforcable, then why even bother with the standard?
>
>>Maybe in a future rfc section 3.8.2 might be taken out, then you would
>be able to do what you want to do (kill the mail) without violating rfc.
> sec 4.4 is put in so that there is a consistent looking received line
>when an smtp server receives something. but 3.8.2 will probably never
>be taken out because of legacy intermediary systems or things like virus
>scanners/mtas.
That doesn't address the question. If a format MUST be used, but one MUST NOT
check that format, how can anyone enforce use of that format?
D. Stussy
>On Mon, 7 Oct 2002, Neil W Rickert wrote:
>> No. I believe he is saying that you cannot enforce STUSSY 2821
>> section 4.4 without violating RFC 2821 section 3.8.2.
>
>I'm afraid that I have to agree with Neil's conclusion. Mr. Stussy seems
>to be reading much more into RFC 2821 section 4.4 than what it actually
>says.
I don't believe that I am. It starts: "When an SMTP server receives a message
for delivery or FURTHER PROCESSING (emphasis added), it MUST insert trace ..."
"This line MUST be structured as follows: ..."
To me, this means that if it is not delivered to a mailbox (including a program
acting as a mailbox) or transmitted to another host via a protocol OTHER THAN [a
flavor of] SMTP, it hasn't left the "SMTP environment" and any other received
header for any other further processing at that host must also comply with the
format specified.
Section 4.4 indicates that a message HAS LEFT the "SMTP environment" when final
delivery occurs (which REQUIRES the "return-path" header be prepended).
For example, is "qmail", when handing a message off from its receiving process
into its queueing process, or subsequently to its queue runner, inserting the
"return path" header? If not (and I suspect not - because I don't see that as a
delivery of the message), then the e-mail has NOT left the SMTP environment, and
therefore, the message it inserts (something like "Received: (qmail uid 10
process 20 queue-id 40) ; <date>") is therefore invalid because RFC 2821
indicates that since it hasn't left that environment, "FROM" and "BY" fields are
required.
There's a real example for you of what I consider a violation of the RFC.
Since "leaving the SMTP environment" also means that the envelope information is
tossed, then just how would "qmail" know the identity of any BCC'ed recipients?
Without an SMTP envelope, it would have to scan the message for the recipients.
If it were passed the recipients on the command line (e.g.) for its pipe, then
it hasn't left the SMTP environment.... The "Return-path" header only logs the
SENDER of the message. Any BCC'ed recipient would be lost.
>I'm sympathetic to Mr. Stussy's intention to write an effective spam
>filter. His techniques may be effective in catching spam. However, these
>techniques will also falsely label legitimate mail as being spam; and when
>(not if) that happens the blame is in the spam filter that uses those
>techniques.
In my case, that has yet to happen. I have yet to reject any mail from any
person I already know.
Mail Guy
>
> I didn't say that. However, what I said is that as long as the message comes in
> by use of SMTP, leaves by use of SMTP, and isn't delivered (i.e. where the SMTP
> envelope information is tossed away in the bit bucket and NOT kept), then the
> message NEVER LEAVES THE SMTP ENVIRONMENT, and thus any received header that is
> inserted must comply with the stricter RFC 2821 format.
Could you demonstrate what the SMTP environment entails, since that is
your big issue? The SMTP environment cannot be an arbitrary thing,
right? I really think your lack of evidence shows you don't know much
about this MTAs in general.
This is an SMTP environment (ie. they must talk SMTP to each other):
One process connects to another process using TCP/IP
Server side process says: 220 crap.com ESMTP
The client says: helo foo.com
The server says: 250 crap.com
The client says: mail from: <f...@foo.com>
The server says: 250 ok
The client says: rcpt to: <f...@crap.com>
The server says: 250 ok
The client says: data
The server says: 354 go ahead
The client sends the email terminating w/a .
The server says: 250 ok 3434343434 qp 343434
At this point, sec 4.4 applies, the mta should insert the mail header
which complies to rfc 2821 sec 4.4.
If the mta decides that it wants to directly deliver the mail, that is
fine. It can insert a second received line if it wants. it is not
forbidden, depite your protestations that 'it is not proper', since it
has already satisfied sec 4.4 already.
If the mta decides that it does not want to directly deliver the mail
(let's say it's too busy to deliver right now), it delivers to another
agent which writes the email to disk. this agent writes a received line
saying 'i got it' then sends it to disk. was the email sent to this
intermediary agent via smtp? nope, it was sent via shared memory.
several hours later, the mail server is not busy anymore. the sending
process can now insert another received header if it wants, it doesn't
have to, but can because sec 4.4 is already satisfied. the sending
process talks SMTP to the next SMTP server, so.... yes, you guessed it,
the receiving server MUST put in a sec 4.4 compliant received header.
after that, it can insert 1000 received lines that are simply comment lines:
Received: ; 8 Oct 2002 02:36:00 -0000
It would suck to receive an email with that many lines, but it isn't
disallowed. See the elementary logic with rfcs is this:
if allowed, then allowed
if not allowed, then not allowed
if not disallowed, then allowed
sec 4.4 is already satisfied, no matter what you believe. so, rfc 2822
complaint headers are allowed.
>
> Piping a message from a mail server's listener to the queue is not an operation
> where the message leaves that environment, especially since the SMTP envelope
> information is kept and passed on as is. That is an INTERNAL operation which
> does not merit a received header, but if one were to be inserted anyway, then it
> must insert a properly formatted header (see above).
According to YOU it doesn't 'merit' a received header. I say it is and
I am sending the email. So, too bad, I will insert a comment cause this
is not an SMTP transaction:
Received: (just cause you don't like it); 8 Oct 2002 02:00:00 -0000
>>>It fails the PURPOSE of the header - to mark host-to-host transfers of the
>>>message. When passing through without delivery, hosts (as relays) are
>>>BLACK-BOXES - i.e. what happens inside is irrelevant.
>>
>>Ok, maybe you don't wanna know that an internal manipulation occurred...
>>so strip it out then. Question: where is it forbidden to add the
>>header? That what an rfc is all about... is there a 'should not' or a
>>'must not' anywhere with regard to this?
>
>
> Try using common sense to fill in what isn't stated. Just what would be the
> purpose of a header such as:
>
> Received: from x.net by x.net ; <some_random_date>
>
> It doesn't tell you anything new because there was already a header indicating
> "... by x.net" when the message came in from a remote host, and there will be a
> header "from x.net ..." added by the next host when the message is transmitted
> to it. Both of the received headers bracketing the above one would already
> contain all the information. Not even the date field produces anything useful -
> because we already know from the other headers that it was there then.
>
hmm... guess i don't see the part of the rfc's that says:
"Received headers must be as informative as possible, and describing
obvious things SHOULD NOT be stated as grounds to insert a header"
find something in the rfcs that says something like this, if you want to
refute it.
>
>>Also, what would convince you that you are wrong besides the fact that
>>hundreds if not thousands of filters/virus scanners/spam filters/mtas
>>put received lines into the headers on their whim? No one (from my
>>searches so far) has ever brought this point up except you. A direct
>>'must not' with regard to your rejection of mail didn't seem to faze
>>you. would a call from the rfc writer telling you were wrong convince
>>you?
>
>
> Just because they are doing it doesn't make it correct usage.
>
> If I must not reject a message based on the format of a received header, then
> just WHY is one required to insert a received header as a result of an SMTP
> transmission formatted in any particular way (let alone RFC 2821 section 4.4)?
> If I cannot reject the message when this format applies but isn't followed, then
> why define the format?
>
>
if you can actually find someone who knows something about email to
actually agree with you, i might give you some leeway. so far, my
interpretation is that you are making assumptions without proof, and
completely twisting the rfc to meet your agenda. if lots of smart
people said i was wrong, you can bet that i would take their word for it.
> So, if "aol.com" (or any large domain with millions of mailboxes) were to start
> using my mail server as a relay, I could KILL any message of theirs at a whim?
No, AOL's MX machine transfers all email to another machine that virus
scans all email (adds received headers like: Received: (virus scanner
2.0); 12 Oct 2002 01:01:00 -0000), kills the ones that have virii, and
pass the rest through to the destination machines.
>
>
>>>So, you have effectively said that I cannot enforce RFC 2821 section 4.4 without
>>>violating RFC 2821 section 3.8.2. Then, what is the point of the standard
>>>defined in 4.4? If it's unenforcable, then why even bother with the standard?
>>
>>>Maybe in a future rfc section 3.8.2 might be taken out, then you would
>>
>>be able to do what you want to do (kill the mail) without violating rfc.
>> sec 4.4 is put in so that there is a consistent looking received line
>>when an smtp server receives something. but 3.8.2 will probably never
>>be taken out because of legacy intermediary systems or things like virus
>>scanners/mtas.
>
>
> That doesn't address the question. If a format MUST be used, but one MUST NOT
> check that format, how can anyone enforce use of that format?
>
They can't enforce the format, as you are starting to understand.
Let's say there are only two ways to get to my house. I tell you what
one of those ways is. You decide not to follow that route, but you take
the other route. I cannot make you follow my route. That does not make
my directions any less correct. I am not going to turn you away at the
door because you didn't follow MY directions that I expected you to
follow. You made it to my house, and that's all that matters.
Mark Crispin
> I didn't say that. However, what I said is that as long as the message
> comes in by use of SMTP, leaves by use of SMTP, and isn't delivered
> (i.e. where the SMTP envelope information is tossed away in the bit
> bucket and NOT kept), then the message NEVER LEAVES THE SMTP
> ENVIRONMENT, and thus any received header that is inserted must comply
> with the stricter RFC 2821 format.
Golly gee.
I have been a developer of email software for 25+ years, contributed to
just about every major email RFC, even invented one of the major email
protocols. Yet, this is the first time that I ever heard of such a thing
of an "SMTP environment" that imposes requirements on the internal
operation of a mailer that Mr. Stussy envisions.
Mr. Stussy's first message was phrased as a question and request for
opinions. Answers and opinions have been provided. I fear that they were
not the answers and opinions that he desired.
I doubt that further discussion would be productive.
D. Stussy
But if the message has not left the SMTP environment, that second (and any
subsequent) received header MUST ALSO COMPLY with RFC 2821 section 4.4.
If the message has left the SMTP environment, then RFC 2821 section 4.4 mandates
that a "Return-Path" header be prepended to the message. Therefore, as long as
a received header anywhere in the message indicates that the message was
"Received: ... with SMTP ..." (or any flavor, ESMTP, SMTP8, etc., ...) and no
"Return-Path" header appears above it, we can clearly tell that it has not left
the SMTP environment (or else the message would be non-RFC anyway).
I agree that the SMTP environment starts no later than a message's first
transmission via SMTP (whether or not reflected with the optional "with" field
in the received headers). We can only confirm when the environment started from
looking at the chronologically first received header that has a "with" field
indicating SMTP, or if none do, then with our host since SMTP is how it is
coming in. We know when it ends by the presence of a "return-path" header. We
can also confirm that a message wasn't in an SMTP environment if there is a
different protocol stated in a "with" field of a received header.
This tells me that any Received header between the first header that has a "with
SMTP" field in it and the "return-path" header (or the top of the message if
none exists) MUST have occurred within the SMTP environment, and therefore must
meet the stricter RFC 2821 format, not the looser RFC 2822 format.
Although any intermediate "return-path" headers may be removed by the host
making a final delivery, they will still be present when the message comes in.
That should take care of any messages resubmitted into the SMTP environment
despite the "SHOULD NOT" clause regarding this. It's possible to have more than
one SMTP environment recorded in the trace headers (and until delivery, this is
evidenced by more than one "return-path" header being present).
>If the mta decides that it does not want to directly deliver the mail
>(let's say it's too busy to deliver right now), it delivers to another
>agent which writes the email to disk. this agent writes a received line
>saying 'i got it' then sends it to disk. was the email sent to this
>intermediary agent via smtp? nope, it was sent via shared memory.
>several hours later, the mail server is not busy anymore. the sending
>process can now insert another received header if it wants, it doesn't
>have to, but can because sec 4.4 is already satisfied. the sending
>process talks SMTP to the next SMTP server, so.... yes, you guessed it,
>the receiving server MUST put in a sec 4.4 compliant received header.
>after that, it can insert 1000 received lines that are simply comment lines:
Wrong. I see nothing that says that only a host's first inserted received
header is bound to the format - as opposed to ALL of them.
>Received: ; 8 Oct 2002 02:36:00 -0000
>
>It would suck to receive an email with that many lines, but it isn't
>disallowed. See the elementary logic with rfcs is this:
>
>if allowed, then allowed
>if not allowed, then not allowed
>if not disallowed, then allowed
>
>sec 4.4 is already satisfied, no matter what you believe. so, rfc 2822
>complaint headers are allowed.
It must be satisfied for EACH received header as long as the message remains
in the SMTP environment.
>> Piping a message from a mail server's listener to the queue is not an operation
>> where the message leaves that environment, especially since the SMTP envelope
>> information is kept and passed on as is. That is an INTERNAL operation which
>> does not merit a received header, but if one were to be inserted anyway, then it
>> must insert a properly formatted header (see above).
>
>According to YOU it doesn't 'merit' a received header. I say it is and
Exactly. However, if one were to be inserted, then it must comply.
>I am sending the email. So, too bad, I will insert a comment cause this
>is not an SMTP transaction:
>
>Received: (just cause you don't like it); 8 Oct 2002 02:00:00 -0000
If it's a non-SMTP transaction for a message previously transmitted by SMTP, it
is valid (RFC 2822 governs) ONLY IF a "return-path" header has been inserted
before (below) it AND the message has not been transmitted by SMTP since.
Otherwise RFC 2821 would govern.
>>>>It fails the PURPOSE of the header - to mark host-to-host transfers of the
>>>>message. When passing through without delivery, hosts (as relays) are
>>>>BLACK-BOXES - i.e. what happens inside is irrelevant.
>>>
>>>Ok, maybe you don't wanna know that an internal manipulation occurred...
>>>so strip it out then. Question: where is it forbidden to add the
>>>header? That what an rfc is all about... is there a 'should not' or a
>>>'must not' anywhere with regard to this?
>>
>>
>> Try using common sense to fill in what isn't stated. Just what would be the
>> purpose of a header such as:
>>
>> Received: from x.net by x.net ; <some_random_date>
>>
>> It doesn't tell you anything new because there was already a header indicating
>> "... by x.net" when the message came in from a remote host, and there will be a
>> header "from x.net ..." added by the next host when the message is transmitted
>> to it. Both of the received headers bracketing the above one would already
>> contain all the information. Not even the date field produces anything useful -
>> because we already know from the other headers that it was there then.
>
>hmm... guess i don't see the part of the rfc's that says:
>
>"Received headers must be as informative as possible, and describing
>obvious things SHOULD NOT be stated as grounds to insert a header"
I agree with that statement, but inserting a header for the sake of absurity
isn't a very good reason.
..But then they wouldn't be using my machine, would they? Therefore, I can't
CENSOR their mail!
>>>>So, you have effectively said that I cannot enforce RFC 2821 section 4.4 without
>>>>violating RFC 2821 section 3.8.2. Then, what is the point of the standard
>>>>defined in 4.4? If it's unenforcable, then why even bother with the standard?
>>>
>>>>Maybe in a future rfc section 3.8.2 might be taken out, then you would
>>>
>>>be able to do what you want to do (kill the mail) without violating rfc.
>>> sec 4.4 is put in so that there is a consistent looking received line
>>>when an smtp server receives something. but 3.8.2 will probably never
>>>be taken out because of legacy intermediary systems or things like virus
>>>scanners/mtas.
>>
>> That doesn't address the question. If a format MUST be used, but one MUST NOT
>> check that format, how can anyone enforce use of that format?
>
>They can't enforce the format, as you are starting to understand.
Then why it is even stated?
Mail Guy
>
> I doubt that further discussion would be productive.
>
I have to agree. Sorry Mr. Stussy that you didn't get what you wanted.
Mail Guy
Neil W Rickert
>On Mon, 7 Oct 2002, Mark Crispin wrote:
>>On Mon, 7 Oct 2002, Neil W Rickert wrote:
>>> No. I believe he is saying that you cannot enforce STUSSY 2821
>>> section 4.4 without violating RFC 2821 section 3.8.2.
>>I'm afraid that I have to agree with Neil's conclusion. Mr. Stussy seems
>>to be reading much more into RFC 2821 section 4.4 than what it actually
>>says.
>I don't believe that I am. It starts: "When an SMTP server receives a message
>for delivery or FURTHER PROCESSING (emphasis added), it MUST insert trace ..."
>"This line MUST be structured as follows: ..."
By your own account, the system in question meets this requirement.
>To me, this means that if it is not delivered to a mailbox (including a program
>acting as a mailbox) or transmitted to another host via a protocol OTHER THAN [a
>flavor of] SMTP, it hasn't left the "SMTP environment" and any other received
>header for any other further processing at that host must also comply with the
>format specified.
Where is RFC 2821 does it proscribe the addition of other trace
headers for tracking other parts of the processing? Certainly not in
section 4.4.
>Section 4.4 indicates that a message HAS LEFT the "SMTP environment" when final
>delivery occurs (which REQUIRES the "return-path" header be prepended).
I think you could read 4.4 as saying that if the software inserts a
"Return-Path:" header, then by virtue of that act it has left the
SMTP environment. It the software later deletes that header and
sends with the SMTP protocol, it has reinjected the message into the
SMTP environment.
But I think you are reading far too much into that. The text is
really just trying to give a practical account of when to create the
"Return-Path:" header.
>For example, is "qmail", when handing a message off from its receiving process
>into its queueing process, or subsequently to its queue runner, inserting the
>"return path" header? If not (and I suspect not - because I don't see that as a
>delivery of the message), then the e-mail has NOT left the SMTP environment, and
>therefore, the message it inserts (something like "Received: (qmail uid 10
>process 20 queue-id 40) ; <date>") is therefore invalid because RFC 2821
>indicates that since it hasn't left that environment, "FROM" and "BY" fields are
>required.
Some people might make the argument that the message has left the
SMTP environment once it has been transferred to qmail :-( .
>Since "leaving the SMTP environment" also means that the envelope information is
>tossed, then just how would "qmail" know the identity of any BCC'ed recipients?
The example is silly. When mail is being transferred via UUCP, it is
not in any reasonable sense within the SMTP environment (if that
expression really means anything). But the envelope is not tossed --
it is transformed into the uucp envelope.
It is quite obvious that you are making this up as you go along.
D. Stussy
>On Tue, 8 Oct 2002, D. Stussy wrote:
>> I didn't say that. However, what I said is that as long as the message
>> comes in by use of SMTP, leaves by use of SMTP, and isn't delivered
>> (i.e. where the SMTP envelope information is tossed away in the bit
>> bucket and NOT kept), then the message NEVER LEAVES THE SMTP
>> ENVIRONMENT, and thus any received header that is inserted must comply
>> with the stricter RFC 2821 format.
>
>Golly gee.
>
>I have been a developer of email software for 25+ years, contributed to
>just about every major email RFC, even invented one of the major email
>protocols. Yet, this is the first time that I ever heard of such a thing
>of an "SMTP environment" that imposes requirements on the internal
>operation of a mailer that Mr. Stussy envisions.
>
>Mr. Stussy's first message was phrased as a question and request for
>opinions. Answers and opinions have been provided. I fear that they were
>not the answers and opinions that he desired.
>
>I doubt that further discussion would be productive.
Those answers may have been consistent with one part of the RFC's but not with
other parts. The RFC's themselves may not be consistent - especially where:
- One MUST format a header as ...
- One MUST NOT check [the same] header's format....
It makes no sense to require a format then specifically fail to enforce it.
D. Stussy
>"D. Stussy" <kd6...@bde-arc.ampr.org> writes:
>>On Mon, 7 Oct 2002, Mark Crispin wrote:
>>>On Mon, 7 Oct 2002, Neil W Rickert wrote:
>>>> No. I believe he is saying that you cannot enforce STUSSY 2821
>>>> section 4.4 without violating RFC 2821 section 3.8.2.
>
>>>I'm afraid that I have to agree with Neil's conclusion. Mr. Stussy seems
>>>to be reading much more into RFC 2821 section 4.4 than what it actually
>>>says.
>
>>I don't believe that I am. It starts: "When an SMTP server receives a message
>>for delivery or FURTHER PROCESSING (emphasis added), it MUST insert trace ..."
>
>>"This line MUST be structured as follows: ..."
>
>By your own account, the system in question meets this requirement.
No, it doesn't. What it inserts is a header formatted per RFC 2822. However,
what is required is the RFC 2821 format. All RFC 2821 possibilities are a
subset of RFC 2822's format, but not vice-versa. Therefore, if a header is
formatted ONLY with respect to RFC 2822 and it is not in the RFC 2821 subset and
an RFC 2821 formatted header is required, it FAILS the requirement.
>>To me, this means that if it is not delivered to a mailbox (including a program
>>acting as a mailbox) or transmitted to another host via a protocol OTHER THAN [a
>>flavor of] SMTP, it hasn't left the "SMTP environment" and any other received
>>header for any other further processing at that host must also comply with the
>>format specified.
>
>Where is RFC 2821 does it proscribe the addition of other trace
>headers for tracking other parts of the processing? Certainly not in
>section 4.4.
Now you understand why I called the offending header extraneous.
>>Section 4.4 indicates that a message HAS LEFT the "SMTP environment" when final
>>delivery occurs (which REQUIRES the "return-path" header be prepended).
>
>I think you could read 4.4 as saying that if the software inserts a
>"Return-Path:" header, then by virtue of that act it has left the
>SMTP environment. It the software later deletes that header and
>sends with the SMTP protocol, it has reinjected the message into the
>SMTP environment.
>
>But I think you are reading far too much into that. The text is
>really just trying to give a practical account of when to create the
>"Return-Path:" header.
But, when the "Return-Path:" header is created, then we know that any
subsequently-inserted "Received:" header need not be RFC 2821 formatted - only
RFC 2822 formatted. That is a primary indication that a non-SMTP operation has
occurred.
>>For example, is "qmail", when handing a message off from its receiving process
>>into its queueing process, or subsequently to its queue runner, inserting the
>>"return path" header? If not (and I suspect not - because I don't see that as a
>>delivery of the message), then the e-mail has NOT left the SMTP environment, and
>>therefore, the message it inserts (something like "Received: (qmail uid 10
>>process 20 queue-id 40) ; <date>") is therefore invalid because RFC 2821
>>indicates that since it hasn't left that environment, "FROM" and "BY" fields are
>>required.
>
>Some people might make the argument that the message has left the
>SMTP environment once it has been transferred to qmail :-( .
Then where is the "Return-Path:" header that is required to be inserted upon
departure from the SMTP environment? If no such header is inserted by qmail,
then the program is non-RFC, and the mail transported through it is then also
non-RFC. If I then subsequently see a message that passed through this program,
then even if my server ends up stripping it out as it performs "final delivery,"
the incoming message should have that "Return-Path:" header embedded between the
"Received:" headers - OR if the non-SMTP->SMTP gateway should strip out the
"Return-Path:" header, then it should also indicate via "WITH" a non-SMTP
protocol. "qmail" does neither (it does not keep any "return-path" header that
must have been previously inserted, nor does it indicate a non-SMTP transport
has taken place).
I find the following parts of RFC 2821 Section 4.4 contradictory:
- A "Return-Path:" header MUST be inserted when a message leaves the
SMTP environment, and
- "a gateway from SMTP->elsewhere SHOULD insert a return-path header,
...:
For a gateway, should it or must it insert the header?
>>Since "leaving the SMTP environment" also means that the envelope information is
>>tossed, then just how would "qmail" know the identity of any BCC'ed recipients?
>
>The example is silly. When mail is being transferred via UUCP, it is
>not in any reasonable sense within the SMTP environment (if that
>expression really means anything). But the envelope is not tossed --
>it is transformed into the uucp envelope.
I agree. A change in the structure of the envelope could also signify a removal
from the SMTP environment. However, I would expect that the SMTP->UUCP gateway
to have inserted (prepended) TWO headers: An RFC 2821 formatted "Received:"
header and a "Return-Path:" header in that chronological order, both resulting
from the fact that the message came into the gateway via SMTP.
>It is quite obvious that you are making this up as you go along.
Hasn't this [also] shown that there is an RFC problem here?
Neil W Rickert
"D. Stussy" <kd6...@bde-arc.ampr.org> writes:
>On 8 Oct 2002, Neil W Rickert wrote:
>I find the following parts of RFC 2821 Section 4.4 contradictory:
> - A "Return-Path:" header MUST be inserted when a message leaves the
>SMTP environment, and
> - "a gateway from SMTP->elsewhere SHOULD insert a return-path header,
>...:
>For a gateway, should it or must it insert the header?
It depends.
>>>Since "leaving the SMTP environment" also means that the envelope information is
>>>tossed, then just how would "qmail" know the identity of any BCC'ed recipients?
>>The example is silly. When mail is being transferred via UUCP, it is
>>not in any reasonable sense within the SMTP environment (if that
>>expression really means anything). But the envelope is not tossed --
>>it is transformed into the uucp envelope.
>I agree. A change in the structure of the envelope could also signify a removal
>from the SMTP environment. However, I would expect that the SMTP->UUCP gateway
>to have inserted (prepended) TWO headers: An RFC 2821 formatted "Received:"
>header and a "Return-Path:" header in that chronological order, both resulting
>from the fact that the message came into the gateway via SMTP.
>>It is quite obvious that you are making this up as you go along.
>Hasn't this [also] shown that there is an RFC problem here?
No, it's a STUSSY problem.
You are over-interpreting.
My reading of 4.4 is very different from yours.
RFC 2821 changes the "Return-Path:" header from what was specified in
RFC 821/RFC 822. The old standard turned out to be impractical, and
RFC 2821 is redefining the standard to be consistent with the
practices that developed.
I read much of 4.4 as an example to illustrate the principles
involved in how this should be handled. Whereas you see a reference
to some formal entity called "the SMTP environment", I see that as
merely informal terminology to aid with the illustration.
The current practice is that when mail is received on SMTP, and sent
on UUCP (at least in sendmail systems), no "Return-Path:" is added.
Instead, the smtp envelope sender is use to generate the UUCP
envelope. If a later step uses SMTP, this will again be converted
into the SMTP envelope. Only on final delivery to a mailbox will
"Return-Path:" be added. I believe that the committed that prepared
RFC 2821 fully approves this behavior. If you are reading 4.4 as
indicating that a "Return-Path:" should be added on the transferral
to UUCP transport, then you are over-interpreting.
As for the supposed contradiction between 3.8.2 and 4.4, I don't
believe any such contradiction exists. The underlying philosophy of
SMTP has always been "be liberal in what you accept, be conservative
in what you send". The requirements of 3.8.2 are part of the "be
liberal" clause, while those of 4.4 are part of the "be conservative"
clause.
D. Stussy
Actually, I read it as the "Return-Path:" header MUST be added....
(That language is literally present in the RFC.)
>As for the supposed contradiction between 3.8.2 and 4.4, I don't
>believe any such contradiction exists. The underlying philosophy of
>SMTP has always been "be liberal in what you accept, be conservative
>in what you send". The requirements of 3.8.2 are part of the "be
>liberal" clause, while those of 4.4 are part of the "be conservative"
>clause.
I understand that. However, that doesn't mean that one should be overly liberal
in acceptance. Does that philosophy extend to accepting non-RFC data too? [I
feel that non-RFC compliance is the other person's problem.]
So far, in what I receive, being overly liberal has the side effect of receiving
spam, while enforcing the parts of the RFC that indicate "MUST" kills that spam
and all legitimate mail remains unaffected. Even if my spam filter is going too
far, it has the desired result (for the moment) - but I am open to considering a
different set of rules. [I consider adding the spammer's domain/mailbox to my
denial-of-access list as the last resort because it only blocks that spammer.]
Neil W Rickert
>On 9 Oct 2002, Neil W Rickert wrote:
>>As for the supposed contradiction between 3.8.2 and 4.4, I don't
>>believe any such contradiction exists. The underlying philosophy of
>>SMTP has always been "be liberal in what you accept, be conservative
>>in what you send". The requirements of 3.8.2 are part of the "be
>>liberal" clause, while those of 4.4 are part of the "be conservative"
>>clause.
>I understand that. However, that doesn't mean that one should be overly liberal
>in acceptance. Does that philosophy extend to accepting non-RFC data too? [I
>feel that non-RFC compliance is the other person's problem.]
As far as "Received:" headers are concerned, the RFC is clear that the only
thing you should be checking for in incoming headers, is the presence of
a mailing loop.
>So far, in what I receive, being overly liberal has the side effect of receiving
>spam, while enforcing the parts of the RFC that indicate "MUST" kills that spam
>and all legitimate mail remains unaffected. Even if my spam filter is going too
>far, it has the desired result (for the moment) - but I am open to considering a
>different set of rules. [I consider adding the spammer's domain/mailbox to my
>denial-of-access list as the last resort because it only blocks that spammer.]
Nobody is saying that the "be liberal" implies that you must accept
spam. Nobody has questioned your right to block spam on whatever
basis. However, your original question was on whether the RFC's will
license the particular filtering you want to do. It seems that
everbody who has responded has said "No" to that question. That
doesn't stop you from blocking, but it suggests that you should not
claim that you are using the RFCs as thebasis for the blocking.
I'm leaving it at that.
D. Stussy
>I'm leaving it at that.
I thank you for your consideration.
Stephen M. Dunn
$I'm having problems with a certain other mailer (MTA) that claims itself to be
$an SMTP transport adding a header which is effectively (comment-stripped):
$
$ Received: ; <date>
$
$The issue I have with this is that if that mail server receives its mail via
$SMTP and passes it on via SMTP, the shouldn't the trace header format of RFC
$2821 apply - thus making this [latter] "Received:" header invalid because it
$lacks the required fields "FROM" and "BY"?
It may be invalid in the context of RFC 2821. But it would be a Bad
Idea to expect that replacing a longstanding, well-known RFC with a new
version suddenly makes all existing software magically upgrade itself
to become compliant with the new RFC :-)
Also, you'll note that RFCs 2821 and 2822 are not official
standards - they're proposed standards. RFCs 821 and 822 are
still the current standards (see http://www.rfc-editor.org/rfcxx00.html
if you don't believe me), and while it would be wise to engineer
any new mail software you write to comply with the proposed
standards, it would also be wise not to disregard the current
standards.
Under section 4.1 of RFC 822, the header which you show above is
perfectly valid; all other clauses which are valid in a Received:
header, including from and by, are explicitly marked as optional.
This minimal Received: header is also shown as an example in
RFC 821.
$I want your opinions on this: Am I justified in denying email that routes
$through a server that uses this MTA because it is inserting an EXTRANEOUS header
$that is not RFC for the SMTP environment? I am aware that this may lower my
$interoperability, but isn't RFC compliance the other person's problem?
There's a principle, called the robustness principle, mentioned
in some RFCs (793 and 1123 for sure; if you go digging you may
find it in others as well). It says that you should be conservative
in what you send and liberal in what you receive. It's good
advice, because even if you were trying to follow the current
standards, there's no guarantee that everyone who might wish to
send you mail uses mail software which is fully compliant (due
to programmers who don't care, or due to simple human error).
By current standards for Internet email, your mail system is
not RFC-compliant. As long as this only affects incoming mail,
it's nobody's business but your own, and it's your call as to
whether you wish to make your mail system comply with the
standards.
--
Stephen M. Dunn <ste...@stevedunn.ca>
>>>----------------> http://www.stevedunn.ca/ <----------------<<<
------------------------------------------------------------------
Say hi to my cat -- http://www.stevedunn.ca/photos/toby/
D. Stussy
>In article <Pine.LNX.4.44.021006...@exp.bde-arc.ampr.org> kd6...@att.net writes:
>$I'm having problems with a certain other mailer (MTA) that claims itself to be
>$an SMTP transport adding a header which is effectively (comment-stripped):
>$
>$ Received: ; <date>
>$
>$The issue I have with this is that if that mail server receives its mail via
>$SMTP and passes it on via SMTP, the shouldn't the trace header format of RFC
>$2821 apply - thus making this [latter] "Received:" header invalid because it
>$lacks the required fields "FROM" and "BY"?
>
> It may be invalid in the context of RFC 2821. But it would be a Bad
>Idea to expect that replacing a longstanding, well-known RFC with a new
>version suddenly makes all existing software magically upgrade itself
>to become compliant with the new RFC :-)
>
> Also, you'll note that RFCs 2821 and 2822 are not official
>standards - they're proposed standards. RFCs 821 and 822 are
>still the current standards (see http://www.rfc-editor.org/rfcxx00.html
>if you don't believe me), and while it would be wise to engineer
>any new mail software you write to comply with the proposed
>standards, it would also be wise not to disregard the current
>standards.
Although that may be true, most of this part of the new RFC's simply carries
over from the old RFC's and is already considered standard. Everyone else I
know personally in the IT world already consider the newer RFC's as in force.
18 Months is an eternity on the Internet!
> Under section 4.1 of RFC 822, the header which you show above is
>perfectly valid; all other clauses which are valid in a Received:
>header, including from and by, are explicitly marked as optional.
>This minimal Received: header is also shown as an example in
>RFC 821.
>
>$I want your opinions on this: Am I justified in denying email that routes
>$through a server that uses this MTA because it is inserting an EXTRANEOUS header
>$that is not RFC for the SMTP environment? I am aware that this may lower my
>$interoperability, but isn't RFC compliance the other person's problem?
>
> There's a principle, called the robustness principle, mentioned
>in some RFCs (793 and 1123 for sure; if you go digging you may
>find it in others as well). It says that you should be conservative
>in what you send and liberal in what you receive. It's good
>advice, because even if you were trying to follow the current
>standards, there's no guarantee that everyone who might wish to
>send you mail uses mail software which is fully compliant (due
>to programmers who don't care, or due to simple human error).
The problem is in being overly liberal with this lets spam in, while being
overly conservative still lets 99% of all legitimate e-mail pass through.