Subject: RE: TECH-ERROR: SenderID sets recomendation for forwarders that are not compatible with RFC 2822

From: "Jim Lyon" <[email-masked]>
Date: Mon, 13 Sep 2004 10:03:14 -0700

On Monday, September 13, 2004 at 1:47:32 AM. William Leibzon quotes RFC
2822 and argues that SenderID's use of Resent-From is inconsistent with
RFC 2822.  I disagree.

RFC 2822 specifies Resent-From is not for "forwarding", in either of the
following two senses:

1. Quoting from RFC 2822:

        One sense of forwarding is that a mail reading program
        can be told by a user to forward a copy of a message to
        another person, making the forwarded message the body
        of the new message.  A forwarded message in this sense
        does not appear to have come from the original sender,
        but is an entirely new message from the forwarder of the
        message.

   This sense of "forwarding" is normally activated by a user
   pressing a "Forward" button in a UI (or similar action.) I
   completely agree that "Resent-From" has no place in this sense
   of forwarding.

2. Quoting from RFC 2822 again:

        On the other hand, forwarding is also used to mean when
        a mail transport program gets a message and forwards it
        on to a different destination for final delivery.

   I believe that what RFC 2822 refers to here as "forwarding" is
   what we today more commonly refer to as "relaying".  That is,
   when an MTA merely accepts a message destined for a particular
   mailbox, and sends the message along to another MTA because
   the first MTA doesn't own the mailbox.

   The position that this sense of "forwarding" really means
   "relaying" is bolstered by the fact that RFC 2822 nowhere
   contains the words "relay" or "relaying".

   I completely agree with RFC 2822 that "Resent-From" has no place
   in this sense of forwarding.

So, what's Resent-From for?  It's for when a message arrives at the
destination mailbox, and the owner of that mailbox gives instructions,
either manually or automatically, that the message should instead or
additionally be delivered to a different mailbox.  A Unix ".forward"
file is the canonical example of such automated instruction.

It would probably be a good idea for Pete Resnick (editor of RFC 2822)
to chime in on this interpretation.

-- Jim Lyon

Internet commerce will never really take off until you can buy something
online without getting spammed by the vendor.

---

From: Paul Iadonisi <[email-masked]>
Date: Mon, 13 Sep 2004 13:51:53 -0400

On Mon, 2004-09-13 at 13:03, Jim Lyon wrote:

[snip]

> file is the canonical example of such automated instruction.

  Well, if your going to use the Unix ".forward" file as a canonical
example, then you should check what that does on a typical Unix box
today (and historically).  Frankly, that's an honest suggestion, because
although I'm not too sure what they've done historically, I did just
check it out on a Linux box of mine and sure enough, there are no
Resent-* fields added whatsoever.
  In at least two places in section 3.6.6 of RFC2822, Resent-* fields
are said to be used to identify a message as re-introduced *by a user*
into the transport system.  Furthermore, it also says this:

   "They MUST NOT be used in the normal processing of replies
    or other such automatic actions on messages."

Which directly contradicts part of your interpretation above.  It's not
for 'manually or automatically' re-inserted messages, but only manually,
given that the automatic re-insertion is excluded from using Resent-*
fields according to the above line I've quoted from section 3.6.6 of
RFC2822.

> to chime in on this interpretation.

  Hmm, agreed.  That would probably put an end to the discussion :-).

--
-Paul Iadonisi
 Senior System Administrator
 Red Hat Certified Engineer / Local Linux Lobbyist
 Ever see a penguin fly?  --  Try Linux.
 GPL all the way: Sell services, don't lease secrets

---

From: "william(at)elan.net" <[email-masked]>
Date: Mon, 13 Sep 2004 11:51:45 -0700 (PDT)

>    completely agree that "Resent-From" has no place in this sense

I also agree with your interpretation of above quoted RFC2822 text.
 
>    the first MTA doesn't own the mailbox.

I disagree with your interpretation here. I think RFC2822 may have meant
that use of Resent-From is inappropriate in all cases of forwarding, 
including for example when MTA finds that new email address is contained 
in its configuration (like /etc/aliases on unix systems) and has to 
forward the message further. I note that sendmail for example treats
forwarding based on aliases in quite similar way to relaying.

I would also like to add that RFC2822 may indicate that use of Resent-From 
is inappropriate for forwarding by other text as well. In particular 
the original use of From header is indicated as follows:

  3.6.2. Originator fields
     In all cases, the "From:" field SHOULD NOT contain any mailbox that
     does not belong to the author(s) of the message

That clearly implies that "From:" is to be used to indicate only the 
author of the message and not anybody else. I understand that we're
talking about "Resent-From:" (and not "From:"), but when RFC2822 
first introduces Resent fields it seems to indicate that their use
should be comparable to the original fields, i.e.:

  3.6.6. Resent fields
     Resent fields SHOULD be added to any message that is reintroduced by
     a user into the transport system.  A separate set of resent fields
     SHOULD be added each time this is done.
     ...
     Each of the resent fields corresponds to a particular field elsewhere
     in the syntax.  For instance, the "Resent-Date:" field corresponds to
     the "Date:" field and the "Resent-To:" field corresponds to the "To:"
     field.

I note that in above quoted text RFC2822 seems to indicate that use of
Resent fields is only appropriate if message is reitroduced by the user
(and its not clear how that implies in case of MTA is doing forwarding
as in example I provided above). Additionally it seems to imply that
Resent fields have the same meaning as original fields which I take
as position that Resent-From field may only be used by user who is
willing to become new author of the message (but does not make any
alterations to the content when forwarding the message). I do not
believe this applies to the case when user is the original recepient
and when MTA automaticly uses his email when adding Resent-From.

>    in this sense of forwarding.
I also agree that "Resent-From" headers should not be used in this 
situaton. 

I would however add that some cases of such relaying email may pass 
through network boundary and in such situation the existing PRA algorithm 
can fail. 

For example email recepient may use several secondary MX servers
that are not on his network or controlled by it. When email comes
in first to one of those secondary MXs, the secondar MX servers
maybe able to verify email based on PRA algorithm (i.e. it comes
directly from the Sender and only has "From:" header and that header
has domain address for which SPF record lists ip address of the sender 
system). Now when this secondary MX has to relay the email to primary
MX (which lets say for simplicity is only MTA in destination network),
then secondary MX per your understanding about relaying should not
add new new "Resent-From:" or other header. In such a case the 
primary MX when it finally receives the email would still derive
same PRA address based on the headers, but ip address would be
that of secondary MX server and would fail SPF test. Now I'm hoping in 
this case the primary MX would be smart enough to recognize that its
one if its secondary MX servers and whiltelist the email, but I can
imagine there could be different simular scenarios where such thing
would no longer work properly and where use of PRA would fail when
relaying between independent networks is involved.
 
> to chime in on this interpretation.
Indeed his help in interpreting RFC2822 would be much appreciated.

And if he's willing to do it, I would also be interested in seeing his
interpretation on if PRA algorithm represents an obvious derivitive
or interpretation of RFC2822.

In particular I note that RFC2822 says in section 3.6.2:

  The "Sender:" field specifies the mailbox of the agent responsible for
  the actual transmission of the message.  For example, if a secretary
  were to send a message for another person, the mailbox of the secretary
  would appear in the "Sender:" field and the mailbox of the actual author
  would appear in the "From:" field.  If the originator of the message can
  be indicated by a single mailbox and the author and transmitter are
  identical, the "Sender:" field SHOULD NOT be used.  Otherwise, both
  fields SHOULD appear.

That clearly indicates that when email leaves MSA, the Sender field
should correspond to the "originator of the message". And if it is
not there, we can assume that it would have been the same as "From:"
and that is why it was omited.

Now during processing email may have been reintroduced by user and
in such cases Resent headers would have been added. As per text
I already quoted, these headers carry similar meaning as original
"From:" and "Sender:" headers, so if they are present, the top set of
these headers would (in some, but not all cases) correspond to the the 
latest reintroduction and so if Resent-Sender and Resent-From are present
they should be checked in the same order and same way as original
Sender and From. So what we get is the following order:
  1. Resent-Sender (only if present as part of set together with Resent-From)
  2. Resent-From
  3. Sender
  4. From
And that is pretty much what PRA algorithm has listed. 

And to me this all does not even appear to apply a derivitive, but simply
an interpretation of RFC2822 when looking for information of the party
responsible for last introduction of email by some user.

---
William Leibzon, Elan Networks:
 mailto: [email-masked]
Anti-Spam Research Worksite:
 http://www.elan.net/~william/asrg/

---

From: David Woodhouse <[email-masked]>
Date: Mon, 13 Sep 2004 19:49:38 +0100

> 2822 and argues that SenderID's use of Resent-From is inconsistent with

Nevertheless, others take a different view, and it is most certainly
inconsistent with current practice. Given such disagreement, is there
some reason why you consider the {ab,}use of Resent-From: header to be
more appropriate than the introduction of a new header about which there
would be no such disagreement? 

If consensus is being sought, surely it makes sense to use a new header
which does not have alternative usage and meaning, unless there is an
overriding reason for it to be 'Resent-From:'?

Btw, since we're quoting RFC2822 at each other, perhaps you should pay
attention to §3.6.4:

   Though optional, every message SHOULD have a "Message-ID:" field.
   Furthermore, reply messages SHOULD have "In-Reply-To:" and
   "References:" fields as appropriate, as described below.

...and to RFC2119:

3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.

Can you explain in accordance with RFC2119 precisely how the full
implications were weighed when you decided to post your reply without a
References: or In-Reply-To: header, thus detaching your reply from the
thread to which you were responding and causing inconvenience to all
members of the mailing list who wish to use a 'threaded' view of its
messages?

Please have the courtesy to use a non-broken mail client if you
participate in public fora. 

-- 
dwmw2

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 16 Sep 2004 04:14:38 +0200

Jim Lyon wrote:

 [2822 interpretation of Resent-*]
> completely agree that "Resent-From" has no place in this
> sense of forwarding.

That covers MIME types like message/rfc822, multipart/digest,
or maybe the proposed application/mbox, where original mails
are "forwarded" as part of the body, as complete as possible.

It's not exactly the sense of "forward" in Outlook, which is
IMHO no "forward" at all.  And that might be the explanation
for the Resent-* headers:

These headers already existed before MIME, in RfC 822.  So my
guess is that they were used in situations where today anybody
(minus Outlook and pre-MIME MUAs) uses message/rfc822:

 [Std 11, 4.7]
| Some systems permit mail recipients to  forward  a  message,
| retaining  the original headers, by adding some new fields.
| This standard supports such a service, through the "Resent-"
| prefix to field names.

| Whenever the string "Resent-" begins a field name, the field
| has  the  same  semantics as a field whose name does not have
| the prefix.  However, the message is assumed to have  been
| forwarded by  an original recipient who attached the
| "Resent-" field.  This new field is treated as being more
| recent  than  the  equivalent, original  field.   For
| example, the "Resent-From", indicates the person that
| forwarded the message, whereas the "From" field indicates the
| original author.

> It would probably be a good idea for Pete Resnick (editor of
> RFC 2822) to chime in on this interpretation.

Maybe it's just a case of RfC 2822 trying to fix something in
Std  11, and the result is not always clearer than the original
text.  We have the author of RfC 822 here, he won't let us
stray too far away from the real meaning of Resent-* headers.

                        Bye, Frank

---

From: "william(at)elan.net" <[email-masked]>
Date: Thu, 16 Sep 2004 03:24:15 -0700 (PDT)

> (minus Outlook and pre-MIME MUAs) uses message/rfc822:

Are you saying use of these headers is supposed to have been
depreciated by now and that they carry mostly historic meaning?
 
> stray too far away from the real meaning of Resent-* headers.

Ok. Can we then have author of RFC822 comment if use of Resent-From
header is appropriate per STD11 for cases of Mail List and different 
types of forwarders as is being recomendated in marid-core? And then 
comment if RFC2822 was supposed to change original STD11 in case of
the Resent-??? headers or not.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Pete Resnick <[email-masked]>
Date: Fri, 17 Sep 2004 15:57:20 -0500

Sorry for my delay in responding. It's been one of those months. 
(Maybe one of those quarters. *sigh*). I had a read through the DRUMS 
archive (boy, was that unpleasant) and some personal mail on the 
topic. Let me attempt to answer a few things. I don't know if this is 
going to help:

1. Yes, the reason we called out "forwarding" as distinct from use of 
the Resent-* fields was in part because 822 (and a good number of 
people for many years) conflated "forwarding" with "redirecting" and 
"resending" and a host of other labels. So we defined the two senses 
of forwarding ((a) including a previous message in an entirely new 
message and (b) relaying from the MTA level) and said that Resent-* 
fields were not to be used in either case.

2. The use of .forward files is a horrible canonical example of 
anything. The problem is that .forward files were historically 
implemented using exactly the same MTA mechanism as alias files used 
in relaying. In both of those cases, the original MAIL FROM was 
preserved by the MTA and therefore was clearly not a case of 
"delivered and then resent." I definitely had some discussion with 
folks about whether or not Resent-* fields would be desirable in the 
.forward case and we didn't come to a strong conclusion.

3. The fact of the matter is that historically .forward files did not 
generate Resent-* and that's part of why we said not to put Resent-* 
fields on forwarded messages. With all due deference to Jim, you do 
*not* read IETF standards like a lawyer. IETF standards are, for most 
intents and purposes, de facto standards: We are documenting what's 
out there (or what we expect to be out there) so that everyone can do 
it the same and interoperate well together, and we're trying to be as 
precise as possible so that programmers can do what they need to do. 
We're not trying to write laws that people should interpret as best 
they can. We're trying to write down exactly what everyone is (and 
should be) doing. Occasionally, we don't do as well as we'd like.

4. I don't think there is anything inconsistent with 2822 in 
implementing an "automatic resend" function and putting Resent-* 
fields on those messages. But in that case, I would expect the bounce 
(MAIL FROM) address to be identical to the Resent-From/Resent-Sender 
address and that the resender would be able to receive and act on 
those bounces. Semantically, I see resending as making the *content* 
of the message appear end-to-end from original sender to resent 
recipient, but that has nothing to do with making the *transport* of 
the message appear end-to-end between original sender and resent 
recipient. MTA-level relaying makes the transport appear end-to-end 
even though there is this "forwarding" step in the middle, and IMO 
that shouldn't be using Resent-* fields.

My read is that the intent of SenderID is to eliminate all MTA-level 
relaying and require it to be changed to automatic resending instead. 
That may be OK, but understand that this is what you are doing.

Does that help?
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: "william(at)elan.net" <[email-masked]>
Date: Fri, 17 Sep 2004 15:57:26 -0700 (PDT)

I'm reading Pete's answer right, then adding Resent-From is ok in some 
forwarding cases but not in others and that in particular:
 1. Its not ok with .forward forwarding (specifically mentioned). This
    is very very common in the deployed base.
 2. Its not ok with any forwarding where RFC2821 Mail From remains the same
    as it was before forwarding (most of the forwarding systems deployed 
    on the net are like this)
 3. Its not ok if message is changed (this would include Mail Lists).

If this is correct I see that most of the cases where marid-core recomends
adding Resent-From is incompabile with RFC2822. If that is so then SenderID
core algorithm has a fundamental technical error and current drafts should 
not move forward and we should review our situation and look for alternative
technologies/algorithms that can meet our requirements.

---

From: Bill McQuillan <[email-masked]>
Date: Fri, 17 Sep 2004 17:03:27 -0700

>  3. Its not ok if message is changed (this would include Mail Lists).

I'm sorry. I must be dense. When *IS* it OK to use Resent-From?
What is it intended to mean?

-- 
Bill McQuillan <[email-masked]>

---

From: "william(at)elan.net" <[email-masked]>
Date: Fri, 17 Sep 2004 17:58:14 -0700 (PDT)

> What is it intended to mean?

My reading is it is ok when email is reintroduced into system by SMTP 
server in the way similar to relaying it but as part of normal relay
process email session and other parameters are never change, but in this
case relaying involves change in envelope parameters Mail From and RCPT TO.
Think of it as relay with change of envelope and possibly change of address
headers (Sender, From, To, CC). It is also probably ok for message to have 
been introduced after certain wait period (in emailqueue presumably) 
triggered by certain event (i.e. like ETRN).

But I maybe wrong, so perhaps  it would be better if Pete answers this 
question.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Pete Resnick <[email-masked]>
Date: Sat, 18 Sep 2004 09:51:15 -0500

On 9/17/04 at 5:03 PM -0700, Bill McQuillan wrote:

>>I'm reading Pete's answer right, then adding Resent-From is ok in 
>>some forwarding cases but not in others and that in particular:

Let's be careful here about "ok" and "not ok". I'd say "intended 
for". Standards are supposed to be more descriptive than prescriptive 
of use.

>>  1. Its not ok with .forward forwarding (specifically mentioned). 
>>This is very very common in the deployed base.

I think it would match the semantics more if the MAIL FROM was not 
preserved. The current deployment of .forward I agree doesn't match 
the semantics of 2822. But that deployment could be changed.

>>   2. Its not ok with any forwarding where RFC2821 Mail From remains 
>>the same as it was before forwarding (most of the forwarding 
>>systems deployed  on the net are like this)

Again, they don't have to be deployed that way. And there might be a 
good argument for them not to be: If there is an error getting the 
mail to beyond the forwarder, should that really be reported back to 
the original sender? Maybe, maybe not.

>>   3. Its not ok if message is changed (this would include Mail Lists).

It has not traditionally been used with mailing lists because mailing 
lists act more like gateways than simple resenders. (That includes, 
but is not limited to, changing the message headers.) I'd prefer to 
use a different mechanism for mailing lists (like the List-* fields: 
perhaps a List-ID could be used if it followed the domain convention).

>What is it intended to mean?

It means, "The person in the Resent-From/Resent-Sender of this 
message received this message and redirected it to you. You should 
treat the message as if it went from the original sender directly to 
you." The Resent-* fields give you information about the resending 
event. Several MUAs have resend or redirect commands. The Resent-* 
fields are to be used when that function is executed.

pr