Subject: TECH-ERROR/DOC-BUG: empty fields in -pra

From: Roy Badami <[email-masked]>
Date: Fri, 3 Sep 2004 01:49:51 +0100

I'm tending to the opinion that the words 'non-empty' should simply be
deleted from section 2 of -pra.  They seem to serve no purpose other
that to add complexity, and for all the header fields in question,
empty field bodies are illegal, and I don't believe they are prevelent
in the wild.

If they must remain, then the wording should be made more accurate, ie

 "Locate the first non-empty Resent-Sender header in the message."

should become

 "Locate the first Resend-Sender field in the message that has a
 non-empty field body"

and similarly for the other references.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 3 Sep 2004 12:05:37 +0200

On Fri, Sep 03, 2004 at 01:49:51AM +0100,
 Roy Badami <[email-masked]> wrote 
 a message of 18 lines which said:

> I'm tending to the opinion that the words 'non-empty' should simply
> be deleted from section 2 of -pra.  They seem to serve no purpose
> other that to add complexity,

The added complexity is very small (using the length of the code as a
metric). The real issue is the definition of "non-empty".

The sendmail milter tests "strlen(hdr->hdr_val) > 0". Mark Kramer uses
the Perl \s expression. I used Python's definition of "empty". They do
not agree and I do not find an authoritative definition.

> empty field bodies are illegal, and I don't believe they are
> prevelent in the wild.
 
Never assume anything about the Real World when it is the Internet :-)
I thought that the provisions of -pra like:

        If that header is malformed (e.g. it appears to contain multiple
        mailboxes, or the single mailbox is hopelessly malformed, or the
        single mailbox does not contain a domain name),

were unecessary until I ran the PRA algorithm against my spam mailbox
and find very strange addresses.

>  "Locate the first Resend-Sender field in the message that has a

Not complete: is a set of whitespaces empty? Is "(Sender not
displayed)" empty or malformed?

---

From: Tony Finch <[email-masked]>
Date: Fri, 3 Sep 2004 15:09:31 +0100

> empty field bodies are illegal, and I don't believe they are prevelent

I'm afraid they are common enough that strict syntax checking has a high
false positive rate.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
LYME REGIS TO LANDS END INCLUDING THE ISLES OF SCILLY: WEST OR SOUTHWEST 2 TO
4. MAINLY FAIR, SMALL RISK FOG PATCHES. MODERATE WITH FOG PATCHES. SLIGHT TO
MODERATE.

---

From: Roy Badami <[email-masked]>
Date: Sat, 4 Sep 2004 19:11:56 +0100

>>>>> "Tony" == Tony Finch <[email-masked]> writes:

    Tony> I'm afraid they are common enough that strict syntax
    Tony> checking has a high false positive rate.

OK, if the empty header check turns out to be useful, then I'm all for
keeping it.

I suggest that an empty field body is defined to be one that is of
zero lenght, or consists entirely of linear whitespace.

     -roy

---

From: Tony Finch <[email-masked]>
Date: Sun, 5 Sep 2004 19:30:06 +0100

> zero lenght, or consists entirely of linear whitespace.

No, it's one that doesn't contain a valid email address.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FAEROES SOUTHEAST ICELAND: SOUTHWESTERLY 5 TO 7, DECREASING 4 AT TIMES.
SHOWERS. GOOD &NBSP;.

---

From: Roy Badami <[email-masked]>
Date: Sun, 5 Sep 2004 20:00:40 +0100

>>>>> "Tony" == Tony Finch <[email-masked]> writes:

    Tony> No, it's one that doesn't contain a valid email address.

I really don't see how you can interpret the phrase 'non-empty' as
meaning 'containing a valid e-mail address'.

Malformed header fields are addressed by step 5; if the field body did
not contain a valid email address, this would constitute a malformed
header field, and the algorithm would not return any PRA.

It's important that two implementations of the PRA algorithm never
return two different answers as to the PRA, even in the presence of a
malformed messages.  (It is acceptable that one implementation returns
an answer and another does not).

What you propose is dangerous, because it depends on the
interpretation of a 'valid e-mail address'; specifically, liberal
implementations typically choose to accept addresses that are not
strictly valid according to RRFC2822.

If two implementations have a different opinion as to the validity of
an e-mail address, that could result in them picking different
headers.  (This is precicely the flaw in the originial PRA algorithm
in the Caller ID draft)

The effect of this is that even if the users MUA is Sender ID aware,
and displays the PRA, there's no guarantee it is displaying the same
identity that was validated by the MTA.

	 -roy

---

From: Graham Murray <[email-masked]>
Date: Sun, 05 Sep 2004 21:14:19 +0100

> identity that was validated by the MTA.

Which is why it would be better for the MTA to create a header to
inform the MUA of the PRA and for the MUA to display that. If this
were done then it would be necessary to specify that the MTA MUST
remove any such headers already in the mail.

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 05 Sep 2004 16:54:21 -0400

> remove any such headers already in the mail.

This would be a problem with multi-hop forwarders.

Yakov

---

From: Graham Murray <[email-masked]>
Date: Mon, 06 Sep 2004 05:51:19 +0100

> This would be a problem with multi-hop forwarders.

Why? Sender-ID only authenticates the last hop anyway, so the PRA
which the MUA would derive should always be the same as that derived
and validated by the last MTA handling the email.

---

From: "Chris Haynes" <[email-masked]>
Date: Mon, 6 Sep 2004 09:24:46 +0100

> remove any such headers already in the mail.

This introduces its own security weakness:

If the senders are forging the additional tag, and the MTA, for some reason or
another, is not removing the headers - the MUA cannot detect that the header is
bad.

See my post:
http://www.imc.org/ietf-mxcomp/mail-archive/msg03118.html
for some of the circumstances in which this can occur.

You end up with the MTA including time-stamps and host-identities in  the
additional header, and then you have to cryptographically sign it, and give the
MUA the means to validate the signature.

It's almost as easy just to sign the entire original message and solve the whole
problem at one stroke!

Chris Haynes

---

From: <[email-masked]>
Date: Mon, 6 Sep 2004 07:25:22 -0400

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Graham Murray
> Sent: Monday, September 06, 2004 12:51 AM
> To: [email-masked]
> Subject: Re: TECH-ERROR/DOC-BUG: empty fields in -pra
>
>
>
The header fields are there for a reason, its an inline log of the email to assist mail
administrators to diagnose email issues (not just forgery/phishing, but also routing issues,
secondary MX issues, etc).  (Or allow skilled email users to analyze validity of incoming mail).

Suppressing this type of information is unacceptable to me, and my admins who are told to look at
the headers BEFORE calling support.

Terry Fielder
Manager Software Development and Deployment
Great Gulf Homes / Ashton Woods Homes
[email-masked]
Fax: (416) 441-9085

---

From: Graham Murray <[email-masked]>
Date: Mon, 06 Sep 2004 13:42:08 +0100

> the headers BEFORE calling support.

I agree with all of that. Yet at the moment there is no PRA header
inserted by the MTA. All the other headers would be left intact and
each email would have just one additional header (showing the PRA as
calculated and authenticated) inserted by the final MTA or MDA. This
would be useful to both the MUA and the admin examining headers. So
nothing would be taken away from the information currently available.

---

From: <[email-masked]>
Date: Mon, 6 Sep 2004 09:09:06 -0400

> Sent: Monday, September 06, 2004 8:42 AM
> To: [email-masked]
> > administrators to diagnose email issues (not just
> forgery/phishing, but also routing issues,
No, that's not what I saw proposed; specifically I was objecting to:  "If this were done then it
would be necessary to specify that the MTA MUST remove any such headers already in the mail."

From the earlier message.

That is what I was objecting to.  I understood that to mean that some or all headers the PRA was
based on would be removed.  Was my interpretation incorrect?

Terry

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 6 Sep 2004 15:18:03 +0200

On Mon, Sep 06, 2004 at 01:42:08PM +0100,
 Graham Murray <[email-masked]> wrote 
 a message of 16 lines which said:

> calculated and authenticated) inserted by the final MTA or MDA.

Why only one? Why not a "Responsible-From" inserted at each
administrative organization boundary? (Like "Received" inserted at
each MTA boundary.)

Each SenderID machine and each MUA would use the first
"Responsible-From" (the last inserted) to find the PRA. But a lot more
information would be available for debugging.

---

From: Graham Murray <[email-masked]>
Date: Mon, 06 Sep 2004 15:20:35 +0100

> based on would be removed.  Was my interpretation incorrect?

Yes, it was incorrect but that was my fault for not making my meaning
clear.

My suggestion was that the MTA should insert a new header (for the MUA
and human) to show the PRA it calculated and authenticated. Also, to
avoid spoofing, that it also removes any pre-existing instances of
this new header so that the user will only see the one inserted by the
MTA/MDA from which the MUA fetched the mail. This being similar
behaviour to SpamAssassin which removes any pre-existing "X-Spam-*"
headers before adding its own.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 6 Sep 2004 17:02:13 +0200

On Mon, Sep 06, 2004 at 03:20:35PM +0100,
 Graham Murray <[email-masked]> wrote 
 a message of 18 lines which said:

> My suggestion was that the MTA should insert a new header (for the
> MUA and human) to show the PRA it calculated and
> authenticated. Also, to avoid spoofing, that it also removes any
> pre-existing instances of this new header so that the user will only
> see the one inserted by the MTA/MDA from which the MUA fetched the
> mail. This being similar behaviour to SpamAssassin which removes any
> pre-existing "X-Spam-*" headers before adding its own.

Pre-existing "X-Spam-*" headers are indeed useless (or dangerous since
they can be forged) but, since SenderID authenticates a channel, not
the end-to-end message, pre-existing "Responsible-From" (or whatever
the name of the PRA header is) are still useful. The situation of
these PRA headers, to me, is closer from the "Received" headers, where
nobody objects there are several instances of them (and sometimes some
forged ones).

---

From: "Alan DeKok" <[email-masked]>
Date: Mon, 06 Sep 2004 12:32:35 -0400

> administrative organization boundary? (Like "Received" inserted at

  That would help, but not enough, due to the current deployment of
SMTP.

  SMTP is "end to end", in that clients talk directly to servers.
It's easy to validate the previous hop: you've got the IP.  email
*messages* aren't "end to end", or, at least, aren't *verifiable* to
be end to end, ias they could have been modified in transit, or
forged.  (Barring signed messages...)

  What you're talking about is changing the forwarding methodology from:

    message -> message' -> message'' -> message''' -> ...

  to:

    forward(forward(forward(forward(message))))

  The first method forwards messages which may, or may not, be related
to each other.  They certainly *appear* to be related, and the
"received" lines indicate some relationship, but that relationship is
difficult to follow or prove, as the whole PRA argument has shown.

  The second method "tunnels" the message though multiple forwarders.
Each forwarder not only accepts responsibility for the message, it
attests to the fact that other forwarders have accepted responsibility
for their behavior.

  In the security world, these methods are used to create multiple
nested virtual tunnels.  The tunneling & layering is *required*,
because the intermediaries know about only their neighbours, and don't
trust anyone else.  To first order, the number of nested tunnels
indicates the number of trust boundaries that the message has crossed.

  This method works for VPN's, because the sites doing the tunnelling
don't futz with the message content.  (They often can't, because the
content is itself signed and encrypted).  This method does NOT work
for SMTP, because there are too many broken MTA implementations which
change the message contents.

  So a "Responsible-From" inserted at each administrative organization
boundary would be a good idea, but it would have to be tied both to
the message, and to the party claiming responsibility.  That's hard.

  Alan DeKok.

---

From: Roy Badami <[email-masked]>
Date: Mon, 6 Sep 2004 20:14:47 +0100

>>>>> "Graham" == Graham Murray <[email-masked]> writes:

    Graham> Which is why it would be better for the MTA to create a
    Graham> header to inform the MUA of the PRA and for the MUA to
    Graham> display that. If this were done then it would be necessary
    Graham> to specify that the MTA MUST remove any such headers
    Graham> already in the mail.

I think this has the potential to make things worse (or at least more
confusing) where a Sender ID aware MUA is used with a non-Sender ID
MTA.

	-roy