Subject: RE: TECH OMISSION: Stronger checks against email forgery

From: Tony Finch <[email-masked]>
Date: Tue, 7 Sep 2004 10:26:48 +0100

On Fri, 27 Aug 2004, Jim Lyon wrote:
>
> I continue to disagree.  There are too many scenarios where the bounce
> address is uncorrelated with the MTA that's delivering a message; this
> means that any scheme that attempts to reject mail based on those two
> inputs (bounce address and IP addr of sending MTA) will have too many
> false rejections.

What makes the PRA different from the bounce address from this point of
view?

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FAIR ISLE: WESTERLY BACKING SOUTHERLY 3 OR 4, OCCASIONALLY 5 AT FIRST. FAIR.
GOOD.

---

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 07 Sep 2004 09:43:03 -0400

> What makes the PRA different from the bounce address from this point of

The PRA algorithm tries to guess the most recent "Sender" for the 
message - i.e. the one that is being used for this SMTP hop. The bounce 
address on the other hand originates from the original hop and stays 
that way throughout multiple SMTP hops.

Two related points I want to stress as well:
1. The verification is no longer being done on bounce address, that 
would mean that domains are not protected from bounces. In a case where 
multiple SMTP hops take place with non-compliant MTAs, and the last hop 
using a compliant MTA, the bounces will be directed to the original 
domain, even though it is using Sender ID.
2. The "Sender" header is being verified over the "From" header. While 
according to the RFC that is the agent introducing the message into the 
email system, the "Sender" header is not displayed in MUAs. Therefore, 
in a compliant MTA but non-compliant MUA, it is still possible to fool 
the system by using a valid "Sender" header with fake "From" headers.

Yakov

---

From: Tony Finch <[email-masked]>
Date: Tue, 7 Sep 2004 14:50:08 +0100

On Tue, 7 Sep 2004, Yakov Shafranovich wrote:
> The PRA algorithm tries to guess the most recent "Sender" for the message -
> i.e. the one that is being used for this SMTP hop. The bounce address on the
> other hand originates from the original hop and stays that way throughout
> multiple SMTP hops.

This is also true for the PRA, since no existing MTAs add Resent-From:
header fields when alias-forwarding.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOGGER: NORTHEAST 4 OR 5. FAIR. GOOD.

---

From: "Chris Haynes" <[email-masked]>
Date: Tue, 7 Sep 2004 15:32:06 +0100

> domain, even though it is using Sender ID.

Or directed to whatever address was given in the Mail-From (which need not be
related to the actual origin of the message). It could be intended that the
bounce (perhaps with malicious content) be sent to the 'victim' identified in
the Mail-From, and an additional 'benefit' of the intentional bounce is that the
reputation of the 'final hop', Sender-ID-compliant node, or of the one before
it, might thereby be damaged.

> the system by using a valid "Sender" header with fake "From" headers.

Chris Haynes

---

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 07 Sep 2004 12:38:54 -0400

> header fields when alias-forwarding.

I think we are going in circles. Validating the bounce address only 
prevents false bounces but allows phishing, validating "Sender" does not 
stop phishing but allows bounces plus does not take into account 
forwarding, validating the "from" headers stops phishing but allows 
bounces. Which one of these do we really want?

Yakov

---

From: Tony Finch <[email-masked]>
Date: Tue, 7 Sep 2004 17:51:53 +0100

On Tue, 7 Sep 2004, Yakov Shafranovich wrote:
>
> I think we are going in circles.

You are missing my point, which is that Jim Lyon's justification is
incorrect. The PRA is not significantly better correlated with the
originating MTA than the bounce address.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
LANDS END TO ST DAVIDS HEAD INCLUDING THE BRISTOL CHANNEL: NORTHEAST VEERING
EAST 4 OR 5, OCCASIONALLY 6 IN SOUTH. FAIR. MODERATE OR GOOD. SLIGHT OR
MODERATE.

---

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 07 Sep 2004 12:56:57 -0400

> originating MTA than the bounce address.

No I got you, I agree with you actually assuming that your comment about 
the forwarding behavior is correct. I was referring to the WG in 
general. I think the current identity has its issues, and all of the 
alternatives have issues as well.

Yakov

---

From: Douglas Otis <[email-masked]>
Date: Tue, 07 Sep 2004 12:01:33 -0700

> bounces. Which one of these do we really want?

Sender-ID is selecting a Mailbox Domain from a series of headers
considered related to the most recent MTA outside of the recipients
administrative region.  Sender-ID header selection prioritizes RFC2822
>From as _last_ in the selection process.  This selection process
attempts to reduce rejection rates, but is no better at stopping
phishing as a result.  At least using MAIL FROM, there will be a better
understanding which header is being inspected. 

Rather than devising a scheme that can safely treat a relationship of
the MTA to Mailbox Domains for From and MAIL FROM as nominal lists, the
attempt to use these lists to also authorize the MTA creates
incompatibility and vulnerability to attack.  Although these Mailbox
Domain lists may color mail with respect to certainty, they will remain
at a weak level of certainty, even after removing Mailbox Domain
portability.  Tony was right in that Sender-ID can not identify forged
messages, nor can Sender-ID reliably identify valid messages.  

The MTA must be authenticated independent of these nominal Mailbox
Domain lists in order to establish a reliable name for reputation
assertions.  Another major benefit in isolating MTA authentication from
Mailbox Domain relationships, is that this greatly simplifies
associating Mailbox Domains with MTA servers.  With an MTA name, an MTA
address list is not needed.  With an MTA name, a secondary lookup for
addresses is not needed to validate the Mailbox Domain.  With an MTA
name, these Mailbox Domain relationships can be expressed within a
single DNS lookup without the use of a text script!

By making MTA authentication independent of the Mailbox Domain
relationships, those wishing to retain their freedom in the use of their
return mailbox address will not impact MTA authentication and reputation
checks.  Those not declaring these nominative lists to be comprehensive,
do not offer an avenue where "open-ended" records become exploited by
spammers.

A real danger with Sender-ID, in addition to making the MTA prone to
attack, is created by false expectations the headers are reliably
checked.  This will only lead spammers to exploit the many Sender-ID
weaknesses.  What is needed is a lightweight method of authenticating
the MTA, where other relationships are then established upon this
relatively solid foundation.  Sender-ID does not offer a suitable
foundation, as it is premised upon the integrity of the RFC2822
information which can not be verified.  If there is a problem, Sender-ID
fails to isolate the source of the problem.  If there is a problem,
Sender-ID may mis-identify the sender and cause serious harm.

-Doug

---

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 07 Sep 2004 15:48:18 -0400

> understanding which header is being inspected. 

Aren't we basically trying to deduce the "mailbox of the agent 
responsible for the actual transmission of the message". Basically you 
want to know who was the original agent or "Sender" was that put that 
email into the email system. To me it sounds like the original 
"purpotable responsible address" is the same as guessing "the agent 
responsible for the actual transmission of the message"?

If so, RFC 2822, section 3.6.2, is very clear that the "Sender" field 
indicates that. That's step 3 in PRA. However, the same RFC states in 
section 3.6.2 "If the originator of the message can be indicated by a 
single mailbox and the author and transmitter are identical, the 
"Sender:" field SHOULD NOT be used". That's step 4 of PRA. The PRA draft 
states explicetely that those 2 steps are taken directly from RFC 2822 
(don't see any IPR).

Now for the resent blocks, the same RFC in section 3.6.6 states that the 
"Resent-XXX" fields are simply placeholders for previous versions of the 
same "Sender" and "From" fields. Therefore, there is a preference for 
"Resent-Sender" just like there is a preference for "Sender" described 
above. That's why steps 1-2 come before 3-4, and step 1 comes before 
step 2 (still no IPR).

Now the interesting part here is deciding how to pick the proper 
"Resent" header. The PRA algorithm picks the first "Resent-Sender" 
unless there is a "Resent-From" before with "Return-Path" or "Received" 
in between. The RFC states that:
A. "Resent fields SHOULD be added to any message that is reintroduced by 
  a user into the transport system."
B. "A separate set of resent fields SHOULD be added each time this is done."
C. "All of the resent fields corresponding to a particular resending of 
the message SHOULD be together."
D. "Each new set of resent fields is prepended to the message; that is, 
the most recent set of resent fields appear earlier in the message."

Picking the earliest Resent block is based on #D and B, picking an 
earlier "Resent-From" if separated from the other Resent headers is 
based on #C (and I still see no IPR).

Yakov

---

From: Douglas Otis <[email-masked]>
Date: Tue, 07 Sep 2004 14:08:05 -0700

On Tue, 2004-09-07 at 12:48, Yakov Shafranovich wrote:
> "purportable responsible address" is the same as guessing "the agent 

The recipient will be looking at the RFC 2822 From, where there is a new
expectation they will be seeing some other header being checked.  It
remains, Sender-ID is not an effective tool against phishing.  Sender-ID
is also not an effective tool for stopping other types of abuse.  To
abate abuse, a reliable reputation mechanism must be possible.  There is
no reliable name obtained with Sender-ID.

> states explicitly that those 2 steps are taken directly from RFC 2822 
> (don't see any IPR).

The EHLO domain tracks the "sender" without the need to sort headers. 
Make this field a visible part of the sender identification.   It does
not require any IPR either and authentication could always result in a
high degree of certainty.

> above. That's why steps 1-2 come before 3-4, and step 1 comes before 

What is the substantial difference between these headers and the name
contained within the EHLO domain?  There a tremendous advantage using
the MTA name to establish MTA to Mailbox Domain relationships for either
MAIL FROM or From.  Why care about these other headers?

> based on #C (and I still see no IPR).

Making a Mailbox Domain list that nominally relates to the MTA name will
encompass a vast majority of the mail without touching upon issues of
whether the Sender-ID IPR is obvious.  By making the MTA authentication
a separate function, then simply marking mail as being outside this MTA
name/Mailbox Domain relationship is little different than selecting an
invisible header compared against an address list unrelated to the From
or MAIL FROM.  Just use the EHLO domain.  By always exposing the EHLO
domain as visible, the recipient will soon recognize these From/EHLO
names used in tandem.  It is too dangerous to block mail based upon
Sender-ID.  A reputation check based upon the MTA name would allow a
safe method of blocking mail however.

If the goal is to assist the recipient decide what is valid mail and
what is not, the visible MTA name compared against a nominal or
restrictive list of MAIL FROM or From Mailbox Domains provides a
superior outcome.

-Doug

---

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 07 Sep 2004 17:45:03 -0400

> no reliable name obtained with Sender-ID.

I think the intent was to introduce the entity of "original sender" of 
the message although I would have to agree with you that by itself it 
might not be too useful. Of course, considering that if Outlook 2000 and 
some other MUAs actually displays that header, it will have some use for 
those MUAs.

But, what about SUBMITTER or RFC2821 MAIL FROM bounce address? Purely 
from the technical viewpoint, are they reliable? All could be forged 
including EHLO.

> high degree of certainty.

The key difference between EHLO and "Sender" is that EHLO is 
MTA-specific while "Sender" is specific to the message itself. A single 
EHLO value may cover messages originating from multiple domains. 
Therefore using the "Sender" field or "SUBMITTER" allows for granularity 
where a specific MTA may be authorized to relay email for one domain and 
not another, like an ISP's outgoing MTA. Of course, like someone has 
mentioned before, there are less MTAs then domains, and it might be 
useful to work with the MTAs alone.

I agree that the reliability is much less for PRA then EHLO, but on the 
other hand SUBMITTER should be no worse than EHLO. So EHLO would not be 
an option, if the granularity described above is desired.

And I still argue that the entire PRA algorithm is obvious to anyone 
reading the RFC so the IPR issue is probably moot.

> MAIL FROM or From.  Why care about these other headers?

See above.

> restrictive list of MAIL FROM or From Mailbox Domains provides a

I am not sure which would be superior. EHLO is not being used today 
anywhere, and similarly for phishing the 2821 MAIL FROM bounce address 
is not displayed to the end user. From the viewpoint of phishing, it 
makes perfect sense to work with the message itself, not the channel. I 
don't see how the invisible EHLO is any better than the invisible 
"Sender". If anything, SUBMITTER or 2821 MAIL FROM, or 2822 "from" would 
be superior.

Yakov

---

From: Douglas Otis <[email-masked]>
Date: Tue, 07 Sep 2004 16:40:12 -0700

> from the technical viewpoint, are they reliable? All could be forged 

It is much more difficult to forge the EHLO name.  Unlike Sender-ID
PRA/Submitter or RFC2821 MAIL FROM, there is no presumption of the
integrity of the mail channel for assessing a particular message based
upon the EHLO name.  The EHLO name, as a host name, must reference a
record that scales to the number of IP addresses of a host, unlike SPF
or Sender-ID that could encompass the entire Internet.  Unlike a record
intended for routing, this MTA record must ensure authentication is
possible.  This record also indicates the host is authorized by the
domain to send mail.

The certainty of the authenticated MTA name allows for asserting
reputation.  An error made establishing an accountable name could be
devastating for a reputation service.  Sender-ID and SPF simple leave
too many avenues open for this name to be spoofed.  In addition, unlike
Sender-ID or SPF, there is no presumption that the MAIL FROM or From
mailbox domain will be constrained.  MTA authentication operates
independently of the mailbox domain relationship.
 
> MTA-specific while "Sender" is specific to the message itself.

Perhaps a poor choice of words.  The sender as identified using the PRA,
such as Resent-From, could easily result in any of the same names as
found in the EHLO name.  Neither offer a direct relationship to the
author of the message.  The MTA name verifies the administrative domain
accountable for delivered messages.  If the mailbox domain is then
related to MTA names, this arrives at the goal of mailbox domains
authorizing these MTA names.  It is also safe to handle this
relationship as being either nominal or absolute.  This MTA name
authorization can be done within a single DNS lookup without the need
for subsequent lookups.

> A single EHLO value may cover messages originating from multiple domains. 

There is no limitation on the EHLO domain with respect to the number of
domains claimed.  A service provider may elect to assert an EHLO name to
correspond to the customer being relayed. 

> an option, if the granularity described above is desired.

Submitter is based upon the PRA.  Submitter can be no better than the
PRA as a result.  Yes, there is much less reliability of the PRA as
compared to the EHLO domain.  There is also a greater likelihood there
will be fewer EHLO domains compared to mailbox domains.  A mail provider
that keeps track of their outbound SMTP logs and quickly terminates the
typical spammer will earn a good reputation and will be offering a
valuable service.  There is no need to restrict the RFC2822 From
address, if the provider authenticates their clients using any number of
methods.  How this control is achieved should be left to the mail
provider, and using the EHLO name affords this flexibility.

> reading the RFC so the IPR issue is probably moot.

I see nothing to justify using the PRA algorithm.  These headers are of
no value when compared to the EHLO domain.

> > MAIL FROM or From.  Why care about these other headers?

There is nothing that limits the granularity of the EHLO name, but there
are reasons for wanting fewer entities to assess reputations.  If the
EHLO name is compared against a name list referenced by the mailbox
domain, this arrives at the desired goal of assessing the mail channel
against the mailbox domains as a separate and optional operation. 

> "Sender". If anything, SUBMITTER or 2821 MAIL FROM, or 2822 "from" would 

The EHLO name is a stronger identity that can safely support reputation
assertions.  If there are any illegal activity, an authenticated MTA
name also identifies where logs can be found as evidence. 
Authenticating the MTA name allows an extremely simple name list to
express the relationship between the mailbox domain and the MTA name. 
This gets rid of the need for dozens or hundreds of DNS lookups and
complex macro scripts needed within SPF or Sender-ID.

These lists can look something like:

_mp._smtp.my-mailbox-domain.com. PTR  big-isp.com
                                      PTR  ads-r-us.com
                                      PTR  webs-r-us.com
                                      
Where a EHLO domain of mx01.nw.big-isp.com is recognized as sending mail
on behalf of my-mailbox-domain.com.  This relationship can be considered
"required" or "nominal" without creating any exploit risk as with either
SPF or Sender-ID.  If there is abuse found, the reputation would be
against the EHLO name and not the mailbox domain, as this mailbox
identity is simply too weak.  There are also advantages and greater
freedoms afforded by establishing a hierarchy of accountability.

-Doug

---

From: Matt Sergeant <[email-masked]>
Date: Wed, 8 Sep 2004 13:49:07 +0100

On 7 Sep 2004, at 17:38, Yakov Shafranovich wrote:

> prevents false bounces but allows phishing, validating "Sender" does 
> not stop phishing but allows bounces plus does not take into account 

Validating the "from" headers in no way stops phishing.

It may give us some comfort to think that in some future we might be 
able to whitelist "from" headers, with some certainty that they aren't 
forged, but that's all it does. A whitelist (or reputation list) does 
not stop phishing.

Matt.

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________