Subject: MTAs should focus on email TRANSPORT not email CONTENT

From: "Terje Petersen" <[email-masked]>
Date: Tue, 20 Jul 2004 10:44:16 +1000

All,

I have tried to read all of the Marid discussions almost since inception but I have not contributed anything to date. I hope what I have to say is not a waste of time or merely repeating what has already been said, however it is a view that I hold quite strongly. 

I am a system administrator that supports email servers for multiple organisations. Mostly small businesses that want fairly automated low maintenance solutions. 

MTAs should focus on email transport. Many proprietary solutions have MTAs dipping inside the email content to perform various tests looking for spam or otherwise. Surfcontrol is the example that I am most familiar with. Antivirus solutions are another obvious example. This is all fine; however I do not believe that any of it should be a standard function of an MTA. An MTA is a "message TRANSFER agent". Its job should not necessitate reading the DATA content of email.

My view is that the MARID standard should proceed as follows:-

1. Use SPF classic to verify the MAIL FROM address. 
2. Use SUBMITTER as an alternative test point if the sending MTA supports it. Otherwise break forwarding after the flag date.

If SUBMITTER does not match the FROM address within the DATA section of an email then the email content can be considered to be malformed. However it should be up to the MUA to deal with this, not the MTA. 

This would mean that most email forgery could be cured by MTAs doing SPF checking. There would still be a phishing opportunity for spammers etc that could claim a false SUBMITTER address. However once this becomes the dominant loophole for spammers and viruses then MUA developers would have user pressure expecting a relevant patch to fix the loophole. The MUA is the correct place to fix this problem. 

Conceptually the postal worker should not be opening mail in order to make the delivery. The content should be private. Maybe even encrypted (if we want to). If the content is all in Latin the postal worker should not be concerned. Just as the postal worker should not need to look at mail content so the MTA should not be looking at the DATA section of email. 

A mismatch between the SUBMITTER address and the FROM address is a content problem not a transport problem. 

MTAs should deal with transport. 
MUAs should deal with content. 

Of course proprietary anti-spam gateways could still do non-standard MUA type testing. However they would remain proprietary. Just as looking for spam keywords in the content of the DATA section is totally proprietary MTA function. Would anybody wish to make the lists of keywords checked by anti-spam gateways a "standard" MTA job? I think not. 

Having the MTA dip inside the DATA section in order to complete its job breaks the layered approach to transport. Breaking the layering limits the future potential of SMTP. Layering should not be broken without good cause. 

MARID should separate what is transport and what is content. Separate what is an MTA job and what is an MUA job. Moving MUA functions into the MTA should be proprietary not standard.   

Regards,
TERJE PETERSEN
ExceLAN
Sydney - Australia

---

From: Michel Bouissou <[email-masked]>
Date: Tue, 20 Jul 2004 03:07:12 +0200

Le mardi 20 Juillet 2004 02:44, Terje Petersen a écrit :
>
> [...] An MTA is a "message TRANSFER agent". Its job should not necessitate
> reading the DATA content of email.

I agree.

> 1. Use SPF classic to verify the MAIL FROM address.

Yes.

> 2. Use SUBMITTER as an alternative test point if the sending MTA supports
> it. Otherwise break forwarding after the flag date.

I would personally rather drop the SUBMITTER concept completely. It looks to 
me like a kludge that tries to give before the DATA stream the results of a 
computation made with this DATA stream contents.

This solution doesn't look right to me.

(Shevek also discusses SUBMITTER at the end of his 
http://www.libsrs2.org/srs/srs.pdf document, finds it to be bad, and I share 
his point of view regarding this)

> Conceptually the postal worker should not be opening mail in order to make
> the delivery. [...] Just as the postal worker should not need to look at
> mail content so the MTA should not be looking at the DATA section of email.

I agree.

> MUAs should deal with content.

Again, I 100% agree with this statement.

> Having the MTA dip inside the DATA section in order to complete its job
> breaks the layered approach to transport. Breaking the layering limits the
> future potential of SMTP. Layering should not be broken without good cause.

Yes, yes, and yes.

Regards.

-- 
Michel Bouissou <[email-masked]> OpenPGP ID 0xDDE8AC6E

---

From: Greg Connor <[email-masked]>
Date: Mon, 26 Jul 2004 13:02:09 -0700

On Mon, 2004-07-19 at 17:44, Terje Petersen wrote:
...
> MUAs should deal with content. 

I think the assumption hidden here is that 2821 defines transport data,
and never content, and 2822 defines content, and never transport. 
Perhaps this is true, but there was LONG discussion during the first
phase of MARID about "Choice of identity to operate on" and it was
decided at the outset that we want to be able to validate BOTH 2821 and
2822 identities.

In my opinion, if you want to define 2821 as Transport and 2822 as
Content, that is fine, but there are still cases where you need to sync
up the two.  How do you solve this puzzle?  If the MTA only has access
to 2821 identities, and the MUA can only see 2822 identities, then
neither of these can validate consistency without stepping out of its
assumed boundaries.

You mentioned spam filters and virus filters that are already blurring
the lines.  There is good reason for that.  Users want to REJECT bad
mail, not accept it and deliver it to an MUA which has no choice but to
file it in another folder.  Administrators want to reject it during the
first conversation so that they don't have to generate a bounce message
later (commonly referred to as "blowback")

Also, RFC2476 defines the concept of an MSA (Submit server) which fills
all the functions of an MTA, but also does some content checking. For
example, it says that an MSA should check all of return path, Sender:,
Message-Id, and more).  So the idea of checking both 2821 and 2822
identities in the same place is not new (RFC2476 is 5.5 years old,
though big ISPs and corporations still routinely skirt it).

Another way of looking at this is: MARID doesn't require implementation
at the MTA level.  SUBMITTER is a helper-function, but not a
requirement, and it is completely possible to implement MARID within the
spec by an MUA or an MUA-like filter like SpamAssassin.  The spec itself
doesn't say "It must be in your MTA".  But, if users are free to
implement it at whatever stage they choose, they are guaranteed to start
phoning Sendmail and asking for a MARID implementation :)

---

From: Douglas Otis <[email-masked]>
Date: Mon, 26 Jul 2004 14:01:41 -0700

> decided at the outset that we want to be able to validate BOTH 2821 and

Subjugation of RFC 2821 MAIL FROM to PRA was "announced" more than
"discussed."  This took place when Ming announcing "he" decided to
relinquish SPF to C-ID.  It has been a hard road to pushing back the
changes this entailed.  From XML syntax to now the RFC 2822 sender
identities.  I see little justification for either of these changes and
may reasons for not allowing such changes.

> neither of these can validate consistency without stepping out of its

There are several proposals that handle RFC 2822 identities in a manner
they SHOULD be handled.  Sender-ID will continue Phishing and "back
door" entry practices.  Something like "DomainKey" or "Identified
Internet Mail" does exactly what a user would expect.  Tell them WHO
sent the mail, or provide STRONG assurances of the domain.  PRA can not
clearly define which identity should be checked, displayed, or how this
relates to the author of the message. 

> later (commonly referred to as "blowback")

The large number of records required for this check and the need to read
the message ensures there is no savings with respect to the incoming
network load.  Closing the "back door" or preventing "blowback" is best
done with RFC 2821 MAIL FROM checks.  Allowing a different identity only
allows a means to side step any protections desired.

> though big ISPs and corporations still routinely skirt it).

RFC 2476: MSA

 Unfinished messages need to be completed to ensure
 they conform to [MESSAGE-FORMAT], and later requirements.  For
 example, the message may lack a proper 'Date' header field, and
 domains might not be fully qualified.  In some cases, the MUA may be
 unable to generate finished messages (for example, it might not know
 its time zone).  Even when submitted messages are complete, local
 site policy may dictate that the message text be examined or modified
 in some way.  Such completions or modifications have been shown to
 cause harm when performed by downstream MTAs -- that is, MTAs after
 the first-hop submission MTA -- and are in general considered to be
 outside the province of standardized MTA functionality.

There is specific wording regarding checking the RFC 2821 return path
but only that compliance to RFC 822 be done to ensure "local only" paths
have the domain added, etc. I would not describe that as advice to
"check" that these RFC 822 paths are valid, only that they conform to
syntax requirements.

> phoning Sendmail and asking for a MARID implementation :)

How easy would it be to spoof, if not done at the MTA?  I did not think
the charter was to aid message filtering.  By making the authorization
based upon the message (RFC 2822), rather than the MTA (RFC 2821), the
overly broad nature of this task removes a clear definition of the
identity being checked.  In addition, the permission sets for this RFC
2822 check obscures which domain is administratively accountable for the
messages being emitted.  At least, when done for the RFC 2821 MAIL FROM,
this had the benefit of directly blocking the "blowback".  Checking the
RFC 2822 identity ensures an ability to continue the practice of using
RFC 2821 MAIL FROM spoofing. 

-Doug

---

From: Michel Bouissou <[email-masked]>
Date: Mon, 26 Jul 2004 23:49:15 +0200

> neither of these can validate consistency without stepping out of its

I don't really understand this necessity of "syncing" envelope with contents, 
unless I missed an essential part of the goals of MARID.

The major problem we have to face today regarding this is the huge quantity of 
spam/viruses that comes from zombie or exploited machines around the world, 
that pick-up a random e-mail address, connect directly to the destination MX 
and say:
HELO destination-domain.com	(or whatever)
MAIL FROM: <[email-masked]>

We aren't out to prevent the user from sending from work a mail which headers 
state "From: <[email-masked]>, we are out to prevent 
the massive disruption of e-mail that all these zombie and trojaned machines 
cause.

And this can perfectly be achived without having to "sync" envelopes and 
contents, or even to worry about contents at MTA level.

It would be enough to use SPF + SRS plus maybe some specifics for bounces, AND 
enforcing readily available state-of-the-art techniques.

If you publish a RFC saying:

1/ A machine that wants to send mail out MUST HELO using its FQDN, and the 
given FQDN MUST resolve in DNS to its actual IP address, otherwise receiving 
side SHOULD reject the connection.

2/ The machine that wants to send mail SHOULD be listed in SPF as a legitimate 
mail sender for the HELO it gave. i.e. MAIL FROM: 
<[email-masked]> SPF check SHOULD give "pass" and MAY give 
"neutral" or "none" (in these 2 last cases, the receiving side policy MAY 
decide to refuse the connection), but MUST NOT give "fail", "softfail", 
"error" or "unknown" otherwise the receving side SHOULD reject the connection 
with a 4xx temporary error.

4/ The given MAIL FROM: MUST be replyable (its right-hand part must resolve in 
DNS either to an A or a valid MX record) otherwise the mail SHOULD be 
rejected (most MTAs already implement this check); but in case of a DSN (MAIL 
FROM: <>), then we should make the check using the given HELO.

3/ The given MAIL FROM: SPF check SHOULD give "pass" and MAY give "neutral" or 
"none" or "softfail" (in these 3 cases the receiveing domain policy MAY 
decide to reject the message or submit it to further checks), and with an SPF 
"fail" the receiving side SHOULD reject the message with a 5xx permanent 
error.

Now, with this, you are already sure that you will only accept mail from 
machines that HELO properly with their real name, that are authorized mail 
sender for their domain (receivers can push towards SPF adoption by deciding 
massively to reject connexions for domains that have no SPF record).

Then the SPF check on the MAIL FROM permits checking that the sender is 
actually allowed to send from this machine, and SRS solves the forwarding 
issue.

At this step, you know that all e-mail you accept to receive comes from a MAIL 
FROM and a sending MTA that are mutually coherent, and authorized.

At this step, all e-mail from forging zombies is eliminated, and this is the 
main result we want to achieve.

Then, we could simply recommend that MUAs SHOULD display the <Return-Path> as 
well as the "From: " header field, and simply educate users in telling them 
that, when they differ, and if in doubt about the authenticity of a given 
message, the <Return-Path> is credible where the "From: " may not be.

We already have almost all the needed tools for achieving all this (and they 
already prove usable and efficient), without having to invent a very COMPLEX 
system such as PRA + SUBMITTER that doesn't give any true assurance that it 
can perform any better than what we already have readily available.

The PRA + SUBMITTER system really looks to me like a kludge on a kludge on a 
kludge, and if I'm not sure it will perform any better than a much simpler 
system such as what I suggest.

In France, we have a Shadok joke that says "Why do simple, when we can do much 
more complicated ?"

And that's exactly what PRA + SUBMITTER evokes to me.

---

From: Michel Bouissou <[email-masked]>
Date: Mon, 26 Jul 2004 23:56:02 +0200

Le lundi 26 Juillet 2004 23:01, Douglas Otis a écrit :
>
> By making the authorization
> identity being checked.

Absolutely. At 2821 stage, we currently have ONE un-ambiguous sender identity, 
which is MAIL FROM.

It is obvious to common sense that we should base our controls upon the 
provided identity that is given to us, rather than having to reinvent, dig 
out from the 2822 headers thru complex circonvolutions or whatever, another 
supposedly "better" identity that we would like to check instead of the one 
that has been given to us in the first place.

Shadok's word in french:
"Pourquoi faire simple quand on peut faire compliqué ?" ;-)

---

From: wayne <[email-masked]>
Date: Tue, 27 Jul 2004 23:07:49 -0500

> decided at the outset that we want to be able to validate BOTH 2821 and

No, it was decided that we would validate the RFC2821 identities
first, and then go on to the 2822 identities.

Yes, I know, the last time I brought this up (see
http://www.imc.org/ietf-mxcomp/mail-archive/msg02598.html ), both PHB
and you said that you didn't see who identities were important.  That
doesn't change the MARID charter and that doesn't change the co-chairs
ruling that 2822 identities would be considered after 2821 identities.

I dunno.  Maybe rulings by co-chairs aren't important and can be
ignored.

-wayne

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 28 Jul 2004 08:24:28 +0200

Michel Bouissou wrote:

> we could simply recommend that MUAs SHOULD display the
> <Return-Path> as well as the "From: " header field

Yes, I like this.  I'd recommend to treat MAIL FROM != From
_without_ Sender like MAIL FROM == Sender.  This would
solve a SPF classic / Sender-Id incompatibility for legacy
software.

For "MAIL FROM" read "address minus old routing stuff", for
"!= From" read "does not match any From address", and for
"== Sender" read "matches the Sender address".  Bye, Frank