Subject: Re: Last Call: draft-klensin-rfc2821bis (Simple Mail Transfer Protocol) to Draft Standard (1)

From: "Frank Ellermann" <[email-masked]>
Date: Sun, 9 Dec 2007 06:07:26 +0100

The IESG wrote:

> <draft-klensin-rfc2821bis-06.txt> as a Draft Standard
 
> The IESG plans to make a decision in the next few weeks,
> and solicits final comments on this action.

I-D.klensin-2821bis 3.9 as before RFC 2821 3.10 state in
essence, that relays, forwarders, and mailing lists have
no business to mess with the header of a mail:

| the message header section (RFC2822 [9]) MUST be left
| unchanged

This appears to be at odds with the mailing list RFCs,
adding various List-* header fields.  

It's disturbing if five IETF mail experts offer three
strictly incompatible opinions what mailing lists are
supposed to do with an existing "Sender" header field,
or if adding a "Sender" (as it's common practice) is
actually permitted.  

 Frank

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Frank Ellermann" <[email-masked]>
Date: Sun, 9 Dec 2007 06:17:14 +0100

> and solicits final comments on this action.

I-D.klensin-2821bis 2.3 contains a list of defined terms
which are *apparently* copied verbatim from RFC 2119 
without stating the source.  This is hostile to readers
trying to figure out what the difference might be, and
if bypassing RFC 2119 chapter 6 really is the goal a
less irritating solution should be possible, compare
<http://article.gmane.org/gmane.ietf.smtp/6260/>

 Frank

---

From: "Frank Ellermann" <[email-masked]>
Date: Sun, 9 Dec 2007 09:33:20 +0100

> and solicits final comments on this action.

To validate the syntax I've extracted all ABNF constructs
to a file, converting various 'Syntax: "verb" param CRLF'
contructs into 'verb = "verb" param CRLF' ABNF.  

Then I added all direct and indirect imports from 4234bis
and 2822upd.  There are three collisions where 2821bis
terms use the same name as similar 2822upd terms, I added
an "U" to these terms:  local-part-U, domain-U, atom-U.

After that the output of Bill's parser started to make
sense, I canonicalized the spelling of some terms to get
rid of all warning, arriving at two ABNF syntax errors in
http://hmdmhdfmhdjmzdtjmzdtzktdkztdjz.googlepages.com/2821bis.ABNF.2
(2821bis.ABNF.1 is the version with different spellings)

Other issues (apart from two typos reported as errors):

1 - <ehlo-greet> permits any character other than CR and
    LF.  That covers NUL and NO-WS-CTL, I'm curious what
    the interoperabilty test results will be for this 
    feature.  Bill's parser is also unhappy with %d0, but
    I didn't count this as error (it would be the third)

2 - <ehlo-param> excludes %d0-32 (NUL..SP), but permits
    %d127, it's not obvious why DEL is no problem here.

3 - <A-d-l> contains 2119 keywords in an ABNF comment.
    It's not obvious why source routes SHOULD NOT be
    generated and SHOULD be ignored, 18 years after
    they were deprecated.  A MUST would be clearer if
    the implementation report doesn't identify a 2821
    MTA with a good reason to violate these SHOULDs.

4 - <esmtp-value> permits DEL for no obvious reason.

5 - <Argument> is by definition an 2821bis <Atom>.  It
    would be clearer to avoid this conflict with the
    different 2822upd <atom> by using 1*atext directly:
    <Dot-string>, <String>, <ID> , <Addtl-Link>, and
    <Attdl-Protocol> are the 5 other <Atom> occurences.

6 - <Attdl-Protocol> and <Addtl-Link> are inconsistent,
    I can imagine cases where keeping a "typo" could be
    a good idea, but I don't see the necessity here.

7 - <Quoted-string> (not the same as the 2822upd term)
    uses 2822upd <qcontent>.  That pulls <qtext> with
    <NO-WS-CTL>, a known 2822upd issue.  It also pulls
    <quoted-pair> with <obs-qp> =  "\" (%d0 / LF / CR)

    NUL, bare LF, bare CR, permitted whereever 2821bis
    uses its <Quoted-string>.  The interoperability
    tests should explore if that really works anywhere.

8 - <General-address-literal> uses <dcontent> again with
    <NO-WS-CTL>, and a quick and dirty test some months
    ago published on the SMTP list already demonstrated
    that this does not work.  It's obscure why 2821bis 
    needs its own syntax for IPv6- and IPvFuture domain
    literals, different from the full Internet STD 66.
    <IPv6-addr> is impenetrable.

9 - <ID> permits a 2822upd <msg-id>, that pulls 2822upd
    <obs-id-left> and <obs-id-right>.  The reason why
    2822upd obsoleted these constructs was that they're
    not used in practice.  Maybe 2821bis needs a general
    "no 2822upd obs" clause, like the Netnews format RFC.

 Frank

---

From: Pete Resnick <[email-masked]>
Date: Mon, 10 Dec 2007 16:31:02 -0600

On 12/9/07 at 6:07 AM +0100, Frank Ellermann wrote:

>I-D.klensin-2821bis 3.9 as before RFC 2821 3.10 state in essence, 
>that relays, forwarders, and mailing lists have no business to mess 
>with the header of a mail:
>
>>the message header section (RFC2822 [9]) MUST be left unchanged
>
>This appears to be at odds with the mailing list RFCs, adding 
>various List-* header fields.

I believe this is a bogus complaint on two counts:

1. RFC 2369 and RFC 2919 (the ones that define the List-* fields) 
both extend [2]821 in the same way that the MIME documents extend 
[2]822. In each case, the former does things that the latter says one 
MUST NOT do (e.g., MIME says that it's OK to use other than US-ASCII 
characters even though neither 822 nor 2822 allow them). That's 
appropriate. Base specifications can be extended by other documents, 
and we may update base specifications without taking into account 
those extensions.

2. The mailing lists described in 2821 are very simple redistribution 
lists, as opposed to the "fairly sophisticated forums for group 
communication" [2919] described in these documents. For simple 
aliasing and redistribution lists, I think the "MUST be left 
unchanged" is appropriate.

>It's disturbing if five IETF mail experts offer three strictly 
>incompatible opinions what mailing lists are supposed to do with an 
>existing "Sender" header field, or if adding a "Sender" (as it's 
>common practice) is actually permitted.

Assuming people here on the general IETF list have no idea what 
you're talking about here, suffice it to say that the discussion that 
occurred on the DKIM list to which you refer is about whether the 
aforementioned more complex mailing list should be adding or 
modifying the "Sender" field. I think there is not consensus on what 
such mailing lists should do, let alone what DKIM should do about 
that, which IMO lends more support for the position of 2821bis: 
Mucking around with a header section leads to bad consequences, and 
until some more work is done in this area to figure out the "correct" 
thing to do, one MUST NOT go changing header sections.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Ned Freed <[email-masked]>
Date: Mon, 10 Dec 2007 15:22:36 -0800 (PST)

> I believe this is a bogus complaint on two counts:

I concur.

> and we may update base specifications without taking into account

I also note that as a practical matter it is necessary for any base
specification update to be stable before extensions that depend on that base to
themselves be updated. The alternative of attempting to do everything at once
is simply unworkable, and of course it makes no sense at all to attempt to
update the extensions to reflect some change in the base at  a point where it
is still possible for the base to change in nontrivial ways.

> unchanged" is appropriate.

The key point is that this is redistribution prior to final delivery.
Sophisticated list processors in effect submit a new message to the transport
infrastructure and do not fall under this prohibition.

Now, having said this, the minor one issue I do see with section 3.10 is that
the "lists MUST change the envelope from" requirement is actually a tautology:
Lists as distinct from aliases as defined as redistributors that change the
envelope from address. The requirement that lists do the one thing that defines
them appears to have originated in the department of redundancy department.

> thing to do, one MUST NOT go changing header sections.

It is also worth pointing out that the extensions defined in RFC 2369 and RFC
2919 are far from the only ones that relax this requirement in some way. MIME
downgrading, submit-time fixups, and content conversions are all processes
defined in various other RFCs that can modify message headers and there are no
doubt a bunch of others that I've forgotten to mention.

				Ned

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 11 Dec 2007 01:19:42 +0100

Pete Resnick wrote:
 
> I believe this is a bogus complaint

And I hope you're right.

> RFC 2369 and RFC 2919 (the ones that define the List-* 
> fields) both extend [2]821

Yes, they didn't bother to say "updates [2]821, but yes,
they obviously overrule the MUST in 3.9 (was 3.10).

> in the same way that the MIME documents extend [2]822.

I'm not aware of any apparent or real incompatibilities
between MIME and [2]822, they are very near to perfect.

> e.g., MIME says that it's OK to use other than US-ASCII 
> characters even though neither 822 nor 2822 allow them

Not over normal 7bit SMTP, it needs an extension or a CTE.
 
> Base specifications can be extended by other documents, 
> and we may update base specifications without taking
> into account those extensions.

SMTP extensions are typically offered by the server, and
accepted / used by the client as specified.  

> The mailing lists described in 2821 are very simple
> redistribution lists, as opposed to the "fairly 
> sophisticated forums for group communication" [2919]
> described in these documents.

Nothing's wrong with simple, especially not in *S*MTP ;-)

The spec. could note that there are mutilating^Wcomplex
lists violating the MUST.  It could also say SHOULD, an
RFC on standards track might be a good excuse to violate
this SHOULD (a SHOULD is also the shortest possible fix).

> For simple aliasing and redistribution lists, I think
> the "MUST be left unchanged" is appropriate.

IBTD, you mention DKIM below.  It's confusing if folks
make up their own definition of "originator", and it's
surreal if they add references to [2]822 or email-arch
with a very different definition.

Admittedly RFC 2369 and 2919 don't reference [2]821,
but as DS 2821bis trumps PS, and a MUST is critical by
definition.  If there are good excuses lets say SHOULD.

I like List-* header fields, they are far better than
manipulations of the body (breaking DKIM, needing work
to remove them in replies, often spam, often breaking
MIME, not what the author wrote, etc.)

> the discussion that occurred on the DKIM list to which
> you refer is about whether the aforementioned more
> complex mailing list should be adding or modifying the
> "Sender" field.

Yes, and 2821bis apparently and you certainly said "NO".

2822 doesn't directly mention the possibility, it offers
a MUST NOT wrt Resent-* header field.  Later resulting
in a confirmed appeal against 4406, and 2822upd will NOT
deprecate them.  Meanwhile the DKIM WG happily ignores
these facts wrt 5016 and SSP.

That's of course very far from 2821bis, but it's a mess,
and it would be good if at least the core spec.s (incl.
the future email-arch) are 100% clear.  A MUST that is
only a MUST in "simple" cases is a new concept for me.

> more support for the position of 2821bis: 
> Mucking around with a header section leads to bad
> consequences

I agree, but apparently nobody else agrees, and common
practice is to add a Sender (unlike the List-* header
fields I don't like this Sender.  Unless it's done as
specified for RFC 4406, which still violates a MUST 
about Resent-* in 2822upd as confirmed in an appeal.)

> until some more work is done in this area to figure
> out the "correct" thing to do, one MUST NOT go 
> changing header sections.

Sorry, I don't get it.  Keeping a MUST when other RFCs
ignore it for the *good* purpose List-* is beyond me.

We're talking about 2821bis, one of the most important
IETF RFCs, that's no stuff for a few geeks, it will be 
analyzed and discussed for years by thousands of users.

We talked *months* about a single *dot* (in the former
one-dot-only rule excluding TLDs) on three lists (SMTP,
SPF, USEFOR).
  
 Frank

---

From: Dave Crocker <[email-masked]>
Date: Mon, 10 Dec 2007 16:35:50 -0800

Pete Resnick wrote:
> Base 
> specifications can be extended by other documents, and we may update 
> base specifications without taking into account those extensions.

+1

> communication" [2919] described in these documents. For simple aliasing 
> and redistribution lists, I think the "MUST be left unchanged" is 
> appropriate.

A mailing list is re-posting a message.  From a purist's standpoint it is a 
new message.  However the simple lists that Pete is describing seek to be MTAs 
that do multi-casting.  They are not intending to more than be a relay, 
mapping what is really an intermediate address into a set of final recipients.

Having mailing lists like these discussed in 2821 seems reasonable, since they 
seek to emulate MTAs.  Not mentioning this class of relay is probably worse 
than the challenge of mentioning them.

Consequently, to Pete's point:  +1.

> Assuming people here on the general IETF list have no idea what you're 
> talking about here, suffice it to say that the discussion that occurred 
> on the DKIM list to which you refer is about whether the aforementioned 
> more complex mailing list should be adding or modifying the "Sender" 
> field. I think there is not consensus on what such mailing lists should 
> do,

+1.  Mailing lists have always presented architectural challenges.  No 
expectation that that will change anytime soon.

  let alone what DKIM should do about that, which IMO lends more
> support for the position of 2821bis: Mucking around with a header 
> section leads to bad consequences, and until some more work is done in 
> this area to figure out the "correct" thing to do, one MUST NOT go 

+1.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 11 Dec 2007 02:26:15 +0100

> and solicits final comments on this action.

JFTR, this is the single dot mentioned in another article:

| Domain         = ( sub-domain 1*("." sub-domain) ) / sub-domain

The new part is the added <sub-domain> alternative.  The
original fix proposed four years ago was to drop the "1":

| Domain         = sub-domain *("." sub-domain)

IMO the latter is clearer, OTOH the long version makes
the modification more obvious.  I forgot to mention this
point in my ABNF comment, and maybe it illustrates that
2821bis is important for the Internet at large.  We even
worried how that might affect the root servers.

 Frank

---

From: John C Klensin <[email-masked]>
Date: Mon, 10 Dec 2007 21:40:54 -0500

--On Tuesday, December 11, 2007 2:26 AM +0100 Frank Ellermann 
<[email-masked]> wrote:

>| Domain         = ( sub-domain 1*("." sub-domain) ) /
>| sub-domain
>

It doesn't affect the root servers, although it might affect the 
TLD servers.

   john

---

From: Ned Freed <[email-masked]>
Date: Mon, 10 Dec 2007 21:12:37 -0800 (PST)

> between MIME and [2]822, they are very near to perfect.

Its about the biggest incompatibility imaginable: MIME allows charsets besides
US-ASCII and both  8bit and binary content, RFC 2822 restricts things to 7bit
US-ASCII.

> Not over normal 7bit SMTP, it needs an extension or a CTE.
 
And that's the point. We're talking about extensions here. Regardless of
whether or not these are negotiated in some way, they are extensions
nevertheless.

> accepted / used by the client as specified.

Typically but not always. I can use MIME to transfer ISO-2022-JP text or binary
data encoded with base64, both of which are specifically  disallowed by
2821/2822 and neither of which require that the server offer anything other
than base SMTP.

> with a very different definition.

Which is DKIM's (or whoever else is doing this) problem to solve.

> definition.  If there are good excuses lets say SHOULD.

I would not object to changing this to a SHOULD, but I don't think there is
consensus to do so, especially since it would require a recycle at
proposed.

> these facts wrt 5016 and SSP.

Again, that's DKIM's problem to solve.

				Ned

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 11 Dec 2007 08:10:40 -0800

> infrastructure and do not fall under this prohibition.

Would it be a reasonable thing to add a note saying that "there exist 
things that call themselves lists but are doing processing other than what 
is described here; such lists should be described as delivery followed by a 
submission, and alterations are therefore outside the rules"?

I'd also be happy with losing the normative language about what lists do 
apart from replacing MAIL FROM; I don't see an upside to having the 
prohibition in here for lists, since it doesn't match widespread practice. 
(I do want to keep the prohibition for aliases!)

> them appears to have originated in the department of redundancy

My memory claims that it originated in the department of terminology war 
peace negotiations - when faced with having to come up with a definition of 
"alias" and "list", the group settled for "if it alters MAIL FROM, it's a 
list, if it doesn't, it's an alias". Thus, the real point of the section is 
the terminology (he said, redundantly).

               Harald

---

From: Pete Resnick <[email-masked]>
Date: Tue, 11 Dec 2007 11:31:27 -0600

Ned said most of what I wanted to, but a couple of little points:

On 12/11/07 at 1:19 AM +0100, Frank Ellermann wrote:

>The spec. could note that there are mutilating^Wcomplex lists 
>violating the MUST.  It could also say SHOULD, an RFC on standards 
>track might be a good excuse to violate this SHOULD (a SHOULD is 
>also the shortest possible fix).

This seems like nonsense lawyering instead of doing something useful 
in this document. Someone who is implementing a minimal SMTP really 
ought to respect the "MUST be left unchanged". Yes, it would be 
lovely if every document referred to every other document related to 
it, except that it would be a nightmare of criss-crossing documents.

Furthermore, see Dave's message re: why the simple "multi-casting" 
lists are worth talking about in 2821bis, whereas the other sorts are 
not.

>Admittedly RFC 2369 and 2919 don't reference [2]821, but as DS 
>2821bis trumps PS, and a MUST is critical by definition.  If there 
>are good excuses lets say SHOULD.

I most emphatically disagree that a Draft Standard "trumps" a 
Proposed Standard in this sense. A DS defines a well-understood and 
stable protocol. What is well understood and stable is that header 
changes cause damage.  A PS may propose that some header changes 
won't cause damage and are a good thing. That's fine. It doesn't mean 
that the MUST in a DS "trumps" a PS doing something different.

The rest of your message is wandering into DKIM and other spam 
related things having nothing to do with this discussion.

pr

---

From: Ned Freed <[email-masked]>
Date: Tue, 11 Dec 2007 16:21:13 -0800 (PST)

> submission, and alterations are therefore outside the rules"?

Seems like a reasonable addition to me.

> (I do want to keep the prohibition for aliases!)

So keep the MUST for aliases but lose it for lists? I could live with that
too but it would probably force a recycle.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Wed, 12 Dec 2007 15:47:43 +0000

> too but it would probably force a recycle.

I thought that removing requirements is OK during document progression.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
PORTLAND PLYMOUTH: SOUTH BACKING SOUTHEAST 4 OR 5, OCCASIONALLY 6 IN PLYMOUTH.
SLIGHT OR MODERATE. FAIR. MODERATE OR GOOD.

---

From: Ned Freed <[email-masked]>
Date: Wed, 12 Dec 2007 11:27:41 -0800 (PST)

> I thought that removing requirements is OK during document progression.

Removing capabilities and features is fine - encouraged even. This is removing
a restriction, which could be seen as adding any number of capabilities. That's
arguably not OK, although the determination is of course up to the IESG.

				Ned

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 12 Dec 2007 21:13:11 +0100

Ned Freed wrote:

> keep the MUST for aliases but lose it for lists? I could 
> live with that too but it would probably force a recycle.

I'd opt for "better standard" instead of "better status"
if these choices are in conflict.  The 2821bis / 2822upd
procedure is somewhat odd, first John insisted on doing
this simultaneously, and I thought that's not necessary.

Now I think he was always right, but what we really do 
is out of sync:  Likely 2822upd will pull NO-WS-CTL, or
move it to its "obs" chapter.  

But the 2821bis in Last Call has a normative reference to
2822, not 2822upd, and so it inherits obscure NO-WS-CTL
not limited to (non-existing) domain literals, it also 
has cruft like DEL and horrors like NUL in a few places.

Originally (2005) John also proposed to do the revision
in a WG, I'm more and more convinced that he was right.
 
 Frank

---

From: Ned Freed <[email-masked]>
Date: Wed, 12 Dec 2007 17:05:36 -0800 (PST)

> if these choices are in conflict.

I question whether they are in conflict. In particular, I question whether
removing this requirement will make this document significantly better, which
IMO is the appropriate criteria for assessing a change at this point in the
process.

> The 2821bis / 2822upd

> has cruft like DEL and horrors like NUL in a few places.

I would prefer to have a reference to 2822upd, but I suspect this decision
dates back to when it wasn't clear that 2822bis was going to get updated more
or less at the same time.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 13 Dec 2007 18:33:58 -0500

--On Wednesday, 12 December, 2007 17:05 -0800 Ned Freed
<[email-masked]> wrote:

> I would prefer to have a reference to 2822upd, but I suspect
> this decision dates back to when it wasn't clear that 2822bis
> was going to get updated more or less at the same time.

This is, of course, the sort of thing that can be changed in an
RFC Editor note if, in fact, 2822 is ready and more or less
concurrent.

   john

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 14 Dec 2007 02:05:18 +0100

John C Klensin wrote:

 [2822 vs. 2822upd] 
> RFC Editor note if, in fact, 2822 is ready and more or less

Modifying the reference is simple.  Removing various NO-WS-CTL,
DEL, and a NUL in 2821bis itself, not inherited from 2822(upd),
for consistent usage with 2822upd (and net-utf8) isn't trivial.

Maybe the interoperabilty report will say "nobody uses it, and
some implementations don't support it".  It could also arrive
at the opposite conclusion, "most implementations survive it,
some even use it" (I'd be curious for *what*).  Or the likely
outcome "who knows, nobody mentioned NO-WS-CTL and NUL in such
obscure places as greetings or domain literals".

Obscure domain literals didn't work everywhere, and that's a
"simple" 2822upd <dcontent> case.  Where it's not yet clear
what 2822upd will do, move it to "obs" or to /dev/null.

If 2822upd moves it only to "obs", then 2821bis needs its own
syntax without NO-WS-CTL, because NO-WS-CTL in IPvFuture (or
rather the 2821bis name for IPvFuture) is already known to be
neither used nor interoperable.

For the other cases of DEL, NO-WS-CTL, and NUL we haven't seen
any tests so far.  For various 127 in the ABNF it's plausible
that they are mere typos, excluding CTL but allowing DEL makes
no sense (i.e. there would be a comment if that's intentional).

 Frank

---

From: Ned Freed <[email-masked]>
Date: Thu, 13 Dec 2007 17:11:54 -0800 (PST)

> RFC Editor note if, in fact, 2822 is ready and more or less

That's what i was hoping you'd say - if 2822upd is ready such a change would be
a very good thing IMO.

				Ned