Subject: Re: Last Call: draft-resnick-2822upd (Internet Message Format) to Draft Standard

From: Brian E Carpenter <[email-masked]>
Date: Fri, 04 Apr 2008 13:23:36 +1300

I am disturbed that the messy situation of X- headers,
created by RFC 2822's silence on the subject, has
not been fixed.

See http://www.ietf.org/IESG/APPEALS/klensin-response.txt
for an example of the issues that this silence can
create.

I believe it would be appropriate to document that although
X- headers are widely used, they are not part of the standard
format and their treatment by Internet MTAs MUST NOT be relied on,
unless registered under RFC 3864.

   Brian
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 4 Apr 2008 14:43:58 +0200

Brian E Carpenter wrote:

> created by RFC 2822's silence on the subject, has not
> been fixed.

As far as 2822 and 2822upd are concerned header fields
not specified in 2822 or 2822upd resp. are covered by
<optional-field> in section 3.6.8.  This section does
not talk about field-names starting with "X-" or not.
 
> for an example of the issues that this silence can create.

Gateways are always a difficult topic, and the 2822upd
syntax *minus* obs-* constructs is hopefully friendlier
to gateways than RFC 2822 *minus* obs-*.  

Including obs-* constructs:  2822upd is slightly better
than before, a few RFC 822 #-cases not covered in 2822
are now accepted as obsolete, ASCII art with commas and
similar oddities.

> I believe it would be appropriate to document that 
> although X- headers are widely used, they are not part
> of the standard format and their treatment by Internet
> MTAs MUST NOT be relied on, unless registered under
> RFC 3864.

RFC 822 said that X- headers will *not* be standardized,
they are reserved for e-X-periments (my interpretation).
Do you propose that 2822upd should copy this rule from
RFC 822 ?  Sorry, but I'm not sure what you are up to.

An MTA not supporting header X-foobar is not forced to
support header foobar only because it has no X-.  As
far as 2822upd is concerned both are <optional-field>s.

 Frank

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 04 Apr 2008 15:03:32 +0200

> created by RFC 2822's silence on the subject, has

Me too.

> format and their treatment by Internet MTAs MUST NOT be relied on,

Agreed.

Further, one could discuss that using X- have caused interop problems
when standardizing the header field.  Old applications only know about
the X- form and would not know the non-X- form, which makes it difficult
to standardize the X- header field.  It may be preferable to avoid using
X- in experiment intended to be standardized later on.  This point of
view may be more contentious than what you propose though.  I'd be
interested to understand if others share this opinion.

> unless registered under RFC 3864.

I'd prefer to avoid this escape mechanism.  Can X-* headers really be
registered under RFC 3864?  RFC 822 says:

        Note:  The prefatory string "X-" will never  be  used  in  the
               names  of Extension-fields.  This provides user-defined
               fields with a protected set of names.
...
     extension-field =
                   <Any field which is defined in a document
                    published as a formal extension to this
                    specification; none will have names beginning
                    with the string "X-">

/Simon

---

From: John C Klensin <[email-masked]>
Date: Fri, 04 Apr 2008 17:47:12 -0400

> are now accepted as obsolete, ASCII art with commas and

Yes, but that has little or nothing to do with Brian's comment,
at least as I understand it.  

>> MTAs MUST NOT be relied on, unless registered under

Yes, but registering them really isn't a good idea either, IMO.
 
> far as 2822upd is concerned both are <optional-field>s.

There are two ways to interpret the "X-" and I think they yield
different answers about what should be done.   

If they are seen as experimental, we promptly run into the
problem that, if I recall, caused DRUMS to remove the text:  If
the experiment succeeds, it is hard to get rid of the "X-" part.
If it doesn't succeed, then the header is likely to vanish with
little trace whether it contains "X-" or not.  And one clearly,
at least in the retrospect of the last many years, wants to have
experimental headers registered (presumably in the 3864
registry) to reduce the odds of header names being reused for
conflicting purposes.

On the other hand, they can be seen as "private-use between
parties who have adequately identified each other and therefore
know what they mean".  For a private-use header, there is no
requirement to register the thing externally because, once one
registers (and ideally describes) them, they are no longer
private.  

Our history of putting "X" (with or without the subsequent
hyphen) in front of things --remembering that there have been
similar provisions at one time or another in FTP, SMTP, and
elsewhere-- has tended to get "experimental" and "private use"
muddled, muddled enough that I think we've often not understood,
or lost sight of, the difference.

I don't know if Brian would agree, but my inference from the
Lemonade appeal and a couple of other rough edges consequent of
removing 822's text about X- headers is that, ideally, we should

(1) Partially restore the 822 text, stressing "private use",
rather than "experiental".

(2) Treat <optional-field> as consisting of either "X-headers"
or "Non-X-headers" (probably there are better names).

(3) Encourage X-headers for strictly private use, i.e., they
SHOULD NOT be used in any context in which interchange or
communication about independent systems is anticipated and
therefore SHOULD NOT be registered under 3683.

(4) Make it clear that the distinction between X-headers and
Non-X-headers is guidance, not firm rules.  Should an X-header
become widely deployed (for a definition of "widely" I don't
think we need to pin down), then it is perfectly reasonable to
treat it as an ordinary (non-X) header, register it, and even
potentially write up RFCs describing it.  We just recommend
against getting into that situation if possible.

Note that this proposed remedy restores the 822 behavior and
recommendations for headers that are _never_ expected to enter
standard use, so it is not a new feature.   It preserves the
2822 recommendations for optional (unspecified) header names,
including preserving it for X-headers that have become popular.
So this is a conceptual improvement and better guidance for
designers of new protocols or features (in or out of the IETF)
without changing anything operationally.

Just my opinion.   And I don't feel strongly about it as
evidenced by the observation that I probably wouldn't have
bothered to say anything had Brian and others not brought it up.

      john

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 05 Apr 2008 17:05:55 +1300

John,

I think I agree with your suggested path.

On 2008-04-05 02:03, Simon Josefsson wrote:
...
> Can X-* headers really be registered under RFC 3864? 

One X- header is provisionally registered under RFC 3864
(and is marked in the registry as 'deprecated').

On 2008-04-05 01:43, Frank Ellermann wrote:
...
> Sorry, but I'm not sure what you are up to.

My underlying concern is that 2822upd should not appear
ridiculous to anyone who looks at a typical mail header
and sees the X-headers. John's message reached me with
X-Original-To, X-Virus-Scanned, X-Spam-Flag, X-Spam-Score,
X-Spam-Level, X-Spam-Status, X-Mailer, X-BeenThere, and
X-Mailman-Version. Leaving this completely undocumented
harms the relevance of the standard.

   Brian

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 5 Apr 2008 17:29:04 +0200

Brian E Carpenter wrote:

> and sees the X-headers.

2822upd specifies only about twenty mail header fields.
The rest is either registered and specified elsewhere,
or to some degree "unknown" (unregistered, unspecified,
proprietary, private use, experimental, spam, and so on).

> John's message reached me with X-Original-To,

A draft about Original-* didn't make it so far, it could
be classified as "could be specified if folks go to the
trouble to try it".  IMO it's no bug in 2822upd if nobody 
tries it.  RFC 3864 would have to be updated for generic
Original-* registrations.

> X-Virus-Scanned, X-Spam-Flag, X-Spam-Score, X-Spam-Level,
> X-Spam-Status,

Hopefully these header fields are specified in the manual
of anti-spam tools used by MTAs on your side.  There are
only two new and specified trace header fields since SMTP
was invented so far.  IMO the 2822upd section about trace
header fields was improved.

A draft Authentication-Results header field specification
exists, as next step to tackle this zoo of "private use"
header fields.  Two years ago we had no precedence of ever
introducing new trace header fields - carefully extracting
one good thing I can say about the 1st new trace field. ;-)

> X-Mailer

An RFC introducing the known HTTP User-Agent also for news
was approved.  From there it's in theory straight forward
to adopt it also in mail.  Adding a User-Agent to 2822upd 
would be sneaky, 2822upd is intended for DS, User-Agent in
mail was never published in a PS.

> X-BeenThere

That's a really interesting beast, claiming that it's only
"private use" would miss the point, and specifying it as
"BeenThere" without X- prefix could have disadvantages.

> X-Mailman-Version

In the direction of User-Agent and tracing.  It's hard to
define what constitutes "potential useful info", and the
security / privacy / I18N considerations for header fields
can be also rather tricky.  Maybe an informative reference
to RFC 4249 in 2822upd would be good.

> Leaving this completely undocumented harms the relevance
> of the standard.

The RFC 3864 reference in 2822upd is informative, would a
normative reference help for your conerns ?

 Frank

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 5 Apr 2008 18:17:27 +0200

John C Klensin wrote:

 [comparison of 2822upd and 2822]
> Yes, but that has little or nothing to do with Brian's
> comment, at least as I understand it.

Sure, for readers not familiar with 2822upd I wanted to
have it on record that 2822upd did something wrt gateways
(one topic of your appeal cited by Brian).  2822upd also
did something wrt trace header fields.  

Admittedly 2822upd doesn't update RFC 3864, or talk about
X- header fields, which might be good or bad depending on
the X- header field.  For X-Mailer I think specifying it
is a waste of time, adopting User-Agent would be better.

Similar for almost all other X- header fields found by
Brian in your message.  The X-BeenThere might be special,
very near to the "above all, avoid loops" in son-of-1036.

 [several proposals] 
> I don't feel strongly about it as evidenced by the 
> observation that I probably wouldn't have bothered to
> say anything had Brian and others not brought it up.

Same here.  I argued that the X-Archived-At registration
is an odd way to terminate this experiment in favour of
Archived-At, but field names are no scarce resource, no
harm done.  I'm still not sure if what Brian wants should
be addressed in 2822upd or in a future 3864bis, though.

 Frank

---

From: John C Klensin <[email-masked]>
Date: Sat, 05 Apr 2008 16:19:10 -0400

> of anti-spam tools used by MTAs on your side.

This is precisely the reason why, although I think the sort of
text I suggested would be useful, I don't think we should go
much further down that path.  Most of these fields would be
reasonable candidates for the "private use" category -- they are
used to communicate between what SMTP sees as the final delivery
MTA and the message store and/or receiving MUA (either
server-side or client-side in split MUA models).    If one tried
to define "private use" (and I suggest that we do not) part of
the process would involve asking the question "could this be
standardized in 2821* or used-on-the-Internet-wire-2822*".   For
these things that communicate among modules on the far side of
the final delivery MTA, the answer is that any standardization
would have to occur in IMAP or POP, or in the never-defined
message store standard, but not in SMTP.   Maybe LTMP, used as a
message store protocol, changes that and maybe it does not (the
fact that it is Informational and not standards-track doesn't
help), but you see where this line of reasoning is headed.

And, speaking personally, I don't care whether private use
headers use non-X field names and are registered.   What I want
to push back against is (i) intentional use of X-headers for
non-private-use applications and (ii) anything (including (i))
that would create a requirement (or strong reason) to register
anything starting with "X-".

In any event, sorting out those special-use header fields is not
a problem for 2822bis.

> normative reference help for your conerns ?

IMO, probably not.  More important, I'd really object to
standardizing a version specification for a single type of
redistribution software.  If this is needed at all, it should
probably be turned into something like
  List-Exploder: <name>, <version>
and that, of course, would be part of a different spec entirely.

    john

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 06 Apr 2008 09:56:51 +1200

Frank,

> proprietary, private use, experimental, spam, and so on).

Sure. But X- is described *nowhere* in current documents;
the one example in provisional registration is marked as
"deprecated" on no basis whatever (there is nothing in
2822 or 2822upd or 3864 to suggest a "deprecated"
registration), and...

> A draft about Original-* didn't make it so far, ...

I don't care about the details of unregistered headers.
I simply want to see a statement of reality in 2822upd.
It can be simple or complicated but IMHO it needs to
say

 X- headers are in widespread use.
 They are not part of this standard.
 They must not be relied on to have any particular semantics
 or even to be delivered.

OK, I've said all I want to say on this.

     Brian

---

From: Dave Aronson <[email-masked]>
Date: Sun, 06 Apr 2008 11:40:47 -0400

Some (though admittedly not all) of the problem could be mitigated if 
software authors were encouraged (not forced!) to have their programs:

  1) accept new X-headers both with and without the X-, and
  2) be configurable to send either, defaulting to "with".

If a given header gets approved, and registered without the X-, the 
software need not be changed and redeployed, or even reconfigured, to 
*accept* the new version.  Having it *send* the new version, would be a 
matter of configuration, hopefully simple.

As new versions of the software are written, support for the X-version 
could even be removed, preferably after some agreeable fairly long 
amount of time.  (A "deprecated" registry would certainly help there. 
That could be cluttered if ALL abandoned X-headers are listed, but it 
could be restricted to those that have been deprecated in favor of 
things that became official.)

The main problem I see is site admins too far behind the times to bother 
flipping the switch to send without an X-, nor upgrade to versions that 
don't support sending the X-.  Their users may send to sites that have 
upgraded to versions that no longer accept the X- version.  In this 
case, the header will not be recognized.

One could take at least approaches here.  First, one could simply say 
"X-headers are experimental, not to be relied upon, so who cares, tough 
luck, if it's that important to you, get your admin to flip the switch." 
  Second, it could be mitigated by simply having the software continue 
to accept the X- version, rather than deliberately removing X- support.

Your thoughts?

-Dave

-- 
Dave Aronson
"Specialization is for insects." -Heinlein
Work: http://www.davearonson.com/
Play: http://www.davearonson.net/

---

From: John C Klensin <[email-masked]>
Date: Sun, 06 Apr 2008 11:58:00 -0400

--On Sunday, 06 April, 2008 11:40 -0400 Dave Aronson
<[email-masked]> wrote:

> Some (though admittedly not all) of the problem could be
> mitigated if software authors were encouraged (not forced!) to
> have their programs:
> 
> If a given header gets approved, and registered without the
> X-, the software need not be changed and redeployed, or even
> reconfigured, to *accept* the new version.  Having it *send*
> the new version, would be a matter of configuration, hopefully
> simple.
> 
> As new versions of the software are written, support for the
> X-version could even be removed, preferably after some
> agreeable fairly long amount of time.  (A "deprecated"
> registry would certainly help there. That could be cluttered
> if ALL abandoned X-headers are listed, but it could be
> restricted to those that have been deprecated in favor of
> The main problem I see is site admins too far behind the times
> to bother flipping the switch to send without an X-, nor
> upgrade to versions that don't support sending the X-.  Their
> users may send to sites that have upgraded to versions that no
> longer accept the X- version.  In this case, the header will
> not be recognized.
> 
> One could take at least approaches here.  First, one could
> simply say "X-headers are experimental, not to be relied upon,
> so who cares, tough luck, if it's that important to you, get
> your admin to flip the switch."   Second, it could be
> mitigated by simply having the software continue to accept the
> X- version, rather than deliberately removing X- support.
> 
> Your thoughts?

Interesting, probably impractical, almost certainly irrelevant
to the text that goes into 2822bis because this would constitute
a new processing requirement (although one might get it through
by insisting that it is just a clarifying suggestion).  The
problem of how to turn "experimental" command verbs, header
lines, etc., has been with us for a very long time.  We have
never figured out how to get it right.  The discussion in RFC
1123 Section 4.1.3.1 about something that the authors though had
been fixed in RFC 959 (four years earlier) is a symptom of the
problem, but by no means the earliest case.

My belief that your idea is impractical is reinforced by the
knowledge that many "firewall" SMTP implementations will simply
remove any X- header and any other header that they do not
recognize on the assumption that it is either private-use and
hence doesn't belong there or that it is a potential vector for
attack or hidden information disclosure.

In my view, RFC 2822 adopted an innovative solution to the
problem, which was to abolish all distinctions and pretend that
there was no history that would lead software to assumptions
about anything that started in an "X-".   My view (I think
consistent with Brian's) is that the 2822 approach has
demonstrably failed and therefore that, for the private-use
cases as least, we need to move the text toward a clear
restatement of the intent of the 822 text. 

best,
    john

---

From: Pete Resnick <[email-masked]>
Date: Mon, 7 Apr 2008 11:14:54 -0700

Coming to consensus on this is going to be messy, as it was in DRUMS, 
which is what landed us with no comment in the document. To wit:

On 4/4/08 at 5:47 PM -0400, John C Klensin wrote:

>If they are seen as experimental, we promptly run into the problem 
>that, if I recall, caused DRUMS to remove the text: If the 
>experiment succeeds, it is hard to get rid of the "X-" part. If it 
>doesn't succeed, then the header is likely to vanish with little 
>trace whether it contains "X-" or not. And one clearly, at least in 
>the retrospect of the last many years, wants to have experimental 
>headers registered (presumably in the 3864 registry) to reduce the 
>odds of header names being reused for conflicting purposes.

I think on this point, folks in DRUMS agreed: If it's experimental 
(in the sense that it is hoped that *this* field name will eventually 
be interoperable), we *don't* want it to have an "X-". We want it 
registered in the provisional registry. And for this, the 
"optional-field" syntax in 2822[upd], which is what 822 called 
"extension-field", is exactly what you want.

>On the other hand, they can be seen as "private-use between parties 
>who have adequately identified each other and therefore know what 
>they mean". For a private-use header, there is no requirement to 
>register the thing externally because, once one registers (and 
>ideally describes) them, they are no longer private.

But this is the real problem that DRUMS got stuck on: It was argued 
by opponents of "X-" fields that "private-use" fields (what 822 
referred to as "user-defined-filed") are never really private; they 
invariably leak onto the net writ large. And once that happens, 
restrictions on the registration or other public description of those 
fields becomes a silly state: I can't register or document it because 
I named it with an "X-", but it's leaked out onto the net and lots of 
people are using it, so it needs documentation. "X-" fields (in the 
private-use sense) were supposed to mean "You will be able to use 
this name without fear of conflicting with another field of the same 
name out on the Internet." In practice, that's simply false.

There are those of us who, during DRUMS, concluded that private-use 
is not only a broken concept, but that 2822 ought to have said 
"trying to use 'X-' fields to indicate private-use is a bad idea 
because it won't work and you shouldn't do it." We could not garner 
consensus for such text.

So, in all honesty, I don't think the problem is because we 
misunderstood the difference between "experimental" and 
"private-use"; it's because (some people claim) that private-use is a 
bogus concept, but we couldn't get consensus to say so.

>(1) Partially restore the 822 text, stressing "private use", rather 
>than "experiental".

I don't think we'll be able to do this; see (3) below.

>(2) Treat <optional-field> as consisting of either "X-headers" or 
>"Non-X-headers" (probably there are better names).

That's what it does now.

>(3) Encourage X-headers for strictly private use, i.e., they SHOULD 
>NOT be used in any context in which interchange or communication 
>about independent systems is anticipated and therefore SHOULD NOT be 
>registered under 3683.

I think this is DOA. There are many folks (myself included) who think 
this should not be encouraged in any way, shape, or form.

>become widely deployed (for a definition of "widely" I don't think 
>we need to pin down), then it is perfectly reasonable to treat it as 
>an ordinary (non-X) header, register it, and even potentially write 
>up RFCs describing it.  We just recommend against getting into that 
>situation if possible.

But if we say that, then one has to ask oneself what the use of "X-" 
fields is in the first place. If you can't depend on any field that 
starts with an "X-" being of private use, then the "X-" hasn't bought 
you anything. In which case, it would be better to include text 
saying, "Since 'X-' field names were once used to mean 'private-use' 
only, and since no field can be guaranteed to ever be private-use 
only, we discourage the use of any field starting with 'X-' so as not 
to confuse folks into thinking it's private-use. It may not be."

I don't think we can get consensus on such text.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Dave Crocker <[email-masked]>
Date: Mon, 07 Apr 2008 11:34:21 -0700

...
> this should not be encouraged in any way, shape, or form.

Folks,

One of the lessons of the community's 30+ years of protocol work is that 
specification details which are actually usage guidance, rather than concrete 
interoperability details, often have little impact on a global community.  The 
community formulates its own preferences.

When X- as original proposed, I thought it was marvelously clever.  I still do.

But it doesn't work.

While it does protect a privately-developed header field label from being 
preempted by a standards process, it creates a much more serious problem of 
moving from private-use to public standards and having to (try to) re-label the 
field.  This is a highly disruptive impact./

In other words, if the model is true that existing practices get standardized -- 
and in this realm they often are, I think -- then we need to design things to 
make the transition from private-to-public be comfortable.  Defining a 
private-use naming space runs counter to that goal.

Valuable lesson.  We should learn it.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf