Subject: [EAI] Proposal for Alternate ABNF (was: Consensus Issue #6: current metalanguage model acceptable?)

From: Dave CROCKER <[email-masked]>
Date: Thu, 20 Jan 2011 16:04:36 -0800

Folks,

The answer to consensus call question 6 should be 'no'.

Ned Freed, Pete Resnick and I have been discussing the working group's basic
model for handling UTF-8 within a legacy, 7-bit world. The current approach is
certainly in line with the long-standing IETF pressure (demand) to have changes
of an existing system be designed in a way that protects and interworks with the
installed base, such as by not damaging a legacy system that receives an
enhanced form of data. And indeed, the working group's SMTP extension is an
example of keeping the new object away from legacy-only systems.

However, the model used in the case of RFC 5335 -- using new syntax elements
instead of extending old ones -- has produced quite a bit of complexity. In this
case, we believe the complexity is not serving the working group's goals all
that well.

That leads us to make a somewhat surprising suggestion to simplify
things considerably. The premise behind the suggestion is that the
world really does have extensive infrastructure for UTF-8 already and
that we can rely on it.

         So, the core suggestion is to have the revision focus
         only on the existing, basic 5322 ABNF constructs and
         simply augment them to support UTF-8.

         This would create what essentially would be an independent,
         UTF-8 email world.

         Interworking with the legacy email world still needs to be
         specified, but it should be done separately.

With the caveat that IDN handling in the spec still needs to be done, this 
produces a candidate to replace for Section 4.3, 4.4 and 4.5 from the existing 
draft:

     [[ --------------------------------------

This section specifies UTF-8 enhancements for the header of an
Internet Mail message, as defined in [RFC5322].

ABNF used in this section is taken from that specification and the
ABNF specification.

This specification retains the [RFC5322] rules for defining header
field names. The bodies of header fields are allowed to contain UTF-8
characters, but the header field names themselves must contain only
ASCII characters.

The following rules extend the corresponding rules in [RFC5322] and
[RFC5234] in order to allow additional Unicode characters.

    VCHAR   =/  UTF8-non-ascii

    ctext   =/  UTF8-non-ascii

    atext   =/  UTF8-non-ascii

    qtext   =/  UTF8-non-ascii

    {{ how to add IDN to this? }}
    domain  =   dot-atom / domain-literal / obs-domain

This means that all the [RFC5322] constructs that build upon these
will permit UTF-8 characters, including comments and quoted strings.

    <field-name>

           [RFC5322] has the rule <field-name> which specifies
permissible names for user-defined header fields. The current
specification defines no changes to that rule.

    <msg-id>

           This ABNF enables Message-ID strings to be full UTF-8.
However the specification directs that Message-ID strings SHOULD be
restricted to ASCII.

      -------------------------------------- ]]

The SMTP extension is essential to this, of course. In addition it's worth
repeating that specification of UTF8-to-ASCII gatewaying is still needed, but we
suggest it be moved to a separate specification.

Since these are very basic ABNF rules, changes to them have an
extensive effect that is not immediately obvious.  So to aid the
reader, it's worth adding an appendix that lists the other ABNF of
RFC5322 that would be affected (ignoring the obs- rules)...

      [[ --------------------------------------

A. Changes to support UTF-8

This section provides a basic audit of the places in a message that
now can permit UTF-8 rather than being restricted to ASCII, based on
the changes to underlying ABNF. The audit ignores rules for "obsolete"
constructs in RFC 5322.

VCHAR:
     quoted-pair, unstructured
     > ccontent, qcontent
     > comment, quoted-string
     > word, local-part
     > phrase
     > display-name, keywords

     > name-addr
     > mailbox, group
     > address

     > address-list, mailbox-list

     > received-token, received

     > from, sender
     > reply-to, to, cc, bcc, resent-*

     > subject, comments, optional-field

ctext:
     ccontent > comment > comments

atext:
     atom, dot-atom-text

     > msg-id

     > local-part, domain
     > addr-spec
     > mailbox

qtext:
     qcontent quoted-string

      -------------------------------------- ]]

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Claudio Allocchio <[email-masked]>
Date: Fri, 21 Jan 2011 12:09:02 +0100 (CET)

> The answer to consensus call question 6 should be 'no'.

the model used in current specificaions was one of my biggest worries in 
my review of the specification (see below);

> example of keeping the new object away from legacy-only systems.

which is correct.

> case, we believe the complexity is not serving the working group's goals all

If we follow the current model, we will legitimate EAI-compliant systems 
to inject into the legacy installed base messages which are suppoused to 
be conformant, because we "updated" some specifications, instead of 
"extending them". As said earlier, however, we cannot expact the installed 
base to "update" so quickly, and this can result in serious problems, 
including loss of data (imagine a logging system which crashes because it 
receives someting which it cannot handle correctly, thus loosing the 
records still in memory).

>        specified, but it should be done separately.

+1

> produces a candidate to replace for Section 4.3, 4.4 and 4.5 from the 
> existing draft:
>
>

another +1
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 21 Jan 2011 12:24:16 -0000

> case, we believe the complexity is not serving the working group's goals  

-1

I find writing different syntax using, for example, <utext> in place of  
the 5332's <text> leads to perfectly clear syntax with no possibility of  
confusion. We took just that approach in RFC 5536.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John C Klensin <[email-masked]>
Date: Sat, 22 Jan 2011 06:23:35 -0500

--On Thursday, January 20, 2011 16:04 -0800 Dave CROCKER
<[email-masked]> wrote:

>...

I can live with this approach.  If it is adopted, I'll help sort
out the domain name part.

    john

---

From: John Levine <[email-masked]>
Date: 22 Jan 2011 20:00:51 -0000

>I can live with this approach. 

Same here.  It seems to me that no matter what the ABNF says, once we
allow UTF-8 in headers, people will put it everywhere so the ABNF might
as well match the likely reality.

R's,
John

---

From: [email-masked]
Date: Sat, 22 Jan 2011 14:11:34 -0800 (PST)

> as well match the likely reality.

Bingo. Just as one example, message ids are often synthesized from available
host and domain information. If that information contains UTF-8, there's going
to be leakage into message ids from careless clients.

One thing both the old and new ABNF refrains from doing is adding utf-8 to
header labels. I think this is a defensible position and thus a reasonable
place to draw the line, although there's bound to be a little leakage even
there.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 22 Jan 2011 22:55:21 +0000

> Folks,

Hi Dave,
Speaking as an individual participant (and not with my AD hat on): I 
think this is a big improvement and I generally support this direction.

I do have one specific issue though, which, if agreed by others, can be 
thought of as a minor detail:
 [...]

>     quoted-pair,

   quoted-pair     =   ("\" (VCHAR / WSP)) / obs-qp

I have to say that I am not happy about any changes to quoted-pair. I 
haven't been around IETF for that long (as compared to other people 
participating in this discussion), so if I am saying something stupid 
here, I plead forgiveness. My view is that any slash escaping mechanism 
needs to be left as is. It only allows escaping of one following octet 
and changing this to escape multioctet sequences is going to require 
changes to parsers of this field for no good reason.
Besides non-ASCII UTF-8 sequences can already be included without escaping.

> unstructured

No objections to extending this one.

---

From: John Levine <[email-masked]>
Date: 22 Jan 2011 23:45:53 -0000

>changes to parsers of this field for no good reason.

No matter what we do here, we'll open a can of worms, so a reasonable
goal is to pick the smallest possible can.

The argument about parsers is unpersuasive. A reasonable way to handle
UTF-8 text is to make a prepass over it to turn the entire string into
16 or 32 bit Unicode characters, then parse those, particularly on
computers like the IBM z/Series that have instructions to do the
translatation at high speed.  In that environment, the obvious thing
to quote is a character, not a byte, and I would argue that the
hackery involved to handle a quoted single byte in that environment is
a lot worse than adjusting the code in a byte at a time system to
handle a UTF-8 character.  Adapting a header parser for EAI messages
will introduce lots of changes to the byte at a time code to handle
UTF-8, and this is just another one.

More importantly, if you can quote an arbitrary byte, you now have the
ability to introduce arbitrary things into the message that are not
valid UTF-8.  Those worms have large painful fangs, and I would just
as soon not let them into my code or my messages.

You're right that quoting a non-ASCII character doesn't do anything
useful, but let's make sure it doesn't do anything awful, either.

So a quoted pair is backslash and a character, not a backslash and a
potentially not-a-character byte, and the ABNF change is correct.

R's,
John

---

From: [email-masked]
Date: Sat, 22 Jan 2011 18:56:24 -0800 (PST)

> Besides non-ASCII UTF-8 sequences can already be included without escaping.

I have to say I share your concern here.

> No objections to extending this one.

Well, the problem is essentially a structural one in RFC 5321 - instead of
defining a single character repetoire for unstructured (which would logically
be called utext), the specification simply references VCHAR, which is a core
ABNF production. THis is a case where an ABNF shortcut ended up having
an undesireable coupling effect.

Had unstructured been defined this way:

    unstructured    =   (*([FWS] utext) *WSP) / obs-unstruct

    utext = VCHAR

we would not have this problem. So, while it isn't as clean, instead of
extending VCHAR we could redefine unstructured this way. The only effect
doing things this way changes is it gets rid of the effect on quoted-pair
and obs-unstruct.

This approach has two other advantages. One, we're not messing with a core ABNF
production any more. Two, we're not modifying the obsolete syntax this way,
which is the same as how ctext works so it's a bit more consistent.

				Ned

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 23 Jan 2011 08:03:04 -0800

On 1/22/2011 6:56 PM, [email-masked] wrote:
>> I have to say that I am not happy about any changes to quoted-pair.
...
...
> which is the same as how ctext works so it's a bit more consistent.

I haven't formed my own opinion about how best to handle these concerns, yet. 
So this note is meant for exploration...

Models for doing quoting vary between "provide an escape mechanism only for 
'special' characters" versus "make the quoting mechanism generic and allow ANY 
character to be quoted'.  There is, as noted, also the difference between 
escaping a collection of bits versus escaping "characters".

The quoting mechanism in RFC 5322 chose the model of allowing any (visible) 
character to be quoted.  And this did mean character, not "bag of bits", I 
believe.  One major advantage of any vs. selective characters is that a user is 
free to use quoting without having to worry about whether it's a special 
character.  So there's a usability benefit that's significant. However, in this 
case, the specification uses the human factors trick of "visibility".

The fact that the existing mechanism is already limited to a subset of full 
ASCII means that continuing to restrict the set of characters that can be quoted 
would merely be a continuation of that model.  It's also true that there is no 
"need" to quote other UTF-8 characters, I suppose.

That said, I'm leaning towards having the mechanism require as little thought by 
the user as possible.  Having to think about whether the character is below x07f 
or above it strikes me as poor human factors.

The modification to define <utext> seems reasonable.

d/
-- 

   Dave Crocker

---

From: "Jiankang YAO" <[email-masked]>
Date: Wed, 26 Jan 2011 17:23:29 +0800

after reading it and the following discussion,

my understanding is that 

RFC5335bis need the new metalanguage model;
 
RFC5336bis can still keep the current metalanguage model.

Is my understanding correct?

Jiankang Yao

---

From: John C Klensin <[email-masked]>
Date: Wed, 26 Jan 2011 09:00:13 -0500

> Is my understanding correct?

Speaking as an individual: having different metalanguage models
in the two documents would be a bad idea and certain to make
things harder for readers.  If we are going to change the model
or terminology in both documents, both (and 5337bis as
appropriate) need to change.

Speaking as co-chair: please see my note to the DT/author's list
yesterday (for WG members not on that list, it summarized a
plan, discussed at around the beginning of this consensus call
cycle, for being sure drafts are aligned with announced WG
consensus and coordinated wrt this sort of "changes to one
require changes to others" situation).

      john

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 26 Jan 2011 07:00:26 -0800

> Is my understanding correct?

The current proposal is for changes to RFC5335bis.

d/
-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 26 Jan 2011 10:29:55 -0800

> RFC5336bis can still keep the current metalanguage model.

My other response to you query was to clarify what has so far been presented.

This note is meant to cover the topic more broadly:

    1. Two documents that share normative content should not repeat that 
content.  One should cite the relevant portion(s) of the other.  This was 
discussed earlier.  Redundancy invites divergence and disparity.

    2. The proposal for changes to RFC5336bis are being put forward first 
because it specifies the email object.  It is therefore the core specification, 
IMO, and needs to be clarified first.

    3. Issues with the ABNF in rfc5336bis are the same as with rfc5335bis.  As 
the group settles on the revisions for the ABNF in rfc5335bis, then rfc5336bis 
should be modified to cite that ABNF.

    4. The two of us who reviewed rfc5336bis made various recommendations.  The 
handling of those is, I believe, separate from concerns about rfc5335bis.

d/

-- 

   Dave Crocker

---

From: "Jiankang YAO" <[email-masked]>
Date: Fri, 28 Jan 2011 13:51:58 +0800

----- Original Message ----- 
From: "Dave CROCKER" <[email-masked]>
To: "Jiankang YAO" <[email-masked]>
Cc: "ietf-eai" <[email-masked]>
Sent: Thursday, January 27, 2011 2:29 AM
Subject: RFC5336bis (was Re: [EAI] Proposal for Alternate ABNF)

....
> should be modified to cite that ABNF.

if I remember it correctly, you recommend that rfc5336bis should not cite the ABNF in RFC5322, and should cite rfc5321 only.

Since rfc5336bis is extension of rfc5321, and rfc5335bis is extension of rfc5322,

it is not very proper to mingle the definition of ABNF in rfc5335bis and rfc5336bis.

the ABNF in rfc5335bis  mainly focus on email envelop; the ABNF in rfc5336bis mainly focus on email headers.

the same name under rfc5335bis and rfc5336bis, rfc5321 and rfc5322 may have different definitions.

for example,

as the problems pointed out by Alexey,

"
definition of dot-atom, domain-literal and obs-domain from RFC 5322 
are going to be problematic in SMTP:

   domain-literal  =   [CFWS] "[" *([FWS] dtext) [FWS] "]" [CFWS]

   obs-domain      =   atom *("." atom)

   atom            =   [CFWS] 1*atext [CFWS]

   dot-atom-text   =   1*atext *("." 1*atext)

   dot-atom        =   [CFWS] dot-atom-text [CFWS]

because of CFWS.

"

so I think that following the ABNF definition mode or pattern in rfc5321 and rfc5322 is a better way to solve the ABNF issues in rfc5335bis and rfc5336bis. 

Jiankang Yao