Subject: Re: [EAI] Proposal for Alternate ABNF (was: Consensus Issue #6: current metalanguage model acceptable?)

From: Shawn Steele <[email-masked]>
Date: Fri, 21 Jan 2011 23:34:57 +0000

IMO this is kind of the same conceptually as what the WG is trying to achieve.  I need whatever gets us stable fastest :)

Re:
    {{ how to add IDN to this? }}
    domain  =   dot-atom / domain-literal / obs-domain

That should be:
    domain  =/  UTF8-non-ascii encoded RFC5890 U-label

There should be a note that a UTF-8 domain label MUST only contain valid RFC5890 u-labels.

-Shawn

-------- Original Message --------
Subject: [EAI] Proposal for Alternate ABNF (was: Consensus Issue #6: current metalanguage model acceptable?)
Date: Thu, 20 Jan 2011 16:04:36 -0800
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: ietf-eai <[email-masked]>

Folks,

The answer to consensus call question 6 should be 'no'.

Ned Freed, Pete Resnick and I have been discussing the working group's basic
model for handling UTF-8 within a legacy, 7-bit world. The current approach is
certainly in line with the long-standing IETF pressure (demand) to have changes
of an existing system be designed in a way that protects and interworks with the
installed base, such as by not damaging a legacy system that receives an
enhanced form of data. And indeed, the working group's SMTP extension is an
example of keeping the new object away from legacy-only systems.

However, the model used in the case of RFC 5335 -- using new syntax elements
instead of extending old ones -- has produced quite a bit of complexity. In this
case, we believe the complexity is not serving the working group's goals all
that well.

That leads us to make a somewhat surprising suggestion to simplify
things considerably. The premise behind the suggestion is that the
world really does have extensive infrastructure for UTF-8 already and
that we can rely on it.

         So, the core suggestion is to have the revision focus
         only on the existing, basic 5322 ABNF constructs and
         simply augment them to support UTF-8.

         This would create what essentially would be an independent,
         UTF-8 email world.

         Interworking with the legacy email world still needs to be
         specified, but it should be done separately.

With the caveat that IDN handling in the spec still needs to be done, this
produces a candidate to replace for Section 4.3, 4.4 and 4.5 from the existing
draft:

     [[ --------------------------------------

This section specifies UTF-8 enhancements for the header of an
Internet Mail message, as defined in [RFC5322].

ABNF used in this section is taken from that specification and the
ABNF specification.

This specification retains the [RFC5322] rules for defining header
field names. The bodies of header fields are allowed to contain UTF-8
characters, but the header field names themselves must contain only
ASCII characters.

The following rules extend the corresponding rules in [RFC5322] and
[RFC5234] in order to allow additional Unicode characters.

    VCHAR   =/  UTF8-non-ascii

    ctext   =/  UTF8-non-ascii

    atext   =/  UTF8-non-ascii

    qtext   =/  UTF8-non-ascii

    {{ how to add IDN to this? }}
    domain  =   dot-atom / domain-literal / obs-domain

This means that all the [RFC5322] constructs that build upon these
will permit UTF-8 characters, including comments and quoted strings.

    <field-name>

           [RFC5322] has the rule <field-name> which specifies
permissible names for user-defined header fields. The current
specification defines no changes to that rule.

    <msg-id>

           This ABNF enables Message-ID strings to be full UTF-8.
However the specification directs that Message-ID strings SHOULD be
restricted to ASCII.

      -------------------------------------- ]]

The SMTP extension is essential to this, of course. In addition it's worth
repeating that specification of UTF8-to-ASCII gatewaying is still needed, but we
suggest it be moved to a separate specification.

Since these are very basic ABNF rules, changes to them have an
extensive effect that is not immediately obvious.  So to aid the
reader, it's worth adding an appendix that lists the other ABNF of
RFC5322 that would be affected (ignoring the obs- rules)...

      [[ --------------------------------------

A. Changes to support UTF-8

This section provides a basic audit of the places in a message that
now can permit UTF-8 rather than being restricted to ASCII, based on
the changes to underlying ABNF. The audit ignores rules for "obsolete"
constructs in RFC 5322.

VCHAR:
     quoted-pair, unstructured
     > ccontent, qcontent
     > comment, quoted-string
     > word, local-part
     > phrase
     > display-name, keywords

     > name-addr
     > mailbox, group
     > address

     > address-list, mailbox-list

     > received-token, received

     > from, sender
     > reply-to, to, cc, bcc, resent-*

     > subject, comments, optional-field

ctext:
     ccontent > comment > comments

atext:
     atom, dot-atom-text

     > msg-id

     > local-part, domain
     > addr-spec
     > mailbox

qtext:
     qcontent quoted-string

      -------------------------------------- ]]

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net