Subject: draft-klensin-unicode-escapes-02.txt

From: John C Klensin <[email-masked]>
Date: Mon, 05 Feb 2007 10:21:53 -0500

Hi.

I've just placed another version of the "unicode escapes"
document into the posting queue.  It should be announced to this
list when posted.

I believe I have incorporated all of the changes suggested so
far that haven't gotten pushback, including adjusting the ABNF
that has been controversial (I even changed 4*4 -> 4; others
seem to be more concerned about that syntax preference than I
am).

It occurs to me now --after I sent the document off-- that the
\u / \U form is the only one for which there is ABNF.  This is
the legacy from its previous featured role.  Recommendations
from others as to whether I should just drop that ABNF, leave
things as they are, or add ABNF to the other forms would be
appreciated.  Anyone who prefers the latter should please send
the ABNF they would like to see.

The thing I have _not_ done is to try to expand this document
into making general suggestions or requirements on the use of
Unicode.  It assumes that the strings that one might want to
escape are valid and reasonable and that the definition of
"valid and reasonable" is the province of other documents.

More comments welcome, but I hope we are converging.

     john

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 9 Feb 2007 14:38:42 +0000

> document into the posting queue.  It should be announced to this

One typo: in section 3, you have "NNMN".

The abstract says "makes a proposal for general use". But you don't any
more, do you? You simply propose rules for consideration.

> "valid and reasonable" is the province of other documents.

Okay. That paragraph might be worth adding to the text.

> More comments welcome, but I hope we are converging.

There's something I found in my email from last year when this topic was
first being discussed.

Consider an explicitly delimited form like \u(NNNN). Is it worth saying
that "customers" MAY allow the use of multiple code-points within the
delimiters. In other words:

    \u(1234,5678,109ABC)

is an acceptable abbreviation for:

    \u(1234)\u(5678)\u(109ABC)

The ABNF for this would be:

    unicode-escape = escape-prefix "(" escape-body ")"
    escape-prefix = "\u"   ; I'm ignore case-sensitivity issues
    escape-body = required-body / permitted-body
    required-body = code-point
    permitted-body = code-point 1*( "," code-point )

See previous email for code-point, though perhaps an implementation MAY
allow leading zeros to be omitted.

[The other point from that old email, though if you say it's too esoteric
to mention I won't complain, is that this also allows other notations to be
mixed in if an implementer wants. For example, explicit character names, so
that:
    \u('e acute')
would be another way to write
    \u(00E9)

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 19 Feb 2007 22:34:44 +0100

John C Klensin wrote:

> Anyone who prefers the latter should please send the ABNF they would
> like to see.

Here's what I'd do with the existing ABNF:

   EmbeddedUnicodeChar =  BMP-form / Full-form
   BMP-form  =  %x5C.75 4HEXDIG   ; starting with lower case "\u"
   Full-form =  %x5C.55 8HEXDIG   ; starting with upper case "\U"

IOW no more <Hex-quad> because you didn't use it elsewhere, and
replacing <HexDigit> by <HEXDIG>, because the latter is already
defined in RFC 4234.

For the XML version I propose to adopt and fix the RFC 4646 ABNF:

   UNICHAR    = %x26.23.78 2*6HEXDIG ";"    ; starts with "&#x"

With a remark in the prose, that a literal "&" can be expressed by
&#x26; when using this style.  Otherwise folks could be tempted to
use &amp; - but we don't want them to try that.

You don't need ABNF in 5.4, it's in essence the same as in 5.1.

For 5.3 (perl) I don't know the correct syntax, if it's like 5.2:

  UNICODEPOINT = %x5C.78 "{" 2*6HEXDIG "}"  ; starts with "\x"

If the x is case insensitive it's simply:

  UNICODEPOINT = "\x{" 2*6HEXDIG "}"

The name "UNICODEPOINT" is horrible, please find something better, I
tried to use a new name.

Please add the [CharMod] reference (informative) with a note about
its C042..C048 conformance criteria in ch. 4.6 (character escaping).

Frank

---

From: Philip Guenther <[email-masked]>
Date: Mon, 19 Feb 2007 17:36:30 -0700

On Mon, 19 Feb 2007, Frank Ellermann wrote:
...
>  UNICODEPOINT = %x5C.78 "{" 2*6HEXDIG "}"  ; starts with "\x"

That looks good to me.  Perl accepts values that don't match that syntax, 
but we don't want to get into trying to define perl's corner cases (or, 
indeed, those of *any* language not originally defined by RFC).  This 
should be a 'generate' syntax, not an 'accept' syntax, IMHO.

> If the x is case insensitive it's simply:
...

The 'x' is case sensitive in perl.  \X has no meaning in plain strings, 
but in regexps it is a built in pattern of sorts.  To quote the 
perlunicode(1) manpage:

        o   The special pattern "\X" matches any extended Unicode
            sequence--"a combining character sequence" in Stan-
            dardese--where the first character is a base character
            and subsequent characters are mark characters that
            apply to the base character.  <...>

Philip Guenther

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 21 Feb 2007 08:44:17 +0000

Frank Ellermann said:
> IOW no more <Hex-quad> because you didn't use it elsewhere,

I would prefer this as well. The <Hex-quad> comes from the C Standard,
where we didn't have the <number><token> notation and so would have had to
write the equivalent of:

    %x5C.55 HEXDIG HEXDIG HEXDIG HEXDIG HEXDIG HEXDIG HEXDIG HEXDIG

---

From: John C Klensin <[email-masked]>
Date: Fri, 23 Feb 2007 12:33:56 -0500

--On Monday, 19 February, 2007 22:34 +0100 Frank Ellermann
<[email-masked]> wrote:

> John C Klensin wrote:
> 
>> Anyone who prefers the latter should please send the ABNF
>> they would like to see.
> 
>    BMP-form  =  %x5C.75 4HEXDIG   ; starting with lower case
> "\u"    Full-form =  %x5C.55 8HEXDIG   ; starting with upper
> case "\U"
> 

Done, including removal of <HexDigit>

> Please add the [CharMod] reference (informative) with a note
> about its C042..C048 conformance criteria in ch. 4.6
> (character escaping).

A reference to [CharMod] has been included, but as part of a
comment (per discussion with Charles on 9 February) about what
this document does not try to address.

I think most (and hope all) of the other comments since -02 was
posted have been picked up, with one major exception.  I believe
that complexity is the enemy of this type of work.  So, while it
is clearly possible to figure out how to write, e.g.,
   \u(NNNN, NNNNN, NNNN)
rather than
   \u(NNNN)\u(NNNNN)\u(NNNN)
or to incorporate named characters into escaped strings, I don't
think doing that is wise.   YMMD, of course, and, if you, I'm
sure I will hear about it.

-03 is now in the posting queue.  I just realized that I didn't
update Section 8 (the change log) for either this version or
-02.  The major changes are the elimination of a specific
recommendation in -02 and cleaning up of a lot of text and
fixing the ABNF in -03.

      john

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 24 Feb 2007 16:56:03 +0100

>>    Full-form =  %x5C.55 8HEXDIG   ; starting with upper case "\U"
[...]
> Done, including removal of <HexDigit>

You kept the multiline comment after <BMP-form>, and Bill's parser
doesn't like this:

BMP-form =  %x5C.75 4HEXDIG ; starting with lower case "\u"
   ; In both this case and the one above, note that the encodings are
   considered to be abstractions for the relevant characters, not
   designations of specific octets.

With more semicolons it's okay:

BMP-form =  %x5C.75 4HEXDIG ; starting with lower case "\u"
   ; In both this case and the one above, note that the encodings are
   ; considered to be abstractions for the relevant characters, not
   ; designations of specific octets.

But actually I think this ABNF comment is unnecessary - or if not it
could be moved into the prose.  Then the contrast \u vs. \U would be
also more directly visible:

Lillyguilding => ready, maybe add "intended status: BCP" and set the
"PubReq" flag while authors are still allowed to manage this flag. ;-)

Frank