Subject: [Ltru] How to represent non-ASCII in Registry I-D?

From: "Doug Ewell" <[email-masked]>
Date: Sun, 11 Mar 2007 14:44:02 -0700

If we decide to change the Registry format from hex NCRs to UTF-8, how 
should non-ASCII characters be represented in RFC 4645bis?  Definitive 
answers only, please; no speculation.

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Addison Phillips <[email-masked]>
Date: Sun, 11 Mar 2007 21:10:25 -0700

Definitive answer:

In the format specified in the "IANA Considerations" section of RFC 
4645bis. [That is: you get to say how the characters are escaped.]

Supporting speculation:

Use the \u#### or %## format (where # == hex digit). The former would be 
in UTF-16 code units. The latter would be in UTF-8 code units (aka 
"bytes" or "octets"). Either of these can be unescaped into UTF-8 quite 
easily. I would tend to suggest the \u format, since the widely 
available JDK tool 'native2ascii' knows it, as does perl (after 5.8).

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 12 Mar 2007 00:33:03 -0700

> 4645bis. [That is: you get to say how the characters are escaped.]

Yes, and thank you, but (a) what if anything does the WG prefer to see, 
and (b) what do we have the best chance of IANA converting correctly? 
(Possible answers below.)

> Use the \u#### or %## format (where # == hex digit). The former would 
> be in UTF-16 code units. The latter would be in UTF-8 code units (aka 
> "bytes" or "octets"). Either of these can be unescaped into UTF-8 
> quite easily. I would tend to suggest the \u format, since the widely 

I'll buy that.  It does make me wonder, of course, why we chose &#xE5; 
instead of \u00E5 if the latter is so much easier to parse.  (Notice the 
number of bytes in each representation is the same.)

If anyone objects to my using \u#### notation in 4645bis, with 
instructions to IANA to convert such sequences into real UTF-8, please 
speak up.  As soon as Addison submits a new 4646bis draft, mine will 
follow close behind.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 12 Mar 2007 09:55:53 +0100

On Sun, Mar 11, 2007 at 09:10:25PM -0700,
 Addison Phillips <[email-masked]> wrote 
 a message of 44 lines which said:

> "bytes" or "octets").

Strongly -1 for me

1) If we use an espace mechanism, it should be to code points, which
are a high-level concept, not to code units, which depends on the
encoding. Apart from URI percent-escape, no standard use escape
mechanisms to encode an encoding!

If we persist in the mistake of URI percent-escape, at least we should
rely on UTF-8, not on the UTF-16 horror with its surrogates.

2) If we use an escape mechanism, it should be with explicit
delimiters like \u(NNNN) because it is unambiguous and easy to parse.

But I suggest to defer this until draft-klensin-unicode-escapes-03
becomes approved (these questions are NOT LTRU-specific and are
currently discussed in the Application Area).

---

From: Peter Constable <[email-masked]>
Date: Mon, 12 Mar 2007 05:21:55 -0700

I prefer to see whatever IANA has the best chance of converting correctly. Have they been asked what that might be?

Peter

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 12 Mar 2007 06:40:01 -0700

Peter Constable <petercon at microsoft dot com> wrote:

> I prefer to see whatever IANA has the best chance of converting 
> correctly. Have they been asked what that might be?

That is a good point, and I suspect nobody has asked them yet.  I'll do 
so.

> mechanisms to encode an encoding!

I'll check with Amanda.

> rely on UTF-8, not on the UTF-16 horror with its surrogates.

UTF-16 surrogates are not an issue for us, since the Registry doesn't 
currently contain any characters outside the BMP.

However, the word "currently" raises new questions for me.  What format 
should be used when requesting new subtags, and what format should be 
used for sending updates to IANA?  (If we go with the proposal to send 
them a complete new Registry each time, the second question is already 
answered.)

> delimiters like \u(NNNN) because it is unambiguous and easy to parse.

\uNNNN is unambiguous: there are exactly four hex digits after the "u". 
To represent a character outside the BMP, one must either encode the 
surrogates as \uNNNN\uNNNN, or use \UNNNNNNNN with exactly eight hex 
digits.  Either one is unambiguous.

> currently discussed in the Application Area).

I hope that draft is at a late enough stage, and close enough to 
approval, that waiting for it does not delay us substantially.

---

From: John Cowan <[email-masked]>
Date: Mon, 12 Mar 2007 10:29:43 -0400

> approval, that waiting for it does not delay us substantially.

I can't speak to process concerns, but here's a summary of the draft's
recommendations (it contains no MUSTs except that each protocol MUST
define the escape syntax it uses).

1) Protocols SHOULD use UTF-8 rather than an escape convention, but if
UTF-8 is not practical, then:

2) Protocols SHOULD encode codepoints, not the UTF-8 or UTF-16 versions
of them, and SHOULD do so in hex, not decimal or octal;

3) The XML style we currently use, the C style (with "\unnnn" and
"\Unnnnnnnn"), and the Perl style (with "\u{nnnn...}") all have their
good and bad points, and they should always be considered.  The draft is
at pains to call the XML style ugly, though why it thinks so does not
appear; the C style lacks a closing delimiter and is readily confused
with the Java convention (which is called out with a SHOULD NOT) and
other conventions involving \u.

-- 
John Cowan  [email-masked]    http://ccil.org/~cowan
Big as a house, much bigger than a house, it looked to [Sam], a grey-clad
moving hill.  Fear and wonder, maybe, enlarged him in the hobbit's eyes,
but the Mumak of Harad was indeed a beast of vast bulk, and the like of him
does not walk now in Middle-earth; his kin that live still in latter days are
but memories of his girth and his majesty.  --"Of Herbs and Stewed Rabbit"

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 12 Mar 2007 16:18:49 +0100

Stephane Bortzmeyer wrote:
 
> I suggest to defer this until draft-klensin-unicode-escapes-03
> becomes approved

It's a one time thing for the 4645bis I-D, Doug can use one of
the formats in I-D.klensin-unicode-escapes-03 without approval,
an informative reference to an I-D is okay.

As it happens one of the formats in this I-D is the existing
4646 format (after fixing the 4646 ABNF), therefore he can use
this format.  If IANA needs a script to "unescape" this once
they could "hire" one of us... :-)

Really, it's no rocket science.  And maybe the whole mad plan
doesn't survive the IETF Last Call.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 12 Mar 2007 16:30:22 +0100

Doug Ewell wrote:

> (a) hat if anything does the WG prefer to see,

I prefer the existing 4646 format, because that's how I check the
registries in your I-Ds.
 
> (b) what do we have the best chance of IANA converting correctly?

They can do anything, a trivial script can convert 4646 format to
Unicode.  A Web browser might do, adding at the begin a dummy line

<title> 4646 format </title><pre>

...and a dummy line </pre> at the end, and "save as text file".
Somebody using IE7 and Office could work out the details.

> It does make me wonder, of course, why we chose &#xE5; instead of
> \u00E5 if the latter is so much easier to parse.

It isn't easier, it's also trivial (at least without surrogates).

> As soon as Addison submits a new 4646bis draft, mine will follow
> close behind.

I'd guess that there will be no new drafts in the next twelve days.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 12 Mar 2007 16:37:40 +0100

John Cowan wrote:

> The draft is at pains to call the XML style ugly, though why it
> thinks so does not appear

Editorial choice, I tried to talk John out of it, but he really
doesn't like XML style, while I don't like C style variations :-)

Perl style had the least opponents, but for a _one time_ 4645bis
I'd really prefer to stick to the known 4646 format working with
my gawk script.

Frank

---

From: Addison Phillips <[email-masked]>
Date: Mon, 12 Mar 2007 13:32:13 -0700

>> in UTF-16 code units. The latter would be in UTF-8 code units (aka 

You oppose either of these escaping mechanisms, I take it. I proposed 
these not for the purity or "correctness" of their approach. Rather, I 
picked them for the availability to IANA of simple-to-use, 
widely-available tools that understood these specific formats.

I note that:

1. The escaped material will be seen only by IANA for the purpose of 
creating the "initial" 4646bis registry. It will be removed from the 
final RFC.

2. There will be no supplemental characters in the registry.

> mechanisms to encode an encoding!

Actually, almost all escaping mechanisms deal with encodings (code 
units) and not character sets (code points). Unicode escaping mechanisms 
tend to be an exception to this rule, at least in *theory* (the \u#### 
formats exposes UTF-16's structure). Examples of other escaping 
mechanisms would be MIME's use of base64 and quoted-printable.

> rely on UTF-8, not on the UTF-16 horror with its surrogates.

I think I said that for URI-encoding, no?

UTF-16's use of surrogate pairs might not be ideal for an escaping 
mechanism. However, support for UTF-16 is widely available and 
supplemental characters are not a problem for *this given instance* of 
the problem. In particular, I proposed the given format because there 
are many tools for dealing with it "off-the-shelf".

Frank makes a good point about using browsers to read XML NCRs, but I 
note that his approach does require dinking around with the 
Internet-Draft. My approach can be applied directly to the text of the 
I-D and then a text or stream editor used to rip out the extra matter.

Having IANA write a custom parser or having Doug provide a script in the 
I-D is less desirable (at least to me) than having a well-known tool 
(native2ascii, perl, etc.) that already digests the given format.

> delimiters like \u(NNNN) because it is unambiguous and easy to parse.

In a perfect world, yes. But *existing* tools were my focus.

> currently discussed in the Application Area).

Why? We might be done before that draft-klensin is approved. And this 
issue will *disappear* forever once our document is published.

Addison

---

From: John Cowan <[email-masked]>
Date: Mon, 12 Mar 2007 17:19:37 -0400

> mechanisms would be MIME's use of base64 and quoted-printable.

Well, Java-style \u is about code units, and so is URI escaping; both
for reasons of backward compatibility.  But C-style, Perl-style, and
XML-style escaping are about code points.

Base64 and QP aren't in this fight; they are intended for general
octet sequences, not just text.

> (native2ascii, perl, etc.) that already digests the given format.

I note that native2ascii doesn't work reliably on random text files;
it knows at least some of Java's syntax, and can do bad things to
backslashes, apostrophes, and quotation marks.  This isn't particularly
relevant for the registry, more of a warning in general.

-- 
John Cowan                                   [email-masked]
        "You need a change: try Canada"  "You need a change: try China"
                --fortune cookies opened by a couple that I know

---

From: Martin Duerst <[email-masked]>
Date: Tue, 13 Mar 2007 14:56:25 +0900

My technical opinion on this issue: I prefer the XML form
that we currently have. For one, it means that we don't have
to change the actual data in case our proposal to go with
UTF-8 for the actual registry gets shot down.

At 05:32 07/03/13, Addison Phillips wrote:

>UTF-16's use of surrogate pairs might not be ideal for an escaping mechanism. However, support for UTF-16 is widely available and supplemental characters are not a problem for *this given instance* of the problem. In particular, I proposed the given format because there are many tools for dealing with it "off-the-shelf".

You mentioned one, you say many. In principle, one is enough, but
what would the others be?

>Frank makes a good point about using browsers to read XML NCRs, but I note that his approach does require dinking around with the Internet-Draft. My approach can be applied directly to the text of the I-D and then a text or stream editor used to rip out the extra matter.

I'd strongly suggest they rip out the extra matter first. Probably
IANA or the RFC editor have tools to do that, but I'd not want
to bet on how UTF-8-safe they are.

>Having IANA write a custom parser or having Doug provide a script in the I-D is less desirable (at least to me) than having a well-known tool (native2ascii, perl, etc.) that already digests the given format.

After reading
http://java.sun.com/j2se/1.5.0/docs/tooldocs/solaris/native2ascii.html,
my understanding is that this would have to be used like this:
   prompt> native2ascii -reverse -encoding UTF-8 inputfile outputfile
or some such. Would these instructions go into Doug's draft,
or what?

>Why? We might be done before that draft-klensin is approved. And this issue will *disappear* forever once our document is published.

[chair hat on] I strongly suggest that we don't create additional
dependencies for our drafts. So I would suggests that we don't
even cite draft-klensin, but just refer to the original spec
of whatever we are using.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]     

hane Bortzmeyer wrote:

>Why? We might be done before that draft-klensin is approved. And this issue will *disappear* forever once our document is published.

[chair hat on] I strongly suggest that we don't create additional
dependencies for our drafts. So I would suggests that we don't
even cite draft-klensin, but just refer to the original spec
of whatever we are using.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 13 Mar 2007 22:42:50 -0700

> UTF-8 for the actual registry gets shot down.

OK, given that end users are never, ever going to see whatever escaping 
convention we use -- only we, the IETF, the RFC Editor, and IANA will 
see it -- does anyone have any strong objections to keeping the same hex 
NCR convention we have now, but adding instructions to IANA to convert 
the hex NCRs to UTF-8?

BTW, I would be opposed to telling IANA what tools to use to perform 
this task; what is important is that it be performed right.