Subject: [Ltru] Re: UTF-8

From: "Doug Ewell" <[email-masked]>
Date: Fri, 15 Sep 2006 21:39:40 -0700

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> This thread is very amusing, but, I think, not very useful.

That's why I marked my last post on it [OT].

> I think that using a widely-recognized, plain-text character encoding 
> (UTF-8: "the new ASCII") would be a Good Thing. But changing which 
> escape syntax we use, especially to use something, um, "unusual" like 
> UTF-1 or BOCU-1, gives us nothing over using US-ASCII with some escape 
> syntax (such as our existing NCR format).

More to the point, changing from record-jar with hex NCR's to *anything* 
else has to be not just better, but "better enough."  The pain and 
instability caused by the change must be considered, and the benefits 
must outweigh them.

> So:
>
> Is there any support for changing to UTF-8?

I would support it on two conditions:

1.  We must be able to use UTF-8 in the I-D (4645bis) that delivers the 
Registry contents to IANA.  (It's OK if that means the I-D can't be an 
RFC.)  Asking IANA to convert NCR's into UTF-8 themselves would be a 
disaster.  The new Registry will have almost 500 non-ASCII characters.

2.  We have to maintain compatibility with the existing format, which 
means we must continue to escape the ampersand as &#x26; if one should 
ever find its way into the Registry.

--
Doug Ewell
Fullerton, California, USA
http://users.adelphia.net/~dewell/
RFC 4645  *  UTN #14

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Sun, 17 Sep 2006 16:57:47 +0900

>More to the point, changing from record-jar with hex NCR's to *anything* else has to be not just better, but "better enough."  The pain and instability caused by the change must be considered, and the benefits must outweigh them.

Agreed. In my point of view, the pain of Bokm&#$!@*l outweights
a lot of things. The main benefit is that native people can
easily check that things are correct, which they won't do
if we show them just a number.

>1.  We must be able to use UTF-8 in the I-D (4645bis) that delivers the Registry contents to IANA.  (It's OK if that means the I-D can't be an RFC.)  Asking IANA to convert NCR's into UTF-8 themselves would be a disaster.  The new Registry will have almost 500 non-ASCII characters.

I agree with the fact that we must be able to deliver UTF-8 to IANA.
I don't think it necessarily needs to be an I-D. I easily can immagine
that you publish an I-D containing just a pointer to an URI and an MD5
for checking.

>2.  We have to maintain compatibility with the existing format, which means we must continue to escape the ampersand as &#x26; if one should ever find its way into the Registry.

As I have said in another mail, I'd agree with the continued need for
escaping the sequence "&#x", but I don't see any practical relevance
for it; we can simply disallow it. For just "&", it may cause a
parsing problem for older software the same way UTF-8 may cause a
parsing problem, so I don't see the need to keep escaping it.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 17 Sep 2006 22:09:26 -0700

Martin Duerst <duerst at it dot aoyama dot ac dot jp> wrote:

> I agree with the fact that we must be able to deliver UTF-8 to IANA. I 
> don't think it necessarily needs to be an I-D. I easily can immagine 

Can you check into this?  It would be nice to say up front what the I-D 
is for, rather than writing it as "This contains the initial contents of 
the Registry" and then having the RFC Editor remove the fun part.

>> means we must continue to escape the ampersand as &#x26; if one 
>> should ever find its way into the Registry.
>

The ABNF in 4646 says the ampersand is not a valid character and must be 
escaped, because it introduces the &#x sequences.  I'm feeling lazy. 
Can someone else verify this, and speak to the impact of allowing a 
character that we previously did not allow?

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 18 Sep 2006 08:03:05 +0200

Doug Ewell wrote:

> The ABNF in 4646 says the ampersand is not a valid character and must
> be escaped, because it introduces the &#x sequences.  I'm feeling lazy.
> Can someone else verify this,

Of course, it's in the <ASCCHAR> production, and in the prose.

> the impact of allowing a character that we previously did not allow?

I refuse to abuse my crystal-ball today.  Registry converters
to XML have to replace "<" by "&lt;", and while they're at it
">" to "&gt;" isn't wrong.  UTF-8-jar converters to XML would
use UTF-8 as encoding, and replace "&" by "&amp;" before the
"<" to "&lt;" and ">" to  "&gt;".

They could throw an error for "&#x", or they can try to get
it right:  "&#x" to ASCII SUB, other "&" to "&amp;", SUB back
to "&#x", after they've checked that it's a plausible NCR.
Or they could replace the NCR by UTF-8. 

The constant thing over all programming languages:  The code
has to be updated.  Not difficult, a minor technical change.

The "keep NCR as is" trick works only for XML as output format.

Frank

---

From: Peter Constable <[email-masked]>
Date: Mon, 18 Sep 2006 00:04:42 -0700

> From: Martin Duerst [mailto:[email-masked]]

> if we show them just a number.

There's lots of software that will interpret an NCR in an HTML, XML or HTML file and display it to the user as the actual character in the UCS for which it is a reference. I don't think there is any software that will do this for the file located at http://www.iana.org/assignments/language-subtag-registry. 

Peter Constable

---

From: Martin Duerst <[email-masked]>
Date: Tue, 19 Sep 2006 22:17:58 +0900

[chair hat on]

Peter, with the observation below, do you want to say
you are in favor of moving to UTF-8, or against, or
did you write that strictly as an observation only?

Regards,    Martin.

At 16:04 06/09/18, Peter Constable wrote:
>There's lots of software that will interpret an NCR in an HTML, XML or HTML 
>file and display it to the user as the actual character in the UCS for 
>which it is a reference. I don't think there is any software that will do 
>this for the file located at 
>http://www.iana.org/assignments/language-subtag-registry. 
>
>
>
>
>Peter Constable

---

From: Peter Constable <[email-masked]>
Date: Tue, 19 Sep 2006 08:29:01 -0700

There's a prior question: what will the content of registry records
contain? (We don't need an encoding that supports the entire UCS if we
don't intend to have records using characters from that entire
repertoire.) I don't recall if that's been discussed and resolved.

I do support using an encoding that directly supports whatever
characters we wish to allow in the registry without use of NCRs or other
such escape mechanisms; and if we wish to allow any UCS character in the
registry then I would support using UTF-8 as the encoding.

Peter

> -----Original Message-----
> Sent: Tuesday, September 19, 2006 6:18 AM
> To: Peter Constable; LTRU Working Group
> Subject: RE: [Ltru] Re: UTF-8
> 
> [chair hat on]
>

---

From: Addison Phillips <[email-masked]>
Date: Tue, 19 Sep 2006 08:45:47 -0700

Descriptions and comments both allow the full range of Unicode today. 
Doug has noted some number of items (perhaps a dozen) that use non-ASCII 
(mostly Latin-script) letters or symbols. There is no check for the 
range of characters permitted, nor (in our discussion in draft-registry 
days) should there be.

I think that limiting the repertoire to anything less than the full 
range of the UCS would prove problematic anyway. And while I don't see a 
pressing need to identify languages in their native form, some people 
might find it offensive if it were banned outright. Similarly, some 
might find comments clearer if they contained some native description. I 
don't know.

What I do know is that I think that the only limitation that could 
reasonably be made that would be non-arbitrary would be a limitation to 
ASCII. And I, personally, will not support any encodings except those 
two for encoding the registry: either US-ASCII or UTF-8.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.