Subject: RE: [Ltru] Re: UTF-8

From: "McDonald, Ira" <[email-masked]>
Date: Fri, 15 Sep 2006 11:42:07 -0700

Hi,

I support changing to unescaped UTF-8 for the registry
(consistent with the IESG policy in BCP 18 / RFC 2277).

But I also think Doug has it right that zipped content
as MIME attachments would be better, because the various
deployed email systems make raw UTF-8 (or any other UTF)
totally unreliable.

And if we move to XML with IANA concurrence, we'll get
UTF-8 anyway.

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked]

> -----Original Message-----
> From: Addison Phillips [mailto:[email-masked]]
> Sent: Friday, September 15, 2006 11:55 AM
> To: Frank Ellermann
> Cc: [email-masked]
> Subject: Re: [Ltru] Re: UTF-8
> 
> 
> This thread is very amusing, but, I think, not very useful. I 
> think that 
> using a widely-recognized, plain-text character encoding (UTF-8: "the 
> new ASCII") would be a Good Thing. But changing which escape 
> syntax we 
> use, especially to use something, um, "unusual" like UTF-1 or BOCU-1, 
> gives us nothing over using US-ASCII with some escape syntax (such as 
> our existing NCR format).
> 
> So:
> 
> Is there any support for changing to UTF-8?
> 
> Addison
> 
> Frank Ellermann wrote:
> > Doug Ewell wrote:
> > 
> >> I'm not sure why you chose 0x86 as your sequence introducer
> > 
> > Six is the number of trailing octets: 91909F9F9F9F (for John's
> > example u+10FFFF).
> > 
> >> you could make each sequence 1 byte shorter by marking the
> >> lead or trail byte specially
> > 
> > Yes, but then only 2 octets (80+81) would never occur (instead
> > of 11), and lost 9x bytes won't cause an error.  UTF-8 has now
> > 13 "impossible" octets, and similar features.
> > 
> 
> -- 
> Addison Phillips
> Globalization Architect -- Yahoo! Inc.
> 
> Internationalization is an architecture.
> It is not a feature.
> 
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ltru
> 

-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.405 / Virus Database: 268.12.4/449 - Release Date: 9/15/2006

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 16 Sep 2006 00:38:45 +0200

> I support changing to unescaped UTF-8 for the registry

I'm opposed.

> (consistent with the IESG policy in BCP 18 / RFC 2277).

That's about new protocols, not text/plain files like
RFCs or the LTRU registry.

> if we move to XML with IANA concurrence, we'll get UTF-8
> anyway.

Several issues here:  Where does the "IANA will convert
everything to XML" rumour come from?  I saw no discussion
about this on the XML-dir list.  Numerous RFCs define
registries with text/plain registration templates, if
those templates are anything it's a kind of "record-jar".

IANA can't simply reformat all their registries when they
feel like it.

XML is perfectly able to support any charset.  UTF-16 and
UTF-8 are only the minimally supported encodings.  With
the required UTF-8 support implementations not supporting
US-ASCII would be odd.  For something like xml2rfc stupid.

Frank

---

From: Martin Duerst <[email-masked]>
Date: Sat, 16 Sep 2006 12:03:15 +0900

>> I support changing to unescaped UTF-8 for the registry

It would be good to know why.

>RFCs or the LTRU registry.

Yes indeed. This means that Ira's argument is a bit weak,
but does not give an argument for not using UTF-8.

>IANA can't simply reformat all their registries when they

As a co-chair, I feel that this is rather irrelevant:
Our charter's timeline is short, and any attempt by
IANA to convert to 'all-XML' will be a long-time effort,
if it is done. If somebody wants to argue for XML, they
should do so based on it's merits.

As a technical contributor, I think that UTF-8 is the right
way to go, but otherwise, we should keep the format as is.
Looking at something like Bokm&#xE5;l is just plain annoying.
I'm sure the logistic problems with IANA can be sorted out
(e.g. using attachments).

As for using compression, this can be done mostly transparently.
browsers tell servers what encodings they support, e.g. like so:
Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
The server can be configured to send the appropriatly encoded
version of the file.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 16 Sep 2006 12:34:47 +0200

Martin Duerst wrote:

  [UTF-8 registry]
> It would be good to know why.

ftp://ftp.iana.org/assignments/language-subtag-registry

I've no clue what FTP clients would do with something like
that.  John's "net-UTF-8" draft isn't ready, and I had some
troubles to understand his last published version.

Another issue are fonts, for RFC 4646 there were proposals
to restrict the registry to Latn.  Fortunately that was
dropped, because the problem doen't exist with US-ASCII
plus NCRs.

> something like Bokm&#xE5;l is just plain annoying.

At least we know what this is.  BokmÃ¥l is less clear -
raw UTF-8 displayed as Latin-1, http://purl.net/net/ucode/E5

> I'm sure the logistic problems with IANA can be sorted
> out (e.g. using attachments).

Yes.  I doubt that there is any logistic problem for mail
submissions.  Only for the bulk update, the fattest I-D in
the history of the Internet using B64, they'd shoot us.

We could try QP, and silently send them Doug's original
data (as mail) in addition to the I-D.  But as you said,
that can be sorted out, it's not the reason why I prefer
to stick to the format as is.

We've no clear idea which tools will exist for the 4646
format in six months, how they do their updates (http or
ftp), and what local conversions they use.  Doug already
proposed that an UTF-8 version might have to keep the
&amp; convention, that could be confusing.  And we'd have
to decide if a signature is required, another change of
the record-jar ABNF.

> compression, this can be done mostly transparently.

ACK, I forgot this, my favourite browser uses HTTP/1.0 ;-)

An explicit gzip variant could still make sense for ftp
and HTTP/1.0 (?)  We can add some "IANA might consider to
offer" blurb in the IANA considerations, but IMO we can't
tell them how to run their servers.

Frank

---

From: John Cowan <[email-masked]>
Date: Sat, 16 Sep 2006 12:07:23 -0400

> (e.g. using attachments).

I am more skeptical about the ability of everyone involved
to keep everything encoded correctly, and repeat into the
indefinite future; consequently, I prefer the current escaping
scheme.  After all, UTF-8 was as useful *and* as problematic
when we adopted the scheme a year or two ago as it is today:
nothing has changed, so we should change nothing.

-- 
John Cowan    http://ccil.org/~cowan  [email-masked]
'Tis the Linux rebellion / Let coders take their place,
The Linux-nationale / Shall Microsoft outpace,
We can write better programs / Our CPUs won't stall,
So raise the penguin banner of / The Linux-nationale.  --Greg Baker

---

From: Addison Phillips <[email-masked]>
Date: Sat, 16 Sep 2006 10:58:50 -0700

> (e.g. using attachments).

Or sending the form in escaped ASCII and telling IANA to unescape it.

> The server can be configured to send the appropriatly encoded

I don't see any reason for us to dabble in transfer encodings!

Addison

---

From: Martin Duerst <[email-masked]>
Date: Sun, 17 Sep 2006 15:59:12 +0900

At 19:34 06/09/16, Frank Ellermann wrote:
>that.

Just save the file. If it's transferred as text, things
may go wrong for an EBCDIC system. If transferred as
binary, it will still be UTF-8.

>John's "net-UTF-8" draft isn't ready, and I had some

That isn't very relevant, except that it suggests
normalization to NFC, which I assume we would do anyway,
and which is already an issue in the case of NCRs.

>to restrict the registry to Latn.

Bad idea. Any average browser installation has way wider
coverage.

>Fortunately that was
>At least we know what this is.  Bokmテ・l is less clear -

Well, and then interpreted as Shift-JIS in the case of my
mailer :-(. Our spec as well as the HTTP headers will say
that it's UTF-8. That should be enough. People who are
okay to see garbage will be as satisfied with Bokm&#xE5;
as they are with Bokmテ・l, or any other weird rendering,
but people who like to see the real thing will be best
served by UTF-8.

>the history of the Internet using B64, they'd shoot us.

We could define a process experiment. As it is only for
an I-D, not for an RFC, that should work. I have seen
non-ASCII stuff in I-Ds, but checks may be a bit more
strict nowadays.

>data (as mail) in addition to the I-D.

If 'them' is IANA, that might work. If 'them' is the
Internet-Drafts Editor, that'd not make sense.

>ftp), and what local conversions they use.  Doug alrcret) with SMTP id
	k8H72S4D001474
	for <[email-masked]>; Sun, 17 Sep 2006 16:02:29 +0900 (JST)
Date: Sun, 17 Sep 2006 15:59:12 +0900
To: Frank Ellermann <[email-masked]>, [email-masked]
From: Martin Duerst <[email-masked]>
Subject: Re: [Ltru] Re: UTF-8
Cc: 

At 19:34 06/09/16, Frank Ellermann wrote:
>that.

Just save the file. If it's transferred as text, things
may go wrong for an EBCDIC system. If transferred as
binary, it will still be UTF-8.

>John's "net-UTF-8" draft isn't ready, and I had some

That isn't very relevant, except that it suggests
normalization to NFC, which I assume we would do anyway,
and which is already an issue in the case of NCRs.

>to restrict the registry to Latn.

Bad idea. Any average browser installation has way wider
coverage.

>Fortunately that was
>At least we know what this is.  Bokmテ・l is less clear -

Well, and then interpreted as Shift-JIS in the case of my
mailer :-(. Our spec as well as the HTTP headers will say
that it's UTF-8. That should be enough. People who are
okay to see garbage will be as satisfied with Bokm&#xE5;
as they are with Bokmテ・l, or any other weird rendering,
but people who like to see the real thing will be best
served by UTF-8.

>the history of the Internet using B64, they'd shoot us.

We could define a process experiment. As it is only for
an I-D, not for an RFC, that should work. I have seen
non-ASCII stuff in I-Ds, but checks may be a bit more
strict nowadays.

>data (as mail) in addition to the I-D.

If 'them' is IANA, that might work. If 'them' is the
Internet-Drafts Editor, that'd not make sense.

>&amp; convention, that could be confusing.

Yes. I'd agree that '&#x' sequences might need escaping,
but I don't expect any of these. Simple & would be okay
unescaped, it would trigger an error in strictly implemented
RFC 4646 software, the same way as UTF-8, and would not
cause any interpretation problems.

>And we'd have

Yes. No signature, please.

>and HTTP/1.0 (?)

Yes.

>We can add some "IANA might consider to

Agreed.

Regards,    Martin.

eady
>&amp; convention, that could be confusing.

Yes. I'd agree that '&#x' sequences might need escaping,
but I don't expect any of these. Simple & would be okay
unescaped, it would trigger an error in strictly implemented
RFC 4646 software, the same way as UTF-8, and would not
cause any interpretation problems.

>And we'd have

Yes. No signature, please.

>and HTTP/1.0 (?)

Yes.

>We can add some "IANA might consider to

Agreed.

Regards,    Martin.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 17 Sep 2006 12:06:51 +0200

Martin Duerst wrote:

>> I've no clue what FTP clients would do with something 
>> like that.

> binary, it will still be UTF-8.

The old FTP server on my box needed an option "-cp none"
for "raw mode", otherwise it would try to "get it right"
with weird results (cp 858 to cp 1252 or vice versa, my
number of UTF-8 plain text files is still *_zero_*)

> and which is already an issue in the case of NCRs.

Let's please just use whatever the source standards use,
garbage in garbage out (drawing the line at "invalid").

In an unrelated (SASLprep) attempt to check how bad this
is some weeks ago my conclusion was "bad enough" after I
found 418 combining characters.

The last thing we need in a 4646bis is a fixed Unicode
version (or convoluted prose why that's unnecessary).

>> for RFC 4646 there were proposals to restrict the 
>> registry to Latn.
> Bad idea. Any average browser installation has way 
> wider coverage.

Or way less, almost certainly not "all Latn", the next
least common denominator after Latin-1 could be MES-1.
Fighting with obscure glyph lists is another issue to
be solved by the source standards, not 4646bis.

 [UTF-8 Bokm&#xE5;l mangled to Latin-1]
> Well, and then interpreted as Shift-JIS in the case of
> my mailer :-(.

Folks will discuss registry entries on the review list,
with any MUA they have.  With NCRs as abstract objects
they'd be at least sure _what_ they discuss.

>> for the bulk update, the fattest I-D in the history of
>> the Internet using B64, they'd shoot us.
> an I-D, not for an RFC, that should work.

Yes, and as long as the AD is okay with it we might get
away without 3933 for this detail - after a bounce from
the secretary ("too long" or "non-ASCII") until they
believe it.

> I have seen non-ASCII stuff in I-Ds, but checks may be
> a bit more strict nowadays.

I'm not sure that it was always UTF-8, in one case where
it was windows-1252 the I-D wasn't published.  If they're
ready to accept any non-ASCII it ought to include UTF-8,
otherwise they've to fix it anyway sooner or later.

> Internet-Drafts Editor, that'd not make sense.

IANA.  I'd guess that the "I-D-editor" is in essence a
mailbot - with a human operator trying to fix some wild
and wonderful submissions where that mailbot isn't sure.
  
QP UTF-8 would be only a trick to get the draft published.
IANA could take the decoded attached file - or roll their
own QP I-D to registry decoder, it's no rocket science.

>> keep the &amp; convention, that could be confusing.

Assuming that we go that way (I still don't like it) we
could "forbid" &#x strings for backwards compatibility.
With "forbid" s/&#x//g.  Or "forbid" s/&#x/u+/g, because
it probably is precisely the same problem in the sources.

To justify such manipulations in 4646bis the WG should 
adopt the record-jar I-D as work item (informational RFC).

I could contribute 212 lines LTRU record-jar to XML as an
example (in the long version allowing to check all subtag
references with the W3C validator).

>> we'd have to decide if a signature is required, another
>> change of the record-jar ABNF.

> Yes. No signature, please.

I could in theory note UTF-8 as an "extended attribute"
of the file.  At some point I've to start that anyway, my
local "if it's not ASCII it's cp 858" rule does not help 
for published files, nobody knows what cp 858 is.

With or without signature, UTF-8 will fail miserably in
some constellations.  With a signature you could say that
it's not the fault of the registry.  Without signature, I
don't see where a misinterpretation as UTF-16 fails, some
"line too long" error ?

Maybe a dummy encoding="UTF-8" XML wrapper could help, one
element with a fixed xml:space="preserve" attribute (?)

Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 17 Sep 2006 18:20:41 +0200

On Sat, Sep 16, 2006 at 12:38:45AM +0200,
 Frank Ellermann <[email-masked]> wrote 
 a message of 36 lines which said:

> > if we move to XML with IANA concurrence, we'll get UTF-8 anyway.
> 
> Several issues here: Where does the "IANA will convert everything to
> XML" rumour come from?

It was announced by David Conrad, director of IANA, at the APRICOT
conference in Perth in january 2006 and confirmed more recently by a
private email from Kim Davies, IANA.

You can email them if you want to check:

David Conrad <[email-masked]>
Kim Davies <[email-masked]>

> IANA can't simply reformat all their registries when they feel like
> it.

I disagree: most registries have no format specification at all, the
RFC is silent about it. I believe that the LTR is an exception, not
the rule, since its authoritative RFC mandates a registry format.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 17 Sep 2006 18:20:41 +0200

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sun, 17 Sep 2006 18:24:51 +0200

On Sat, Sep 16, 2006 at 12:34:47PM +0200,
 Frank Ellermann <[email-masked]> wrote 
 a message of 62 lines which said:

> I've no clue what FTP clients would do with something like that.
> John's "net-UTF-8" draft isn't ready, and I had some troubles to
> understand his last published version.

John Klensin's draft-klensin-net-utf8-02.txt addresses a very
important point: how to represent the end-of-line? Try yourself now:
the registry transferred from the IANA site with FTP or with HTTP does
not have the same end-of-line (and that's what crashed your awk file,
in the bug report I sent you privately).

I did not read the record-jar draft yet. I hope it addresses the
issue. Otherwise, this is one more reason to move to XML :-)
 
> We've no clear idea which tools will exist for the 4646 format in
> six months, how they do their updates (http or ftp), and what local
> conversions they use.  Doug already proposed that an UTF-8 version
> might have to keep the &amp; convention, that could be confusing.
> And we'd have to decide if a signature is required, another change
> of the record-jar ABNF.

This is the sort of things that happen when you reinvent the wheel,
and create a new format for a registry, instead of trusting existing
formats...

---

From: Addison Phillips <[email-masked]>
Date: Sun, 17 Sep 2006 10:18:53 -0700

By the way: nothing says we have to put the actual updated registry into 
the I-D. Yes, we did it that way last time. But that is NOT the only way 
to transmit data to them. And I purposely selected the text in 
4646bis-00 to *not* specifically state what form the update would take.

I would suggest that it would be just as easy and/or better to specify 
that the LSTR (via his technical helper, aka Doug) create and verify the 
updated registry before forwarding it.

In fact, I've been thinking about text that Doug could use in 4645bis to 
manage the interregnum process and I'll send that to the list shortly.

Addison

---

From: John Cowan <[email-masked]>
Date: Sun, 17 Sep 2006 14:38:21 -0400

Martin Duerst scripsit:

> as they are with Bokm?$B%F!&l, or any other weird rendering,

Bokm&#xE5;l is at least reconstructible without knowing anything
except the SGML character reference conventions and the
codepoints of Unicode characters; Bokm?$B%F!&l (which is the
way it got to me, sans ESC characters) is nothing but rubbish.

This is an *excellent* example of why we need explicit escaping
(for which SGML is as good a convention as any) rather than
encoding, given the present state of email.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
In the sciences, we are now uniquely privileged to sit side by side
with the giants on whose shoulders we stand.
        --Gerald Holton

---

From: Addison Phillips <[email-masked]>
Date: Sun, 17 Sep 2006 12:32:33 -0700

> way it got to me, sans ESC characters) is nothing but rubbish.

Well, that's mozibake: it's ISO 2022-JP encoding of the Shift-JIS 
characters made by mis-interpreting UTF-8 bytes. Lovely.

> encoding, given the present state of email.

But that consideration doesn't apply to the registry file. It only 
applies to the discussion of a registry entry on ietf-languages. 
Admittedly MUAs vary greatly in their support for encodings, but we can 
certainly specify directions for how to send the registration request to 
the list so that everyone can tell what code points are intended and 
other instructions for how to encode those characters in the registry.

Suggestions to limit the character repertoire strike me as counter 
productive too. Why would we do that? I could see, for example, that we 
might end up with a few Chinese descriptions or comments in the registry 
to disambiguate Chinese variations. Why artificially restrict it 'ab 
initio'?

I find this fascination with ASCII slightly quaint.

Addison

---

From: John Cowan <[email-masked]>
Date: Sun, 17 Sep 2006 17:05:13 -0400

Addison Phillips scripsit:

> >Bokm?$B%F!&l (which is the

Not even.  My mailer suppresses ESC characters altogether,
so the result is pure printable ASCII again, ergo unreconstructible.

> other instructions for how to encode those characters in the registry.

Directions which work uniformly across all native encodings, operating
environments, and mailers?  I venture to doubt it.

> productive too. 

+1

---

From: Addison Phillips <[email-masked]>
Date: Sun, 17 Sep 2006 15:58:54 -0700

> environments, and mailers?  I venture to doubt it.

Uh... how about something like "in your registration form, non-ASCII 
code points must be encoded as an XML-like NCR"

I didn't say that the registration forms had to support non-ASCII. Only 
that it be possible for everyone to tell what the code points were 
supposed to be and for IANA to arrive at a UTF-8 registry from them.

Addison

---

From: John Cowan <[email-masked]>
Date: Sun, 17 Sep 2006 19:10:50 -0400

> supposed to be and for IANA to arrive at a UTF-8 registry from them.

Okay, I'm fine with that, provided the NCR version remains as the
normative version of the registry; this is justified by *stare decisis*.

-- 
John Cowan  http://ccil.org/~cowan    [email-masked]
There are books that are at once excellent and boring.  Those that at
once leap to the mind are Thoreau's Walden, Emerson's Essays, George
Eliot's Adam Bede, and Landor's Dialogues.  --Somerset Maugham

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 18 Sep 2006 03:05:26 +0200

> I find this fascination with ASCII slightly quaint.

I vaguely recall a day when I read a beginner's book about
PASCAL, and was annoyed by its ASCII table, because it had
nothing to do with "real world standards" (at that time a
"real world standard" had a IBM GA number and used EBDCIC ;-)

Charsets in general are fascinating.  Admittedly I'm less
thrilled by SCSU (implementing it takes more than 5 minutes,
so far I got it).

Frank
-- 
<http://purl.net/xyzzy/src/bocu.cmd>

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 18 Sep 2006 04:20:02 +0200

Stephane Bortzmeyer wrote:

> This is the sort of things that happen when you reinvent the
> wheel, and create a new format for a registry, instead of
> trusting existing formats...

...I trust that the 4646 record-jar format is fine.  Actually
I'm sure.  We discussed XML vs. "STD 10 header style" in the
last round, and also UTF-8 vs. NCRs.  So far I haven't seen a
compelling reason to change this.  Folks who really want XML
obviously can convert it.  Converting it to a variant of CSV
should be also trivial, ASCII offers 0x1C .. 0x1F separators
if all else fails.

Frank

P.S::  Your lineend problem beats me, my two gawks (2.15pl5 and
3.0.6) don't care if it's Lf or CrLf.  Officially it should be
CrLf as in RFC 4234 (and as specified in RFC 4646).