Subject: [Ltru] UTF-8 In Registry Considered Harmful, or Doom! Doom! I Cry Doom!

From: John Cowan <[email-masked]>
Date: Tue, 20 Mar 2007 15:45:35 -0400

I wish to oppose in the strongest terms the proposal to change the
registration process in RFC 4646bis to publish the registry in UTF-8.
This is *not* a matter of a one-time conversion from an ASCII I-D to a
UTF-8 registry.  It also affects every addition and update thereafter.

The essential link in adding and modifying subtags is an email sent from
the Language Subtag Reviewer to IANA requesting the change.  Now either
(1) that email will contain NCRs or (2) it will not.  In case (1), a
*continuing* capacity to convert the NCRs to UTF-8 on the part of IANA is
assumed, something they seemingly do not have today.  Are we confident
that this will reliably be performed as part of registry updating?
We are creating a whole new risk for ourselves.  I foresee a time when
the registry contains a mixture of UTF-8 and NCR representations.

In case (2), the email contains the characters directly.  Then either
(2a) it is in UTF-8, or (2b) it is in some other encoding sufficient
to represent the particular characters needed.  In either subcase, we
are taking risks: email is notoriously the most i18n-broken part of the
Internet, and probably every one of us has received at least one broken
email of this kind.  In subcase (2b) we also add the risk that IANA (or
their mail-processing tools) will fail to interpret the encoding, or will
interpret it wrongly due to problems at the sender's end.  We are, once
again, creating a whole new risk for ourselves.  I foresee a time when
the registry contains a mixture of UTF-8 and non-UTF-8 representations.

There is also the matter of the implicit contract which RFC 4646
created about the registry: that it would be in ASCII with NCRs.
Are we *convinced* that we have *no choice* but to break this contract?
The current system may not produce pretty results, but it *works*.
Let's not break it gratuitously.

Yours emphatically,

-- 
Mos Eisley spaceport.  You will never           John Cowan
see a more wretched hive of scum and            [email-masked]
villainy -- unless you watch the                http://www.ccil.org/~cowan
Jerry Springer Show.   --georgettesworld.com

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 20 Mar 2007 14:10:39 -0700

I have some qualms along the same line. I don't think we should expect IANA
to do anything but paste some text into a position in the file. (Frankly, I
think even that is too much -- it would be better if we just handed them a
replacement file which we've reviewed, but people didn't like that idea.) So
processing NCRs and extracting UTF-8 is too much.

The only process for producing UTF-8 that seems tenable to me is where the
text is sent as an email file attachment, where the attachment is in UTF-8.

Mark

-- 
Mark

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 21 Mar 2007 18:18:05 +0100

> processing NCRs and extracting UTF-8 is too much.

We have a reliable solution in RFC 4646 (excl. the ABNF bug)...
 
> The only process for producing UTF-8 that seems tenable
> to me is where the text is sent as an email file attachment,
> where the attachment is in UTF-8.

...but actually I'm sure that the subtag reviewer and IANA
will find ways to send and receive UTF-8 working among them.

>> The essential link in adding and modifying subtags is an
>> email sent from the Language Subtag Reviewer to IANA 
>> requesting the change.

>> Now either (1) that email will contain NCRs or (2) it
>> will not.

It won't.

>> In case (2), the email contains the characters directly.
>> Then either (2a) it is in UTF-8, or (2b) it is in some
>> other encoding sufficient to represent the particular
>> characters needed.

It's UTF-8 (2a), nothing else.  Installing a mail program
getting this right is a duty for the subtag reviewer.  In
rare cases it might be ASCII if the template happens to be
ASCII-only.  It's the duty of IANA to upgrade their email
program to something supporting B64 or QP encoded UTF-8 in
an Internet Message.  Probably they've already done this
in the previous millennium.

> email is notoriously the most i18n-broken part of the
> Internet, and probably every one of us has received at
> least one broken email of this kind.

Yes, but there are only two critical installations, from
the subtag reviewer to IANA.

> In subcase (2b) we also add the risk that IANA (or their
> mail-processing tools) will fail to interpret the encoding

Yes, (2b) is unacceptable, excl. the special case US ASCII.

> There is also the matter of the implicit contract which
> RFC 4646 created about the registry: that it would be in
> ASCII with NCRs.

> Are we *convinced* that we have *no choice* but to break

I'm 100% sure that we do NOT need to break this contract,
and that it will create unnecessary problems for some users
of the registry.

But subtag reviewer and IANA should IMO get their part right
after testing their setup.

> The current system may not produce pretty results, but it
> *works*. Let's not break it gratuitously.

+1

Frank

---

From: "McDonald, Ira" <[email-masked]>
Date: Wed, 21 Mar 2007 10:49:48 -0700

Hi,
 
My two cents - John Cowan's right - this (native UTF-8 registry)
is a dangerous road that's several years ahead of a reliable
Internet email infrastructure - even UTF-8 _attachments_ force
the sender and ALL commenters to be able to correctly display
and print UTF-8 on their machines - not happening anytime soon!
 
Please consider sticking with ugly but SAFE use of NCRs.
 
After all, we're talking about edge conditions here - not the whole
body of the registry.
 
Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Chair - Linux Foundation Open Printing WG
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked] 

-----Original Message-----
From: Mark Davis [mailto:[email-masked]]
Sent: Tuesday, March 20, 2007 4:11 PM
To: John Cowan
Cc: [email-masked]
Subject: Re: [Ltru] UTF-8 In Registry Considered Harmful,or Doom! Doom! I Cry Doom!

I have some qualms along the same line. I don't think we should expect IANA to do anything but paste some text into a position in the file. (Frankly, I think even that is too much -- it would be better if we just handed them a replacement file which we've reviewed, but people didn't like that idea.) So processing NCRs and extracting UTF-8 is too much. 

The only process for producing UTF-8 that seems tenable to me is where the text is sent as an email file attachment, where the attachment is in UTF-8. 

Mark

On 3/20/07, John Cowan < HYPERLINK "mailto:[email-masked]"[email-masked]> wrote: 

Yours emphatically,

--
Mos Eisley spaceport.  You will never           John Cowan
see a more wretched hive of scum and             HYPERLINK "mailto:[email-masked]"[email-masked]
villainy -- unless you watch the                 HYPERLINK "http://www.ccil.org/~cowan"http://www.ccil.org/~cowan
Jerry Springer Show.   -- HYPERLINK "http://georgettesworld.com"georgettesworld.com

_______________________________________________
Ltru mailing list 
HYPERLINK "mailto:[email-masked]"[email-masked]
HYPERLINK "https://www1.ietf.org/mailman/listinfo/ltru"https://www1.ietf.org/mailman/listinfo/ltru

-- 
Mark 

-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.446 / Virus Database: 268.18.16/729 - Release Date: 3/21/2007 7:52 AM

---

From: David Conrad <[email-masked]>
Date: Wed, 21 Mar 2007 14:20:47 -0700

Hi,

Ah ha!  So this was a trick question... :-)

We're discussing this with the IESG folks right now.  It isn't clear  
to me what the consensus of the working group is on this issue, so  
I'll retract previous statements for the time being.  This isn't to  
say we've made a decision one way or the other, of course.  Rather we  
simply don't have sufficient information.

Rgds,
-drc

On Mar 21, 2007, at 10:49 AM, McDonald, Ira wrote:
> I have some qualms along the same line. I don't think we should  
> expect IANA to do anything but paste some text into a position in  
> the file. (Frankly, I think even that is too much -- it would be  
> better if we just handed them a replacement file which we've  
> reviewed, but people didn't like that idea.) So processing NCRs and  
> extracting UTF-8 is too much.
>
> The only process for producing UTF-8 that seems tenable to me is  
> where the text is sent as an email file attachment, where the  
> attachment is in UTF-8.
>
> Mark
>
> On 3/20/07, John Cowan <[email-masked]> wrote: I wish to oppose in  
> the strongest terms the proposal to change   the