Subject: [Ltru] UTF-8

From: "Reshat Sabiq (Reşat)"<[email-masked]>
Date: Thu, 21 Jun 2007 12:48:55 -0500

I'd like to get a feel for what the concerns are about using UTF-8 for
subtag registry. Based on my testing locally, that works fine on IE, and
firefox (the latter both on Windows and Linux). If anybody happens to
use a shell or shell client that can't handle UTF-8, all they have to do
 when they see a mangled character is use one of many different clients
that can handle UTF-8. There's hardly any problem, as far as i can see.

P.S. I'm not a member of the list, but i'll try to track it down thru
http interface.

- --
My public GPG key (ID 0x262839AF) is at: http://keyserver.veridis.com:11371

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Thu, 21 Jun 2007 14:01:58 -0400

> that can handle UTF-8. There's hardly any problem, as far as i can see.

The issue is that email is an essential part of the process, and
guaranteeing that email gets through the whole of the infrastructure
unmangled is still difficult.

-- 
Possession is said to be nine points of the law,                John Cowan
but that's not saying how many points the law might have.       [email-masked]
        --Thomas A. Cowan (law professor and my father)

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 21 Jun 2007 11:28:27 -0700

Hi -

As a technical contributor...

> From: "John Cowan" <[email-masked]>
> To: ""Reshat Sabiq (Re?at)"" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Thursday, June 21, 2007 11:01 AM
> Subject: Re: [Ltru] UTF-8
...
> unmangled is still difficult.
...

A UTF-8 registry doesn't require that the format used to communicate
additions and changes also be UTF-8.  I think the real question is that
of how much one is willing to rely on the good folks at IANA to handle
the translation of whatever the updates look like into UTF-8.

Randy

---

From: John Cowan <[email-masked]>
Date: Thu, 21 Jun 2007 14:31:42 -0400

> the translation of whatever the updates look like into UTF-8.

Quite.  My notion is that the less IANA has to do (considering what
all else it has to do), the better.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If I have seen farther than others, it is because I am surrounded by dwarves.
        --Murray Gell-Mann

---

From: GerardM <[email-masked]>
Date: Thu, 21 Jun 2007 20:53:30 +0200

Hoi,
It happens often that exactly because of the content is NOT UTF-8 that I get
the information mangled. With UTF-8 you either get the message or it will be
indicated that you have insufficient fonts installed. The notion that the
current encoding always works is wrong.
Thanks,
    Gerard

---

From: John Cowan <[email-masked]>
Date: Thu, 21 Jun 2007 14:57:38 -0400

GerardM scripsit:

> It happens often that exactly because of the content is NOT UTF-8 that
> I get the information mangled. With UTF-8 you either get the message
> or it will be indicated that you have insufficient fonts installed. The
> notion that the current encoding always works is wrong.

It always works for ASCII text, and the Registry is restricted to
ASCII text.

-- 
John Cowan                              [email-masked]
            http://www.ccil.org/~cowan
Humpty Dump Dublin squeaks through his norse
                Humpty Dump Dublin hath a horrible vorse
But for all his kinks English / And his irismanx brogues
                Humpty Dump Dublin's grandada of all rogues.  --Cousin James

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 21 Jun 2007 12:37:22 -0700

Hi -

As co-chair...

We've had this discussion  several times.  As long as there is
an insistance that the format for the communication of updates
is the same as that of the registry itself, I'm pretty sure the conclusion
won't change in the forseeable future.  Just on this list, we've all
seen the picturesque results of various incompatible mailers
(and even different configurations of the same mailer) mangling
what should be simple UTF-8.  Unfortunately, it doesn't appear to
just be a question of having the right fonts.

If this thread is to move forward to a different conclusion, the only
path by which I can see it succeeding is to recognize that the
email communication of changes might have to rely on a different
coding of non-ASCII things from what the registry itself uses.
 If we can agree that the existing email format is adequate for purposes
of communicating changes, then there are three decisions needed:
    (0) do we want to stick with email as our mechanism for communicating
          updates?
   (1) is a UTF-8 registry indeed desirable?
   (2) if so, are we willing to allow IANA to convert the apropriate bits
        of an update directive conveyed in ASCII into the "real" code points?

But please, let's not rehash the discussion we've already had.

Randy

---

From: Addison Phillips <[email-masked]>
Date: Thu, 21 Jun 2007 12:48:33 -0700

> all else it has to do), the better.

The question is: can the LSR and/or his/her proxy (i.e. Doug) manage to 
send email to IANA encoded as UTF-8. All other emails are "nice to 
have", but not required. I think the answer here is that they can manage it.

The "compelling reason" against UTF-8 that I've seen has nothing to do 
with email or browser availability or IANA's skill set. It is simply 
that it is a change in the registry format and could break existing 
implementations that rely on the ASCII format.

As the original (and long time) proponent of a UTF-8 registry, I find 
that argument to be the one sticking point for which I cannot find a 
good counter argument. If we change it, we will have to make a conscious 
decision to break with the past on that point.

I note also that UTF-8 has failed to reach consensus on at least three 
separate occasions.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: Addison Phillips <[email-masked]>
Date: Thu, 21 Jun 2007 12:52:09 -0700

I think we should close this discussion. We're unlikely to make it 
further this time than last. I think UTF-8 is desirable and the 
procedural points can be solved to deal with it. But I don't think we'll 
find consensus down this path (or at least consensus that doesn't have 
multiple loud objections). Ditto on the XML proposal.

Addison

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 21 Jun 2007 22:40:12 +0200

On Thu, Jun 21, 2007 at 12:52:09PM -0700,
 Addison Phillips <[email-masked]> wrote 
 a message of 58 lines which said:

> But I don't think we'll find consensus down this path (or at least
> consensus that doesn't have multiple loud objections). Ditto on the
> XML proposal.

For UTF-8 and XML, a possible path is to have unofficial,
automatically produced, copies of the registry in other formats (see
http://www.langtag.net/registries.html). It would solve most of the
use cases.

XML is already there, UTF-8 is certainly possible (any volunteer to
write the program?)

---

From: Peter Constable <[email-masked]>
Date: Fri, 22 Jun 2007 09:20:04 -0700

> From: Addison Phillips [mailto:[email-masked]]

> further this time than last...

Before we kill this off too quickly...

I don't remember all the discussion from the last time around, but seems to me we at least have identified a small set of core issues:

1) Would UTF-8 be desirable (all other things being equal)?

2) Is such a breaking change in the registry acceptable?

3) Does the content of a registry edit have to be exchanged in email bodies on IETF-languages and be in the same encoding in those emails?

Randy broke 3 into two parts, which I'll modify slightly:

3a) Do we need or want to stick with communicating proposed registry edits solely in the body of emails?

3b) If "yes" to 3a, then can we use a different representation in discussion on IETF-languages and depend on the LST Reviewer (or his assistants) and IANA between them to convert the content as needed?

I think there is consensus on 1. If we are willing to discuss this a bit further, it might be helpful to focus on 2 before 3.

Wrt 2: Non-ASCII content would appear in certain attribute fields accompanying a subtag: only in description or comment fields. Would there really be parsers that make assumptions about the content of these fields, or that assume only byte values within the range 0x20 - 0x7E would be encountered?

Peter

---

From: "Reshat Sabiq (Reşat)"<[email-masked]>
Date: Fri, 22 Jun 2007 12:13:03 -0500

> It always works for ASCII text, and the Registry is restricted to
Well, i think for me the interest is whether the next release could
mandate UTF-8, so that we don't have to deal w/ escapes.
I guess there are 2 issues:
1. Email clients
2. Browser clients

On 1:
If you can see s w/ cedilla on the first line above, then your email
client handled UTF-8; if not maybe you could try choosing UTF-8 encoding
manually. I know that Thunderbird and Outlook don't have a problem w/
UTF-8. What other email clients are we talking about? If your pine
client thru your shell can't handle it, then really you will only have
few characters mangled, because charactes overlapping w/ ASCII are
rendered the same whether it's UTF-8 or ISO-8859-1. Then you still have
an option of using another email client to UTF-8 appropriately when you
really care.

On 2:
I don't think any up-to-date browsers have any problem w/ UTF-8 (IE,
firefox, seamonkey, opera). The only ones that would have problem could
be cell-phone based ones that don't belong to one of the brands above
and some shell-based ones, which probably depends on what shell is being
used. And then again i'd say it's not a big deal, because you can use
another client to read stuff if you care. Besides the pros of not having
to deal w/ escapes outweigh the cons of not having UTF-8 support on some
cell-phone devices and some shells.

So what exactly email and browser clients are we talking about that
don't support UTF-8? If it's only a matter of email, we could say: "when
using email, you can send requests using escapes, but these escapes will
be posted in UTF-8 in the registry"? If we adopt UTF-8 for new
submissions, we could also say "starting on <someDate/> all character
escapes in the registry will be converted to UTF-8."

Most importantly, if UTF-8 is going to be easier for 95% of mainstream
users, couldn't we tell 5% of peculiar users to deal w/ it? The registry
for the internet should probably be based on up-to-date technologies,
rather than trying to accommodate obsolete ones as well, which i think
are used to rarely to be cared about anyway.

- --

---

From: "McDonald, Ira" <[email-masked]>
Date: Fri, 22 Jun 2007 10:34:09 -0700

Hi,

A suggested refinement to 3b below:

* Email discussion remains 7-bit ASCII with some escape
  convention.

* LSR (and helpers) send all official approved updates to
  IANA as plaintext email in US-ASCII with an attachment
  of the UTF-8 text for cut-and-paste into the Registry.

The necessary effort to use the ASCII to UTF-8 conversion
tool is moved back into the LSR's scope.

Hmm?

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Chair - Linux Foundation Open Printing WG
Blue Roof Music / High North Inc
PO Box 221  Grand Marais, MI  49839
phone: +1-906-494-2434
email: [email-masked]

-----Original Message-----
From: Peter Constable [mailto:[email-masked]]
Sent: Friday, June 22, 2007 11:20 AM
To: Addison Phillips; Randy Presuhn
Cc: LTRU Working Group
Subject: RE: [Ltru] UTF-8

<...snip...>

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.472 / Virus Database: 269.9.4/860 - Release Date: 6/21/2007 5:53 PM

---

From: "Reshat Sabiq (Reşat)"<[email-masked]>
Date: Fri, 22 Jun 2007 12:35:32 -0500

> are used to rarely to be cared about anyway.
I just read the updates thru http, and i agree w/ Randy and Addison's
idea of converting escapes
to UTF-8 after the email phase. The submitter can always verify the
correctness thereof thru http,
and object if there are any mistakes. If this is done by a little
program, that is written w/ some unit tests, and tested, then errors
should be minimal.
As far as passivity, if this takes effect w/ the next release of the
registry spec, and people know about it in advance, then it's natural
for it to not be passive. For temporary passivity, the old registry
could continue to exist w/o updates at the existing URL, and the new
registry to be updated could be exposed thru a new URL.
That happened in java betweem 1.3 and 1.4, and 1.4 and 5: it's
inevitable at early stages, and maybe even beyond. Deep down, do we not
all know that it will be UTF-8 and XML at some point? I think we do, so
passivity question will come up anyway, it's just a matter of managing
it well.

- --