Subject: [idn] Which lanuages/scripts to reorder?

From: Erik Nordmark <[email-masked]>
Date: Mon, 22 Oct 2001 16:06:10 +0200 (CEST)

One issue with reordering that hasn't been put on the table
is that, as a process, I'm concerned that it might never complete.
This concern is in addition to the other concerns I've seen expressed on
the list.

Suppose that we start of defining a reordering scheme that allows for
longer labels for some set of languages/scripts.
Suppose this gets tested etc. and finished in the WG.
Duing IETF last call somebody objects that their favorite language/script
doesn't get the same benefits. What happens?

If we have accepted to do reordering for some languages/scripts what technical
argument do we have for saying "no" to any particular language/script?

In worst case the above can repeat over and over - each new attempt at
improving the maximum length labels for some subset of the worlds
languages/scripts can result in requests to add yet another language/script
to the set that needs to be supported.

Thus as far as I can tell embarking on solving reording as a pre-cursor to
getting IDN specified and usable, might result in delaying IDN for an
undetermined amount of time.

   Erik

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Mon, 22 Oct 2001 10:58:25 -0400

Erik,

The process, if any, of future revision of thie WG's product, again, if any,
is important. Either there is none, and it is "permanent" until discarded,
or there is some, which may work (allowing semi-graceful revisioning).

We have the assertion that reordering (blocks of the UTC's product) is useful
for labels composed in a well-defined range of code-points (aka "Chinese").

We have the assertion that the UTC's product is incomplete.

Both appear to be correct.

One possible course of action is to discard the first assertion (utility for
labels using Chinese characters), until the second assertion is false.

One possible course of action is to accept the first assertion, while the
second assertion remains true.

It is my view that the second course of action is the better of the two.

Eric

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 23 Oct 2001 04:14:18 +0900

Hi, Erik

I answered for your concerns on reordering supports for new scripts 
in the new postings  titled with
"idn] suggestion: two prefices scheme for unassigned code points treatments.".

I will answer for your other concerns later time.

Thanks.

Soobok Lee

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 22 Oct 2001 22:01:58 +0200 (CEST)

Eric, 

> or there is some, which may work (allowing semi-graceful revisioning).

This is assuming that the WG actually produces something.
My concern in the area of reordering is that incremental requests for
supporting additional laguages/scripts for reordering benefits might mean
that the WG never gets to deliver anything - there are just too many
scripts/languages that can appear.

Thus my concern is completely unrelated to the version issues
others have brought up with respect to UTC.

If the WG embarks on doing reordering for 1,2,3,4 languages/scripts
how do we know that this will be sufficient? Next month there might be
requests for a few more languages/scripts. Etc.

  Erik

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Mon, 22 Oct 2001 16:55:02 -0400

Erik,

It is an assumption that this WG will actually produce something, which is
why "if any" appears twice in the first line of my remarks.

I appreciate the concern you expressed. I also appreciate that the question
"is IDNA/Nameprep/ACE enough?" can be answered in the negative, without the
reordering, even in the context of an ACE framework.

Restating the cautionary statement may helpful:

	Because an additional reordering request may be made for some
	as yet unspecified block of code-points, rough consensus for
	the reordering a specified block of code-points is undesireable.

This doesn't seem like prudent process. An alternative statement is:

	Because additional reordering requests will be made for some
        as yet unspecified blocks of code-points over time, rough
	consensus for a version identifier is desireable.

Eric

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 23 Oct 2001 09:26:25 +0900

----- Original Message ----- 
From: "Erik Nordmark" <[email-masked]>
To: "Eric Brunner-Williams in Portland Maine" <[email-masked]>
Cc: "Erik Nordmark" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2001 5:01 AM
Subject: Re: [idn] Which lanuages/scripts to reorder? 

> requests for a few more languages/scripts. Etc.

seamless upgrade idea for adding new script reordering tables and new nameprep rules (NF/casemappings) are described in the posting titled with "suggestion : two prefices ....". No newer prefix other than (zq-- and uq--) will be needed forever.
I recommend you to comment on the suggestion first before you go further.

Soobok Lee

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 23 Oct 2001 12:25:25 +0200 (CEST)

> seamless upgrade idea for adding new script reordering tables and new
> nameprep rules (NF/casemappings) are described in the posting titled with
> "suggestion : two prefices ....". No newer prefix other than (zq-- and uq--)
> will be needed forever. I recommend you to comment on the suggestion first
> before you go further.

Soobok,

We seem to have a failure to communicate.
I've placed one new concern on the table which does not have anything to
do with how you could technically upgrade to new reordering tables.

My concern is about that the process of definition reordering tables
might never terminate (or might take a decade or so) since there might very
well be a request to add more and more languages/scripts to the tables before
we ever get to produce a single RFC.

I do not have anything to add to the techical issue of how upgrade
to new reordering tables as that was not the issue over which 
I expressed concern. Thus it makes no sense to me to comment on that issue.

Sincerely,
  Erik

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 23 Oct 2001 21:34:43 +0900

----- Original Message ----- 
From: "Erik Nordmark" <[email-masked]>
To: "Soobok Lee" <[email-masked]>
Cc: "Erik Nordmark" <[email-masked]>; "Eric Brunner-Williams in Portland Maine" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2001 7:25 PM
Subject: Re: [idn] Which lanuages/scripts to reorder? 

Erik,

new reordering tables can be added only when there comes new nameprep
revision for newly added scripts, because reordering tables is proposed as
one candidate of nameprep/ACE components. that is, reordering tables are
bound by the versioning of nameprep.
 
nameprep (as one profile of stringprep) upgrade paths also never terminate
because there will be more and more scripts pending for approval over time.

If you look into nameprep/stringprep document, each version of nameprep
should specify  the used unicode version and the list of unassigned code points in 
in that version. If new versions of unicode accumulate enough set of new scripts for justifying major nameprep revision, then IETF IDN WG should work again to give birth to new nameprep profile with new list (maybe reduced by the amount of new assigned code points) of unassgined code points and new version number of unicode.

Therefore, your concerns about long path of reordering revisioning is NOT new problem,but rather just one of never-terminaing nameprep/ACE revisioning problem.

if we finalize IDN  before TAGALOG is assigned and added, current nameprep's architecture for unassigned code points, does not allow us to register tagalog domains   until new nameprep version come out by IETF activities in the undefined time in the future.

My two prefix(uq-- and zq--) scheme proposal tries to solve this inherenet problem of user inconveniences in treatment of unassigned code points,  even before distributions of new version of nameprep which often take *too much time* to serve the TAGALOG users' need.

Soobok Lee

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 23 Oct 2001 15:12:57 +0200 (CEST)

> Therefore, your concerns about long path of reordering revisioning is NOT
> new problem,but rather just one of never-terminaing nameprep/ACE revisioning
> problem.

I think they are very different.

nameprep/stringprep would probably need to be revised if there are Unicode code
points added before the document becomes an RFC.

My point about reordering is that it might need to be revised
each time somebody requests adding support for one of the 
languages/scripts that are already supported in Unicode, even if Unicode
doesn't add anything. I don't know how many languages/scripts
that Unicode currently support but the number is presumably a few hundered.

  Erik

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 23 Oct 2001 23:12:56 +0900

----- Original Message ----- 
From: "Erik Nordmark" <[email-masked]>
To: "Soobok Lee" <[email-masked]>
Cc: "Erik Nordmark" <[email-masked]>; "Eric Brunner-Williams in Portland Maine" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2001 10:12 PM
Subject: Re: [idn] Which lanuages/scripts to reorder? 

> that Unicode currently support but the number is presumably a few hundered.

Hundreds of languages, but, Not so many scripts. AFIAK,extended Latin scripts are shared by hundreds of languages as their writing systems. For latin labels which are already most favored by ACE, reordering does not help ( only 1% save on ACE length ). MY REORDERING I-D includes reordering on latin script but recommends not to include it because of its marginal gain on latin.

Each script has its ACE-Z label length constant which is listed in my
REORDERING I-D 2.0.
many scripts has relatively small set of basic alphabets, and their
ACE label length is less than 2*N.  han/hangeul ACE label length is
greater than 3*N without reordering. With reordering, it falls to 2.2*N.

MY REORDERING I-D 2.0 suggests one option of supporting only Han/HANGEUL 
which are most dis-favored scripts in ACE. If we decide to support all
major scripts in the current UNICODE, it will be done in the my I-D 3.0.

if we setup a certain threshold value on the label length constant for each script to determine the necessity of reordering supports, we could also restrict the set of new added scripts for which we should prepare reorderings in a regular fashion. I propose the threshold should be 1.7~ 2.2
with which threshold, only han/hangul/ethiopic/katakana/hiragana would be
supported in reordering.

And many script are archaic or used by too few native-speakers or scholars.
cost/benefits analysis based on the population using each script can be
performed and helps. That's already contained in My REORDERING I-D 2.0.

I enclose some proposed scripts in unicode. Most of them from minorities or 
un-industrialized countries or extinct/archaic ones. 
I expect those scripts would take fair amounts of time to be actively 
used in IDN in the future. That would determine when we will have major upgrades on  
reordering supports and namepre/ACE supports for them.

Soobok Lee
----------

For the BMP 
  a.. Cham 
  b.. Tai (Dai) scripts 
  c.. Glagolitic 
  d.. Coptic 
  e.. Buginese 
  f.. Old Hungarian 
  g.. Phoenician 
  h.. Avestan 
  i.. Tifinagh (Berber) 
  j.. Javanese 
  k.. Lepcha (Rong) 
For Plane 1 (surrogates) 

  a.. Basic Egyptian Hieroglyphics 
  b.. Meroitic 
  c.. Old Persian Cuneiform 
  d.. Tengwar 
  e.. Cirth 
  f.. Brahmi 
  g.. Old Permic 
  h.. South Arabian 
  i.. Pollard 
  j.. Blissymbolics 
  k.. Soyombo 

Approved ones.

Philippine Scripts 

  a.. Tagalog 
  b.. Hanunóo 
  c.. Buhid 
  d.. Tagbanwa 

Pending..
  a.. Linear B (see also the overview for Aegean scripts) 
  b.. Ugaritic Cuneiform 
  c.. Shavian 
  d.. Osmanya 
  e.. Cypriot syllabary (see also the overview for Aegean scripts) 
For the BMP

  a.. Limbu

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 23 Oct 2001 07:59:34 -0700

>Therefore, your concerns about long path of reordering revisioning is NOT
new >problem,but rather just one of never-terminaing nameprep/ACE
revisioning >problem.

I disagree completely. Nameprep is designed so that it will work even if a
server and a client are on different versions, and without any version
number involved. That would absolutely fail if reordering were changed in
any future version.

Introducing explicit version numbers (and it could not be limited to 2)
would not help -- it would just make things worse.

Mark
—————

Δός μοι ποῦ στῶ, καὶ κινῶ τὴν γῆν — Ἀρχιμήδης
[http://www.macchiato.com]

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "Erik Nordmark" <[email-masked]>
Cc: "Eric Brunner-Williams in Portland Maine" <[email-masked]>;
<[email-masked]>
Sent: Tuesday, October 23, 2001 05:34
Subject: Re: [idn] Which lanuages/scripts to reorder?

----- Original Message -----
From: "Erik Nordmark" <[email-masked]>
To: "Soobok Lee" <[email-masked]>
Cc: "Erik Nordmark" <[email-masked]>; "Eric Brunner-Williams in
Portland Maine" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2001 7:25 PM
Subject: Re: [idn] Which lanuages/scripts to reorder?

Erik,

If you look into nameprep/stringprep document, each version of nameprep
should specify  the used unicode version and the list of unassigned code
points in
in that version. If new versions of unicode accumulate enough set of new
scripts for justifying major nameprep revision, then IETF IDN WG should work
again to give birth to new nameprep profile with new list (maybe reduced by
the amount of new assigned code points) of unassgined code points and new
version number of unicode.

if we finalize IDN  before TAGALOG is assigned and added, current nameprep's
architecture for unassigned code points, does not allow us to register
tagalog domains   until new nameprep version come out by IETF activities in
the undefined time in the future.

My two prefix(uq-- and zq--) scheme proposal tries to solve this inherenet
problem of user inconveniences in treatment of unassigned code points,  even
before distributions of new version of nameprep which often take *too much
time* to serve the TAGALOG users' need.

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 00:17:08 +0900

----- Original Message ----- 
From: "Mark Davis" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "Erik Nordmark" <[email-masked]>
Cc: "Eric Brunner-Williams in Portland Maine" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2001 11:59 PM
Subject: Re: [idn] Which lanuages/scripts to reorder? 

> number involved. That would absolutely fail if reordering were changed in

Reordering won't be changed and  remains as sub-optimal frequency tables
forever without new prefix, as i repeated many time in this list.
 
What i mean is adding NEW reordering tables for *NEW SCRIPTS*
in accordance with new SCRIPT support in newer nameprep.
I agree with you that reordering table for old existing scripts should not be changed .

each Nameprep version defines unassigned code points which reordering should
not touch at all. That enforce  reordering  upgrade path strictly to  follow  that of nameprep.

Soobok Lee

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 23 Oct 2001 23:35:41 +0800

The problem is that there will be people who will request reordering (or
they complain unfair treatment). It is very hard to explain to X why
there is reordering for Y but not X.

Why create a potential timebomb for ourselves?

-James Seng

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "Mark Davis" <[email-masked]>; "Erik Nordmark"
<[email-masked]>
Cc: "Eric Brunner-Williams in Portland Maine" <[email-masked]>;
<[email-masked]>
Sent: Tuesday, October 23, 2001 11:17 PM
Subject: Re: [idn] Which lanuages/scripts to reorder?

----- Original Message -----
From: "Mark Davis" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "Erik Nordmark"
<[email-masked]>
Cc: "Eric Brunner-Williams in Portland Maine" <[email-masked]>;
<[email-masked]>
Sent: Tuesday, October 23, 2001 11:59 PM
Subject: Re: [idn] Which lanuages/scripts to reorder?

What i mean is adding NEW reordering tables for *NEW SCRIPTS*
in accordance with new SCRIPT support in newer nameprep.
I agree with you that reordering table for old existing scripts should
not be changed .

each Nameprep version defines unassigned code points which reordering
should
not touch at all. That enforce  reordering  upgrade path strictly to
follow  that of nameprep.

Soobok Lee

> > > nameprep rules (NF/casemappings) are described in the posting
titled
> with
> > > "suggestion : two prefices ....". No newer prefix other than (zq--
and
> uq--)
> > > will be needed forever. I recommend you to comment on the
suggestion
> first
> > might never terminate (or might take a decade or so) since there
might
> very
> > well be a request to add more and more languages/scripts to the
tables
> before

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 00:35:08 +0900

----- Original Message ----- 
> would not help -- it would just make things worse.

Mark, 
You may missed my point.
two prefix uq-- and zq-- does *not*  limit  the versioning into 2 times, but infinite times.

zq-- is for valid ACE labels  for the all version of nameprep and
uq-- is for invalid ACE labels including at least one unassigned code points for all versions of 
nameprep.

This differentiating ace prefixing scheme works  in all version of nameprep from 1 to infinity.

newly supported script in nameprep/ACE version 5 will be
encoded using  uq--????  with old nameprep/ACE version 4  and below.
But with namepre/ACE version 5 or above, it will be encoded zq--???? .

Likewise,
labels contain newly supported script in nameprep/ACE version 6 will be
encoded using  uq--????  with old nameprep/ACE version 5 and below.
But with namepre/ACE version 6 or above, it will be encoded zq--???? .

And so on.

There is no need for version numbered prefix  for each of version 2,3,4,5,6, .....

Current nameprep does not encode unassigned code points into ACE for
"saved strings", but my two prefix scheme allow it with "uq--" prefix  which
provide with  rooms for incorporating  some useful fallback mechanims.
Even with dealing with ACE foor lookup "query", it provide some clues
about the version of nameprep which encode the ACE lables by the
distintion of "uq--" and "zq--".
I am proposing new nameprep behavior for unassigned code points for 
"saved string" and "query" through out  nameprep versioning path. 
Not about  versioning nameprep itself!

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 00:51:12 +0900

----- Original Message ----- 
From: "James Seng/Personal" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "Mark Davis" <[email-masked]>; "Erik Nordmark" <[email-masked]>
Cc: "Eric Brunner-Williams in Portland Maine" <[email-masked]>; <[email-masked]>
Sent: Wednesday, October 24, 2001 12:35 AM
Subject: Re: [idn] Which lanuages/scripts to reorder? 

> there is reordering for Y but not X.
I will try to include as many script as possible in the final REORDERINg I-D.

But, Even without reordering,
some tagalog people will come and say  why ACE-Z  favors latin  with special
 literal mode supports.
Do you think this timebomb invalidates ACE-Z literal mode ?  No.

cost/benefits-based trade-off  can be the answer for this disputes on
special treatment in ACE-Z as well as REORDERING.

REORDERING's memory requirement is minimal  while it give 30% boostup
for hangul names. I don't recommend  reordering supports for latin and runic script.
You can find the rationale in my REORDERING I-D 2.0.

Soobok Lee

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 23 Oct 2001 10:09:52 -0700

That may work. However, I believe that it implies some limitiations on
reordering to preserve compatibility across versions.

Example: Suppose that at some time in the future there are 4 versions: A, B,
C, D.

I send you zq--XXX. It contains a character UA that is unassigned in A (but
not B), a character UB that is unassigned in B (but not C), and a character
UC that is unassigned in C (but not D). With normal nameprep this doesn't
matter. You should use version D of nameprep to read the IDN. If you use a
back-rev'd version, you will simply recognize that it has an unassigned
character, and reject the name.

Now suppose that we have reordering, and you are back-rev'ed, and support C
but not D. When you get UC, you have to be able to recognize that it is an
unassigned character to know to reject it. That means that within any
version (I think after the first), isUnassigned(x) if and only if
isUnassigned(reordered(x)). Thus any future changes in reordering have to
preserve the "unassigned" status: so the future reorderings have to be, in
some sense, "in place".

Mark
—————

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "Mark Davis" <[email-masked]>; "Erik Nordmark"
<[email-masked]>
Cc: "Eric Brunner-Williams in Portland Maine" <[email-masked]>;
<[email-masked]>
Sent: Tuesday, October 23, 2001 08:35
Subject: Re: [idn] Which lanuages/scripts to reorder?

----- Original Message -----
> would not help -- it would just make things worse.

Mark,
You may missed my point.
two prefix uq-- and zq-- does *not*  limit  the versioning into 2 times, but
infinite times.

zq-- is for valid ACE labels  for the all version of nameprep and
uq-- is for invalid ACE labels including at least one unassigned code points
for all versions of
nameprep.

This differentiating ace prefixing scheme works  in all version of nameprep
from 1 to infinity.

Likewise,

And so on.

There is no need for version numbered prefix  for each of version 2,3,4,5,6,
.....

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 03:28:41 +0900

----- Original Message ----- 
From: "Mark Davis" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "Erik Nordmark" <[email-masked]>
Cc: "Eric Brunner-Williams in Portland Maine" <[email-masked]>; <[email-masked]>
Sent: Wednesday, October 24, 2001 2:09 AM
Subject: Re: [idn] Which lanuages/scripts to reorder? 

> preserve the "unassigned" status: so the future reorderings have to be, in

No problem.

REORDERING  is defined  on  single contiguous block of  assigned characters
within a script block for one version of nameprep.

If  a part of contiguous block of  UNassigned characters of a script block was
 changed into  ASSIGNED in the new nameprep, REORDERING for new nameprep
might contain the REORDERING defined on the changed block part.

REORDERING can be  defined NOT across script blocks, nor,  across
unassigned sub-block and assignedsub-block in a script block at any time.
Since assigned and unassigned code points , in most cases, would not much orrelations
between their character usages, that is very natural consequence,too.

Soobok Lee