Subject: [idn] (bias) summary of reordering discussion

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 24 Oct 2001 00:32:11 +0800

On the questions posted:

1. Efficient - I accepted that reordering produce a shorter ACE string,
sometimes as much as 20%. This means instead of a "zq--zxcvbnmasdfg"
label, I get a "zq--zxcvbnmasd". I do not buy the arguments it is
helpful to naked eye, to memorised or save RAM however.

2. Compression efficient in future since statistic - Lee's counter that
compression "always SHORTER labels than usual". Mathematically, it can
be proven this is wrong (very basic pigeon hole principle).

3. Referencing from established I18N organisation - ISO14651 is deem
inappropriate and I agree with it. No alternative was proposed.

4. Stability of reordering - Lee's countered with the arguments that
reordering tables would never changed. I am not sure if that is possible
but I agree with the assessment that it is possible to design reordering
to be stable. However, I like to see explictive statement in future
draft.

5. Future additional of code points / changes to reordering - Lee's
proposal is a two prefix solution, using prefix as a versioning tag. I
do not like to solve a problem by creating others, especially one which
makes it even more complex. Lee have yet to address the process of how
future additional of code points or changes to reordering could be done.
(IDN WG is not going to exist forever...I dream of finishing our work
one day).

6. Reordering is never ending task - Lee's countered that so is
Nameprep. My thoughts is two wrong dont make one right. (OTOH, Nameprep
which is based on UTC work have explicit principles on how it can be
done. And Nameprep is not subjective to frequency analysis changes which
reordering is)

Of the people participate in the reordering discussion

Martin Duerst - explicit objection
Mark Davis - not very supportive, not explicit
Adam Costello - (no conclusion from comments)
Doug Ewell - explicit objection
Paul Hoffman - explicit objection
Eric Brunner-William - somewhat supportive, not explicit
Kenny Huang - (no conclusion from comments)
James Seng - explicit objection
Karlsson Kent - explicit objection
Erik Nordmark - not very supportive
David Hopwood - (no conclusion from comments)

This is my rough read of the discussion on reordering so far (please
correct me, and apologise in advance, if I am wrong).

(Wearing my co-chair's hat) It is not a vote or even a strawpoll here
obviously but I am trying to get a feel of the group consensus. But if
these discussion is any indication of the group consensus, it does not
indicate very little support for reordering. If there is other comments,
please bring them forward soon. Thanks.

-James Seng

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 23 Oct 2001 09:53:07 -0700

Count me as against the reordering; I think there has been some interesting
work done on it, but I don't believe it is necessary for IDN.

Mark

—————

Δός μοι ποῦ στῶ, καὶ κινῶ τὴν γῆν — Ἀρχιμήδης
[http://www.macchiato.com]

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 02:23:09 +0900

----- Original Message ----- 
From: "James Seng/Personal" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, October 24, 2001 1:32 AM
Subject: [idn] (bias) summary of reordering discussion

> helpful to naked eye, to memorised or save RAM however.

  the 20% is just the mean average. 40% improvement is not rare for
   most frequency words, especiall in the case of hangul 9-letter KRNIC ( 29->16).
   and Tokyo Univ in Han ( 11->7 ).

  i mentioned the limit in mobile device's narrow LCD panel and precious ram/rom resources.  
  Also Good for visually impaired one or childrens and elder ones and for me .
  
  In the case of  sub-ML.ML.com, The sum of saved characters
   are multiplied by 2 with reordering.

  Take this into your considerations.

> be proven this is wrong (very basic pigeon hole principle).
  ALready answered. You may have missed that.
   0.3% hangul labels get longer by 1 character, while 99.6% labels got shorter
    by greater amounts. the net gain is always positive for each length N.

> inappropriate and I agree with it. No alternative was proposed.

  sub-optimal tuning parameters not revealed to outside of ACE. they  need authorities ?  i think it need not.

> (IDN WG is not going to exist forever...I dream of finishing our work

 Already answered. read my recent articles. 
 new reordering support of new script in Tandem with newer nameprep for
 the new script. no new prefix needed if there will be no change 
 in existing stable reordering tables.

 You have misunderstanding of my proposal. 

> done. And Nameprep is not subjective to frequency analysis changes which

 Statistical Frequency is always sub-optimal tuning parameters. better than not.
 What's your position on engineering tuning paramters in various 
 protocols ?, eg. Ethernet packet length, ttl, flag length in tcp/udp etc.... 

Soobok Lee

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 23 Oct 2001 19:41:11 +0200

--On 2001-10-24 00.32 +0800 "James Seng/Personal" <[email-masked]>
wrote:

> Nameprep.

I have not read all details in this long thread of reordering, but, I have
a strong feeling about some principal issues:

o Nameprep mappings is defined by UTC, not IETF.
  -> I am nervous when we in IETF talk about these things
o Versioning in Nameprep is stable for assigned codepoints, as
  the mappings for an assigned codepoint is never changed.
  -> My reading of the reordering is that some reordering rules
     might be added later for assigned codepoints
o Better compression is a good thing, but I don't see reordering
  gaining so much that it is worth it (see bullets above)

So, to conclude, I also oppose reordering.

For more details on versioning, and my view on it, see separate response to
Soobok Lee.

              paf

---

From: David Hopwood <[email-masked]>
Date: Wed, 24 Oct 2001 00:39:24 +0100

-----BEGIN PGP SIGNED MESSAGE-----

> Of the people participate in the reordering discussion
[snip]
> David Hopwood - (no conclusion from comments)

Make that an explicit objection - I think it's far too complicated to
justify the marginal benefits. Also, any changes or additions to
reordering would have disastrous consequences for name comparison, and
Soobok Lee's zq--/uq-- proposal does not fix this.

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 24 Oct 2001 09:44:21 +0800

>   the 20% is just the mean average. 40% improvement is not rare for

The mean is 20% from your data. The result I got using my data is around
12%, which I send you. Either way, I accept it improves efficiency
somewhat.

>     by greater amounts. the net gain is always positive for each
length N.

Using *existing* data. I am talking about data in future if/when we
reach saturation. (No, never say we never going to run out. That is what
we say of IPv4).

>   sub-optimal tuning parameters not revealed to outside of ACE. they
need authorities ?  i think it need not.

Yes, I feel strong enough that we should.

>  in existing stable reordering tables.

Okay. Then who and how you going to define these new reordering table?
What is the process? Do we publish new version of reordering RFC? Or do
we do this as an IANA action (if so, where is the IANA consideration).

Does this explain why I really like to see an authoritive source who
would ensure continouity in the work?

>  You have misunderstanding of my proposal.

If you stop changing your proposal in every email, I think I have a
better chance to keep up to date with it.

>  protocols ?, eg. Ethernet packet length, ttl, flag length in tcp/udp
etc....

My position? What have protocol tuning have to do with code point
tuning?

IETF is a well-known wire-protocol expert group. We exist original to do
so.
But are we code point expert?

-James Seng

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 11:30:42 +0900

----- Original Message ----- 
From: "David Hopwood" <[email-masked]>
> Soobok Lee's zq--/uq-- proposal does not fix this.

I expect you clarify this assertions with detailed examples soon. 

Soobok Lee

---

From: David Hopwood <[email-masked]>
Date: Wed, 24 Oct 2001 04:22:11 +0100

-----BEGIN PGP SIGNED MESSAGE-----

> I expect you clarify this assertions with detailed examples soon.

The problem is this: how does an application that doesn't support the
latest version check whether a zq-- label is equivalent to a uq-- label?

For nameprep without reordering, an old application can perform matching
that will be correct provided that the inputs are nameprepped (i.e.
case folded NFC or NFKC), and this is sufficient for most purposes. In
your proposal, OTOH, it must just give up, because it has no information
about how characters are reordered in uq-- labels. I consider this to be
a fatal flaw in the proposal.

This flaw would not arise if reordering only applied to characters that
are assigned now (i.e. in Unicode 3.1), and new scripts were never
reordered. In that case the zq--/uq-- versioning wouldn't be needed.
However, I still don't think that reordering is worthwhile even in that
case.

- --

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 13:55:22 +0900

----- Original Message ----- 
From: "David Hopwood" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Wednesday, October 24, 2001 12:22 PM
Subject: Re: [idn] (bias) summary of reordering discussion

> a fatal flaw in the proposal.

That's  not new problem in uq--/zq-- scheme, but already in current single zq-- scheme.

let's assume new nameprep with new added script  which contains mapping X-->Y.
The mapping may be one of case folding/NFC/NFKC or others.
Old and New nameprep applications  exchanged  differenetly-encoded two 
X.com ACE labels  and  each  did comparisons on two labels. What happens
with single  zq-- scheme ?  Both applications will fail.
even users of new applications  are left clueless for the failure.

But with zq--/uq-- scheme, the new application would succeed, but old application
should give up and issue warning to users that new nameprep labels come in.
zq--/uq-- scheme provides the clues for the failure/inpasse and teach what to to for fallback
mechanisms. That's the virture of uq--/zq-- scheme.

Even without its intended connection to REORDERING issues,  zq--/uq-- scheme are still
worth to research for its pros/cons  for the sake of  consistent/safer handling of 
unassigned code points.  Let's think about more...

Soobok Lee

> -----BEGIN PGP SIGNATURE-----
> Version: 2.6.3i
> Charset: noconv
> 
> iQEVAwUBO9YzrzkCAxeYt5gVAQHtVAgAmjLhWyYvhTnBRHoIcN55JoTY+cTc6JlB
> +kv6kYJ6x2CHJd4iG/3OyIY5J7ix4tqWbn9Zynhk0F4hleCZT+Yn1l7BRSNAnq7i
> TERrhaj2L+0Aw6cI/S1/OCL9TVEBDKA70eSXQH8nzCejG1Zjy3N5dheKwNTpbCyV
> UxpJ3g6CRsFzV1O7z3pItIS2M7DSDeKQzTYa3rJo42bepYv75oTmDfDEvHKLvcRD
> o5qaEnYWOhrWkdvyPPGiG6eW5D7by4kWxyYsnsKJ4ZpThYqRyB70YnWcuBT+/7J1
> k93JBDsgOg8iqCoXVVAxvhFB6IjN4knwIdkuBWCSmudQUleHYiJ4yg==
> =bt61
> -----END PGP SIGNATURE-----
>

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 24 Oct 2001 07:54:53 +0200

--On 2001-10-24 13.55 +0900 Soobok Lee <[email-masked]> wrote:

> let's assume new nameprep with new added script  which contains mapping
> X-->Y. The mapping may be one of case folding/NFC/NFKC or others.

This is not correct.

Say that the mapping exists in version 3, but not in version 2 of nameprep.

This implies that X is not an assigned codepoint in version 2.

Y can have been assigned in version 2.

An application which read X from a keyboard or something else where the
name "enters the system" have to do nameprep. Two things can now have
happened. Either the application understand version 3 of nameprep, and X is
mapped to Y, or the application does not understand version 3, and then X
is not allowed, as no non-assigned codepoints is allowed.

In neither of these cases X will be transferred over to the other
application.

   paf

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 15:26:34 +0900

----- Original Message ----- 
From: "Patrik Fältström" <[email-masked]>
> In neither of these cases X will be transferred over to the other

Do you mean that old nameprep application cannot encode unassigned code
points into ACE label at all ?  What do you mean by "query" that can 
contains unassigned code points  in your stringprep document ? 
Is it native label  or ACE one ?  Please suggest real applications 
which will process input native labels as "query"  or  "stored strings" .

Soobok Lee

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 24 Oct 2001 08:33:11 +0200

> which will process input native labels as "query"  or  "stored strings" .

As I responded to your question for clearifications in the stringprep
draft, and never got comments or questions back, I thought you understood.
And, as I said in that mail, I am happy to get other words than "query" and
"stored strings".

Please go back to that message of mine which already responds to this and
ask questions on that mail instead.

   paf

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 24 Oct 2001 17:25:26 +0900

----- Original Message ----- 
From: "Patrik Fältström" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "David Hopwood" <[email-masked]>; <[email-masked]>
Sent: Wednesday, October 24, 2001 3:33 PM
Subject: [idn] Re: stringprep and unassigned code points

> ask questions on that mail instead.

Thanks Patrick.

I finished careful reading your clear answers titled with
 "Re: [idn] questions: unassigned code points in nameprep".

What I understand is:

  "saved string" :  generated ACE labels by applications 
                      with known version of nameprep

  "query"        :  received ACE label from other applications
                      the ACE label may have been generated by 
                         unknown version of nameprep

 from this understanding,

 1) I will prefer to use "generated ACE label" and "received ACE label".
 They seem to be  more intuitive.

 2) Current stringprep prohibits old nameprep to encode into ACE labels 
  any unassigned code points. (MUST NOT).

 3) Therefore, Unassigned code points cannot even begin its trip to other applications from the old nameprep applications. 

 4) If we finalize IDN before TAGALOG is added, there is no way to support
  seamlessly support of TAGALOG input in older nameprep applications 
  but for users to upgrade their s/w.
 
 if anyone read and got more  from the answers, add them to this list please.

 Soobok Lee

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 24 Oct 2001 12:46:47 +0200

>                          unknown version of nameprep

Yes. Sounds like a good definition.

>  They seem to be  more intuitive.

Ok.

>   any unassigned code points. (MUST NOT).

Yes.

>  3) Therefore, Unassigned code points cannot even begin its trip to other
> applications from the old nameprep applications. 

Correct.

>   but for users to upgrade their s/w.

I would like to rephrase this. Your wording is very drastic.

When the stringprep/nameprep documents is done, they will reference one
specific version of the Unicode standard. It looks like version 3.1 at the
moment. Applications made for this version will not be able to handle
codepoints which are unassigned in this version of Unicode.

Now, we all know that new versions of the Unicode Standard will come, with
more codepoints assigned.

If I were an application developer I would not hardcode the tables needed
in the application itself. This so when a new version of the
stringprep/nameprep document comes out after a new version of Unicode
tables is done, the user should be able to upgrade his software in a very
easy way. When the software is upgraded, he can use the newly assigned
codepoints.

And, yes, I have already dealt with this problem once as a programmer, and
saw that this was the easiest way of handling versioning in the software.

    paf

---

From: Martin Duerst <[email-masked]>
Date: Thu, 25 Oct 2001 12:54:21 +0900

Hello Patrick,

Just some clarifications below:

>   -> I am nervous when we in IETF talk about these things

The main mappings used by nameprep (casemapping and normalization)
are defined by the UTC. But it is IETF that chooses which mappings
to select (e.g. NFC vs. NFKC), it is IETF that may request other
mappings to be created (NFC was created based on a request by W3C,
an additional file was added to help IETF with case mapping,
a (tentative, implicit) request for TC/SC mappings was turned down).

Also, nameprep contains a few additional mappings and prohibitions
that are motivated by the specific situation of domain names,
without taking these from anywhere.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 25 Oct 2001 12:12:08 +0900

> >  They seem to be  more intuitive.

I'm a bit confused here. This is in the stringprep draft, isn't it?
But stringprep is a general framework, so it should be independent
of ACE or any character encoding or other form of encoding.

Either I'm totally confused (please tell me where I went wrong)
or this has to be changed.

> > applications from the old nameprep applications.

 From the old, single-draft nameprep, I was under the strong impression
that clients may use codepoints for which they don't know whether
they are unassigned or not (they were unassigned when the client
software was released, but may have been assigned in the meantime).

It could be that what Soobok means is that unassigned (even in the last
version) codepoints cannot be entered into registries (because server
have more strict rules), and something that isn't registered won't
get on a billboard,... and therefore won't get to a client and will
therefore not be entered.

>codepoints which are unassigned in this version of Unicode.

The way I was understanding things is that the server side where the
new name is registered has to be upgraded to the new version including
Tagalog (and this may be done by a new table, or whatever). The client,
and in particular the stringprep/nameprep/ace part of it, won't need
any upgrades. Of course, there may be a need for installing a Tagalog
keyboard driver or other components to input Tagalog, but that's
a separate problem.

Regards,  Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 25 Oct 2001 10:08:48 +0200

--On 01-10-25 12.54 +0900 Martin Duerst <[email-masked]> wrote:

> At 19:41 01/10/23 +0200, Patrik F舁tstr仁 wrote:
> 

You are correct, but I am still nervous.

   paf

---

From: "James Seng/Personal" <[email-masked]>
Date: Sun, 28 Oct 2001 13:23:17 +0800

I hope we can do a strawpoll on reordering draft before Salt Lake
meeting. We will announce the process to do so in the next few weeks. In
the meantime, if there is any more discussion, argument for/against on
reordering, please bring them forward.

Thanks.

-James Seng