Subject: Re: Inputting mixed SC/TC (Re: [idn] A question...)

From: liana Ye <[email-masked]>
Date: Mon, 11 Feb 2002 09:34:23 -0800

Dear John:

I appreciate your understanding of all the problems the IDN 
does not solve. The differences between the WG and the 
CDN group is not a political issue at its base.  WG is trying 
to push TC/SC out of IDN.  The CDN and the others argue
for them to be dealt with in IDN. 

If you consider IDN is a part of DNS, then the CDN group 
say "NO, NO, NO" with force for you and the WG to raise 
eyeballs.   And I hope they have accomplished it with 
all their protests.  

If you consider IDN is above DNS then you are agreeing 
with CDN group. 

If you admit that you are not an expert in dealing with 
Han charaters processing then you should give a good 
and hard study regarding what they have been saying 
all along.

If you are understand what Xiang Deng is saying then 
don't introduce political arguements.  It doesn't help the 
communication at all. 

Regards,

Liana Ye

On Fri, 08 Feb 2002 22:35:33 -0500 John C Klensin <[email-masked]>
writes:
> --On Saturday, 09 February, 2002 10:10 +0800 xiang deng
> <[email-masked]> wrote:
> 
> > Chair's responsibily is not to create conflict but to collect
> > all different argument and coordinate to find a possible
> > solution.
> 
> Under normal IETF procedures, which apply to this Working Group
> as well, the Chair's responsibility, at this stage in the
> process, is to move work along.  That includes, quite
> explicitly, 
> 
> 	* Discouraging the further discussion of topics on which
> 	the WG has already reached a conclusion.
> 	
> 	* Discouraging discussion of problems that are either
> 	out of the WG's scope, or for which realistic solutions
> 	(e.g., solutions that deal with all appropriate cases)
> 	have not been proposed.
> 
> James (and Marc) have, if anything, been more tolerant of
> extended discussions of topics the WG can't fix, or has already
> reached consensus that it won't try to fix, than is normal in
> the IETF.
> 
> Since I have started on this, I am very concerned about the very
> real and serious problems with TC<->SC.  I am also very
> concerned about problems with look-alike characters in
> alphabetic character sets.  I am also very concerned about the
> implications of non-language characters in Unicode that, from my
> point of view, don't belong in the DNS (however coded).  
> 
> I have raised those issues with the WG, discovered that the WG
> isn't interested in the problems, and gone on to worry about
> contexts in which those problems _can_ be solved.   That does
> not make me worry less, nor does it mean that I think the
> problems are unimportant.  But, until and unless I think I have
> new arguments or evidence, I don't bring them up again, and I
> would _expect_ James or Marc to tell me that I'm out of order if
> I did.  I also don't try to convince the WG that those topics
> are important by getting many people --who have not been
> participating in the WG's work-- to send identical notes
> complaining, nor do I circulate notes to other groups in the
> hope that they will somehow overrule the WG.  I avoid doing
> those things for two reasons: they are considered in very bad
> taste in the IETF and they never convince anyone; instead, they
> tend to irritate people enough that they stop listening.
> 
> What is strange, to me, about this discussion is that there is a
> way to delay CDN deployment without impacting Korean or
> Japanese.  It is fairly simple, and does not require that the
> IDN WG agree. And I had assumed it would be obvious to everyone.
> Specifically, the four NICs simply refuse to register such
> names.  You try to convince Singapore and other countries with
> large Chinese-speaking populations to join them in that
> prohibition.  You expand on Erin's document, some of Prof
> Tseng's notes, and other comments and examples which have been
> introduced here to provide other registries cautions about the
> problems they could cause by registering such names.  And you
> try to convince ICANN to prohibit those names in gTLDs.   From
> the perspective of this WG, that is a policy move, made in
> policy forums.  The fact that the WG provides a _mechanism_ for
> encoding CDNs does not imply that any registry need _permit_
> CDNs.  And it seems to me that, at this late date and given the
> overlapping Japanese and Korean issues, that may be your most
> efficient way forward on this particular problem.
> 
> regards,
>      john
> 
>

---

From: John C Klensin <[email-masked]>
Date: Mon, 11 Feb 2002 14:44:41 -0500

--On Monday, 11 February, 2002 09:34 -0800 liana Ye
<[email-masked]> wrote:

> CDN group is not a political issue at its base.

I think I understand that, as do, I think, most of the
participants in the WG.  If I correctly understand what they are
saying when they make comments about political issues, those
comments are based on three things:

(i) Whomever and whatever initiated the process, much of the
IETF reacts very badly to attempts to influence a working group
(or other effort) by letter-writing campaigns that involve a
large number of people sending the same note or notes,
especially when those people have not be participating in the
WG.  Such note-sending campaigns are a political action, not an
engineering one.  They are also considered in such bad taste
that they may damage the position they are trying to advocate:
the reaction is, more or less, "they have completely run out of
technical arguments and  are falling back to creating a lot of
noise; this action indicates that even they do not believe in
their technical arguments".

(ii) We have a continuing communications disconnect in which
some members of the Chinese-speaking community are, apparently,
so astonished that their positions are not immediately accepted
by the WG that they assume that no one [else] in the WG cares
about the impact of this work on names written in Chinese
characters.  Again, that leads to statements being made that
sound to others as if reasoning has been abandoned in favor of
political actions.

(iii) There is also a perception that the CDNC group, and some
others, are ignoring or dismissing the problems with Japanese
and Korean that would be induced by special mappings for Chinese
and, similarly, that many careful explanations of why
specialized TC<->SC handling is not analogous to case-mapping in
alphabetic character sets.  Again, this is a failure to
communicate (perhaps on both sides) at a very fundamental level
that is leading to ill-feeling and beliefs that decision are
being made on a political or emotional basis, rather than with
full understanding and acceptance of the engineering constraints
and tradeoffs.

> WG is trying 

Again, let me try to explain this a little bit differently.
There is a very strong conviction in the IETF that one of the
issues, perhaps the key issue, in Internet protocol design is
scaling.  The scaling concerns are usually though of as applying
to "size of the Internet" issues -- e.g., we try to avoid
standardizing anything that can work only in a restricted
environment or under other "small network" constraints.  But
they also apply to partial solutions: if a particular aspect of
a problem or design activity seems to require a comprehensive
solution, but none is immediately available, the IETF will tend
to avoid a partial solution until the comprehensive one and a
migration strategy are well understood.

The other difficulty with the "consumer confusion" aspects of
the CDN discussion is that many of the comments seem to ignore a
point that has been made repeatedly in the WG: these issues
exist, in some form, for almost every language and script we
have investigated. All of the "equivalent character"
discussions, the similarity of some characters among, e.g.,
"Latin", Cyrillic, and Greek scripts, and related issues point
to considerable possibility of user confusion, misrouted
queries, and so on.  One might even suggest that the CDNC
complaints have been inappropriately focused on Chinese
characters and that the statement should have been "this will
cause problems all over the world, and the IDN effort should
just stop".  The people in the WG who have put in huge amounts
of effort trying to get _something_ to work probably would not
accept that either, but the position would more nearly recognize
the realities of likely IDN usage.

And that brings me to what I think it is the real problem here,
and it a problem that many of us, including those taking the
lead on IDNA (and nameprep, etc) have understood for most of the
time that the WG has been working on these problems.  There are
many problems with the use of words in languages that cannot be
dealt with in the DNS and some other solution will be needed.
Full TC<->SC matching (of all combinations) appears to fall into
that category.  So does the Greek-Cyrillic-Latin problem, and
the ASCII 0<->O and 1<->l problems.  So does a really
satisfactory solution to the "invisible character" problem, some
issues with Arabic/Hebrew/Yiddish vowels, and so on.  Mark's
notion of using presentation mechanisms to identify unusual
character combinations is, I think, quite ingenious but it isn't
part of the DNS either.  Your notions about phonetic mappings
fall into this category as well.  But the two things that all of
these problems and approaches have in common is the potential
for user confusion (or fraud) --some of it serious-- and the
fact that none of them can be dealt with by the simplistic
matching rules of the DNS.  Instead, we need user choices, or
likelihood functions, or fairly serious language recognition, or
knowledge the DNS doesn't have, or heuristics of some [other]
variety.  And the problems aren't limited to Chinese, even
though they manifest themselves differently.

Now I suggest that almost everyone who has been participating
actively in this effort knows that by now.  We know that other,
non-DNS, mechanisms are going to be needed to accomplish
satisfactory internationalization.  Given that, the hard problem
is whether it makes sense to tamper with the DNS at all.  And,
if one is going to do that, what the right set of constraints is
to prevent some of the worst possible damage.   My reading of
the consensus in the WG is that they believe that eliminating
(or postponing) every script that raises problems is not a
solution: we would rapidly have to eliminate all scripts,
possibly including the LDH subset of ASCII.

I think the WG believes that there are three options, and that
most of its members has eliminated the third from consideration:

	(a) Use Nameprep, or something very much like it, which
	deals with mappings that are complete (i.e., there is no
	controversy about them and no need for examination of
	surrounding characters, other context, or language
	information to do them properly) and which excludes
	characters that are problematic in any context in which
	they appear.
	
	(b) Adopt a very strict "identifier" rule in which
	nameprep becomes unnecessary: characters from the
	permitted repertoire can be intermixed in any way,
	without restrictions, comparisons are made on those
	characters only, and the users will somehow learn to
	cope with the results.
	
	(c) Give it up, keep text labels in the DNS restricted
	to the LDH rules, and do _all_ internationalization
	"somewhere else".

> eyeballs.   And I hope they have accomplished it with 

I fear not.  Many of us were fully aware of the problems, and
very concerned about them, before now, and even before Salt Lake
City.    For some of us, the protests have been successful only
in taking up time that should have gone into the "above DNS"
work.  For others, they have distracted from the effort to
carefully review and adjust the details of what nameprep should
be doing in areas where it can constitute a nearly-100%
solution. And, as I have suggested above and elsewhere, they
have led many people to the conclusion (whether correct or
incorrect) that the real goal of the CDNC protest was to disrupt
the work of the WG without offering real alternatives or a way
forward.

> If you consider IDN is above DNS then you are agreeing 

As I trust you know, I have been suggesting that much critical
IDN work will have to be done "above DNS" since before either
the WG or MINC got started.  But to agree on that principle does
not necessarily imply agreement on details of what the WG should
do.

> don't introduce political arguements.  It doesn't help the 

I didn't raise any political arguments in the note of mine that
you quoted, nor did I use that word.  "Policy" issues and
decisions are another matter entirely.   And, while I assume
neither you nor Xiang Deng did so either, I didn't set off the
letter-writing campaign.

regards,
    john

---

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Feb 2002 21:13:43 -0000

John C Klensin writes:
> many careful explanations of why specialized TC<->SC handling is not
> analogous to case-mapping in alphabetic character sets.

Here is James Seng's answer to the TC/SC complaints:

   The user behavior education about domain names should be that domain
   names are identifier, not names. They should enter into the computer
   exactly as they seen it or reference it.

This answer destroys the stated reasons for case-mapping. (Case-mapping
was, in turn, the primary excuse for sticking IDNA into applications.)

This is not just an analogy. It is a literal application of Seng's
principle. Do you think that the principle is incorrect? If so, what
exactly is wrong with it?

I agree with your comment that these issues go beyond Chinese. We have
vowels in Hebrew, accents on capital letters in French, o-umlaut and oe
in German, etc. But I do not agree with your claim that this has all
been settled in previous IDN discussions. Nobody has given a coherent
explanation of _why_ IDNA needs to do certain mappings and not others.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 11 Feb 2002 22:28:55 +0100

--On 2002-02-11 21.13 +0000 "D. J. Bernstein" <[email-masked]> wrote:

> Nobody has given a coherent

Because we do comparison of Unicode strings.

See http://www.unicode.org/unicode/faq/normalization.html

I.e. not our choice (more than choosing to use Unicode).

   paf

---

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Feb 2002 21:48:55 -0000

``Nobody has given a coherent explanation of _why_ IDNA needs to do
certain mappings and not others,'' I wrote.

> I.e. not our choice (more than choosing to use Unicode).

Patrik is lying. The web page that he cites does _not_ require, or even
suggest, that uppercase and lowercase should be considered equivalent.
Unicode normalization is about things like e-with-a-separate-egu-accent
matching the combined e-egu character.

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 11 Feb 2002 22:13:41 +0000

"D. J. Bernstein" <[email-masked]> wrote:

> Patrik is lying.

Unlikely.  He probably just didn't understand your question exactly the
way you intended it.  Language isn't perfect, misunderstandings happen.

> The web page that he cites does _not_ require, or even suggest, that
> uppercase and lowercase should be considered equivalent.

Right.

The reason IDNA does case-folding is to be consistent with the existing
standard for domain names, which says they are case-insensitive.  Not
everyone considers that a sufficient reason, and we've been through
all the arguments on both sides, but that is the reason.  There are
many ways IDNA could have been designed differently and still work, and
this is one of them, but at the end of the day we need to use our best
collective judgement and make a choice.  There does not exist a solution
that everyone would agree is the best possible in all respects.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 11 Feb 2002 18:42:45 -0600

"Adam M. Costello" wrote:

> The reason IDNA does case-folding is to be consistent with the
> existing standard for domain names, which says they are
> case-insensitive.

More specifically, the encoded representation of an uppercase i18n domain
name is different than the encoded representation of a lowercase i18n
domain name, so the only way that case-neutral matching of the encoding
sequence can work is if the i18n domain name is case-neutered before it is
encoded.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "D. J. Bernstein" <[email-masked]>
Date: 12 Feb 2002 02:50:05 -0000

Adam M. Costello writes:
> standard for domain names, which says they are case-insensitive.

What the existing standard actually says is ``domain name comparisons
for all present domain functions are done in a case-insensitive manner,
assuming an ASCII character set, and a high order zero bit.''

Similarly, the Internet mail standards specifically require that bytes
in message headers---including domain names---be interpreted as ASCII
characters.

Complete consistency with the existing standards would mean continuing
to use only bytes 0-127, continuing to interpret those bytes as ASCII,
and continuing to compare names as case-insensitive ASCII names.

But we don't _want_ to follow those rules. We want to see glyphs that
simply aren't available in the ASCII character set.

Of course, we have to maintain INTEROPERABILITY with all strings used
today, so we'll have to continue accepting A-Z and a-z as equivalent.
But there are many possible equivalence rules for non-ASCII strings.
Here are several examples---certainly not a complete list:

   (1) Exactly what software uses now: no equivalences outside ASCII.

   (2) Equivalence of characters that have duplicate glyphs but that
       were kept separate by Unicode for one of the reasons described in
       http://www.unicode.org/unicode/standard/where.

   (3) American-biased equivalences according to Mark Davis's UTR 21,
       which is _not_ part of the Unicode standard.

   (4) German equivalences: for example, o-umlaut equivalent to oe, and
       the German ss equivalent to the two-byte Latin sequence SS, which
       in turn is equivalent to the two-byte Latin sequence ss.

   (5) Hebrew equivalences: for example, aleph-bar equivalent to aleph.

   (6) Various Chinese equivalences for the benefit of Chinese users.

   (7) Some combination of the above.

All of these are INTEROPERABLE with the existing use of ASCII. None of
them are CONSISTENT with the existing standards. One of them, #1, has
the advantage of being by far the easiest to implement---but provides
the most opportunities for confusion and fraud.

What exactly is the rational line between, for example, #3 and #4? For
ASCII characters they both boil down to A-Z matching a-z. Why is #3 a
better extension of the current situation than #4, or #3+#4?

James Seng states that #6 is pointless because ``domain names are
identifier ... should enter into the computer exactly as they seen it or
reference it.'' Under exactly the same principle, #3 and #4 and #5 are
all pointless, so IDNA has no excuse for the costs of #3.

Another approach, allowing the software simplicity of #1 but eliminating
user confusion, is to allow _selected_ non-ASCII characters. We don't
have to map all characters to the selected set; we simply have to make
sure that the selected characters won't be confused by the users. This
neatly dodges the difficulty of defining a broad equivalence rule.

The decisions here have to be based on rational assessments of costs and
benefits. Costello's notion of ``consistency'' is obviously not helpful:
it leads to such huge costs for Chinese users that it has already drawn
objections from _three hundred_ people.

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 11 Feb 2002 21:15:04 -0800

>        which is _not_ part of the Unicode standard.

(a) These are not American-biased equivalences.
(b) It is hardly "my" UTR. I'm the author, but the content is produced
under direction of the UTC and is approved at every stage by the UTC.
(c) UTR 21 was approved by the UTC for incorporation into Unicode 3.2.
Its new status will be reflected in Unicode 3.2, which will be final
very shortly.

Mark
—————

Πόλλ’ ἠπίστατο ἔργα, κακῶς δ’ ἠπίστατο πάντα — Ὁμήρου Μαργίτῃ
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "D. J. Bernstein" <[email-masked]>
To: <[email-masked]>
Sent: Monday, February 11, 2002 18:50
Subject: Re: Inputting mixed SC/TC (Re: [idn] A question...)

> > The reason IDNA does case-folding is to be consistent with the
existing

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 12 Feb 2002 08:24:46 +0100

> sequence can work is if the i18n domain name is case-neutered before it is

Correct. Thanks Eric.

What I should have said.

  paf

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 12 Feb 2002 08:22:16 +0100

> matching the combined e-egu character.

You just asked for "certain mappings and no others". I pointed to the
source for nameprep mappings. You did _not_ say "arguments for case
folding".

I don't understand where you get "lying" from?

I took for granted you knew that case folding is done already in DNS, and
that people has asked for case folding to continue to be valid also for
other scripts.

   paf

---

From: "D. J. Bernstein" <[email-masked]>
Date: 12 Feb 2002 08:37:47 -0000

Patrik writes:
> You just asked for "certain mappings and no others".

Liar. I started by quoting two lines from Klensin saying that TC/SC was
``not analogous to case-mapping.'' I then quoted Seng's TC/SC response,
and said ``This answer destroys the stated reasons for case-mapping.''

> I don't understand where you get "lying" from?

I attribute your behavior to malice because it cannot be adequately
explained by stupidity. It is simply not possible to read my message
without observing that I was talking about case-mapping.

This is not the first time that you have blatantly misrepresented the
statements of people who object to IDNA.

I have heard, by the way, that some of your company's products are not
8-bit-clean: in particular, they fail to handle UTF-8 transparently. Is
this true? Your position on the IESG gives you substantial power to
elevate your company's interests above the interests of the community;
you have a clear conflict of interest if you are making decisions on
proposals that affect your company's products.

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 13 Feb 2002 09:10:30 +0800

Dan,

It is my responsibility to remind you of your language in the group.

Disagreement is okay. Arguments over technical point is typical. But
lets keep it away from ad hominem attack.

-James Seng