Subject: RE: [idn] Reality Check

From: "Russ Rolfe" <[email-masked]>
Date: Mon, 16 Jul 2001 11:03:27 -0700

Let me first apologize for not answering sooner.  First my message that
showed up on Saturday was actually sent on Wednesday.  Some kind soul
re-submitted my E-mail and E-mail address to allow me to contribute
again to the list.

Adam M. Costello wrote:

> Are you arguing that ACE is unnecessary?  

Yes

> If so, please explain how email 
> is going to work.  What is supposed to happen when IDN-capable person
A 
> sends a message to IDN-capable person B, who then forwards the message
to 
> IDN-incapable person C, who would like to reply to A?  

To follow my example given.  In the past when people did not have
matching E-mail systems, person C would send the reply back to person B
to forward it on to person A.  Not efficient, but the message got back
to person A.  Thus it would work the same IDN-incapable person C would
send to IDN-capable person B who would forward it to Person A. 

Another solution is if Person A also has a non-IDN address, he can added
it to his signature (just like most of us add a phone number, address,
fax or other way to communicate to our signature) and then person C can
E-mail his reply via the non-IDN address directly to person A.

If it is important enough for person C and A to continue with
communications, they either stay with the non-IDN address or person C
contacts his IT people and suggests that they need to upgrade their
system to allow him to communicate with person A via the IDN-address.

> What is supposed to 
> happen when IDN-capable people send email to mailing lists?  Are they 
> supposed to use their non-IDN address if there are any IDN-incapable 
> subscribers?  How will they know?

Same as mentioned above, during the transition, some people will have
both E-mail addresses, IDN and non-IDN, (by the way it is not unusual
for people to have and manage more than one E-mail address -- example
they have both a private and work related addresses) those who do, would
put their non-IDN and IDN address as part of their signature and those
who chose to reply to them would use the address they are capable of
using.

If it is a true mailing list, all the non-IDN subscribers who wish to
reply to the sender would actually send to the mailing-list and the
mailing list would forward it to the IDN address.  If the mailing list
didn't handle IDN addresses, the person with the IDN-address would know
from the very beginning because he would not get any replies (including
the one that said that he had joined) and would use his non-IDN address
to send to the mailing list.  He may then contact the mailing list owner
and ask why he doesn't handle IDN address and a long with other request,
the mailing list my be updated.

Again this is my whole point, if there is no pressure to upgrade, then
it will either take a long time (10 years or longer as people mention)
or it won't happen at all to move to UTF-8, and IMHO if we go with an
ACE only solution, it will never go away in the long run.  

I see no wrong asking users to take some of the responsibility of moving
to a new technology (i.e. having two addresses, adding a line as part of
the signature pointing to non-IDN address, requesting System-Operators
to upgrade)?  I know that this will be uncomfortable for users in the
beginning, but because of this, there will be more pressure to upgrade
to the new IDN standard and thus the final solution (UTF8) will be
reached sooner.

Regards, Russ
Microsoft -- [email-masked]

-----Original Message-----
From: Adam M. Costello [mailto:[email-masked]] 
Sent: Saturday, July 14, 2001 2:18 PM
To: [email-masked]
Subject: Re: [idn] Reality Check

Russ Rolfe <[email-masked]> wrote:

> ...people will use both IDN and non-IDN address until every they need 
> to communicate can...

Are you arguing that ACE is unnecessary?  If so, please explain how
email is going to work.  What is supposed to happen when IDN-capable
person A sends a message to IDN-capable person B, who then forwards the
message to IDN-incapable person C, who would like to reply to A?  What
is supposed to happen when IDN-capable people send email to mailing
lists?  Are they supposed to use their non-IDN address if there are any
IDN-incapable subscribers?  How will they know?

> I have read a lot of comments on this list that state that ACE is just

> an intermediate step to help us get to a UTF-8 solution.

I don't agree with those comments.  While I can see ACE becoming less
common over time, I wouldn't expect it to completely die out in the
forseeable future.

AMC

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 16 Jul 2001 22:21:27 +0200

> to the new IDN standard and thus the final solution (UTF8) will be

Being a bit pessimistic on the ability for end users to make software
developers and system managers (yes, I was one of them for some 10 years)
to upgrade, I say we should look at the bad handling of MIME which make
client and server software still not handling mime correctly.

But, due to the ascii encoding MIME has (for example in the headers) makes
it possible for people to communicate just because the encodings are 100%
backward compatible with the standard.

I did interoperability tests for MIME during the 90's, and even though we
had minimal requirements, it took software manufacturers _years_ to handle
the things correctly.

It's also the case that I only see "I support UTF-8" when a real solution
with UTF-8 would be something like IDNA, with UTF-8 as the output and not
ACE encoded strings, let's call it INDU. :-)

Because of the fact that some software already do UTF-8 (without nameprep),
and people on this list which should know better say "UTF-8 and not ACE"
when they should point at the actual algorithm used, I am extremely worried
that what I see on this list is IDNA (which include nameprep) or "just send
UTF-8 without nameprep". In reality that is.

Is that what people want?

Is that not what people will get?

If not, explain how the software which today send UTF-8 on the wire will
stop doing that the day we release the IDNU proposal? Who will suffer and
discover what software do nameprep and not?

  paf

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 16 Jul 2001 21:53:43 +0000

Russ Rolfe <[email-masked]> wrote:

> I see no wrong asking users to take some of the responsibility of
> moving to a new technology (i.e. having two addresses, adding a line
> as part of the signature pointing to non-IDN address, requesting
> System-Operators to upgrade)?

In other words, you'd rather create technical difficulties for users in
order to ease the technical burden on engineers (by freeing them from
ACE).  I'll concede that it may be possible to do so, but in my opinion
it's the wrong tradeoff.  Users are far more numerous than engineers,
and less able to deal with technical challenges.  I think it makes more
sense to increase the technical burden on the engineers in order to make
things easier for the users.

> IMHO if we go with an ACE only solution, it will never go away in the
> long run.

If the engineers really hate ACE, they will move toward other mechanisms
at every opportunity, and ACE will become less common over time.  If
the engineers don't mind using ACE, then there's no problem with it
remaining common forever.

In either case, I think the desire of users to see the native characters
will be sufficient pressure to make ACEs very rarely visible to users.
After that, users won't care whether ACE is truly gone or merely hidden.

Patrik Fältström <[email-masked]> wrote:

> I am extremely worried that what I see on this list is IDNA (which
> include nameprep) or "just send UTF-8 without nameprep".

Nitpick:  The problem isn't sending non-nameprep'd UTF-8.  The problem
is receiving UTF-8 and neglecting to run nameprep on it before comparing
it with something.  Even with ACE, after a receiver decodes the ACE
to obtain a Unicode name X, it must verify that X == nameprep(X),
otherwise it's susceptible to spoofing.  But you still have a point:
The ACE-decoding function is likely to have the nameprep check built in,
whereas nameprep is more likely to be forgotten if UTF-8 is received.
Still, I don't think we could stop new protocols from using UTF-8,
and I don't think we'd even want to, so we just have to emphasize the
importance of using nameprep for comparing IDNs.

AMC

---

From: Martin Duerst <[email-masked]>
Date: Tue, 17 Jul 2001 16:48:04 +0900

>ACE encoded strings, let's call it INDU. :-)

Yes. Actually it would be quite easy to convert the IDNA draft to an IDNU
draft, more or less by replacing ACE with UTF-8. Of course a second pass
to smoothen some edges may be required.

>UTF-8 without nameprep". In reality that is.

Just send UTF-8 with nameprep is definitely what's needed, much better
than just send UTF-8 without nameprep.

>discover what software do nameprep and not?

This software will most probably be upgraded. If it's browsers,
I'll definitely contribute by talking to the right people. I'm very
sure browser vendors prefer UTF-8 with nameprep over ACE with nameprep.

Anyway, I think that nameprep is in various ways very important, but
on the other hand, its importance has also been highly overestimated.
So the amount of suffering is much more limited than it may seem.

It is very important to understand that in most contexts, the chance
that a label is changed (except for lowercasing, which was part of
the UTF-8 proposal from the start) by nameprep when somebody takes a
proper domain name from paper and inputs it, is *very* small.
There are some well-known and important exceptions such as 
half-width/full-width
kana, and some implementations of Vietnamese (windows-1258), but for
many areas of the world, nameprep, in its relevant parts, just
conforms what's done anyway.

The reason for this is that this property is *designed* into NFC, and
that most of NFK (but not all of it) is garbage collection and includes
many things that are in some way similar but that the user would never
want to type in (because they really look different from the real thing),
and in addition are difficult to type; a lot of examples can e.g. be
found in blocks U+32xx and U+33xx).

So what we very much need is a very clear definition of which names are
acceptable and which names are not acceptable, and a high-enough
checking/compliance rate (somewhere between 50% and 90%) on the request
side to put enough pressure on the registry side to make that side of
compliance 100%.

What is also quite beneficial is to have some clear guidelines as to
what characters should be mapped to others before lookup and what not.
Half-width/full-width is a typical example. But currently, we are e.g.
forbidding somebody who makes Turkish software to use the case mapping
that a Turkish user would expect, just because we want to avoid
problems if ever a user without an idea about Turkish casing rules
uses that software. The current tendencies of 'better check once too much
than not enough times' (to which I agree in principle) and 'better uniform
and sometimes wrong than according to user expectations' (about which I
have serious doubts) seem to have lead us to overshoot our goals.

Regards,    Martin.

---

From: "Sun Guonian" <[email-masked]>
Date: Tue, 17 Jul 2001 20:28:56 +0800

hello, all,

Last Thursday I had sent this document to [email-masked]
and co-chairs of IDN WG, but I can't see it in IDN group mail list
till now.
I think it is important for the people on the mail list to share it, so
I
send it to the list.

regards,
Sun Guonian

--
*****************************************************
* China Internet Network Information Center (CNNIC) *
*   Sun Guonian                    No.4, S.4 Street *
*                                  Zhong Guan Cun   *
* Phone: 86-10-62553604            Haidian District *
* Email: [email-masked]          Beijing          *
* http://drop.cnnic.net.cn/~sun/   P.O.Box: 100080  *
*****************************************************
* ÖÐ¹ú»¥ÁªÍøÂçÐÅÏ¢ÖÐÐÄ    Ëï¹úÄî                    *
*****************************************************

Internet Draft                                               Sun Guonian
draft-ietf-idn-ace-eval-cn-00.txt                                  CNNIC
Jul 10, 2001
Expires Jan 10, 2002

     Evaluation of various ACEs with existing Chinese Domain Names

Status of this memo

    This document is an Internet-Draft and is in full conformance with
    all provisions of Section 10 of RFC2026.

    Internet-Drafts are working documents of the Internet Engineering
    Task Force (IETF), its areas, and its working groups.  Note that
    other groups may also distribute working documents as
    Internet-Drafts.

    Internet-Drafts are draft documents valid for a maximum of six
    months and may be updated, replaced, or obsoleted by other documents
    at any time.  It is inappropriate to use Internet-Drafts as reference
    material or to cite them other than as "work in progress."

    The list of current Internet-Drafts can be accessed at
    http://www.ietf.org/ietf/1id-abstracts.txt

    The list of Internet-Draft Shadow Directories can be accessed at
    http://www.ietf.org/shadow.html.

Abstract

    The ACE (ASCII Compatible Encoding) Design Team in IDN Working Group
    is selecting an appropriate ACE proposal.  To push forward the work of
    ACE Design Team, this document illustrates the results of various ACEs
    that being applied to Chinese Domain Names.

1. Test data and tools

    These test data were sampled from CNNIC's current registry database, with
    the total number of 100,000.

    Applied ACEs are RACE-03 [RACE], BRACE-00 [BRACE], LACE-01 [LACE],
    UTF-6-00 [UTF6], DUDE-02 [DUDE], AMC-ACE-M-00 [AMCACEM], AltDUDE-01
    [AltDUDE], AMC-ACE-O-00 [AMCACEO], AMC-ACE-R-01 [AMCACER],
    AMC-ACE-V-00 [AMCACEV], AMC-ACE-W-00 [AMCACEW], MACE-01 [MACE] and
    LDUDE-00[LDUDE].

2. Result of each ACE

    HANZI_len is the characters in the Chinese string before UCS2-ACE coding,
    max_len is the maximum length that a ACE-coded string can reach, min_len
    is the minimum length that a ACE-coded string can reach, aver_len is the
    average length of all ACE-coded strings converted from the Chinese string
    with the same HANZI_len-size.
 
  2.1 AMC-ACE-M

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            9        9        9.0000  
    2            13       10       12.7385 
    3            17       11       15.8155 
    4            20       12       18.8141 
    5            23       13       21.7565 
    6            26       14       24.6305 
    7            29       16       27.5662 
    8            33       16       30.4084 
    9            35       30       33.3293 
    10           39       27       36.1929 
    11           41       36       39.1082 
    12           45       34       42.0063 
    13           48       42       44.8869 
    14           50       43       47.8597 
    15           53       48       50.7955 
    16           57       51       53.8091 
    17           59       53       56.5031 
    18           62       57       59.7467 
    19           65       61       62.5106 
    20           68       63       65.6471 

  2.2 AMC-ACE-O

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            9        9        9.0000  
    2            13       10       12.7401 
    3            17       11       16.2954 
    4            21       12       19.8272 
    5            25       13       23.3622 
    6            29       14       26.6862 
    7            32       16       30.3160 
    8            36       16       33.3543 
    9            40       30       36.9435 
    10           44       27       40.1309 
    11           48       36       43.5358 
    12           51       34       46.9934 
    13           55       41       50.1921 
    14           59       31       53.6718 
    15           62       48       56.4908 
    16           66       51       60.5728 
    17           68       53       63.0881 
    18           73       57       66.3467 
    19           76       61       70.4894 
    20           79       67       73.4706 

  2.3 AMC-ACE-R

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            9        9        9.0000  
    2            13       10       12.7401 
    3            17       11       16.4921 
    4            21       12       20.3006 
    5            25       13       24.0866 
    6            29       14       27.7128 
    7            33       16       31.5339 
    8            37       16       34.8798 
    9            41       31       38.7328 
    10           45       27       42.2981 
    11           49       40       45.9261 
    12           53       34       49.7214 
    13           57       45       53.1325 
    14           61       43       56.8372 
    15           65       52       60.1350 
    16           69       56       64.5534 
    17           73       53       67.0566 
    18           75       63       71.5600
    19           80       67       74.8936
    20           84       72       79.2647 

  2.4 AMC_ACE_V

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            8        8        8.0000  
    2            12       10       10.9848 
    3            16       11       13.9613 
    4            20       12       16.9441 
    5            23       13       19.9398 
    6            25       14       22.9069 
    7            28       16       25.9121 
    8            32       16       28.8512 
    9            35       29       31.8418 
    10           37       27       34.8368 
    11           41       35       37.7769 
    12           44       35       40.8079 
    13           47       40       43.7668 
    14           49       32       46.7032 
    15           52       45       49.6626 
    16           56       50       52.8026 
    17           58       53       55.6604 
    18           65       57       58.9733 
    19           64       60       61.8085 
    20           67       63       64.8235 

  2.5 AMC_ACE_W

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            8        8        8.0000  
    2            12       10       10.9848 
    3            16       11       13.9790 
    4            19       12       16.9746 
    5            24       13       19.9955 
    6            27       14       22.9755 
    7            29       16       26.0036 
    8            34       16       28.9908 
    9            37       29       32.0108 
    10           41       27       35.0201 
    11           46       36       38.0537 
    12           49       28       41.0596 
    13           52       42       44.1294 
    14           56       44       47.1385 
    15           56       47       50.1472 
    16           62       51       53.3204 
    17           66       54       56.3082 
    18           72       58       60.1333 
    19           69       61       62.7234 
    20           71       64       65.2059 

  2.6 AltDUDE

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            8        8        8.0000  
    2            12       9        11.7401 
    3            16       10       15.4679 
    4            20       11       19.2468 
    5            24       12       23.0600 
    6            28       13       26.7517 
    7            32       15       30.6092 
    8            36       15       34.1284 
    9            40       30       37.9871 
    10           44       29       41.5955 
    11           48       38       45.2093 
    12           52       26       49.0893 
    13           56       45       52.7266 
    14           60       49       56.3480 
    15           64       53       59.7935 
    16           68       57       63.8964 
    17           72       61       67.2956 
    18           76       65       71.6400 
    19           79       71       75.3404 
    20           83       72       78.3529 

  2.7 BRACE

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            8        8        8.0000  
    2            11       9        10.9402 
    3            14       11       13.9295 
    4            18       12       17.8337 
    5            21       14       20.9788 
    6            24       15       23.9537 
    7            27       16       26.9647 
    8            30       18       29.9676 
    9            34       31       33.8942 
    10           37       29       36.9766 
    11           40       36       39.9731 
    12           43       36       42.9665 
    13           46       43       45.9814 
    14           50       43       49.9406 
    15           53       51       52.9877 
    16           56       54       55.9806 
    17           59       52       58.8868 
    18           62       59       61.9200 
    19           66       65       65.9787 
    20           69       69       69.0000 

  2.8 DUDE

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            8        8        8.0000  
    2            12       9        11.7401 
    3            16       10       15.4679 
    4            20       11       19.2468 
    5            24       12       23.0600 
    6            28       13       26.7517 
    7            32       15       30.6092 
    8            36       15       34.1284 
    9            40       30       37.9871 
    10           44       29       41.5955 
    11           48       38       45.2093 
    12           52       26       49.0893 
    13           56       45       52.7266 
    14           60       49       56.3480 
    15           64       53       59.7935 
    16           68       57       63.8964 
    17           72       61       67.2956 
    18           76       65       71.6400 
    19           79       71       75.3404 
    20           83       72       78.3529 

  2.9 LACE

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            9        9        9.0000  
    2            12       11       11.9642 
    3            16       12       15.9868 
    4            19       14       18.9913 
    5            22       16       21.9985 
    6            25       17       24.9972 
    7            28       19       27.9995 
    8            32       20       31.9990 
    9            35       35       35.0000 
    10           38       36       37.9994 
    11           41       41       41.0000 
    12           44       33       43.9934 
    13           48       48       48.0000 
    14           51       51       51.0000 
    15           54       54       54.0000 
    16           57       57       57.0000 
    17           60       60       60.0000 
    18           64       64       64.0000 
    19           67       67       67.0000 
    20           70       70       70.0000 

  2.10 LDUDE

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            8        8        8.0000  
    2            12       9        11.3573 
    3            16       10       14.4717 
    4            20       11       17.5182 
    5            24       12       20.5703 
    6            28       13       23.5265 
    7            32       15       26.6787 
    8            36       15       29.6551 
    9            40       21       32.7320 
    10           44       23       35.9557 
    11           48       28       38.9694 
    12           51       26       42.1635 
    13           55       34       45.1844 
    14           60       37       48.4613 
    15           63       40       51.3865 
    16           66       43       54.9903 
    17           70       45       58.2767 
    18           75       50       61.8133 
    19           76       51       62.7234 
    20           78       57       67.9706 

  2.11 MACE

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            8        7        7.9850  
    2            12       9        10.9905 
    3            16       10       13.9855 
    4            19       11       16.9912 
    5            24       12       19.9935 
    6            26       14       22.9938 
    7            29       15       26.0009 
    8            34       15       29.0167 
    9            37       29       32.0114 
    10           41       30       35.0183 
    11           46       37       38.0455 
    12           49       29       41.0432 
    13           52       42       44.0883 
    14           56       45       47.1493 
    15           56       48       50.1718 
    16           62       53       53.3042 
    17           66       55       56.3837 
    18           70       59       60.1467 
    19           69       62       62.7234 
    20           71       64       65.2941 

  2.12 RACE

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            9        8        8.0014  
    2            12       9        11.8931 
    3            16       11       15.9835 
    4            19       12       18.9981 
    5            22       14       21.9994 
    6            25       16       24.9987 
    7            28       17       27.9993 
    8            32       19       31.9989 
    9            35       35       35.0000 
    10           38       38       38.0000 
    11           41       41       41.0000 
    12           44       44       44.0000 
    13           48       48       48.0000 
    14           51       51       51.0000 
    15           54       54       54.0000 
    16           57       57       57.0000 
    17           60       60       60.0000 
    18           64       64       64.0000 
    19           67       67       67.0000 
    20           70       70       70.0000 

  2.13 UTF-6

    HANZI_len    max_len  min_len  aver_len
    -----------  -------- -------- --------
    1            8        8        8.0000
    2            12       9        11.9582
    3            16       10       15.9889
    4            20       11       19.9985
    5            24       17       23.9995
    6            28       13       27.9984
    7            32       21       31.9993
    8            36       23       35.9989
    9            40       40       40.0000
    10           44       44       44.0000
    11           48       48       48.0000
    12           52       52       52.0000
    13           56       56       56.0000
    14           60       60       60.0000
    15           64       64       64.0000
    16           68       68       68.0000
    17           72       72       72.0000
    18           76       76       76.0000
    19           80       80       80.0000
    20           84       84       84.0000

3. Summary

    ACE_name   max_cstring
    AMC-ACE-M  18
    AMC-ACE-O  15
    AMC-ACE-R  14
    AMC_ACE_V  17
    AMC_ACE_W  16
    AltDUDE    14
    BRACE      18
    DUDE       14
    LACE       17
    LDUDE      15
    MACE       16
    RACE       17
    UTF-6      14

    The max_cstring is the maximum length of Chinese Domain Name when
    max_len is less than 64.

    For Chinese Domain Name, the max_len is most significant.

4. References

    [RFC1035]  "DOMAIN NAMES - IMPLEMENTATION AND SPECIFICATION",
               RFC1034, Nov 1987, P. Mockapetris
    [RACE]     "RACE: Row-based ASCII Compatible Encoding for IDN",
               draft-ietf-idn-race-03.txt, Nov 2000, P Hoffman
    [BRACE]    "BRACE: Bi-mode Row-based ASCII-Compatible Encoding for IDN
               version 0.1.2"
               draft-ietf-idn-brace-00.txti, Sep 2000, A Costello
    [LACE]     "LACE: Length-based ASCII Compatible Encoding for IDN"
               draft-ietf-idn-lace-01.txt, Jan 2001, M Davis, P Hoffman
    [UTF6]     "UTF-6 - Yet Another ASCII-Compatible Encoding for IDN"
               draft-ietf-idn-utf6-00, Nov 2000, M Welter, B Spolarich
    [DUDE]     "Differential Unicode Domain Encoding (DUDE)"
               draft-ietf-idn-dude-02.txt, Jun 2001, M Welter, B Spolarich,
	       A Costello
    [AMCACEM]  "AMC-ACE-M version 0.1.0"
               draft-ietf-idn-amc-ace-m-00.txt, Feb 2001, A Costello
    [AltDUDE]  "AltDUDE version 0.0.2"
               draft-ietf-idn-altdude-00.txt, Mar 2001, A Costello
    [AMCACEO]  "AMC-ACE-O version 0.0.3"
               draft-ietf-idn-amc-ace-o-00.txt, Mar 2001, A Costello
    [AMCACER]  "AMC-ACE-R version 0.2.1"
               draft-ietf-idn-amc-ace-r-01.txt, May 2001, A Costello
    [AMCACEV]  "AMC-ACE-V version 0.1.0"
               draft-ietf-idn-amc-ace-v-00.txt, May 2001, A Costello
    [AMCACEW]  "AMC-ACE-W version 0.1.0"
               draft-ietf-idn-amc-ace-w-00.txt, May 2001, A Costello
    [MACE]     "MACE: Modal ASCII Compatible Encoding for IDN"
               draft-ietf-idn-mace-00.txt, Jun 2001, M Ishisone, Y Yoneya
    [LDUDE]    "Improving ACE using code point reordering v0.9"
               draft-ietf-idn-lsb-ace-00.txt, Jun 2001, Soobok Lee
    [MDNKIT]   "Multilingual Domain Name tool Kit",
               http://www.nic.ad.jp/jp/research/idn/mdnkit/download/

5. Acknowledgements

    CNNIC Chinese Registry Service Department provided registered
    Chinese Domain Names.
    XiaoDong LEE, [email-masked]
    Wang Yanfeng, [email-masked]
    Deng Xiang, [email-masked]
    
6. Author's Address

    Sun Guonian
    China Internet Network Information Center
    No.4, South 4th street, Zhongguancun,
    Haidian District, Beijing,
    China  100080
    [email-masked]

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Jul 2001 09:32:52 -0400

> Just send UTF-8 with nameprep is definitely what's needed, 

I find that statement completely irresponsible.  It completely ignores 
the need for deployed applications to keep working through this transition.

Keith

---

From: Martin Duerst <[email-masked]>
Date: Tue, 17 Jul 2001 23:47:22 +0900

>the need for deployed applications to keep working through this transition.

Hello Keith,

If you can explain to me why ACE should be better than NAT,
and why deployed applications should stop working with the
domain names they always have worked with until now,
I'll accept your comments.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 17 Jul 2001 23:44:30 +0900

>     UTF-6      14

Not an ACE, but probably worth mentioning anyway:

UTF-8       21

With the additional advantage that it's much easier to explain
to anybody why some names are too long and others are short enough.

Regards,   Martin.

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Jul 2001 11:59:24 -0400

> I'll accept your comments.

ACE and NAT are completely orthogonal, so comparisons are almost meaningless.
Even as analogy it doesn't fit - e.g. ACE is end-to-end and applications that
need to know about it are aware of it, while NAT corrupts data in midstream
without the endpoints' awareness or consent.

As to the second question, the burden is on you to explain why widely used
applications should break because of a few folks' instistence on using UTF-8.  
To break these applications for the sake of preserving compatibility for a 
relatively few people who have been violating the protocols is simply insane.  
It's more far important to maintain operation of services that are standardized 
and widespread, than it is to maintain operation of services that are nonstandard 
and in limited use.

Keith

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 17 Jul 2001 08:50:26 -0700

>I'll accept your comments.

I'll do it for Keith:

IDNA lets applications use the domain names that they have always 
worked with just as they are. Your proposal it to break all current 
applications by requiring the application to both do UTF-8 (which 
many don't do) and to do nameprep (which those that currently do 
UTF-8 don't do).

How did you get these two mixed up?

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Jul 2001 12:07:30 -0400

> to anybody why some names are too long and others are short enough.

is it really?  would an average Greek, Russian, Chinese, Japanese,
or Korean speaker really understand the mapping between characters 
and length of UTF-8 representation any better than the mapping 
between characters and length of (say) DUDE representation?

seems like if the limits are specified in any other form than 
"maximum number of symbols" then users will find this mysterious.
and for many languages UTF-8 requires different octet lengths for 
different characters.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Jul 2001 12:24:53 -0400

> As I have remarked earlier, we don't need this WG to allow
> people to create random domain name labels. ACE is not
> 'domain names just as they are'. It's a clear layer violation.

That's simply BS.  ACE is an *encoding" just like UTF-8 is an *encoding*.

> You can't do any serious kind of processing (e.g. sorting,
> searching,...) on ACEd domain names.

You can do comparison of names with ACE.  Sorted ACE names are almost
as useful to humans as sorted names in UTF-8, which is to say - not very.
It's true that you cannot search for substrings of an ACE name as
easily as you can do so in UTF-8, but this hardly compels use of UTF-8
where it will break existing applications.

> >Your proposal it to break all current applications by requiring the
> >application to both do UTF-8 (which many don't do)
> 
> There are mainly three cases:
> - Those that just pass data along. Most implementations nowadays are
>    8-bit transparent, for most others, it's easy to clean them.
> - Those that do some actual character processing, but not display.
>    These will require work, but considerably less with UTF-8 than
>    with ACE.
> - Those that do display. Again, less work with UTF-8
>    than with ACE. And more work already done.

For the latter two, the burden in supporting ACE in addition to supporting
UTF-8 is small.  The application has to call API routines in either case -
the differences between ACE and UTF-8 are mostly hidden under those routines.

The other thing that you're leaving out is the additional complexity that
would be required to add UTF-8 negotiation to existing applications that
could not be expected to deal with UTF-8 names.

> >and to do nameprep (which those that currently do UTF-8 don't do).
> 
> Again, nameprep only is easier than nameprep+ACE.

It requires more code, but the application writer doesn't really notice
the difference.

Keith

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Jul 2001 01:12:29 +0900

At 08:50 01/07/17 -0700, Paul Hoffman / IMC wrote:
>IDNA lets applications use the domain names that they have always worked 
>with just as they are.

Again, nameprep only is easier than nameprep+ACE.
And as I have explained to Patrick, while on the registration
side, 100% nameprep and nothing less is extremely important,
on the application side, it's not really that important.

Regards,    Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 17 Jul 2001 19:04:33 +0200

> on the application side, it's not really that important.

The day an online stock trader doesn't get his bid in to the broker because
he typed the wrong version of what according to nameprep is the wrong
character, you loose.

  paf

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Jul 2001 01:45:45 +0900

At 12:07 01/07/17 -0400, Keith Moore wrote:
>is it really?

Very significantly, yes.

>would an average Greek, Russian, Chinese, Japanese,

Greek and Russian may be a bit easier for some of the ACEs,
but for Chinese, Japanese, and Korean, you either just use
some program and believe the output, whatever it may be, or
you go look up the numbers and do the calculations by hand,
which may take you something like 20-30min.

All these cases are very easy with UTF-8.

>"maximum number of symbols" then users will find this mysterious.

Well, because different users may even see different numbers
of symbols, there will always stay some bit of mistery.

>and for many languages UTF-8 requires different octet lengths for

Yes. But it's something that isn't too difficult to know, if
you want to know. The toughest case is probably Vietnamese,
because you have ASCII (1 byte), Latin-1/Latin extended A
(2 bytes) and Latin Extended Additional (3 bytes).

Regards,   Martin.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 17 Jul 2001 22:43:39 +0000

Martin Duerst <[email-masked]> wrote:

> I'm very sure browser vendors prefer UTF-8 with nameprep over ACE with
> nameprep.

If that's true, then they can opt to rely on updated HTML and HTTP
standards that support UTF-8 domain names (using nameprep to compare
them), deploy browsers and servers that conform to the new standards,
and not bother with ACE at all.  (Of course, browsers that include mail
user agents would also need to rely on updated standards for message
headers and SMTP.)  The only time they would need to use ACE would be
to use a legacy resolver, but if they can convince the operating system
providers to add a UTF-8 API to the resolver then they don't have to use
ACE anywhere.

This argument applies to any application or protocol.  ACE is there
if you want to interoperate with 7-bit protocols/formats, but if you
don't want to, then don't, just go ahead and stick to your new 8-bit
protocols.

There are obviously people here who think ACE is important, and
people who think nameprep is important, and people who think UTF-8 in
application protocols is important, and people who think UTF-8 in DNS
is important.  There are obviously people who think other people are
wasting their time on some of these.

Perhaps we should stop trying to convince people that what they're
working on is a waste, and just get out of each other's way.  We can
end up with a standard for each of the above four items, and they need
not all be ready at the same moment.  As an implementor or application
protocol designer, if you think one of the standards is worthless to
you, then don't use it.  (Except for nameprep--I think everyone needs to
use that one.)

AMC

P.S.

> Not an ACE, but probably worth mentioning anyway:

UTF-8 has a slight edge for Han (AMC-ACE-Z can almost always fit at
least 19 Han characters) and Latin, but it's terrible for Indian scripts
(AMC-ACE-Z can fit about twice as many characters) and pretty bad for
other non-Latin alphabetic scripts (AMC-ACE-Z can fit about 1.5 to 1.7
times as many characters).

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Jul 2001 08:19:09 +0900

>he typed the wrong version of what according to nameprep is the wrong

This is just spreading FUD. No halfway serious online trader types
in the broker address every time he trades. For that, there are
things such as bookmarks and autocompletion.

Also, if it were true, there would be a lot of law suits related to
this because currently in Japan, only half-width ASCII (and not the
full-width equivalents) work for domain names. And because the 'fi'
ligature doesn't work with domain names (but it will with nameprep
as currently defined).

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Jul 2001 08:33:23 +0900

At 12:24 01/07/17 -0400, Keith Moore wrote:
>ACE is an *encoding" just like UTF-8 is an *encoding*.

Sorry, no. ACE can be reapplied to ACE (the drafts of course prohibit this),
but please try to apply UTF-8 to UTF-8. Also, when the term 'UTF-5' became
known for my original proposal, there was strong pushback in the Unicode
community, and similar for UTF-7, which again maps 'more than ASCII' to
'ASCII'.

Also, ACE is used piecemeal. Maybe the lhs of an email address will
use a different ACE than the rhs, or a different prefix. Maybe it
will restart at dots, or not. There is no definition of how to apply
ACE to a text file. Content-Type: text/plain; charset=XXXACE cannot
be use. If you try to get down that path, you get the horrible
dependencies between mail header syntax and the =? ? ? ?= encoding.

Converting a file from Latin-1 to UTF-8 (or chose your encoding)
is independent of the content of the file. Converting ACE to
UTF-8 needs to parse the file (i.e. go to a different layer).

>could not be expected to deal with UTF-8 names.

The main application that really needs this is SMTP. Most others don't.
And those that do would need it for other purposes, anyway.

Regards,    Martin.

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Jul 2001 20:45:03 -0400

> working on is a waste, and just get out of each other's way.  

that sounds like a fine strategy. 

It's painfully clear that we need an ACE format for IDNs to be used in 
legacy appliactions, and that queries for ACE-encoded IDNs have to work.

If people can make a good case that we also need to support queries that 
use UTF-8 (or some other general purpose encoding of 10646), more power 
to them.  I won't swear that the case cannot be made, it's just that when
I did my own analysis of the effort involved (with the requirement that
ACE queries produce consistent results with native queries), it looked 
to be a fair amount of effort for a marginal amount of gain.   

The one thing that I don't want is for the folks who want UTF-8 on the
wire to block progress on deployment of an ACE solution.  Because it's
clear that we need to move forward on this, and that we will need the 
ACE solution in any case.

We can probably afford to have two ways of looking up IDNs.  We 
cannot afford a tug-of-war in this group that delays adoption of
any solution.

--

What I'd really like to see us work on is the transcribability problem.
This is a problem that all of the proposals have in common - there
are still too many similar glyphs with different code points that are not
folded by nameprep.  I see this as the biggest remaining problem that
must be solved before we can standardize an ACE lookup scheme for IDNs.
(even if we standardize an alternate one later that uses UTF-8 or some
other encoding)

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Jul 2001 20:36:00 -0400

> but please try to apply UTF-8 to UTF-8. 

I fail to see how this is relevant.

> Also, ACE is used piecemeal. 

That's true.  ACE is an encoding, but is not a general purpose encoding.
It's designed for a narrow but important purpose, and it's sufficiently ugly 
that you really would not want to use it except when necessary.  But there 
are cases where it is necessary. 

> The main application that really needs this is SMTP. Most others don't.

that's simply false.  any deployed application that exchanges DNS names
as protocol elements needs ACE if those deployed codes are expected to
interoperate (and exchange DNS names) with new codes that use IDNs. 

true, there are sometimes other ways to solve the problem than ACE - 
explicit negotiation for instance.  But this isn't feasible with all
applications, and it's very unattractive for most of those in which it is
feasible.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 17 Jul 2001 21:48:54 -0500

> legacy appliactions, and that queries for ACE-encoded IDNs have to

Yes.

> If people can make a good case that we also need to support queries
> that use UTF-8 (or some other general purpose encoding of 10646),
> more power to them.

Future applications will want to be totally internationalized from the
ground up, without any legacy crap in the soup. Ease-of-deployment in
programming and operating environments where UTF-8 is easily accessed.
Overall "servicability" of DNS, where problems with UTF-8 will be much
simpler to analyze than similar problems with ACE. More to the point,
however, it does not need to be "justified" if there is a sufficient
amount of interest from knowledgable people, which there is.

And while it would be nice to have the just-send-UTF8 camp recognize the
merits of ACE, it would also be nice to have the ACE-only camp recognize
that many of their peers find some measurable amount of merit in UTF-8.

> We can probably afford to have two ways of looking up IDNs.

There have been many proposals for a dual-mode approach, and this should
illustrate that many folks recognize the merits of both camps already. And
if it is done right, it doesn't suffer from compromise syndrome, since
folks can deploy the one they wish without suffering any real penalties
(you can only-send-ACE or just-send-UTF8, as you choose, although it would
be self-defeating to ignore half the benefits). And we can do it cheaply.

If we can at least agree to work on a dual-mode approach -- even if we
think the other half is nuts -- we will get the kinks worked out that much
faster and to everyone's benefit.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Jul 2001 23:14:18 -0400

> ground up, without any legacy crap in the soup. 

No doubt about that at all.   I have some doubts about whether it will
really matter if they get it, but I don't doubt that people want it.

(I don't think it affects such applications in any significant way 
if they use ACE only under the hood in DNS queries. After all, the format 
of DNS names within queries is *already* very different than that used by 
applications, without the dots, with each label prefixed by a byte count,
with label compression, etc.  But very few apps need to care about this.)

> More to the point,

Well, it still needs a cost-vs-benefit justification.  But perhaps those
knowledgable people can provide one.  

> that many of their peers find some measurable amount of merit in UTF-8.

I doubt that anyone fails to see the merit in just-send-UTF-8 within 
applications, under some conditions.  It's the notion that it's always 
okay for applications to just-send-UTF-8 that seems to have little merit.

> folks can deploy the one they wish without suffering any real penalties

That seems like a stretch.  But perhaps the proponents should go off and
work out the details for themselves.

I suggest some essential criteria for a dual-mode solution: 

- ACE lookups have to work from legacy apps for all ACE-encoded IDNs, 
  without additional support on the part of applications, hosts, caches, 
  resolvers, or servers.  

- Native lookups from IDN-aware apps have to work for all Native IDNs, 
  though it might be acceptable to impose some new requirements on 
  those applications, their hosts, caches, resolvers, etc.  

- ACE lookups and Native lookups have to produce results that are
  consistent with one another in both results and failure rates.

> faster and to everyone's benefit.

Right.  But if we do this, working out the details of the native approach
should not hold back the ACE solution.

Keith

p.s. I still think the argument about what to send over the wire in DNS
is specious.  I get the strong impression that people who are arguing
in favor of using UTF-8 over the wire in DNS are really wanting to insist
that we use UTF-8 over the wire, immediately, in all applications - or
at least that this is the default behavior for all applications even if
a few applications make an exception to this rule.  I don't think this is
feasable, and even if it were, it's not within this WG's purview.

But if the proponents of UTF-8 over the wire in DNS can come up with a
sound proposal and make a good case for it, we ought to seriously consider
it.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Jul 2001 14:10:04 +0900

>cannot afford a tug-of-war in this group that delays adoption of

Thanks, Keith!
This is an excellent proposal for a forward-looking compromise.

>(even if we standardize an alternate one later that uses UTF-8 or some

Very good point. Do you have any particular kind of similarities
in mind, or can you give some examples? Or do you have some kind
of principles or tests in mind that should be applied? Or any
particular kind of procedure that we should follow?

Also, there are some things that get folded in nameprep that should not.
This is not so important if nameprep is mainly seen as the absolute
check for registrations. But if it is supposed to be strictly applied
on every occasion, in particular every time a name is resolved,
as e.g. Patrick is describing it, these foldings may lead to people
believing that these characters are acceptable in a domain name the
same way an upper-case character is.

What should be eliminated are:

1) Characters that are very clearly visually distinct from the ones
    they are mapped to. (for obvious reasons)
2) Characters that completely map to ASCII-only characters
    (to make sure that current applications and applications doing
    nameprep behave the same way for ASCII-only).

An example of a character for which both of the above apply is
U+2460 CIRCLED DIGIT ONE. This is a simple digit 'one' in a circle.
Obviously, everybody can immediately see that it's different from
just a '1'. Also, if it's mapped to '1' by nameprep applications,
these applications will behave differently from applications not using
nameprep (i.e. everything out there now).

There are quite a few of these, all identified with <circle> (or
similar) in Unicode. There are also quite a few to which only
1) applies (circles with things other than ASCII letters in them,...)
and a few to which only 2) applies (e.g. fi, ffi,... ligatures).

I propose that all of them should be rejected, for the reasons above,
rather than folded.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Jul 2001 14:05:41 +0900

>I fail to see how this is relevant.

Architecturally, because it's folding one layer back onto the same layer.
Operationally, because some implementations will get it wrong, and we will
see ACE over ACE the same way we have seen Mime over Mime.

>are cases where it is necessary.

What I see comming more and more is that this 'narrow but important'
argument will be applied over and over: It's not worth to do anything
to upgrade protocol A because the DNS is ACE, then it's not worth to
upgrade protocol B because protocol A uses ACE, and so on.

If for example the IDNA draft could say that resolvers should provide
a way (a new API) to resolve UTF-8 directly, and could say that UTF-8
(nameprepped) is accepted as an alternate encoding for zone files and
is preferred where possible, that would help quite a bit to avoid forcing
protocols and operations into ACE unnecessarily.

What do you think of it?

Similarly for the DNS protocol: Send UTF-8, and if it's not found,
try again with ACE. An existing DNS server with no IDN names (in
whatever encoding) that behaves in any other way than returning
'unknown' for UTF-8 is definitely broken and has to be fixed anyway
(and I'm still really and seriously looking for somebody to give
an actual report on such a case).

I agree with you that when thinking about whether it's worth to
do a dual implementation in a DNS server, it looks like a pure
overhead, and just looking at DNS, it is. But overall and in the
long run, it's not an overhead, it's an investment. The past is
important, but the future is more important than the past.

Regards,   Martin.

---

From: Keith Moore <[email-masked]>
Date: Wed, 18 Jul 2001 02:28:13 -0400

> in mind, or can you give some examples? 

this is well outside of my area of expertise, but several people have cited 
examples that haven't been refuted.  

> Or do you have some kind

one idea (which I don't particularly like) is to assume that all characters
within a single label are from a single langauge, and if the same glyph
maps to different code points (indicating characters from differnet languages)
then you resolve the ambiguity by using the code point that creates the
fewest number of language changes.  I won't even begin to list the problems
with this; I mention it only because I think that this approximates the
behavior that is most natural for human beings.

another idea (which I likely only slightly better) is to have two kinds
of ACE - one (using nameprep) for name-to-whaterver lookups and another 
(not using nameprep) for IDNs returned in PTR records.  That way, 
nameprep can be more agressive about folding together codepoints with
similar glyphs, because it doesn't affect names *returned* from DNS.
(unfortunately, you still have to nameprep names returned in CNAME 
records, NS records, MX records, etc.)

actually, the ability to do lookups without having to force the names
to suffer nameprep munging - in other words, the ability to do 
name comparisons without requiring that the representations be identical -
might be the most compelling reason to eventually deploy a native
IDN query interface to DNS.  

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 18 Jul 2001 02:11:48 -0400

> > > Sorry, no. ACE can be reapplied to ACE (the drafts of course prohibit 
> > > this), but please try to apply UTF-8 to UTF-8.
> >

I'm not convinced; the cases aren't even similar.  MIME was designed
to be applied recursively, so it's hardly surprising that it's
used in this way.  ACE is designed to be easily distinguished from non-ACE
forms, so it seems unlikely that implementors will accidentally re-encode
an ACE very often.  The closest thing to ACE is RFC 2047 encodings,
and I've never seen one of those recursively applied except as an example
of what not to do.  

Also, I assume that the WG will produce one or more high-quality reference 
implementations of the chosen ACE - actually I'd regard that as essential.  
So it won't often be necessary for folks to implement it from scratch.

> upgrade protocol B because protocol A uses ACE, and so on.

I don't know where you're seeing that argument - I haven't seen it at all.
But even if it's true to some extent, it says nothing about whether
applications will get upgraded and therefore it says nothing about 
whether users will be able to use IDNs in their applications.  They are
completely separate issues.  If anything, the appearance of ACE in
legacy appliations will drive deployment of IDN-aware versions of those
applications.

I do think that most widely-used legacy protocols will get upgraded - 
but on their own schedules, depending on the additional perceived 
utility in doing so, the perceived difficulty of doing so, and also
depending on what other demands there are that the protocol get updated.
And the last thing we want from an operational perspective is for
large numbers of protocols to get updated concurrently - producing
the maximum disruption.
 
> protocols and operations into ACE unnecessarily.

Regarding the API, I've always assumed that an API to resolve UTF-8 
IDNs was essential.  As for zone files, I've already proffered a suggestion 
to have an exchange format (using ACE) distinct from the native 
format.  The native format could use UTF-8 if that were the platform's 
native charset.  But we shouldn't presume that all platforms will use 
UTF-8.  A separate UTF-8 exchange format, or a way of labelling
native format to say that it is using UTF-8, might also be a possibility.

> try again with ACE. 

no, that's simply not acceptable.  DNS queries are already too 
slow and too unreliable, and you've just introduced some very 
interesting failure modes.

> long run, it's not an overhead, it's an investment. 

okay, but what's the return on the investment?  it appears to be 
purely psycological, since using UTF-8 on the wire in DNS doesn't 
make it any easier to upgrade either applications or protocols to 
use UTF-8 natively.    you could argue that it reduces complexity
in the long run, but the day that some benefit will be derived
for that reason is decades away.  and by then we should have
upgraded DNS (for other reasons) anyway.

Keith

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Jul 2001 16:46:26 +0900

>behavior that is most natural for human beings.

I think this is worth trying, in order to get rid of the famous 'A' for
Latin, Greek, and Cyrillic. It's of course to be done on a per script
base, not per language. I wouldn't actually resolve by tweaking
codepoints (sometimes it will be very difficult to decide which
codepoint to tweak), but just by rejecting strange combinations.
You have to do a keyboard switch to get from one script to the other,
so the chance of getting a mixture accidentally isn't great.
Doing the check only on the registration side may also be a very
good idea; that may allow us to start with very tight rules and
expand them later (e.g. allow scripts separated by a hyphen,...).
It would also help a lot to address some bidirectionality problems.

>records, NS records, MX records, etc.)

Well, 'not using nameprep' for PTR records would actually mean
to use it to make sure the mapping is right, but not to put the
result into the PTR record. It may be that it's better to use
a new record type.

In terms of feasibility, the idea to check that each label only
contains letters from a single script sonds much more doable,
although there are a few special cases (languages written with
Cyrillic using some Latin letters).

Regards,    Martin.

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 18 Jul 2001 17:28:42 +0900

Hi,
----- Original Message -----
From: "Martin Duerst" <[email-masked]>
> It would also help a lot to address some bidirectionality problems.

Good idea.

Now, let's think about another case of   all-Greek "oo.com" and all-Latin
"oo.com":
Either of the two consists of scripts from only single character sets.
But the two still look very similiar. Do you have any good idea about this ?

Regards, Soobok Lee

---

From: Martin Duerst <[email-masked]>
Date: Wed, 18 Jul 2001 17:38:09 +0900

>But the two still look very similiar. Do you have any good idea about this ?

1) Not try to do any business in Greece with this (either way),
    or in any region where Cyrillic is used (at least until IDNs
    are deployed widely enough).
2) Use an appropriate equivalent to .com for disambiguation.
3) Disallow all combinations of any of the three letters only.

Probably 1) or 2) are easier than 3).
2) will take quite some time.

Regards,   Martin.

---

From: "tsenglm" <[email-masked]>
Date: Wed, 18 Jul 2001 23:04:55 +0800

----- Original Message -----
From: "Keith Moore" <[email-masked]>
To: "Soobok Lee" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; "Martin Duerst" <[email-masked]>;
<[email-masked]>
Sent: Wednesday, July 18, 2001 11:10 PM
Subject: Re: [idn] Re: Back to work (Nameprep) (was: Re: Just send UTF-8
with nameprep (was: RE: [idn] Reality Check))

> > But the two still look very similiar. Do you have any good idea about
this ?
Assume there are no mixed type of language scripts  except the ASCII.
At client side , the input method environment knows what language the user
used.
To differentiate them ,   a lookup table can be  used , they can be
represented as:
      Greek(oo.com)  => unicode(oo).greek.com
      Latin(oo.com)   =>  unicode(oo).latin.com

Just like ACE is mapped to  ASCII LDH, so ACE(IDN)  need  bq--  tag for IDN
to differentiate them from traditional ascii domain-name.

Li-Ming Tseng,

---

From: John C Klensin <[email-masked]>
Date: Wed, 18 Jul 2001 11:54:38 -0400

--On Wednesday, 18 July, 2001 08:22 -0700 Mark Davis
<[email-masked]> wrote:

> I think the whole notion of trying to prevent cross-script
> confusions in domain names is a morass. It would invariably
> result in complicated rules that would invariably have false
> positives and negatives; it also depends greatly upon the font
> in use on the particular user's machine. We even have that
> now, with one and ell -- depending on the user's font, those
> can look identical.
> 
> Better would be to have useful GUIs that detect and signal
> possibly confusing names. For example, in the URL field of a
> browser, the spelling-check-style wavy underline could be used
> under terms like "intÐµl.com" vs "intel.com" (where in the
> first the "e" is the Cyrillic letter U+0435), to alert the
> user that the URL might be odd. Such tools would not get in
> the way of legitimate domain names that mix scripts or symbols.

Mark,

I think your conclusion is correct, but its implications are
quite broad.

This set of conversations has been very interesting to me, for
the unfortunate reason that it confirms the tentative, but
painful, conclusion I reached a few months ago.  Once we get
down to the really fine details of which characters match and
which do not, how to disambiguate glyphs from different scripts
that look (or are) identical, and so on, we need to rely on user
interfaces and human intelligence (or very close approximations
to the latter), rather than depending on absolute matching rules.

If we have to do that (and I agree that we will), then we almost
certainly need a non-DNS mechanism to support it.  If nothing
else, the performance costs of making multiple "maybe it is this
one" probes into the DNS to try to sort out ambiguities will
almost certainly be unacceptable (remember that DNS timeouts are
on the order of seconds and that cached negative responses
cannot have long durations).

If we _are_ going to fix the UIs to handle these cases, the
amount of work required to use a completely different mechanism
that supports human-assisted disambiguation tools (rather than
one encoding or another within the DNS) is almost certainly
quite small in comparison to other aspect of the effort required.

And, if we need a non-DNS mechanism, the current ACE versus
UTF-8 debate has the potential to turn into "look in three or
more different ways" story.   That, in turn, dramatically
increases the odds of false positives, and false positives cause
both security and sanity problems (as well as driving trademark
lawyers crazy).

    john

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 18 Jul 2001 08:22:05 -0700

I think the whole notion of trying to prevent cross-script confusions in
domain names is a morass. It would invariably result in complicated rules
that would invariably have false positives and negatives; it also depends
greatly upon the font in use on the particular user's machine. We even have
that now, with one and ell -- depending on the user's font, those can look
identical.

Better would be to have useful GUIs that detect and signal possibly
confusing names. For example, in the URL field of a browser, the
spelling-check-style wavy underline could be used under terms like
"intеl.com" vs "intel.com" (where in the first the "e" is the Cyrillic
letter U+0435), to alert the user that the URL might be odd. Such tools
would not get in the way of legitimate domain names that mix scripts or
symbols.

Mark
—————

πάντων μέτρον ἄνθρωπος — Πρωταγόρας
[http://www.macchiato.com]

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 18 Jul 2001 10:35:01 -0500

Keith Moore wrote:

> As for zone files, I've already proffered a suggestion to have an
> exchange format (using ACE) distinct from the native format. The
> native format could use UTF-8 if that were the platform's native
> charset.  But we shouldn't presume that all platforms will use UTF-8.
> A separate UTF-8 exchange format, or a way of labelling native format
> to say that it is using UTF-8, might also be a possibility.

I think that the exchange format (master files and AXFR/IXFR) can and
should be one the last point we discuss. It doesn't need to be decided on
first, and if we debate it later we will have arguments that don't lend
themselves to an "X is preferred, so Y must suck" interpretation.

However, a couple of points.

 a) There will be some requirement for the UTF-8 namespace to
    provide manually-encoded ACE names directly. If we are
    converting UTF-8 to ACE, these fixed names are pretty easy to
    implement. Conversely, if we are doing ACE to UTF-8, then we
    have to come up with a way to prevent the ACE name from being
    converted to its UTF-8 equivalent in order for the ACE
    encoding to be preserved in the UTF-8 namespace.

 b) The exchange format applies to a specific zone. A zone which
    consists of IDNs only has to be consistent across the servers
    for that zone. If we are supporting a UTF-8 namespace for a
    particular zone, then it is okay to use UTF-8 as the default,
    since any additional servers which join the zone must be able
    to use it.

Like I said, this should probably be one of the last things we do. But
given the impact of (a) and the permission of (b), the default encoding
for zone management operations should probably be UTF-8.

> interesting failure modes.

The default encoding for application functions needs to be ACE, except in
those cases where the application protocol explicitly provides in-stream
support for UTF-8 IDNs.

However, it's also important to remember that many lookups are not
dictated by application protocols (IETF or otherwise). Ping should
certainly be allowed to use the UTF-8 encoding on its command line, for
example. Hostnames in simple apps (like POP3 server) could also use UTF-8,
even though the POP3 protocol doesn't address this. It might be prudent to
simply say that "A lookups MAY default to UTF-8 if the operating
environment allows it, but the environment SHOULD explicitly indicate that
it supports this usage." We may need to encourage a usage whereby the
application can do a binding test to check the resolver.

---

From: Keith Moore <[email-masked]>
Date: Wed, 18 Jul 2001 11:10:09 -0400

> But the two still look very similiar. Do you have any good idea about this ?

first, how likely is this in practice that a label of all Greek letters
will accidentally collide with a label of all Latin letters?   (as opposed
to a deliberate collision)

for second-level domains at least, and for some third-level domains,
it would make sense for the registry to disallow labels whose appearances
collide with all-ASCII labels.   there are already some such rules in place
even for ASCII, these would just be extended.

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 18 Jul 2001 09:45:34 -0700

John,

I agree for the most part; in that we can do the bulk of the "heavy-lifting"
in nameprep, handling case issues and obvious variants. However, for the
fine issues of mixtures of scripts and symbols, I think that at this point
we are better off not having absolute rules, but rather leaving it up to
some intelligence on the client side, assisted by the GUI.

I don't know that we need to specify the precise mechanism: after all, there
was none for "1" and "l", which may look different on your machine, but look
identical on mine right now. I agree that such intelligence should be on the
client side.

I agree. If we leave it up to humans to make the choice (assisted by GUI)
then probably something as simple as:

  Using http://www.unicode.org/Public/3.1-Update/Scripts-3.1.0.txt

  - find the first script that is neither COMMON nor INHERITED
  - flag any character that has a different script.

(flagging could be: different color, wavy underline, or some other
mechanism)

This should, however, be a recommendation not a requirement. More
sophisticated mechanisms could be developed in the future, either by this
body or by application vendors. For example, the consortium could develop
"confusible" mappings: sets of characters that in common fonts, at common
sizes, could be easily confused with one another. Given them, an application
could have a finer-grained GUI assist. However, the development of such
mappings would take time and effort, and continue to be refined over an
extended period.

After all,
(a) the majority of characters in different scripts are not confusible:
there is no Latin letter that looks like katakana KA.
(b) more importantly, not all confusible characters are separated by script:
especially in the area of symbols.

If it is impossible to register names *except those that have been
nameprepped*, then whether or not UTF-8 names go over the wire unprepped or
not should not cause much of a security problem.

    john

---

From: Keith Moore <[email-masked]>
Date: Wed, 18 Jul 2001 14:36:05 -0400

John,

> to the latter), rather than depending on absolute matching rules.

I think this is true to a large degree.  But I think that to come
up with a practical solution to the problem will require a 
combination of not only user interfaces and human intelligence
on the client side, but also client APIs that reject ambiguous
symbol sequences, registration rules that discourage conflicts, 
nameprep-like canonicalization and folding, and perhaps even
fuzzy matching on the server side.
 
> certainly need a non-DNS mechanism to support it.

I agree that we can't support this with current DNS protocol 
mechanisms.  However there is so much investment and mind-share
in the DNS name space, that I don't think starting another 
name space that is similar to DNS will get very far.  And if
we're going to have DNS-like names, then we might well be better
off augmenting DNS servers (with appropriate protocol extensions
that allow them to do fuzzy matching) than to try to build a 
completely separate service.    Separate services seem far more
likely to get out of sync with each other and produce inconsistent
results.

Keith

---

From: John C Klensin <[email-masked]>
Date: Wed, 18 Jul 2001 14:58:29 -0400

--On Wednesday, 18 July, 2001 14:36 -0400 Keith Moore
<[email-masked]> wrote:

>>...
> I think this is true to a large degree.  But I think that to
> come up with a practical solution to the problem will require
> a  combination of not only user interfaces and human
> intelligence on the client side, but also client APIs that
> reject ambiguous symbol sequences, registration rules that
> discourage conflicts,  nameprep-like canonicalization and
> folding, and perhaps even fuzzy matching on the server side.

I believe in all of these but the last.  Perhaps it is just lack
of knowledge on my part, but I don't understand how to do fuzzy
matching in a deterministic way.  Fuzzy matching, to me, always
implies heuristics and, in a changing environment (as new names
are registered and others drop out of the DNS space), heuristics
imply that a given search string might resolve to one name and
target on one day and a different one on a different day.  That
is a property I hope to never see in the DNS.

>> If we have to do that (and I agree that we will),  then we
>> almost certainly need a non-DNS mechanism to support it.
> 
> we're going to have DNS-like names, then we might well be
> better off augmenting DNS servers (with appropriate protocol
> extensions that allow them to do fuzzy matching) than to try
> to build a  completely separate service.    Separate services
> seem far more likely to get out of sync with each other and
> produce inconsistent results.

Well, I don't know anyone who has seriously proposed a non-DNS
name space.  The most radical proposals on the table are, to my
knowledge, both my strawmen:

	* Staying within the DNS, but using a different Class to
	force application changes and create an unambiguous
	distinction between "upgraded" and "non-upgraded"
	clients.  The arguments against that include important
	"slow deployment" and "difficult transition" issues, a
	risk-free environment (even the proponents of ACE see
	some risks), but are otherwise painfully similar to the
	ACE-versus-UTF8 discussion we have seen in the last week
	or so.
	
	* Layering additional search-type facilities on top of
	the DNS, with most or all of the "internationalization"
	work being done in those additional sublayers, rather
	than relying exclusively on the DNS.

I hope it is obvious that neither of these is a "discard the
DNS, find another namespace and mechanism" approach.

And I note for the record that I haven't suggested, this week,
that the IDN WG declare success and shut down without proposing
any modifications to the DNS or the way it is used.

   john

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 18 Jul 2001 23:28:52 +0000

This message contains responses to Martin Duerst, Mark Davis, and Keith
Moore.

Martin Duerst <[email-masked]> wrote:

> But if [nameprep] is supposed to be strictly applied on every
> occasion, in particular every time a name is resolved, as e.g. Patrick
> is describing it, these foldings may lead to people believing that
> these characters are acceptable in a domain name the same way an
> upper-case character is.

What's wrong with actually considering them to be acceptable in the same
way as uppercase characters?

>    they are mapped to. (for obvious reasons)

The reason is not obvious to me.  B and b are very clearly visually
distinct, yet we live happily in a world where both berkeley.edu and
Berkeley.EDU are commonly used, and are equivalent.

> 2) Characters that completely map to ASCII-only characters (to make
>    sure that current applications and applications doing nameprep
>    behave the same way for ASCII-only).

I don't see the problem, and you are apparently proposing to eliminate
the mapping from full-width Latin to half-width Latin, which I think
would be unfortunate.

> nameprep (i.e. everything out there now).

Well of course, if you try to compare names without doing nameprep, then
you will fail to see matches where you should.  Of course feeding a
non-ASCII hostname to an IDN-unaware application is asking for trouble.

If you decide to write the domain name foo1.com using a circled 1, then
obviously you are using IDN functionality and you assume the same risk
as anyone using any IDN: the risk that someone will try to paste it into
an IDN-unaware application.

Mark Davis <[email-masked]> wrote:

> I think the whole notion of trying to prevent cross-script confusions
> in domain names is a morass.
>
> confusing names.

I agree.  There's really no way to guarantee that the distinction
between non-equivalent names is apparent to users.  Even with a font
that distinguishes 1 from l and 0 from O, people still overlook
misspellings, even blatant ones like "whitehouse.com" instead of
"whitehouse.gov".

What nameprep can do, however, is help make sure that users are able
to type the names they have in mind.  So a user who wants to type
<omicron><omicron> can do so, and a user who wants to type oo (Latin)
can do so (regardless of whether their keyboard interface defaults to
full-width or half-width).  A user who configures their browser to
accept cookies from oo.com (Latin) will not accidentally get cookies
from the Greek look-alike.

> nameprepped*, then whether or not UTF-8 names go over the wire
> unprepped or not should not cause much of a security problem.

Here's another way to think of it:  Whenever I use any domain name
for any purpose, I am trusting the name servers of that zone (and
all its ancestor zones) to give out true information when that name
is looked up.  With IDNs, I must now also trust those servers not to
give out bogus information in response to non-nameprep'd equivalent
representations of that name, but my exposure has not increased--I was
completely at the mercy of those servers before, and I still am.  So
as you say, allowing non-nameprep'd UTF-8 on the wire does not cause a
security problem.

> fuzzy matching on the server side.

Fuzzy matching at lookup time (as opposed to registration time) would
not help with the problem of domain name spoofing (like tricking someone
to follow a link to yah<omicron><omicron>.com).  It would help the
problem of someone seeing oo.com in print and not knowing whether
it's Latin or Greek, but that problem is much rarer, usually resolved
by context, and people can choose to register names that are not so
confusing.  I think fuzzy matching in servers would be more trouble than
it's worth.

AMC

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 19 Jul 2001 00:54:02 +0000

"Eric A. Hall" <[email-masked]> wrote:

>  a) There will be some requirement for the UTF-8 namespace to provide
>     manually-encoded ACE names directly.  If we are converting
>     UTF-8 to ACE, these fixed names are pretty easy to implement.
>     Conversely, if we are doing ACE to UTF-8, then we have to come up
>     with a way to prevent the ACE name from being converted to its
>     UTF-8 equivalent in order for the ACE encoding to be preserved in
>     the UTF-8 namespace.

There is only one namespace.  Every name in that single namespace has
both an ASCII representation and a UTF-8 representation (which are
identical in the case of ASCII names).  Every protocol, message format,
file format, etc. is free to require the ASCII representation or to
allow both representations.  If both representations are allowed, then
they must be treated the same by the receiver, because they are the
same name.  There is no need to tag some names as must-use-ACE.  Are
you trying to support names that really do begin with the ACE signature
prefix, so that the user sees the ACE prefix even inside IDN-aware
applications?  Such names are forbidden; they simply do not exist in the
namespace.  Trying to support them would be much more trouble than it's
worth.

> Ping should certainly be allowed to use the UTF-8 encoding on its
> command line, for example.

This is not a protocol issue, but a user-interface issue, and operating
systems have their ways of dealing with it.  For example, under UNIX,
commands should assume that their command-line arguments use the
encoding indicated LC_ALL or LANG.

> Hostnames in simple apps (like POP3 server) could also use UTF-8, even
> though the POP3 protocol doesn't address this.

I don't know what you mean by the POP3 protocol not addressing this.
The POP3 protocol exchanges a variety of commands, none of which contain
hostnames, and also exchanges RFC 822 messages, which are required to be
ASCII.  I see no way for a POP server to use UTF-8 hostnames until there
is an updated message header format and an updated Post Office Protocol
that uses it.

AMC

---

From: "D. J. Bernstein" <[email-masked]>
Date: 19 Jul 2001 01:35:39 -0000

> domain names is a morass. It would invariably result in complicated rules

I see a domain name on a business card. I type it in. I'll be unhappy if
I don't get the right name. Isn't that the whole point of nameprep?

If I see WWW.AOL.COM, where the first dot is actually U+3002, then I'm
sure I'll type it incorrectly. Isn't this why nameprep prohibits U+3002?

If I see WWW.AOL.COM, where the A is actually a capital Alpha, then I'm
sure I'll type it incorrectly. So why doesn't nameprep prohibit Alpha?

It seems safe to allow lowercase Greek alpha (and not the other alphas)
as long as users _don't_ think they can type it in uppercase. We really
should deprecate the use of uppercase characters in domain names.

---Dan

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 19 Jul 2001 02:19:06 +0000

"D. J. Bernstein" <[email-masked]> wrote:

> I see a domain name on a business card.  I type it in.  I'll be
> unhappy if I don't get the right name.  Isn't that the whole point of
> nameprep?

In my opinion (and this is just my opinion, not an interpretation of
the nameprep spec or the IDNA spec) the point of nameprep is to make
sure that users are able to type what they intend to type, and to make
sure that names are not corrupted by transcoders.  I think the visual
ambiguity problem is too much for nameprep to solve.  What do you do
about l and 1?  What about katakana ka and the Han character for power?
What about the katakana long vowel mark and the Han character for one
(and the em dash, and the minus sign...)?  I think Mark Davis is right
that this is a morass.

> If I see WWW.AOL.COM, where the first dot is actually U+3002, then
> I'm sure I'll type it incorrectly.  Isn't this why nameprep prohibits
> U+3002?

No, the nameprep spec is explicit about the reason for prohibiting
U+3002:

    U+3002 is used as if it were U+002E in many input mechanisms,
    particularly in Asia.  This prohibition allows input mechanisms to
    safely map U+3002 to U+002E before doing nameprep without worrying
    about preventing users from accessing legitimate host name parts.

> If I see WWW.AOL.COM, where the A is actually a capital Alpha, then
> I'm sure I'll type it incorrectly.  So why doesn't nameprep prohibit
> Alpha?

[Nitpick:  I wouldn't say you typed it incorrectly, I'd say you read it
incorrectly, then typed exactly what you intended to type.]

No one is going to put <Alpha>OL.COM on a business card and seriously
expect people to read it correctly.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 18 Jul 2001 21:12:04 -0500

"Adam M. Costello" wrote:
> identical in the case of ASCII names).

You are correct in that there is only one DNS namespace. Using that term
was a bad idea, and as much as I complain about conflicting metaphors, I
should have known better.

> Every protocol, message format,
> applications?  Such names are forbidden; they simply do not exist in
> the namespace.  Trying to support them would be much more trouble than
> it's worth.

Well, since you brought it up, this is my current thinking:

Dual-mode servers would flag specific RRs as having legacy or IDN names,
or having both flags. Some legacy domain names will look like UTF-8 names
(any eight-bit binary label is legal), and so there must be a way to flag
specific entries as being either a legacy name or an IDN (or both).

Any domain name which is entered as an IDN would need to have the IDN flag
enabled, but must not have the legacy flag enabled. These IDNs must only
be used to answer IDN queries, and must not be processed through legacy
query paths, as they would likely be used to answer queries being issued
by older applications in that scenario. We must not allow UTF-8 IDNs to to
be processed by older applications, since they may be issuing queries
based on a local encoding (such as 8859-n or windows-nnnn or MacRoman or
whatever). Using these IDNs to answer legacy queries will absolutely
definitely cause some kind of unintended consequence, and we will get a
lot of heat for the failure.

ACE domain names which are automatically created from the UTF-8 names
should only have the legacy flag enabled, to prevent bleed-over in the
opposite direction. It is also feasible to allow these entries to exist
with both flags, but see below.

It should also be possible to have an ACE name with both flags by allowing
users to enter it manually, although this should be cautioned against (but
not prohibited). We don't want applications that always re-apply ACE to an
IDN to do that, but moreover we don't want to make it common or simple to
make that mistake (although in truth, it might be more effective to let
people shoot themselves in the feet to really drive this message home).

For things like Dynamic DNS, this mechanism is actually pretty simple:
what message format was used for the registered name? Legacy names get the
legacy flag, while IDNs get the IDN flag (while the ACE conversions get
the legacy flag). For human operators, it is less clear. Perhaps one
product will allow users to specify the datatype on entry, while another
will simply store all names as IDNs and automatically create a legacy
version of the zone whenever the IDN zone is [re]loaded (with the caveat
that such a mechanism will break any non-IDN eight-bit labels). UI stuff
is probably outside of the scope, but some discussion would be fruitful as
it would likely result in a better management method than the one
presented here.

It would be a lot simpler to simply declare non-ASCII labels as IDNs, but
I don't think this is feasible. Perhaps it is?

> encoding indicated LC_ALL or LANG.

Good approach.

> > Hostnames in simple apps (like POP3 server) could also use UTF-8,
> > even  though the POP3 protocol doesn't address this.
> 

*hostnames* for POP3 servers are not specifed by the protocol. You have to
tell your POP3 mailer where it should go to pickup mail; that hostname is
what I'm referring to.

Also, from the quoted message:

 | It might be prudent to simply say that "A lookups MAY default to
 | UTF-8 if the operating environment allows it, but the environment
 | SHOULD explicitly indicate that it supports this usage."

This should be the general rule for all lookups (not just A RRs). In the
case of application-specific queries, the application must be sure to only
use domain names which are legal for that application. IOW, if an
application is presented with an IDN but the protocol specifications do
not allow for the use of IDNs, then the application should down-convert
the IDN and use that (or return an error, or whatever is appropriate).

---

From: "Soobok Lee" <[email-masked]>
Date: Thu, 19 Jul 2001 11:47:38 +0900

----- Original Message ----- 
From: "Adam M. Costello" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, July 19, 2001 11:19 AM
Subject: Re: [idn] nameprep failures

> expect people to read it correctly.

At least, a cheater or a fraud would deliberately try  it.

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 19 Jul 2001 03:42:29 +0000

"Eric A. Hall" <[email-masked]> wrote:

> Dual-mode servers would flag specific RRs as having legacy or IDN
> names, or having both flags. Some legacy domain names will look like
> UTF-8 names (any eight-bit binary label is legal), and so there must
> be a way to flag specific entries as being either a legacy name or an
> IDN (or both).

Okay, now I see what you meant by multiple namespaces.  On the one hand,
there's the world of ASCII names and IDNs.  On the other hand, there are
things that are neither:  8-bit names that predate any IDN standard.
Let's call such things "freaks".  :)

First, I question whether there really are any such names, and if there
are, whether we should care if we break them.

But assuming for the moment that we want to support these freak names,
we only need one flag, the freak flag.  An old-style DNS query can match
ASCII names (including ACEs) and freaks, but not IDNs.  A UTF-8 EDNS
query can match IDNs and ASCII names (including ACEs), but not freaks.
(I suppose you could also define another EDNS query for freaks, if you
wanted everything to be accessible via EDNS.)

There is no need for a second flag.  If a label contains only LDH
characters and conforms to ACE syntax, then it *is* an ACE, regardless
of whether it was obtained via an ASCII transport or a UTF-8 transport
or whatever.  There are no labels whose native (Unicode) form conforms
to ACE syntax.  To allow that would just confuse people.

> *hostnames* for POP3 servers are not specifed by the protocol. You
> have to tell your POP3 mailer where it should go to pickup mail; that
> hostname is what I'm referring to.

Okay, then this is a user-interface issue just like the ping example.

> This should be the general rule for all lookups (not just A RRs).  In
> the case of application-specific queries, the application must be
> sure to only use domain names which are legal for that application.
> IOW, if an application is presented with an IDN but the protocol
> specifications do not allow for the use of IDNs, then the application
> should down-convert the IDN and use that (or return an error, or
> whatever is appropriate).

I'm not sure what you mean here.  Let's take POP3 for example.  The
protocol doesn't allow for the use of IDNs, but that's unrelated to
the question of whether a POP3 client can connect to a server named by
an IDN.  If the client is IDN-aware, and has a UTF-8 resolver at its
disposal, there's no reason it shouldn't accept a UTF-8 server name from
the user and look it up.

AMC

---

From: "Bruce Thomson" <[email-masked]>
Date: Thu, 19 Jul 2001 13:12:42 +0900

These kind of visual ambiguities can only be resolved by
knowing the language. It's not likely that a business card
recipient will be able to type an IDN properly if the
script is exotic for him. He needs a second romanized
domain name to use.

On the other hand, one of the main good things about
nameprep is that it prohibits "bad" names at registration
time, preventing the unscrupulous from making scam sites
similar to legitimate ones. But with the abundance of
supported scripts, mixing and matching weird characters
will allow registration of things that we would prefer not
to have registered with the potential of confusion.

Unfortunately, the only way to stop this is to further
burden nameprep with rules that disallow scripts in
bizarre combinations. Is this worth it? I don't know
but I would be interested in hearing what people think.

Bruce

---

From: Martin Duerst <[email-masked]>
Date: Thu, 19 Jul 2001 16:25:26 +0900

>At least, a cheater or a fraud would deliberately try  it.

Cheaters won't try to put in on namecards.
They may try to put it into an email, and get you to
click on it.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 19 Jul 2001 16:04:22 +0900

>to a deliberate collision)

Very low for lower case. Between Latin and Greek, or Cyrillic and Greek,
it's only the 'o'.
Between Latin and Cyrillic, it's the 'a', 'e', 'o', 'p', 'c', 'y', 'x',
plus 's', 'i', and 'j' in some languages.

For upper-case, it's more. Regards,   Martin.

>collide with all-ASCII labels.

Most probably yes.

>there are already some such rules in place

Can you tell me more about these, or give some pointer?

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 19 Jul 2001 16:19:58 +0900

>lot of heat for the failure.

The chance that when a legacy resolver passes through a local encoding
and this matches with a UTF-8 entry is extremely low. The reason for this
is that UTF-8 has a very peculiar byte pattern. For details, please see my
paper at
http://www.ifi.unizh.ch/mml/mduerst/papers/PDF/IUC11-UTF-8.pdf.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 19 Jul 2001 16:34:57 +0900

>It seems safe to allow lowercase Greek alpha (and not the other alphas)

There are many accented Greek alphas; am I right that you would allow
them (in lower case)? There is a Latin alpha (in IPA), what about that?
There is a turned alpha (also in IPA), I guess that's no harm.

>should deprecate the use of uppercase characters in domain names.

Deprecating them looks like a good point, something that should go
in some of the drafts (maybe we need something like an 'advice to
users' draft?).

The nameprep draft might contain some advice to implementers about
whether to show the pre-nameprep or the post-nameprep string.
But I'm not sure it's worth throwing out the display of upper-case
(assuming about the same frequency as we have it currently)
to prevent the accidental confusion.

Regards,   Martin.

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 19 Jul 2001 08:20:49 -0700

> Very low for lower case. Between Latin and Greek, or Cyrillic and Greek,

This is inaccurate. For example, phi (φ,ϕ) and o-slash (ø) are confusable depending on the font, i.e., where the slash in the phi is at an angle. Rho (ρ) and p (p) are confusible in certain fonts, etc. And you can't forget all the extended Latin characters. There is Latin small letter open e (ɛ) and epsilon (ε). For Cyrillic and Greek, you are omitting many characters that are confusable (к, п, р, ф, є, ѡ, ѱ, ѳ, ...) . For capitals, don't forget the extended Latin and Cyrillic characters, like O with tilde (Ɵ) vs. theta (Θ), or Latin letter Esh (Ʃ) and Sigma (Σ).

But there are also many confusable characters even within the same script, like b (b) and Latin letter tone 6 (ƅ), p (p) and latin letter wynn (ƿ), many hooked letters which can fool someone who isn't looking for them: k (k) vs hooked k (ƙ), capital I or one and dental clicks (ǀ), variants like script g (ɡ), etc. Forbidding script mixing would not solve those.

It would take some real work to look at all the Unicode characters under a variety of common fonts at 9-12 point (normal body text) to determine which are confusable and which are not, and cross-check all the results. I'm not saying it can't be done -- and it would be a worthwhile effort in the long run -- but it is not trivial.

That is why I think the most practical solution is for GUI support that allows for human judgment.

Mark

P.S. Depending on what font your are using, of course, the above examples may or may not be distinguishable.

—————

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 19 Jul 2001 15:36:57 -0500

> > Using these IDNs to answer legacy queries will absolutely
> > definitely cause some kind of unintended consequence, and we
> > will get a lot of heat for the failure.
> 
> and this matches with a UTF-8 entry is extremely low. The reason for
> this is that UTF-8 has a very peculiar byte pattern.

I agree that it is low probability. However, for charsets like MacRoman
and Windows-1252 that reuse the C1 codepoint space, there is a measurable
probability of it happening.

IMO, if we are to take a risk, we should simply say that any label which
contains eight-bit characters must be processed through the IDN path, and
that any eight-bit queries which arrive via the legacy path are to be
rejected (we may also want a new RCODE to signify the exact failure).
Otherwise we end up in a situation where freak labels have to be compared
to the UTF-8 profile and then processed or rejected based on match level.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 20 Jul 2001 17:44:19 +0900

At 08:20 01/07/19 -0700, Mark Davis wrote:
>This is inaccurate.

This came in in an extremely small font. Was that on purpose :-) ?

Regards,   Martin.

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 20 Jul 2001 06:59:53 -0700

Yes, it was in 9pt, which is the lower end of normal body text (9-12pt).
What I couldn't show in the message is the variation among the typefaces
that people could be looking at domain names with.

Mark
—————

----- Original Message -----
From: "Martin Duerst" <[email-masked]>
To: "Mark Davis" <[email-masked]>; "Keith Moore" <[email-masked]>;
"Soobok Lee" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; <[email-masked]>
Sent: Friday, July 20, 2001 01:44
Subject: Re: [idn] Re: Back to work (Nameprep) (was: Re: Just send UTF-8
with nameprep (was: RE: [idn] Reality Check))