Subject: [idn] Newbie's questions implementing the [IDNA]

From: Seungho Lee <[email-masked]>
Date: Mon, 09 Dec 2002 10:47:18 +0900

Hello.

Implementing [IDNA] spec, I found an incomprehensible parts,
so I ask for your helps...

Question1.
What kind of sequence of code points can be the input of ToUnicode 
Operation?
Only punycoded one or all the other ones?
If in case of only punycoded one, it seems that the [NAMEPREP] operation 
is unnecessary...
(They will be in the ASCII range...)
And if all the other ones are permitted, isn't it that the [NAMEPREP] 
can't convert it to punycoded one?
(There is no punycode encoding process in Step2,
and From Step3, it is solid that the input is the ACE code points 
starting with ACE prefix.)

Question2.
What is the return value of DNS?
I think it is the IP address just like the usual DNS return value...
For example, when I send the query 'PREFIX-089a.com'(this is a Korean 
character) to the resolver,
the DNS will return the IP address for 'PREFIX-089a.com', won't it?

But the [IDNA] spec says in page 3,
- The ToUnicode opeartion is used when displaying names to users,
for example "names obtained from a DNS zone".

Does the DNS return a name?

Question3.
I came to think that the ToUnicode opeation is not included in the major 
flow of 'sending query and receiving the return values'.
This is the flow I pictured... (only when a person is to use IDN for web 
surfing...)
1) An user inputs an IDN to the browser... ( Let's suppose the input was 
'?.com' )
2) The IDNA client program takes the input.
3) Through the conversion operation ( the Step 4 would be ToASCII ), the 
input is changed to a punycoded one.
( '?.com' is converted to 'zq-089a.com' ; suppose the prefix is 'zq--' )
4) The IDNA client program sends the query to the resolver.
5) The resolver sends the query to the designated name server.
6) The name server sends the query to the .com name server.
( Suppose the user-designated name server has no cache for the query and 
also is not the authoratitive name server)
7) .com name server sends the query to the zq--089a.com name server.
8) The zq--089a.com name server returns the IP address.
9) The client's resolver receive the return value(IP address)

So, even though ToUnicode indispensible process of [IDNA], (because the 
spec requires that the uses should not see
the incomprehensible punycoded names), I think it is outside the major 
flow of 'sending query and receiving the reutrn values'...

Is this an acceptable thought?

And where exactly the ToUnicode operation should be done?
Between 3) and 4)?

Regards,
Lee
-----
Seungho Lee
Netpia.com, Korea.

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 9 Dec 2002 02:40:49 +0000

Seungho Lee <[email-masked]> wrote:

> Operation?  Only punycoded one or all the other ones?

Any sequence of Unicode code points can be the input of ToUnicode.  The
first paragraph of the ToUnicode section (4.2) says:

    The ToUnicode operation takes a sequence of Unicode code points that
    make up one label and returns a sequence of Unicode code points.
    If the input sequence is a label in ACE form, then the result is
    an equivalent internationalized label that is not in ACE form,
    otherwise the original sequence is returned unaltered.

> can't convert it to punycoded one?

Right, Nameprep does not perform Punycode encoding.

> From Step3, it is solid that the input is the ACE code points starting
> with ACE prefix.

Step 3 says "Verify that the sequence begins with the ACE prefix."
If the sequence does not in fact begin with the ACE prefix, then the
verification fails.  The second paragraph of the ToUnicode section says:

    ToUnicode never fails.  If any step fails, then the original input
    sequence is returned immediately in that step.

> What is the return value of DNS?

It depends on the query.  You can ask the DNS for A records, which
contain IP addresses, or PTR records, which contain domain names, or MX
records, which contain mail exchanger information, or NS records, which
contain name server information, and so on.

> I came to think that the ToUnicode operation is not included in the
> major flow of 'sending query and receiving the return values'.

ToUnicode is useful for displaying names to users.  That's the only
purpose mentioned in the IDNA spec for ToUnicode.  The "major flow"
you have in mind does not involve displaying names to users, right?
Therefore ToUnicode is not useful there.

> And where exactly the ToUnicode operation should be done?

Anywhere between where the application receives the domain name and
where the application displays the domain name to the user.  Domain
names might be recieved in message headers, or in HTML markup, for
example.

AMC

---

From: Soobok Lee <[email-masked]>
Date: Mon, 9 Dec 2002 11:44:50 +0900

> so I ask for your helps...

This list is not for a tutorial session, but i'll answer for your
questions as a fellow korean.

> Only punycoded one or all the other ones?

any legitimate labels. LDH domains or PREFIX--***.
utf-8 and legacy-encoded labels are for presentation-only, and not for
the wire format according to IDNA.

> (They will be in the ASCII range...)

Necessary for verification so that the output of ToUnicode must not contains
unnameprepped/prohibited/(unassigned) codepoints. IF the output contain such ones,
the input label should be displayed as it is in the form of the punycode encoding.

> And if all the other ones are permitted, isn't it that the [NAMEPREP] 
 
 What do you mean by "all the other ones"? Only LDH and PREFIX--** ones
are allowed as valid inputs to ToUnicode according to IDNA.

 ToUnicode does not  perform "legacy to unicode conversion", which are done
before ToASCII is performed.

> starting with ACE prefix.)

I can't understand your question.

> I think it is the IP address just like the usual DNS return value...

Right.

> the DNS will return the IP address for 'PREFIX-089a.com', won't it?

Yes. PREFIX--** works like a usual LDH label.

> Does the DNS return a name?

"DNS zone" dones not mean the DNS server query response.
It just mean "Visual representaion of zone file contents".

You should read IDNA again. ToASCII is performed before DNS resolving.
ToUnicode is for display and verification purpose for the punycode
encoded labels later.

> And where exactly the ToUnicode operation should be done?

Sorry. Nowhere in your all steps. When an user inputs an IDN label, he/she
already see native IDN label , not punycode-encoded one. Why does he
need  toUnicode  there?  If prenameprepped form and nameprepped form have
different glyphs (uppercase inputs?), step 10) will be added to perform toUnicode
for displaying the nameprepped (lowercase?) one. But even in this case,
i think respecting the user input in the display/location bar is preferrable.

Soobok Lee

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 9 Dec 2002 23:18:28 +0000

Soobok Lee <[email-masked]> wrote:

> Only LDH and PREFIX--** ones are allowed as valid inputs to ToUnicode
> according to IDNA.

According to the first paragraph of the ToUnicode section, any sequence
of Unicode code points can be the input of ToUnicode.

> the output of ToUnicode must not contains
> unnameprepped/prohibited/(unassigned) codepoints.

The output of ToUnicode can contain unnameprepped, prohibited, and
unassigned code points.  Simply feed such a string as input to
ToUnicode, and the string will be output unaltered by ToUnicode.

> ToUnicode is for display and verification purpose for the punycode

IDNA uses ToUnicode for exactly two purposes: to force labels to
human-friendly form before displaying them to users, and to define the
concept of ACE label (which is any label that ToUnicode would alter).

IDNA uses ToASCII for exactly two purposes: to define the concept of
internationalized label (which is anything that ToASCII can be applied
to without failing), and to force labels to ASCII form before putting
them into slots (protocol fields, function arguments, etc).

AMC

---

From: "William Blackwood" <[email-masked]>
Date: Mon, 9 Dec 2002 19:10:53 -0500

Adam,
Do you have any estimate when the RFC is going to be published?  How far
behind, in terms of calendar months/years is this working group's output
according to the original schedule?

Thank you,

William

---

From: Soobok Lee <[email-masked]>
Date: Tue, 10 Dec 2002 09:03:48 +0900

> of Unicode code points can be the input of ToUnicode.

You are right. but, many novices would carelessly misinterpret "toUnicode()" 
as doing some legacy to unicode conversion (if he did not carefully read drafts
or API manuals), 
and they may actually call "toUnicode("legacy encoded IDN")" expecting that.

> ToUnicode, and the string will be output unaltered by ToUnicode.

right. IDNA states that such outputs should not be displayed as native
ones, but just as ASCII ones as it is.  "must not" is  meant for that.
I think it is clear enough in drafts.

> them into slots (protocol fields, function arguments, etc).

Thanks.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 10 Dec 2002 00:29:46 +0000

> I think it is clear enough in drafts.

I don't know what you could be referring to.  Suppose X is a string
consisting entirely of uppercase Greek letters.  X is not nameprepped,
because nameprepped things don't contain uppercase.  X contains no ASCII
characters.  ToUnicode(X) equals X, exactly, which means it is perfectly
acceptable to display X.

AMC

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 10 Dec 2002 01:03:34 +0000

William Blackwood <[email-masked]> wrote:

> Do you have any estimate when the RFC is going to be published?

I don't.  It's now in the hands of the RFC editor.  People with more
IETF experience than I have might be able to guess how long it will
take.

> How far behind, in terms of calendar months/years is this working
> group's output according to the original schedule?

I joined the working group after it was already a year old, so I'm not
sure I ever saw the original schedule.  The only schedule I can find now
is shown here:

http://www.ietf.org/html.charters/idn-charter.html

It gives 2001-Jun as the goal for working group last call (which
actually happened in 2002-Jan).  The schedule gives no goals for IESG
approval (which happened in 2002-Oct) or RFC publication (which has yet
to happen).

AMC

---

From: Soobok Lee <[email-masked]>
Date: Tue, 10 Dec 2002 09:55:50 +0900

> characters.  ToUnicode(X) equals X, exactly, which means it is perfectly

I mean that Punycode(X) [not Punycode(NAMEPREP(X)), not X ] can be inserted into 
 RFC822 message headers. In that case, ToUnicode(Punycode(X)) should be 
treated differently than ToUnicode(Punycode(Nameprep(X))) .

You are right if X is non-ASCII input, because toUnicode(X)==X.

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Tue, 10 Dec 2002 13:12:31 +0100

>You are right if X is non-ASCII input, because toUnicode(X)==X.

if (ToASCII(nameprep(ToUnicode(ascii_text))) == ascii_text) babelname=true;
if (babelname) "iesg--ascii_text" will display in ASCII mode on most of the 
systems while having been registered, and possibly TMed, as 
ToUnicode(ascii_text).
I documented Adam with cases where babelname==true.
Among them "coca-cola", "ibm", "vint-cerf", "adam-costello".

IMHO multi-sub-profiling would keep interoperability while limiting such a 
practical inoperability. We could also investigate sub-profile SLDs as 
permitting better management and support and having ASCII labels as non 
protected 3LDs.
jfc

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 11 Dec 2002 01:11:04 +0000

"JFC (Jefsey) Morfin" <[email-masked]> wrote:

> if (babelname) "iesg--ascii_text" will display in ASCII mode on most
> of the systems while having been registered, and possibly TMed, as
> ToUnicode(ascii_text).

Your reference to "iesg--ascii_text" suggests that you expect ascii_text
not to include the ACE prefix.  But in that case, ToUnicode(ascii_text)
is simply ascii_text itself, because ToUnicode won't alter a string that
doesn't begin with the ACE prefix.

> I documented Adam with cases where babelname==true.  Among them
> "coca-cola", "ibm", "vint-cerf", "adam-costello".

babelname, as you've defined it, would be true for *every* lowercase
ASCII string (up to 63 characters).  I think the feature of those
example strings that you're interested in is that if you prepend the ACE
prefix to them, the result is an ACE (which is, by definition, something
that ToUnicode would alter).  So the test you're looking for is:

    ToUnicode(IESG--ascii_text) != IESG--ascii_text

which is roughly approximated by this test:

    Punyencode(Nameprep(Punydecode(ascii_text))) == ascii_text

which is similar in form to the test you proposed, but ToASCII is
very different from Punyencode, and ToUnicode is very different from
Punydecode.

> if (babelname) "iesg--ascii_text" will display in ASCII mode on most
> of the systems while having been registered, and possibly TMed, as
> ToUnicode(ascii_text).

IESG--ascii_text will be displayed in non-ASCII form by IDN-aware
applications capable of displaying the characters, and will be displayed
in ASCII form otherwise.

AMC

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Wed, 11 Dec 2002 05:05:22 +0100

Dear Adam,
>doesn't begin with the ACE prefix.

True. I was interested in the general idea.

>     Punyencode(Nameprep(Punydecode(ascii_text))) == ascii_text

actually it is:
- punycode_encode ()  and punnycode_decode.()
- and I am not interested in nameprep but just in making
   sure the Unicode stay invarient as per Paull Hoffman's draft.
   so it would be Paul_Hoffman_invarient() if I am correct.

>very different from Punyencode, and ToUnicode is very different from

I am only interested to knowi if such a program permits me to
correctly determine Unicode IDNs of which the

>applications capable of displaying the characters, and will be displayed

as "iesg--coca-cola.com" or "iesg--jonathan-cohen.net" or
"iesg--vint-cerf.org".

Thank you.
jfc

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 11 Dec 2002 14:37:59 +0100

On Wed, Dec 11, 2002 at 05:05:22AM +0100,
 JFC (Jefsey) Morfin <[email-masked]> wrote 

> as "iesg--coca-cola.com" or "iesg--jonathan-cohen.net" or

The Unicode strings which IDN-encodes into "iesg--coca-cola.com" is:

LATIN SMALL LETTER C (Basic Latin)
LATIN SMALL LETTER O (Basic Latin)
ORIYA DIGIT THREE (Oriya)
ORIYA DIGIT THREE (Oriya)
LATIN SMALL LETTER C (Basic Latin)
LATIN SMALL LETTER A (Basic Latin)
FULL STOP (Basic Latin)

LATIN SMALL LETTER C (Basic Latin)
LATIN SMALL LETTER O (Basic Latin)
LATIN SMALL LETTER M (Basic Latin)

In what language does it make sense? It is certainly possible to find
funny ACE encodings which have a meaningful Unicode form but it is not
obvious.

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Wed, 11 Dec 2002 16:50:00 +0100

>LATIN SMALL LETTER M (Basic Latin)

My program tells:
u+0063 u+006F u+0B69 u+0B69 u+0063 u+0061 . fr
are we in agreement?

>funny ACE encodings which have a meaningful Unicode form but it is not

The Unicode form is of no real interest except that the more character sets 
the more chances the IDN displays in ACE form and the less chances it is 
already registered as a multilingual design, TM or even patent (?).

The target is only to see printed "From: iesg--coca-cola.com" when the 
registrant sends a mail.
I feel multi-sub-profiling could drastically reduce that capacity. But what 
about u+A317.com? Even then it will print in most of the cases as iesg-ibm.com.

Again, unless I have misunderstood something. This is what I ask.
jfc

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 11 Dec 2002 17:22:58 +0100

On Wed, Dec 11, 2002 at 04:50:00PM +0100,
 JFC (Jefsey) Morfin <[email-masked]> wrote 

> u+0063 u+006F u+0B69 u+0B69 u+0063 u+0061 . fr

Yes. That's the good thing with standards :-) But I used the official
Unicode names to make easier to see there are no semantics in the
Unicode string.
 
> about u+A317.com? Even then it will print in most of the cases as 

It's u+3A17 and it does not seem registered at this time (although I
agree that such unfortunate collisions are possible, I simply believe
they are very uncommon).

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Wed, 11 Dec 2002 20:36:02 +0100

My question expected just a simple response:
- either yes, the proposed draft permits to defeat TMs rights on a great 
scale in using what you call "unfortunate collisions" and others call 
"babel names".
- or no, it does not (and may be why).
jfc

At 17:22 11/12/02, Stephane Bortzmeyer wrote:

>---
>Incoming mail is certified Virus Free.
>Checked by AVG anti-virus system (http://www.grisoft.com).
>Version: 6.0.427 / Virus Database: 240 - Release Date: 06/12/02