Subject: [idn] IDN WG Last Call on two major changes to Stringprep

From: "James Seng" <[email-masked]>
Date: Sun, 7 Jul 2002 16:06:58 +0800

The IETF last call have been concluded and the latest version of
Stringprep is in the I-D directory:

http://www.ietf.org/internet-drafts/draft-hoffman-stringprep-04.txt

Other then editorial changes, there are two significant changes*:
- Reference Unicode 3.2, updating all the tables from 3.1 to 3.2.
- Additional Bidirection characters handling, Section 6 and Appendix D.

The co-chairs is conducting a working group last call on these two
significant changes. Please send any proposed modification of text to the
wg mailing list as well as the wg co-chairs, stating clearly the text to
be modified and the new text.

This three weeks WG last call will end on 28th July 2002, 23h59 GMT-5.
(We have a three weeks last call instead of the usual two weeks for those
who is busy preparing for IETF Yokohama)

Marc & James
co-chairs

* The previous of Stringprep can be found at
http://www.potaroo.net/ietf/old-ids/draft-hoffman-stringprep-03.txt

---

From: "Edmon Chung" <[email-masked]>
Date: Sun, 7 Jul 2002 09:58:17 -0400

A minor change:

> Other examples are
> simplified Chinese spellings of names (for example,
> "<U+7EDF><U+4E00><U+7801>") vs. the equivalent traditional Chinese
> spelling (for example, "<U+7D71><U+4E00><U+78BC>").

I dont think Chinese phrases should be called "spelling".
I would suggest using the word "representation" instead of "spelling" so
that it makes more sense.

-----Update to-----
Other examples are simplified Chinese representation of names (for example,
"<U+7EDF><U+4E00><U+7801>") vs. the equivalent traditional Chinese
representation (for example, "<U+7D71><U+4E00><U+78BC>").
-----

Edmon

---

From: "Edmon Chung" <[email-masked]>
Date: Mon, 8 Jul 2002 07:37:38 -0400

I dont quite understand why we need to have 3.
Isnt 3 a subset of 2?
Also

-----
2) If a string contains any Right-to-Left character (defined as
belonging to Unicode bidirectional categories "R" and "AL"), the string
MUST NOT contain any Left-to-Right character (defined as belonging to
Unicode bidirectional category "L").

3)  If a string contains any Right-to-Left character (as defined above),
a Right-to-Left character MUST be the first character of the string, and
a Right-to-Left character MUST be the last character of the string.
-----

Also this will mean that there cannot be a mixture between RTL and LTR
characters.  While I am not familiar with Arabic, I sure have seen English
words mixed with Arabic in phrases, albeit rare.  I didnt see much
discussion on the list before on bidi issues, but I did see an example used:

> Assume there were two labels inside the DNS, one reading ABCdef
> and the other reading defABC, and both would be displayed
> CBAdef. Who would consider that usable for the DNS?

Why would both be displayed the same?  Shouldn't defABC be displayed as
defCBA?  Given the context of a domain name within a DNS.

Furthermore, the section for bidi in Nameprep should be updated to specify
that it is meant to say within a label there cannot be both RTL and LTR
characters (eventhough it is mentioned early in the doc that Nameprep deals
with labels), because in Stringprep that wasnt clear and also a given
"string" can be a "part" of a label, so there could be two "strings", one
containing LTR one RTL in the same label.

Edmon

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Mon, 8 Jul 2002 07:04:47 -0700

>I dont quite understand why we need to have 3.

No. There are many categories, not just R, AL, and L, as detailed in 
the Unicode standard.

>. . . Why would both be displayed the same?

Read the Unicode standard for information on this.

>characters

That's not true.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Edmon Chung" <[email-masked]>
Date: Mon, 8 Jul 2002 11:27:49 -0400

I think I am lost.  (which also suggests that the wordings need to be
improved)
Please clarify two simple things:
a. Are mixed RTL and LTR characters allowed within a label?
b. If there are more categories than R, AL and L that we are discussing,
then in point 3 it should not say "As defined above":

-----

Edmon

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 16 Jul 2002 07:47:23 +0000

> I dont quite understand why we need to have 3.

No, because there are characters that are neither Right-to-Left nor
Left-to-Right.

> characters.

Correct.  That is exactly rule 2 above.  "If X appears then Y must not
appear" is exactly equivalent to "X and Y must not both appear".

> While I am not familiar with Arabic, I sure have seen English words
> mixed with Arabic in phrases, albeit rare.

I don't doubt that.  I know almost nothing about the bidi algorithm, but
the bidi experts concluded that this was the price that needed to be
paid to prevent distinct labels from being displayed identically.

> I didn't see much discussion on the list before on bidi issues, but I
> did see an example used:
>
> > Assume there were two labels inside the DNS, one reading ABCdef and
> > the other reading defABC, and both would be displayed CBAdef. Who
> > would consider that usable for the DNS?
>
> Why would both be displayed the same?

I don't know. :) The answer can presumably be inferred from the bidi
tech report.

> a given "string" can be a "part" of a label, so there could be two
> "strings", one containing LTR one RTL in the same label.

Nameprep doesn't need to know whether it's input string is a label or
part of a label or whatever.  Nameprep could operate on any string.

But in IDNA, the IDNA spec specifies that nameprep is applied to labels,
not to substrings of labels.  Therefore a label cannot contain both LTR
and RTL characters.

> a. Are mixed RTL and LTR characters allowed within a label?

No.

> then in point 3 it should not say "As defined above":

Yes it should.  Rule 3 is using the same definition of Right-to-Left
character as stated in rule 2: 'belonging to Unicode bidirectional
categories "R" and "AL"'.

AMC

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 27 Jul 2002 05:28:44 +0200

Quoting the draft:

,----
| In any profile that specifies bidirectional character handling, all
| three of the following requirements MUST be met:
...
| 2) If a string contains any Right-to-Left character (defined as
| belonging to Unicode bidirectional categories "R" and "AL"), the string
| MUST NOT contain any Left-to-Right character (defined as belonging to
| Unicode bidirectional category "L").
| 
| 3)  If a string contains any Right-to-Left character (as defined above),
| a Right-to-Left character MUST be the first character of the string, and
| a Right-to-Left character MUST be the last character of the string.
`----

There is little rationale for the last two requirements.  Without
knowing the rationale, it is difficult to understand how to implement
this, not to speak of understanding and evaluating the specification.

It is not difficult to construct various strings that violates these
requirements, but seem like valid identifiers to me (e.g., U+05D0
U+0966, contemplate it being written by a mathematically inclined
writer in India).  Why is U+05D0 a R/AL character but U+2135 not?
U+2135 is NFKC'd into U+05D0.  It thus seems like the identifier is a
valid IDN if NFKC is not used, but if NFKC is used, it is not a valid
identifier.  A bidi user thus seem to require NFKC not to be used in
order to have the bidi string accepted.

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 26 Jul 2002 21:17:47 -0700

There are two issues. (a) rationale, (b) NFKC interaction.

For the rationale, I include at the bottom an old email. It proposed a
somewhat more complicated solution than eventually appeared in the
text; I think what is in StringPrep is better (just for simplicity).

For the NFKC interaction, you bring up a good point. The conditions on
the string, if the goal is to always have a consistent appearance
(both on keyboard entry, and when displaying a name fetched from the
server) should be in effect for the string both before and after
StringPrep. In practice, it only has any effect in a few isolated
cases, since there are very few characters whose BIDI class changes.

Mark

----- Original Message -----
From: "Mark Davis" <[email-masked]>
To: <[email-masked]>; "Paul Hoffman / IMC" <[email-masked]>
Sent: Saturday, September 15, 2001 19:03
Subject: Re: First attempt at problem statement

BIDI IDN

I will try to recap some of the discussions in the ad hoc on BIDI IDN
last week.

The BIDI algorithm is designed to deal with normal text. Within any
string, sequences of LTR (left to right) and RTL characters will
always appear in the correct order. However, the order of other
characters (such as a period) will depend on their context. For
details, see http://www.unicode.org/reports/tr9/.

URLs are not normal text, and thus may have odd display. This is
complicated by the fact that the overall paragraph direction has an
effect on the display. Whether the URL is displayed in a RTL or LTR
context will change the order of the components. [In this and other
examples we will use the convention that uppercase letters stand for
right-to-left characters: Arabic, Hebrew, etc.] Example:

Memory:         http://SOME.LARGE.mixed.CORP.org
Display (LTR):  http://EGRAL.EMOS.mixed.PROC.org
Display (RTL):  org.PROC.mixed.EGRAL.EMOS//:http

Notice that "SOME.LARGE" always appears from RTL: the period adopts
the order of the surrounding characters. Characters on boundaries
(such as "//:" take on the overall display direction.

For example:

(1) characters in different fields may mix across fields:

Memory:         http://SOME.veryLARGE.CORP.org
Display (LTR):  http://EMOS.veryPROC.EGRAL.org
Display (RTL):  org.PROC.EGRALvery.EMOS//:http

(2) two different sequences of characters in the same field can have
the same order when displayed. Thus a user would not know how to type
a URL that he sees printed.

Memory1:        http://123CORP.org
Memory2:        http://CORP123.org
Display (LTR):  http://123PROC.org

The following are proposed requirements.

1. Consistent fields.

(a) If you have fields such as http://XXX.YYY.ZZZ, characters from the
same field should not be displayed in different fields and vice versa.
(b) It should be possible to deduce the order of the backing-store
fields from the order of the display fields.

2. Order within fields.

Within a field, tt should be possible to deduce the order of the
backing-store characters from the order of the display fields.

3. No algorithm change

This should require no change to BIDI algorithm. Using a separate
algorithm for display of URLs would be difficult, since they are found
within flowing text. Getting people to update the BIDI algorithm would
also be quite difficult (and changes to make URLs work might have
repercussions on other text).

4. Simplicity

Whatever solution we have should have a simple algorithm (according to
Paul).

Ideally, some reasonable restrictions on the contents of a field would
meet all of these requirements.

 * * *

During the meeting, I thought that the most straightforward method was
to force the periods to be LTR. However, after looking at the results
in both a RTL and LTR context, I concluded that Mati's approach would
be better. The results both in terms of field order and order within a
field would still be determinate. With any complete URL (with http://,
ftp://, etc.) it would be easy to recognize the order in context,
since the position of those initial letters would show the order. The
only bad case would be where the end of the string reversed because of
its surroundings, e.g.:

Memory:         the url http://SOME.LARGE.mixed.CORP.org is the one
Display (LTR):  the url http://EGRAL.EMOS.mixed.PROC.org is the one
Display (RTL):  org is the one.PROC.mixed.EGRAL.EMOS//: the url http

Thus in flowing text, users would be recommended to bracket any BIDI
URL with RLM (or embed the URL). E.g.

Memory:         the url <RLM>http://SOME.LARGE.mixed.CORP.org<RLM> is
the one
Display (LTR):  the url http://EGRAL.EMOS.mixed.PROC.org is the one
Display (RTL):  is the one org.PROC.mixed.EGRAL.EMOS//:http the url

Software, such as email clients, that recognizes URLs could do this
automatically.

Here is what I have now for an explicit algorithm to be added as a
step to NamePrep, after the regular Prohibition step.

A. Characters are classified into RTL, LTR, DIGIT, OTHER.

These categories are drawn from the BIDI algorithm. The precise lists
of characters in each category would be added to NamePrep as an
appendix. The composition is as follows (See
http://www.unicode.org/reports/tr9/#Bidirectional_Character_Types).

LTR   := L ; # including LRM

RTL   := R | AL ;

DIG   := EN | AN ;

OTH := all other characters: NSM, ON, etc.

Note: The characters in categories LRM, RLM, LRO, RLO, LRE, RLE, PDF,
B, S, and some other BIDI categories are prohibited anyway.

B. In any field that contains any RTL characters:
B0. no LTR characters can occur.
C1. a sequence of characters of type DIG can only occur at the end.
C2. a sequence of characters of type OTHER can occur only between
characters of type RTL.

The following is an example of an algorithm that implements (B). EOS
stands for "end of string".

1. Let S be 0
2. Get the next character, then get its numeric type T
3. Let S be Map[S, T]
4. If S = T or F, exit with OK or FAIL respectively
5. Goto Step 2

Map is defined by the following table:

  T  LTR RTL DIG OTH EOS
S +---------------------
B |   L,  R,  L,  L,  F   // begin
L |   L,  F,  L,  L,  T   // left
R |   F,  R,  D,  O,  T   // right
O |   F,  R,  F,  O,  F   // right + other
D |   F,  F,  D,  F,  T   // right + digit

Mark
__________
http://www.macchiato.com
◄  “Eppur si muove” ►

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 27 Jul 2002 06:57:03 +0200

"Mark Davis" <[email-masked]> writes:

> For the rationale, I include at the bottom an old email. 

Thanks.

(If people want to look at more discussions, I found an archive at
<URL:http://www.imc.org/ietf-bidi/entire-arch.txt>.)

> It proposed a somewhat more complicated solution than eventually
> appeared in the text; I think what is in StringPrep is better (just
> for simplicity).

Somehow I wish this was discussed in the specification, if people are
supposed to understand it without digging in mailing lists archives.
The fact that bidi is difficult to understand (as is normalization and
other IDN topics) isn't an excuse to be sparse with details, rationale
and examples; on the contrary.

> cases, since there are very few characters whose BIDI class changes.

What could a solution be?  Perform the bidi step before NFKC too?  If
the intention is to not ignore the issue, that is.

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 27 Jul 2002 07:51:09 -0700

That would be a solution, if it was felt to be important that the
input also obey the conditions. Whether it is important or not, I
would defer to Mati.

Mark
__________
http://www.macchiato.com
◄  “Eppur si muove” ►

----- Original Message -----
From: "Simon Josefsson" <[email-masked]>
To: "Mark Davis" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; "IETF/IDN
WG" <[email-masked]>; "Matitiahu Allouche" <[email-masked]>
Sent: Friday, July 26, 2002 21:57
Subject: [idn] Re: IDN WG Last Call on two major changes to Stringprep

Thanks.

>> | MUST NOT contain any Left-to-Right character (defined as
belonging
> to

---

From: John C Klensin <[email-masked]>
Date: Sat, 27 Jul 2002 12:20:54 -0400

--On Saturday, 27 July, 2002 07:51 -0700 Mark Davis
<[email-masked]> wrote:

>> field would still be determinate. With any complete URL (with
>> http://, ftp://, etc.) it would be easy to recognize the order 
>> in context, since the position of those initial letters would
>> show the order.
>> The
>> Memory:         the url http://SOME.LARGE.mixed.CORP.org is
>> the one Display (LTR):  the url
>> http://EGRAL.EMOS.mixed.PROC.org is the one Display (RTL):
>> org is the one.PROC.mixed.EGRAL.EMOS//: the url http
>> 
>> Thus in flowing text, users would be recommended to bracket
>> any BIDI URL with RLM (or embed the URL). E.g.
>...

Mark,

As I understand it, this essentially implies that an IDN, or at
least an IDN that might contain RTL characters, is not
interpretable unambiguously unless it appears as part of a
fully-spelled-out URL.  I can't imagine that being acceptable.  I
can't even imagine it being acceptable to require that IDNs
appear in URLs at all to be valid.

To repeat something I seem to need to say in every talk I give
about these issues,
   "the web is not the net".
This WG is tasked with internationalizing _domain names_, not
URLs. An acceptable URL solution may also be a requirement, but
that is a side-effect.

       john

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 27 Jul 2002 17:06:41 -0700

> fully-spelled-out URL.

No, it is the converse.

> URLs. An acceptable URL solution may also be a requirement, but

Understood. However, having these restrictions in domain names is a
step towards internationizing URLs, and one without which it would be
difficult.

Mark
__________
http://www.macchiato.com
◄  “Eppur si muove” ►

----- Original Message -----
From: "John C Klensin" <[email-masked]>
To: "Mark Davis" <[email-masked]>
Cc: "Simon Josefsson" <[email-masked]>; <[email-masked]>;
<[email-masked]>; "IETF/IDN WG" <[email-masked]>;
"Matitiahu Allouche" <[email-masked]>
Sent: Saturday, July 27, 2002 09:20
Subject: Re: [idn] Re: IDN WG Last Call on two major changes to
Stringprep