Subject: [idn] Comments on IDNA/stringprep/nameprep

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 8 Feb 2002 00:25:25 +0100

Comments on IDNA/stringprep/nameprep

1. stringprep and nameprep should be rejoined to a
hostnameprep. They are only about host name preparation,
not any other name preparation.  Similar preparations
may still take advantage of the hostnameprep document,
by declaring "deltas", small changes that may be needed
for other (Internet, DNS) names.  That would likely
minimise the size of the "reuse" documents.

2. hostnameprep should be applied to the *entire*
hostname; i.e. the entire name should be 'mapped'
in the same way *before* it is parsed into parts.

3. Various FULL STOPs should be mapped to FULL STOP,
which must be allowed. Some of this is accomplished via
NFKC, but some mappings need be added specially for
hostnameprep, e.g. IDEOGRAPHIC FULL STOP. (Note that
parsing into parts should come after the mapping and
prohibition steps.)

4. Various Pd (punctuation dash) should be mapped to
HYPHEN-MINUS by hostnameprep.  Future keyboards may
generate HYPHEN rather HYPHEN-MINUS (except perhaps
in "programming language mode", which few will use).
At least, hostnameprep should not prevent such a development.

5. Symbols/punctuation/dingbats (except the hyphen-like
dashes) should not be allowed by hostnameprep; and
all of that prohibition should be in hostnameprep,
not some of them handled differently elsewhere.
Punctuatuation in particular, in contexts where hostnames
are embedded, may in future syntaxes use non-ASCII
punctuation adjacent to the hostname. At the very least
such a development should not be prevented by hostnameprep.
Symbols are at present excluded, and should remain so also
for non-ASCII symbols, for the same reason as punctuation
should be excluded.

6. Hangul syllables (with conjoining characters, not
non-conjoining compatiblity characters) that represent
the same syllable must be mapped to the same representation.
Due to unfortunate historic reasons, this does no longer
happen automatically with NFKC (though for drafts for
NFKC it did).  Mappings should be added so that "syllabically"
equivalent Hangul conjoning characters are mapped to a common
representation.  Hangul compatibility letters should be
prohibited though.  Correctly mapping those is more complicated
than can be expressed in the (current form of) (host)nameprep
mappings. Hangul compatibility letters should instead be
prohibited. (Mapping table for Jamos, and prohibition table for
Hangul compatibility characters, are available upon request.)
Future keyboard, e.g., input may generate only single letter
Jamos, rather than any "cluster letter" Jamos or precomposed
Hangul syllable characters.  At the very least, hostnameprep
should not prevent such a development.

7. No document associated with hostnameprep should make any
further restrictions on domain/host names than hostnameprep
itself. (In addition, duplicating some of the restrictions
elsewhere is confusing and should not be done.)

8. Note: The SC/TC issue cannot be solved at a near-impossible-
to-change (once deployed) technical level, but should instead
be solved at a policy level (which may employ software with
relatively easy to change mappings).

9. User interfaces that encounter mixed script hostname *parts*
should be recommended to "flag" them (ballon warning, color
differentiate, make blinking, bounce automatic registratations, ...).

		/Kent Karlsson

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 7 Feb 2002 18:36:20 -0800

Ken, you are coming in very late in this process. A lot of this was
debated back and forth early in the process, both on mail and in
personal contact. I suggest for a start that you review all of the
archives so that you don't simply retread issues.

Mark
—————

Πόλλ’ ἠπίστατο ἔργα, κακῶς δ’ ἠπίστατο πάντα — Ὁμήρου Μαργίτῃ
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "Kent Karlsson" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, February 07, 2002 15:25
Subject: [idn] Comments on IDNA/stringprep/nameprep

/Kent Karlsson

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 07 Feb 2002 19:06:19 -0800

Kent Karlsson <[email-masked]> wrote:

> 1. stringprep and nameprep should be rejoined to a hostnameprep. They
> are only about host name preparation, not any other name preparation.

I think they are about domain name preparation.  Some domain names are
textual but not host names, and don't obey the host name syntax rules.
For example, _ldap._tcp.foo.net (see RFC 2782 about SRV records).  I see
no reason why IDNA should restrict its attention to host names when it
can work perfectly well for all textual domain names.

That said, there could still be arguments for recombining the stringprep
and nameprep documents.  But you haven't yet presented any.

> 2. hostnameprep should be applied to the *entire* hostname; i.e. the
> entire name should be 'mapped' in the same way *before* it is parsed
> into parts.

Can you present any arguments favoring that approach?  I can think of a
few problems with it:

Sometimes programs count the number of dots in a name.  Nameprep can
change the number of dots (see the "Co." character, for example).  This
would be asking for trouble.

After running nameprep on the whole string, then splitting into labels,
individual labels might be invalid Unicode strings.  For example, a
label might start with a combining character.  So you'd have to do
another round of checking on the individual labels anyway.

In general, software often operates on labels individually, splitting
names, joining names, comparing labels, etc.  If IDNA were to do
anything to create interdependencies and interactions between labels, it
would be asking for trouble.

> 3. Various FULL STOPs should be mapped to FULL STOP

This is motivated by the previous suggestion, so we can put it aside for
now.

> 4. Various Pd (punctuation dash) should be mapped to HYPHEN-MINUS by
> hostnameprep.

Presumably this is to avoid visual ambiguity.  But there are many many
visual ambiguities created by the addition of the Unicode repertoire
(for example, several characters that look exactly like B).  Does this
one really warrant special treatment?  Why?

> 5. Symbols/punctuation/dingbats (except the hyphen-like dashes) should
> not be allowed [in host names]...  Punctuatuation in particular, in
> contexts where hostnames are embedded, may in future syntaxes use
> non-ASCII punctuation adjacent to the hostname.

That's a reasonable argument.  I argued something similar myself long
ago, but I didn't persuade the group.  Maybe it's too late now.

> 6. Hangul syllables (with conjoining characters, not non-conjoining
> compatiblity characters) that represent the same syllable must be
> mapped to the same representation.  Due to unfortunate historic
> reasons, this does no longer happen automatically with NFKC (though
> for drafts for NFKC it did).  Mappings should be added so that
> "syllabically" equivalent Hangul conjoning characters are mapped to a
> common representation.

I know nothing about this issue, but it sounds like you want the IETF to
redo Unicode work.  The Unicode folks know more about characters than we
do; I'm skeptical that we could do a better job on that sort of thing.

> 7. No document associated with hostnameprep should make any further
> restrictions on domain/host names than hostnameprep itself.

That's simply impossible.  Nameprep has no way of knowing whether the
name is too long.  You can't know that until after Punycode has been
applied.  Also, nameprep is not designed to apply special rules at the
beginning/end of the string, like prohibiting leading/trailing hyphens,
or prohibiting the IDNA prefix.  Those checks are better left where they
are now, in ToASCII.

> to-change (once deployed) technical level, but should instead be
> solved at a policy level

I tend to agree.

> 9. User interfaces that encounter mixed script hostname *parts* should
> be recommended to "flag" them (ballon warning, color differentiate,
> make blinking, bounce automatic registratations, ...).

Generally a good idea, although some scripts might be mixed often.
Maybe just talk about "labels containing combinations of characters
likely to be misleading", and give a couple of examples, and leave it to
the software developers to figure out what in general is misleading and
what isn't.

AMC

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 9 Feb 2002 14:54:22 +0100

> -----Original Message-----
> From: Adam M. Costello
> Sent: den 8 februari 2002 04:06
> To: IETF idn working group
> Subject: Re: [idn] Comments on IDNA/stringprep/nameprep
> 
> 
> hostnameprep. They

A *delta* of the hostnameprep whould/should be used for SRV records;
i.e. it would use hostnameprep with a specified modification:
"LOW LINE is not prohibited" and whatever other changes needed 
for SRV records compared to hostnames, only listing the differences
(that is a delta), not the full result of applying the delta.

> and nameprep documents.  But you haven't yet presented any.

I'm not sure why the document was split into "stringprep" and a
"profile".  At least I would find a "hostnameprep" with deltas
for similar domain name preps a much more manageable approach.

> Can you present any arguments favoring that approach?  I can 

Point 3 was the motivation.  Also: most users are likely to see a
hostname as an entity that would be treated the same by the system.
If fullwidth letters are perfectly ok to enter (mapped to normal width
by nameprep), why is not FULLWIDHT FULL STOP ok?  I think most users
that encounter that or similar (IDEOGRAPHIC FULL STOP in particular)
will find the currently specified behaviour idiosyncratic.

> think of a
> change the number of dots (see the "Co." character, for 
> example).  This would be asking for trouble.

The "counting of dots" for an IDN should be done (as if) after the
mapping+NFKC, not before.

> another round of checking on the individual labels anyway.

Yes, so?  "The other round" here does completely different things
than the "first round".  What's the problem?

> would be asking for trouble.

Where was that implied in my suggestion?  As I said, parsing into parts
must be done as if after hostnameprep (most implementations may actually
do it after hostnameprep).  

> This is motivated by the previous suggestion,

No, it's the other way around: This point motivates the point 2.
See above.

> one really warrant special treatment?  Why?

If you had read my point 4 in its entirety, you would have found
the motivation: "Future keyboards may generate HYPHEN rather HYPHEN-MINUS
(except perhaps in "programming language mode", which few will use).
At least, hostnameprep should not prevent such a development."
What is not clear about that motivation?  It is not the business of
the IDN WG to prevent otherwise possibly desired developments of
keyboards.  The current suggestion puts up an unnecessary stumbling
block for this at least, which is inappropriate for this WG to do.

> dashes) should

Again, it is not the business of the IDN WG to put up stumbling blocks
for developments that may otherwise be desirable.  It is quite
conceivable that future designers of syntaxes that embed host names
may wish to do so using non-ASCII punctuation and symbols adjacent
to the hostname (before being ACE encoded).  With the current proposal
that is, without reason, prevented.

> do; I'm skeptical that we could do a better job on that sort of thing.

This is due to a compatibility mapping removal.  The compatibility mapping
was present in Unicode 2.1, but was removed for Unicode 3.0, not for
Hangul specific reasons, but for technical trouble with the NFKC definition.
A better, and for Hangul more reasonable resolution would have been to
make those compatibility mappings canonical instead, which would have
been appropriate both for Hangul and for the NFKC definition issue.

This is again an issue of not putting up stumbling blocks for future
possible developments.  Future Hangul keyboards *may* (if so desired)
generate just single letter jamos (Hangul is an alphabetic script
with 14 consonant letters and 11 vowel letters; the alphabet *itself*
does not have hundreds/thousands of characters).

> or prohibiting the IDNA prefix.  Those checks are better left 

Let me rephrase:  "No document associated with hostnameprep should
make any further restrictions on the principal makeup of domain/host
names than hostnameprep itself. They make make size or tagging restrictions
though (those aspects would normally not be directly visible to the user)."

> the software developers to figure out what in general is 

That's fine by me.

		/Kent K

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 9 Feb 2002 15:03:10 +0000

Kent Karlsson <[email-masked]> wrote:

> i.e. it would use hostnameprep with a specified modification:  "LOW
> LINE is not prohibited"

But what about when the exact classification of a domain name is
unknown?  Maybe the software knows that it's a textual domain name but
does't know whether it's a host or a SRV or something else that was
invented after the software was written.  Should the software err on the
side of being too lenient or too strict?  Should it default to enforcing
the hostname syntax, which is probably the most restrictive syntax there
will ever be?  How could it default to anything else if nothing else
were defined in IDNA?

Nameprep defines a least common denominator--the most lenient
restrictions that *all* textual domain names must adhere to.  Software
can always apply those restrictions without fear of disabling legitimate
domain names, and software can depend on the fact that all legitimate
domain names adhere to those restrictions.  Hostnameprep would not have
those useful properties.

Your idea of having one basic preparation for domain names and deltas
against it for particular kinds of domain names is a good idea, but I
think the deltas should always add restrictions, never remove them.  In
this model, the host restrictions in step 3 of ToASCII are a delta.

> "profile".

So that the stringprep framework could be reused for other things
(things that are not domain names at all).

> by nameprep), why is not FULLWIDTH FULL STOP ok?  I think most users

What currently specified behavior?  IDNA says nothing about the
separators between labels except that they are "usually" dots.  IDNA
says nothing about how to split domain names into labels, or join labels
into domain names.  It neither requires nor prohibits the acceptance of
fullwidth full stop as a domain label separator.

I'm not sure there is any wide-reaching standard on this.  STD13 (the
DNS standard) specifies that labels are dot-separated in DNS master
files, but doesn't say they should use that syntax in other places.  RFC
(2)822 says that domain labels are dot-separated in email headers.  But
those two standards disagree about the trailing dot: it is required
in master files and forbidden in email headers.  They also disagree
about whether whitespace is allowed around the dots.  Syntax rules for
delimiting domain labels appear many separate standards, but I'm not
aware of any overall requirement regarding domain label delimiters that
applies to all applications/protocols.

> "Future keyboards may generate HYPHEN rather HYPHEN-MINUS (except
> perhaps in "programming language mode", which few will use).  At
> least, hostnameprep should not prevent such a development."

I have no strong opinion either way.  Anyone else?  Should hyphen-like
characters get special treatment just as whitespace characters get
special treatment?

AMC

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sat, 9 Feb 2002 08:41:51 -0800

>I'm not sure why the document was split into "stringprep" and a

The stringprep document says so explicity:

Other protocols can create profiles of these rules; these profiles 
will allow users to enter internationalized text strings in 
applications and have the highest chance of getting the content of 
the strings correct.

Note the first two words. Stringprep is already being used by other 
protocols that have nothing to do with DNS, such as iSCSI names.

>   At least I would find a "hostnameprep" with deltas

But would make all other string preparation documents much harder to read.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 9 Feb 2002 22:30:06 +0100

> protocols that have nothing to do with DNS, such as iSCSI names.

It's a "00" DRAFT.  It cannot possibly (reasonably) already be *used*.

> But would make all other string preparation documents much 

The 'deltas' approch works very well for collation.  One big
'default' table and a number of (most often small) deltas.
I don't see why that approach would make things difficult here.
To me, and I guess to others, it's easier to handle just *one*
targe table (with parts; here: map, prohibit, unassigned) and
a number of small changes, than to handle multiple large tables
that may have arbitrary and unemphasised differences.

		Kind regards
		/kent k

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 9 Feb 2002 22:29:39 +0100

> Adam M. Costello

> unknown?  Maybe the software knows that it's a textual domain name but

draft-ietf-idn-nameprep-07.txt has the title "Stringprep profile for
Internationalised ***Host Names***" (my emphasis)...

> will ever be?  

You have a point there.  But then the "nameprep" document should
say that this is the "default table" (or "tailorable template"
which is politically correct ISO-speak for that) and "default
normalisation".  Another document should then define the "host name
delta", This should be quite separately from IDNA, which talks about
other things like the use of "punnycode" [I liked that misspelling...;
or was that "pennycode" ;-); sorry], something most users should
(though they will not) be shielded from.

However there is a problem with that: one has to decide *now* what
additions are allowable in the "textual formal Internet names",
and which are not.  With the approach I favor (most restrictive,
i.e. hostnames as "default", and no punctuation/symbols in the hostnames,
with deltas adding permissions to use) additional characters allowed
in "deltad" names or instead used for future IDN embedding syntaxes
can be decided as one goes along with this at future times.

> this model, the host restrictions in step 3 of ToASCII are a delta.

See above, on both issues.

> (things that are not domain names at all).

I'm all in favour of reuse; note that a "deltas" approach has a much
higher degree of reuse!

> fullwidth full stop as a domain label separator.

Hmmm, this kind of contradicts your statements in your other e-mail.
I think something should be said about this, namely that the entier
IDN is processed the same way before any kind of part splitting.
When everything else is so detailed about mappings etc., this should
be the same.  Otherwise how does anyone know if FULLWIDTH STOP is
allowed (as a part separator) in an IDN or not?

		Kind regards
		/kent k

---

From: Dave Crocker <[email-masked]>
Date: Sat, 09 Feb 2002 13:46:21 -0800

>It's a "00" DRAFT.  It cannot possibly (reasonably) already be *used*.

This represents both a failure to understand how things can be done in the 
IETF and what specific things are being done now.

>The 'deltas' approch works very well for collation.

The choice for separation was made carefully, after failure to make the 
split caused problems.

You are, of course, entitled to your own preferences.  However personal 
preferences are not all that relevant during a Last Call.

d/
----------

Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.273.6464

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 11 Feb 2002 21:37:27 +0000

I wrote:

> IDNA says nothing about the separators between labels except that they
> are "usually" dots.  IDNA says nothing about how to split domain names
> into labels, or join labels into domain names.  It neither requires
> nor prohibits the acceptance of fullwidth full stop as a domain label
> separator.

> Hmmm, this kind of contradicts your statements in your other e-mail.

I don't see how.  Maybe you're thinking of comments made by someone
else.

> IDN is processed the same way before any kind of part splitting.  When
> everything else is so detailed about mappings etc., this should be the
> same.  Otherwise how does anyone know if FULLWIDTH STOP is allowed (as
> a part separator) in an IDN or not?

That is a sensible concern.  The difficulty is that there is no existing
standard for this, even for LDH host names, so we have no starting
point to internationalize.  DNS queries and responses don't use dots at
all, they use length bytes.  DNS master files use ASCII dots as label
separators, and require a trailing dot.  URIs also use dots as label
separators, but the trailing dot is optional.  In email message headers
the trailing dot is forbidden, and the other dots can be surrounded by
optional whitespace and parenthesized comments (no kidding!).

Thus, domain names are defined as a sequence of labels, but the
representation of that sequence is not standardized, so it would be
difficult for IDNA to say precisely how to internationalize this
non-standard.  Furthermore, given that applications often pick domain
names apart and put them together, it is safest if IDNA operates on
labels independently.

However, I suppose we might want to consider having the IDNA spec to say
something vague like:

    Applications that use U+002E (FULL STOP) to separate domain labels
    are encouraged to accept all the full-stop characters as label
    separators:

    U+002E FULL STOP
    U+0589 ARMENIAN FULL STOP
    U+06D4 ARABIC FULL STOP
    U+0701 SYRIAC SUPRALINEAR FULL STOP
    U+0702 SYRIAC SUBLINEAR FULL STOP
    U+1362 ETHIOPIC FULL STOP
    U+166E CANADIAN SYLLABICS FULL STOP
    U+1803 MONGOLIAN FULL STOP
    U+1809 MONGOLIAN MANCHU FULL STOP
    U+3002 IDEOGRAPHIC FULL STOP
    U+FE52 SMALL FULL STOP
    U+FF0E FULLWIDTH FULL STOP
    U+FF61 HALFWIDTH IDEOGRAPHIC FULL STOP

This calls into question the current nameprep prohibition of U+3002.
Why pick on just that one?  If we make a special case for U+3002, why
not do the same for all the other full-stop characters?  On the other
hand, nameprep does *not* prohibit U+002E (and it shouldn't--notice that
STD-13 includes an explicit demonstration of how to put ASCII dots in
domain labels).  U+002E is prohibited for host names in ToASCII, but
allowed for non-host domain names.  Maybe the same should be true of the
other full-stop characters.  Or maybe they should simply be allowed,
and registrants should just be advised that domain labels containing
full-stop characters will be difficult or impossible to use in many
situations.

AMC

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 12 Feb 2002 01:41:36 +0100 (CET)

> minimise the size of the "reuse" documents.

Kent,

There is work going you in other WGs to use stringprep (that you might not
be aware of).
Is the high-order bit of this comment that you'd like nameprep
be renamed hostnameprep?

> should not prevent such a development.

This sounds like you are proposing to fix/change a decision made by the
Unicode consortium. Why is that in scope?

> differentiate, make blinking, bounce automatic registratations, ...).

By "*parts*" do you mean labels or something else?

  Erik

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 12 Feb 2002 12:36:04 +0100

> From: Erik Nordmark [mailto:[email-masked]]
...
> Is the high-order bit of this comment that you'd like nameprep

Both the title and abstract of nameprep says that it is about
host names (or even host name parts).

However, the gist of the comment you are referring to is that:

There should be a basic approach (currently stringprep)
*and* a basic table of mappings and prohibitions, and that
deltas should be made from that.  Currently each use of stringprep
must have complete tables, and cannot reuse a common basic
table (per version of Unicode).

The idea is to have more reuse through having such a common
table.  It would make the similar (in other WGs) uses easier to
review and (once approved) use.

To make that workable, the basic table should be very restrictive
and not allow any punctuation or symbols (other than full stops,
and hyphen-like characters).  The latter can then be taken into
use as either syntax around domain names or be included in
certain other kinds of domain names than hostnames on a per
punctation/symbol basis.

Going the other way around (as seems to be suggested by the
authors), frezes at time of first deployment which
punctuation/symbols can and cannot be used in future
syntaxes that embed domain names.  In particular, the current
suggestion *forbids* as special use (i.e. surrounding syntax or
use in special purpose (non-host) domain names) any non-ASCII
symbol/punctuation.

> This sounds like you are proposing to fix/change a decision 
> made by the Unicode consortium. Why is that in scope?

Even though e.g. [gg] and [g][g] (there are a few hundred other examples)
are not canonically or compatibility equivalent, they still represent
the same sequence of Hangul letters, and thus "mean" the same.
Even though not all systems display the decompostions correctly yet,
there is no reason to believe that that they will not be supported
by most rendering engines for Hangul.  Unfortunately, the normal
forms cannot be changed at this time, even though that would have
been better.  There have been many misconceptions around about,
e.g., where Hangul syllable break are, which may have lead
to the current situation.  Once those misconceptions are cleared
up, you will likely see more comments to the fact that [gg] and [g][g]
(etc.!) are in fact equivalent from a Hangul perspective.

> By "*parts*" do you mean labels or something else?

stringprep: "DNS domain name parts";

idna: "A label is an individual part of a domain name";

nameprep: "This document describes how to prepare
	internationalized host name parts" (I think that's the
	wrong approach, e.g. it should apply to the entire name,
	but I'm just quoting for the term here.)

		Kind regards
		/kent k

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 12 Feb 2002 16:47:23 +0100 (CET)

> certain other kinds of domain names than hostnames on a per

I understand that alternative.

> use in special purpose (non-host) domain names) any non-ASCII

I don't understand this issue - stringprep is likely to be used for
many things that use Unicode and need ways to test for equality etc
and not all of these are domain names.
I think (but would gladly accept to be corrected) that the IPS use of
stringprep is intended to be for names of storage volumes (or something
like that).
Given that this intent exists your concern about freezing the set
of punctuation can't possibly be something that the people working
on non-host/domain name use of stringprep see as a problem.

> the same sequence of Hangul letters, and thus "mean" the same.

Yes, same argument is used for SC/TC needing to be addressed in IDN.

> (etc.!) are in fact equivalent from a Hangul perspective.

I still don't see why you think this adjustment/fix to Unicode is in scope
for IDN.

> 	but I'm just quoting for the term here.)

I didn't ask what semantics others assign to "parts" - I asked specifically
what you meant in your use of the term.

  Erik

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 12 Feb 2002 22:39:15 +0100

> and not all of these are domain names.

Yes, sorry for sayin "domain" there.

> stringprep is intended to be for names of storage volumes (or 
> something like that).

In the scenario I suggest, "volume names", "file names", or even 
programming language identifers should be able to use a tailoring
of a common talbe (that carries most of the data).  Things like
case fold or not case fold should be made easy to express in a
tailoring (delta).

> on non-host/domain name use of stringprep see as a problem.

"freezing the set of punctuation"?  Where did I say or imply that?
What I said was that punctuation should be disallowed by the default
(but some, selectively, punctuation may be allowed by a delta/tailoring).

> Yes, same argument is used for SC/TC needing to be addressed in IDN.

No, no, no!!  This issue is comparable to the *canonical* equivalences
that already exist for Hangul syllable characters, and for other 
characters that have a canonical decomposition (some "double latin
letters" have compatibility decompositions, but the relationship here
is much stronger; and it is much much stronger than case insensitivity).
Unfortunately, due to historic events, that equivalence is no longer
recorded in Unicode 3.0 and later property data.

This is in no way comparable to the SC/TC issue which is a spelling
preference issue, where the "spellings" are actually different.
Here it is just about the underlying representation for the **same**
spelling (in terms of sequence of letters; there is not even any
case difference or font variant difference [for correctly constructed
fonts that cover Hangul]).

> > up, you will likely see more comments to the fact that [gg] 
> > and [g][g] (etc.!) are in fact equivalent from a Hangul perspective.
> 
> I still don't see why you think this adjustment/fix to 
> Unicode is in scope for IDN.

Since it can no longer be corrected by changing the normalisation
forms, just about everything else has to do the correction instead.

> what you meant in your use of the term.

The quotes motivated not only how, but why I used that term.
Except for the definition in the idna document, why don't
you complain when others use the same term without definition?

Suggestion: All the documents should include a sufficiently
complete "definitions of terms" clause, where all technical
terms are defined in a dictionary like format (not with the
term inlined into a paragraph).

		Kind regards
		/kent k

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 13 Feb 2002 11:06:32 +0900

Thanks, Kent.

----- Original Message ----- 
> fonts that cover Hangul]).

True. the canonical equivalence between [gg] and [g][g] is defined in the
unicode 3.0 . They should have been unified by NFC, but haven't correctly.

Too late to be changed. and It should be solved in new normalizatio forms.
But If applications use the new normalization before nameprep,
As i warned  in the last call comments, the following condition will be
 trigerred silently,

  stringprep(newnormalization(Hangul)) != stringprep(Hangul) 

If stringprep would be neutral to new normalization adopted by applications,
 stringprep should be perfect and inclusive of all kinds of mature normaliztions,
that is,  the universal set of all kinds of normalizations built upon unicode.
Impossible?

Applications implementors should be cautious when applying normalizations forms
to data/texts portions that contain IDN. If some applications already adopted some
normalizations forms that are not compatible to stringprep as above,
backward compatibility requirements are not met in that case.
IDNA's backward compatibility claim doesn't come without costs.

Don't build our grand castle on the moving sand dune, on which a tiny tent is more adequate
and wise choice. :-)

Soobok Lee

---

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 13 Feb 2002 19:02:43 +0100

...
> The Unicode consoritium debated making the canonical decomposition
> from <gg> to <g><g> for a long time. The deciding feedback was from
> the Korean national body at the Seoul SC2/WG2 meeting, where they said
> it should not be done; that it was akin to canonically decomposing "w"
> to "vv". 

With all due respect, I don't think that is the correct analysis.  The
original alphabet consisted of 17 consonant and 11 vowels, none of them
"double" per se (later complemented with a few variants, no longer in
use). Using doubled consonants for particular sounds is very much like
using doubled consonants in the Latin script, even though the phonemic
differences are of a different nature.  Yes, the doubled consonants
in Hangul collate as separate entities, but that is a consequence of
the general approach of collating by consonants cluster (and vowels
cluster; irrespective of how that is achieved technically; Mark and
I have had a long debate about the latter...).

If [g][g] and [gg] really are different, how would [g][g] and [gg] be
differentiated within properly typeset Hangul syllable blocks?

> They also objected to combinations like <gs> being
> canonically decomposed, principally so that modern syllables could
> always be decomposed into 3 pieces. 

And why would that be essential?  Especially since some of them still
decompose into two pieces...  Hangul is a very elegant writing system,
but its computer encoding is far from elegant, I'm sorry to say, even
though a subset remains very elegant.  However that subset interacts
with the rest when normalising, and the normalisation is not complete
(different normal forms for the same (logical) sequence of letters).

> The (weaker) compatibility
> decompositions in Unicode until the time that NFC was formed; those
> were removed because they would have prevented the formation of Hangul
> Syllables in NFKC.

No, but them being compatibility instead of canonical decompositions
prevented the *preservation* of Hangul syllable characters in NFKC.

It is not the first time we hear that that would be a syllable
break between two consecutive lead consonants, or between a lead
consonant and a Hangul syllable character, even though that
has never, AFAICT, been the case in Unicode.  It is not unlikely
that the misconception of spurious syllable breaks has lead astray.

		Kind regards
		/kent k

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Wed, 13 Feb 2002 11:43:44 -0800

At 7:02 PM +0100 2/13/02, Kent Karlsson wrote:
>With all due respect, I don't think that is the correct analysis.

It seems a tad inappropriate for the IETF to consider not only going 
against the Unicode standard but *also* against the desires of the 
Korean national body to ISO because of the desires of a small number 
of people who are neither representatives of standards bodies or the 
national governments that are most interested in the script in 
question.

--Paul Hoffman, Director

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 13 Feb 2002 07:31:44 -0800

The Unicode consoritium debated making the canonical decomposition
from <gg> to <g><g> for a long time. The deciding feedback was from
the Korean national body at the Seoul SC2/WG2 meeting, where they said
it should not be done; that it was akin to canonically decomposing "w"
to "vv". They also objected to combinations like <gs> being
canonically decomposed, principally so that modern syllables could
always be decomposed into 3 pieces. The (weaker) compatibility
decompositions in Unicode until the time that NFC was formed; those
were removed because they would have prevented the formation of Hangul
Syllables in NFKC.

Mark
—————

Γνῶθι σαυτόν — Θαλῆς

http://www.macchiato.com

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "Kent Karlsson" <[email-masked]>; "'Erik Nordmark'"
<[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, February 12, 2002 18:06
Subject: Re: [idn] Comments on IDNA/stringprep/nameprep

---

From: "Kent Karlsson" <[email-masked]>
Date: Sat, 16 Feb 2002 13:24:02 +0100

> national governments that are most interested in the script in 

It seems to be much more than a tad inappropriate (for anyone) to go
against the *design* of a script. Note that Hangul was really designed,
by a working group, resulting in a document describing the (then) new
script.  So there is no after-construction involved when referring to
"the design" here.  There have been some minor changes of use, like
IEUNG as trail now stands for what YESIEUNG ("ng") used to stand for.
But that does not affect the structure of the design.  See "The Korean
Language" [1], section 6.3 gives an English translation of the original
description of the script design.  (Note that I'm not suggesting that
YESIEUNG be folded to IEUNG, they are different letters, albeit similar.)

I think I know why the letter cluster characters were invented, and 
why one did not want them to be decomposed at the time: it has to do
with collation.  But getting Hangul correctly collated *can and
should* be done without resorting to these letter cluster characters.
(There are several methods, irrelevant for this group.)

Note that it is not entirely unusual for SC2/WG2 to go "against the
desires of the NNN national body to ISO" that is asking for some
rearrangement or precomposition of characters, referring said
NB to 14651 for how to achieve correct collation order.

For all other (alphabetic or syllabic) scripts we have that: same
sequence of letters (including diacritics) = same "nameprepped" form =
same "name"; indeed, for IDN we even disregard all case differences
(for historic reasons; which we therefore must generalise; even though,
as has been pointed out, there are still some technical problems
with that [Turkish, and Greek; dotted i and adscript iota...]).

Now why should we in this instance at all regard multiple low
level representations of exactly the ***SAME*** sequence of
letters (and [tone] marks), according to the design, as being
different?  They will not only look the same (if properly
supported), they *ARE* the same, even if one representation
uses letter cluster characters and the other does not.

		Kind regards
		/kent k

PS
Note still that this is very different from the SC/TC case where the
spelling is different for the same words.

[1] SOHN, Ho-Min, "The Korean Language", Cambridge University
    Press, 1999.

---

From: "James Seng/Personal" <[email-masked]>
Date: Sun, 17 Feb 2002 12:39:51 +0800

this is the kind of discussion that should be carried out in unicore and
not in idn, really.

-James Seng