Subject: [idn] Editorial comments on stringprep

From: Erik Nordmark <[email-masked]>
Date: Fri, 3 May 2002 20:57:07 +0200 (CEST)

draft-hoffman-stringprep-02.txt
-------------------------------

I've reviewed the stringprep document to make sure it is
ready for IETF last call.

Here are some editorial comments that it would be useful to
fix before the last call.

The nameprep document contained some text which I think should belong
in stringprep. The text in nameprep (in two different contexts)
was:
	The lists in Appendix E MUST be used by
	implementations of this specification. If there are any discrepancies
	between the lists in Appendix E and subsections below, the lists in
	Appendix E always takes precedence.

	This profile lists the unassigned code points in the
	range 0 to 10FFFF for Unicode 3.1 in
	Appendix F. The list in Appendix F MUST be used by implementations of
	this specification. If there are any discrepancies between the list in
	Appendix F and the Unicode 3.1 specification, the list in Appendix F
	always takes precedence.

The document needs to make it more clear that it will be an exception
when profiles define additional tables. The intent is that stringprep
define the tables and that  the stringprep profiles will select among 
those defined tables. Adding some text to section 1.2
about this would be helpful for folks defining profiles. E.g.
add "(in exceptional cases)" for those items in the bulleted list.

It isn't obvious that table B.2 and B.3 derive directly
from the Uncode tables when in fact they do. 
I think it would be useful to state this.
In fact the text about "NormalizeWithKC(Fold...)" in section 3.2 describes
how table B.2 is created, but it needs to explicitly say this.
Likewise it would be helpful to say that B.3 is created from the Unicode
table for case folding (CaseFolding.txt??)

The document needs a table of content based on the rfc-editors
criteria and the length of the document.

Section 1: Is there a missing "would" before "cause an error"?

The RFC 2119 citing text does not include all the 2119 words.
The document uses "SHOULD NOT" and "MUST NOT" which are not included.
The safest thing is to use the boiler plate that lists all the 2119 words.

Section 1.2:
	- The tables
	froom this document of characters that are prohibited as
s/froom/from/

Section 3: talks about "mapped to nothing" but the tables refer to
this as "map out". Either use a single term or explicitly state e.g.
in section 3 that they mean the same. Latter could perhaps be done
by adding "map out" in parenthesis after "map to nothing".

3.1 says "because their presence or absence should not
make two strings different".
The statement makes sense in the context of protocol elements
(for the protocols whose stringprep profile select this mapping)
but it might not make sense for text in general.
Thus I think it makes sense adding this context to the description.

Section 3.2 talks about the tradition of using lowercase
but my understanding is also that there is a technical argument 
in that downcasing is more well-defined in Unicode than upcasing.
Is this not the case? If so if would make sense to add that to
the description of the motivation.

Section 3.2 missing close parenthesis after "statuses C, F, and I."

Section 3.2 talks about "the table is stable from that point on."
But it doesn't actually state that this is how the table in B.2 is generated.
Perhaps this should be made explicit both here and at the top of
appendix B.2.

Section 3.2 and elsewhere: I don't think "unupdated" is a word.
How about 
	in both those systems which have been updated and those which have not.

Section 4 talks of "three options for Unicode normalization"
but there are only two bullets below that text.

Section 4 doesn't refer to which Uncode table to use (e.g. the filename)
In fact, partially because of this I originally read the document as 
if table B.2 including the NFKC mapping when it in fact does not.
This needs to be made more clear.

Section 5.2. Says
	displayed. Note that additional 
	control characters (U+0000 through U+001F, and U+007F) are not listed
	below. They are listed in Appendix C-2.
The second sentence makes one believe the ASCII control characters are
in appendix C.2 when in fact they are not.

The references to the parts of appendix C use "C-<n>" but the
appendices are named "C.<n>".

Section 5.3. The argument for the replacement character is quite weak -
it does have some semantics.
Isn't there a stronger argument that one would never expect to use
this character as an *input*? The intent is to only use it for
*output*. As such it doesn't make sense to have it be used
in a protocol element.

Section 5.4 missing "in" before "the Proplist.txt file ..."

Section 6 talks of "names of SNMP objects" in multiple places.
But SNMP objects are named by OIDs - not character strings. Drop it?

Section 6 says
	Using two different policies for where unassigned code points can
	appear prevents the need for versioning in protocols that use
I think "removes" is better than "prevents" here.

Section 6 seems to use the term "query" in some places
and "request" in others. Please make it use a consistent term.

Section 6.2 seems to use the term "correct" with two rather different
meanings:
1. The correct answer is returned (positive or negative answer)
2. In order to prevent false positives the "correct" approach is
   to return false negatives in some cases.

I think #2 should use a different term. ("predictable"? conservatively? I'm
open to suggestions).
This appears in two places: the XY vs. YX and the X vs. nX.

Section 7 reads like the designers were just too lazy 
to handle similar looking characters, which clearly isn't the case. 
Thus I think it makes sense to add some language saying that
similar-looking is a very hard problem and is (believed to be) impossible
to solve without some additional context, and that context is
not available in DNS lookups.

Appendix E.
I think it makes sense to move this to a section i.e. before the character
tables.

"MUST be an RFC" is not one of the choices in RFC 2434.
"IETF Consensus" seems closest to what the text says, but "IESG approval"
might also match. And add a citation to RFC 2434.

Split normative vs. non-normative references.

---

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 3 May 2002 21:24:30 -0700

In the midst of some very well-thought-out comments, Erik Nordmark

> the description of the motivation.

I can't think of anything that would lead to that impression, in Unicode
or anywhere else.  Uppercase, lowercase, and titlecase mapping are all
handled on a completely equal footing in Unicode.

-Doug Ewell
 Fullerton, California

---

From: Erik Nordmark <[email-masked]>
Date: Sat, 4 May 2002 10:02:08 +0200 (CEST)

> handled on a completely equal footing in Unicode.

I didn't mean "undefined" as poorly specified.
Sorry for my lack of Unicode clue - perhaps I've misunderstood something.
But I recall hearing that given two arbitrary Unicode code points X and Y
one could do a comparison of
	MaptoUpperCase(X) and MaptoUpperCase(Y)
or
	MaptoLowerCase(X) and MaptoLowerCase(Y)

Both of these are well-defined, but the results of the comparison
will be different in some cases. I haven't verfied this but I think
U+0131 and U+0049 would compare as equal in the first case
but not in the second case.

But I now realize that the above isn't germane to the text on which I
commented. The above is just an argument that the spec must pick one or the
other techniques for case insensitive comparison.

Hmm - I don't recall if Patrik has some additional arguments for why
lowercase is better in his plenary presentation in Minneapolis.

  Erik

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 04 May 2002 11:17:49 +0200

> lowercase is better in his plenary presentation in Minneapolis.

Historically, Unicode Consortium has always stated that mapping to
lowercase were more consistent and better than mapping to uppercase. At
least for Unicode before version 2. Don't remember any exact reasons, but
we can dig it up if you want us to.

There are also a number of codepoints which are lowercase which doesn't
have uppercase versions.

Last, some codepoints (like the german sharp-s, ß) turns to "SS" in
uppercase, and my guess is (with my limited knowledge of German, only 2
years of studies) that one when comparing don't want that similarities.

Anyway, my point is that previously UTC has always recommended mapping to
lowercase, and people seems to be happy with it.

And, personally, I rather see bq-asdqwe123 than BQ-ASDQWE the few times I
hope I see a domain name used in protocols natively in its ACE encoding.

    paf

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 4 May 2002 08:34:18 -0700

For most case-insensitive binary comparison, the recommendation is to
use case folding, as defined in the Standard. For more info, see
http://www.unicode.org/unicode/reports/tr21/.

Mark

—————

Γνῶθι σαυτόν — Θαλῆς
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "Patrik Fältström" <[email-masked]>
To: "Erik Nordmark" <[email-masked]>; "Doug Ewell"
<[email-masked]>
Cc: "Erik Nordmark" <[email-masked]>; <[email-masked]>
Sent: Saturday, May 04, 2002 02:17
Subject: Re: [idn] Editorial comments on stringprep

Historically, Unicode Consortium has always stated that mapping to
lowercase were more consistent and better than mapping to uppercase.
At
least for Unicode before version 2. Don't remember any exact reasons,
but
we can dig it up if you want us to.

There are also a number of codepoints which are lowercase which
doesn't
have uppercase versions.

Last, some codepoints (like the german sharp-s, ß) turns to "SS" in
uppercase, and my guess is (with my limited knowledge of German, only
2
years of studies) that one when comparing don't want that
similarities.

Anyway, my point is that previously UTC has always recommended mapping
to
lowercase, and people seems to be happy with it.

And, personally, I rather see bq-asdqwe123 than BQ-ASDQWE the few
times I
hope I see a domain name used in protocols natively in its ACE
encoding.

    paf

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 4 May 2002 10:26:25 -0700

Patrik Fältström <[email-masked]> wrote:

> At least for Unicode before version 2. Don't remember any exact
> reasons, but we can dig it up if you want us to.

and later:

> Anyway, my point is that previously UTC has always recommended
> mapping to lowercase, and people seems to be happy with it.

I couldn't remember UTC ever saying such a thing, so when Mark Davis
<[email-masked]> wrote:

> For most case-insensitive binary comparison, the recommendation is
> to use case folding, as defined in the Standard. For more info, see

naturally I took a look.  The only thing I found that could be construed
as a preference for lowercase was in the CaseFolding details:

> Each equivalence class is completely disjoint from all the others,
> and together they form a partition of the entire Unicode code space.
> From each class, one representative element (a single lowercase
> letter where possible) is chosen to be the common form.
> [CaseFolding] thus contains the mappings from other characters in
> the equivalence characters to their common forms.

This passage describes an *internal process* only.  This has nothing to
do with the Unicode Consortium or Technical Committee preferring
lowercase forms over uppercase for user-visible operations, and it
definitely does *not* claim that mapping to lowercase is more consistent
or "better" than mapping to uppercase.  There is no explanation given
for the phrase "a single lowercase letter where possible" to explain why
lowercase was selected.  It appears to be an arbitrary choice.

> doesn't have uppercase versions.

Which ones?  I can think of a character that looks uppercase but has no
lowercase form (U+04C0 CYRILLIC LETTER PALOCHKA).  But such letters,
despite their appearance, are neither uppercase nor lowercase; they are
caseless, and immune to the effects of any casing operation.

> uppercase, and my guess is (with my limited knowledge of German,
> only 2 years of studies) that one when comparing don't want that
> similarities.

German speakers are forced to deal with that mapping every day.  It is a
natural part of the language.

> times I hope I see a domain name used in protocols natively in its
> ACE encoding.

No argument there.  All-lowercase is widely recognized as being easier
to read than all-uppercase, primarily because of the greater variation
in letterforms.  But again, there doesn't seem to be any evidence that
the Unicode Consortium has made any of the claimed statements about
"preferring" lowercase or about the mapping to lowercase being more
"consistent."  Please dig up the relevant references, if possible.

-Doug Ewell
 Fullerton, California

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 4 May 2002 20:59:11 +0000

I don't think it matters whether Unicode has ever expressed any
preference for lowercasing versus uppercasing.  What matters is that
Unicode defines a case folding algorithm that happens to use lower
case, and Stringprep simply uses it.  Although Unicode defines both
lowercasing and uppercasing operations, the only case folding operation
it defines uses lower case, so we really have no choice unless we want
to define our own case folding algorithm (which we don't).

(The case folding operation is designed for doing case-insensitive
comparisons.  The lowercasing operation is not.)

AMC

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 4 May 2002 15:14:57 -0700

> lowercase was selected.  It appears to be an arbitrary choice.

The choice of whether to use upper or lower for folding was originally
somewhat arbitrary; either could have been chosen.* However, the
choice of lowercase was made for folding operations, and was
encapsulated into the case folding mappings in the UCD. This does not
indicate an fundamental sort of 'preference' by the consortium for
either case in general text processing, but for a folding operation
one must chose, and the choice needs to be stable.

*The main reason for choosing lowercase was that there are more
characters with lowercase-only than uppercase-only. These characters,
mostly IPA, are lowercase in form but have no corresponding uppercase
equivalent. Occasionally corresponding uppercase characters are
added -- when the character is added to the orthography of some
language and ceases to be only IPA (when this happens, it is typically
for an African language), and an uppercase is required.

Mark

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 05 May 2002 10:02:27 +0200

> <[email-masked]> wrote:

I said _at_least_before_version_2_.

In 1995 I had a long discussion with Unicode Consortium when I worked at
Bunyip Information System how to do normalization and case folding
together. Their response was to covert to lower case, and our
implementation of Whois++ named Digger also ended up in a paper which was
presented at a Unicode Conference around 1995-1996.

I also saw Mark only refering to case folding, not lower case perticularly,
and that's why I talk about handwaving, historical artifacts etc etc.

> caseless, and immune to the effects of any casing operation.

Quote from page 142 in Unicode version 3 book:

"Also, because many characters are really caseless (most of the IPA block,
for example), uppercasing a string does not mean that it will no longer
contain any lowercase letters."

I only quote the text.

Yes, when reading it, one might think it should have been written "...that
it will only contain uppercase letter." but it doesn't.

> natural part of the language.

I know. I just gave it as one example where I _thought_ people from Germany
rather wanted lower case than upper case. I wait until I hear from someone
from Germany saying they prefer mapping to upper case before I say
something else.

We talk about what is preferred. Not wether people are used to.

> "consistent."  Please dig up the relevant references, if possible.

After some digging, I see the paper Philippe and I wrote was presented on
A5 on Unicode Conference number 9, on september 5, 1996:

   Text Searching Across Multiple Character Sets in Unicode
   Philippe Boucher, Senior Programmer,
   Bunyip Information Systems, Montreal, Quebec, Canada

I can not find the document though.

   paf

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 6 May 2002 12:07:50 +0200 (CEST)

> we can dig it up if you want us to.

*If* there is such additional reasons it wouldn't
hurt to state that as part of the motivation in the draft.

  Erik

---

From: Dave Crocker <[email-masked]>
Date: Fri, 10 May 2002 14:01:44 -0700

>hurt to state that as part of the motivation in the draft.

Erik, et al,

IETF specifications sometimes explain the reasons for particular choices, 
but mostly they don't.  On the average, we explain far more than ITU or ISO 
specifications, but far less that a complete historical narrative.

I think that the typical guidance is to provide whatever explanation that 
will help people understand WHAT the specification is, and to avoid 
explaining the WHY behind a choice unless the choice is so unusual readers 
will be distracted until they understand the reason for the choice.

That is, a specification could explain every decision in great detail, and 
it always is helpful to explain reasons... or is it?  All that explanation 
can get in the way of reading the actual specification.

And the explanation for my sending this long-winded note is that the recent 
postings on the mailing list frankly seem to be focusing on minor 
items.  All are valid but many/most are minor.

Your formal editorial comments were wonderfully diligent.  Far better than 
the IETF norm for Area Director feedback.  In such documents, adding minor 
suggestions is fine, because they are only part of a comprehensive set of 
comments.

The issue is follow-on discussion.  We should leave handling of the minor 
items to the document editors.  The minor items do not really affect 
normative content of the specifications.

I find myself wishing that the online discussion would focus only on the 
essentials, in order to get the specifications published sometime soon.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 13 May 2002 15:29:16 +0200 (CEST)

Dave,

I agree with the essence of your note.
This issue is a minor editorial one.
So let's move on.

  Erik

>----- Begin Included Message -----<

Date: Fri, 10 May 2002 14:01:44 -0700
From: "Dave Crocker" <[email-masked]>
Subject: Re: [idn] Editorial comments on stringprep
To: "Erik Nordmark" <[email-masked]>
Cc: [email-masked]

Erik, et al,

d/

----------

>----- End Included Message -----<