Subject: Comments on draft-klensin-net-utf8-06

From: Marcos Sanz/Denic <[email-masked]>
Date: Tue, 16 Oct 2007 16:45:38 +0200

To my eyes the document is in good shape, but it still leaves one of the 
issues at stake partially open:

Section 2, bullet 2 says "CR SHOULD NOT appear except when followed by 
LF". The last paragraph of the section 2 says "CR MUST NOT appear unless 
it is immediately followed by LF (...) or NUL". To me the first of these 
statements is much less restrictive than the second.

Nitpicking:

* Section 1.1: s/variable length/variable-length/ for coherence with other 
text appearances

* Section 3: s/to convert all canonically equivalent sequences a single 
unique form/to convert all canonically equivalent sequences into a single 
unique form/

* Section 4: The definition of the normalization stability is misleading, 
actually even wrong.

  Old text: 

 That is, if a string does not contain any unassigned
 characters, and it is normalized according to NFC, it will always be
 normalized according to all future versions of the Unicode Standard.

  Suggested text:

 That is, if a string does not contain any unassigned
 characters for a given version of Unicode, and it is normalized according 
to
 the definition of NFC in that version, it will always result in the same
 normalized string according to all future versions of the Unicode 
Standard.

* Section 4: "the string order of RFC 3629". It's not very clear to me 
what is meant with this. Byte order? Sorting order?

* Section 4: I would drop the last paragraph, since it is a repetition of 
what is exhaustively explained in section 5.2. I got a parsing error at 
the last sentence of that paragraph anyway.

* Section 5.2: s/[RFC3454])/[RFC3454]),/

* Section 5.2, bullet 4: "This process has been discussed in the Unicode 
Consortium under the name 'Stable NFC'". That might very well be but the 
only hits I get when googling for that string are this very draft and some 
contributions of the draft author to some mailing lists. So I cast doubt 
on the utility of introducing this new term here which is a pointer to 
nowhere. Is this again referring to the normalization stability policy of 
unicode? http://unicode.org/standard/stability_policy.html

Best regards,
Marcos Sanz

---

From: John C Klensin <[email-masked]>
Date: Tue, 16 Oct 2007 11:51:27 -0400

--On Tuesday, 16 October, 2007 16:45 +0200 "Marcos Sanz/Denic"
<[email-masked]> wrote:

> To my eyes the document is in good shape, but it still leaves
> one of the  issues at stake partially open:
> 
> Section 2, bullet 2 says "CR SHOULD NOT appear except when
> followed by  LF". The last paragraph of the section 2 says "CR
> MUST NOT appear unless  it is immediately followed by LF (...)
> or NUL". To me the first of these  statements is much less
> restrictive than the second.

Marcos (and others who have been trapped by this),

While I would welcome suggestions about other text and ways to
organize this, the two statements are perfectly consistent with
each other, not a more restrictive/ less restrictive
contradiction.

It won't fit with the existing text and layout in this form, but
what is being said is:

   if there is a CR, 
       it MUST be followed by either LF or NUL
       however, NUL SHOULD be avoided too, so 
       CR LF is the only recommended context in which CR 
          SHOULD be used.

> Nitpicking:
> 
> * Section 1.1: s/variable length/variable-length/ for
> coherence with other  text appearances

fixed in working draft.

> * Section 3: s/to convert all canonically equivalent sequences
> a single  unique form/to convert all canonically equivalent
> sequences into a single  unique form/

s/into/to/, but fixed.
 
> * Section 4: The definition of the normalization stability is
> misleading,  actually even wrong.
> 
>   Old text: 
> 
>  characters, and it is normalized according to NFC, it will
> always be  normalized according to all future versions of the
> Unicode Standard.
> 
>   Suggested text:
> 
>  characters for a given version of Unicode, and it is
> normalized according  to
>  the definition of NFC in that version, it will always result
> in the same  normalized string according to all future
> versions of the Unicode  Standard.

The text that was used was supplied by Mark Davis after my first
attempt didn't come out right.   I'm happy to put anything in
there that people agree to be correct, but please sort this out
with him and/or other UTC members.

> * Section 4: "the string order of RFC 3629". It's not very
> clear to me  what is meant with this. Byte order? Sorting
> order?

3629 specifies a byte order (in section 4).  It does not address
or mention sort order except to note (in the introduction) that
UTF-8 preserves it and that sort order based on code point
sequence is likely to be fairly useless.

I _think_ I would welcome text to clarify this but please note
that it is not likely to be possible to use this spec without
understanding and following 3629 (that is what the normative
reference is all about).  So I am loathe to cover things that
are well-covered in 3629 lest more confusion be created.

> * Section 4: I would drop the last paragraph, since it is a
> repetition of  what is exhaustively explained in section 5.2.
> I got a parsing error at  the last sentence of that paragraph
> anyway.

Hmm.  It parses for me.   But I agree about the redundancy,
except for that last sentence, which makes a normative assertion
about this specification that does not appear in Section 5.
That last sentence could be restated, less formally, as:

	If one encounters a UTF-8 string in a protocol, and its
	syntax and properties are not specifically defined, then
	it is reasonable to assume that it conforms to this
	specification.

That assumption might, of course, be wrong, which is one reason
it is important to be careful about having string-receivers
assume that the string-transmitter normalized it.

I would appreciate input from others about what to do about this.

> * Section 5.2: s/[RFC3454])/[RFC3454]),/

It is correct as written ("..stored in normalized form if...")
although the long parenthetical note is a little obnoxious.
Suggestions welcome.
 
> * Section 5.2, bullet 4: "This process has been discussed in
> the Unicode  Consortium under the name 'Stable NFC'". That
> might very well be but the  only hits I get when googling for
> that string are this very draft and some  contributions of the
> draft author to some mailing lists. So I cast doubt  on the
> utility of introducing this new term here which is a pointer
> to  nowhere. Is this again referring to the normalization
> stability policy of  unicode?
> http://unicode.org/standard/stability_policy.html

No.   I will recheck the terminology.

I'm going to hold the document for a few days before re-posting
in the hope of getting comments from others.

thanks and regards,
     john

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 17 Oct 2007 08:21:08 +0200

> I _think_ I would welcome text to clarify this

Please simplify the remark and remove one "that":

-| Were Unicode to be changed in a way that violated these
-| assumptions, i.e., that either invalidated the string order
-| of RFC 3629 or that that changed the stability of NFC as
-| stated above, this specification would not apply.

+| Were Unicode to be changed in a way that violated these
+| assumptions, i.e., that changed the stability of NFC as
+| stated above, this specification would not apply.

UTF-8 as specified in STD 63 is stable.  

> So I am loathe to cover things that are well-covered in 
> 3629 lest more confusion be created.

Yes, just don't mention the "byte-value lexicographic sorting
order of UTF-8 strings", it's covered in STD 63, and besides
not very interesting.

>> I got a parsing error at  the last sentence of that paragraph

Indeed, that paragraph is unnecessary.  I also can't parse its
first sentence.

> it is reasonable to assume that it conforms to this

I still don't understand this.  What is an UTF-8 string with
"unspecified syntax" ?  STD 63 specifies the syntax of UTF-8,
anything not following this syntax is invalid.

The net-utf8 I-D doesn't specify any default properties, what
is an assumption that "unspecified properties" conform to
net-utf8 supposed to do ?  

If you're talking about unassigned code points please say so.
In that case it's covered in 5.2, and you can simply delete
the last paragraph of section 4.

> I'm going to hold the document for a few days before 
> re-posting in the hope of getting comments from others.

Please update the [NFC] reference, s/March 2005/2006-10-12/
for the version belonging to TUS 5.0.

 Frank

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Thu, 18 Oct 2007 10:44:30 +0200

John,

> organize this,

What about making a forecasting note right away under bullet 2:

OLD TEXT:

  CR SHOULD NOT appear except when followed by LF.

SUGGESTED TEXT:

  CR SHOULD NOT appear except when followed by LF. The other only allowed 
appearance is in the combination CR NUL, which is not recommended (see 
note at the end of this section).

There is a similar contradictory (less restrictive at the beginning and 
suddenly more restrictive at the end) situation in bullet 3. The first 
sentence goes

  [...] control characters (U+0000 to U+001F and U+007F to U+009F) SHOULD 
generally be avoided

vs the last sentence

  the so-called "C1 Controls" (U+0080 through U+009F) MUST NOT appear

This is the nightmare for any implementor. The double negation doesn't 
provide for clarity either. What about changing 

OLD TEXT:

  control characters (U+0000 to U+001F and U+007F to U+009F) SHOULD 
generally be avoided.

SUGGESTED TEXT

  control characters (U+0000 to U+001F and U+007F) SHOULD NOT be used and 
the so-called "C1 controls" (U+0080 to U+009F) MUST NOT be used.

Then you can drop the last sentence of the bullet.

> attempt didn't come out right.

He'll certainly know better.

> I _think_ I would welcome text to clarify this

I support Frank's suggestion, which I'll copy here again for clarity:

And again, UTF-8 as specified in STD 63 is stable.

> So I am loathe to cover things that

We fully agree. I only think that the reference in the old text is not 
necessary.

>    it is reasonable to assume that it conforms to this

The old formulation mentioned "unidentified UTF-8 strings", the new 
formulation mentions a UTF-8 string with syntax and properties "not 
specifically defined". I am sure you have something in mind, but it still 
doesn't get through. And you are aiming at a normative assertion, then 
normative language should be used and not something vague like "it is 
reasonable to assume".

Thanks and best regards,
Marcos

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 18 Oct 2007 10:11:27 +0100

>   the so-called "C1 Controls" (U+0080 through U+009F) MUST NOT appear
[...]

Agreed. In addition, at the end of item 2 of section 2:

OLD TEXT:

The newer control characters IND (U+0084) and NEL
("Next Line", U+0085) might have been used to disambiguate the
various line-ending situations, but, because their use has not
been established on the Internet and many protocols require CRLF,
they SHOULD NOT be used.

SUGGESTED TEXT:

The newer control characters IND (U+0084) and NEL
("Next Line", U+0085) might have been used to disambiguate the
various line-ending situations, but, because their use has not
been established on the Internet, because many protocols require CRLF,
and because they fall within the "C1 Control", they MUST NOT be used.

Nit-pick: right at the end of section 2, NUL should be described as U+0000
and not as X'00' (a notation which isn't well-defined). Or, if you are
trying to distinguish octets from characters, use the RFC 4234 notation of
%x00. Similarly, at the end of appendix B, IAC should be %xFF.

Incidentally, %xFE is also not a problem for UTF-8, and %xF5-FD are
probably not problems.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |