Subject: draft-klensin-net-utf8-06

From: John C Klensin <[email-masked]>
Date: Thu, 11 Oct 2007 15:49:22 -0400

Ok, this document just went into the queue and should appear
momentarily (probably before this message goes up).  It is
clearly less ready than the unicode-escapes one, partially
because of the addition of a lot of new or revised text, but I
hope we are converging: I will probably not have time to
generate another draft --except possibly for _very_ minor
changes-- prior to at least the middle of next month.

I've tried to deal with the comments from Frank, Stephane,
Clive, a few offlist comments, and the very extensive analysis
from Martin.  The comments that follow address some suggested
changes that I did not make plus a few things that seem to
deserve comments.

(1) We obviously have a tension between those who like the
history, context, and explanations and those who don't.  Oddly,
at least one person who asked for less then turned around and
complained that one decision wasn't explained.   Given people on
both sides, I've tried to strike a balance, but it clearly
reflects my/our editorial preferences.

(2) There are many ways in which the document could be more or
less radically reorganized.  Some might make it better.  Barring
really compelling arguments, we aren't going to do it: the move
of some material to appendices already fouled up some references
(I hope this iteration fixes all of them) and the payoff just
doesn't seem worth the additional time and iterations.

(3) This document affects protocols, which the unicode-escapes
one almost exclusively affects documentation and how the IETF
talks about things.  I have consequently tagged this one as
destined for Proposed Standard and the other one as BCP.  If you
disagree, this is probably the time to speak up.

(4) I've tried to clarify the text in several places, but I'm
not sure I got one bit adequately.  For CR, the rule is
   MUST NOT appear bare (i.e., MUST be followed by LF or NUL)
   SHOULD NOT appear with NUL
I believe that is consistent.

(5) I've added some words about both HT and FF.  One may need to
look in the appendices to find all of them.

(6) I've eliminated references to IDNA entirely, substituting
one to Stringprep where that appeared necessary.  

(7) The document is now linked to Unicode 5.0, with previous
versions, notably 3.2, referenced only non-normatively in
discussion.   I hope the revision was done correctly; please
tell me if it was not.  Moving to 5.0 eliminates any possible
need to go into detail about NFC changes after 3.2 but not the
need to caution against the risk of similar changes in the
future.

(8) While a few people advocated it, I can't see making the NFC
requirement a MUST.  First, there are certainly reasonable cases
in which it is not appropriate.  Second, if NFC on sending is a
MUST, then receivers have the right to expect that they will see
only NFC-normalized forms on input.  That assumption is
dangerous, since, in general, there is no way to know whether a
sender is following this spec.  I hope the text is now
consistent with that position.

best,
     john

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 12 Oct 2007 10:10:18 +0200

On Thu, Oct 11, 2007 at 03:49:22PM -0400,
 John C Klensin <[email-masked]> wrote 
 a message of 66 lines which said:

> I hope we are converging:

My opinion is that, yes, the document is better and more stable than
it was.

I've read and reviewed -06 and I find it suitable for Last Call, with
no serious pending issues.

> (2) There are many ways in which the document could be more or less
> radically reorganized.

The "historical" or "rationale" sentences (such as "In retrospect, one of
the advantages of ASCII [X3.4-1978] when it was chosen was that the
code space was full when the Standard was first published") could
certainly be more separated from the normative text but I do not see
this issue as a serious problem. The text seems clear.

> requirement a MUST.  [...] Second, if NFC on sending is a MUST, then
> receivers have the right to expect that they will see only
> NFC-normalized forms on input.  That assumption is dangerous, since,
> in general, there is no way to know whether a sender is following
> this spec.

While I agree with the SHOULD, I do not think the reason given above
is a good one. If we follow it, *all* MUST in RFCs could be changed to
SHOULD, in the name of the robustness principle. 

IMHO, the best reason for keeping the SHOULD ("applications should
send a NFC-normalized stream") is the argument about libraries that
the application does not control.

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 15 Oct 2007 10:17:33 +0200

John C Klensin wrote:

> less ready than the unicode-escapes one, partially because of
> the addition of a lot of new or revised text, but I hope we 
> are converging

Looking only at the diff, yes.  You've added a recommendation
to avoid private use code points, because they "do not have
standard definitions or normalization interpretations".

I think the latter is incorrect, they're by decree normalized.
Maybe what you want to say is that they can't have canonical
or compatible decompositions.  IMO the obvious "no standard
definition" alone is clearer and good enough to justify your
recommendation.

Not directly related to our draft, in 5.2 you write:
| The latter is important because an unassigned code point 
| always normalizes to itself.

Something's really odd with this in the Unicode standard:

For some unassigned code points it's "almost obvious" that
they'll never end up in any NFX, unless they also get the
non-character property.  What I have in mind are "obvious
gaps" in dingbats, mathematical symbols, and other blocks,
where the abstract character corresponding to the given
unassigned code point was already encoded elsewhere.  

One of the earlier examples is u+2073.  In theory they
could say "whatever we'll do with that code point, it
will be either a non-character, or have a canonical
decomposition, or stay as is (unassigned) forever".

Back to the draft, s/have been be tied/have been tied/
in 5.2. 
  
> I've added some words about both HT and FF.  One may need to

Fine now.  Over the weekend I read a related chapter
in the Unicode standard, and I think you have to 
mention LS u+2028 and maybe also PS u+2029 somewhere.

Admittedly banning LS might upset some folks.  IMO it's
similar to the NEL case, LS might be even more obscure.

OTOH I think that you shouldn't discuss IND u+0084,
it's dead.  TUS 5.0 says "formerly known as INDEX".

 Frank

---

From: John C Klensin <[email-masked]>
Date: Mon, 15 Oct 2007 13:49:34 -0400

> definition" alone is clearer and good enough to justify your

Frank, in both this and the notes below, we are, I think,
getting dangerously close to providing an interpretation and
profile of Unicode.   The bits of that that are necessary to get
line-endings nailed down and to make the NFC recommendation
clear are necessary but, when we start going into detail on how
individual code points or blocks are interpreted, we are getting
into territory that is very uncomfortable for me and, I believe,
should be uncomfortable for the IETF.

>...

The set of issues involved here is the reason that the "Stable
NF..." discussions started in the wake of some of the
pre-RFC4690 discussions a year or more ago.  I think we should
continue to try very hard to avoid having it turn into an IETF
problem.

> decomposition, or stay as is (unassigned) forever".

They could say lots of things in theory and perhaps in practice.
But they are also in the position in which, being human, it is
possible for them to make mistakes.  And, when they make a
mistake of sufficient importance, they reserve the right to
change things.  Becoming more dependent than absolutely
necessary on changes, or certain types of changes, being
forbidden, impresses me as unwise... and probably unfair to both
the IETF and Unicode communities.

> Back to the draft, s/have been be tied/have been tied/

Fixed in working draft for -07.   Unless more changes come up, I
won't post that draft except as part of a Last Call process --
too much clutter from too many drafts.
   
> mention LS u+2028 and maybe also PS u+2029 somewhere.

Sorry.  This gets into the picking and choosing of individual
code points that we have tried to avoid, for the reasons above
and otherwise.  Note that the prohibition on IND, etc., is part
of a general prohibition on C1 controls -- a clearly-defined
range.  The comments on those characters are just that, comments
(the text was bad on that -- applying a SHOULD NOT to those
characters and then a MUST NOT to the entire block.  That has
been fixed.

> it's dead.  TUS 5.0 says "formerly known as INDEX".

But it was taken into Unicode, if I recall, from ISO 8859, the
associated supplemental controls standard, and the various other
flavors of "Latin-NN".  It went into those documents, if my
memory is correct, precisely to correct ambiguities in the
interpretation of LF and CR (LS presumably arose when even that
wasn't sufficient in some environments).   I think mentioning it
because of the relationship to the pre-Unicode C1 controls might
be helpful to some people and is harmless at worst.  But I'd  be
happy to hear from others about this if people think it is
important enough to delay a last call still longer.

     john

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 16 Oct 2007 08:45:39 +0200

John C Klensin wrote:

> when we start going into detail on how individual code points or
> blocks are interpreted, we are getting into territory that is very
> uncomfortable for me and, I believe, should be uncomfortable for
> the IETF.

Yes, I'm not proposing to make the TUS "newline guidelines" worse,
they are already bad enough.  But it's easy (at least for me) to
forget the odd LS case, but "users" of the net-utf8 RFC have to be
aware that LS exists (in addition to the controls already covered
in your I-D).
 
Most "users" of net-utf8 will be protocol designers, they need to
know some relevant TUS dragons like LS, NFC, annotations, language
tags, etc.  [[And when EAI moves to standard track in some years
we have to do something about LS in display names and addresses,
before it hits say mailto-ter IRIs]].

 [various kinds of unassigned code points] 
> continue to try very hard to avoid having it turn into an IETF

+1  It might still make sense if the next TUS version identifies
such "unnormal unassigned gaps", it could simplify the IDNAbis
work (just an example).

> And, when they make a mistake of sufficient importance, they 
> reserve the right to change things.

In some areas.  Elsewhere they can't fix an obvious typo in a
character name s/KC/CK/, and had to invent a "normative alias"
mechanism to mitigate it.

> I'd be happy to hear from others about this if people think
> it is important enough to delay a last call still longer.

Let's start with the BCP on unicode-escapes, they don't depend
on the net-utf8 PS.  If you don't care about LS at the moment
you could also decree that it's a potential issue for a later
DS, but I think adding some "there be dragons" note about LS
to net-utf8 now could be simple.  Certainly no showstopper.

 Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 16 Oct 2007 10:11:14 +0200

On Mon, Oct 15, 2007 at 10:17:33AM +0200,
 Frank Ellermann <[email-masked]> wrote 
 a message of 51 lines which said:

> I think you have to mention LS u+2028 and maybe also PS u+2029
> somewhere.

<über-troll>We should drop the ambiguous, old, ASCII and
typewriter-specific CR and LF and standardize only the Unicode proper
characters, PS and LS for Net-UTF8.</über-troll>

#define TROLL 0
I suggest the following text in 2.1 "Definitions":

For compatibility reasons with the oldest practice, the Unicode
characters U+2028, Line separator, and U+2029, Paragraph separator,
are similarly NOT RECOMMENDED.

---

From: Bill McQuillan <[email-masked]>
Date: Tue, 16 Oct 2007 09:38:53 -0700

In section 2.1, I understand that the ONLY way to indicate the end of a
line, for text parsing purposes, is the pair <CRLF>. Any other appearance
of either CR or LF, whether or not allowed by a SHOULD, MUST NOT be
considered to be a "line-ending" and only can be considered as text mark
up.

For example, this<LF>
                  sentence is a single, malformed line.<CRLF>

Right?

A typo: Appendix B, bullet 4, penultimate line:

  interpreted as one or spaces, but, in general, it cannot be
                       ^
                       more

-- 
Bill McQuillan <[email-masked]>

---

From: "Tim Bray" <[email-masked]>
Date: Mon, 22 Oct 2007 03:23:21 -0700

My apologies for arriving late to the party.   I got on a
trans-Pacific flight with the draft in a browser window, but
unfortunately not much else of the conversation, so this may be
duplicative.

This review of the draft raises, with one exception, issues of style
and explication, and only one of these seems really serious to me: the
term "code point" needs to be introduced early in the document to
facilitate its use later; you really can't talk about Unicode
intelligently without flinging the term around.

I have one material point: standardizing on CRLF, rather than LF, for
line-ending, seems questionable:
- Lots of payload formats have their own rules about line ends; it's
not clear to me that coupling line-ending rules to character-encoding
rules is natural or beneficial.
- XML, which constitutes a noticeable proportion of Internet protocol
payloads, allows CRLF but requires parsers to convert it to LF before
passing it on to the software.  So this choice is going to waste (an
admittedly small amount of) space and require extra data processing;
the cost of not being able to pass a buffer-full of text to the
receiving program without performing the CRLF->LF mapping and thus
necessitating an additional copy of each byte seems to me worth
worrying about.

Now to style points:

Section 1.1

"Historically, the Internet has been largely ASCII-based"

I suggest "Historically, Internet protocols have been largely
ASCII-based."  The actual payload of the Internet has contained a high
volume of non-English (and thus necessarily non-ASCII) text for some
decades now.

For reasons of brevity and clarity, I'd simply remove the following,
which bulks up the paragraph and doesn't really add much value: "and
no longer need to deal with per-script standards for character sets
(e.g., one standard for each of Arabic, Cyrillic, Devanagari, etc., or
even standards keyed to languages that are usually considered to share
a script, such as French, German, or Swedish). "

Here's a proposed slight redraft of the text introducing the
encodings, which provides a bit of motivation, usefully introduces the
term "code point", and is more precise:

Unicode identifies each character by an integer, called its "code
point", in the range 0-0x10ffff.  These integers can be encoded into
byte sequences for transmission  in at least three standard and
generally-recognized encoding forms, all of which are completely
defined in The  Unicode Standard and the documents cited below:

- UTF-8 [RFC3629] defines a variable-length encoding that may be
applied uniformly to all code points.
- UTF-16 [RFC271] encodes the range of Unicode characters whose code
points are less than 65536 straightforwardly as 16-bit integers, and
provides a "surrogate" mechanism for encoding larger code points in 32
bits.
- UTF-32 (also known as UCS-4) simply encodes each code point as a
32-bit integer.

Section 2.

"Characters must be coded"... would "encoded" be more consistent with
usage elsewhere?

"  2.  Line-endings MUST be indicated by the sequence Carriage-Return
       (CR, U+000D) followed by Line-Feed (LF, U+000A), often known just
       as CRLF. "

See above.

"  5.  As suggested in Section 6 of RFC 3629, the Byte Order Mark
       ("BOM") signature MUST NOT appear at the beginning of these text
       strings."

It might be worth adding a note something along these lines: "The BOM
is useful in establishing the endian-ness of UTF-16 and UTF-32
encodings, but serves no useful purpose in the context of UTF-8."

"Systems conforming to this specification MUST NOT transmit any string
   containing any code point"

- This is the first appearance of the term "code point", which needs
defining.  The edit I suggested above for section 1 would fix this.
- Also, this sentence (which is important) seems to sit uneasily in
this section about NFC; I would hoist it and promote it to near the
top of section 2.

Section 6.

"The same security issues that apply to UTF-8, as
   discussed in [RFC3629], could be argued to apply to this standard form,"

That last clause is klunky, how about: "It could be argued that the
same issues that apply to UTF-8, as discussed in [RFC3629], apply to
this specification."

Generally: I'd ruthlessly whack about 80% of the history and suchlike
explication.  Future consumers of this document, of which I predict
there will be many, will need to consult and then cite the meat:
"Unicode characters, UTF-8 encoding, NFC normalization upstream, be
careful about Unicode versioning", and will benefit from some
explanation of the technical advantages of the choices.  The rest is
of questionable value.  -Tim

---

From: Julian Reschke <[email-masked]>
Date: Mon, 22 Oct 2007 12:36:36 +0200

> encodings, but serves no useful purpose in the context of UTF-8."

The BOM doesn't server any purpose *here* (because we know it's UTF-8).

In general, a BOM starting a UTF-8 encoded octet stream can be a useful 
hint to the recipient if out-of-band encoding information has been lost 
(such as when storing an HTTP response body in a simple file).

Best regards, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 22 Oct 2007 13:45:18 +0200

Tim Bray wrote:

> - Lots of payload formats have their own rules about line ends

The draft is about Internet protocols based on or closely related
to "telnet" like "whois", it's not about payload formats like XML.

 [your "code point" proposal]
> introduces the term "code point", and is more precise:

>| Unicode identifies each character by an integer, called its
>| "code point", in the range 0-0x10ffff.

That might backfire, not all code points are characters, and not
all abstract characters can be encoded by a single code point.
The glossary in TUS 5.0 offers:

| Code point: Any value in [...] the range of integers from 0
| to 10FFFF [hex.]. 
 
 [BOM]
> It might be worth adding a note something along these lines: "The
> BOM is useful in establishing the endian-ness of UTF-16 and UTF-32

It's an important "signature" for plain text files on platforms 
where UTF-8 is not the local codepage.  One of these platforms is
quite popular. :-)  For the purpose of the net-utf8 I-D just saying
NO to BOMs is IMO good enough, RFC 3629 contains the fine print.

> I'd ruthlessly whack about 80% of the history and suchlike
> explication.

Oops, no, the history is essential to understand old telnet issues
like IAC without reading pre-historic RFCs with numbers below 821.

> Future consumers of this document, of which I predict there will
> be many, will need to consult and then cite the meat

I think the future consumers are standard developers and protocol
designers trying to update say RFC 3912.  Appendix C lists future
work, indirectly that defines the main audience for net-utf8.

 Frank

---

From: Tony Finch <[email-masked]>
Date: Mon, 22 Oct 2007 20:06:31 +0100

> like IAC without reading pre-historic RFCs with numbers below 821.

I also like the historical appendix.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: NORTHERLY 5 AT FIRST IN EAST, OTHERWISE SOUTHEASTERLY 4 OR
5 INCREASING 6 OR 7. MODERATE OR ROUGH. RAIN LATER. GOOD, OCCASIONALLY
MODERATE LATER.