Subject: Form feed in Net-UTF8? (Was: FWD: Re: Comments on Unicode Format for Network Interchange

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 5 Oct 2007 17:12:27 +0200

Doug Ewell said:

> > I recommend that the control code FF (form feed, U+000C) be
> > likewise permitted.  The form feed function is well known and well
> > defined in almost all printing functions, and all RFCs issued in
> > modern times use FF to separate pages.  It would ironic indeed for
> > the Internet-Draft to retain the requirement that form feeds
> > "SHOULD NOT be used unless required by exceptional circumstances"
> > while advancing toward publication as an RFC, complete with form
> > feeds!

And I did not see anywhere a discussion of this specific point. I
notice that draft-klensin-net-utf8-04 still disallows (actually, it is
a SHOULD NOT in section 2.1) form feeds and I'm not sure of the
rationale. Why end-of-lines and not end-of-pages?

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Oct 2007 11:57:59 -0400

--On Friday, 05 October, 2007 17:12 +0200 Stephane Bortzmeyer
<[email-masked]> wrote:

>> > likewise permitted.  The form feed function is well known
>> > and well defined in almost all printing functions, and all
>> > RFCs issued in modern times use FF to separate pages.  It
>> > would ironic indeed for the Internet-Draft to retain the
>> > requirement that form feeds "SHOULD NOT be used unless
>> > required by exceptional circumstances" while advancing
>> > toward publication as an RFC, complete with form feeds!
> 
> And I did not see anywhere a discussion of this specific
> point. I notice that draft-klensin-net-utf8-04 still disallows
> (actually, it is a SHOULD NOT in section 2.1) form feeds and
> I'm not sure of the rationale. Why end-of-lines and not
> end-of-pages?

Because line breaks are required even for unformatted streams.
FormFeeds are not.  

But I find the above arguments reasonably persuasive and will
put FF back in unless others feel strongly about this.

    john

---

From: Martin Duerst <[email-masked]>
Date: Sat, 06 Oct 2007 12:12:17 +0900

FF is a tricky one. It's definitely used in RFCs, and can
be helpful for printing (although less and less so). But
in a protocol context, where pages are nonexisting, it will
only cause trouble. So I'd suggest that the SHOULD NOT is kept,
but give a bit more detail. For some shot at wording, I'd change:

to something like:

"SHOULD NOT be used unless the protocol or format has a need
to express page breaks in plain text (as e.g. done in the format
for RFCs)."

Regards,   Martin.

>put FF back in unless others feel strongly about this.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Oct 2007 04:17:35 -0400

--On Saturday, 06 October, 2007 12:12 +0900 Martin Duerst
<[email-masked]> wrote:

> to express page breaks in plain text (as e.g. done in the
> format for RFCs)."

Martin, since I already shipped -05 and, thanks to the new
automated system, it has been posted, please see how you like
the treatment there.   I think we agree on the principles, so
would welcome suggestions for further tuning.

While I could live with the text you propose above, it seems to
me to lead onto a slippery slope.  Try substituting, for "page
breaks" above, terms like "alert sounds", "visual emphasis such
as highlighting or colored characters", or "flashing lines".
FormFeeds are much more frequent in text, and do appear in RFCs,
but the principles are, I think, the same.

      john

---

From: Dave Crocker <[email-masked]>
Date: Sun, 07 Oct 2007 08:18:53 -0700

> but the principles are, I think, the same.

Seems like the discussion is really distinguishing among different classes of 
use, where each might be labeled distinctly and have variants of permitted 
characters.

(I'm not expressing an opinion about whether this is a good thing to do, but 
merely noting that I think it is what is driving the discussion.

So the set for "data stream", such as protocol commands, needs minimal format 
effectors.  Whereas "document stream" probably needs all the format effectors 
and other visual indicators.

The reference to beeps suggests another role, which I suppose is a superset of 
format effectors and might be classed as "user interface" or somesuch.

Mumble...

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 8 Oct 2007 11:28:31 +0200

On Fri, Oct 05, 2007 at 05:12:27PM +0200,
 Stephane Bortzmeyer <[email-masked]> wrote 
 a message of 18 lines which said:

> I notice that draft-klensin-net-utf8-04 still disallows (actually,
> it is a SHOULD NOT in section 2.1) form feeds and I'm not sure of
> the rationale. Why end-of-lines and not end-of-pages?

The new text in -05 is fine for me:

       Other than CR, LF, Space (SP, U+0020), and, with caution, FF
       (U+000C), control characters (U+0000 to U+001F and U+007F to
       U+009F) SHOULD generally be avoided.  FF should be used only with
       caution: if its use assumes a page length, such assumptions may
       not be appropriate in international contexts (e.g., considering
       8.5x11 inch paper versus A4).  Other control characters are used
       to affect display format, control devices, or to structure files.
       None of those uses is appropriate for streams of plain text.  In
       particular, the so-called "C1 Controls" (U+0080 through U+009F)
       MUST NOT appear.

---

From: Bill McQuillan <[email-masked]>
Date: Mon, 8 Oct 2007 09:39:45 -0700

> use, where each might be labeled distinctly and have variants of permitted

It seems to me that this draft is trying to cover two separate areas of
concern:

1 - How to express information (UTF-8, NFC, CRLF line-endings, no BOM).

2 - What information to express.

Is it necessary to tell prospective users of Network Text which characters
are allowed for their application? Wouldn't it be simplier to merely
specify the encoding to be used and then mention that use of things like
control characters should be specified within the protocol definition?

Since the use of "text" occurs in many varied contexts, not all of which
are merely for display, I think that alerting users to the problems of
control characters and making it their responsibility to address them is
sufficient.

-- 
Bill McQuillan <[email-masked]>

---

From: John C Klensin <[email-masked]>
Date: Mon, 08 Oct 2007 13:03:45 -0400

--On Monday, October 08, 2007 9:39 AM -0700 Bill McQuillan 
<[email-masked]> wrote:

>>> While I could live with the text you propose above, it seems
>>> to me to lead onto a slippery slope.  Try substituting, for
>>> "page breaks" above, terms like "alert sounds", "visual
>>> emphasis such as highlighting or colored characters", or
>>> "flashing lines". FormFeeds are much more frequent in text,
>>> and do appear in RFCs, but the principles are, I think, the
>>> same.
>> Seems like the discussion is really distinguishing among
>> different classes of use, where each might be labeled
>> distinctly and have variants of permitted characters.
>
> It seems to me that this draft is trying to cover two separate
> areas of concern:
>
> Is it necessary to tell prospective users of Network Text
> which characters are allowed for their application? Wouldn't
> it be simplier to merely specify the encoding to be used and
> then mention that use of things like control characters should
> be specified within the protocol definition?
>
> Since the use of "text" occurs in many varied contexts, not
> all of which are merely for display, I think that alerting
> users to the problems of control characters and making it
> their responsibility to address them is sufficient.

Bill, Dave,

To me, the purpose of trying to develop an IETF specification is 
to promote interoperability.  If you send me encoded characters 
that I don't know how to interpret, or don't reliably know how 
you intend that I interpret, we have an interoperability 
problem.  That problem may be more or less severe depending on 
the range of possible interpretations and their consequences. 
The reason this document was developed was to reduce those 
interoperability problems and also to reduce the number of 
"N-squared" problems in which my receiving system is expected to 
understand all of the possible formats and interpretations that 
other systems can send out, even if they are explicitly 
identified.

I hope it is obvious that, if one identifies each body of text, 
in-band, with a description of what characters are used and how 
they are to be interpreted, then the net-utf8 format is not 
needed.  Everything that is needed is in the UTF8 and Unicode 
specs themselves.  I assume it is equally obvious that it makes 
little difference --at least to the utility of this spec-- 
whether such a description is explicit and local or provided in 
the form of some label that points to an external definition of 
a class of characters.

It seems to me that we have three plausible paths from here:

(1) Go ahead with net-Unicode (net-utf8), with the understanding 
that some protocols and contexts will find it useful and others 
won't.

(2) Try to insist that every protocol that transmits Unicode 
"plain-text" explicitly identifies the forms it is using (or the 
category to which it belongs, after defining those categories).

(3) Decide that interoperability of text interpretation is not 
an issue and just drop these ideas.

     john

---

From: Martin Duerst <[email-masked]>
Date: Tue, 09 Oct 2007 18:17:39 +0900

>would welcome suggestions for further tuning.

I think the treatment of FF looks reasonable. What I'm missing
is any mention of HT (horizontal tab). HT is often part of white
space, e.g. in formats such as XML, and heavily used in programming
languages,.... It may not be part of NVT ASCII; in that case, maybe
a short note would help. (I later found HT mentioned in the Appendix,
probably a pointer to that would be enough).

As for the overall document, here are a few more comments:

- 1.1, in my view, is still too much history. If the idea is just to
  establish the need for the new format, the details for UTF-8,
  UTF-16, and UTF-32, as one example, are not needed there.

- In section 3, I don't see any MUST for NFC, although from reading
  the text, it very much seems like that was intended (and I'd agree
  that's the way to go). Looking around, I found this in Section 2.
  Putting it in Section 2 is okay, but then there should be some
  pointer in Section 3. Also, if the definition is in Section 2,
  adding more requirements in other sections isn't very helpful.
  It would be better to either include the "must not transmit
  unassigned codepoints" to Section 2, or to change point 4 in
  section 2 to say "normalized according to details described in
  section 3".

  BTW, Section 2 should not say "Unicode method "NFC""; this would
  be the first spec I see that calls NFC a 'method'.

- Section 3 is written as if NFC were the only normalization form.
  E.g. "The section above requires that all Net-Unicode strings be
  transmitted in normalized form." or "The Unicode Consortium
  specifies a normalization method, known as NFC [NFC]...".
  This document doesn't need to got into details about other forms,
  but it shouldn't keep the informed or half-informed reader wondering
  "so, what about the other forms", or "any reason why they choose
  NFC?" or so.
  As the reason for NFC, I'd suggest some text like the following:
  "Of all normalization forms defined by Unicode, NFC is closest
  to actual use in practice and, in general requires the least
  work when converting from a non-Unicode encoding."

- "Systems conforming to this specification MUST NOT transmit any string
  containing any code point that is unassigned in the version of
  Unicode and NFC on which they are dependent.": The 'and' here is
  somewhat problematic. First, the statement only works for
  'version of Unicode', because NFC as such doesn't come with a list
  of assigned codepoints. The wording from
  http://unicode.org/standard/stability_policy.html#Normalization
  is better: "s only characters from a given version of the Unicode, and it
  is put into a normalized form in accordance with that version of Unicode".
  Second, the 'and' means that we are fine if at least one of the
  conditions is met, i.e. I can use a character from Unicode 5.0
  with a normalization implementation from Unicode 3.0 and I'm still
  okay with the above text.

- Section 4 starts with: "In retrospect, one of the advantages of ASCII".
  This is again too much focus on history. This fact can be mentioned
  e.g. at the end of the first paragraph, but the start of the section
  should concentrate on the problem at hand. Otherwise, this document
  already *reads* outdated now, just immagine how it will read in
  five or ten years.

- Section 4 nails down normalization at Unicode version 3.2. Also,
  it says that an update may be necessary for any changes to normalization.
  In fact, after Unicode version 3.2, there are already corrigenda to
  normalization, and it would be good if this document said clearly
  how they should be dealt with. I would propose the following:

  Corrigendum #4: Five CJK Canonical Mapping Errors
  (see http://www.unicode.org/versions/corrigendum4.html):
  This fixes five obvious mapping errors. Obviously, the new mappings
  should be used, because the old mappings lead to data corruption.
  No explicit labeling of the new mappings is necessary.

  Corrigendum #5: Normalization Idempotency
  (see http://www.unicode.org/versions/corrigendum5.html):
  This fixes a logical inconsistency in the textual description of
  the normalization algorithm. No reasonably occurring text is affected
  (by an extremely generous definition of reasonably occurring).
  No explicit labeling of the new mappings is necessary, because
  data isn't actually affected.

  I also think that the discussion on normalization versions should be
  in the normalization section.

- Section 4 says that non-assigned codepoints should not be used.
  Does that include or exculde codepoints in the private use areas?

- Section 5.1 says:
  "During the development of this specification, there was some
   confusion about where it would be useful given that, e.g., MIME and
   HTTP have their own rules about UTF-8 character types."
  This reads as if there were special provisions in MIME or HTTP about
  UTF-8. Such a misunderstanding should be avoided. I'm not sure
  I understand what you are referring to, but if you mean line-ending
  conventions, then MIME is with you, and for HTTP, it's not an issue
  of UTF-8, but more general. If you are referring to the fact that
  MIME/HTTP don't require NFC normalization, then don't blame your spec;
  NFC wasn't around yet when 

- Later in section 5.1, it says:
  "In particular, if this proposal is approved, or even appears to be
   getting significant traction,": Obviously, such text will look
  weird in an eventual RFC, and should be reworded, e.g. to something
  like a simple applicability statement:
  This specification is intended for use by other specifications that
  have not yet defined how to use Unicode. Examples could be Telnet...
  or FTP... This specification is not inteded for use with specifications
  that already allow the use UTF-8, such as MIME or HTTP.

- I remember a discussion about the fact that Unicode in theory can
  contain a base character followed by an unlimited number of combining
  characters, and that in a protocol context, checking such combinations
  may not be feasible (you in theory need a buffer of unlimited length).
  There was the idea to restrict this format to a certain number of
  combining character; where has this idea gone? From a protocol
  implementer's viewpoint, I think it is still necessary. It is actually
  defined at
  http://www.unicode.org/reports/tr15/tr15-26.html#Stream_Safe_Text_Format
  but I saw some problems there, too, so I'm pinging the Unicode guys.
  (the main problem is that it looks like this form ends up to always
   be based on NFKC, which would be counter-productive)

- I suggest a different title for 5.2. The current title sounds too much
  like FUD. The title should make clear that it is about design decisions,
  and maybe it actually belongs into an appendix (it's not normative,
  as far as I understand). To reduce any potential remaining FUD, it
  should also be pointed out that although the current stability rules
  do not rule out that a new script with both precomposed and decomposed
  representation is encoded, the chance that this happens is pretty
  slim, because the precomposed/decomposed dichotomy mainly came from
  legacy encodings. Also, unless you add a new base letter or a new
  combining character, creating a new precomposed/decomposed pair
  for an existing script is impossible, and one can fairly assume that
  virtually all base letters AND combining characters are already encoded,
  and that for new ones, no precomposed encodings will be created, again
  for the same reasons as above.

- 5.2 mentions "IETF Standard UTF-8" without a definition or a reference.
  Not sure what is inteded, so please clarify.

- Sentences such as these:
  "While not specifically a security issue, the requirement in NVT, and
   hence here, that, except as "newline" (CR LF), the CR character never
   appear alone but only when followed by ASCII NUL (an octet with all
   bits zero) may be problematic for some programming languages, and
   hence a trap for the unwary, unless caution is used."
  should be reworded. If the RFC Editor does even just a halfway job,
  this sentence won't remain as it is, and it's better to fix it now
  than leave that to the RFC Editor (at least in my experience).

I haven't read the Appendix in detail, but I don't think it affects
the above comments.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 09 Oct 2007 18:17:39 +0900

- In section 3, I don't see any MUST for NFC, although from reading
  the text, it very much seems like that was intended (and I'd agree
  that's the way to go). Looking around, I found this in Section 2.
  Putting it in Section 2 is okay, but then there should be some
  pointer in Section 3. Also, if the definition is in Section 2,
  adding more requirements in other sections isn't very helpfu3b
	for [email-masked]; Tue, 09 Oct 2007 05:43:52 -0400
To: [email-masked]
From: "Frank Ellermann" <[email-masked]>
Subject: Re: draft-klensin-unicode-escapes-05 and draft-klensin-net-utf8-05
Date: Tue, 9 Oct 2007 11:40:48 +0200
Lines: 23

John C Klensin wrote:

 [bare CR vs. bare LF]=20
> I could try to incorporate a little more text along those lines=20
> in the document

Maybe.  The main issue why I mentioned it are some inconsistencies
in the normative language (MUST vs. SHOULD) wrt CR.  And my odd
impression that net-utf8 tries to "ban" HT and bare CR, but still
"permits" FF and bare LF.  I'd like it better if both bare CR and
bare LF are in one group, and both HT and FF in another group.

> please remember that it has already been criticized for=20
> containing too much history and explanation

I think at least Stephane and I liked the historical parts very
much, e.g. the IAC (0xFF) info at the end of appendix B is not
only fascinating, it's important.  OTOH I'm not sure that net-utf8
is a good place to discuss general Unicode and NFC issues, let
alone implications for IDNAbis.  Maybe an informative reference
to section 3 in RFC 4690 could cover most of this.

 Frank

l.

- Section 5.1 says:

Regards,   Martin.