Subject: draft-klensin-net-utf8 and draft-klensin-unicode-escapes

From: John C Klensin <[email-masked]>
Date: Thu, 04 Oct 2007 17:10:43 -0400

Hi.

After a long delay, I'm in the process or updating both of these
documents.  Very high-level summary of changes:

  ------
draft-klensin-unicode-escapes
  -04 has been posted.   It has been reorganized to make it
easier to follow and to respond to comments from several people
on and off the list.  It now recommends either \u'NNNN..' or the
XML form and recommends against the others (and explains why).
I think (hope?) that is consistent with general consensus.  The
alternative is to make no recommendation at all, which rather
misses the point, I think (and some others seem to think so as
well).

Chris Newman suggested one additional change, which was to move
the syntax for the not-recommended options into an appendix so
as to further discourage their use.  I think that is reasonable,
but it didn't make -04.   -05 will be posted, presumably within
the next 24 hours if the online tool works this time, but that
is the only substantive change from -04.

 -------

draft-klensin-net-utf8
  This document has been considerably reworked, with some
requirements lowered to SHOULD and others promoted to MUST per
online discussions.  Most of the former introductory material
and explanations of Net-ASCII have been moved to appendices and
the text streamlined.  Normative recommendations to avoid the
printer-driven idiosyncrasies of Net-ASCII have been added and
clarified and a prohibition against the C1 controls has been
added as well.  The document should be posted once my
collaborator and co-author has had a chance to review it.

My hope is that we can fine-tune these in the next week or two
and then put the documents out for Last Call as Proposed
Standards.  I think they are valuable or I wouldn't be putting
effort into them.  However, they have reached the point that, at
least for me, significant further investments would exceed that
value.  So, if the community wants them, let's focus on any
remaining important issues and get them resolved.   If not,
well, I'm prepared to just let one or both drop at this point.

     john

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 5 Oct 2007 16:24:34 +0200

On Thu, Oct 04, 2007 at 05:10:43PM -0400,
 John C Klensin <[email-masked]> wrote 
 a message of 49 lines which said:

> It now recommends either \u'NNNN..' 

[Small syntax detail, do not spend too much time on it.]

Section 5.1 describes the content of the "Backslash-U with Delimiters"
form as "4*6HEXDIG". Why not "2*6HEXDIG"? It would be more compact for
the first characters and would be more consistent with the other forms
such as the "XML and HTML" one.

My personal taste is that \u'20' is better than \u'0020'.

Section 6.2 contains a note which seems related (the risk that small
numbers are thought to represent octets, in the current locale) but I
do not think it is a serious risk since section 2 clearly states that
we encode Unicode code points, not octets.

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Oct 2007 11:30:23 -0400

--On Friday, 05 October, 2007 16:24 +0200 Stephane Bortzmeyer
<[email-masked]> wrote:

> Section 5.1 describes the content of the "Backslash-U with
> Delimiters" form as "4*6HEXDIG". Why not "2*6HEXDIG"? It would
> be more compact for the first characters and would be more
> consistent with the other forms such as the "XML and HTML" one.
> 

First, I want to be clear that I don't feel strongly about any
of this and will do what the community wants.   However...

One could use the two-digit form, but I think it is trouble.
The Unicode folks,
whose guidance I'm trying to follow except when it seems to
violate good Internet or applications sense, never use less than
four hex digits.  And we have the problem with Perl (noted in
that section now) that two-digit values are sometimes
interpreted as ASCII and sometimes as "local one-octet character
encoding, whatever that is".  So the four-digit form helps
emphasize that this is, in fact, Unicode and avoids any risk of
confusion with local one-octet (or less) character sets, decimal
encoding of octets, etc.

> Section 6.2 contains a note which seems related (the risk that
> small numbers are thought to represent octets, in the current
> locale) but I do not think it is a serious risk since section
> 2 clearly states that we encode Unicode code points, not
> octets.

And we all know that people often read specs in a sloppy or
incomplete fashion.    So I suggest that requiring four digits
(or more) is more robust than permitting two.  Whether the
advantages of being able to write two digits outweigh that...
well, I don't think so, but, if others strongly disagree, I'll
change it.

      john

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 5 Oct 2007 17:20:38 +0100

John C Klensin said:
> on and off the list.

First thing I spotted: the syntax in 5.1 is missing the trailing quote mark.

Section 4 is missing the references to ISO-C and Java.

> draft-klensin-net-utf8

Section 2.1 item 3: "control characters" needs to be properly defined,
since it's a SHOULD. Unicode.org is currently offline, but at the least
you either need to make it explicit ranges (say U+0000 to U+001F and U+007F
to U+009F), or base it on a Unicode character class.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Oct 2007 12:31:37 -0400

--On Friday, 05 October, 2007 17:20 +0100 "Clive D.W. Feather"
<[email-masked]> wrote:

>> easier to follow and to respond to comments from several
>> people on and off the list.
> 

Noted already and fixed.

> Section 4 is missing the references to ISO-C and Java.

I will happily accept and incorporate such references.  However,
I note that I've been encouraged to remove or minimize Section 4
and that anyone who is reading this and doesn't know what C or
Java are probably won't want to use those forms (which is the
idea).

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Oct 2007 12:57:57 -0400

--On Friday, 05 October, 2007 17:20 +0100 "Clive D.W. Feather"
<[email-masked]> wrote:

>> draft-klensin-net-utf8
> 
> Section 2.1 item 3: "control characters" needs to be properly
> defined, since it's a SHOULD. Unicode.org is currently
> offline, but at the least you either need to make it explicit
> ranges (say U+0000 to U+001F and U+007F to U+009F), or base it
> on a Unicode character class.

Once upon a time, the criterion for the level of referencing and
specificity in an RFC was "everyone, or at least everyone
competent to use the spec, knows what one is talking about".
Being pedantic for its own sake or on principle was not
considered useful or desirable.

If anyone is writing code that uses Unicode and doesn't know
what a control character is, they are in much bigger trouble
than this document can fix.   And "C0" and "C1" are defined in
Unicode as the names of parts of blocks.

I'll make this change, but would encourage everyone to think
carefully about whether we really want to go in the direction I
think it represents.

    john

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 5 Oct 2007 19:49:04 +0100

John C Klensin said:
> I will happily accept and incorporate such references.

Sorry, I wasn't clear. They're referred to in the References section. They
just aren't tagged in section 4.

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Oct 2007 15:11:47 -0400

--On Friday, 05 October, 2007 19:49 +0100 "Clive D.W. Feather"
<[email-masked]> wrote:

> Sorry, I wasn't clear. They're referred to in the References
> section. They just aren't tagged in section 4.

Ah.  Editing glitch.  Will fix
    john