Subject: draft-klensin-unicode-escapes-05 and draft-klensin-net-utf8-05

From: John C Klensin <[email-masked]>
Date: Fri, 05 Oct 2007 20:11:28 -0400

Hi.

New versions of these documents have been posted for your
reading pleasure, reflecting comments and discussions on this
list (and some private correspondence about editorial issues)
today.

     john

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 8 Oct 2007 11:30:03 +0200

John C Klensin wrote:

> reading pleasure

The unicode-escapes-05 are fine.  One observation:  In 5.2
you mention how to encode a literal '&' as "&#x26;", that's
enough to solve all issues with literal "&#x" constructs.

In section 5.1 you don't mention a similar trick for "\u'"
constructs, and "implementors" (i.e. protocol designers)
might pick some '\\' convention to get a literal '\'.  Is
that as it should be?

==========================================================
In net-utf8-05 2.1 I'm not sure about this recommendation:
| CR SHOULD NOT appear except when followed by LF.

There's no similar statement about a bare LF.  If you want
to permit bare LF for cursor control on dumb terminals,
then you could also permit bare CR for this purpose.  

If you want to prohibit bare CR as "line ending" (the topic
of item 2 in section 2.1), then you could also prohibit a
bare LF for this purpose, maybe you could also mention NEL.

Combined with item 3 in 2.1 I think what you really want
is to ban both bare CR _and_ bare LF, something like this:

| CR SHOULD NOT appear except when followed by LF, and v.v.

In 2.1 item 3 please s/FF/Formfeed/ and simplify this part:
"FF should be used only with caution: if its use assumes a
 page length, such assumptions may not be appropriate in
 international contexts (e.g., considering 8.5x11 inch
 paper versus A4)".

There's more about this (fonts, font sizes), you could
simply state:  "Formfeeds should be used only with
caution, e.g. in legacy formats like Internet Drafts".

At the end of 2.1 you use a "MUST NOT" about the CR.
Either it's MUST NOT or SHOULD NOT.  ITYM the latter,
otherwise the following CR NUL "SHOULD" makes no 
sense (for me).  

There's no section 2.2 and no intro before section 
2.1, please get rid of the unused subsection title.

Section 3:
| The section above requires that all Net-Unicode
| strings be transmitted in normalized form.

s/requires/recommends/ (SHOULD in 2.1 item 4)

Section 4:
| The normalization specified here, NFC

s/specified/recommended/ (you don't specify NFC,
section 3 explicitly says that NFC is specified
by Unicode in UAX #15).

Section 5.2:
| IETF Standard UTF-8 is dependent on some definitions
| not changing after Unicode Version 4.0.

I'm not aware of such dependencies.  UTF-8 depends on 
two points:  The will to limit Unicode to 21 bits, and 
the will to outlaw "overlong" encodings.  Both points
are guaranteed in STD 63, they don't depend on anything
defined by say UTC.

The complete IDNA issue is IMO unrelated to "net-utf8".
I'd delete the complete section 5.2, rewrite it from
scratch, or something.  

Appendix A:
s/2068/2616/ or s/2068/1945/ (?)  ITYM 1945 HTTP/1.0.

Back to section 2.1:
Before discussing FF (Formfeed) at all you should talk
about HT.  Many decent protocols use WSP as defined in
the future ABNF 4234bis standard, i.e. SP or HT.

 Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 8 Oct 2007 11:51:43 +0200

On Mon, Oct 08, 2007 at 11:30:03AM +0200,
 Frank Ellermann <[email-masked]> wrote 
 a message of 150 lines which said:

> The complete IDNA issue is IMO unrelated to "net-utf8".  I'd delete
> the complete section 5.2, rewrite it from scratch, or something.

+++1

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 8 Oct 2007 12:08:09 +0200

It seems well-covered in the current draft:

> Since this specification does not recommend one specific syntax,
> protocols specifications that use escapes MUST define the syntax
> they are using, including any necessary escapes to permit the escape
> sequence to be used literally.
[...]
> In addition, when an escape is needed for the escape mechanism
> itself, the optimal one of those might differ from one context to
> another.

I would have preferred an official recommandation but it seems there
is no consensus for one specific 'escape the escape' format.

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 8 Oct 2007 13:53:01 +0200

Stephane Bortzmeyer wrote:
 
> It seems well-covered in the current draft

ACK, I forgot the overall clause in section 4
when I looked at the 5.1 and 5.2 details.

 Frank

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 8 Oct 2007 15:14:53 +0100

> is no consensus for one specific 'escape the escape' format.

However, since we're mentioning "&#x26;" as *a* syntax that always works,
even though the more usual approach is "&amp;", I think for completeness
we should mention "\u'005C'" as *a* guaranteed way of getting a backslash,
even if "\\" is more usual.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 8 Oct 2007 15:19:46 +0100

> list (and some private correspondence about editorial issues)

unicode-escapes section 7 has a spurious closing double quote.

Appendix A.1., comment after "BMP-form", what does "the one above" refer
to?

Nitpick: some slashes in the definition of HEXDIG have a space before them
and some don't. You've also got two spaces after some = and three after
others.

You don't have an A.3 for Java.

---

From: John C Klensin <[email-masked]>
Date: Mon, 08 Oct 2007 12:42:41 -0400

> unicode-escapes section 7 has a spurious closing double quote.

Will check on this.

> Appendix A.1., comment after "BMP-form", what does "the one
> above" refer to?

Probably the result of pulling that text out to a separate 
section.  Will check.

> Nitpick: some slashes in the definition of HEXDIG have a space
> before them and some don't. You've also got two spaces after
> some = and three after others.

If the RFC Editor thinks this is important, I'm sure they will 
fix it.

> You don't have an A.3 for Java.

Because there was no specific syntax in the last draft that had 
the syntax in line.

Re the FF and CR discussions: please understand that, if one 
were able to erase the entire history of text on the network and 
start over today, one might reasonably do something different. 
Or one might not.  But, because this whole business is intended 
for untagged text, making incompatible changes in interpretation 
of strings is just not, IMO, a plausible option.  CR was 
discussed more intensely in the net-utf8 draft because it is 
discussed more intensely in the NVT / Net-ASCII documents.  Bare 
LF was discussed less because it wasn't considered much of an 
issue and because the negative effects of interpreting it 
differently than the sender intended were much less severe 
(e.g., one-line vertical indexing versus NewLine) than the 
consequences of getting the interpretation of bare CR wrong 
(NewLine versus overstriking).  Anyone who has

 seen a terminal
                or other display
                                 render bare-LF
                                                text like this

versus having lines overstuck or, worse, rewritten multiple 
times with each one being quickly erased, should understand this 
immediately.  The LF case is readable, if annoying, and the 
source of the problem is almost immediately clear.  The CR case 
obscures text: on a fast enough device, or on a system that 
canonicalizes before transmitting, one ends up looking only at 
the last line of a paragraph and has to guess what might have 
happened.

I could try to incorporate a little more text along those lines 
in the document, but please remember that it has already been 
criticized for containing too much history and explanation 
rather than just laying out the rules.

General comment on these two documents.  In both cases, I 
started them because it appeared that they would be useful to 
the community.  I am not strongly committed to them and my 
patience for nitpicking has just reached zero.  So, I will 
generate -06 drafts if they are really needed, but that is it: 
if the community, or selected individuals, want the quest for 
perfection to drive out the opportunity to have something useful 
(but probably imperfect), this is your opportunity.  Or, if 
there is someone who has more patience and time than I do who 
would like to take over and polish them to a high luster, please 
let me know.

      john

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 9 Oct 2007 11:40:48 +0200

John C Klensin wrote:

 [bare CR vs. bare LF] 
> in the document

Maybe.  The main issue why I mentioned it are some inconsistencies
in the normative language (MUST vs. SHOULD) wrt CR.  And my odd
impression that net-utf8 tries to "ban" HT and bare CR, but still
"permits" FF and bare LF.  I'd like it better if both bare CR and
bare LF are in one group, and both HT and FF in another group.

> please remember that it has already been criticized for 
> containing too much history and explanation

I think at least Stephane and I liked the historical parts very
much, e.g. the IAC (0xFF) info at the end of appendix B is not
only fascinating, it's important.  OTOH I'm not sure that net-utf8
is a good place to discuss general Unicode and NFC issues, let
alone implications for IDNAbis.  Maybe an informative reference
to section 3 in RFC 4690 could cover most of this.

 Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 9 Oct 2007 15:20:25 +0200

On Mon, Oct 08, 2007 at 12:42:41PM -0400,
 John C Klensin <[email-masked]> wrote 
 a message of 82 lines which said:

> I could try to incorporate a little more text along those lines in
> the document, but please remember that it has already been
> criticized for containing too much history and explanation rather
> than just laying out the rules.

Allow me a personal soapbox. I find that typical RFCs are terribly
lacking rationales, history and background details. The focus on 'pure
specification' makes difficult to understand the choices and is a sure
recipe for reinventing bad ideas from time to time (or, worse, to
violate a specification because it seems suboptimal).

The standard IETF reply 'Read the archives of the working group, they
are public, anyway', is a joke: no normal human being can digest two
or three years of a typical WG mailing list archive and still do
actual work.

I understand the need to separate the specification from the rationale
(for large RFC, this could be done by a split between two RFC, one
standard and one informational) but this should not prevent us to have
a 'Rationale and design choices' section.

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Oct 2007 14:49:29 -0400

--On Tuesday, October 09, 2007 3:20 PM +0200 Stephane Bortzmeyer 
<[email-masked]> wrote:

>> I could try to incorporate a little more text along those
>> lines in the document, but please remember that it has
>> already been criticized for containing too much history and
>> explanation rather than just laying out the rules.
>
> Allow me a personal soapbox. I find that typical RFCs are
> terribly lacking rationales, history and background details.
> The focus on 'pure specification' makes difficult to
> understand the choices and is a sure recipe for reinventing
> bad ideas from time to time (or, worse, to violate a
> specification because it seems suboptimal).

We also tend to forget that compliance with IETF standards is 
voluntary.   If someone looks at a specification and says "that 
makes no sense to me", the odds of their implementing it just 
because the IETF says so are not high, certainly not as high as 
if they could look somewhere and say "ah, that was done for the 
following reason... even if I don't completely agree, it makes 
sense".

It seems to me to be particularly important if we are modifying, 
rather than just following, some practice or existing form of a 
standard that is out there.  One of the best reasons for doing 
so is if the existing practices are not consistent and the IETF 
work is an attempt to draw things back together or outline a 
path for the future, but that makes it all the more important to 
provide a rationale, lest people simply continue to do whatever 
occurs to them first.

> The standard IETF reply 'Read the archives of the working
> group, they are public, anyway', is a joke: no normal human
> being can digest two or three years of a typical WG mailing
> list archive and still do actual work.
>
> I understand the need to separate the specification from the
> rationale (for large RFC, this could be done by a split
> between two RFC, one standard and one informational) but this
> should not prevent us to have a 'Rationale and design choices'
> section.

I think we are in violent agreement.

     john

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Thu, 11 Oct 2007 07:46:04 +0100

> Because there was no specific syntax in the last draft that had 

  Appendix A.3.  Java Form

     EmbeddedUnicodeChar = %x5C.7A 4HEXDIG

> rather than just laying out the rules.

I am in favour of the history and explanation.

---

From: John C Klensin <[email-masked]>
Date: Thu, 11 Oct 2007 10:05:50 -0400

--On Thursday, 11 October, 2007 07:46 +0100 "Clive D.W. Feather"
<[email-masked]> wrote:

>> Because there was no specific syntax in the last draft that
>> had  the syntax in line.
> 
> 
>   Appendix A.3.  Java Form
> 

You mean %x5C.75, don't you?  7A is "z".

> I am in favour of the history and explanation.

Thanks.

    john