Subject: Re: New draft (Was: I-D ACTION:draft-klensin-unicode-escapes-00.txt

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 2 Feb 2007 11:46:58 +0000

>> In section 5.2 it is said HTML uses the &#xNNNN; form and that
>> this form has a clear terminator. This is not really false but
>> HTML allows to omit the terminator if it is not needed, for
>> example <p>Bj&#xf6rn</p> is also valid. I would suggest to
>> mention only XML or note that HTML's mechanism is similar to
>> that of XML.

If you check the HTML specification (section 5.3), it says that SGML
allows the semicolon to be omitted in certain contexts, but "strongly
suggest" not to do that.

In particular, the example
    <p>Bj&#xf6rn</p>
is not valid because it lies in the middle of a word. A permitted case
would be:
    <p>Bj&#xf6</p>
where the tag begin symbol < ends the entity.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 2 Feb 2007 11:38:53 +0000

John C Klensin said:
> I've just submitted draft-klensin-unicode-escapes-01.txt and
> assume it will show up in the posting directory today or
> tomorrow.  

Some comments for you.

* In 1.1, rather than saying that Unicode occupies "two or more octets",
wouldn't it be better to say "21 bits - rather than the 7 bits of ASCII -"?

* Somewhere in the last two paragraphs of 1.1 you should be talking about
mini-languages (e.g. Cosmogol) as well as protocols and UIs.

* In 3, you're inconsistent between "U+NNN[N[N]]" and "NNN...". Indeed,
shouldn't the former actually be "U+[[N]N]NNNN"? (Note both the order and
the number of Ns.) I would suggest that better wording might be:

    ... U+NN syntax for code point references specified in the Unicode
    Standard, where NN is between four and six hexadecimal digits.

* In 4, second bullet, "string terminators" should be "string delimiters".

* In 5.2, you've said "generally considered ugly and awkward" but I'm not
aware of anyone else who's made that complaint.

* In 6 you need to copy in all the security stuff from Unicode; the stuff
that says that you must use shortest-form UTF-8 (so not using %xC1.A1 for
'A') because of the problems of filters and firewalls not spotting longer
forms.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 2 Feb 2007 13:50:12 +0100

On Fri, Feb 02, 2007 at 11:38:53AM +0000,
 Clive D.W. Feather <[email-masked]> wrote 
 a message of 35 lines which said:

> * Somewhere in the last two paragraphs of 1.1 you should be talking
> about mini-languages (e.g. Cosmogol) as well as protocols and UIs.

This is clearly a difficult issue since RFC 2277 is clear on:

* text carried by a protocol (i18n is necessary)
* protocol elements (i18n is optional)

but does not mention formats, mini-languages and so on. RFC 4234 is a
good example of a format whose i18n rules are unclear.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Feb 2007 14:30:52 +0100

Clive D.W. Feather wrote:

> In 3, you're inconsistent between "U+NNN[N[N]]" and "NNN...". Indeed,
> shouldn't the former actually be "U+[[N]N]NNNN"? (Note both the order
> and the number of Ns.)

+1

>     Standard, where NN is between four and six hexadecimal digits.

No, folks could misinterpret U+NN as "anything up to 6 digits".

> In 5.2, you've said "generally considered ugly and awkward" but I'm
> not aware of anyone else who's made that complaint.

+1  Obviously John hates it, that would justify "often".  Others don't
like backslash-U for various reasons, not only ugly and awkward, also
confusing (due to various conventions), unclear (lack of delimiter),
and a royal PITA in conjunction with <quoted-string>, when it results
in multiple backslashes.  "Harmful" is worse than "ugly and awkward".

> In 6 you need to copy in all the security stuff from Unicode

IMO not "all", folks are supposed to know RFC 3629, it's a STD.  So far
all attacks on the "three steps" model fortunately failed, STD is STD.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Feb 2007 14:05:34 +0100

> allows the semicolon to be omitted in certain contexts, but "strongly

Yes, never ever mention that HTML exists, it's horrible.  The [Charmod]
bible requires no (SGML) nonsense in http://www.w3.org/TR/charmod/#C044

The I-D should IMO adopt and cite [Charmod] C042 up to C048 verbatim.

A few other conformance criteria in [Charmod] might be also interesting:
http://www.w3.org/TR/charmod/#C070  Don't exclude arbitrary code points
http://www.w3.org/TR/charmod/#C077  Don't allow anything above U+10FFFF
http://www.w3.org/TR/charmod/#C078  Don't (ab)use surrogates
http://www.w3.org/TR/charmod/#C079  Don't (ab)use non-characters

http://www.w3.org/TR/charmod/#C015  n/a (covered by the better RFC 2277)
http://www.w3.org/TR/charmod/#C016  n/a (covered by the better RFC 2277)
http://www.w3.org/TR/charmod/#C017  Stick to working encoding rules
http://www.w3.org/TR/charmod/#C018  n/a (covered by the better RFC 2277)

http://www.w3.org/TR/charmod/#C049  n/a (for the I-D US-ASCII is given)
http://www.w3.org/TR/charmod/#C026  n/a (covered by the better RFC 2277)

Etc.  The "better RFC 2277" idea is a single default UTF-8, instead of a
choice between UTF-8, UTF-16, UTF-16LE, UTF16-BE, UTF-32, UTF-32LE, and
UTF-32BE in [Charmod], let alone hypothetical UTF-32 "2143" or "3412".

Probably the I-D should mention that one famous exception from its rule
to avoid encoded UTF-8 is the URL form of IRIs.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Feb 2007 15:07:37 +0100

Stephane Bortzmeyer wrote:

> RFC 4234 is a good example of a format whose i18n rules are unclear.

| NOTE:
|
|     ABNF strings are case-insensitive and the character set for these
|     strings is us-ascii.

That's clear.  A tricky part could be <name> in chapter 2.2, because...

|     rulename       =  ALPHA *(ALPHA / DIGIT / "-")

...in chapter 4 could be interpreted as different from <name>.  Now I've
found a typo in 4234 chapter 2.4:

   although Appendix A (Core) provides definitions for a 7-bit US-ASCII
   environment as has been common to much of the Internet.

It's Appendix B, not Appendix A (Acknowledgements).  Chapter 4 is based
on Appendix B, a <rulename> is clearly ASCII.  IMO RFC 4234 is fine, its
LWSP is an exception, FWS as in RFC 2822 (excl. obs-FWS) would be better.

Unlike an utter dubious variant in RFCs 2068, 2069, 2616, 2617, and 2831,
where nobody sees the potential damage caused by <LWS> hidden in a #rule.

Frank

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 2 Feb 2007 18:25:06 +0000

John C Klensin said:
> 	* I have not touched the ABNF associated with the \u /
> 	\U case.  I have inserted an explicit placeholder but,
> 	as discussed on this list, I think we need to figure out
> 	what we want to do and then go back and adjust the
> 	metalanguage productions.   In particular, there has
> 	been one strong suggestion, with which I agree, that we
> 	not take the obvious approach of substituting %x5C.75
> 	for "\u", since the intent is a character string
> 	abstraction (independent of the implementation character
> 	set) rather than specific octets.

I asked Paul Overell about this and got the following answer (in part):

>        "By separating external encoding from the syntax, it is intended
>        that alternate encoding environments can be used for the same
>        syntax."
>
> So although "\" means us-ascii %x5C, the same ABNF may still be used to
> specify the syntax of strings expressed in a different character set by
> specifying the mapping between %x5C and the encoding used, but that is
> outside the scope of ABNF.
   
In other words, you write the ABNF as if the target encoding was ASCII, but
then state somewhere in the document that other encodings may be used and
the ABNF is meant to represent the abstract characters, not specific octet
values.

Or else we either drop ABNF (wrong, I think) or state explicitly that the
notation is "ABNF except for case-sensitivity".

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 2 Feb 2007 18:47:27 +0000

> The I-D should IMO adopt and cite [Charmod] C042 up to C048 verbatim.

C042 would require &#x1234; rather than allowing us to invent \u'1234'.

> http://www.w3.org/TR/charmod/#C079  Don't (ab)use non-characters

Those are worth including, I think.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 2 Feb 2007 18:51:04 +0000

> No, folks could misinterpret U+NN as "anything up to 6 digits".

Um, the wording I've used is almost identical to that in the charmod
document (section 1.3).

> confusing (due to various conventions), unclear (lack of delimiter),

Solvable if we include a delimiter.

> in multiple backslashes.

Um, every scheme has that problem, surely? See "&amp;#x1234;".

> IMO not "all", folks are supposed to know RFC 3629, it's a STD.

Okay, then the security section needs to explicitly point at the security
section of 3629; it's not enough to say "people should know it".

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 2 Feb 2007 18:58:56 +0000

Frank Ellermann said:
> A tricky part could be <name> in chapter 2.2, because...
> 
> ...in chapter 4 could be interpreted as different from <name>.

That's stretching it somewhat.

> Now I've
[...]

I pointed this out to Paul a while ago.

> IMO RFC 4234 is fine, its

I'm not sure what point you're trying to make here.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Feb 2007 22:08:19 +0100

> I pointed this out to Paul a while ago.

Thanks, apparently not yet noted as RFC 4234 "erratum".

But something happened, they've finally processed Ned's
simple 2045 erratum - as "unverified", now what's that, 
do they try to verify that the submitter is the author ?

>> IMO RFC 4234 is fine, its LWSP is an exception, FWS
>> as in RFC 2822 (excl. obs-FWS) would be better.

> I'm not sure what point you're trying to make here.

| LWSP =  *(WSP / CRLF WSP)

That allows multiple empty lines consisting only of WSP.

For FWS minus obs-FWS that's impossible (and a loophole
in CFWS is covered by a MUST NOT in the RFC 2822 prose):

| FWS  = ([*WSP CRLF] 1*WSP) /   ; Folding white space

The real damage is in the #rule in RFC 2068 ff. up to 2831bis:

| a rule such as "( *LWS element *( *LWS "," *LWS element )) "
| can be shown as "1#element".
[...]
| implied *LWS
|  The grammar described by this specification is word-based. Except
|  where noted otherwise, linear whitespace (LWS) can be included
|  between any two adjacent words (token or quoted-string), and
|  between adjacent tokens and delimiters (tspecials), without
|  changing the interpretation of a field. At least one delimiter
|  (tspecials) must exist between any two tokens, since they would
|  otherwise be interpreted as a single token.
[...]
| LWS   = [CRLF] 1*( SP | HT )

The same issue as in LWSP, but in RFC 2068 etc. it's hidden in the
#rule and the odd "implied *LWS":  It allows to insert constructs
like <CRLF><SP><CRLF><HT> anywhere.  RFC 2822 tries to eliminate
this possibility in its obs-FWS chapter (4.2), and in the USEFOR
I-Ds obs-FWS is explicitly verboten.  Trailing white space is evil.

Frank

---

From: John C Klensin <[email-masked]>
Date: Fri, 02 Feb 2007 16:37:01 -0500

In order to reduce the number of threads consisting of
incremental micro-messages, I'm going to try an omnibus
response....

--On Friday, 02 February, 2007 18:47 +0000 "Clive D.W. Feather"
<[email-masked]> wrote:

>> A few other conformance criteria in [Charmod] might be also
>> interesting: http://www.w3.org/TR/charmod/#C070  Don't
>> exclude arbitrary code points
>> http://www.w3.org/TR/charmod/#C077  Don't allow anything
>> above U+10FFFF http://www.w3.org/TR/charmod/#C078  Don't
>> (ab)use surrogates http://www.w3.org/TR/charmod/#C079  Don't
>> (ab)use non-characters
> 

Folks, someday, someone will write a document called "using
Unicode on the Internet".  To some extent, the now-suspended
draft-klensin-net-utf8 is a step in that direction (Mike and I
will get back to it, but, right now, I've got some other things
on my plate).  

But _this_ document is about escaping characters, or really code
points, not about what characters should be permitted or how
they should be used.  Or one of you might take some of this
energy and divert it into an RFC-ish version of "CharMod".   But
for this, just taking the above list as examples:

  C070: irrelevant.   if it is in range, this works.
  C077: needs to be fixed somewhere else.  This is just about a
syntax for escapes
  C078: Already prohibited
  C079: Precisely the reason why one might want escapes is to be
able to deal with non-characters in a sensible way.  Whether
they should be used or not depends on the relevant protocol.

--On Friday, 02 February, 2007 14:05 +0100 Frank Ellermann
<[email-masked]> wrote:

> Yes, never ever mention that HTML exists, it's horrible.  The
> [Charmod] bible requires no (SGML) nonsense in
> http://www.w3.org/TR/charmod/#C044

Sorry, but, IMO, for a document like this which is merely giving
examples (at this point), "very widely deployed and used" trumps
"horrible".
 
>...
> Probably the I-D should mention that one famous exception from
> its rule to avoid encoded UTF-8 is the URL form of IRIs.

Why?  I don't see that (and a few other cases) as "exceptions"
(famous or not), but as mistakes from which we should learn and,
I hope, have learned.   The document is reasonably clear that it
is not a proposal to retrofit any existing protocol.

>> assume it will show up in the posting directory today or

Thanks

> * In 1.1, rather than saying that Unicode occupies "two or
> more octets", wouldn't it be better to say "21 bits - rather
> than the 7 bits of ASCII -"?

No, because of net-ascii and some other issues, probably not.
Again, because the important issue here is that this stuff is
about escapes, you are picking nits that belong elsewhere (see
rant at the end).

> * Somewhere in the last two paragraphs of 1.1 you should be
> talking about mini-languages (e.g. Cosmogol) as well as
> protocols and UIs.

Why?  In principle I could talk about Japanese business cards
and all sorts of other things too.  But that is introductory
material to help the reader understand the context.  Trying to
create an exhaustive list would add nothing to the document
except more pages and making it harder to read.

> * In 3, you're inconsistent between "U+NNN[N[N]]" and
> "NNN...". Indeed, shouldn't the former actually be
> "U+[[N]N]NNNN"? (Note both the order and the number of Ns.) I
> would suggest that better wording might be:

Partially fixed in -02.   U+NNNN[N[N]] versus U+[[N]N]NNNN is a
matter of taste.  Clearly, in that pseudo-notation, one "N" is
as good as another.   See rant at end.

>     ... U+NN syntax for code point references specified in the
> Unicode     Standard, where NN is between four and six
> hexadecimal digits.

I agree with another comment -- too easy to misunderstand.

> * In 4, second bullet, "string terminators" should be "string

I was deliberately trying to be general.  If one has a form like
H'nnnn', one is clearly talking about "string delimiters".
However, if one has, e.g., &#xNNNN;, it is clear that ";" is a
string terminator, but whether there is a starting delimiter at
all depends on how one defines things in metalanguage or words.
E.g., using BNF (_not_ ABNF), one could reasonably have
   <XML-like-Unicode-escape-string> ::=  <type-introducer>
<value> <terminator>
   <type-introducer> ::= "&#x"
   <value> ::= ....
   <escape-terminator> ::= ";"
or you could construct it in other ways.  Matter of taste, see
rant at the bottom.

> * In 5.2, you've said "generally considered ugly and awkward"
> but I'm not aware of anyone else who's made that complaint.

I could trundle out several others, but it is probably more
efficient to write off to editor's privilege.   See rant at end.

> * In 6 you need to copy in all the security stuff from
> Unicode; the stuff that says that you must use shortest-form
> UTF-8 (so not using %xC1.A1 for 'A') because of the problems
> of filters and firewalls not spotting longer forms.

Absolutely not.  Again, this document is about escapes for
strings of Unicode characters, not about the general use and
appropriateness of Unicode.   And shortest-form UTF-8 is
especially irrelevant because this document is an attempt to
prohibit escapes for UTF-8 entirely where that is still
possible.  The business about different, almost-equivalent,
forms of UTF-8 arguably should go into Section 1 or 2.1 as
further evidence that escaped UTF-8 is a bad idea, but I think
the point has been made.

---------------
<rant>
There is a lot of work to be done in the internationalization
area in the IETF.  Whether one likes what RFC 2277 says or not,
it has become fairly clear that it is not as close to the last
word on the subject as many of us assumed it would be when it
was written.   The suggestions about about CharMod, the
shortest-form string issues that caused us to need to revise the
UTF-8 specs, and the recent work on comparators and comparator
registries, are just the beginning of a very long list.

Speaking personally, I'd be really pleased if I were doing a
much smaller percentage of the document-writing in this area.
But, if I'm going to do it, then my editorial judgment and
preferences are going to prevail... at least until the document
gets far enough along that I have to start arm-wrestling with
the RFC Editor and _their_ judgment and preferences.  I really
appreciate comments about how to make a document more clear but
an argument about, e.g., U+NNNN[N[N]] versus U+[[N]N]NNNN is
only about taste and is hence a waste of everyone's time.

If you don't like my writing style -- and many people don't --
please take on these efforts yourselves and let me complain
about your style (or not) some of the time.  Sniping and
nit-picking is easy and may be fun, but it tends to block,
rather than contribute to, progress.   If you are not going to
pick up some of the writing work, unless your purpose is to
introduce delays and lay down obstacles -- which I assume it is
not-- can we please concentrate on substantive issues and
document changes that would have a clear positive impact on
clarity?
</rant>

thanks,
     john

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Feb 2007 23:23:27 +0100

[U+NN]
> Um, the wording I've used is almost identical to that in the charmod

The wording is okay, but IMO your reversed mnemonic U+[[N]N]NNNN is
better than only U+NN.  With readers you never know, some like me never
look into the prose if the ABNF is apparently clear, while others also
including me look for the examples before ever reading a single word of
the prose or ABNF, and if I understood John correctly his approach is
more like the opposite, he looks into the ABNF if prose and examples
are hopeless... ;-)  

>> a royal PITA in conjunction with <quoted-string>, when it results
>> in multiple backslashes.
 
> Um, every scheme has that problem, surely? See "&amp;#x1234;".

Yes, but it doesn't have to fight with putting <quoted-string>s into
MIME parameter values and similar horrors, compare the RFC 3696 errata.

Escaping backslashes is a pain, the USEFOR WG needed some months^Wtime
to figure this out.  And for 2831bis it strikes again.  Probably it's
a matter of taste, I recall times when I desperately tried \\ or \\\\
or worse with sh or csh scripts.

> the security section needs to explicitly point at the security

Yes.  Only copying the same old UTF-8 security considerations again and
again is boring, a distraction from "real" (specific and fresh) issues.

Frank

---

From: John C Klensin <[email-masked]>
Date: Fri, 02 Feb 2007 18:02:13 -0500

--On Friday, 02 February, 2007 23:23 +0100 Frank Ellermann
<[email-masked]> wrote:

>> Um, the wording I've used is almost identical to that in the
>> charmod document (section 1.3).
> 
> The wording is okay, but IMO your reversed mnemonic
> U+[[N]N]NNNN is better than only U+NN.  With readers you never
> know, some like me never look into the prose if the ABNF is
> apparently clear, while others also including me look for the
> examples before ever reading a single word of the prose or
> ABNF, and if I understood John correctly his approach is more
> like the opposite, he looks into the ABNF if prose and examples
> are hopeless... ;-)  

Pretty close.  I'll look at formal and semi-formal definitions
first, but only if they cover semantics in addition to syntax.
If they don't, I tend to focus on semantics and worry about the
syntax details later.  Probably too many years spent worrying
about formal definitions of programming languages.

>>> a royal PITA in conjunction with <quoted-string>, when it
>>> results in multiple backslashes.
>  
> Yes, but it doesn't have to fight with putting
> <quoted-string>s into MIME parameter values and similar
> horrors, compare the RFC 3696 errata.
> 
> Escaping backslashes is a pain, the USEFOR WG needed some
> months^Wtime to figure this out.  And for 2831bis it strikes
> again.  Probably it's a matter of taste, I recall times when I
> desperately tried \\ or \\\\ or worse with sh or csh scripts.

I have to confess that an early pre-posting draft of
unicode-escapes-00 had a section that was supposed to be titled
something like
   Recommendation for \UNNNNNNNN
Processing various combinations of slashes, escapes, and named
characters got two slashes in the output, occasionally three,
and occasionally the escapes themselves, but never one.  I
imagine, given enough time, that I could figure it out, but I'm
really sympathetic your concern above.

> Yes.  Only copying the same old UTF-8 security considerations
> again and again is boring, a distraction from "real" (specific
> and fresh) issues.

See other note about the distinction between a spec about the
use of Unicode characters and one about escapes.

        john

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Feb 2007 03:03:48 +0100

> examples (at this point), "very widely deployed and used" trumps

When I mentioned hex. NCRs I meant XML, not SGML and its many ways
to save keystrokes.

> I hope, have learned.

The overall RFC 2277 rule is IMO "if you don't know what it is or
can't say what it is, it's UTF-8, and if that theory fails it's
UNKNOWN-8BIT".  And unlike RFC 2231 an URL can't say what it is.

> about escapes, you are picking nits that belong elsewhere

That nit is closely related to escape mechanisms providing for
31 or 32 bits, and attempts to get rid of leading zeros in these
mechanisms, which could fail without explicit delimiters.

> U+NNNN[N[N]] versus U+[[N]N]NNNN is a matter of taste.

If the idea is to reflect appendix A of Unicode 5, it talks about
stripping leading zeros until four digits are left.  In table A.1
it has U+HHHH vs. U-HHHHHHHHH, saying that this is the same as
\uHHHH vs. \UHHHHHHHH.  I haven't seen U-HHHHHHHH before, and I
won't miss it if you don't want to talk about it in the draft.

> my editorial judgment and preferences are going to prevail...
> at least until the document gets far enough along that I have
> to start arm-wrestling with the RFC Editor and _their_ judgment
> and preferences.

Oops, sorry, I thought the intended status was BCP, "an attempt
to prohibit escapes for UTF-8 entirely".

> about your style (or not) some of the time.

I like it, your drafts are almost always very interesting.  I was
really surprised when I stumbled over RFC 2345 some day ago, it
supports UTF-8 for whois.  So that wasn't a DeNIC invention after
all, and it's clearly older than RFC 3912 chapter 4.

But "interesting" sometimes includes "controversial", and then the
intended status is relevant.  As you say it makes no sense to nit-
pick your personal preferences.

> rather than contribute to, progress.

I don't read 99% of all drafts, and I don't try to contribute to
99.9%.  The remaining 0.1% somehow attracted my attention, often
because I like them, the opposite is also possible.  No idea how
easy it generally is, but this article alone took me about three
hours, because I tried to find out what "net ascii" really is, why
your "net utf-8" mentions an RFC that's not more available, where
the Unicode 5 notational conventions are, etc.

> unless your purpose is to introduce delays and lay down obstacles

Escaping that with <rant> and adding a qualifier doesn't make it
better.  If you don't want a discussion about the draft it's okay,
as you say we're free to submit our own drafts about this and / or
related topics.

Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 5 Feb 2007 23:15:03 +0100

On Fri, Feb 02, 2007 at 02:05:34PM +0100,
 Frank Ellermann <[email-masked]> wrote 
 a message of 34 lines which said:

> Probably the I-D should mention that one famous exception from its
> rule to avoid encoded UTF-8 is the URL form of IRIs.

And the protocol described in RFC 2324 :-)

---

From: John C Klensin <[email-masked]>
Date: Mon, 05 Feb 2007 19:34:59 -0500

--On Monday, 05 February, 2007 23:15 +0100 Stephane Bortzmeyer
<[email-masked]> wrote:

>> Probably the I-D should mention that one famous exception
>> from its rule to avoid encoded UTF-8 is the URL form of IRIs.
> 

Now _that_ is clearly the important one, probably sufficiently
so to make an exception to my principle that we should not start
enumerating protocols that might or might not need escapes and
might or might not use encoded UTF-8.

On the other hand, perhaps it is time to start working on
RFC2324bis to permit it to accept other escape characters and to
fully generalize its device control interface formats.

Thanks.  I needed that.

     john

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 7 Feb 2007 17:49:41 +0000

[...]

> they should be used.

Agreed.

>   C070: irrelevant.   if it is in range, this works.

Okay.

>   C077: needs to be fixed somewhere else.  This is just about a

Mumble. Anything above U+10FFFF is, by definition, not Unicode (okay, not
Unicode today). But I'm not going to fight this one any more.

However, it does bring something else to mind: whatever escape mechanism is
chosen, should there be a limit on the length of the hex string? Should an
implementation be expected to correctly handle:

  \u'000000000000000000000000000000000000000000000000000000000000000001234'
or
  \u'000000000000000000000008000000000000000000000000000000000000000001234'
?

What is "correct" in the latter case?

>   C078: Already prohibited

Okay.

> able to deal with non-characters in a sensible way.

Very true, and a point I should have realized.

> examples (at this point), "very widely deployed and used" trumps

Um, I'm not sure what Frank was thinking, but I would say:
- it's fine to mention HTML;
- we should *not* mention the construct "<B>&aring</B>" (note the missing
  semicolon), which SGML allows and HTML says SHOULD NOT be used.

> is not a proposal to retrofit any existing protocol.

As someone else asked, what about new protocols based on IRIs?

> No, because of net-ascii and some other issues, probably not.

"net-ascii"? The only reference I can find is in RFC 1350, where it appears
to mean "ASCII with top bit clear, plus some codes from RFC 764 which have
the top bit set".

My point is that Unicode is not octet-based at all - encodings like UTF-8
are, but Unicode isn't. It's a numbering of characters from 0 to 0x10FFFF
just like ASCII is a numnbering from 0 to 127. Neither are octet based.

> about escapes, you are picking nits that belong elsewhere (see

Nits have to be picked at some point, and sometimes earlier is better than
later.

> and all sorts of other things too.

Because they are something that RFCs often use or contain, and for which
this is highly relevant. Unlike Japanese business cards.

> But that is introductory

I'm not suggesting exhaustive; I'm suggesting that mini-languages are a
relevant target for this specification.

> as good as another.

I don't agree, but ....

> string terminator,

No, it's a delimiter. A string terminator marks the end of the *string* -
in C, for example, it's the terminating " in the source code or the zero
byte at run-time. A delimiter is something that separates one part of the
string from another.

> but whether there is a starting delimiter at

Delimiters don't have to be in pairs.

> possible.

You've completely missed my point.

The security section needs wording along the lines of:

    An escape mechanism such as the one specified in this document can
    allow characters to be represented in more than one way. Where
    software interprets the escaped form, there is a risk that security
    checks are done at the wrong point.

    For example, a security system might prohibit the substring
    "/../" within certain strings. If so, an attacker could attempt to
    avoid the test by sending "/\u'002E'\u'002E'/" instead. If the
    security check is made before interpretation of escaped characters,
    the attack will be successful.

> the RFC Editor and _their_ judgment and preferences.

Or until Last Call shows that you're in a very small minority.

> I really

No, it isn't. It's showing the *semantics* of the omission.

> about your style (or not) some of the time.

Oh no. The last time I accepted that challenge, I ended up authoring a 125
page RFC.

> Sniping and

Excuse me, I'm not sniping (at least, not deliberately; if I'm giving that
impression, I apologise). Yes, I'm nit-picking sometimes, but that's
something that should be done if the resulting document is to be clear,
precise, and useful.

Note, for example, that I'm not nit-picking about what I consider to be
wrong spelling and grammar where I'm aware that it's a dialectic variant.

> If you are not going to
> pick up some of the writing work,

Where I have text to offer - be it 3 words or 3 pages - be assured that I
will. See above, for example. But sometimes it's a lot easier to make the
general point and let the author apply it.

> unless your purpose is to
> not--

It is not.

> can we please concentrate on substantive issues and

If I didn't think my suggestions had a positive impact, I wouldn't make
them. Equally, sometimes I am asking questions or trying to get an idea of
what people think. Perhaps *I*'m in the minority.

> I'm willing
> to be convinced but note that every time I put something into a
> document that is not strictly necessary I get attacked for
> excessive length, etc.                             

Not by me. Examples and explanations are often a good thing. I could have
made RFC 3977 about 40 to 50 pages shorter if I'd taken that approach.

Frank wrote:
> Yes, the matter of 21 vs. 31 bits was recently discussed on the
> Unicode list in conjunction with a (hypothetical) "UTF-21", maybe
> Clive had that discussion in mind.

No. I just object to people introducing octets (or, even worse, "bytes")
when they're unnecessary and - to my mind - confuse the issue.

I note, for example, that the entire POSIX process didn't understand the
difference between the two until I pointed it out, and pointed out some of
the wording changes needed to deal with it. At which point they decided
that POSIX would have to be limited to implementations where they are the
same.

---

From: John C Klensin <[email-masked]>
Date: Wed, 07 Feb 2007 16:25:42 -0500

--On Wednesday, 07 February, 2007 17:49 +0000 "Clive D.W.
Feather" <[email-masked]> wrote:

> However, it does bring something else to mind: whatever escape
> mechanism is chosen, should there be a limit on the length of
> the hex string? Should an implementation be expected to
> correctly handle:
> 
> \u'00000000000000000000000000000000000000000000000000000000000
> 0000001234' or
> 
> \u'00000000000000000000000800000000000000000000000000000000000
> 0000001234' ?
> 

I have assumed that compatibility with existing practice and
good sense suggests that the string should be at least four and
no more than eight hex digits in length.  Obviously, I didn't
write that down.  This is probably a strong argument for
including ABNF for all of the even slightly-recommended forms
(see previous note).   It is not fixed in -02, which has been in
the posting queue since Monday morning, but I'll try to get it
in -03.

>> Sorry, but, IMO, for a document like this which is merely
>> giving examples (at this point), "very widely deployed and
>> used" trumps "horrible".
> 
> - we should *not* mention the construct "<B>&aring</B>" (note
> the missing   semicolon), which SGML allows and HTML says
> SHOULD NOT be used.

It is not mentioned.   Even &aring; is not mentioned.  That is a
whole different sort of abstraction than what the document talks
about.  Perhaps it should be mentioned; if you think so, text
would be helpful.

>> (famous or not), but as mistakes from which we should learn
>> and, I hope, have learned.   The document is reasonably clear
>> that it is not a proposal to retrofit any existing protocol.
> 

Words here won't help, IMO.  Put differently, this is not the
right place to put the words.  If we can get reasonable
consensus on some variation of this and a few other things
wrapped up, I think the right approach is to revisit IRI itself
and see if we want to make changes, narrow scope, insert
warnings, or whatever.  And, in the meantime, I trust that these
conversations have raised the issues with the right group to
prevent new work from going forward without at least their being
considered.

I just don't know what else to suggest.  I have personally never
been happy about aspects of the IRI spec but, up through and
after the time that it was in Last Call, most of my concerns
were just intuitions that there were things in it that would get
us in trouble in the long term -- intuitions I wasn't able to
explain in terms of, e.g., concrete examples.   I also have
concerns that IRIs addressed a problem that didn't really need
solving and ignored the important ones.   Today, I could
probably do somewhat better at explanation and examples and
others reading this could probably do better than I can.   But
I'm still not ready to try to assemble either "IRIbis" or "IRIs
considered harmful".  If someone else is, he or she should go to
it.

> "net-ascii"? The only reference I can find is in RFC 1350,
> where it appears to mean "ASCII with top bit clear, plus some
> codes from RFC 764 which have the top bit set".

Long discussion.  But "ASCII with top bit clear", etc., implies
7-bit ASCII in an 8-bit unit.  If one were to transmit ASCII in
7-bit units, there is no top bit to be clear or set.  Some
systems, of course, actually did that, even though ASCII on the
network was defined in octet terms.

> My point is that Unicode is not octet-based at all - encodings
> like UTF-8 are, but Unicode isn't. It's a numbering of
> characters from 0 to 0x10FFFF just like ASCII is a numnbering
> from 0 to 127. Neither are octet based.

On the other hand, UCS-4 was certainly octet based, and UTF-32
is now presented as new terminology for UCS-4.   So the
situation isn't as clear, or as pure, as your comment above
suggests.  A bit more on this below.

> Nits have to be picked at some point, and sometimes earlier is
> better than later.

Ok.   It is just wearing me out on a document I somewhat
accidentally volunteered to assemble but don't feel strongly
committed to.   My problem, not yours, unless I give up and you
do care about it.   Co-authors would be welcome at this point.
 
> Because they are something that RFCs often use or contain, and
> for which this is highly relevant. Unlike Japanese business
> cards.

I still don't see what the stopping rule is.  And, while I may
not be looking in the right places, I don't see Cosmogol (for
example) playing the role in IETF protocol definitions that,
e.g., ABNF or ASN.1 do.
 
> I'm not suggesting exhaustive; I'm suggesting that
> mini-languages are a relevant target for this specification.

If you suggest text that explains why they are relevant and what
the impact is, and others agree that it is important enough to
justify the added length, I'll happily drop that text in.

>> Partially fixed in -02.   U+NNNN[N[N]] versus U+[[N]N]NNNN is
>> a matter of taste.  Clearly, in that pseudo-notation, one "N"
>> is as good as another.
> 
> I don't agree, but ....

Maybe this is connected to our slightly different views of
octets/ bytes (and perhaps "nibbles") above.  Or maybe not.
And, fwiw, if there were a requirement along the lines of "no
leading zeros in strings unless they are needed to make the
string at least four digits long", then I would certainly want
to write
   U+[[1]M]NNNN
with N in the range 0..F and
M in the range 1..F
or something like that.

But, at least so far, we don't have that rule and, so far, I'm
happy to leave deciding on whether or not it is needed to those
specifying particular escapes for particular protocols.

>> I was deliberately trying to be general.  If one has a form
>> like H'nnnn', one is clearly talking about "string
>> delimiters". However, if one has, e.g., &#xNNNN;, it is clear
>> that ";" is a string terminator,
> 
> No, it's a delimiter. A string terminator marks the end of the
> *string* - in C, for example, it's the terminating " in the
> source code or the zero byte at run-time. A delimiter is
> something that separates one part of the string from another.

Sigh.  I think that "terminator" is still right.  If I read your
definition, whether it is correct or not depends on the
definition of strings and substrings. And part of that
definitional question goes back to discussions long ago that
I'll discuss only over appropriate beverages (coming to Prague?).

However, I don't think this is worth shedding blood over so,
unless someone else objects, -03 will use "delimiter" throughout.
 
>     An escape mechanism such as the one specified in this
> document can     allow characters to be represented in more
> than one way. Where     software interprets the escaped form,
> there is a risk that security     checks are done at the wrong
> point.

While I do not believe this is necessary or the right place to
put these sorts of warnings (IMO, they belong in something like
2277bis or a "safely using Unicode" doc), it is at worst
harmless, so I'm invoking the "not willing to shed blood"
principle.  Text included in -03 (with an added comment about
checks for minimal or normalized forms).
 
>     "/../" within certain strings. If so, an attacker could
> attempt to     avoid the test by sending "/\u'002E'\u'002E'/"
> instead. If the     security check is made before
> interpretation of escaped characters,     the attack will be
> successful.

See above.

>> preferences are going to prevail... at least until the
>> document gets far enough along that I have to start
>> arm-wrestling with the RFC Editor and _their_ judgment and
>> preferences.
> 

Which, in the case of _purely_ editorial preferences, involves
some probability of the author saying "I don't need this; find
someone else to finish the document or let it drop".

>> I really
>> appreciate comments about how to make a document more clear
>> but an argument about, e.g., U+NNNN[N[N]] versus U+[[N]N]NNNN
>> is only about taste and is hence a waste of everyone's time.
> 

Not without more words or a more complete definition, IMO.  See
the comments above about U+[1[M]]NNNN.  Do you think that is
worth belaboring?  Do others?

As I think about it, our difference in view may rest in my
viewing U+NNNN[N[N]] pretty much as a name for a concept that I
assumed everyone reading the document was likely to already
understand.  I could, equally comfortably, have just used U+NNNN
or U+NNN... and put in some words, once, about actual lengths.
In fact, that may have been where the original use of U+NNN came
from.   You are viewing it, I guess, as a piece of semi-formal
metanotation with associated semantics, etc.  If I had seen it
that way, I would probably have gone all the way to ABNF on the
theory that U+[[N]N]NNNN does little more than give a strong
hint.

No commitment yet, but, if no one but Clive and I have strong
opinions about this, I am likely to invoke the "no bloodshed"
rule and change this too... but I'd really like to hear from
others about whether to go to
   U+[[N]N]NNNN
or something along the lines of 
   U+[[1]M]NNNN
or to giving this critter a name and writing ABNF, thereby
ridding ourselves of trying to communicate a lot of information
in a 12-character notational form.  If we do go to ABNF, I also
need guidance as to whether to write a leading zero rule into it
(preferably in the form of the ABNF that people would prefer).

> Oh no. The last time I accepted that challenge, I ended up
> authoring a 125 page RFC.

And here I assumed that 2821 would never qualify for a brevity
award in the messaging space.  Time to go add 50 pages to
2821bis :-)

> Excuse me, I'm not sniping (at least, not deliberately; if I'm
> giving that impression, I apologise). Yes, I'm nit-picking
> sometimes, but that's something that should be done if the
> resulting document is to be clear, precise, and useful.
> 
> Note, for example, that I'm not nit-picking about what I
> consider to be wrong spelling and grammar where I'm aware that
> it's a dialectic variant.

Ok.  For various historical reasons, I often actually prefer
what I assume is your dialect to my native one.  The RFC Editor
generally doesn't.  If you are so inclined, we should have an
offline discussion and then I'll send you the XML and let you
have at it -- part of what is going on here is that this
document is a very marginal activity for me right now and the
time needed to insert trivial and low-priority changes feels
burdensome.  But, if you felt inclined to make a stylistic
editing pass, we should chat to find out whether we can agree on
criteria and limits.

> Where I have text to offer - be it 3 words or 3 pages - be
> assured that I will. See above, for example. But sometimes
> it's a lot easier to make the general point and let the author
> apply it.
>...

Ok.   As long as we can both understand that there are limited
here, even if we don't precisely agree about where they are,
keep those cards and letters coming.  Just for planning
purposes, I'll probably get -03, containing the changes
discussed above and whatever else comes up (including the
results of corrections or arguments about things I've put into
-02, which was finally announced about five minutes ago), out in
the next week or two. But, if more versions are needed after
that, I will have to put the thing aside until post-Prague.

> Not by me. Examples and explanations are often a good thing. I
> could have made RFC 3977 about 40 to 50 pages shorter if I'd
> taken that approach.

See snide comment about its length relative to that of 2821
above.  We probably mostly agree, but several people in the
community don't... or are willing to insist on their favorite
example and then complain about length vis-a-vis everyone else's.
 
> No. I just object to people introducing octets (or, even
> worse, "bytes") when they're unnecessary and - to my mind -
> confuse the issue.

And most of us who first encountered the Internet or ARPANET, or
computer systems generally, at a time when "characters" could
come in 5, 6, 7, 8, 9, or 12 bit units (and maybe some others),
with or without padding, are sensitive to that issue too.  I
just don't know whether this document is the right place to make
that point, partially because of the UCS-4 problem mentioned
above (and the corresponding original definition of ISO 10646 as
a 32-bit character set) and partially because I have no real
confidence that the 21-bit limit will last (any more than seven
or eight bit limits on characters or eight or 32 bit limits on
address have lasted).  So, in practical terms, I'm strongly
inclined to say that, if we have a variable-length delimited
string, it can be up to eight hex digits long and folks need to
be prepared to parse that much although individual protocols
adopting escapes can impose a leading zero rule.
 
> I note, for example, that the entire POSIX process didn't
> understand the difference between the two until I pointed it
> out, and pointed out some of the wording changes needed to
> deal with it. At which point they decided that POSIX would
> have to be limited to implementations where they are the same.

Sigh.  Unfortunately, not the only thing wrong in the POSIX
process.  We should have a drink sometime and swap stories.

best,
    john

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Fri, 9 Feb 2007 14:24:17 +0000

John C Klensin said:
>> the hex string?

Okay.

> about.

Sorry, I shouldn't have mentioned &aring; in my example. I see that -02
talks about the missing semicolon, so I'm happy that this be closed.

> suggests.

Not quite, no.

> do care about it.

Sorry, I didn't mean to stick a huge load on you by my comments.

> Co-authors would be welcome at this point.

Not at the moment, I'm afraid. If things change at work, then perhaps.

> example) playing the role in IETF protocol definitions that,

Well, ABNF is another mini-language that might benefit from a way to handle
Unicode.

> justify the added length, I'll happily drop that text in.

In -02, change the last paragraph of 1.1 to:

    In addition to the protocol contexts addressed in this specification,
    escapes to represent Unicode characters could also be useful in formal
    languages (such as ABNF and Cosmogol) and in presentations to users
    (i.e. user interfaces). The formats specified in, and the reasoning of,
    this document may be applicable to these contexts as well, but this is
    not a proposal to standardise them.

That would satisfy me.

> with N in the range 0..F and

Something like that, yes, since as written you've ruled out U+10xxxx.

That's also, perhaps, going too far in putting the detail in the text.

[...]
> I'll discuss only over appropriate beverages (coming to Prague?).

Regrettably not - I can't justify the time. But if you're ever in London or
Cambridge ....

> principle.

Thanks.

I can agree with "not necessary"; I just think it's a good idea (see other
discussions on minimalism).

>>> preferences are going to prevail...

True.

> metanotation with associated semantics, etc.

Or, more precisely, I'm concerned that others may read it that way. As you
say, you and I know what it's supposed to mean.

[My day job involves doing this sort of analysis on draft legislation.
You'd be amazed just how badly people can misinterpret even the slightest
ambiguity.]

> in a 12-character notational form.

I think that the ABNF approach is probably a good one.

> If we do go to ABNF, I also

    unicode-notation = "U+" code-point
    code-point = bmp-code-point / extended-code-point
    bmp-code-point = 4hex-digit
    extended-code-point = (non-zero-hex-digit / "10") 4hex-digit
    hex-digit = "0" / non-zero-hex-digit
    non-zero-hex-digit = "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" /
        "9" / "A" / "B" / "C" / "D" / "E" / "F"

> have at it

No thanks. I actually view this as being very much an author's privilege.
Throughout 3977 I used *my* choices as to correct spelling and grammar, and
fought the RFC Editor when necessary. You're the author, so these are your
choice IMAO.

> come in 5, 6, 7, 8, 9, or 12 bit units (and maybe some others),

[5 and a half in my case: I learned my programming on a system with 39 bit
words, and a character encoding with 5 bits with two shift states - there
were two separate packing systems, one of 7x5 with explicit shifts, the
other of 6x6 with shift bits on each character.]

> partially because I have no real
> confidence that the 21-bit limit will last

Point.

> So, in practical terms, I'm strongly

Okay.

See above.