Subject: Re: Comments on Unicode Format for Network Interchange

From: "Markus Scherer" <[email-masked]>
Date: Mon, 18 Jun 2007 10:34:42 -0700

Dear Frank,

Sorry for the very late reply.

I would like to point out that my main interest in the Net-Unicode
internet-draft and in sending comments is with the discussion of
Unicode normalization and versioning. I happily note that you didn't
disagree with my suggestions in that area, except for one suggested
text deletion (where I now mostly agree with you).

As for the issues on line endings and signatures:

On 4/24/07, Frank Ellermann <[email-masked]> wrote:
> Markus Scherer wrote:
>
> > *** Suggested change:
> >    2.  Line-endings MUST be indicated by the sequence Carriage-Return
> >        (U+000D) followed by Line-Feed (U+000A), or by a single
> >        Carriage-Return (U+000D), or by a single Line-Feed (U+000A).
>
> -1F
>
> > Justification: We believe that single CR and LF are common because of
> > implementation practice on a variety of platforms, and that it is both
> > unrealistic and unnecessary to try to legislate them away.
>
> No, it causes havoc.
>
> > Applications already commonly handle all of CR, LF and CR+LF, and some
> > support even more characters according to the Unicode Newline
> > Guidelines.
>
> The draft isn't about arbitrary text or XML (where you'd also need NEL),
> it's about telnet.  It tries to extend ALPHA and DIGIT as used in some
> syntax constructs for text in Internet protocols, it doesn't try to
> introduce a new concept of "line" in these protocols.

I didn't understand from the internet-draft that it was only geared
towards telnet.

If it is indeed only about telnet, then the end-of-line defininition
is of course fine, and apologies for causing confusion.

If it is not intended just for telnet, then I believe allowing several
already-common forms of line endings is simply pragmatic.

> > *** Suggested change:
> >    4. The UTF-8 signature byte sequence (EF BB BF, UTF-8 encoding of
> >       U+FEFF, sometimes called Byte Order Mark ("BOM")), when it
> >       appears at the beginning of the text, SHOULD be deleted by the
> >       recipient.
>
> I don't think that works.  The draft isn't about local text or XML
> files,
> it's about Internet protocols, especially telnet, over the wire.

Fair enough, and I am not suggesting that Unicode signatures should be
used there. My suggestion was intended to clarify the specification
for when a Unicode signature is in fact included even when it is not
legal, not recommended, or simply not necessary. I find it easier, in
practice, to specify how to handle a situation than to just require
that it not occur.

> >       If a Word Joiner is needed in the text, U+2060 WORD JOINER SHOULD
> >       be used instead of U+FEFF ZERO WIDTH NO-BREAK SPACE.
>
> Already covered by STD 63 (RFC 3629).

Ok, thanks.

> > *** Suggested change:
> >    1.  Control codes from both the "C0" (U+0000..U+001F, U+007F)
> >        and "C1" (U+0080..U+009F) ranges,
> >        with the exception of HT (09), LF (0A) and CR (0D),
> >        SHOULD NOT be used unless required by exceptional circumstances.
>
> > Justification: The sets of C0 and C1 control codes that should and
> > should not be used should be defined explicitly, and with code point
> > values. Only HT, LF and CR are very widely used.
>
> Makes sense, but HT can have surprising effects if it's "expanded" into
> one or more spaces, that would need a "security consideration".  Does

Possible, although HT is rarely expanded to spaces in low-level processing.
The most important point here is that the set of control codes be
specified very explicitly.

> DEL really belong to the C0 set ?  Maybe avoiding these old terms is
> clearer for readers today.

I don't actually know if DEL is formally a C0 control code.
I would be perfectly happy with specifying the control codes via
Unicode code point ranges and not using the terms "C0" and "C1".
(Although for charset old-timers these terms might be useful.)

> > *** Suggested change:
> > Remove points 2. and 3.
>
> See above.  Try to edit a plain text file using LF as line-end with the
> tool styling itself as "editor" on a Windows box, and you'll see what I
> mean.  IIRC there were some hot debates why the IETF ftp server sends
> text files with (only) LF, in theory breaking the ABNF in these files.
> This is a rathole, please just accept it as some IETF oddity.  We're
> not forced to use CRLF in local files if we hate it.

Again, the importance of the line ending encoding is way below that of
the normalization and versioning discussion. Here I was simply
suggesting to allow what's already common practice - if not for
telnet, then for many other text formats.

> > *** Suggested change:
> > Drop this second bullet and the following paragraph.
>
> No, folks need to know that Unicode is a moving target to some degree,
> that only small and different subsets are supported by most devices,
> and that it's horribly complex in comparison with ASCII or many legacy
> charsets.  The advantage is obvious, some disadvantages are not.

I agree with you about not deleting the bullet ("The normalization
specified here, NFC (see Section 3), performs...") and the following
paragraph ("The NFC tables may be updated over time"), or not
entirely. However, the last sentence of that paragraph, which reads

  The stability of the Net-Unicode format is thus guaranteed when any
  implementation that converts text into Net-Unicode format does not
  permit unassigned characters.

should be deleted, because with a SHOULD for normalization the
stability of the Net-Unicode format does not depend on normalization
stability any more.

> > Suggested change: Please add a reference for [RFC3629] UTF-8, a
> > transformation format of ISO 10646
>
> There is a RFC 3629 (STD 63) reference, it's in the first part with
> the normative references.

Thanks, and sorry for the oversight.

Best regards,
markus

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 18 Jun 2007 23:18:14 +0200

> Sorry for the very late reply.

No issue, I still have that "ping" dated 2007-03-29 sitting in my 
"Todos" folder and waiting for better times when nobody tries to
update the core e-mail RfCs or similar emergencies... ;-)

> I didn't understand from the internet-draft that it was only
> geared towards telnet.

> is of course fine, and apologies for causing confusion.

The "only" might be misleading, a bunch of other protocols used
to be based on telnet, and inherited some features more or less
clearly.  I guess John has to explain in a new version of this
I-D what exactly he's talking about (a long string of "updates
RFCs x, y, z" in the header), and where he only proposes to
update other protocols (like "whois") in the spirit of his I-D.

Maybe 2821bis (SMTP) is an interesting example, at the moment it
still "allows" control char.s in places where I'd prefer to get
rid of them (as recommended in the "net-Unicode" I-D).

Of course 2821bis will insist on CR LF for lineends.  Just an
example, it's the same issue for many IETF protocols.  An IMO
bad case is FTP, where "a" (= ascii text, and convert lineends
for whatever passes as "local" convention) is the default.

The "local" convention supported on my box is "only CR doesn't
work as expected".  And besides I rarely use FTP for text and
get garbage when I forgot the "b" (binary).  Fortunately I was
never forced to grab a text file from an EBCDIC FTP server. :-)

> If it is not intended just for telnet, then I believe
> allowing several already-common forms of line endings is
> simply pragmatic.

It would cause havoc for some scripts I use (POP3, HTTP, SMTP,
ident, whois, simple stuff).  The SMTP and whois scripts fix
a body LF to CR LF on the fly, but they'd break miserably if 
somebody thinks that a bare ASCII CR is a lineend.  Let alone
Latin-* NEL, or weirder scenarios.

So it's not "only" telnet, but it still is about something
you can completely ignore, unless you need to post a NetNews
article with telnet to the NNTP port, or similar stunts.

 [BOM]
> My suggestion was intended to clarify the specification for
> when a Unicode signature is in fact included even when it is
> not legal, not recommended, or simply not necessary. I find
> it easier, in practice, to specify how to handle a situation
> than to just require that it not occur.

Okay, admittedly I don't like it if I get a mail where the
body starts with a BOM, because my obsolete MUA treats UTF-8
as windows-1252 claiming that it's Latin-1.  Something on the
side of the sender could have silently removed this BOM.  

But the I-D is more general, it has no concept of "body" or
SDU.  Maybe it should mention the BOM issue in an example (?)

For telnet it's IMO pointless, that's lines or rather control
sequences plus text defining the content of a screen, the I-D
can't say that a BOM should be silently removed when it would
end up at the begin of a line.  Ignoring BiDi, and besides the
draft deprecates all control char.s, even HT, allowing only
CR LF in this order.

Is it your idea to eliminate BOM like the I-D eliminates SUB
(= EOF on some platforms) ?  If that's your point I think it's
better solved in RFC 3629, that's already at STD.

Otherwise the I-D could arguably also talk about non-characters,
surrogates, overlong UTF-8, more-than-21-bits UTF-8, etc., all
addressed in RFC 3629.

> the last sentence of that paragraph, which reads
 
>   The stability of the Net-Unicode format is thus guaranteed
>   when any implementation that converts text into Net-Unicode
>   format does not permit unassigned characters.
 
> stability of the Net-Unicode format does not depend on 
> normalization stability any more.

I can't judge it, my last attempt to implement something in this
direction ended again with giving up.  IOW the SHOULD is wishful
thinking as far as my scripts are concerned.  Maybe I try it
again with "level one" later, actually I was only curious what
SASLPREP really does, in addition to its NFC part (it uses case
sensitive NFKC plus tons of prohibited and mapped to nothing or
mapped to space rules).

Frank