Subject: Re: [art] Modern Network Unicode

From: John C Klensin <[email-masked]>
Date: Tue, 09 Jul 2019 14:16:05 -0400

Carsten,

I will study this more when I get time, but a few quick
observations.  I have glanced at your revised versions, but
started this response before either -01 or (obviously) -02 were
published so will just finish it and get it off.

If it makes a difference, I am writing as co-author of RFC 5198.
I regret that my late, much-lamented, and far more articulate
co-author is not not available to do so.  Those you knew Mike
might develop a smile in his memory by contemplating how he
would have expressed himself about some of the issue below.

Some things to think about that the drafts do not appear to
reflect...

(1) The network acquired the CRLF convention because the
earliest versions of ASCII defined LF strictly as a movement to
the same character position on the next line.  For anyone old
enough to remember typewriters, that is the function of rolling
the carriage up one line.   From the same period and
interpretation, CR alone means "back to first character position
on the current line".   In that interpretation, if one wanted
the first character position on the next line (sometimes known
as the "new line" function), one needed CR LF.  To make things a
bit more complicated, some systems at the time used LF alone as
new line and others used CR alone.  If a few lines were entered
on one such system and transferred to another, CR LF would
always produce the expected result, but bare CR, bare LF, or LF
CR might cause surprises.

There are symmetric relationships among SP and HT, LF and VT,
and CR and BS; considering them might help explain how we got
here (and, for those who don't know, part of the reason why BS
(U+0008) is defined as non-destructive on many systems).

Later versions of ASCII changed several of the definitions to
be, essentially, "interpret it however you like".  Of course,
from an interchange standpoint, that just made things worse.
ISO/IEC 6429 tried to fix the problem by defining new control
codes with specific meanings (see U+0084 and U+0085), but they
never took off and hence added to the confusion.   Unicode,
having learned from that experience, solved it by adding yet
more codes, notably U+2028, which, as you note, didn't go
anywhere either.

Telnet (and FTP TYPE A and perhaps email) may be history
although I think we lost quite a bit in going from the
network-standard forms specified at the core of those protocols
to the "everything is like Unix" or "everything is like me"
models that seem to be pervasive in many more recent
developments.  It may be a bit ironic that your solution to some
other problems is to devise yet another network-standard form,
especially one that has options that basically encourage (or
require) profiles (or, if you prefer, combinations of variances).

As long as one only intends to store or display the results of
whatever is transmitted and the display device follows more less
the same conventions as the transmitting one, none of that makes
any difference, at least in the overwhelming number of cases.
If, however, there is any requirement to compare a set of
strings, things get more complicated in a hurry (see below).

(2) Especially given recent knowledge and discoveries, requiring
NFC is probably a mistake and might have been a mistake when
5198 was written.  NFKC is worse.   If you read the Unicode
Standard, you will find that the goal of normalization is
generally to permit application to convert two strings to a
common form before they are compared to provide comparisons that
would nearly approximate what people would expect.  That
pre-comparison conversion is probably why Unicode uses the term
"canonical form" although it doesn't include some of the things
the rest of us have historically means by that term in specific
applications.   However, converting strings to normalized form
before transmission may lose information that was intended by
the user to affect how characters are displayed.  Because of
some characteristics of the DNS and IDNA, there was a special
need for stored NFC-strings.  That might generalize to some of
the PRECIS work and possibly to identifiers in general, but
certainly does not extend to transmission of general Unicode
strings.  For the NFC case, it is easy to make the wrong
generalization from Latin script, where it is generally harmless
at worst but that is not the case more generally.  NKKC is even
more problematic: eliminating half-width characters in favor of
their normal-width equivalents may be harmless (or may not), but
dropping, e.g., mathematical symbols or rarely-used CJK
characters in favor of Latin or Greek character or more commonly
used CJK ones with similar interpretations can be a disaster.
If you want a variation for "loses information but maybe no one
cares" then you might put NFKC in there along with several other
things :-(

(3) The comparison problem is, as mentioned above, far more
difficult.  At the most superficial level, the web and some
other modern applications allow either LF or CRLF.  That might
have been a mistake -- or maybe it is ok.  It does means that
strings (or whole files) have to be converted to some canonical
form for comparison, that sizes change, etc.  The difficulties
pervade the many issues we've had with IDNA over the years,
including the question of whether or not NFC is adequate to deal
comprehensively with the problems (it isn't) and what should be
done about the cases where it is not.  For a discussion of some
of the issues, see https://github.com/w3c/charmod-norm

You might want to try to encourage the i18n directorate (copied
on this note) to review this before you get much further along.  

best,
   john

--On Sunday, 07 July, 2019 22:14 +0200 Carsten Bormann
<[email-masked]> wrote:

> This week, I have been asked twice (for independent drafts)
> what these drafts should say about their usage of Unicode.  It
> turns out, we have an RFC that is almost, but not entirely
> unlike what is needed to reference for a new protocol that
> uses text: RFC 5198. I have tried to build on that in the very
> short draft:
> 
> 
> 
> I would expect that such a specification, once completed,
> would be a useful point of reference for any new protocols
> that need to use text without any specific requirements
> imposed by their applications. (Protocols that already have
> been around probably benefit less, because there always are
> legacy considerations.)
> 
> I'm not yet asking for dispatching this document; right now I
> would just like to receive feedback from this esteemed
> community to further improve the document.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 9 Jul 2019 23:50:44 +0200

Hi John,

thank you for chiming in.  I would have summoned you (and i18ndir) in time; in the current phase of rapid respins of the document I was first trying to make it useful before trying to nail down the details.

I was a toddler when ASA X3.4-1963 was released, but I caught up to ASCII about a dozen years later.  I have used ASR33s (although I spent more time on LA36s), and I have fully internalized the teleprinter model that led to NVT.  (I also spent some time on glass tty models of teleprinters [1].)  But while the teleprinter model has originally shaped what became “text files”, the latter are now living a life of their own.  MNU in effect tries to codify the emancipation of text from the NVT model, and I’m not surprised that this breaking away causes some sorrow.

> require) profiles (or, if you prefer, combinations of variances).

Most new applications will be fine with (1D) CMNU or (2D) MNU with lines.
But sometimes legacy rears its ugly head, and therefore additional variances are defined.  I believe it is much better to expressly define these variances than to lump all the legacy into one big blob (single profile) that confuses the cleaner applications of MNU.

Now what is text being used for?  Some is exclusively for display to humans in 1D (CMNU) or 2D (MNU with lines) environments.
Some is then also used by machines, and that is where easy comparison comes in.
But the advantages of reasonably predictable encoding go way beyond that; which is why NFC coding is a de facto standard in most places that would use MNU.
(NFKC is in the current document mostly as a reminder that variances can be made in normalization, as well; it is probably the only reasonable one beyond NFC among the normalization forms, but has its own problems as you note.)

The CRLF (which really has survived only because the ASR33 needed up to 200 ms for a CR and the LF was a good excuse to waste those other 100 ms) is no longer needed; with the exception of a single popular operating system family, bare LF has won the line ending competition.  CRLF is one legacy feature that it’s worth getting rid of (except maybe as the “CR-tolerance" variance), as is HT.  (I’m assuming FF and VT are no longer relevant in most of today’s applications.)  All this may be a bit opinionated, but it is also forward-looking.  If full NVT is needed, we can always reference RFC 5198; MNU is for the cases where that is not needed.

I hope this doesn’t come over as too brash — I really like RFC 5198 and the careful tradeoffs it makes, but there are lots of applications that need something simpler and MNU reflects current practice and is a sane design for the 2020s+ (*).

Grüße, Carsten

[1]: https://en.wikipedia.org/wiki/GNU_Screen
(*): Yes, I’m fully aware that nothing about text and writing systems is genuinely simple; Einstein/Sessions applies.

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 09 Jul 2019 21:43:15 -0400

Carsten Bormann <[email-masked]> writes:
> (although I spent more time on LA36s)

My memory is that the LA36 had a configuration button on it titled "AUTO
LF".  Which rather symbolizes the difficulties of this topic.

Dale

---

From: John C Klensin <[email-masked]>
Date: Wed, 10 Jul 2019 03:09:54 -0400

--On Tuesday, July 9, 2019 23:50 +0200 Carsten Bormann
<[email-masked]> wrote:

> Hi John,
> 
> thank you for chiming in.  I would have summoned you (and
> i18ndir) in time; in the current phase of rapid respins of the
> document I was first trying to make it useful before trying to
> nail down the details.

The counterargument is that it is better to catch any
significant misconceptions early.  I haven't seen those in your
draft(s), but couldn't know without looking through them.

> I was a toddler when ASA X3.4-1963 was released, but I caught
> up to ASCII about a dozen years later.  I have used ASR33s
> (although I spent more time on LA36s), and I have fully
> internalized the teleprinter model that led to NVT.  (I also
> spent some time on glass tty models of teleprinters [1].)  But
> while the teleprinter model has originally shaped what became
> "text files", the latter are now living a life of their
> own.  MNU in effect tries to codify the emancipation of text
> from the NVT model, and I'm not surprised that this breaking
> away causes some sorrow.

Understood.  And, as you have probably figured out, 5198 was an
attempt to update NVT but to do so in as direct a way as
possible, which meant carrying some of the NVT baggage along.
I also suspect from your explanation that I'm close to 20 years
older that you are and have a larger selection of
contemporaneous scars from that period.  Probably less sorrow or
nostalgia than you might assume, but a keen sense of why some
things got to be the way they are.

I will warn you that another effect on me of that period is an
inclination toward simplicity and a minimum of options.  That is
not specifically an NVT issue, but NVT (and early Telnet) show
some of that thinking.  I would even claim that one of the
reasons the Internet succeeded is that many people treated
"profile" as nearly an obscenity.

In principle, I have no difficulty with what I think you are
trying to do.  I do think it is going to need more explanation
about why, scope of applications, etc., than the I-D has now.
Probably that can wait although getting to at least some of it
relatively early may help clarify what this is all about.

> Some specific comments:
> 
>> developments.  It may be a bit ironic that your solution to
>> some other problems is to devise yet another network-standard
>> form, especially one that has options that basically
>> encourage (or require) profiles (or, if you prefer,
>> combinations of variances).
> 
> Most new applications will be fine with (1D) CMNU or (2D) MNU
> with lines. But sometimes legacy rears its ugly head, and
> therefore additional variances are defined.  I believe it is
> much better to expressly define these variances than to lump
> all the legacy into one big blob (single profile) that
> confuses the cleaner applications of MNU.

I will need to think more about this.  It may depend a bit on
how the variances are presented and what you encourage and
discourage.  And I'd be happier if, e.g., what is needed for 2D
were separated more clearly from accommodations for legacy
situations.

> Now what is text being used for?  Some is exclusively for
> display to humans in 1D (CMNU) or 2D (MNU with lines)
> environments. Some is then also used by machines, and that is
> where easy comparison comes in. But the advantages of
> reasonably predictable encoding go way beyond that; which is
> why NFC coding is a de facto standard in most places that
> would use MNU. (NFKC is in the current document mostly as a
> reminder that variances can be made in normalization, as well;
> it is probably the only reasonable one beyond NFC among the
> normalization forms, but has its own problems as you note.)

This difference in perspective may or may not be useful, but NFC
is also a close approximation to what any sensible terminal
driver or IME is going to produce natively from a plausible
keyboard layout for the relevant script.  It would just make no
obvious sense to start decomposing characters selected by a
single keystroke (even if multiple fingers were involved), table
entry selection, or equivalent.  If you stick to
Unicode-specified normalization (and there are no
generally-known alternatives), there is nothing else "beyond"
NFC.  NFD and NFKD decompose, which has disadvantages implied
above by both your causal hypothesis and mine.  On the other
hand, there is at least one modern operating system that, IIR,
prefers NFD internally. 

> The CRLF (which really has survived only because the ASR33
> needed up to 200 ms for a CR and the LF was a good excuse to
> waste those other 100 ms) is no longer needed; with the
> exception of a single popular operating system family, bare LF
> has won the line ending competition.  CRLF is one legacy
> feature that it's worth getting rid of (except maybe as the
> "CR-tolerance" variance), as is HT.  (I'm assuming FF and
> VT are no longer relevant in most of today's applications.)
> All this may be a bit opinionated, but it is also
> forward-looking.  If full NVT is needed, we can always
> reference RFC 5198; MNU is for the cases where that is not
> needed.

I think you are wrong about the above although I could provide
an even better justification for obsolescence based on trying to
simulate "extended Latin" characters by overstriking.  For
example, in Unicode, we have "ó" (U+00F3) and U+006F U+0301,
which turns into U+00F3 if NFC is applied, but consider the
ASCII 
   o BS '
an example that turns out to be historically very important.
 
> I hope this doesn't come over as too brash — I really like
> RFC 5198 and the careful tradeoffs it makes, but there are
> lots of applications that need something simpler and MNU
> reflects current practice and is a sane design for the 2020s+
> (*).

As I said, I thoroughly approve of this effort.

For the record, I do have one other concern.  The examples above
use extended Latin script.  Because of its NVT origins, much of
5198 makes assumptions about that script or scripts closely
related to it.  If you are doing something for this century and
beyond, you should really think carefully about the implications
of scripts that are very different.   For example, does anything
change when using a script that is primarily RtoL?  Do you need
to be concerned about non-spacing breaks and other
usually-invisible separators for strings that might be wrapped
in display or where graphemes are intrinsically not monospaced?
Do you need rules about rational string composition or character
sequences for complex scripts?  At one level, those questions
are where your Einstein/Sessions comment really fits in.  At
another, I  think what you are trying to do gets much less
useful if you blow those questions off so think you should at
least try to work through them.

best,
   john

> (*): Yes, I'm fully aware that nothing about text and
> writing systems is genuinely simple; Einstein/Sessions applies.
>

---

From: John C Klensin <[email-masked]>
Date: Thu, 11 Jul 2019 12:31:31 -0400

(top post and adding the ART list back in)

Carsten, 
I think Asmus and I are in complete agreement, but you might
find an attempt to summarize the state of work for relevant
characteristics and behaviors on the web, some of which interact
with this discussion, illuminating.  It is a chart with
explanation, not a long document.  See
https://w3c.github.io/typography/gap-analysis/language-matrix.html 

best,
   john

--On Thursday, July 11, 2019 08:51 -0700 "Asmus Freytag (c)"
<[email-masked]> wrote:

> On 7/11/2019 12:03 AM, Carsten Bormann wrote:
>> On Jul 11, 2019, at 01:00, Asmus Freytag
>> <[email-masked]> wrote:
>>> There are a few scripts where un-normalized text is
>>> "preferred" by the user community over NFC. In some cases,
>>> the most natural ordering of combining marks does not match
>>> NFC's canonical ordering. I other cases, NFC does not
>>> compose some sequences while local user communities strongly
>>> prefer the precomposed code points (e.g. Bengali).
>> Interesting.  There are also a few writing systems where NFC
>> is what is strongly preferred by the user community.
>> 
>> Is there any effort to capture these preferences in a formal
>> way so they become more accessible to international developer
>> communities?
> 
> No. Note least because these preferences may not be stable
> over (longish) periods of time as they depend a bit on what
> keyboards / rendering platforms and apps handle best today.
> 
> A./
> 
>> 
>>> Those scripts would be an exception to John's statement: "
>>> NFC is also a close approximation to what any sensible
>>> terminal driver or IME is going to produce natively from a
>>> plausible keyboard layout for the relevant script", a
>>> statement that otherwise holds well.
>> Right.  It is interesting to see how Apple recently moved
>> from a normalizing file system (HFS+ was normalizing [to NFD
>> unfortunately, and using UTF-16 on disk]) to a
>> normalization-preserving, normalize-before-comparison
>> ("normalization-insensitive") file system (APFS).  It is
>> also interesting how the problem is "solved" in
>> applications such as git (core.precomposeunicode needs to be
>> set to true on platforms that tend to generate non-NFC names
>> so other platforms can pretend to stay blissfully unaware).
>> 
>> Grüße, Carsten
>> 
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 11 Jul 2019 18:44:26 +0200

> https://w3c.github.io/typography/gap-analysis/language-matrix.html 

Thank you for this interesting link.
This is very condensed information; what column should I look for specifically on the issue of normalization and applicability of NFC?
E.g., in the linked page https://w3c.github.io/iip/gap-analysis/beng-gap I find something about grapheme clusters but nothing about normalization — normalization of course is not an immediate need for Web interoperability.

Grüße, Carsten

---

From: John C Klensin <[email-masked]>
Date: Thu, 11 Jul 2019 13:25:09 -0400

--On Thursday, July 11, 2019 18:44 +0200 Carsten Bormann
<[email-masked]> wrote:

>> characteristics and behaviors on the web, some of which
>> interact with this discussion, illuminating.  It is a chart
>> with explanation, not a long document.  See
> This is very condensed information; what column should I look
> for specifically on the issue of normalization and
> applicability of NFC? E.g., in the linked page
> https://w3c.github.io/iip/gap-analysis/beng-gap I find
> something about grapheme clusters but nothing about
> normalization — normalization of course is not an immediate
> need for Web interoperability.

I'll try to hunt for it when I have time unless someone else
gets to it first, but there are W3C recommendations that
specifically recommend against trying to insert a normalization
step before transmission or immediately before storage but
focusing on it as a comparison-time operation.   That is
consistent with what Asmus and I have been trying to tell you.  

Let me try to summarize those comments in a different way:

(1) For most language-script combinations, the form in which a
network application gets a string is usually going to be the
right form for that string.  If there is a strong preference
that the string be normalized, it will probably come to the
network application (and probably when it leaves something
rather close to the keyboard) normalized.  If there is a strong
preference that it not be normalized, it will probably come to
you in _that_ form.  If there is no preference at all,
normalizing the string (as long as you stay away from NFKC/NFDC)
will probably not cause harm, but it won't buy much either.

(2) Checking as to whether a string is normalized before sending
it is unlikely to produce any safely actionable information
unless you know a lot about the language, script, and
circumstances.

(3) If a string is unnormalized for what whomever created it
thinks is a good reason, then normalizing on general principles
is going to lose whatever information that decision was intended
to convey.  

The above may be different for strings that are specifically
intended as identifiers, but I don't think that is what you are
talking about.    I used to believe that "normalize everywhere"
and "store strings only in normalized form" was a good idea, but
I've been gradually (and sometimes quite unpleasantly) educated
by a long series of cases and counterexamples.   If I were
updating 5198 today, that recommendation is the thing I would be
most likely to either change or at least moderate and qualify
considerably.

    john

---

From: Ira McDonald <[email-masked]>
Date: Thu, 11 Jul 2019 17:41:58 -0400

Hi,

Carsten - thanks for working on this MNU draft.

John - thanks for you excellent background comments.

FWIW - IEEE-ISTO PWG 5100.3 IPP Everywhere (a very widely implemented
driverless printing profile of IETF STD92 Internet Printing Protocol 1.1
and a few
other PWG IPP extensions) makes RFC 5198 and NFC normalization mandatory
for IPP Server generated names, keywords, and text strings and mandatory
for
IPP Client submitted names and keywords.  But IPP's purpose is reliable
document
printing (the document content may, of course, be in some legacy charset,
but must
be tagged) and most string attributes are names (like job names) or keywords
(fonts, media, etc.), so pure NFC w/ restricted C0 characters makes sense.

IPP Everywhere does have this warning in Internationalization
Considerations:

"WARNING – Performing normalization on UTF-8 strings received from IPP
Clients and
subsequently storing the results (e.g., in IPP Job objects) could cause
false negatives in
IPP Client searches and failed access (e.g., to IPP Printers with
percent-encoded UTF-8
URIs now 'hidden')."

Which speaks to John's point about the disadvantages of automatic NFC
(or any other normalization).

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Co-Chair - TCG Trusted Mobility Solutions WG
Co-Chair - TCG Metadata Access Protocol SG
Chair - Linux Foundation Open Printing WG
Secretary - IEEE-ISTO Printer Working Group
Co-Chair - IEEE-ISTO PWG Internet Printing Protocol WG
IETF Designated Expert - IPP & Printer MIB
Blue Roof Music / High North Inc
http://sites.google.com/site/blueroofmusic
http://sites.google.com/site/highnorthinc
mailto: [email-masked]
PO Box 221  Grand Marais, MI 49839  906-494-2434

On Thu, Jul 11, 2019 at 1:25 PM John C Klensin <[email-masked]> wrote:

> _______________________________________________
> art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/art
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 12 Jul 2019 01:46:41 +0200

This is a great discussion.

To me, it seems to converge on the following.

(1) Sending sane data is the job of the data originator.  

(2) Do not include gratuitous normalization steps in your processing, once the data have been originated in a sane form.

(2a) If you broke it, you fix it (as far as possible): If your processing steps did involve gratuitous normalization, you have to renormalize to NFC before sending.

Here, “sane” is defined as:

(0) Data SHOULD be originated in NFC, unless that would be inappropriate for the specific script, in which case the community consensus rules for the script govern.

For Latin script, this happens to collapse to what 5198 says.

This set of rules places the onus on the place where the data is generated, which is usually the place that knows most about the specific script and about the intent of the originator.  If you know that place isn’t doing its job, add the rule:

(1a) If the data originator does not do (0), the software placing the data on the network may need to sanitize (normalize towards sane).

1a is similar to 2a in that it doesn’t create perfect results, so both SHOULD be avoided — there is no way to, after the fact, perfectly sanitize data that weren’t originated sane or that were gratuitously normalized on the way.

With these definitions, MNU can direct towards: 
(A) Senders: send sane data
(B) Recipients: break as little as reasonable when data received isn’t sane
(C) B is not a valid excuse not to do A, and specifically: recipients are not expected to clean up after senders (because there is no correct way to do that).

(Rule C is the often forgotten third rule of the Postel principle.
It also means that an entity that is a recipient of MNU and then sends the data on as MNU has no need to gratuitously normalize, but it does not entirely get rid of rule 1a for recipients of data from places known not to be sane.)

Grüße, Carsten

---

From: John C Klensin <[email-masked]>
Date: Thu, 11 Jul 2019 22:36:28 -0400

--On Friday, July 12, 2019 01:46 +0200 Carsten Bormann
<[email-masked]> wrote:

> (2) Do not include gratuitous normalization steps in your
> processing, once the data have been originated in a sane form.
> 
> (2a) If you broke it, you fix it (as far as possible): If your
> processing steps did involve gratuitous normalization, you
> have to renormalize to NFC before sending.
> 
> Here, "sane" is defined as:
> 
> (0) Data SHOULD be originated in NFC, unless that would be
> inappropriate for the specific script, in which case the
> community consensus rules for the script govern.
> 
> This set of rules places the onus on the place where the data
> is generated, which is usually the place that knows most about
> the specific script and about the intent of the originator.
> If you know that place isn't doing its job, add the rule:
> 
> (1a) If the data originator does not do (0), the software
> placing the data on the network may need to sanitize
> (normalize towards sane).
> 
> 1a is similar to 2a in that it doesn't create perfect
> results, so both SHOULD be avoided — there is no way to,
> after the fact, perfectly sanitize data that weren't
> originated sane or that were gratuitously normalized on the
> way.
> 
> (B) Recipients: break as little as reasonable when data
> received isn't sane 
> (C) B is not a valid excuse not to do A,
> and specifically: recipients are not expected to clean up
> after senders (because there is no correct way to do that).
> 
> (Rule C is the often forgotten third rule of the Postel
> principle. It also means that an entity that is a recipient of
> MNU and then sends the data on as MNU has no need to
> gratuitously normalize, but it does not entirely get rid of
> rule 1a for recipients of data from places known not to be
> sane.)

I think this is exactly right.  And, fwiw, I agree with your
observation about the Postel principle.

   john