Subject: [I18nrp] Minutes / summary?

From: "Paul Hoffman" <[email-masked]>
Date: Sun, 22 Jul 2018 17:48:34 -0700

Greetings. I missed the BoF due to a conflict. Will minutes be coming 
soon? If not, could someone summarize on the mailing list what was 
"decided" at the meeting?

--Paul Hoffman

---

From: Peter Saint-Andre <[email-masked]>
Date: Sun, 22 Jul 2018 19:01:20 -0600

Pete is working on minutes, but a fair summary can be found here:

https://etherpad.tools.ietf.org/p/notes-ietf-102-i18nrp

On 7/22/18 6:48 PM, Paul Hoffman wrote:
> _______________________________________________
> i18nRP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i18nrp

---

From: Larry Masinter <[email-masked]>
Date: Mon, 23 Jul 2018 20:05:41 -0700

Is it time to talk about specific drafts?

This seems more general:

There is a problem in RFC 6365 “Terminology used…” that I think is at the core: It seems to imply that the way to compute equivalence of strings REQUIRES normalization to allow determining a~b iff normalize(a)==normalize(b). 
But in general, the equivalence relations one might want, for usability reasons, don’t have an acceptable normal form.

Continuing to put in normalization as an essential step is doomed to failure. Some protocols may need to be redefined to allow non-normalized strings. 

--
http://LarryMasinter.net

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 23 Jul 2018 20:38:15 -0700

On 7/23/2018 8:05 PM, Larry Masinter wrote:
> There is a problem in RFC 6365 “Terminology used…” that I think is at 
> the core: It seems to imply that the way to compute equivalence of 
> strings REQUIRES normalization to allow determining a~b iff 
> normalize(a)==normalize(b).
>
> But in general, the equivalence relations one might want, for 
> usability reasons, don’t have an acceptable normal form.
>
> Continuing to put in normalization as an essential step is doomed to 
> failure. Some protocols may need to be redefined to allow 
> non-normalized strings.
>

If you define equivalence relations (e.g. variants) based on code 
points, then you'll need to apply these / compare strings in NFD 
otherwise you will miss out on the effect of re-ordering of combining 
sequences.

Separate from that are equivalences between sequences so that some 
distinctions that are never normalized away can be treated as 
equivalent. For example some ordering differences in complex scripts.

A./

---

From: Nico Williams <[email-masked]>
Date: Tue, 24 Jul 2018 02:02:13 -0500

On Mon, Jul 23, 2018 at 08:05:41PM -0700, Larry Masinter wrote:
> normalize(a)==normalize(b).  But in general, the equivalence relations
> one might want, for usability reasons, don’t have an acceptable normal
> form.
> 

Well, somewhere the equivalence check has to be done, else you have
problems.  But you're absolutely right that always normalizing isn't
always the right answer.

For protocols like remote filesystem protocols, the best thing to do is
to implement form-insensitive / form-preserving semantics in the
filesystem (but not the server component).

For security protocols where relying parties need never re-encode to
verify signatures, then form-insensitivity can work.  In many cases it's
easy and convenient enough to make issuers/authorities normalize anyways
-- so might as well do that.

For something like DNS, normalization really is necessary, and has to be
done on the client-side.  (Though with on-the-fly signing one could also
just put ToUnicode() and ToASCII() functionality on the server, but I
don't think this is needed, nor desirable, nor desired.)

I speculate that in most cases in Internet protocols where Unicode
strings are carried, the best behavior is form-insensitive /
form-preserving, with a few exceptions where either an authority
normalizes, or the clients do.

In particular, it was a mistake to require normalization for NFSv4.

Quite often NFSv4 clients (Unix and Unix-like clients) have no idea
whether the application is even providing Unicode strings (because the
locale information is not visible to the NFS client).

NFSv4 servers are almost never very tightly integrated with the
filesystem -- typically there is a pluggable filesystem layer.  And with
other access methods to content with (local POSIX API access, other
protocols), the servers are just not in a position to authoritatively
perform normalization -- this alone argues for putting all normalization
burdens on the filesystem, not the server component, not the clients.
And the realities of actual use and implementation prevent us from
effectively dictating a normal form: HFS+ normalizes to ~ NFD; input
methods generally produce pre-composed sequences.

Only f-i/f-p can work for NFSv4.

I have been pushing form-insensitive / form-preserving behavior for well
over a decade now.  It's been lonely.  Are you joining me? :)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 24 Jul 2018 02:12:09 -0500

On Mon, Jul 23, 2018 at 08:38:15PM -0700, Asmus Freytag wrote:
> If you define equivalence relations (e.g. variants) based on code points,
> then you'll need to apply these / compare strings in NFD otherwise you will
> miss out on the effect of re-ordering of combining sequences.

Or NFC.  It doesn't matter which form you choose as the internal form
for comparison, provided that the comparison is form-insensitive.

Now, since normalization to NFC is defined in terms of an initial
normalization to NFD, it's obviously best to use NFD internally anyways.

Note, BTW, that form-insensitive comparison is not necessarily enough to
implement form-insensitivity.  If you have hash tables, say, and you're
hashing strings for hash table lookups, then you have to normalize
during hashing as well in order to have form-insensitive behavior.

The nice thing about this form-insensitive comparison/hashing is that it
can be done with fixed and small memory allocation, as you can step
strings character by character, and there is a maximum length in
codepoints to a character normalized to NFD, thus the amount of memory
needed is fixed, and that memory can be allocated on the stack (where
you have one).

Indeed, you can even optimize form-insensitive string comparison so that
for the mostly-ASCII and mostly-not-equivalent or mostly-equal input
cases... most characters do not require canonical decomposition.

Nico
--

---

From: "Pete Resnick" <[email-masked]>
Date: Wed, 25 Jul 2018 10:12:52 -0400

And Pete being on a short holiday has slowed down that process. I’ll 
submit them by the end of the week.

(And Peter is right: Matt did a fine job with the Etherpad notes. Makes 
my task pretty easy.)

pr

---

From: Larry Masinter <[email-masked]>
Date: Wed, 25 Jul 2018 18:54:27 -0700

Here's a wild idea, flame away

> problems.  But you're absolutely right that always normalizing isn't always the
> right answer.

It isn't clear how Unicode defines "normalization" compared to how
the term is used in mathematics. If you start with an equivalence relation
https://en.wikipedia.org/wiki/Equivalence_relation 
you can define a "canonicalization" if the set has a total ordering,
and pick the "least" or "most". But the form you might get by
doing so might not be desirable, or suitable to be called "normal",
even if it's useful for determining equivalence.

You don't need to normalize locally for using hash-table caching
if you allow the hash-table to reflect a cached redirect,
and put the onus on the origin to supply a canonical string.

You might think this would bloat the table with all of the
possible redirects, but as long as you look up on data entry
and the origin is consistent, there should only be one redirect.
In fact, you could limit it to one, or just periodically clear LRU.

I'd even include case folding, punycode and %xx-URL percent encodings
as possible input forms. The times they are typed in are
rare.

It would allow different TLDs to define their own additional
equivalences and case folding.

Some variation of this could be made to work for DNS, HTTP,
and email addresses with some enormous deployment problems.
Would need to prove that it works.

Sorry if this is terse

Larry
--
https://LarryMasinter.net

---

From: Nico Williams <[email-masked]>
Date: Thu, 26 Jul 2018 14:51:12 -0500

On Wed, Jul 25, 2018 at 06:54:27PM -0700, Larry Masinter wrote:
> > [...]
> It isn't clear how Unicode defines "normalization" compared to how the
> term is used in mathematics. If you start with an equivalence relation

The important thing is that Unicode defines canonical normalization
forms.  I don't believe there is an argument that those forms aren't
normal -- that is, that they allow multiple equivalent forms in one NF,
which clearly they do not.

> and pick the "least" or "most". But the form you might get by doing so
> might not be desirable, or suitable to be called "normal", even if
> it's useful for determining equivalence.
> 

Take as an example LATIN LETTER U WITH DIAERESIS AND MACRON U+0304.
This character can be expressed in this many ways:

  - U+1E7B
  - U+0075 U+0308 U+0304
  - U+0075 U+0304 U+0308
  - U+016B U+0308

That's four different ways, only two of which are in one NF or another.

Now imagine a string that has four such characters.  You'd not have 4^4
possible combinations -- 256(!!) of them only two (or maybe four) of
which would be in one NF or another (well, four, if any of those
characters have different K forms).

I'm really not familiar enough with input methods to know how many of
all those combinations are actually likely in the real world, but
because most input methods pre-compose, and because HFS+ decomposes,
you'll have at least two forms.  Any more than two forms and this can't
scale, and it saves you nothing.

Better to normalize as you hash (and only if necessary -- all-ASCII
strings, for example, require no normalization) then.

> possible redirects, but as long as you look up on data entry

Not just bloat the table.  Also bloat compute, and if you have to have
these sent to you, then bloat network bandwidth.

> In fact, you could limit it to one, or just periodically clear LRU.

The form-insensitive string comparison algorithm I described is actually
quite fast in most cases, as it can manage to avoid normalization in
most cases.  Normalizing as you hash is not as fast, but it's OK.

> as possible input forms. The times they are typed in are

If you're hash table allows duplicates, sure, then you can still be
case-sensitive.  Otherwise only if you want case-insensitivity.

> Would need to prove that it works.

IMO it would not work at all.

> Sorry if this is terse

I got it.

I don't push form-insensitivity only as an optimization.  IMO it has the
best, most user-friendly semantics and interoperability characteristics,
and it minimizes the number of locations where normalization
capabilities are needed.  That's at least three big reasons to prefer
f-i.

Nico
--