Subject: [xmpp] 6122bis: normalization form

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 19 Mar 2012 13:56:20 -0600

We've previously discussed what to do about the normalization form. In
RFC 3920 and RFC 6122, we used normalization form KC (NFKC), which you
can think of as the most aggressive normalization form since it finds
matches when less aggressive forms don't; for details, see here:

http://www.unicode.org/reports/tr15/

http://www.saint-andre.com/ietf/i18n-intro.pdf

RFC 5198 recommends NFC as a general rule, and that's what IDNA2008
uses, too.

In the past we've talked about possibly using NFD (which does not
perform recomposition) instead of NFC. As I recall, the reasoning in
favor of NFD was (1) servers don't really need to perform
recomposition in order to compare JID parts, so we can save some
processor cycles by using NFD instead of NFC, and (2) client
interfaces can easily perform recomposition before showing JIDs to
humans, if needed. The reasoning in favor of NFC was (1) everyone else
is doing it, specifically we'll need to do it for domainparts when
resolving them anyway, and (2) the processing savings of NFD are
unproven. Other than oversimplifying, am I missing anything here?

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: [email-masked]
Date: Tue, 20 Mar 2012 08:52:07 +0000 (GMT)

I think we should use NFC. I suspect that NFD would cause some friction, and the benefits are not clear to me.

In general, I think we need compelling arguments to strike out on our own here.

Sent from my android device.

-----Original Message-----
From: Peter Saint-Andre <[email-masked]>
To: XMPP <[email-masked]>
Sent: Mon, 19 Mar 2012 7:56 PM
Subject: [xmpp] 6122bis: normalization form

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

_______________________________________________
xmpp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/xmpp

---

From: Joe Hildebrand <[email-masked]>
Date: Tue, 20 Mar 2012 13:33:28 -0600

Recomposition is only useful for crappy font renderers that can't deal with
the most popular combining characters, which don't really exist anymore.  It
can also decrease the size of a couple of JIDs very slightly on uncompressed
streams, at the expense of:

- added tables that need to be tracked for correctness
- extravagant complexity to compress the tables to manageable size
- annoyingly slowness in the implementations I've seen

The argument against NFD is "status quo", which doesn't seem adequate at
what amounts to our last chance to chance this stuff for the next decade or
two.

On 3/20/12 2:52 AM, "[email-masked]" <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAk9nj2QACgkQNL8k5A2w/vzUrACaA7enXXCEy5EZDZggGtMJak/K
> NGEAn2TgLDA+cjhVIZOnV5dKiRJhLcve
> =7OZD
> -----END PGP SIGNATURE-----

-- 
Joe Hildebrand

---

From: Matthew Wild <[email-masked]>
Date: Tue, 20 Mar 2012 19:34:52 +0000

> In general, I think we need compelling arguments to strike out on our own

Based on Peter's email (which incidentally is below this reply...) I
agree. If the main reason in favour of NFD is performance - as long as
NFC isn't worse than what we do now (and I believe it isn't) then I
think it's fine. Stringprep (if comparable) is not really a
performance bottleneck when done right.

Regards,
Matthew

---

From: Joe Hildebrand <[email-masked]>
Date: Tue, 20 Mar 2012 13:36:34 -0600

Note: that was intended to be marked as:

Hat: individual

-- 
Joe Hildebrand

---

From: Joe Hildebrand <[email-masked]>
Date: Tue, 20 Mar 2012 13:38:00 -0600

> performance bottleneck when done right.

I'll argue that it's easy to get wrong; it *is* a bottleneck in some of our
software.

-- 
Joe Hildebrand

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 20 Mar 2012 16:37:14 -0600

On 3/20/12 1:38 PM, Joe Hildebrand wrote:
>> Based on Peter's email (which incidentally is below this
>> reply...) I agree. If the main reason in favour of NFD is
>> performance - as long as NFC isn't worse than what we do now (and
>> I believe it isn't) then I think it's fine. Stringprep (if
>> comparable) is not really a performance bottleneck when done
>> right.
> 
> I'll argue that it's easy to get wrong; it *is* a bottleneck in
> some of our software.

It would be helpful to gather some numbers here, but that might not be
so easy (you'd need to run the same tests on the same software, but
swap out the normalization form).

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Dave Cridland <[email-masked]>
Date: Wed, 21 Mar 2012 06:16:26 +0000

On Tue Mar 20 19:38:00 2012, Joe Hildebrand wrote:
> reply...) I
> some of our

I agree it's a significant chunk of performance, but changing the  
normalized form of a jid will mean lots of complex handling in order  
to transition "right".

NFC, on the other hand, should be OK.

Addressing performance is, I think, best done by simply minimizing  
the amount of normalization we require - that is, ensure that jids  
are normalized when needed, instead of as a routine on any input.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 21 Mar 2012 00:30:10 -0600

> NFC, on the other hand, should be OK.

Can you say more about this?  We're still going to have to deal with
transition with JIDs that contains codepoints with kompatibility mappings.
Anything that is canonicalizing can deal with NFKC inputs just as easily as
NFD inputs, and will get the same outputs for each.

-- 
Joe Hildebrand

---

From: Dave Cridland <[email-masked]>
Date: Wed, 21 Mar 2012 06:32:22 +0000

> NFD inputs, and will get the same outputs for each.

Yeah, I'm assuming K is a much smaller problem.

I'm also assuming the problem isn't input jids, per se, but stored  
jids.

Dave.

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 21 Mar 2012 06:28:18 -0600

On 3/21/12 12:32 AM, Dave Cridland wrote:
>>> I agree it's a significant chunk of performance, but changing
>>> the normalized form of a jid will mean lots of complex handling
>>> in order to transition "right".
>>> 
>> Can you say more about this?  We're still going to have to deal
>> with transition with JIDs that contains codepoints with
>> kompatibility mappings. Anything that is canonicalizing can deal
>> with NFKC inputs just as easily as NFD inputs, and will get the
>> same outputs for each.
> 

I tend to agree. I'd be happy to gather some real-life data from the
jabber.org service to see what percentage of our ~1 million users have
usernames containing codepoints that have compatibility equivalents.

> I'm also assuming the problem isn't input jids, per se, but stored

Yes, some form of migration / JID scrubbing will be necessary.

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/