Subject: [precis] Enforcement as an Idempotent operation

From: William Fisher <[email-masked]>
Date: Wed, 12 Oct 2016 12:56:21 -0700

Should enforcing a string using PRECIS be idempotent?  If I apply the
enforce operation to a string twice, should I get the same result as
applying it just once?

The nickname profile is NOT idempotent for some inputs.

1. Certain characters are NFKC normalized to sequences with ASCII spaces.
This can lead to nicknames that begin with a space or contain adjacent
interior spaces that are removed if you apply the nickname profile again.

  U+00A8  =>  U+0020 U+0308  =>  U+0308

 2. Some characters can be further case mapped after NFKC normalization.

  U+1F11  => (K) => (k)

I also noticed that the RFC 7700 has case-mapping defined only when
comparing nicknames.  I thought this was confusing. I didn't understand why
username is split into two profiles (CasePreserved and CaseMapped), but
nickname is not.

If not all PRECIS profiles are idempotent, it would help to mention this in
the IANA Profile registry, e.g.

   Idempotent:  No.

As an implementer, I would prefer profiles that are idempotent.

Regards,
Bill

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 12 Oct 2016 22:03:29 -0600

On 10/12/16 1:56 PM, William Fisher wrote:
> Should enforcing a string using PRECIS be idempotent?

As far as I know, that was not a design criterion for PRECIS. Naturally, 
it might be a desirable property nonetheless.

> If I apply the
> 1. Certain characters are NFKC normalized to sequences with ASCII
> spaces. This can lead to nicknames that begin with a space or contain
> adjacent interior spaces that are removed if you apply the nickname
> profile again.
>

That's a good example.

One could argue that the leading/trailing space and adjacent interior 
space rules are application-specific and don't really belong in the 
nickname profile (indeed, I seem to recall a message to the WG about 
that years ago). I've been on the fence about that several times.

>  2. Some characters can be further case mapped after NFKC normalization.

It's not clear to me that U+1F11 has the problem you describe; perhaps 
could you sketch it out further?

> comparing nicknames.  I thought this was confusing. I didn't understand
> why username is split into two profiles (CasePreserved and CaseMapped),
> but nickname is not.

We try really hard not to multiply profiles beyond necessity. In this 
instance, we deemed acceptable not to apply the case mapping rule for 
enforcement (e.g., "StPeter" is a fine nickname) but would like to avoid 
nicknames in the same address space (e.g., a chatroom) that differ only 
by case because that would be confusing (e.g., "StPeter" and "stpeter"). 
As you suggest, we could have accomplished the same result by defining 
two separate profiles.

> If not all PRECIS profiles are idempotent, it would help to mention this
> in the IANA Profile registry, e.g.
>
>    Idempotent:  No.
>

Thanks for your input. Personally I will think about it further and post 
again after I do so.

Peter

---

From: William Fisher <[email-masked]>
Date: Thu, 13 Oct 2016 12:33:14 -0700

On Wed, Oct 12, 2016 at 9:03 PM, Peter Saint-Andre <[email-masked]> wrote:
> It's not clear to me that U+1F11 has the problem you describe; perhaps could you sketch it out further?

Oops, that should be U+0001F11A. The full example is:
"\U0001f11aevin" => "(K)evin" => "(k)evin"

I wrote a program to categorize characters that are not idempotent
under Nickname "ToLower" (ignoring white space). The numbers are the
same for Unicode 6.3, 8.0 and 9.0.

{
  '<font>': 467,
  '<square>': 90,
  '<compat>': 35,
  '<super>': 27,
  '<circle>': 4
}

The following two characters also appear to fail the idempotent test.
The initial decompositions do not begin with '<'.

\u03d3 GREEK UPSILON WITH ACUTE AND HOOK SYMBOL
\u03d4 GREEK UPSILON WITH DIAERESIS AND HOOK SYMBOL

> Thanks for your input. Personally I will think about it further and post again after I do so.

To me, the problem is to take untrusted input, validate it using
specified rules, and transform it into a stable, unambiguous format.
I'm still learning more about Unicode. Is there a reason that the case
mapping rule has to be applied *before* the normalization rule?  The
order appears to make a difference for NFKC.  I suppose the Nickname
"comparison" profile could re-apply the case mapping rule after the
normalization rule?

Thanks,
-Bill