Subject: [precis] toLower() vs. toCaseFold()

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 4 May 2016 16:43:05 -0600

In a previous thread, we (mostly John) discussed the relative 
desirability of using toLower() vs. toCaseFold(); see for instance:

https://www.ietf.org/mail-archive/web/precis/current/msg01158.html
https://www.ietf.org/mail-archive/web/precis/current/msg01159.html

I suggested that we add some text about this to 7564bis. Here is a 
proposed paragraph for insertion in §5.2.3 ("Case-Mapping Rule"):

    The Unicode toCaseFold() operation defined by the Unicode Default
    Case Folding algorithm is most appropriate when an application needs
    to compare two strings.  When an application merely wishes to convert
    uppercase and titlecase code points to the lowercase equivalents
    while preserving lowercase code points, the Unicode toLower()
    operation is more appropriate and is less likely to violate the
    "Principle of Least Astonishment".  Therefore, application developers
    are advised to carefully consider whether they truly need to use the
    toCaseFold() operation in a given situation, or whether the toLower()
    operation would be more appropriate than the toCaseFold() operation.

Suggestions for improvement are welcome, especially from John. (E.g., we 
might want to more explicitly call out comparison vs. other contexts in 
the normative text elsewhere in §5.2.3).

Thanks,

Peter

---

From: Martin J. Dürst <[email-masked]>
Date: Fri, 6 May 2016 15:54:02 +0900

Hello Peter,

> the normative text elsewhere in §5.2.3).

I think 'compare' should be changed to 'search'. That's the prototypical 
use case for CaseFold.

Also, the language in the "Therefore" sentence is somewhat convoluted. 
It's unclear which alternative this text prefers. I suggest that if we 
want to put the two alternatives on an equal footing (i.e. make sure the 
application designer thinks carefully), then a more parallel sentence 
structure, avoiding words such as "carefully", "truly", and "would", 
would be more appropriate. What about:

                                        Therefore, application developers
    are advised to carefully consider whether toCaseFold() or
    toLower() is more appropriate.

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Fri, 06 May 2016 22:38:53 -0400

--On Friday, May 06, 2016 15:54 +0900 "Martin J. Dürst"
<[email-masked]> wrote:

>> I suggested that we add some text about this to 7564bis. Here
>> is a proposed paragraph for insertion in §5.2.3
>> ("Case-Mapping Rule"):
>> 
>>    The Unicode toCaseFold() operation defined by the Unicode
>>    Default Case Folding algorithm is most appropriate when an
>>    application needs to compare two strings.  When an
>>    application merely wishes to convert uppercase and
>>    titlecase code points to the lowercase equivalents while
>>    preserving lowercase code points, the Unicode toLower()
>>    operation is more appropriate and is less likely to
>>    violate the "Principle of Least Astonishment".  Therefore,
>>    application developers are advised to carefully consider
>>    whether they truly need to use the toCaseFold() operation
>>    in a given situation, or whether the toLower() operation
>>    would be more appropriate than the toCaseFold() operation.
>> 
>> Suggestions for improvement are welcome, especially from
>> John. (E.g., we might want to more explicitly call out
>> comparison vs. other contexts in the normative text elsewhere
>> in §5.2.3).
> 
> I think 'compare' should be changed to 'search'. That's the
> prototypical use case for CaseFold.

Hmm.  If we have to choose, I think I prefer "compare".  I just
looked at the subsections on "Default Case Folding" and "Default
Caseless Matching" in Section 3.13 of TUS 8.0 and it says a lot
about comparison and nothing about search.   Recommended
compromise:  Make the relevant sentence fragment read "most
appropriate when an application needs to compare two strings
such as in search operations."

I'd still prefer to denounce toCaseFold completely, especially
where identifiers are concerned.  It just has far too much
potential for being destructive and creating false results
(either positive or negative) when the language context is
unknown.  People/designers/implementers who are not prepared to
understand those issues and their implications should really not
be using the thing.

> Also, the language in the "Therefore" sentence is somewhat
> convoluted. It's unclear which alternative this text prefers.
> I suggest that if we want to put the two alternatives on an
> equal footing (i.e. make sure the application designer thinks
> carefully), then a more parallel sentence structure, avoiding
> words such as "carefully", "truly", and "would", would be more
> appropriate. What about:
> 
>                                         Therefore, application
> developers

For the reasons above, I'm not sure that an even footing is
appropriate.  I'd rather have the guidance be closer to "use
toLowerCase, which your users are likely to understand, unless
you need CaseFolding for some particular reason and understand
its implications"

best,
    john