Subject: Re: [precis] local case mapping

From: Andrew Sullivan <[email-masked]>
Date: Fri, 4 Oct 2013 23:17:46 -0400

Dear colleagues,

I reviewed draft-ietf-precis-mappings-03 today, at long last.  I
apologise for being so late.

I put together a number of incoherent questions about the case folding
stuff, but fortunately I re-read the mailing list archives on this
topic before posting a long message.  I agree with Peter: I find this
section of the document very confusing, and I think it may be wrong.
In particular …

On Thu, Sep 19, 2013 at 04:39:12PM +0900, Takahiro Nemoto wrote:

> Considering the maintenance and preservation of the document, 
> leaving it the way it is now is not a bad idea.

…I am pretty sure it shouldn't be left the way it is.

It seems to me that our principle generally needs to be that Unicode
is the thing we use, and if Unicode is broken it's Not Our Problem.
So we should figure out how to say, "Do the Unicode-y right thing
here," and then put that in.  I especially don't want to get into
specifying special language-specific tables ourselves: we don't have
the expertise, I think.

I can't think of any better suggested text than what Peter already
sent, so I think that's the right direction.  In the unlikely event
something clearer comes to me in the night, I promise to write it
down.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 08 Oct 2013 13:55:27 -0600

Hi Andrew, thanks for helping to move things forward.

> apologise for being so late.

No worries. Better that we get it right than that we finish it quickly.
Or maybe that's just a convenient excuse. :-)

> something clearer comes to me in the night, I promise to write it

I suggested text to clear up the first paragraph. However, my message
merely asked the key question, but left it unanswered: what are we
trying to accomplish here?

As I noted, Appendix B.1 simply matches the Language-Sensitive Mappings
from the SpecialCasing.txt file in the Unicode Character Database. If
that's *all* we're trying to accomplish, then we could simply say "apply
the Language-Sensitive Mappings in SpecialCasing.txt".

However, I get the sense that we're actually trying to accomplish more,
e.g., applying at least the context-sensitive mapping for Greek final
sigma -- in my example, a nickname of "ΦΙΛΟΣ ΜΟΙ" would be case folded
to "φιλος μοι" (with a Greek final sigma, which is correct in Greek) and
not to "φιλοσ μοι" (with a Greek medial sigma, which is incorrect in Greek).

It's also not clear to me if we have a position on full case folding vs.
simple case folding (e.g., ẞ = U+1E9E to "ss" instead of "ß" = U+00DF).
It seems to me that we might want to suggest a consistent approach here
so that we have improved interoperability.

So IMHO one approach would be:

1. Apply the language-sensitive mappings from SpecialCasing.txt
2. Apply the context-sensitive (i.e., "language-insensitive") mappings
from SpecialCasing.txt

I'm still not sure what to do about about full vs. simple case mapping,
but I see no strong reason to prefer simple case mapping because I don't
see a problem with our algorithm resulting in two characters (e.g.,
"ss") instead of one.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 8 Oct 2013 17:16:12 -0400

> trying to accomplish here?

Yeah, sorry that I wasn't clear.  I think the issue may actually be
that we can't decide which is needed, and there's a temptation to make
this local policy.  I think that's a bad idea: we should have one
method.  Decisions like this are what "the customers" wanted, I
think.  Therefore,

> see a problem with our algorithm resulting in two characters (e.g.,

I think all of these are right (so I think we should say that we don't
prefer simple case mapping also).

A

-- 
Andrew Sullivan
[email-masked]

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 08 Oct 2013 16:38:17 -0600

On 10/08/2013 03:16 PM, Andrew Sullivan wrote:
> think.
Agreed.
>   Therefore,
Well it's a bit mealy-mouthed for us to say we don't prefer simple case 
mapping. Better, I think, to say "apply full case mapping".

Peter

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 8 Oct 2013 18:44:53 -0400

On Tue, Oct 08, 2013 at 04:38:17PM -0600, Peter Saint-Andre wrote:
> case mapping. Better, I think, to say "apply full case mapping".

Sure, yes.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Takahiro Nemoto <[email-masked]>
Date: Fri, 18 Oct 2013 16:41:19 +0900

I would like to address some of the comments that I have received from Peter, Alexey, and Andrew in this email.

First, I would like to do some revision to the document according to Peter's suggestions.
Also, I am planning to split the "References" section into "Normative References" and "Informative References".
One other thing, as Andrew has pointed out, I am thinking of local case mapping as followings:

Purpose:
The purpose of local case mapping is to increase the probability of matching-result from the comparison between 
uppercase and lowercase characters,  targeting language-dependent characters.

Regarding the final sigma,

Currently, the handling order of PRECIS Framework is local case mapping to case mapping.
Because of this, when Σ=U+03A3 is handled by local case mapping that apply the context-sensitive (i.e., "language-insensitive") mappings
from SpecialCasing.txt, it becomes ς=03C2, in the end, when ς=03C2 is case mapped, it will still result in σ=03C3.

For this reason, final sigma is not one of the targets for local case mapping.

Also, in the next I-D, I would like to add "local case mapping can be selected only when case mapping is selected 
using the PRECIS Framework profile" and "casefolding in this document means full casefolding described in 
the Casefolding.txt file".

Regards,

Nemo

On 2013/10/09, at 7:44, Andrew Sullivan <[email-masked]> wrote:

> _______________________________________________
> precis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/precis

---

From: Takahiro Nemoto <[email-masked]>
Date: Fri, 18 Oct 2013 19:49:32 +0900

Yoneya-san and I submitted new version of precis mappings document. 
We reflected comments from this ML in this version.
Please read and give your comments/suggestions whether this version 
is an accurate reflection of the comments.
BTW, Final sigma is too Unicode-y, so there is not final sigma in this document.

Regards,

Nemo

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 18 Oct 2013 10:58:16 -0600

On 10/18/13 1:41 AM, Takahiro Nemoto wrote:
> I would like to address some of the comments that I have received
> from Peter, Alexey, and Andrew in this email.
> 
> First, I would like to do some revision to the document according to
> Peter's suggestions. Also, I am planning to split the "References"
> section into "Normative References" and "Informative References". 

That's all good.

> One
> other thing, as Andrew has pointed out, I am thinking of local case
> mapping as followings:
> 
> Purpose: The purpose of local case mapping is to increase the
> probability of matching-result from the comparison between uppercase
> and lowercase characters,  targeting language-dependent characters.

Works for me.

The text in the I-D is now:

   The purpose of local case mapping is to increase the probability of
   matching-result from the comparison between uppercase and lowercase
   characters, targeting locale and locale and context-dependent
   characters.

I stumbled over the last clause. I think it would be a bit clearer as
follows:

   targeting characters whose mapping depends on locale or on locale
   and context.

> Currently, the handling order of PRECIS Framework is local case
> mapping to case mapping. Because of this, when Σ=U+03A3 is handled by
> local case mapping that apply the context-sensitive (i.e.,
> "language-insensitive") mappings from SpecialCasing.txt, it becomes
> ς=03C2, in the end, when ς=03C2 is case mapped, it will still result
> in σ=03C3.
> 

OK, I understand that logic. However, it implies that if a protocol or
PRECIS profile uses local case mapping then it will benefit from the
nice language-sensitive mappings in SpecialCasing.txt but it can't
handle Greek final sigma in a context-sensitive way.

I *think* that's probably fine for comparison purposes. To use my
previous example, a nickname of "ΦΙΛΟΣ ΜΟΙ" would be case folded
to "φιλοσ μοι" (not "φιλος μοι"). That comparison would be applied
consistently, without attention to context (i.e., we wouldn't get
"φιλοσμοι" if there's no space but "φιλος μοι" if there is a space).

This seems to be an acceptable approach, I just wanted us to be clear
that this is what we're doing. :-)

> Also, in the next I-D, I would like to add "local case mapping can be
> selected only when case mapping is selected using the PRECIS
> Framework profile" and "casefolding in this document means full
> casefolding described in the Casefolding.txt file".

That's sensible.

Peter

--
Peter Saint-Andre
https://stpeter.im/