Subject: Re: [precis] PRECIS profiles appear underspecified

From: Dan Chiba <[email-masked]>
Date: Tue, 02 Sep 2014 17:42:28 -0700

Thank you Peter for the explanation. Yes, I am looking for a way that 
allows clients to tell in advance precisely what rules the server will 
apply with regard to case mapping.

Precis operations for the same kind of string can take place in various 
places in different tiers and their behavior should be consistent.  The 
usernames "fussball" and "fußball" should be treated as equal for case 
insensitive matching, or as distinct for case sensitive matching. In a 
worst case, distinct but confusable accounts might be confused. Such 
problems would never happen if the components designated the complete 
spec of the username string matching. If the profile doesn't fully 
specify the spec but it's up to the deployment/service policies, I would 
like the components to be able to identify the policies precisely. The 
components that perform precis processing are distributed and they have 
to be coordinated with respect to the deployment/service policies. With 
a standard naming scheme for them it would be much easier to coordinate 
them. Implementers would have to make that up anyway, and it would be 
hard for application developers to tell the difference between different 
UsenameIdentifierClass implementations.

I am aware the ICU project has implemented SASLprep. It appears the SASL 
implementation has no ambiguity regarding case mappings; cases are 
preserved in the example:

    2  user             user       no transformation
    3  USER             USER       case preserved, will not match #2

I expect the same clarity in PRECIS. I do believe case insensitive 
matching should be enabled. I think a small number of popular recipes 
for usernames should be identified and standardized as a profile or a 
deployment/service policy, however it is called.

Regards,
-Dan

On 8/29/2014 3:15 PM, Peter Saint-Andre wrote:
> On 8/25/14, 7:52 PM, Dan Chiba wrote:
>> Hi Peter,
>>
>> This is essentially generic but I think the degree of impact would vary,
>> depending on the profile. UsenameIdentifierClass would be one of those
>> severely affected because it is important to evaluate usernames
>> correctly and there are various common practices of handing them.
>> Sometimes case insensitive, sometimes sensitive, among others.
>
> Correct. Which is why it's difficult to formulate one rule for all 
> treatments of usernames, and why it took quite a bit of discussion to 
> come to consensus on the text in Section 4.2.1 of the SASLprepbis 
> specification:
>
> http://tools.ietf.org/html/draft-ietf-precis-saslprepbis-07#section-4.2.1
>
> If I understand your original message correctly, you are looking for a 
> way that, say, client software can know in advance how a server will 
> treat usernames with regard to case mapping, based on the SASL 
> mechanism or application protocol in use. I was looking for that, too. 
> Unfortunately, our friends in the KITTEN WG (which works on SASL) were 
> insistent - and correct - that there is no deterministic formula here 
> because case mapping can even be a matter of deployment or service 
> policy and thus not determined by the SASL mechanism or application 
> protocol in use. Thus our carefully-crafted text in Section 4.2.1.
>
> I wish I could report happier news.
>
> Peter
>
>

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 3 Sep 2014 09:46:47 +0300

On Tue, Sep 02, 2014 at 05:42:28PM -0700, Dan Chiba wrote:
> insensitive matching, or as distinct for case sensitive matching. 

I don't see any case in those two strings.  What am I missing?

But anyway, it's not clear to me that the usernames "fussball" and
"fußball" should in fact always and everywhere be treated as equal.
For my understanding is that the use of ß is very much a localization
issue: the ligature is never used in Swiss written German.  One could
argue that it'd be extremely unwise to permit distinct user
registrations that are otherwise identical except for ss and ß
(certainly I would).  But if we think that's a good reason to insist
on a mapping, then that would also seem to be a good reason to say,
"No case folding, period."  Indeed, that's one of the big differences
between IDNA2003 and IDNA2008; IDNA2008 violates a deep tradition in
the DNS in this way, basically because we couldn't make the mapping
work the way we'd like.

> username string matching. If the profile doesn't fully specify the spec but
> it's up to the deployment/service policies, I would like the components to
> be able to identify the policies precisely. 

While I agree with that in principle, it sounds like an
interoperability and implementation nightmare (which issue is, I
think, what got us the current compromise text).

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Wed, 03 Sep 2014 08:35:24 -0400

--On Wednesday, September 03, 2014 09:46 +0300 Andrew Sullivan
<[email-masked]> wrote:

>> usernames "fussball" and "fußball" should be treated as
>> equal for case insensitive matching, or as distinct for case
>> sensitive matching. 
> 

That applying ToCaseFold(fußball) yields "fussball".  The
reasons why that made sense were clear, although questioned in
some quarters, when it was done.  Whatever else can be said
about it, it (and similar issues such as those with dotless "i")
are generally much less problematic in running text and/or when
the language and locale context are known than with, e.g., short
identifiers with no language context (such as with IDNA and at
least some PRECIS contexts).    At this stage, stability rules
prevent any reconsideration in the Unicode context: if the
ToCaseFold behavior is not satisfactory for a given context,
that operation either needs to be avoided (as IDNA2008 does) or
replaced by something else.

> But anyway, it's not clear to me that the usernames "fussball"
> and "fußball" should in fact always and everywhere be treated
> as equal. For my understanding is that the use of ß is very
> much a localization issue: the ligature is never used in Swiss
> written German.  One could argue that it'd be extremely unwise
> to permit distinct user registrations that are otherwise
> identical except for ss and ß (certainly I would).  But if we
> think that's a good reason to insist on a mapping, then that
> would also seem to be a good reason to say, "No case folding,
> period."  Indeed, that's one of the big differences between
> IDNA2003 and IDNA2008; IDNA2008 violates a deep tradition in
> the DNS in this way, basically because we couldn't make the
> mapping work the way we'd like.

I would question the "deep tradition" because, in reality, it
applies only to the ASCII repertoire, i.e., a subset of the set
of undecorated Latin characters.  I note that, for scripts that
do not have case as that term is understood in
Greek-Latin-Cyrillic but that do have
Initial-Medial-Final-Isolated distinctions, no one of whom I'm
aware has seriously suggested the CharPositionFormFold operation
and we have dealt with those cases inconsistently by banning one
of the forms, quietly mapping them into another form and losing
whatever information might be present (the option IDNA2008 quite
deliberately eliminated), or treating them as distinct.

>...

best,
   john

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 3 Sep 2014 16:08:21 +0300

On Wed, Sep 03, 2014 at 08:35:24AM -0400, John C Klensin wrote:
> That applying ToCaseFold(fußball) yields "fussball".

Yes, I know.  But this is sort of my point.  If the matching behaviour
of those two strings differ because of _case_ treatment, it's a recipe
for user puzzlement at best:

"Oh, that's because of case folding."

"Um, it's all lower case?"

> of undecorated Latin characters.

Yes, but surely for someone German it's handy to be able to use
CamelCase to make strings somewhat more memorable, and having it work
when the string has no decorartions and not work when there are
decorations is, at least, hard to understand.  The only responsible
advice in light of IDNA2008, in my opinion, is, "Use domain names are
in lower case, always."

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Dan Chiba <[email-masked]>
Date: Wed, 03 Sep 2014 10:07:52 -0700

Hi Andrew,

> I don't see any case in those two strings.  What am I missing?
My bad, both of them are lowercase. My intent was to present two 
distinct strings that are evaluated as equal in case insensitive 
matching. "fussball" and "Fussball" would be good examples.
> "fußball" should in fact always and everywhere be treated as equal.
In my understanding that is not always the case.
> the DNS in this way, basically because we couldn't make the mapping
I used the ß character as a case in which distinct strings are to be 
evaluated equal after precis preparation. I think it is a good idea to 
disallow confusable characters like this. I do not know the background 
of IDNA well, but I think there could be a precis profile for it.
> think, what got us the current compromise text).
I am concerned about interoperability problems that may happen due to 
the ambiguity in the profile spec. Right now, an implementation of 
UsernameIdentifierClass may use arbitrary character mappings. I agree 
the compromise was required, but I think it has introduced the problems 
while partially resolving the other requirement to match the various 
current practices of mapping some characters in username handling.

Best regards,
Dan

---

From: John C Klensin <[email-masked]>
Date: Wed, 03 Sep 2014 15:42:16 -0400

--On Wednesday, September 03, 2014 16:08 +0300 Andrew Sullivan
<[email-masked]> wrote:

> Yes, I know.  But this is sort of my point.  If the matching
> behaviour of those two strings differ because of _case_
> treatment, it's a recipe for user puzzlement at best:
> 

Indeed.  We are in complete agreement... I was just trying to
explain how we got here.    And part of that is that our
practice has been to apply not "toLowerCase" to strings (which
many people think they understand even though it can be a little
ambiguous in common practice) rather than the case folding
operation which few people not familiar with Unicode understand.

>> applies only to the ASCII repertoire, i.e., a subset of the
>> set of undecorated Latin characters.
> 
> CamelCase to make strings somewhat more memorable, and having
> it work when the string has no decorartions and not work when
> there are decorations is, at least, hard to understand. 

Sure.  But that is, IMO, for two reasons.  First, we "trained"
them to expect those variations to match by requiring it for
ASCII, thereby setting a very bad precedent for the i18n cases.
Second, and more important, it is unclear to me why the users of
slightly-decorated Latin text (German included) should expect
case matching to work while we tell (in no particular order) 

(i) Those same Germans that strings containing "ö" and those
containing "oe" don't match.

(ii) Our Greek, Hebrew, and Arabic script-using colleagues that,
if Unicode supplies different code points for final (or other)
forms than whatever is considered the base, then strings
containing the final form are not equal to otherwise-identical
ones that don't.

(iii) Our ASCII-using colleagues that, while they can write
MyFavoriteMovie.example and treat it as equivalent to
"myfavoritemovie.example", their preferred form will usually be
preserved by the DNS but sometimes won't.

(iv) Our English-speaking colleagues that "color" and "colour"
are different, even though most of their everyday experience
says otherwise.

(v) Lots of groups that joespizza.example and
joeʼspizza.example (that is not a single quote/apostrophe) are
not equivalent to each other, that JoesPizza.example is
equivalent to the first but JoeʼsPizza.example is invalid, and
that pizzeriajoe and pizzeragiovanni are not equivalent to any
of them or each other.  FWIW, I would not be at all surprised if
some clever search engine did consider all of the above to be
plausible alternatives to each other.

(vi, ff)  <endless numbers of additional examples)

> The
> only responsible advice in light of IDNA2008, in my opinion,
> is, "Use domain names are in lower case, always."

And that advice is part of what I meant in an earlier note when
I suggested that users avoid edge case and cases in which they
expected the computer system to figure out what they meant and
do it.  Users who don't want surprises and name registrants or
creators who don't either will stick to lower case and strings
as uncomplicated as possible, and will expect strings to match
only if what is in the database and what they type are
identical.    Since month ago, I would have written "modulo
normalization", but the recent discussions about U+08A1 and
related code points make even that suspect.

    john

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 09 Sep 2014 19:18:38 -0600

> of IDNA well, but I think there could be a precis profile for it.

Disallowing confusable characters is a laudable goal, but very difficult 
in practice. Section 9.5 of the precis-framework document covers these 
issues in some detail. Here's one of my favorite examples:

https://stpeter.im/journal/1420.html

> current practices of mapping some characters in username handling.

As far as I can see, the only alternative to the compromise we've 
brokered here is (my understanding of) what John suggests: lock down the 
base string classes, effectively disallow profiles, and tell existing 
applications to conform to the new rules. If they want exceptions, then 
they don't have to use PRECIS.

In my opinion, that would be slightly at odds with the spirit of how the 
PRECIS WG was formed, since all of the Stringprep "customers" in the 
room at the NewPrep BoF had existing profiles and wanted to figure out 
how to use modern versions of Unicode while supporting as many of their 
existing strings/identifiers as possible. If we now tell those customers 
that we can't meet their needs, I don't think they will be particularly 
happy. But, as John also says, this is about pleasing (or at least 
minimizing confusion for) end users.

Peter

---

From: John C Klensin <[email-masked]>
Date: Wed, 10 Sep 2014 14:09:12 -0400

--On Tuesday, September 09, 2014 19:18 -0600 Peter Saint-Andre
<[email-masked]> wrote:

>...
> As far as I can see, the only alternative to the compromise
> we've brokered here is (my understanding of) what John
> suggests: lock down the base string classes, effectively
> disallow profiles, and tell existing applications to conform
> to the new rules. If they want exceptions, then they don't
> have to use PRECIS.

I've suggested several slightly different things in the hope of
finding a solution that would be acceptable to everyone (or at
least "many").  I don't think the above is one of them, but it
is possible.  I am really no more interested than anyone else
sensible in invalidating applications that work now.  My goals
are to keep the user experience predictable and to minimize
maintenance responsibilities and costs.    What I have suggested
would be a slight variation on the above, replacing "tell
existing applications to conform" and "don't have to use" with
an explicit section of the PRECIS standard that would say, more
or less,

	"While it is recommended for predictability that
	applications conform narrowly to the base strings
	classes and rules, applications may need to make
	exceptions to allow or disallow particular characters
	for historical or other reasons.  Such applications are
	still conformant to PRECIS if they their protocol
	specifications precisely identify the characters
	affected."

I assume that we would than either add a per-application list of
exceptions or put in language passing on that responsibility to
the applications.  I also assume that any applications that
intended to expand their lists in the future (i.e., not simply
have exceptions as a legacy matter) would create IANA registries
for their exceptions lists (not complete tables) and that text
about exception for those applications would point to the
registries.

> In my opinion, that would be slightly at odds with the spirit
> of how the PRECIS WG was formed, since all of the Stringprep
> "customers" in the room at the NewPrep BoF had existing
> profiles and wanted to figure out how to use modern versions
> of Unicode while supporting as many of their existing
> strings/identifiers as possible. If we now tell those
> customers that we can't meet their needs, I don't think they
> will be particularly happy.

I agree with that goal and have all along.    But, again, I'd
like to move as much as possible toward a minimum number of
definitions/ tables/ profiles with exception lists as needed.
The latter are not ideal -- there is no way to get to ideal
while preserving backward compatibility -- but, if they are
treated as necessary exceptions and about edge cases (which they
mostly are), then I think we will be ok.

> But, as John also says, this is
> about pleasing (or at least minimizing confusion for) end
> users.

FWIW, I do see the goal of minimizing user confusion to be
diametrically opposite to the notion that we should have many
profiles and the potential for variations on each that I believe
Dan has suggested.  Either that or I still don't understand his
suggestion/ request.

     john

---

From: Dan Chiba <[email-masked]>
Date: Thu, 18 Sep 2014 13:05:32 -0700

On 9/9/2014 6:18 PM, Peter Saint-Andre wrote:
> Disallowing confusable characters is a laudable goal, but very 
> difficult in practice. Section 9.5 of the precis-framework document 
> covers these issues in some detail. Here's one of my favorite examples:
>

Unicode defines what characters are confusable (this is arguable imho) 
so I thought it theoretically possible to achieve this goal. Being 
confusable should not disqualify as a valid string. i.e. a confusable 
string should be "valid" and not "disallowed" in terms of Precis 
classification.

For example, both of "ᏚᎢᎵᏋᎢᏋᏒ" and "STPETER" should be valid and they 
are to be evaluated as confusable. Similarly, Turkish dotless i and the 
regular I/i should be valid and equivalent.
> brokered here is (my understanding of) what John suggests: lock down 
> the base string classes, effectively disallow profiles, and tell 
> existing applications to conform to the new rules. If they want 
> exceptions, then they don't have to use PRECIS.
>
> In my opinion, that would be slightly at odds with the spirit of how 
> the PRECIS WG was formed, since all of the Stringprep "customers" in 
> the room at the NewPrep BoF had existing profiles and wanted to figure 
> out how to use modern versions of Unicode while supporting as many of 
> their existing strings/identifiers as possible. If we now tell those 
> customers that we can't meet their needs, I don't think they will be 
> particularly happy. But, as John also says, this is about pleasing (or 
> at least minimizing confusion for) end users.
>
I think the standard username profile should not disalllow commonly used 
characters so the "customers'" existing profiles will be valid. Then 
their individual policies regarding disallowed characters may be handled 
as a sub-profile or variant. I think it would be desirable to enable 
identifying the complete specification (ideally registered in an IANA 
registry) using a compact identifier.

Regards,
-Dan

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 18 Sep 2014 17:10:01 -0400

On Thu, Sep 18, 2014 at 01:05:32PM -0700, Dan Chiba wrote:

> Unicode defines what characters are confusable (this is arguable imho) so I
> thought it theoretically possible to achieve this goal. 

Unicode's list is a good start, and it certainly includes the worst
cases, but it may not be applicable to every case.  My Arabic-using
friends tell me that it doesn't solve some of their worries, also.  I
do not know enough about practical Arabic use to evaluate those
claims.

A

-- 
Andrew Sullivan
[email-masked]