Subject: [precis] One change before Last Call

From: Pete Resnick <[email-masked]>
Date: Tue, 3 Feb 2015 20:43:44 -0600

Greetings,

Given the late-breaking excitement of the IAB Statement[1], I have been 
chatting with a few folks (John Klensin, Andrew Sullivan, Patrik 
Fältström) about text we can add to the document to explain the state of 
affairs before I send it out for a second Last Call. We've worked out 
the following (with a bit of editorial help from PSA):

    As part of the review of Unicode 7.0 for IDNA, a question was raised
    about a newly-added code point that led to a re-analysis of the
    Normalization Rules used by IDNA and inherited by this document
    (section 5.2.4). Some of the general issues are described in
    [IAB-Statement] and pursued in more detail in
    [draft-klensin-idna-5892upd-unicode70], but the result is that in
    the future, this specification is likely to be updated such that the
    range of characters in the LetterDigits category (Sections 4.2.1 and
    9.1) may be narrowed, some characters with special properties that
    are now allowed may be excluded, more Additional Mapping Rules
    (Section 5.2.2) may be added, and/or alternative normalization
    methods may be added. Even so, implementations that are sensitive to
    the advice given in this specification (to be careful to only allow
    characters whose implications they actually understand and,
    especially for the LetterDigit case, characters that actually are
    used to write relevant human languages) are unlikely to run into
    significant problems as a consequence of these issues or potential
    changes.

That is, we're aware of the issue, it's not just about Hamza, it's 
documented elsewhere, and the spec might change down the road because of 
it, but it's unlikely to affect an implementer who does the right thing 
anyway.

Does that text work for folks? If I don't hear much in the way of 
objection in the next 24 hours, I'll ask the authors to put that in and 
do the Last Call.

pr

[1] 
https://www.iab.org/documents/correspondence-reports-documents/2015-2/iab-statement-on-identifiers-and-unicode-7-0-0/

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 04 Feb 2015 04:12:21 +0100

* Pete Resnick wrote:
>    [...] Even so, implementations that are sensitive to

You did not say which document this is for, but if it actually gives the
advice above, then it probably should be abandoned. As an example, even
though I had seen any number of mispellings of my name, it was not until
a Google employee trying to recruit me kept addressing me as "Bjoem"
that I realised "rn" can be confused with "m" a couple of years ago. So,
the text above pretty much comes across as mockery, there is no way for
me to fully understand the implications of dozens of characters, and the
Unicode range spans across over a million of characters. I think it is
not appropriate for the specification to try and shift responsibility to
implementers in this extreme fashion. Might as well suggest that stereo-
typical monolingual citizens of the United States reject non-ASCII input
and even then still blame them for confusions like "rn" versus "m".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/

---

From: Pete Resnick <[email-masked]>
Date: Tue, 3 Feb 2015 21:24:43 -0600

On 2/3/15 9:12 PM, Bjoern Hoehrmann wrote:
> You did not say which document this is for...

-framework.

> ...but if it actually gives the advice above...
>    

Might I suggest that before commenting on what you think the text might 
mean, you actually read over the document and see if it says something 
that is in fact problematic. Since the documents cited in the text I 
provided are *not* about confusability, and the present document's 
section 12.5 discusses the issue and what can, and more importantly 
can't, be done about it, I'd suggest that the rest of what you write is 
not applicable to either the document or the text to be added.

pr

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 04 Feb 2015 05:04:35 +0100

>not applicable to either the document or the text to be added.

I think the text to be added is very clear that implementations should
disallow all characters whose implications are not actually understood.
If you cannot produce a complete list of all characters implementations
should disallow with the current version of the Unicode standard, then
my comment seems very applicable to me. Note that I have reviewed the
document before and did not come across the advice the proposed text
claims to be there, and I've checked section 12.5 right now and do not
see it there either.

Regardless, as an example, if, instead of saying

    Even so, implementations that are sensitive to the advice given in 
    this specification (to be careful to only allow characters whose 
    implications they actually understand and, especially for the 
    LetterDigit case, characters that actually are used to write 
    relevant human languages) are unlikely to run into significant 
    problems as a consequence of these issues or potential changes.

the specification were to say something like

    Implementations that heavily restrict which characters they allow,
    like limiting the set of permissable characters to a single script,
    are unlikely to run into significant problems as a consequence of 
    these issues or potential changes.

I would find that much less objectionable.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/

---

From: Pete Resnick <[email-masked]>
Date: Tue, 3 Feb 2015 22:40:39 -0600

> disallow all characters whose implications are not actually understood.

So, if you read the text below again, you'll see that it says that the 
*rest of the document* says that implementers need to be careful in 
certain ways. It is referring to advice in 4.1 and 12.3 regarding 
free-form class, to give two particular examples, as well as by 
reference the discussion in the IDNA document. Perhaps you are saying 
that the text I proposed does not properly capture what's in the 
document; that's a fine thing to discuss. Let's just not get off on the 
tangent of confusables.

> Note that I have reviewed the

I mentioned 12.5 in reference to your discussion of confusables, which 
is not part of what this text is meant to point to.

> I would find that much less objectionable.

Your suggestion I think is making too strong a claim, but I see where 
you're going. You needn't limit to a single script or otherwise heavily 
restrict to stay clear of potential problems; you simply have to stick 
to the more restrictive classes provided, or if you need to use 
free-form, then restrict to something more limited. So perhaps this 
would be clearer, and capture your concern:

    Even so, implementations that are sensitive to the advice given in
    this specification (to use the more restrictive String Classes, or
    otherwise to only allow a restricted set of characters, particularly
    ones whose implications they actually understand) are unlikely to
    run into significant problems as a consequence of these issues or
    potential changes.

pr

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 04 Feb 2015 06:18:34 +0100

>    run into significant problems as a consequence of these issues or

This is better, but if I and my implementation do run into significant
problems because I did not disallow some characters that I could have
disallowed, do you really think that I am to blame? I18N folks tell me
to have as few restrictions on characters as possible, while the text
above suggests to have as many restrictions as possible. If I fail to
find the proper balance myself, in the void that is left by the draft,
I think the specification is also responsible.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Wed, 04 Feb 2015 07:50:57 -0700

> I think the specification is also responsible.

I'm not sure who these i18n folks are who are telling you to have as few 
restrictions as possible. The very point of the PRECIS work, in part, 
has been to define a restricted, safe set of characters that is still 
expressive enough for most humans - i.e., the IdentifierClass. (The 
FreeformClass goes in the opposite direction, but then we provide some 
strong warnings about using that.)

Yes, the balance between safety and expressiveness is hard to strike 
properly, but that's why we've done all this work.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Feb 2015 12:02:07 -0500

--On Tuesday, February 03, 2015 22:40 -0600 Pete Resnick
<[email-masked]> wrote:

>...
> Your suggestion I think is making too strong a claim, but I
> see where you're going. You needn't limit to a single script
> or otherwise heavily restrict to stay clear of potential
> problems; you simply have to stick to the more restrictive
> classes provided, or if you need to use free-form, then
> restrict to something more limited. So perhaps this would be
> clearer, and capture your concern:
> 

Pete,

It seems to me that "particularly ones whose implications they
actually understand" in this newer phrasing essentially
encourages people to allow/use characters whose implications
they know they don't understand.  I don't think we should
encourage that.  Ever.  It may well happen with FreeFormClass,
but I still think we shouldn't encourage it.

Bjoern wrote,
 
	"I think the text to be added is very clear that
	implementations should disallow all characters whose
	implications are not actually understood. If you cannot
	produce a complete list of all characters
	implementations should disallow with the current version
	of the Unicode standard, then my comment seems very
	applicable to me."

We cannot produce such a list, especially in the presence of the
issue of decomposability being less predictable than we had
assumed, but it is equally possible to have characters in "a
single script" (even the script in which one's first language is
written) that one does not understand and characters in other
scripts that one does.

To illustrate the problem within the Latin script, while I might
be able to make guesses from his use of a domain in the DE.
tree, I would need to claim significantly more understanding
than I have to be sure whether "Bjoern" is most properly
"Bjoern", "Björn" or "Bjørn" and, if the latter, whether it is
expected to be coded as U+00F8 or as \u'006F'\u'0338' (or, for
that matter, something else).  Anyone who doesn't fully
understand that remark probably doesn't fully understand the
code points involved.  I expect that Bjoern (sic) does
understand them, but that many users of the Latin script might
not.

    john

---

From: Pete Resnick <[email-masked]>
Date: Wed, 4 Feb 2015 13:44:20 -0600

> but I still think we shouldn't encourage it.

Ah, so perhaps reversing it, and being more specific, would be better:

    Even so, implementations that are sensitive to the advice given in
    this specification (to use the more restrictive IdentifierClass
    whenever possible, or otherwise to only allow a restricted set of
    characters in the FreeformClass, particularly avoiding ones whose
    implications they don't actually understand) are unlikely to run
    into significant problems as a consequence of these issues or
    potential changes.

Is that clearer for everyone?

pr

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Feb 2015 14:46:39 -0500

> Is that clearer for everyone?

For me, at least, much clearer and better.
So this works for me.
    john

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 4 Feb 2015 15:32:46 -0500

> Ah, so perhaps reversing it, and being more specific, would be better:

[…]

I am happy with that.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Wed, 04 Feb 2015 13:44:30 -0700

>> Ah, so perhaps reversing it, and being more specific, would be better:

Ditto.

Peter

---

From: Pete Resnick <[email-masked]>
Date: Wed, 4 Feb 2015 14:47:52 -0600

>>> Ah, so perhaps reversing it, and being more specific, would be better:

I love the sound of growing consensus. :-)

Give Björn (and anyone else, for that matter) a bit of time to yelp, but 
then feel free to publish a new version with the agreed text and I'll 
send out the Last Call.

pr

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Wed, 04 Feb 2015 14:05:18 -0700

> then feel free to publish a new version with the agreed text and I'll

Excellent.

BTW, I am thinking that this new text would be best in the 
Interoperability Considerations section. Does that seem right?

Peter

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 05 Feb 2015 07:04:18 +0100

>Is that clearer for everyone?

Yes, thank you.

---

From: Pete Resnick <[email-masked]>
Date: Thu, 5 Feb 2015 00:44:13 -0600

On 2/4/15 3:05 PM, Peter Saint-Andre - &yet wrote:
> I am thinking that this new text would be best in the Interoperability 
> Considerations section. Does that seem right? 

I think that's reasonable. Perhaps split it, and make "13.1 General 
Considerations" and "13.2 ???". It's a bit separate, so I don't know 
that it should just be added as a paragraph, but I don't have a good 
subsection title. I'll leave it to the authors.

pr

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 05 Feb 2015 19:23:23 +0900

>> Is that clearer for everyone?

I think the above should work. Giving that it's the most busy time of 
the Japanese academic year, I haven't been able to follow the discussion 
in as much detail as I would have wanted, but I remember that quite some 
time ago, I was very much concerned by phrases along the line of "only 
allow those characters whose implications they fully understand".

Except maybe for two or three people on the Unicode Technical Committee 
I know, I wouldn't want to claim that anybody knows the implications of 
even a significant (in terms of size and use) part of the Unicode 
repertoire. And for the average implementer or system administrator, 
it's of course much less. But we definitely don't want that to lead to a 
situation where we go back to (some time) last century and ASCII only.

Given that the text is now very clearly predicated on FreeformClass, 
which is indeed wide open, it looks okay.

Regards,   Martin.

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 5 Feb 2015 08:46:04 -0500

If you're reading the text as implying that IdentifierClass is somehow guaranteed safe, then I _think_ we still have a problem, because I don't think it says that and anyway it isn't true.  

A

-- 
Andrew Sullivan 
Please excuse my clumbsy thums. 

> On Feb 5, 2015, at 5:23, "Martin J. Dürst" <[email-masked]> wrote:
> 
> I think the above should work. Giving that it's the most busy time of the Japanese academic year, I haven't been able to follow the discussion in as much detail as I would have wanted, but I remember that quite some time ago, I was very much concerned by phrases along the line of "only allow those characters whose implications they fully understand".
> 
> Except maybe for two or three people on the Unicode Technical Committee I know, I wouldn't want to claim that anybody knows the implications of even a significant (in terms of size and use) part of the Unicode repertoire. And for the average implementer or system administrator, it's of course much less. But we definitely don't want that to lead to a situation where we go back to (some time) last century and ASCII only.
> 
> Given that the text is now very clearly predicated on FreeformClass, which is indeed wide open, it looks okay.
> 
> Regards,   Martin.
> 
> _______________________________________________
> precis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/precis

---

From: John C Klensin <[email-masked]>
Date: Thu, 05 Feb 2015 10:58:45 -0500

--On Thursday, February 05, 2015 19:23 +0900 "\"Martin J.
Dürst\"" <[email-masked]> wrote:

>...
> I think the above should work. Giving that it's the most busy
> time of the Japanese academic year, I haven't been able to
> follow the discussion in as much detail as I would have
> wanted, but I remember that quite some time ago, I was very
> much concerned by phrases along the line of "only allow those
> characters whose implications they fully understand".
> 
> Except maybe for two or three people on the Unicode Technical
> Committee I know, I wouldn't want to claim that anybody knows
> the implications of even a significant (in terms of size and
> use) part of the Unicode repertoire. And for the average
> implementer or system administrator, it's of course much less.
> But we definitely don't want that to lead to a situation where
> we go back to (some time) last century and ASCII only.

Martin,

I don't see how you get from there to "ASCII only".  First,
there are a lot of people in the world who don't "understand the
implications of" Latin Script, even the basic undecorated Latin
characters and even though they might use them.  I think that,
while it may require some effort on their part, it is reasonable
to expect implementers and system administrators who establish
rules for identifiers to take responsibility for understanding
the use and possible risks associated with the characters of
their own scripts, especially the subset of those characters
that are relevant to their own languages.  

I recognize that makes it hard to design software systems that
are somehow internationally script-insensitive where identifiers
are concerned, but I think we have to live with that as the
price of the diversity of human languages and writing systems.
It may also imply a need for software implementations that are
far more rule-driven, possibly with locally-tailorable rules for
individual scripts, languages, and context, rather than an
approach that is construed as "this magic table of characters is
ok".   Again, that may be the price of the diversity of human
writing system and, by looking at tables and global profiles, we
may just be in denial about that diversity and its implications.

None of the above is made any easier by Unicode decisions,
however justified by the same diversity issues, pushing us from
design principles that apply to all of the coding system, to
design principles that are different on a per-script basis, to
specific and exception-driven rules such as "normalization does
all of the right comparison things within the Latin script
except for the following code points for which decomposition is
appropriate under some circumstances but not others" or "there
are case matching rules that are universally applicable except
for certain specific locales and code points, where special
treatment is needed".

It may be that we have been in denial, that the whole concept of
identifiers without language context is unworkable for at least
some protocols, and that we should be thinking of an
"internationalized identifier" as a tuple with a string and
language identifier.  Comparisons would then depend, not on
catenation and bit-by-bit comparison but on 
consideration of the language identifier based on RFC 4647 and
then interpretation and comparison of the string based on that
information.

That suggests that we should finish the PRECIS work based on
current documents rather than looking for a more prefect
solution (or textual phrasing) now.  However, it does also
suggest that, for at least some purposes, the PRECIS work may be
a waypoint rather than a final answer.

    john

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 05 Feb 2015 09:24:52 -0700

> subsection title. I'll leave it to the authors.

Here is what I have in my working copy. Please note that to improve 
readability I broke up some of the sentences in the long paragraph we 
all agreed to. Please review.

###

13.  Interoperability Considerations

13.1.  Encoding

    Although strings that are consumed in PRECIS-based application
    protocols are often encoded using UTF-8 [RFC3629], the exact encoding
    is a matter for the application protocol that uses PRECIS, not for
    the PRECIS framework.

13.2.  Character Sets

    It is known that some existing systems are unable to support the full
    Unicode character set, or even any characters outside the ASCII
    range.  If two (or more) applications need to interoperate when
    exchanging data (e.g., for the purpose of authenticating a username
    or password), they will naturally need to have in common at least one
    coded character set (as defined by [RFC6365]).  Establishing such a
    baseline is a matter for the application protocol that uses PRECIS,
    not for the PRECIS framework.

13.3.  Unicode Versions

    Changes to the properties of Unicode code points can occur as the
    Unicode Standard is modified from time to time.  For example, three
    code points underwent changes in their GeneralCategory between
    Unicode 5.2 (current at the time IDNA2008 was originally published)
    and Unicode 6.0, as described in [RFC6452].  Implementers might need
    to be aware that the treatment of these characters differs depending
    on which version of Unicode is available on the system that is using
    IDNA2008 or PRECIS.  Other such differences might arise between the
    version of Unicode current at the time of this writing (7.0) and
    future versions.

13.4.  Potential Changes to Handling of Certain Unicode Code Points

    As part of the review of Unicode 7.0 for IDNA, a question was raised
    about a newly-added code point that led to a re-analysis of the
    Normalization Rules used by IDNA and inherited by this document
    (Section 5.2.4).  Some of the general issues are described in
    [IAB-Statement] and pursued in more detail in
    [I-D.klensin-idna-5892upd-unicode70].

    At the time of writing, these issues have yet to be settled.
    However, implementers need to be aware that this specification is
    likely to be updated in the future to address these issues.  The
    potential changes include:

    o  The range of characters in the LetterDigits category
       (Section 4.2.1 and Section 9.1) might be narrowed.

    o  Some characters with special properties that are now allowed might
       be excluded.

    o  More "Additional Mapping Rules" (Section 5.2.2) might be defined.

    o  Alternative normalization methods might be added.

    Nevertheless, implementations and deployments that are sensitive to
    the advice given in this specification are unlikely to run into
    significant problems as a consequence of these issues or potential
    changes - specifically the advice to use the more restrictive
    IdentifierClass whenever possible, or if using the FreeformClass to
    allow only a restricted set of characters, particularly avoiding
    characters whose implications they do not actually understand.

###

---

From: Pete Resnick <[email-masked]>
Date: Thu, 5 Feb 2015 11:32:29 -0600

Looks good to me.

pr

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 05 Feb 2015 10:41:21 -0700

> Looks good to me.

OK, I shall submit -22.

Peter

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sat, 07 Feb 2015 21:10:11 +0900

> that are relevant to their own languages.

Hello John - You are right that not all people who use a script 
understand it. I could give some specific examples. But they usually 
think they understand it, so that will lead to the same outcome 
regarding what we write. Also, you are correct that not all implementors 
and system administrators are from the Latin writing parts of the world. 
But a good majority is, and has a huge influence on the rest of the 
world, and once one is in a defensive mindset, it's easy to come to the 
conclusion "let's use ASCII, that has worked for decades, that can't be 
wrong (and even if it is, nobody can be blamed/fired for choosing it)." 
So "ASCII only" was somewhat of a simplification, but not a big one.

> may just be in denial about that diversity and its implications.

I agree, at least in theory.

> for certain specific locales and code points, where special

I spent quite a bit of my time in colleague to learn Kanji. Quite a bit 
later, I spent some time to analyse Kanji shapes in an attempt to create 
some software for font design. I published a few papers, but didn't get 
much farther. But one thing I learned was that although Kanji were built 
up highly regularly, once you had to account for them in their full 
numbers, there was always some kind of edge case or exception that broke 
(or confirmed, as the saying goes) the rules.

What you talk about above shows that the same applies to Unicode 
overall: Even with a very strong attempt of keeping everything in line 
with simple rules, there will always be some corner case or exception.

> then interpretation and comparison of the string based on that

I think we have very good reasons to have rejected this approach 
virtually since the first moment we thought about internationalized 
identifiers. Human eyesight doesn't see invisible BCP 47 'color' painted 
over text, and people don't think that way.

> a waypoint rather than a final answer.

As I tried to explain, I just wanted to make sure we don't throw out the 
baby with the bathwater. The new text is fine with me.

Regards,   Martin.