Subject: [Idna-update] FWD: Expiration impending: <draft-klensin-idna-rfc5891bis-01.txt>

From: John C Klensin <[email-masked]>
Date: Mon, 05 Mar 2018 09:18:13 -0500

Hi.

Given that there has been no discussion of this draft in the
last few months, that there seems to be no interest in it in the
IESG (although we have not pressed hard on it specifically), and
that the possibly-complementary
draft-freytag-troublesome-characters draft has already expired,
I'm just going to let this expire unless someone gives me a good
reason why not in the next few minutes (if a revision is not
posted today, the draft will expire because, without special
permission, nothing can be posted between tomorrow and the start
of IETF).

I hope no one interprets this as complete lack of interest in
pursuing IDN issues, or even clarification of the IDNA
standards, in the IETF.

   john

---------- Forwarded Message ----------
Date: Monday, March 5, 2018 04:42 -0800
From: IETF Secretariat <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Expiration impending:
<draft-klensin-idna-rfc5891bis-01.txt>

The following draft will expire soon:

Name:     draft-klensin-idna-rfc5891bis
Title:    Internationalized Domain Names in Applications (IDNA):
Registry Restrictions and Recommendations State:    AD is
watching
Expires:  2018-03-16 (in 1 week, 3 days)

---------- End Forwarded Message ----------

---

From: Kim Davies <[email-masked]>
Date: Mon, 5 Mar 2018 11:15:29 -0800

> pursuing IDN issues, or even clarification of the IDNA

With the expiry of these efforts, I am concerned more generally that
the standing IAB guidance in
https://www.iab.org/documents/correspondence-reports-documents/2015-2/iab-statement-on-identifiers-and-unicode-7-0-0/
has kept the pre-computed IDNA tables published at IANA downgraded
to Unicode 6.3.0 for a number of years. It seems appropriate to revisit
publishing tables against contemporary Unicode editions if there is no
active work on publishing clarifying RFCs in this regard.

kim

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 05 Mar 2018 20:38:26 +0100

On 5 Mar 2018, at 20:15, Kim Davies wrote:

> With the expiry of these efforts, I am concerned more generally that the standing IAB guidance in
> has kept the pre-computed IDNA tables published at IANA downgraded to Unicode 6.3.0 for a number of years. It seems appropriate to revisit publishing tables against contemporary Unicode editions if there is no active work on publishing clarifying RFCs in this regard.

Speaking as the by IAB appointed expert that do approve new tables, I simply can not unless there are some guidance and agreement in the IETF on how to move forward.

The few people involved that spend time on these issues makes it hard to draw conclusions on "the right path forward". This given the situation we are in, as described by John.

   Patrik

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 05 Mar 2018 15:02:36 -0500

Does the IAB know that, and have an opinion?

A

-- 
Please excuse my clumbsy thums

----------
On March 5, 2018 14:38:48 Patrik Fältström <[email-masked]> wrote:

>> has kept the pre-computed IDNA tables published at IANA downgraded to 
>> Unicode 6.3.0 for a number of years. It seems appropriate to revisit 
> Speaking as the by IAB appointed expert that do approve new tables, I 
> simply can not unless there are some guidance and agreement in the IETF on 
> how to move forward.
>
> The few people involved that spend time on these issues makes it hard to 
> draw conclusions on "the right path forward". This given the situation we 
> are in, as described by John.
>
>    Patrik
>
>
>
> ----------
> _______________________________________________
> IDNA-UPDATE mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idna-update
>

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 05 Mar 2018 21:09:08 +0100

I hope they know, at least I have the view I have reported we have a stall. I have not heard any opinion part from more architectural questions and ideas on "what should we do in general as this is a hard question". Which is not answering this specific question.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 05 Mar 2018 21:22:22 +0100

I did send the question, and gave choices, to IAB on July 9 2014.

Message-Id: <[email-masked]>

Since then the question has been stalled waiting for consensus on how to move forward.

   Patrik

---

From: "John R. Levine" <[email-masked]>
Date: 5 Mar 2018 15:40:24 -0500

> The few people involved that spend time on these issues makes it hard to draw conclusions on "the right path forward". This given the situation we are in, as described by John.

I am certainly interested in this stuff, but the only language I speak 
fluently is English so my knowledge of the topic is largely theoretical, a 
problem that I'd guess a lot of IETFers have.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Suzanne Woolf <[email-masked]>
Date: Mon, 5 Mar 2018 16:23:51 -0500

Patrik,

That was before my time on the IAB, sorry.

But as it happens, I’m looking at an update to the IAB statement, so have been pondering this very question. Do you think that https://www.ietf.org/id/draft-klensin-idna-5892upd-unicode70-05.txt <https://www.ietf.org/id/draft-klensin-idna-5892upd-unicode70-05.txt> covers the options reasonably well? 

Thanks,
Suzanne
("member of, not speaking for, the IAB")

> On Mar 5, 2018, at 3:22 PM, Patrik Fältström <[email-masked]> wrote:
>

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 5 Mar 2018 16:26:09 -0500

> problem that I'd guess a lot of IETFers have.

The problem is worse than that, in that the problem for speakers whose
first or only language is _not_ English is also largely theoretical.
Writing sytems are way harder than one might think from being familiar
with any of them.

One important input to the IAB might be an observation by IANA that it
thinks it needs more guidance given the state of affairs.

I wish I had any evidence that there were any good answers here.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 5 Mar 2018 14:12:24 -0800

The emerging conclusion after a lot of back-and-forth among people with 
deep and very deep expertise in this area was the that the issue that 
caused the stall wasn't really new; that there are, sprinkled through 
the writing systems, a number of cases of characters of identical 
appearance that are not covered by normalization (with and without 
involving combining marks); that some of these cannot be easily 
mitigated by simple repertoire restrictions; and that, ultimately, there 
is a need to do additional due diligence at the registry policy level 
(or LGR level).

In many cases, using the variant mechanism (blocked variants) would be 
the appropriate remedy - but this is a tool not available to the IANA 
review.

In any case, there is little benefit in keeping the IANA tables stuck at 
Unicode 6.3.0 -- the number of pre-existing cases (going as far back as 
the earliest Unicode version covered by IDNA2008) definitely exceeds the 
expected incremental addition from pending versions of Unicode. And a 
comprehensive solution lies outside the methodology of property-based 
inclusion; however, the property-based IANA tables would make a solid 
base on which to implement additional mitigation.

Therefore, the rationale to maintain this process in a stalled state is 
tenuous at best.

What is needed is for someone to summarize the state of the discussion 
and give a public signal that updating the IANA tables is not causing 
irreparable harm and can therefore move forward.

Secondary to that is finding a way to communicate the to consumers of 
these tables that simply allowing all PVALID code points isn't a robust 
solution for many writing systems and additional due diligence needs to 
be preformed - for example along the same lines as is being done now for 
the Root Zone (which is quickly defining the state of the art in that 
respect).

A./

---

From: John C Klensin <[email-masked]>
Date: Mon, 05 Mar 2018 18:25:59 -0500

--On Monday, March 5, 2018 16:23 -0500 Suzanne Woolf
<[email-masked]> wrote:

> But as it happens, I'm looking at an update to the IAB
> statement, so have been pondering this very question. Do you
> think that
> https://www.ietf.org/id/draft-klensin-idna-5892upd-unicode70-0
> 5.txt
> covers the options reasonably well? 

Suzanne,

Patrik may have a different answer, but, as the author of that
document, let me provide two opinions:

(1) No.  There is a newer version that covers more of the issues
and has a better discussion of options, but I have no intention
of posting it before there is a plan about how the document will
be processed.   I believe I told the IAB that while Andrew was
still Chair and told the late and lamented IAB I18N Program that
while Andrew was still chairing it.   I do have some ideas about
how it, and some other relevant documents might be processed,
but, until there is evidence that someone in "the leadership" is
interested, it has seemed to me that trying to write those
things down (or, e.g., flying to London to try to discuss the
issues and possibilities) would be a waste of time 

My amusement level is decreasing rapidly about the IAB's going
off and discussing issues like this internally without making
any attempt to involve members of the community who have actual
expertise and a history of doing most of the work.  In recent
months, I've been reminded by four well-known colleagues (two of
them former IAB Chairs) that the IAB got itself into a situation
in the early 1990s in which the community concluded that it had
gotten out of touch, thought its opinions were sufficient onto
themselves, and that wasn't interested in listening to experts
in the community who were not on the IAB.  Whether that
characterization was accurate or not and whatever one things of
the outcomes of the changes that followed, the results clearly
did not go well for the IAB or much of its then-membership.

(2) As much to Kim as to you... Although the IAB statement was
based on a limited understanding of only part of the problem,
the underlying problems, as described in the unposted
draft-klensin-idna-5892upd-unicode70-06.txt and, to a
considerable degree, in the expired but available
draft-klensin-idna-5892upd-unicode70-05.txt, are unchanged.
Given some of the behavior being exhibited by selected TLD
registries (I note from recent news that one of them is not only
selling names that are invalid under IDNA2008 (and that would
remain invalid if the tables were updated to Unicode 10.0
without changes being made to the Standard itself) but that
their record-keep systems are weak enough that they managed to
sell the same invalid strings multiple times and had to withdraw
the sales and that they are apparently also running a futures
market in such names using graphemes that are expected to be
assigned as code points in Unicode 11 and 12.  

There is another issue with Unicode 7 onward as far as domain
names are concerned.  Most, if not all, of the scripts that are
used for contemporary, widely-used, languages that are
well-represented on the Internet were coded long before now.  As
a result, most of the characters added in Unicode 7 and later
fall into one of three categories:

(i) Scripts that are less well-known in the Internet community
and which therefore carry even greater risk of "surprises"
(including but certainly not limited to, confusion with other
code points, issues with combining characters and what
normalization does or does not do, and special rendering issues)
if used in domain names.   Andrew's comment about this being
hard apply much more strongly to such scripts than to
better-known ones.

(ii) Letter- or digit-like characters added to existing scripts.
These potentially raise all of the issues listed above but, if
added to replace or supplement characters or character sequences
that are in use already, the may raise issues similar to the
relationship between Traditional and Simplified Chinese.

(iii) Characters (or, more precisely code points) that would not
qualify for use in domain name labels because of the properties
Unicode assigned to them.  That list includes emoji (because
they are symbols) but is definitely not limited to them.

Now, there are constituencies within ICANN would would like all
of those characters added and to start selling them.  But it
would be irresponsible (at best) on the IETF's part (or that of
ICANN) to encourage such things.

    john

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 5 Mar 2018 19:34:22 -0800

Last summer, I spent some time doing a detailed analysis of modern 
scripts (the ones that the lion's share of IDNs, other than some vanity 
ones, will be sold in), and limited to the current approved subset 
(Unicode 6.3.0). This survey surfaced a considerable fraction of 
potentially "troublesome" characters; fundamentally not different from 
the one that caused the halt in updating the IDNA208 tables. This survey 
benefited from input beyond my own personal experience, including input 
and data both from Unicode experts engaged in reviewing Unicode's tables 
of code intentionally identical code points as well as expertise and 
data generated as part of ICANN's Root Zone LGR project.

A draft version of the survey was made public as ID 
https://datatracker.ietf.org/doc/draft-freytag-troublesome-characters/
The results were discussed ad-hoc with IETF and IAB experts at various 
occasions and by e-mail. These discussions lead to some conclusions.

The following points are worth noting:

(1) A small, but significant number of both *existing* code points and 
combining sequences exhibit the same issues as the code point that lead 
to the IAB recommendation to halt the update of IDNA2008 tables. This 
problem can be characterized as code points having identical (or near 
identical) appearance with the code point differences not folded by 
normalization. (The problem case found in 2015 only had "near identical" 
appearance.)

(2) Objectively, halting the process did and does nothing about existing 
"troublesome characters". Common to all is the issue that disallowing 
repertoire elements is generally a poor way of mitigating the issues - 
mainly because doing so would arbitrarily favor letters over digits, or 
one language or writing system over another. The exception is that 
disallowing some combining marks intended for more technical purposes 
would prioritize regular text, and that would address a subset of these 
cases. All others require different mitigation approaches well within 
the scope of registration policies.

(3) For existing scripts, the pending additions are not expected to 
significantly add to the existing problem. Compared to the number of 
existing cases, the pending additions are few, and would be addressed 
with the same mitigation approaches. (Pending additions are not 
predominantly identical to existing code points; the vast majority are 
added because they are, in fact, different, isolated exceptions 
notwithstanding).

(4) For new scripts, John's observation that they are mostly not 
modern-use means that problems are inherently limited by several 
factors. Most of these scripts do not represent useful markets, as few 
people can read them. The exception are decorative scripts like 
hieroglyphics, which in turn have distinct appearance. For 
non-decorative ones, some may exhibit code points that are accidentally 
close in appearance to code points in other scripts. Whether there would 
be font resources to present code points in these obsolete scripts is 
doubtful; other attack vectors would promise easier success. (User 
agents flagging unusual obsolete scripts would not create many false 
positives.)

(5) The issue of non-normalized identical appearance pales in 
significance compared to other issues with existing code points for 
modern, widely used scripts. For Chinese and Arabic, not implementing 
variants is arguably not state of the art; doing so allows unregulated 
registration of confusing labels on a large scale. For all South East 
Asian / Indic scripts, not implementing context rules for a good portion 
(about a third) of the characters for each script is not state of the 
art; doing so would allow labels that can't be interpreted by either 
layout engines or readers, as in these scripts, characters do not exist 
in isolation. For zones implementing multiple scripts, not blocking 
cross-script homoglyphs is arguably also not state of the art; doing so 
leads to whole-label confusables that are impossible to detect by 
inspection.

(6) By freezing the update of IDNA2008 tables, IAB effectively declares 
that IDNA2008 is "stuck in the past". This incrementally increases the 
pressure on / temptation for various operators to unilaterally move 
beyond IDNA2008. If such "wild catting" can cloak itself in the moral 
mantle of support for some minority languages, it provides cover for 
those cynically selling emoji labels.

(7) The only way forward is to re-synchronize the updates with the 
current version of Unicode (11.0 is being readied and there will be a 
new version every year). This must be coupled with simultaneously 
addressing the gaps between "PVALID" and the state of the art.

(8) The ability to define blocked variants (code points that can be 
substituted for each other by an unsuspecting user) and to robustly 
exclude later registrations differing only by such a variant from an 
earlier registration can take care of nearly all of the troublesome 
characters, as well as of the problem of cross-script homoglyphs. This 
is currently being implemented in the Root Zone for a growing list of 
modern scripts (eventually 28 of them). Other zones can build on that 
work (for example by integrating digits and hyphen, and optionally by 
extending that analysis to scripts that "look alike" some modern ones).

(9) The security implications of not defining context rules for the 
South East Asian/Indic scripts need to be publicized in a way that 
non-specialists can understand the need and know where to look for 
examples (such as the Root Zone LGRs) that successfully implement these. 
(While IDNA2008 provides context rules for some special code points the 
protocol level may not be the most appropriate place for general context 
rules; some rules could be made stricter/looser for zones that are 
limited to specific scripts/languages).

(10) The security implications of emoji are not understood and need to 
be publicized in ways that is accessible to those that might be tempted 
to flout the rules to get a "cute" label. There is enormous public 
interest in emoji and it is spread across users of nearly all modern 
scripts. Given the enormous pressure driving demand for these it is 
frankly surprising how limited the wild catting has been up to now. The 
concern would be that simply pointing to an existing standard 
(especially one that artificially limits itself the the past - that is 
Unicode 6.3.0) may not prove sufficient counter; there is a chance that 
a clear explanation why emoji labels cannot be made secure, might give 
at least some users pause.

(11) Finally, none of the issues discussed here exist in isolation. The 
next layer of the onion is "similarity" or plain "confusable" labels. In 
an environment where .com and .corn or apple.com and app1e.com may 
legitimately coexist (as far as the protocol is concerned) and where 
that situation is much worse in many scripts, it is simply questionable 
that an action as drastic as anchoring the IDNA2008 protocol at an 
arbitrary and (as we have seen) not problem-free point is justified or 
in proportion.

This reality calls for a layered approach where each layer takes care of 
its slice of the problem:

(a) The protocol takes care of all cases that can be remedied by 
property-based inclusion applicable to all languages, and the occasional 
context rule of global applicability. RFC 5891, 5892 etc.

(b) The next layer out consists of some tailoring of the repertoire for 
each zone, but importantly, also implementing the state of the art when 
it comes to blocking identical variants. This would rely on the 
technology around LGRs and RFCs 7940 and 8228

(c) The outer layer would finally address the issue of the halo of 
"confusables". There's room for new technologies allowing automated 
string similarity scoring and review.

A./

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 5 Mar 2018 22:56:38 -0500

Hi,

Having failed to hold up my end of the log on this work, I guess I
should say something here.  Also, I was on the IAB for some of this
and lead on the i18n program(s) during some of it, so I should take
some blame.

On Mon, Mar 05, 2018 at 07:34:22PM -0800, Asmus Freytag wrote:

> combining sequences exhibit the same issues as the code point that lead to
> the IAB recommendation to halt the update of IDNA2008 tables.

Yes.  Note the IAB couldn't actually halt anything (Asmus didn't
suggest we at the time did, but I want to make that clear).  What the
IAB did was ask IANA not to do more updates until there was some
consensus.

> "troublesome characters".

Sure, but I don't think anyone thought it did.  I think the idea was
that the IAB observed a problem that it believed had been solved by
IDNA2008, but which had not been.  That problem remains unsolved.

> mainly because doing so would arbitrarily favor letters over digits, or one
> language or writing system over another.

I am not sure I am prepared to concede that such limitations is
"generally poor", since in fact general arbitrary limitations are what
we have in the "host name" or "LDH" rules, historically.  STD13 is at
some pains to point that out, so I don't think it is controversial.

The problem is when that arbitrariness rubs up against both commercial
and political realities.

> require different mitigation approaches well within the scope of
> registration policies.

But registration policies are not uniform and can't be assured.  This
is how the "DNS is distributed" and "everyone should be safe"
principles come into conflict.

> (6) By freezing the update of IDNA2008 tables, IAB effectively declares that
> IDNA2008 is "stuck in the past". This incrementally increases the pressure
> on / temptation for various operators to unilaterally move beyond IDNA2008.
> If such "wild catting" can cloak itself in the moral mantle of support for
> some minority languages, it provides cover for those cynically selling emoji
> labels.

Worse, of course, IDNA2003 provides no principled reason _not_ to add
emojis, and I think this argument from Asmus is an important (and
too-overlooked) point.  Even if IDNA2008 draws the line imperfectly,
the identifier/non-identifier code point line is _way clearer_ in it
than in IDNA2003.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 06 Mar 2018 07:04:28 +0100

On 5 Mar 2018, at 22:26, Andrew Sullivan wrote:

> The problem is worse than that, in that the problem for speakers whose first or only language is _not_ English is also largely theoretical.
> Writing sytems are way harder than one might think from being familiar with any of them.
>
> One important input to the IAB might be an observation by IANA that it thinks it needs more guidance given the state of affairs.

Exactly, and sometimes just understanding that there IS a problem might be very very good things to understand. So for me, just because one know one language only, and it is english, one might still suit in a group/discussion like the one described. Knowing that one only know one language is in certain cases a strength and not what everyone understands... :-)

To some degree the problem collapses to:

- List the number of difficult incompatible paths forward, each one in isolation being hard

- Choose one

- Implement, move in that direction, execute

> I wish I had any evidence that there were any good answers here.

Agree. Completely agree.

   paf

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 5 Mar 2018 23:45:41 -0800

On 3/5/2018 1:23 PM, Suzanne Woolf wrote:
> But as it happens, I’m looking at an update to the IAB statement, so 
> have been pondering this very question. Do you think that 
> https://www.ietf.org/id/draft-klensin-idna-5892upd-unicode70-05.txt covers 
> the options reasonably well?

There is one statement here (and it's the premise of the whole affair) 
that is actually not 100% correct:

    Unicode 7.0.0 introduces the new code point U+08A1, ARABIC LETTER BEH
    WITH HAMZA ABOVE.  As can be deduced from the name, it is/*visually identical*/  to the glyph that can be formed from a combining sequence
    consisting of the code point for ARABIC LETTER BEH (U+0628) and the
    code point for Combining Hamza Above (U+0654).

(/*emphasis */added)

In fact, in many (all?) fonts that do support this code point, there's a 
noticeable difference between the combining sequence and the precomposed 
character. Here is one example:

similar systematic differences, although slight, obtain in many cases of 
combining sequences that /appear at first to be identical/ to 
precomposed Latin code points without being normalized:

where the right side is 'o' + 0x0337   [this case has been in Unicode 
since IDNA2008 first started]

Obviously much depends on the font, but because the entire premise rests 
on the *absolute identical appearance* of the sequences to the code 
points in question, I think we need to take a step back here.

Perhaps, one difference since the time this was first raised is that the 
code point 08A1 is now supported in a number of fonts, which may not 
have been the case immediately after Unicode 7.0 was released and it may 
not have been apparent that the precomposed code point would be treated 
differently from a sequence that was so apparently made up from the same 
elements.

To put this into perspective: there are thousands of PVALID Chinese 
characters for which the distinctions are equally subtle, just saying. 
(And no, for the most part, they are not variants of each other). Yet 
this didn't stop IDNA from tracking many thousands of Han additions over 
the years.

I think the best thing for the IAB is to find a way to climb down from 
seeing this as an earth-shattering crisis and accepting such near 
similarities as unavoidable consequences of trying to implement mnemonic 
identifiers that are readable to the world population.

The right way forward is simply not to leap to the assumption that 
action on the protocol level is called for for these cases, but that 
they are in some ways the end points of a continuum that includes cases 
like .com/.corn and apple.com/app1e.com.

The right way forward also means that one cannot blithely assume that 
identifiers are always made from a small set of very distinct shapes 
(like most of the Latin alphabet) and to perhaps find ways to mitigate 
the worst cases.

A./

>> On Mar 5, 2018, at 3:22 PM, Patrik Fältström <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Message-Id: <[email-masked] 
>> <mailto:[email-masked]>>
>>
>>> On March 5, 2018 14:38:48 Patrik Fältström <[email-masked] 
>>> <mailto:[email-masked]>> wrote:
>>>
>>>>> to Unicode 6.3.0 for a number of years. It seems appropriate to 
>>>>> revisit publishing tables against contemporary Unicode editions if 
>>>>> there is no active work on publishing clarifying RFCs in this regard.
>>>>
>>>> Speaking as the by IAB appointed expert that do approve new tables, 
>>>> I simply can not unless there are some guidance and agreement in 
>>>> the IETF on how to move forward.
>>>>
>>>> The few people involved that spend time on these issues makes it 
>>>> hard to draw conclusions on "the right path forward". This given 
>>>> the situation we are in, as described by John.
>>>>
>>>>   Patrik
>>>>
>>>>
>>>>
>>>> ----------
>>>> [email-masked] <mailto:[email-masked]>
>> [email-masked] <mailto:[email-masked]>

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 06 Mar 2018 11:30:58 +0100

Suzanne, others,

In general I agree with the comments from John.

Here is the mail I did send (scroll down to see my comments today):

Forwarded message:

> From: Patrik Fältström <[email-masked]>
> To: Internet Architecture Board <[email-masked]>, [email-masked], John Klensin <[email-masked]>, Pete Resnick <[email-masked]>, Andrew Sullivan <[email-masked]>
> Subject: Fwd: PRELIMINARY  APPEAL -- Re: [i18n] Exper Review - New Release of the Unicode Standard, Version 7.0.0
> Date: Wed, 9 Jul 2014 06:03:57 +0200
>
> As the appointed expert for this IANA registry I reacted by pulling the hand brake when John sent his email.
>
> I have also clarified with IANA what that implied, i.e. nothing more than that I did "revoke" my earlier decision and that they should wait for me to give them more clear instructions. See below.
>
> I have myself had a discussion with John on the matter and I do understand his point. I have recommended him to propose something concrete in the form of an I-D that explain the issue and the concrete actions (which should be to add a contextual rule, exceptions in the form of Disallowing certain code points, or both) that can be discussed. Consensus on such an I-D say what to do.
>
> Unfortunately we are in the quiet period of I-D due to the upcoming IETF meeting, so the I-D might take a few weeks (I am unclear on todays rules).
>
> If IAB do want some other action, do not hesitate contacting me to discuss the issue.
>
> To summarize:
>
> - Unicode has released a new version, 7.0.0, of the Unicode Code Points.
>
> - When applying the algorithm in the current RFCs of IDNA to the new table, myself and IANA did reach the same result.
>
> - When inspecting the set of valid and non valid code points, no problematic backward compatibility is found.
>
> - Because of this, the derived property values could be released as calculated.
>
> - BUT, what is found is that there is (at least) one new code point (U+08A1) added that also can be represented by two other code points (U+0628 and U+0621), but no normalization rule is added for them. Equivalence rules exists for other such triples, for example U+0623, U+0627 and U+0254 (below from Unicode 7.0.0):
>
> 0621;ARABIC LETTER HAMZA;Lo;0;AL;;;;;N;ARABIC LETTER HAMZAH;;;;
> 0623;ARABIC LETTER ALEF WITH HAMZA ABOVE;Lo;0;AL;0627 0654;;;;N;ARABIC LETTER HAMZAH ON ALEF;;;;
> 0627;ARABIC LETTER ALEF;Lo;0;AL;;;;;N;;;;;
> 0654;ARABIC HAMZA ABOVE;Mn;230;NSM;;;;;N;;;;;
> 08A1;ARABIC LETTER BEH WITH HAMZA ABOVE;Lo;0;AL;;;;;N;;;;;
>
> - Unicode consortium refer to explanation of Hamza in the "Combining Hamza Above" subsection, pp. 263-264 in <http://www.unicode.org/versions/Unicode6.2.0/ch08.pdf> (version 7.0.0 of this is not released yet) and in particular, Table 8-11. They say that U+08A1 will be added to that table explicitly in the Unicode 7.0 version of the core specification.
>
> - Question for IETF is whether this addition of U+08A1 should result in:
>
> 1. No action
>
> 2. Addition of U+08A1 to the list of exceptions with derived property value DISALLOWED
>
> 3. A contextual rule implementing the so far non-existing table that explains use of Hamza, which is in table 8-11 in Unicode 6.2.0.
>
> I am waiting for a consensus call on this, and do only see this being possible by having John Klensin proposing an I-D that is explaining the situation and suggesting one outcome (possibly 2). I have myself requested to be co-author of this I-D for conformity to the set of RFCs that do describe the algorithm for the derived property value.
>
> - In the mean time I have asked IANA to wait for me to give them further instructions. If IAB do believe some IANA action should happen already now, please let me know.
>
>    Patrik
>
> Begin forwarded message:
>
>> From: Patrik Fältström <[email-masked]>
>> Subject: Re: PRELIMINARY APPEAL -- Re: [i18n] Exper Review - New Release of the Unicode Standard, Version 7.0.0
>> Date: 8 juli 2014 22:54:33 CEST
>> To: Pearl Liang <[email-masked]>
>> Cc: Michelle Cotton <[email-masked]>
>>
>> You only have to send it to me. What I do is to inform IAB and i18n list. In this case they did pull the hand brake. Not your problem. They pulled, and I informed you to stop the process (revoked my earlier decision). You should just wait for me (again).
>>
>> We might get an I-D with proposal on what to do, but we are due to the IETF in a silent period for new drafts. :-(
>>
>>   Patrik
>>
>> On 8 jul 2014, at 22:35, Pearl Liang <[email-masked]> wrote:
>>
>>> Further, I wonder if this "additional" review will become part of the
>>> "expert review" from now on before we can post a new unicode version.  As
>>> now, I only need to send the request to Patrik for approval.
>>>
>>> Thanks,
>>> ~pl
>>>
>>>
>>> On 7/8/14 12:24 PM, "Michelle Cotton" <[email-masked]> wrote:
>>>
>>>> Patrik,
>>>> What do we need to do here?
>>>> We will wait to make any changes until this is sorted out.  Is that
>>>> correct? Let us know.
>>>> Thanks, Michelle
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On Jul 8, 2014, at 9:22 AM, "Pearl Liang" <[email-masked]> wrote:
>>>>>
>>>>> Michelle,
>>>>>
>>>>> Cc: Patrik
>>>>>
>>>>> FYI, John escalated this.
>>>>>
>>>>> So, our main table is still with version 6.3.0.
>>>>>
>>>>> Thanks,
>>>>> ~pl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On 7/7/14 3:22 PM, "John C Klensin" <[email-masked]> wrote:
>>>>>>
>>>>>> Attention AppsADs and IANA: Please treat this as a preliminary
>>>>>> appeal of Patrik's decision to allow the IANA tables to be
>>>>>> updated.  I will formalize that with a note to Jari if necessary
>>>>>> after we have the discussion suggested below.
>>>>>>
>>>>>> --On Monday, July 07, 2014 20:54 +0200 Patrik Fältström
>>>>>> <[email-masked]> wrote:
>>>>>>
>>>>>>> From Ken:
>>>>>>>> Patrik,
>>>>>>>>
>>>>>>>> O.k. The hamza above is an explicit edge case, where a
>>>>>>>> determination needs to be made in each instance.
>>>>>>>>
>>>>>>>> See the "Combining Hamza Above" subsection, pp. 263-264 in:
>>>>>>>>
>>>>>>>> http://www.unicode.org/versions/Unicode6.2.0/ch08.pdf
>>>>>>>>
>>>>>>>> and in particular, Table 8-11. U+08A1 will be added to that
>>>>>>>> table explicitly in the Unicode 7.0 version of the core
>>>>>>>> specification.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> --Ken
>>>>>>>
>>>>>>> I read Ken's response as if we already have such situations
>>>>>>> for Hamza.
>>>>>>>
>>>>>>> If an exception is to be added, that requires RFC action.
>>>>>>
>>>>>> Ok.  In the absence of other information, I think Ken is
>>>>>> referring to something called the Normalization Stability Rule.
>>>>>> To review earlier notes that none of this contradicts, part of
>>>>>> that rule, as I understand it, requires that, if a new
>>>>>> precomposed abstract character is added that previously could
>>>>>> have been assembled with a base character and a combining one,
>>>>>> NFC of the new character is required to decompose back the prior
>>>>>> sequence.  The other part of that rule is the one I think Mark
>>>>>> refers to: if the combining sequence normalized to itself in
>>>>>> version N, it cannot normalize to anything else (and, in
>>>>>> particular, the new precomposed character) in version N+1.
>>>>>>
>>>>>> Now, let's call that second part "normalization stability",
>>>>>> i.e., once a code point or sequence exists and has a
>>>>>> normalization the latter doesn't change.  That is elementary and
>>>>>> not at issue here.  And, of course, unassigned code points don't
>>>>>> have normalizations (or much of any other properties) -- that
>>>>>> was precisely the reason why IDNA2008 couldn't allow them to be
>>>>>> processed or looked up.
>>>>>>
>>>>>> The first part, however, is about "keeping normalization working
>>>>>> consistently".  The main purpose of normalization is to allow
>>>>>> different ways to represent "the same string" to compare equal
>>>>>> (and collate in similar ways, etc., but that is secondary).  If
>>>>>> a precomposed character and a combining sequence represent the
>>>>>> same abstract character and they don't both normalize to the
>>>>>> same thing (same thing under NFC, same thing under NFD, even
>>>>>> though those two sets of results may be different), then
>>>>>> normmalization does not work consistently for its intended
>>>>>> purpose, i.e., fails in that case.
>>>>>>
>>>>>> And there are a whole series of late-added precomposed
>>>>>> characters in earlier versions of Unicode where precisely that
>>>>>> "decompose the new precomposed character back to the sequence"
>>>>>> rule was included in NFC, so this case is neither new nor
>>>>>> theoretical.
>>>>>>
>>>>>> Now, I don't know what gets Ken to "edge case".  No special
>>>>>> treatment would be needed if the [new] precomposed character
>>>>>> always looked different from the combining sequence.  The choice
>>>>>> of name for the precomposed character would be stupid, but that
>>>>>> would be the _only_ problem.  But I've checked with a couple of
>>>>>> users of Arabic script and they assure me that, as far as they
>>>>>> know/ can tell, the two forms will normally yield exactly the
>>>>>> same printed results.  We'd have an edge case if, for example,
>>>>>> the considerable rendering differences between Arabic Script as
>>>>>> used in Arabic language and Arabic Script as used in
>>>>>> Perso-Arabic languages produced a rendering difference.  Again,
>>>>>> I don't even pretend to be an Arabic expert, but that would
>>>>>> constitute an edge case of the "sometimes" variety and, to me,
>>>>>> it would be lots safer to apply the "decompose the new
>>>>>> precomposed character" rule than not.  More on that below.
>>>>>>
>>>>>>
>>>>>> --On Monday, July 07, 2014 21:21 +0200 Patrik Fältström
>>>>>> <[email-masked]> wrote:
>>>>>>
>>>>>>>> Well, yes, but it also reads as though U+0654 should have
>>>>>>>> required a contextual rule or something.  At least, it does
>>>>>>>> to me.
>>>>>>
>>>>>> Again, I am not an Arabic script expert and won't even pretend,
>>>>>> but this seems to me to have nothing to do with contextual
>>>>>> rules.  We use those rules to restrict the use of characters to
>>>>>> some specific script context when they might plausibly be
>>>>>> applied elsewhere or look like something used elsewhere.  I
>>>>>> suppose we could use a contextual rule to prevent the use of a
>>>>>> combining Hamza with Latin or Han script, but we tend to assume
>>>>>> that things like that would look so stupid and out of place that
>>>>>> no such precautions are needed.  In either event, it wouldn't
>>>>>> help with this case which is, as explained above, entirely about
>>>>>> normalization stablility and predictability.
>>>>>>
>>>>>> There is a possible exception under which we could use a
>>>>>> combining rule, but I think it would be bad news and a horrible
>>>>>> kludge (see [1] below).
>>>>>>
>>>>>>> Yes, that was my point. What has happened in 7.0.0 is not a
>>>>>>> new situation. It might be that we missed the Hamza when
>>>>>>> discussing the contextual rule(s) and if that is the case, a
>>>>>>> contextual rule is what we should add (should have added). Not
>>>>>>> exception rule.
>>>>>>
>>>>>> It is not a new situation and it is not, as discussed above, a
>>>>>> contextual rule one either.  The only thing that is different is
>>>>>> that UTC has decided to not apply their existing/historical
>>>>>> "make normalization keep working right rule".  If we do nothing
>>>>>> and unless _every_ valid rendering of the new precomposed
>>>>>> character will look significantly different from _every_ valid
>>>>>> rendering of the combining sequence, their failure to apply that
>>>>>> rule means that we could have two perfectly valid and
>>>>>> identical-looking DNS labels, in the same script, etc., that do
>>>>>> not compare equal.  IDNA was designed to avoid that case and
>>>>>> categories F (Section 2.6 of RFC 5892) and G (Section 2.7) are
>>>>>> there precisely to deal with the "Unicode messed up and we need
>>>>>> to apply a different rule than their properties alone would give
>>>>>> us" cases.  In this case, the only rules that don't cause other
>>>>>> problems would be to either ban the use of the newly-added
>>>>>> precomposed character(s) (I can't remember from April whether
>>>>>> there is more than one) entirely or to
>>>>>> ban the combining sequences.  The disadvantage of the first is
>>>>>> that it will cause some inconvenience going forward; of the
>>>>>> second that it might make a now-valid label invalid, something
>>>>>> that we promised to not do again.
>>>>>>
>>>>>> There is a small procedural problem, which is that Category G,
>>>>>> where I think this really belongs, says "changes in property
>>>>>> values" rather than "changes in property values or normalization
>>>>>> screwups".  To make that a little more complicated, should we
>>>>>> want to pick the second options, I don't believe we have
>>>>>> anywhere that we ban a particular combining sequence without
>>>>>> completely banning the use of one or more of the characters that
>>>>>> make it up [1].  But nothing would prevent our writing a new
>>>>>> document that updates 5892 by adding this (these) new
>>>>>> precomposed characters to the Category F "Disallowed" list,
>>>>>> using the discussion above as a basis.
>>>>>>
>>>>>> If that is what we want to do, I'm willing to write the I-D.  If
>>>>>> not, I see no way to forestall this action other than to
>>>>>> generate a formal appeal, one that would necessarily call for
>>>>>> changes in the review process for these revisions and tables.
>>>>>>
>>>>>> regretfully, but I think the precedent here is really important.
>>>>>>
>>>>>> john
>>>>>>
>>>>>>
>>>>>>
>>

So, where are we and what to do?

I think:

1. A decision "JUST" must be made regarding 7.0.0 according to my email above. Of course updated to the situation today (as John wrote) and findings by others (as Asmus said) that though are just "different positions" and one of the reasons I see "it is hard to find consensus". And THAT is ultimately what IAB must make a very very very careful decision about.

2. ICANN Board finally made a decision to be strict regarding IDNA2008, but the question then ends up being "how to ensure it is implemented"? ICANN Board have now reached out to IAB (and a few others -- SSAC included) and I saw IAB asked for people a few days ago. I will present the issue with ccTLDs (and emojis) that John explained where ultimately who controls a ccTLD is the hidden question.

I think, if IAB *DO* believe (at least until we have something better), they:

A. MUST make a recommendation/decision regarding how to move forward with IDNA2008/Unicode

B. MUST take the flag and advocate with other players (W3C, ICANN etc) strict implementation of IDNA2008

And if IAB do not want to take this decision "just like that" because of risks John lay out between the lines (no new Kobe) then IAB I think should discuss carefully, now with the help of ICANN, "how to move forward". Maybe the group ICANN Board asked for might help? I do not know.

But the situation is pretty serious. For many reasons. And I must say (positively) that the last couple of messages in this thread lay out the issue in a relative good way.

   Patrik