Subject: Genart telechat review of draft-faltstrom-unicode11-08

From: Dan Romascanu via Datatracker <[email-masked]>
Date: Mon, 18 Mar 2019 00:31:36 -0700

Reviewer: Dan Romascanu
Review result: Ready with Issues

I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed
by the IESG for the IETF Chair. Please wait for direction from your
document shepherd or AD before posting a new version of the draft.

For more information, please see the FAQ at

<https://trac.ietf.org/trac/gen/wiki/GenArtfaq>.

Document: draft-faltstrom-unicode11-08
Reviewer: Dan Romascanu
Review Date: 2019-03-18
IETF LC End Date: 2019-03-05
IESG Telechat date: Not scheduled for a telechat

Summary:

This document is Ready. However, I have signaled one minor issue in my LC
review which was not addressed in the revision, neither have I seen an answer
to the clarification that I requested (unless I missed some mail).

Major issues:

Minor issues:

(from my previous review, not addressed)

The IANA Considerations text is not clear to me.

> IANA is requested to update the IDNA Parameters registry of derived
   property values, after the expert reviewer validates that the derived
   property values are calculated correctly.

What does this exactly say? Are these recommendations for future actions, or do
they refer specifically to property values described in this document? Are the
expert reviewer validation and IDNA Parameters registry update required to be
performed prior to approval of this document?

Nits/editorial comments:

---

From: Barry Leiba <[email-masked]>
Date: Mon, 18 Mar 2019 08:31:26 -0400

Hi, Dan.  Thanks for the review, and I’m sorry you didn’t get a response to
your question before.

Last part first: IANA does not normally make changes until the document is
approved, and this is not an exception.

The changes to the derived property values are described in detail in the
text in Section 3.  The designated expert will need to check all that and
generate a list of specific chaanges to the table here:

https://www.iana.org/assignments/idna-tables-6.3.0/#idna-tables-properties

...and IANA will then apply those changes.

Barry

---

From: Barry Leiba <[email-masked]>
Date: Mon, 18 Mar 2019 08:43:05 -0400

(And I was still not sufficiently clear...)
The list for the expert to check is in the appendices.

Barry

On Mon, Mar 18, 2019 at 8:31 AM Barry Leiba <[email-masked]> wrote:
>
> Hi, Dan.  Thanks for the review, and I’m sorry you didn’t get a response to your question before.
>
> Last part first: IANA does not normally make changes until the document is approved, and this is not an exception.
>
> The changes to the derived property values are described in detail in the text in Section 3.  The designated expert will need to check all that and generate a list of specific chaanges to the table here:
>
> On Mon, Mar 18, 2019 at 3:31 AM Dan Romascanu via Datatracker <[email-masked]> wrote:
>>
>> Reviewer: Dan Romascanu

---

From: Alissa Cooper <[email-masked]>
Date: Mon, 18 Mar 2019 08:51:19 -0400

> On Mar 18, 2019, at 8:31 AM, Barry Leiba <[email-masked]> wrote:
> 

Indeed, this document got moved to a later telechat and in the midst of that I failed to thank you for your reviews, Dan. I entered a Yes ballot. IANA had been tracking this for some time so I believe the instructions are clear. 

Alissa

> https://www.iana.org/assignments/idna-tables-6.3.0/#idna-tables-properties <https://www.iana.org/assignments/idna-tables-6.3.0/#idna-tables-properties>
> On Mon, Mar 18, 2019 at 3:31 AM Dan Romascanu via Datatracker <[email-masked] <mailto:[email-masked]>> wrote:
> Reviewer: Dan Romascanu
> <https://trac.ietf.org/trac/gen/wiki/GenArtfaq <https://trac.ietf.org/trac/gen/wiki/GenArtfaq>>.
> 
> _______________________________________________
> Gen-art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/gen-art

---

From: Dan Romascanu <[email-masked]>
Date: Mon, 18 Mar 2019 16:04:41 +0200

Hi Barry,

Thank you for the clarification. Probably adding something like ('... as
detailed in Section 3') would have helped to make clear the scope of what
is referred to in the IANA Considerations section.

Regards,

Dan

On Mon, Mar 18, 2019 at 2:43 PM Barry Leiba <[email-masked]> wrote:

> > Last part first: IANA does not normally make changes until the document
> is approved, and this is not an exception.
> >
> > The changes to the derived property values are described in detail in
> the text in Section 3.  The designated expert will need to check all that
> and generate a list of specific chaanges to the table here:
> >
> >

---

From: John C Klensin <[email-masked]>
Date: Mon, 18 Mar 2019 13:21:37 -0400

Dan,

I have read Barry's two responses, Alissa's response, and your
response to Barry, but this note seems the easiest one to which
to respond.  This particular draft is, as I understand it, OBE,
but there are (still) a few things going on that are probably
not visible from this particular I-D alone and that were even
less visible in draft-faltstrom-unicode11-07.   The draft and
those issues  has been discussed extensively in the
recently-created I18n directorate and many, perhaps most, of the
changes between -07 and -08 are the result of those discussions
(the summary below might be consistent with those directorate
discussions, but I'm not sure there is sufficient consensus
there to make such a claim, so take the summary as my personal
opinion).   A bit of that perspective follows for those who have
not been following IDNA or i18n issues closely; the IDNA-update
list is copied in the hope that I (and Patrik and others) won't
need to go through this too many more times.

(1) This review of new Unicode versions and these IANA tables
are inherently odd.  Usually, when information is sent to IANA
and put, by them, in a registry, the contents of that registry
are considered the authoritative source of the information.  As
one of the older examples still in use, if there is a list of
assigned port numbers floating around the Internet somewhere and
the values in it are different from the values in the IANA
registry (
https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml
), the IANA registry is authoritative and that list is simply
wrong.   That is not the case for these IDNA-related tables.
For them, according to the IDNA2008 specs (and RFC 5892 in
particular) the authoritative information for a particular
application or implementation is to pick the version of Unicode
that application or implementation is using (in theory, at a
given instant if that is changing) and then calculate the
derived properties as specified from RFC 5892, and then use that
information (if the implementation needs a table, that is how it
is supposed to get the table).  The WG was persuaded, after
what, IIR, was considerable demand, to create the IANA registry
and put tables in it, but that does not make the IANA tables
authoritative, nor do they distinguish between supported code
points or supported versions of Unicode, etc.  

(2) In a more perfect world, the review called for by RFC 5892
and reflected in RFC 6452 and this I-D would not be necessary at
all.  Unicode would announce a new version, assorted entities
would calculate tables for their use with that version, and
everyone would go merrily on their way.  The IDNA-update WG was
painfully aware of the imperfection of the world and, for this
particular case, was worried about two possibilities (both of
which we hoped would be very, very rare).  One was that, for a
new version of Unicode, the Unicode Consortium would make
changes in the properties or category values they assigned to
particular code points in a way that would change the derived
properties calculated by the RFC 5892 algorithm for those code
points.  This I-D primarily reflects a review designed to detect
such changes.  As I read RFC 5892, if such cases are detected,
the normal response is to introduce additional backward
compatibility exceptions into the list in RFC 5892 to preserve
stability (Patrik reads that a bit differently than I do).  The
other option is to leave the exception list in 5892 unchanged,
but that means that an application using Unicode version YY and
tables corresponding to it (and possibly a current set of IANA
tables) is not going to behave in a way that is completely
consistent (except for code points not allowed earlier because
they were unassigned) with an application using Unicode version
XX and corresponding IDNA tables).   Ultimately, the choice is
whether, for a particular code point whose Unicode properties
have changed, to maintain consistency with Unicode changes or to
maintain consistency within implementations that calculated IDNA
derived properties based on an earlier Unicode version and are
now recalculating.   

The second possible problem, to which this I-D pays little
attention, is one in which it is discovered that some action
(such as assigning a new code point) or other discovery exposes
an issue or other behavior that was not anticipated by IDNA2008.
This is obviously a much more complicated problem, harder to
review for and detect and much harder to figure out what to do
with.  It was, however, precisely the sort of situation that we
discovered when reviewing the additions in Unicode 7.0.  That
review led to the 2015 IAB statement and to
draft-klensin-idna-5892upd-unicode70.   If the issue were just
about one code point, we could either make it an exception or
ignore it, and move on.  However,  it isn't and examining it led
to a series of still-unresolved issues about assumptions
IDNA2008 made about Unicode and its evolution.  Since that
discovery in 2014, time and Unicode versions have moved on and
applications and libraries that are based on doing their own
derived property calculations (as IDNA2008 requires) have
diverged more and more from any assumption that the tables
available from IANA are somehow "current" or that other
inferences can be drawn from them and the version of Unicode
they reference.

(3) As long as there are no changes or issues of either of the
types described in (2) above, a reasonable reading of RFC 5892
is that the Designated Expert shrugs his or her shoulders,
passes a new table off to IANA (or validates a new table that
IANA calculates) and moves on, with no report to IESG or I-D
that is expected to become an RFC required.   Viewed that way,
RFC 6452 was published only to confirm that, for the first
version of Unicode to appear after the IDNA2008 docs were
finalized, things were working as predicted and no IETF action
was required.  draft-faltstrom-unicode11 is significant, less
because several years have passed and we need to catch up (and
maybe explain) but because, between Unicode 6.x and 11.0 (or
12.0) we've discovered examples of both changed derived
properties for code points that existed before the relevant
Unicode version and a case of discovering an "other problem"
that opens one or more cans of worms.

(4) Because the IDNA requirement is that applications calculate
derived properties (or find and use an appropriate table) that
corresponds to whatever version of Unicode they are using,
IANA's having a registry that consists only of the table
corresponding to one particular version of Unicode seems
inappropriate, unwise, and maybe inconsistent with that
provision of the standard.

It is fairly clear to me that sorting the above out and
clarifying things to the point that confusion is considerably
reduced and expectations of the Designated Expert(s) are clear
should now be a reasonably high priority.   It is almost equally
clear that putting those clarifications in the critical path in
front of publication of draft-faltstrom-unicode11 (or
-unicode12), much less getting relatively more current tables
into IANS's hands, would be unwise pragmatically, however much
more satisfying it might be procedurally.

Specific suggestions about how this might be untangled, text and
ideas about where to put it, etc., would be very welcome (at
least by me).

best,
    john

--On Monday, March 18, 2019 00:31 -0700 Dan Romascanu via
Datatracker <[email-masked]> wrote:

> I am the assigned Gen-ART reviewer for this draft. The General
> Area Review Team (Gen-ART) reviews all IETF documents being
> processed by the IESG for the IETF Chair. Please wait for
> direction from your document shepherd or AD before posting a
> new version of the draft.
> 
> This document is Ready. However, I have signaled one minor
> issue in my LC review which was not addressed in the revision,
> neither have I seen an answer to the clarification that I
> requested (unless I missed some mail).
> 
> Major issues:
> 
> Minor issues:
> 
>    property values, after the expert reviewer validates that
> the derived    property values are calculated correctly.
> 
> What does this exactly say? Are these recommendations for
> future actions, or do they refer specifically to property
> values described in this document? Are the expert reviewer
> validation and IDNA Parameters registry update required to be

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 19 Mar 2019 07:47:01 +0100

On 18 Mar 2019, at 18:21, John C Klensin wrote:

> (2) In a more perfect world, the review called for by RFC 5892 and reflected in RFC 6452 and this I-D would not be necessary at all.  Unicode would announce a new version, assorted entities would calculate tables for their use with that version, and
> everyone would go merrily on their way.  The IDNA-update WG was painfully aware of the imperfection of the world and, for this particular case, was worried about two possibilities (both of which we hoped would be very, very rare).  One was that, for a new version of Unicode, the Unicode Consortium would make
> changes in the properties or category values they assigned to particular code points in a way that would change the derived properties calculated by the RFC 5892 algorithm for those code points.  This I-D primarily reflects a review designed to detect such changes.  As I read RFC 5892, if such cases are detected, the normal response is to introduce additional backward
> compatibility exceptions into the list in RFC 5892 to preserve stability (Patrik reads that a bit differently than I do).

Correct, and let me try to explain this in slightly different words. And yes, we do read 5892 slightly differently, but I claim the outcome is the same. Let me come back to that.

1. We MUST remember that it is the algorithm in 5892 that is normative. Not any table of derived property values. As John wrote which I agree with, having IANA having a table with non-normative data was probably a very bad idea. And maybe that registry should simply be removed. It just confuses people. If it confuses people on IESG, of course it confuses other people as well. Maybe the description of the table at IANA should be changed? Let me think and discuss with IANA.

2. The whole idea with the algorithm was that the property values that the algorithm uses would never ever be changed as Unicode Consortium said those would be stable. At least the property values IETF did use in the algorithm. So having the algorithm normative, and "just" let Unicode consortium release new versions of Unicode was EXACTLY one design criteria for IDNA2008 that is vastly different from the "pick code points" that IDNA2003 did.

3. RFC 5892 was based on Unicode 5.2.0, and since then Unicode have released versions 6.0.0, 6.1.0, 6.2.0, 6.3.0, 7.0.0, 8.0.0, 9.0.0, 10.0.0, 11.0.0, 12.0.0 and a pre-relase of 12.1.0. Of all of these versions, two include incompatible changes: 6.0.0 and 11.0.0.

4. As John wrote, the simple solution for the IDNA2008 standard to move forward would be to change the algorithm so that the IDNA2008 algorithm when applied to any Unicode version would produce the same result, and indeed did we think about that when doing RFC 5892. See section 2.7 <https://tools.ietf.org/html/rfc5892#section-2.7>. By adding code points and explicit derived property values to rule set [G], we would have backward compatibility.

5. When 6.0.0 was released, I as appointed expert did flag that for IETF (I do not remember exactly how, but I might be able to dig up those email messages) and after some discussions the conclusion was that rule set [G] should NOT be updated. I think I was convinced that at that time some software libraries where changed, and it was more important to get IDNA2008 out the door and we had the risk of two different incompatibilities:

5.1. If we added a rule to [G], the incompatibility would be between software packages that did use the original version of 5892 algorithm without the added rule and the ones which did have the updated rule.

5.2. If we did not add a rule to [G], the incompatibility would be between software packages that calculated the derived property value on Unicode 5.2.0 and the ones that did the calculation on 6.0.0.

Ultimately [5.2] above did win, and that was explained in RFC 6452. And I as the appointed expert and editor of 5892 I feel good having the discussion about this incompatible change that *would never happen* recorded in 6452.

6. The fact it was an issue was of course also brought up to the Unicode Consortium as I am (handy enough) also the liaison from IETF to Unicode Consortium. No response. They do push, as we know, TR#46 which is a different kind of animal than IDNA2008. Oh well. Things where fine, and we moved on. And IETF could focus on what IETF really should focus on, to ensure the algorithm itself was still "ok". I.e. without looking at individual code points, but more in general terms, was IDNA2008 still good enough? Lots of good feedback from people like Asmus that was fighting like mad (and still is) trying to a. convince registries they MUST come up with a subset of IDNA2008 permissible code points when they decide what can be used in their zones and b. come up with processes and rules for how that work should be done. Specifically work did go on for a long time regarding what code points can be used in the root zone, i.e. for TLDs. This is managed in the IAB document, and the Klensin draft about and more.

6. Now in version 11.0.0, the non-backward-compatible change happened again. Sigh. I was at that time on my way to just give up, and recommend IETF that what we do today, to reference something that we thought was normative, stable etc (i.e. the Unicode Standard) did not work. That IETF instead would pick up and use some relationship with ISO that should be formalized, and instead of referencing the Unicode Consortium IETF should reference ISO, and then ISO should have in its rules that backward compatibility would be a requirement. If then Unicode did ship something to ISO for "approval", then ISO would say "no, try again" or else ISO would make a process violation that members could appeal within ISO. But after sleeping on this for a bit, I felt the issue with 11.0.0 also happened with 7.0.0 and maybe one should first have a discussion within IETF again before the bolts are blown regarding relationship with Unicode Consortium. That just started when the I18N directorate was closed, and I ended up in a void. After long time, pushing and what not (lets discuss in the bar) this group finally was created were this VERY SERIOUS ISSUE could be discussed, and as John explained, there are many things to discuss, where we can boil it down to two:

A. Should the non-backward compatible change in 11.0.0 result in a change in rule [G] in RFC 5892, or should we accept a non-backward compatible change? To trigger the discussing, I proposed the same result as for 7.0.0, to NOT update [G], and simply accept it. This also because the next issue is the important one.

B. What should we do with IDNA2008? Obviously Unicode Standard is not stable enough. Or is it? What should we do with review? Should we have to start do what Martin just did with 12.0.0? Do we IETF have the expertise? Can we rely on individuals like Martin, Asmus, myself and John and very few more to be around? Can we rely on Unicode Consortium? Is this the time to instead move to ISO? Like keeping IDNA2008 algorithm BUT tie it to ISO approved charset, and then ask ISO to protect the backward compatibility? Or are we done, so that we simply freeze IDN to a specific Unicode version, and simply ignore all added code points after that version? I.e. at 5.2.0 we knew some interesting code points where still to be added, but after 11.0.0?

What I now know is that IESG have told me that we do have agreement in IETF on draft-faltstrom-unicode11-08.txt which implies moving to IDNA 11.0.0 without adding things to [G] is the path forward for now.

What to do with 12.0.0 and future versions is still up in the air.

See (some of) you in Prague.

See you in the bar...

   Patrik