Subject: Re: Last Call: <draft-faltstrom-5892bis-04.txt> (The Unicode code points and IDNA - Unicode 6.0) to Proposed Standard

From: John C Klensin <[email-masked]>
Date: Wed, 25 May 2011 13:32:22 -0400

Just for the record,..

I favor the publication of this document with a minimum of
further fuss.

I read and studied this document before the first I-D was
published, participated in discussion of it on the IDNABIS WG
list and again on the APPSAWG list.  In each case, essentially
the same arguments were repeated and the same conclusion
reached.  While the consensus is definitely rough, we are
putting a tremendous amount of of review cycles and energy into
a document whose essence is "we have agreed to do nothing".  We
rarely publish documentation of a decision to do nothing in the
RFC Series.  Instead, we simply do nothing and move on.

   john

_Background and Reprise (in case needed or wanted)_

The argument for doing something --that this document makes and
describes the wrong decision-- is rooted in a set of decisions
the IDNABIS WG made (also with rough, rather than perfect,
consensus) a rather long time ago.  One of the key differences
between IDNA2003 and IDNA2008 was the decision to try to be
independent of Unicode versions rather than tied to a specific
one.  That decision, in turn, required that the IDNA properties
of characters -- whether they are Protocol-VALID in IDNs,
DISALLOWED, or valid only in particular contexts -- became a
matter of rules that, in turn, utilize the values of selected
Unicode properties.  If Unicode never changed the properties for
a particular character once it was allocated (and avoided
introducing new characters that required contextual evaluation),
the rule-set would never need to be changed and we would never
have to debate, and write, documents keyed to those property
changes: new versions of Unicode would add new characters but
not change the IDNA properties of previously-assigned ones.

The world is not that perfect: Unicode does change properties of
already-assigned characters.  It is rare (how rare depends a bit
on perspective), but it does happen.  The reason is typically to
correct errors made in earlier property assignments, but other
reasons are possible (or one could have a
philosophically-interesting, but otherwise useless, debate about
how far the concept of "error" can extend).  IDNA2008 recognized
that such changes might occur and might be disruptive, so
provided for a special "exception" rule that could be used to
list characters that needed special treatment because of changes
in Unicode properties.  

The WG discussed the question of how to fill the table
associated with that rule in, a discussion that included the
realization that having the  having the table become large would
simply create a new, and hard-to-explain, Unicode-version
dependence.  One option was to populate it automatically: if
Unicode made changes, the table would be filled in to preserve
the behavior at the time the character was first allocated or
that appeared in Unicode 5.2, whichever came later, whatever
that was.  Another was to turn the decision-making over to the
Unicode consortium, which might have resulted in either that
same rule or a rule that treated changes from DISALLOWED to
PVALID differently from changes from PVALID to DISALLOWED.  The
WG rejected both of those ideas as well as the idea that some
changes were inherently more attractive than others.

Instead, the WG concluded that Unicode changes of character
properties that affected IDNA needed to be evaluated in the IETF
on a case-by-case basis as to whether they were important enough
to justify an addition to that exceptions table or whether the
balance fell on the side of keeping the table small (and IDNA
reflecting as short and obvious an exception list as possible).  

In this particular instance, rough consensus has been reached in
multiple groups that it is better to keep the rules (including
that exception table stable and compact than to change the rules
in the interest of preserving the character classification
associated with Unicode 5.2.  Had the characters involved been
less obscure, or more obviously used in domain names for other
than demonstrations and equivalent purposes, perhaps the
decision would have been different (or, if they were less
obscure, perhaps Unicode either would not have made the mistake
in the first place or would have decided to not correct it).

But wanting to consider the tradeoffs and balance between those
alternatives, probably with a slight bias toward not making
changes in the rules,a is precisely why the WG decided that
Unicode property changes needed to be considered by the IETF and
on a case-by-case basis.   

There is no perfect answer to the tradeoff involved unless
Unicode never makes an error and concludes that a correction is
needed.  Because no perfect answer exists, it is possible to
reopen the issue, bring up variations on the same old arguments,
and debate them endlessly.  In that case, we've done that at
least twice, and, depending on how one counts, probably several
more times.  We've decided.  Let's get the document published
rather than seeing how many words and person-hours we can devote
to saying "we considered the issue and decided to do nothing".

     john

--On Monday, 23 May, 2011 15:19 -0700 The IESG
<[email-masked]> wrote:

> 
> The IESG has received a request from the Applications Area
> Working Group WG (appsawg) to consider the following document:
> - 'The Unicode code points and IDNA - Unicode 6.0'
>   <draft-faltstrom-5892bis-04.txt> as a Proposed Standard
> 
> The IESG plans to make a decision in the next few weeks, and
> solicits final comments on this action. Please send
> substantive comments to the [email-masked] mailing lists by
> 2011-06-06.

---

From: Mark Davis ☕ <[email-masked]>
Date: Wed, 25 May 2011 10:54:20 -0700

I also favor publication of the document with a minimum of further fuss.

[For a somewhat different reason. I do believe that
draft-faltstrom-5892bis-04.txt
makes an incorrect choice, breaking compatibility when that isn't at all
necessary. However, I don't think that any further discussion will change
the decision-maker's minds. Since further discussion appears to just be a
waste of everyone's time,  the document might as well just get issued.]

Mark

*— Il meglio è l’inimico del bene —*

On Wed, May 25, 2011 at 10:32, John C Klensin <[email-masked]> wrote:

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

---

From: Simon Josefsson <[email-masked]>
Date: Thu, 26 May 2011 13:19:42 +0200

The IESG <[email-masked]> writes:

> The IESG has received a request from the Applications Area Working Group
> WG (appsawg) to consider the following document:

Dear IESG,
Is the intention that this document will update RFC 5892 or not?
The document does not contain a "Updates:" header but the draft name
suggests otherwise to me, hence my question.

If the document does not update RFC 5892 (or some other document), I
support publishing this document because it will not affect my
implementation of RFC 5892.

If the document updates RFC 5892, in order to remain compliant with the
RFCs I would have to modify my implementation and make a backwards
incompatible change.  Today U+19DA converts to xn--pkf.  With this
document, I would have to raise an error for that input instead.  I
believe a case-by-case evaluation for each modified code-point is a good
way to determine whether or not to add an exception in the IDNA tables.
I haven't seen any discussion why U+19DA is so harmful that it has to be
disallowed.  On the contrary, everyone appears to agree that the code
point is not widely used and the implications of continue permitting it
are minor.  Thus I would support publication of the document after
adding U+19DA to table BackwardCompatible (G) as PVALID.

I do realize that I may be in the rough part of the consensus here,
which happens, but I want to provide my feedback for the record and
allow the decision process to proceed.  At least I will be able to shift
blame to someone else if/when my users gets confused by this. :-)

/Simon

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 26 May 2011 08:37:47 -0400

For all those people just dying to know about this character (U+19DA),
the latest Unicode code chart listing it is here
http://www.unicode.org/charts/PDF/U1980.pdf
and the name of the character is NEW TAI LUE THAM DIGIT ONE.

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street
 Milford, MA 01757 USA
 [email-masked]

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 26 May 2011 08:26:07 -0700

> suggests otherwise to me, hence my question.

As document co-editor, let me say definitively: this document does *not* update RFC 5892. The filename is an artifact of the early consideration and, as you know, disappears once the document is published as an RFC. Note the information at <https://datatracker.ietf.org/doc/draft-faltstrom-5892bis/>: this is what the IESG is going on. There is no mention of "updates" anywhere there.

> implementation of RFC 5892.

It is always nice to hear support from actual implementers. :-)

--Paul Hoffman

---

From: Simon Josefsson <[email-masked]>
Date: Thu, 26 May 2011 21:06:00 +0200

Paul Hoffman <[email-masked]> writes:

> As document co-editor, let me say definitively: this document does
> *not* update RFC 5892. The filename is an artifact of the early
> consideration and, as you know, disappears once the document is
> published as an RFC. Note the information at
> <https://datatracker.ietf.org/doc/draft-faltstrom-5892bis/>: this is
> what the IESG is going on. There is no mention of "updates" anywhere
> there.

Thanks for sharing your view.  Earlier discussion suggested that the
decision of document status would be punted to the IESG, but it seems
they will only make an aggregate decision about the entire document.

/Simon

---

From: Pete Resnick <[email-masked]>
Date: Sat, 28 May 2011 17:06:42 -0500

> suggests otherwise to me, hence my question.

The IESG has not yet discussed this topic, and I will be bringing it to 
their attention. We may decide to include "Updates: RFC 5892"; we may not.

> document, I would have to raise an error for that input instead.

My understanding is that the above statements are, if not false, at 
least incomplete. It is not true, at least with regard to RFC 5892, that 
"today U+19DA converts to xn-pkf" because RFC 5892 doesn't do any 
"converting". It *is* true that the code point U+19DA is PROTOCOL VALID 
using the algorithm in RFC 5892 as applied to Unicode 5.2, and is 
DISALLOWED using the algorithm in RFC 5892 as applied to Unicode 6.0. If 
what you mean above is that your implementation intends to raise an 
error for DISALLOWED code points and would, in a Unicode 6.0 
environment, evaluate U+19DA as PVALID and therefore not raise that 
error, then it is not "compliant" with RFC 5892, irrelevant of the 
"Updates" status of the present document.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Simon Josefsson <[email-masked]>
Date: Sun, 29 May 2011 08:58:48 +0200

Pete Resnick <[email-masked]> writes:

> The IESG has not yet discussed this topic, and I will be bringing it
> to their attention. We may decide to include "Updates: RFC 5892"; we
> may not.

Thanks for clarifying.

> least incomplete. It is not true, at least with regard to RFC 5892,
> that "today U+19DA converts to xn-pkf" because RFC 5892 doesn't do any
> "converting".

Sure.  RFC 5892 is part of the IDNA2008 set and primarily RFC 5891
describe the conversion.  RFC 5891 uses the properties defined in RFC
5892.

> It *is* true that the code point U+19DA is PROTOCOL VALID using the
> algorithm in RFC 5892 as applied to Unicode 5.2, and is DISALLOWED
> using the algorithm in RFC 5892 as applied to Unicode 6.0.

Right.

> If what you mean above is that your implementation intends to raise an
> error for DISALLOWED code points and would,

That is required by RFC 5891 section 4.2.2:

4.2.2.  Rejection of Characters That Are Not Permitted

   The candidate Unicode string MUST NOT contain characters that appear
   in the "DISALLOWED" and "UNASSIGNED" lists specified in the Tables
   document [RFC5892].

and section 5.4:

   Putative U-labels with any of the
   following characteristics MUST be rejected prior to DNS lookup:

   o  Labels containing prohibited code points, i.e., those that are
      assigned to the "DISALLOWED" category of the Tables document
      [RFC5892].

> in a Unicode 6.0 environment, evaluate U+19DA as PVALID and
> therefore not raise that error, then it is not "compliant" with RFC
> 5892, irrelevant of the "Updates" status of the present document.

I don't see how.

My code uses the tables from RFC 5892 which were generated in an Unicode
5.2 environment.  My IDNA2008 code may eventually run in an Unicode 6.0
environment, or any other future version of Unicode.  I can't control
the Unicode version used, and from what I understand this is one of the
features of IDNA2008.  Implementations need not lock down the Unicode
version to a single Unicode version, as they had to do for IDNA2003.

If this model is not permitted, I believe there are bigger problems.

To avoid doubt, and to back up your assertment that my implementation is
non-compliant, please point to the "MUST" or "SHOULD" in RFC 5892 that
forbis this, to me, logical implementation approach.

/Simon

---

From: John C Klensin <[email-masked]>
Date: Sun, 29 May 2011 11:20:19 -0400

--On Sunday, May 29, 2011 08:58 +0200 Simon Josefsson
<[email-masked]> wrote:

>> therefore not raise that error, then it is not "compliant"
>> with RFC 5892, irrelevant of the "Updates" status of the
>> present document.
> 
> I don't see how.
> 
> My code uses the tables from RFC 5892 which were generated in
> an Unicode 5.2 environment.  My IDNA2008 code may eventually
> run in an Unicode 6.0 environment, or any other future version
> of Unicode.  I can't control the Unicode version used, and
> from what I understand this is one of the features of
> IDNA2008.  Implementations need not lock down the Unicode

It seems to me that this is exactly where we are having a
misunderstanding.   In terms of determining conformance, those
tables are not normative, so it is not possible to say "I
implemented the tables in RFC 5892 and therefore I conform to
the standard".  The closest you can get would be to say "I
implemented the rules and tested against the tables when those
rules were applied to Unicode 5.2 and therefore have great
confidence in my implementaton", but conformance statements stop
with "implemented the rules correctly".  

For practical reasons, we expect to see production
implementations using tables or other abstractions of the rules
that are somewhat pre-compiled, not applying the rule set each
time.   One consequence of this is that a given table-based
implementation is inevitably dependent on versions of Unicode
even if the Standard (and its conformance requirements) is not.
That would be true even if the type of change (correction) that
occurred with version 6.0 of Unicode had not occurred. It would
still be necessary to construct version-dependent tables to deal
with newly-assigned code points.

>From the perspective of those who argued that the document
titled "...5852bis.." should not be produced and published
because it is unnecessary, the point is that we would not have
generated the document at all had the only changes been the
addition of new PROTOCOL-VALID and DISALLOWED code points by
virtue of new code points being added to Unicode.  But, in
practical terms, that is a much greater change to an
implementation than anything related to these few characters
with changed properties.

And, again, this situation would be true of virtually any
specification that depends on Unicode, regardless of whether the
definition is in terms of  rules/properties or tables. There
would be an exception if the specification depended on code
point assignments alone and was okay with treating unassigned
code points as if they had been assigned if they turned up in
the data stream (IDNA2003 attempted to lay the foundation for
the latter but failed because all of the properties that an
unassigned code point will have when it is assigned cannot be
known).  For anything else, working properly with a given
version of Unicode requires updating of code point tables,
normalization tables, and assorted property tables.   As Mark
points out, defining things in terms of the tables, with the
rules providing only guidance, has some important advantages in
this regard.  However, it guarantees the need to talk about
conformance to a Unicode version, not just "Unicode".

> To avoid doubt, and to back up your assertment that my
> implementation is non-compliant, please point to the "MUST" or
> "SHOULD" in RFC 5892 that forbis this, to me, logical
> implementation approach.

The key is the text in Section 4 that says:

	"The table in Appendix B shows, for illustrative
	purposes, the consequences of the categories and
	classification rules, and the resulting property values.
	
	"The list of code points that can be found in Appendix B
	is non-normative.  Sections 2 and 3 are normative."

It seems to me that is very clear about the relationship between
the rules and the tables.   That relationship is reiterated in
Section 7.1.1 of RFC 5892.

You could reasonably say that your implementation is conformant
but current only to Unicode 5.2.   If you are willing to say
that, I guess you don't need to change anything.   While we
recognize that you have no control over the Unicode version in
use, good sense suggests that systems will update versions of
Unicode (including all of the associated tables and support
routines as applicable) and versions of your library together,
While that should be clear from the context of the discussions
in RFC 5891 and 5892, RFC 5894 is quite explicit about it in the
second bullet of Section 7.1.2:

 "o The Unicode tables (i.e., tables of code points,
	character classes, and properties) and IDNA tables
	(i.e., tables of contextual rules such as those
	that appear in the Tables document), must be
	consistent on the systems performing or validating
	labels to be registered.  Note that this does not
	require that tables reflect the latest version of
	Unicode, only that all tables used on a given
	system are consistent with each other."

Similarly, the first bullet of 7.1.3 reads:

 "o Maintain IDNA and Unicode tables that are consistent
	with regard to versions, i.e., unless the application
	actually executes the classification rules in the Tables
	document [RFC5892], its IDNA tables must be derived from
	the version of Unicode that is supported more generally on
	the system.  As with registration, the tables need not
	reflect the latest version of Unicode, but they must be
	consistent."

I hope that helps.

best,
     john

---

From: Simon Josefsson <[email-masked]>
Date: Sun, 29 May 2011 20:29:34 +0200

> even if the Standard (and its conformance requirements) is not.

Right, and that describes my implementation.  There is no difference in
behaviour of an implementation that uses the informative tables in RFC
5892 directly or one that pre-computes the table at compile time using
Unicode 5.2.  The data and output are the same in both cases.  So I
don't follow where you think the misunderstanding is?  I agree with what
you say here.

> Section 7.1.1 of RFC 5892.

s/5892/5894/

Sure.  But that does not prove (or disprove) Pete's claim that my
implementation is non-compliant.

> that, I guess you don't need to change anything.

I claim my implementation is compliant to all requirements in RFC 5890,
RFC 5891, RFC 5892 and RFC 5893.

> While we recognize that you have no control over the Unicode version
> in use, good sense suggests that systems will update versions of
> Unicode (including all of the associated tables and support routines
> as applicable) and versions of your library together,

That is unrealistic.  Traditional operating systems are already so
complex that upgrading them to one Unicode versions across all software
pieces (Java, Perl, SQL databases, web browsers, word processors, etc)
is economically infeasible.

Modern operating system rely so much on network services that it is not
even useful to decouple the local system from external systems.
Essentially "the system" is identical to "the Internet".  A flag day to
upgrade to the latest Unicode version across the Internet is, despite
how infinitely pleasant that would be, impossible.

If it was possible to upgrade software components to the latest Unicode
version in a controlled way, the IDNA2003 model would have worked fine.

Fortunately, I believe IDNA2008 does not require tight Unicode version
synchronization.  In fact, I believe one of the features with IDNA2008
is exactly that it doesn't require all Unicode versions to be in sync in
all parts of the Internet.

> While that should be clear from the context of the discussions in RFC
> 5891 and 5892, RFC 5894 is quite explicit about it in the second
> bullet of Section 7.1.2:
>

That is about registration of labels, not lookup.  Registration is a
centralized process where you can control the software used more easily.

> Similarly, the first bullet of 7.1.3 reads:

You forgot to quote the paragraph before the one you quoted:

   Any application processing a label through IDNA so it can be looked
   up in a DNS zone is required to (the exact rules appear in Section 5

> 	reflect the latest version of Unicode, but they must be

I don't see any similar text about IDNA and Unicode version consistency
requirements in section 5 of RFC 5892.  I'm sure you recall that RFC
5894 is as non-normative as the RFC 5892 tables are.

/Simon

---

From: Pete Resnick <[email-masked]>
Date: Sun, 29 May 2011 13:45:06 -0500

On 5/29/11 1:29 PM, Simon Josefsson wrote:
> John C Klensin<[email-masked]>  writes:
>
>    
>>> an Unicode 5.2 environment.

Then you are, in my terminology, implementing RFC 5892 in a "Unicode 5.2 
environment". Your implementation is carrying the "5.2 environment" with 

> RFC 5891, RFC 5892 and RFC 5893.

There's the problem. You can't claim that your implementation is 
compliant with the above RFCs without also mentioning the version of 
Unicode you are using, precisely because the RFCs are now Unicode 
version independent. Your implementation that evaluates U+19DA as PVALID 
is complaint with the RFCs *as applied to Unicode version 5.2*. Your 
implementation that evaluates U+19DA as PVALID is *not* complaint with 
the RFCs *as applied to Unicode version 6.0*.

pr

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 30 May 2011 13:44:51 +0200

Pete Resnick <[email-masked]> writes:

> Then you are, in my terminology, implementing RFC 5892 in a "Unicode
> 5.2 environment". Your implementation is carrying the "5.2
> environment" with it.

The Unicode library used during run-time, for RFC 5891, is version 6.0
though.

> But I now think I see the source of the misunderstanding:
>
> version independent. Your implementation that evaluates U+19DA as
> PVALID is complaint with the RFCs *as applied to Unicode version
> 5.2*. Your implementation that evaluates U+19DA as PVALID is *not*
> complaint with the RFCs *as applied to Unicode version 6.0*.

The correct claim would then be that I use Unicode 5.2 (for tables) and
Unicode 6.0 (for run-time).

I believe this is typical of how IDNA2008 will be deployed: the IDNA2008
implementation uses pre-computed tables for one Unicode version fixed at
compile-time, and the Unicode library on the system may be more rapidly
changing and could support a later version of Unicode.

Can you point to some (normative) requirement in IDNA2008 that forbids
this?

/Simon