Subject: [precis] Category changes with Unicode 6.3

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 16 Oct 2013 11:34:07 +0900

Excuse me if this has been checked and/or discussed already, but I just 
downloaded the Unicode 6.3 version (officially published a few days ago) 
of http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt and 
found several changes in character classification:

OLD
180E;MONGOLIAN VOWEL SEPARATOR;Zs;0;WS;;;;;N;;;;;
NEW
180E;MONGOLIAN VOWEL SEPARATOR;Cf;0;BN;;;;;N;;;;;

OLD
1A1B;BUGINESE VOWEL SIGN AE;Mc;0;L;;;;;N;;;;;
NEW
1A1B;BUGINESE VOWEL SIGN AE;Mn;0;NSM;;;;;N;;;;;

OLD
2308;LEFT CEILING;Sm;0;ON;;;;;Y;;;;;
2309;RIGHT CEILING;Sm;0;ON;;;;;Y;;;;;
230A;LEFT FLOOR;Sm;0;ON;;;;;Y;;;;;
230B;RIGHT FLOOR;Sm;0;ON;;;;;Y;;;;;
NEW
2308;LEFT CEILING;Ps;0;ON;;;;;Y;;;;;
2309;RIGHT CEILING;Pe;0;ON;;;;;Y;;;;;
230A;LEFT FLOOR;Ps;0;ON;;;;;Y;;;;;
230B;RIGHT FLOOR;Pe;0;ON;;;;;Y;;;;;

Can somebody check whether and how they affect IDNA 2008 and/or precis?

Again, if that has already been done, sorry for the noise.

Regards,   Martin.

P.S.:
All the other changes in UnicodeData.txt:

Change in numerical value only:

OLD
12456;CUNEIFORM NUMERIC SIGN NIGIDAMIN;Nl;0;L;;;;-1;N;;;;;
12457;CUNEIFORM NUMERIC SIGN NIGIDAESH;Nl;0;L;;;;-1;N;;;;;
NEW
12456;CUNEIFORM NUMERIC SIGN NIGIDAMIN;Nl;0;L;;;;2;N;;;;;
12457;CUNEIFORM NUMERIC SIGN NIGIDAESH;Nl;0;L;;;;3;N;;;;;

New characters (my understanding is that these are taken care of 
automatically):

061C;ARABIC LETTER MARK;Cf;0;AL;;;;;N;;;;;

2066;LEFT-TO-RIGHT ISOLATE;Cf;0;LRI;;;;;N;;;;;
2067;RIGHT-TO-LEFT ISOLATE;Cf;0;RLI;;;;;N;;;;;
2068;FIRST STRONG ISOLATE;Cf;0;FSI;;;;;N;;;;;
2069;POP DIRECTIONAL ISOLATE;Cf;0;PDI;;;;;N;;;;;

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 15 Oct 2013 21:15:16 -0600

> downloaded the Unicode 6.3 version (officially published a few days ago)

I had not seen the new release, so I will check over the code points you
have highlighted. Thank you very much for the report!

Peter

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 16 Oct 2013 08:26:43 +0300

I have in earlier beta versions of 6.3 not seen any problems, but I am running my scripts <http://stupid.domain.name/idna> now to see what's up.

   Patrik

On 16 okt 2013, at 05:34, Martin J. Dürst <[email-masked]> wrote:

> Excuse me if this has been checked and/or discussed already, but I just downloaded the Unicode 6.3 version (officially published a few days ago) of http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt and found several changes in character classification:
> 
> OLD
> New characters (my understanding is that these are taken care of automatically):
> 
> _______________________________________________
> Idna-update mailing list
> [email-masked]
> http://www.alvestrand.no/mailman/listinfo/idna-update

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 16 Oct 2013 08:40:35 +0300

Executive summary: Does not impact IDNA2008.

Longer explanation:

> 180E;MONGOLIAN VOWEL SEPARATOR;Cf;0;BN;;;;;N;;;;;

No change:

$ grep '^180E;' ../6.[23].0/allcodepoints.txt 
../6.2.0/allcodepoints.txt:180E;DISALLOWED;I;C;MONGOLIAN VOWEL SEPARATOR
../6.3.0/allcodepoints.txt:180E;DISALLOWED;I;C;MONGOLIAN VOWEL SEPARATOR

> 1A1B;BUGINESE VOWEL SIGN AE;Mn;0;NSM;;;;;N;;;;;

No change:

$ grep '^1A1B;' ../6.[23].0/allcodepoints.txt 
../6.2.0/allcodepoints.txt:1A1B;PVALID;I;A;BUGINESE VOWEL SIGN AE
../6.3.0/allcodepoints.txt:1A1B;PVALID;I;A;BUGINESE VOWEL SIGN AE

> 230B;RIGHT FLOOR;Pe;0;ON;;;;;Y;;;;;

No change:

$ egrep '^230[89AB];' ../6.[23].0/allcodepoints.txt 
../6.2.0/allcodepoints.txt:2308;DISALLOWED;I;;LEFT CEILING
../6.2.0/allcodepoints.txt:2309;DISALLOWED;I;;RIGHT CEILING
../6.2.0/allcodepoints.txt:230A;DISALLOWED;I;;LEFT FLOOR
../6.2.0/allcodepoints.txt:230B;DISALLOWED;I;;RIGHT FLOOR
../6.3.0/allcodepoints.txt:2308;DISALLOWED;I;;LEFT CEILING
../6.3.0/allcodepoints.txt:2309;DISALLOWED;I;;RIGHT CEILING
../6.3.0/allcodepoints.txt:230A;DISALLOWED;I;;LEFT FLOOR
../6.3.0/allcodepoints.txt:230B;DISALLOWED;I;;RIGHT FLOOR

> 12457;CUNEIFORM NUMERIC SIGN NIGIDAESH;Nl;0;L;;;;3;N;;;;;

$ egrep '^1245[67];' ../6.[23].0/allcodepoints.txt 
../6.2.0/allcodepoints.txt:12456;DISALLOWED;I;;CUNEIFORM NUMERIC SIGN NIGIDAMIN
../6.2.0/allcodepoints.txt:12457;DISALLOWED;I;;CUNEIFORM NUMERIC SIGN NIGIDAESH
../6.3.0/allcodepoints.txt:12456;DISALLOWED;I;;CUNEIFORM NUMERIC SIGN NIGIDAMIN
../6.3.0/allcodepoints.txt:12457;DISALLOWED;I;;CUNEIFORM NUMERIC SIGN NIGIDAESH

> 061C;ARABIC LETTER MARK;Cf;0;AL;;;;;N;;;;;

$ egrep '^061C;' ../6.[23].0/allcodepoints.txt 
../6.2.0/allcodepoints.txt:061C;UNASSIGNED;I;J;<reserved>
../6.3.0/allcodepoints.txt:061C;DISALLOWED;I;C;ARABIC LETTER MARK

> 2069;POP DIRECTIONAL ISOLATE;Cf;0;PDI;;;;;N;;;;;

$ egrep '^206[6789];' ../6.[23].0/allcodepoints.txt 
../6.2.0/allcodepoints.txt:2066;UNASSIGNED;I;CJ;<reserved>
../6.2.0/allcodepoints.txt:2067;UNASSIGNED;I;CJ;<reserved>
../6.2.0/allcodepoints.txt:2068;UNASSIGNED;I;CJ;<reserved>
../6.2.0/allcodepoints.txt:2069;UNASSIGNED;I;CJ;<reserved>
../6.3.0/allcodepoints.txt:2066;DISALLOWED;I;C;LEFT-TO-RIGHT ISOLATE
../6.3.0/allcodepoints.txt:2067;DISALLOWED;I;C;RIGHT-TO-LEFT ISOLATE
../6.3.0/allcodepoints.txt:2068;DISALLOWED;I;C;FIRST STRONG ISOLATE
../6.3.0/allcodepoints.txt:2069;DISALLOWED;I;C;POP DIRECTIONAL ISOLATE

   Patrik

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 16 Oct 2013 16:04:31 +0900

Hello Patrick,

Many thanks for checking. Great to see that everything is okay.

Regards,   Martin.

On 2013/10/16 14:40, Patrik Fältström wrote:
> On 16 okt 2013, at 05:34, Martin J. Dürst<[email-masked]>  wrote:
>

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 16 Oct 2013 10:16:08 +0300

Yeah, I had to do it anyways -- as I am expert reviewer of the IANA tables. New versions of the IANA tables where created by IANA the other day, and I just approved them as they match my own calculations. I.e. we have two completely independent implementations of IDNA2008 (one at IANA, one that I have -- see link below) and I compare the output of the two. If the output matches, then we are pretty sure we are correct.

   Patrik

---

From: Mark Davis ☕ <[email-masked]>
Date: Wed, 16 Oct 2013 09:50:04 +0200

FYI: We are considering a change in UTS46 (
http://www.unicode.org/reports/tr46/proposed.html) for the new
Bidi_Controls. However, this does *not* affect IDNA2008.

Mark <https://plus.google.com/114199149796022210033>
*
*
*— Il meglio è l’inimico del bene —*
**

On Wed, Oct 16, 2013 at 7:40 AM, Patrik Fältström <[email-masked]> wrote:

> downloaded the Unicode 6.3 version (officially published a few days ago) of
> http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt and found
> several changes in character classification:
> >
> > OLD

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 16 Oct 2013 16:49:09 +0300

On 16 okt 2013, at 16:46, John C Klensin <[email-masked]> wrote:

> This is great.   To a considerable extent, it reinforces the
> utility and importance of the IDNA2008 approach.

Absolutely!

Having the original IDNA locked with one version of Unicode is so weird I ask myself quite often how I could come up with that idea :-P

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 Oct 2013 09:46:51 -0400

--On Wednesday, October 16, 2013 10:16 +0300 Patrik Fältström
<[email-masked]> wrote:

> Yeah, I had to do it anyways -- as I am expert reviewer of the
> IANA tables. New versions of the IANA tables where created by
> IANA the other day, and I just approved them as they match my
> own calculations. I.e. we have two completely independent
> implementations of IDNA2008 (one at IANA, one that I have --
> see link below) and I compare the output of the two. If the
> output matches, then we are pretty sure we are correct.

This is great.   To a considerable extent, it reinforces the
utility and importance of the IDNA2008 approach.   YOu (and
IANA) reapply the rules to a new version of Unicode, verify that
nothing  significant has changed and your results are the same,
and we move on.

In addition, because mapping is defined on top of those rules
and the collection of allowable code points they create, we
don't end up with different profiles for different domains or
applications.

It remains to be seen whether the PRECIS approach will work as
well.  On the one hand, generation rules have replaced the
normative Stringprep table.  On the other, that system leads,
not to a definitive list but to string classes that are then
profiled, potentially on an application-by-application or even
implementation-by-implementation basis, to yield actual
practices.  Different behavior in different applications may be
inevitable but it is certain that it will confuse at least some
users.  And, because different profiles will address different
parts of the basic PRECIS code space, it is possible that the
same evaluations that are run against the generation rules for
IDNA (and exceptional adjustments added if needed) will need to
be performed for each profile, resulting in temporal
instability.    Perhaps it will all work out and the profiles
will be completely stable, but it is hard to be completely
confident about that.

best,
   john

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 17 Oct 2013 07:04:03 +0900

On 2013/10/16 22:49, Patrik Fältström wrote:
> On 16 okt 2013, at 16:46, John C Klensin<[email-masked]>  wrote:
>

I think there are simple explanations, but they are non-technical.

When we worked on IDNA2003, there were so many people asking for so many 
things (many of them rather impossible, dangerous, or otherwise of 
doubtful value, but never mind), that any idea that was able to restrict 
the solution space looked like a good idea.

Also, there seems to be a psychological value in the newest version. In 
economic terms, no difference between 10 cents and 5 cents, and 5 cents 
and free, but because of psychological reasons, people behave quite 
differently once something is free. Likewise, with versions, even though 
the differences from version 3.1 to version 3.2 and from version 3.2 to 
3.3 or 4.0 may be of the same nature or magnitude, if version 3.2 is the 
newest, that's seen as more stable and permanent that the others.

Thinking about many standardization efforts I have seen over time, there 
also seems to be a tendency to ignore versioning in the first version, 
because it's seen as the only one, and people are really concerned with 
getting the basics right. Occasionally, one sees a lot of effort being 
put into versioning and extensibility, but it usually doesn't work out 
exactly as intended.

So in summary, versioning is hard, and maybe it's easier to get right in 
the second version than in the first.

Regards,   Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 17 Oct 2013 01:47:14 +0300

On 17 okt 2013, at 01:04, Martin J. Dürst <[email-masked]> wrote:

> So in summary, versioning is hard, and maybe it's easier to get right in the second version than in the first.

If you do a second version, it better be better than the first :-P

   paf

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 24 Oct 2013 11:27:47 -0600

On 10/16/13 1:16 AM, Patrik Fältström wrote:
> Yeah, I had to do it anyways -- as I am expert reviewer of the IANA
> tables. New versions of the IANA tables where created by IANA the
> other day, and I just approved them as they match my own
> calculations. I.e. we have two completely independent implementations
> of IDNA2008 (one at IANA, one that I have -- see link below) and I
> compare the output of the two. If the output matches, then we are
> pretty sure we are correct.

Patrik, if I may ask, which implementation does IANA use?

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 24 Oct 2013 11:42:45 -0600

[ removing idna-update ]

> will be completely stable, but it is hard to be completely

Unfortunately, I have to agree. It seems to me that with IDNA2008 we
don't end up with different profiles for different applications because
there's only one application. With PRECIS we're again attempting to
generalize beyond the IDN case to a wider variety of applications, and
different applications have different requirements with regard to case
preservation and case mapping, normalization, excluded characters, etc.
It strikes me that we'll need code to test all of the PRECIS profiles,
not just the string classes. I haven't had time to keep my PrecisMaker
code up to date, let alone extend it for all of the profiles, but I will
try to work on those tasks after the Vancouver meeting.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 24 Oct 2013 20:00:18 +0200

> Patrik, if I may ask, which implementation does IANA use?

Not mine :-)

I know Marc Blanchet was involved in developing it, because he asked me some questions, but honestly I do not know. You must ask IANA.

I had my code ready (I actually have two versions of the code, ruby and python -- where the ruby code is completely written from base up while python use some external libraries for a few things) when IANA started to implement theirs. I asked them to ensure they would develop their code completely independent from mine.

What we do though is that when IANA has calculated a new version of the tables (for a new version of Unicode) I do the calculations as well with my code. I do a diff, and if there are no differences, then I say "go" to IANA and they make the new version available.

   Patrik

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 24 Oct 2013 12:13:54 -0600

On 10/24/13 12:00 PM, Patrik Fältström wrote:
>>> implementations of IDNA2008 (one at IANA, one that I have -- see
>>> link below) and I compare the output of the two. If the output
>>> matches, then we are pretty sure we are correct.
>> 
> I know Marc Blanchet was involved in developing it, because he asked
> me some questions, but honestly I do not know. You must ask IANA.
> 
> I had my code ready (I actually have two versions of the code, ruby
> and python -- where the ruby code is completely written from base up
> while python use some external libraries for a few things) when IANA
> started to implement theirs. I asked them to ensure they would
> develop their code completely independent from mine.
> 
> What we do though is that when IANA has calculated a new version of
> the tables (for a new version of Unicode) I do the calculations as
> well with my code. I do a diff, and if there are no differences, then
> I say "go" to IANA and they make the new version available.
> 
> Patrik
> 

Thanks, that's helpful. We'll need to establish a similar workflow for
PRECIS.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: John C Klensin <[email-masked]>
Date: Thu, 24 Oct 2013 15:42:32 -0400

(changing return address in the hope of avoiding filtering)

--On Thursday, October 24, 2013 11:42 -0600 Peter Saint-Andre
<[email-masked]> wrote:

>...
>> practices.  Different behavior in different applications may
>> be inevitable but it is certain that it will confuse at least
>> some users.  And, because different profiles will address
>> different parts of the basic PRECIS code space, it is
>> possible that the same evaluations that are run against the
>> generation rules for IDNA (and exceptional adjustments added
>> if needed) will need to be performed for each profile,
>> resulting in temporal instability.    Perhaps it will all
>> work out and the profiles will be completely stable, but it
>> is hard to be completely confident about that.
> 
> Unfortunately, I have to agree. It seems to me that with
> IDNA2008 we don't end up with different profiles for different
> applications because there's only one application.

Yes, although it wasn't for lack of trying by people who,
instead, wanted different profiles for different scripts and/or
languages.

> With PRECIS
> we're again attempting to generalize beyond the IDN case to a
> wider variety of applications, and different applications have
> different requirements with regard to case preservation and
> case mapping, normalization, excluded characters, etc. It
> strikes me that we'll need code to test all of the PRECIS
> profiles, not just the string classes. I haven't had time to
> keep my PrecisMaker code up to date, let alone extend it for
> all of the profiles, but I will try to work on those tasks
> after the Vancouver meeting.

I completely agree, but was actually trying to make a slightly
different point.  It is certainly the case that different
applications have evolved differently and developed different
ways of dealing with non-ASCII characters.  Characterizing those
as "different requirements", except in the sense of what those
evolutionary processes came up with, is another matter.  All of
those different rules and profiles are probably ok today, while
non-ASCII strings are still relatively unusual and most of those
who are using them either understands the issues or understands
that they are wandering around on thin ice.   But, in the long
run, they are almost certain to cause a huge amount of user
confusion as those users make inferences about what should work
in application X based on what works in application Y.  

I recognize that there are real differences between, e.g., a set
of rules that can serve to maximize entropy in a password-like
application and rules for what we normally think of as system
identifiers (including, but not limited to the DNS) or user
names.  But it seems to me that, if we end up needing
per-application profiles rather than a very small number of
profiles (I'd guess four or five including the IDN one would be
too many) for application categories that can be explained to
users, we will be doing ourselves and the Internet a real
disservice.    If that means some applications will need to
undergo a difficult transition around some edge cases, better
sooner than later: if enough users and confused and cry out in
pain, it will be only a matter of time before applications start
applying their own restrictions to reduce the number of painful
surprises.

best,
   john