Subject: Re: [iucg] Unicode 7.0.0, (combining) Hamza Above, and normalization for comparison

From: JFC Morfin <[email-masked]>
Date: Wed, 06 Aug 2014 14:01:29 +0200

At 07:03 06/08/2014, Patrik Fältström wrote:
>To be honest, I do not think it matters where it is discussed.

I suggest we keep it discussed here. The reason why is the ICANN 
response to the plaintiffs in the .ir, etc. case. "the DNS provides a 
human interface to the internet protocol addressing system". This 
seems to be a good definition to commonly sustain as it is 
technically true, easy to understand, and makes a clear distinction 
between the human and the non-human issues.

The most complex issue of the human confusability of the ISO 10646 
code points calls for a visual to binary anti-phishing algorithm. 
Such an algorithm should be added to the idna table allowing 
registries to accept xn-- registrations or not, based upon the domain 
names already registered.

To start the debate on this issue I would suggest a possibilty for 
such an algorithm: a mathematical proximity confusability 
discrimination between character 32x32 rasterizations (i.e. 1024 bits 
structured strings). I note that this also implies a common font of 
reference: I do not think this is a problem as it is on the human 
side and that conflicts will be subject to courts: what counts is the 
font local law will consider. Up to each ccTLD to provide that 
information and to have it added to ISO 3106, which already includes 
the administrative languages we should get renamed anyway as 
standardization languages coupled with the accepted script(s).

Initial question:
1. what is the URL of the complete Unicode code point table value/description?
2. I found rasterisations made for different scripts but not for all.

jfc

---

From: John C Klensin <[email-masked]>
Date: Thu, 07 Aug 2014 12:09:40 -0400

--On Wednesday, August 06, 2014 14:01 +0200 JFC Morfin
<[email-masked]> wrote:

> I suggest we keep it discussed here. The reason why is the
> ICANN response to the plaintiffs in the .ir, etc. case. "the
> DNS provides a human interface to the internet protocol
> addressing system". This seems to be a good definition to
> commonly sustain as it is technically true, easy to
> understand, and makes a clear distinction between the human
> and the non-human issues.
>...

Jefsey,

I am not sure I understand what you are talking about but, if I
do, it is about an almost completely different topic.  

The actual issue here is extremely narrow and not associated
with subjective (or computable via a distance function) visual
conformability at all.  The issue also applies to a very small
(compared to the total Unicode repertoire) number of characters.
It also involves the relationship between characters/ code
points within a single script, while most conversations about
confusability have related to characters (more specifically,
code point sequences) within a given script.

In the hope that we can at least all be talking about the same
thing, let me try to summarize the issue with the hope that Mark
and I can at least agree about the summary.  At the risk of
being harsh, while I think more informed discussion of the
issues would be helpful, getting to an informed discussion
requires some serious effort to understand the Unicode Standard,
its construction, and its specific treatment of Arabic.  The
explanation below may be helpful to those who have at least a
large fraction of that understanding; those who don't have that
much would be, IMO, well-advised to do some serious reading
before trying to participate.

In some situations, the Hamza combining character is used with a
base character as a pronunciation indicator.  I'm told that, for
the "BEH" base character and a few others, the most common such
use is when Arabic (language) words are written in a
Perso-Arabic context or similar writing system environments.
Hamza is less often used this was for writing contemporary
Arabic language in an Arabic language context, but that usage
has changed over time.   That usage as a pronunciation indicator
has been supported in Unicode for years and years by a combining
sequence using the base character and Hamza Above (a combining
character).

In the lead-up to Unicode 7.0.0, the Unicode Consortium
apparently got a request to include some characters that are
needed for a North African language that is sometimes written in
Arabic script.  While it looks just like the BEH WITH HAMZA
ABOVE combining sequence (and Unicode even decided to give it
that name), it is really a conceptually separate character.   

I think there is no disagreement up to that point, including
about the abstract form of the combining sequence looking "just
like" the newly-assigned character.   Again, this is within the
same script and there are no issues about what is confusingly
similar and what is not.

There are several (I think equivalent) versions of where the
disagreement sets in, but let me try what seems today to be the
most clear.

Section 2.2 of The Unicode Standard seems to be quite clear that
coding in the standard is independent of language,  and similar
considerations (see, in particular, the subsection titled
"Unification").   Some of us who read that believe that a new
code point for this letter should not be assigned at all and
that, if it is, it should be subject to other rules that
decompose it back into the combining sequence.  However, section
2.2 makes clear that there are multiple considerations or
"design principles" (it lists ten of them), that it may not be
possible to apply them all and get a consistent result in a
particular case, and that it is necessary to strike a balance.
Presumably on the basis of that balance, and with the precedent
of at least three other characters in the Arabic script that are
also distinct characters for some languages but combining
sequences (if used at all) for others, a new code point, U+08A1,
was assigned without a decomposition back to the existing
combining sequence.

Mark (and others associated with the decision) cite the language
issues, the precedents in the other Arabic characters, and so
on.   Those of us who aren't enthused about the new character at
all but who are really concerned about two code point sequences
that, once language identification or considerations are
removed, yield the identical characters are very concerned that
normalization doesn't create an equality relationship.  

IMO, that can't be "resolved" or consensus reached because the
criteria for making the decision are different and lead to
different results.  There are still parts of the criteria that
Unicode is applying that confuse me (with frequent examples
about use of Latin script among European languages and even some
Latin characters being used to denote rather different phonemes
in Hanyu Pinjin than they denote in Western Europe as examples
of the confusion).  I'd like to be wrong.   But, if I'm not,
then we have a mechanism in IDNA for dealing with newly-added
Unicode code points that are problematic for IDNA.   It seems to
me that there is little question that this new character (and at
least its three predecessors) are problematic for IDNA (Mark may
disagree, but I don't think he has said that yet).  The question
then becomes whether the damage that would be done by just
accepting the Unicode decision and allowing U+08A1 to be PVALID
would be greater or less than the potential damage from
excluding it or writing special rules, rules that might, in some
ways, parallel the discussion of Hamza in the "Arabic" section
of The Unicode Standard.

Again, more general discussion about confusable characters,
especially between scripts, are relevant, but not to this thread
and discussion and maybe not in/to this WG or the IETF.  As an
aside, if you dig back through the literature in optical (or
equivalent) character recognition in the pre-Kurzweil period,
you will find quite a bit about abstract character properties
and distance functions that might explain why the latter never
worked very well even with a character repertoire limited to a
single language.  Some things have clearly changed in more than
a half-century, but a lot has not.

best,
    john

---

From: Jefsey <[email-masked]>
Date: Thu, 07 Aug 2014 20:07:21 +0200

>do, it is about an almost completely different topic.

John,

as you may remember I am not interested in Unicode per se. I am 
interested in the open pragmatic use of the digisphere through 
whatever is available and people want to use. This calls for external 
(fringe to fringe) innovations not to be constrained by any internal 
(end to end) MUST. As far as I undedrstand, the issue being raised 
concerns orthotypography in a  specific language?

This has been ruled out of the IDNA2008 scope. Because it was ruled 
out the Unicode scope.

Now, what may have to be clarified is what a user calls confusable. 
For users, "confusables" are different codepoint sequences that look 
the same (whatever the reason). If the Hamza added sequences are not 
creating internet use confusion, we are not concerned. If they are, 
we are concerned. However, we MUST not decide for others: we only are 
to give them the possibility to decide by themselves.

A digital name supports a semantic address through group of visual 
signs. Whatever the underlying code, version, etc. For the time being 
the IETF has chosen a single underlying typographic code to support 
the digital names' signs. That code does not consider orthotypography 
(i.e. semantic constraints that are language dependant). So the IETF 
has chosed that its end to end protocol are not concerned by 
orthotypographic issues. Tomorrow we can chose an additional code to 
Unicode: we need the use of these two codes to be transparent to our 
own use, independently from any orthotypographic issue. This is the 
metric of our choice.

1. TLD Managers must be able to use their own add-ons to support or 
not orthotypographic aspects in their zone. How do you know if you 
will not create conflicts today with Hamza, or in "equivalent" other cases?
2. If you consider Hamza you must consider French majucules. Or am I wrong?

Best.
jfc

>At 19:37 07/08/2014, John C Klensin wrote:
>p.s. RFC 5895 was never intended to be standards track.  Talking 
>about "denying" that status to it is misleading at be

Short memories. Anyway, the point is that without Paul's and Pete's 
solution I would never have given up the orthotypographic requirement 
and IDNA would have been technically and politically blocked. I 
consider that RFC 5895 is one of the most fundamental architectural 
RFCs of the internet as it illustrates how the internet architecture 
survives external diversity, by subsidiarity at the fringe. This is 
where French majuscules can be treated (layer six experimentation).

---

From: John C Klensin <[email-masked]>
Date: Thu, 07 Aug 2014 16:29:11 -0400

--On Thursday, August 07, 2014 20:07 +0200 Jefsey
<[email-masked]> wrote:

>> I am not sure I understand what you are talking about but, if
>> I do, it is about an almost completely different topic.
> 
> John,
> 
> as you may remember I am not interested in Unicode per se. I
> am interested in the open pragmatic use of the digisphere
> through whatever is available and people want to use. This
> calls for external (fringe to fringe) innovations not to be
> constrained by any internal (end to end) MUST. As far as I
> undedrstand, the issue being raised concerns orthotypography
> in a  specific language?

Actually, perhaps the opposite because one view of the problem
is a desire to keep the IDN uses of Unicode language-independent.

> This has been ruled out of the IDNA2008 scope. Because it was
> ruled out the Unicode scope.

Yes.  And that makes it out of scope for this list.  not just
the Unicode 7.0.0 topic thread.  

For whatever it is worth, the Internet is rapidly moving in a
direction in which there will be two kinds of Web browsers:
those that work (more or less exclusively) with Unicode encoded
in UTF-8 and those that don't work with contemporary tools and
facilities.  That set of issues is obviously much broader than
IDNs.   If you feel a need to support or rely on non-Unicode
interfaces you should probably be standing in front that that
juggernaut, not trying to fine-tune IDNA edge cases.

> Now, what may have to be clarified is what a user calls
> confusable. For users, "confusables" are different codepoint
> sequences that look the same (whatever the reason). If the
> Hamza added sequences are not creating internet use confusion,
> we are not concerned.   If they are, we are concerned. However,
> we MUST not decide for others: we only are to give them the
> possibility to decide by themselves.

Then you are discussing yet another problem.   People who survey
the written forms of languages indicate that the language in
which the particular character in question is written (variously
known as Fula,  Fulan, Peul, and other names) is almost always
written in this century in Latin script.  I haven't even tried
to do the research but a guess from my recollections about the
history of the area is that Latin script started to predominate
about the time of the beginning of French presence in Northern
Africa.  The Unicode discussion (and various online articles
that may or may not be independent) say that the language is
still written in Arabic script by "Islamists" (a category whose
definition one can guess at but cannot be certain of its meaning
in this context).   The first question is whether that community
is likely to be registering and using domain names that use this
particular character.  I have no way to guess the answer to that
question but want to stress that this character is a non-issue
for anyone who reads and writes the Fula language exclusively in
Latin characters (and may be less of an issue from those who
don't read or write, and maybe haven't heard of, that language).

The second part is more complicated.  Even in the Unicode
context as of Version 6.3 and earlier, one can create something
that looks just like this character by using BEH and Combining
Hamza Above.  That is clearly inconvenient and annoying, in the
same sense that it would be inconvenient and annoying to have to
figure out a way to enter U+0063 and then U+0327 any time you
wanted to write "ç" (U+00E7) (I'm assuming a French-relevant
example will work better for you than the usual Swedish one).  

That brings us to the crux of this rather subtle problem.   If
you (or the users you are concerned about) think of "ç" as a
special and unique latter, rather than as a decorated "c", then,
following the Fula example that leads to U+08A1, then that
letter should not compare equal, even after normalization, to
the U+0063 U+0327 combination and, unless the latter is
meaningful in some language that does not consider the
combination a letter, the combination should not be allowed at
all... any more than we would expect "w" to compare equal to the
sequence "vv" or "uu".   But, if both U+00E7 and the combining
sequence are allowed (as is the case today) and both are
normalized using the same method, the results will compare
equal... not because they look alike, but because they are the
same character.  Moreover, the IDNA requirement that all
A-labels be in NFC form (which doesn't affect your ordinary
text) effectively makes it impossible to incorporate the
combining sequence into the DNS, so no potential for two
representations of the same character that do not compare equal. 

However, because of the distinction that was explained earlier,
and perhaps because Arabic script is subject to different rules
in practice than Latin script, the intent (or at least the
effect of letting the existing rules work without introducing an
exception) is that the DNS be able to accommodate both ARABIC
BEH WITH HAMZA ABOVE as a single, atomic, character coded as
U+08A1 and ARABIC BEH with HAMZA ABOVE as a two-character
combining sequence with identical appearance and without the two
comparing equal after normalization.  

_That_ is the present issue.   

Now, in a world in which one were not using Unicode but instead
used, e.g., national language code pages, both the French and
Fula cases above are entirely non-issues.   In ISO 8859-1 and
its descendants, "ç" appears as a single character, there is no
combining cedella, and it makes no difference whether that ISO
8859-1 character is mapped to Unicode by using the single code
point or the combining sequence as long as one is consistent
about it.  Similarly, if one created a seven or eight bit code
page for Fula written in extended Arabic script, the combining
Hamza Above would probably not be included, the single code
point probably would, and it would make no difference how the
character was mapped to Unicode as long as one was consistent
about it.

The only problem with the above paragraph is that it is
questionable whether continued use of ISO 8859-1 is viable today
much less whether there would be international acceptance of new
code pages.  Whether it is viable today or not, it is clear that
it is getting rapidly less plausible no matter how much you (or
others) might wish for it.

> A digital name supports a semantic address through group of
> visual signs. Whatever the underlying code, version, etc. For
> the time being the IETF has chosen a single underlying
> typographic code to support the digital names' signs.

This may be a vocabulary issue, but I would associate the term
"typographic code" with the combination of some sort of
reference to a character repertoire (a coded character set,
encoding, and code point would be one such reference, but
definitely not the only possibility) with a reference to a type
style or type family or member thereof (or, less precisely, a
"font").   The IETF has _never_ made such a choice.

>  That
> code does not consider orthotypography (i.e. semantic
> constraints that are language dependant). So the IETF has
> chosed that its end to end protocol are not concerned by
> orthotypographic issues. Tomorrow we can chose an additional
> code to Unicode: we need the use of these two codes to be
> transparent to our own use, independently from any
> orthotypographic issue. This is the metric of our choice.

I think that, in practice, the window on your making and using
that choice is closing rapidly if it has not closed already.
See above.  The only constraint existing IETF work is going to
impose on you is that there be an orderly and consistent mapping
between whatever you decide to use and the Unicode repertoire.
If there is not, you will find that you are introducing your own
form of confusion, one that will be very hard for you users to
understand.   However, if you want to create or adopt a
language-specific or even typography-specific coding, don't let
me discourage you from trying as long as you don't expect most
people on this list (or even me) to discuss it with you.

> 1. TLD Managers must be able to use their own add-ons to
> support or not orthotypographic aspects in their zone. How do
> you know if you will not create conflicts today with Hamza, or
> in "equivalent" other cases?

Such TLD managers would face at least two problems: One would be
getting equivalent plug-ins into all of the browsers that
potential users of the TLD (or URIs that include it) might use
or reference.  Otherwise, they would expose users to wildly
inconsistent behavior, which is generally not a good idea.
Reducing the chance of that sort of confusion is one of many
things driving the "UTF-8 only" movement.  The second is that
the TLD manager has very little control over the TLD
environment.  If the plug-ins merely change the appearance of
characters to make them a little more attractive, that might be
a non-problem.  But, if they create distinctions that don't
exist elsewhere or map some characters into others, there is a
lot of potential for very bad things to happen.   In particular,
the propagation of such plugins would create a wonderful
opportunity for people with malicious intent because, at least
conceptually, any plug in that can alter the form of a domain
name or URI can alter it all the way to something completely
different.

> 2. If you consider Hamza you must consider French majucules.

You are mostly wrong because the issue with French majuscules is
that, if you had a CCS that distinguished them from conventional
upper and lower case letters, information would be lost in
mapping that CCS to Unicode, causing the "Consistency" condition
mentioned above to fail.  There is no loss of information in
this Hamza case because the mappings to and from Unicode are
trivial (because the discussion is entirely about Unicode) and,
as discussed above, any specialized code page would behave in a
consistent and predicable way.   Interestingly --and part of the
challenge the Unicoce Consortium faces-- if normalization
brought U+08A1 together with the combining sequence but the
"separate character" argument continued to hold, the
normalization process would lose the information about whether
the original was that "separate character" or the combining
sequence.  But that situation is much like the counterfactual
c-with-cedilla case mentioned above, not anything to do with
majuscules.

And I suspect this list has now had enough of this discussion.

best,
   john