Subject: Re: [Idna-update] Expiration impending: <draft-klensin-idna-rfc5891bis-01.txt>

From: John C Klensin <[email-masked]>
Date: Thu, 08 Mar 2018 12:11:48 -0500

One more thought about something it may be helpful to remind
ourselves periodically.

One of the notions floated when IDNs were first being discussed
and raised a few times after that was to simply ban combining
forms of all types.  The theory was that there was no
entitlement to write any character form (e.g., any "word") in
DNS labels, that those labels were all about mnemonics and
nothing but mnemonics, and that a "no combining sequences" rule
would eliminate most issues about normalization and grapheme
clusters and make everything a lot easier to explain to
implementers and others who wanted to conform but were not
willing to make the investment to actually understand all of the
complex issues and rules with which we are now dealing.  

If that were the rule and someone really, really, wanted a
grapheme that could only be formed in Unicode with a combining
sequence, it would be up to them to convince the Unicode
Consortium that their favorite character (grapheme) needed to be
added to Unicode as a single code point.   However hard that
might be, it would not be our problem.

FWIW, a "nothing but precombined characters" rule is essentially
the recommendation for Arabic IDNs in RFC 5564 and, I
understand, in the emerging Arabic script rules for the root
zone.

We didn't go down that path, not only because of impassioned
pleas for some of the character forms that might be excluded but
because of precisely the reassurance that arguably led to the
non-decomposing characters thread -- assurances that no new code
points would be added to Unicode if there were already a
combining sequence that could reasonably substitute for it in
the same script except under very unusual circumstances and,
when those circumstances occurred, the new code points would
decompose to those sequences.

I don't suggest that we try to reverse that decision at this
point.   I assume that, if nothing else, it would just be too
disruptive.  However, it is worth pointing out that a "no
combining sequences" rule would eliminate the non-decomposing
character problem and at least a few other potential spoofing
and related cases.   It might also be worth examining as a
guideline or advice for registries who are interesting in
raising the safety level of what they allow to be registered
without having to understand the underlying issues more deeply.

best,
    john

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 8 Mar 2018 12:47:04 -0500

> might be, it would not be our problem.

I thought the argument at the time was not just that it was safe, but
that some characters (in conceptual sense that Unicode uses it) were
necessarily made of combining characters.  This was also I guess the
reason that the two joiners ended up in under context rules, I think,
yes?

Moreover, there was the principle that we didn't want to make
restrictions low in the tree even if we thought they were mostly a bad
idea near the top (this was literally the reasoning for including
hieroglyphs as I recall).

A

-- 
Andrew Sullivan
[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Mar 2018 14:11:21 -0500

--On Thursday, March 8, 2018 12:47 -0500 Andrew Sullivan
<[email-masked]> wrote:

>> Consortium that their favorite character (grapheme) needed to
>> be added to Unicode as a single code point.   However hard
>> that might be, it would not be our problem.
> 
> I thought the argument at the time was not just that it was
> safe, but that some characters (in conceptual sense that
> Unicode uses it) were necessarily made of combining
> characters.  This was also I guess the reason that the two
> joiners ended up in under context rules, I think, yes?

Yes.  The joiners normally (and pre-emoji) had more to do with
what would ordinarily be considered phrase or sentence
construction than with character formation, but, yes, they were
included because of strong arguments that the use of some
scripts (at least with some languages) was impossible without
them.  The distinction is, IMO, worth making only because it
would have been plausible to prohibit combining sequences and
allow ZWJ and ZWNJ (with or without contextual conditions)anyway
== really separate issues.

> restrictions low in the tree even if we thought they were
> mostly a bad idea near the top (this was literally the
> reasoning for including hieroglyphs as I recall).

Certainly it was a key part of the reasoning for allowing
archaic scripts.  Whether hieroglyphs were the critical issue or
merely a common example probably depends on who you ask.

I should stress that I'm not at all convinced that we made the
wrong decision about combining sequences, nor that we would have
made a different decision had we had access to better and more
accurate information.   I would feel better now if we could look
back at the decision and its consequences and say "there was an
informed discussion based on full information and we decided to
do it and accept the consequential risks" -- just as we did
with, e.g., the characters that were classified as valid (PVALID
or CONTEXTx) in IDNA2008 but mapped to something or discarded in
IDNA2003.

On the other hand, the same kind of thinking  that argues for a
"troublesome characters" list (at least as I've understood it)
-- more "you SHOULD NOT use these unless you are extra-sure you
know what you are doing and what the implications might be" than
any sort of absolute prohibition -- might imply a similar
statement about combining sequences.  In other words, whether
they are prohibited by the requirement that putative labels be
in NFC form or not (and many are), one should not be using them
without really good reason.  Not a protocol prohibition, but an
extra layer of protection against edge cases and some
potentially-malicious behavior.

best,
    john

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 8 Mar 2018 14:37:44 -0500

On Thu, Mar 08, 2018 at 02:11:21PM -0500, John C Klensin wrote:
> statement about combining sequences.

That actually doesn't seem like a terrible idea to me.  Of course,
that was _supposed_ to be the point of the general exhortation to have
a clue about the code points you were putting in a zone.  And that
doesn't seem to have panned out :-/

A
-- 
Andrew Sullivan
[email-masked]

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 8 Mar 2018 12:02:52 -0800

> complex issues and rules with which we are now dealing.

It would also very  nicely prevent IDNs on the entire Indian Subcontinent.

Ironically, it is Arabic, where most (all) of the combining can safely be
excluded from IDNs. This is being done for the Root Zone, for example,
as you acknowledge below.

The reason is that in Arabic, these marks are generally optional and/or
used for specific purposes in specialized text. Therefore, leaving them
out is not detrimental to the usability of IDNs and the Root Zone will
not allow them (extending the set of prohibited ones from RFC 5564)

Therefore, in the Root Zone the Unicode 7 addition of the U+08A1 would
not be an issue.
> might be, it would not be our problem.

Well, we see what happens if someone does get Unicode to add a pre-
composed form: the entire process is derailed. That's what happened
with the addition of U+08A1 even though it is NOT the case that this is
a true "precomposed" form - while the same graphical elements are
involved, the result does not look identical. There is a strong similarity
of course, but despite what people read into the Unicode character
name, this is not a case of an exact homoglyph.
> understand, in the emerging Arabic script rules for the root

That is because of the way these function in Arabic.

Unicode could not generally add precomposed forms of, say, Latin code
points, say a new letter with a dot above, because of normalization
stability. In the Latin script, a combining dot above and a precomposed
dot above are identical.

However, even there you have a number of combining marks that are
not considered as part of possible decompositions: they are the code
points for various (stroke) overlays and some attached extenders.

Like the Arabic combining marks, they could (and should) be disallowed
from LGRs. (The Latin LGR for the Root Zone will not allow combining
marks other than in enumerated combinations - that's something that
works for Latin, Greek, Cyrillic and practically all scripts that are not
South or South East Asian complex scripts.
> decompose to those sequences.

Other participants in that discussion remember this claim differently.
Unicode Normalization forms C and D were never about "reasonable
substitutions" but about "exact equivalents" or "the same thing,
except for the encoding".

As there is generally no benefit in encoding another representation of
"the same thing", Unicode does not allow addition of precomposed
code points that can be decomposed into something that is the
exact equivalent.

> without having to understand the underlying issues more deeply.

A useful set of recommendation for handling combining marks safely in LGRs
would consist of:

1) in all non-complex scripts: allow only fixed enumerations of base 
code point
and combining marks. (The number of required combinations is small, even
for a sprawling script like Latin).

2) in all complex scripts (where the number of combinations is too large),
provide context rules that assure combining marks are not placed in the
wrong part of a syllable (such wrong contexts cannot be "read" by humans
and not rendered correctly by machines). The Root LGR presents suitable
examples of this for SEA scripts, Indic scripts in preparation.

3) in scripts where combining marks express optional elements (vowels, etc.)
disallow all of them. (Arabic, see Root Zone LGR for example)

4) in scripts where combining marks are used for historical/ special 
purposes
disallow those (diacritics for classical Greek, stroke overlays and 
other marks
for linguistics).

5) in LGRs supporting variants, consider mutually blocking labels that vary
only in the presence or absence of some combining diacritic; some 
diacritics
are not reliably distinguished from each other (comma below, cedilla) or 
from
an undecorated base character (forms with/without Nukta).

What is a complex script: a good approximation is whether it contains a code
point with ccc=virama, plus Thaana, which isn't really complex but has 
mandatory
vowels classed as combining marks. Effectively all South and South East 
Asian
scripts that are abugidas.

In addition, you will want recommendations that actually address the 
homoglyph
issues: there are a fair number of non-combining mark exact visual 
duplicates in
Unicode. These are not exact equivalents, because Unicode does not treat
code points that differ in script, case or digit/letter properties as 
equivalent.

A./

>
> best,
>      john
>
>
>
> _______________________________________________
> IDNA-UPDATE mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idna-update
>

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 8 Mar 2018 12:26:45 -0800

> doesn't seem to have panned out :-/

A general exhortation ("know what you are doing") is too vague to be 
helpful.
Something on the level of what I just sent out as "recommendations" for
safe use of combining marks is much more specific and actionable.

I think it might be close enough to something that could become part of an
enforceable policy.

A./

---

From: John C Klensin <[email-masked]>
Date: Fri, 09 Mar 2018 11:27:25 -0500

Asmus,

Thanks for the more detailed explanation.  I don't think we
really disagree.   Three observations:

* The Unicode Standard is very clear that code points are
assigned to abstract characters and not to particular fonts,
type styles, or type families.  While it specifies rendering
rules for particular scripts (and, if I recall without going
back and checking, sometimes even particular languages0, those
rules are much more readily applied to actual blocks of text
than to relatively short mnemonic strings that may not obey the
word or phrase formation rules of the writing system for some
human language.  We've also seen applications and
implementations that are trying to do as little as possible (or,
stated a bit more positively, to incorporate as little code that
is exercised only for very specific cases as possible) and hence
are unlikely to carefully apply rendering rules to strings even
if they support the fonts needed to display the code points in
those strings.  There are also applications and formats out
there that allow incorporating fonts of the author's choice,
rather than picking from those we, or a different application
implementer, might consider optional. Because of those factors
and because users will often see what they expect to see, I
think we need to be very careful when making statements about
what might or might not look like something else.  

* In retrospect, we missed an important opportunity when the
"preferred syntax" rules were written into RFC 1034 and its
predecessors.  It would have been possible at some time decades
ago to specify a rule for combining words for the purpose
mnemonics to form DNS labels.  porky-pig or porkypig, usc-isi or
uscisi, etc.   For those old enough to remember the host table
and NIC-approved names, we actually had such a convention, but
it was, at least as far as I can remember, never written down...
and it certainly did not make it to RFC 1034.   So we now have
internetsociety.org and not internet-society.org, but, if one
wants to have a discussion about confusion, remember which of
those two labels is being used is at least as confusing to a
typical user as any of the more subtle character or
character-representation issues we have been discussing (if
there is anyone reading this with an inclination toward evil or
chaos, it might be interesting to see if internet-society.org
could be registered and, if so, whether it would cause any
interesting havoc).   Noting that, like so many other things,
IDNs don't create new problems but may make old ones more
complex, I observe that, AFAIK, ICANN has never seriously
discussed a guideline prohibiting having both porky-pig and
porkyig as labels in the same zone, a situation that adds to my
skepticism about blocked variants as a major part of any
solution.  All of that aside, had there been a discussion in RFC
1034 about the relationship between, e.g., "xerox-copier" and
"xeroxcopier", it certainly would have been reflected in the
extrapolations that generated most of the rules of IDNA2008.

* Your explanation below is very helpful, at least to me.  I am
confident that, if it was what we were given when we started the
IDNA2008 design, some of the rules would have reflected it.   We
were given a different explanation and answer to our questions,
one much closer to "there is no problem and will be no problem
in  the future, so don't worry - no special rules are needed".  

The question is what to do now.  If we were to decide that your
discussion would made a good addition to the protocol, we would
run into at least two problems.  One is that we promised the
registry community that there would be no more disruptive
changes and, at least as important, the Unicode Consortium
outrage at our altering the way a handful of code points were
treated would presumably be much greater if we were to make a
change that would invalidate a large number of labels that might
now be registered and in use somewhere in the tree.  Another is
the problem that started this thread -- there appears to be no
energy in the IETF to consider and process changes to IDNA, even
fairly trivial clarifications.

See my next note for a different aspect of this.

best,
   john

--On Thursday, March 8, 2018 12:02 -0800 Asmus Freytag
<[email-masked]> wrote:

>> One of the notions floated when IDNs were first being
>> discussed and raised a few times after that was to simply ban
>> combining forms of all types.  The theory was that there was
>> no entitlement to write any character form (e.g., any "word")
>> in DNS labels, that those labels were all about mnemonics and
>> nothing but mnemonics, and that a "no combining sequences"
>> rule would eliminate most issues about normalization and
>> grapheme clusters and make everything a lot easier to explain
>> to implementers and others who wanted to conform but were not
>> willing to make the investment to actually understand all of
>> the complex issues and rules with which we are now dealing.
> 
>> FWIW, a "nothing but precombined characters" rule is
>> essentially the recommendation for Arabic IDNs in RFC 5564
>> and, I understand, in the emerging Arabic script rules for
>> the root zone.
> 
>> pleas for some of the character forms that might be excluded
>> but because of precisely the reassurance that arguably led to
>> the non-decomposing characters thread -- assurances that no
>> new code points would be added to Unicode if there were
>> already a combining sequence that could reasonably substitute
>> for it in the same script except under very unusual
>> circumstances and, when those circumstances occurred, the new
>> code points would decompose to those sequences.
> 
> 1) in all non-complex scripts: allow only fixed enumerations
> of base code point
> 4) in scripts where combining marks are used for historical/
> special purposes
> disallow those (diacritics for classical Greek, stroke
> overlays and other marks
> for linguistics).
> 
> only in the presence or absence of some combining diacritic;
> some diacritics
> are not reliably distinguished from each other (comma below,
> cedilla) or from
> point with ccc=virama, plus Thaana, which isn't really complex
> but has mandatory
> vowels classed as combining marks. Effectively all South and
> South East Asian
> In addition, you will want recommendations that actually
> address the homoglyph
> issues: there are a fair number of non-combining mark exact
> visual duplicates in
> code points that differ in script, case or digit/letter
> properties as equivalent.
> 
> A./
> 
> 
>> 
>> best,
>>      john
>> 
>> 
>>

---

From: John C Klensin <[email-masked]>
Date: Fri, 09 Mar 2018 12:13:06 -0500

--On Thursday, March 8, 2018 14:37 -0500 Andrew Sullivan
<[email-masked]> wrote:

>> On the other hand, the same kind of thinking  that argues for
>> a "troublesome characters" list (at least as I've understood
>> it) -- more "you SHOULD NOT use these unless you are
>> extra-sure you know what you are doing and what the
>> implications might be" than any sort of absolute prohibition
>> -- might imply a similar statement about combining sequences.
> 
> That actually doesn't seem like a terrible idea to me.  Of
> course, that was _supposed_ to be the point of the general
> exhortation to have a clue about the code points you were
> putting in a zone.  And that doesn't seem to have panned out
> :-/

Well, and this may apply to Asmus's recent notes (including a
more nuanced view of combining sequences) too, I'm trying to
think in terms of, and maybe adjust, a distinction that goes
back to at least the earliest discussions of IDNs and IDN
policies.   The distinction is arguably closely related to the
difference between RFCs 1034/1035 and a number of subsequent
"this is how we are actually going to do this" statements, of
which RFC 1591 may be a high point.   The version that might be
relevant today is that there are three (or four) levels or
layers of IDNA-related rules affecting what domain names can be
registered:

(0) Nothing gets to violate the fundamental architecture of the
DNS, e.g., one doesn't get to have two labels that compare equal
in the same zone and it really is a tree.   By and large, we
expect this set of rules to be self-enforcing: it is hard to
even think about ways to break those rules and still have
anything work (although we've seen some speculation and wishful
thinking about aliases that would interleave branches of the
tree and that would therefore come close).

(1) The IDNA protocol collection specifies what is and is not
allowed and what operations work and how, and what is and is not
valid, at a fundamental level.  That isn't just
registration-time requirements.  IDNA2008 specifies checks to be
done at lookup time and we expect systems looking up names to
reject at least a subset of invalid strings before looking them
up.   There is some implementation flexibility there: while we
don't require, e.g., contextual checks at lookup time, I don't
imagine anyone would severely criticize an implementation for
rejecting a string outright that didn't conform to those rules
because, if lookup succeeded, the application would be safe in
concluding that either something was broken or that it was
likely that something malicious was going on.  That is true even
of IDNA2003, which effectively has no lookup-time requirements:
to take an extreme example, if an implementation were given
something in ACE (aka presumably "Puncode encoded") form;
decided to convert it back to native characters to be sure it
could generate native-character error messages if needed;
discovered it wouldn't convert or converted to something that
required mapping; and then generated an error message rather
than trying to look the thing up, I don't imagine anyone would
complain (other than perhaps a registrant who had bought such a
string in its Punycode encoded from -- something many registries
won't allow).  

The important thing about these protocol requirements is that no
sane person should expect the problem domain name or label to
actually work, or work predictably, for what we normally
consider DNS functions such as doing name to address
translation.  If someone were buying labels as trophies in the
hope that their trophy-value would increases over time that
would be another matter but I'm not sure that, from a protocol
standpoint, we really need to care.

(2) Advice, guidelines, or policy requirements applicable to a
broad variety of domains.  Unless we are going to have chaos,
such guidelines must allow only a subset of what is allowed
under the protocol limitations above.  For zones that are well
and carefully run, with name use and the best interests of
Internet users in mind, it is reasonable to expect that
well-designed guidelines will be followed, as least unless the
zone administrators understand things well enough to carefully
allow exceptions.   The meta-rule that zones are required to
understand what they are doing is one such guideline and, at
least to a first approximation, I think most registries
(including zones below the second level that are using IDNs)
try.   For those who are not inclined (for business or other
reasons), to follow such guidelines, the guidelines are only as
good as the existence of someone who is willing and able to
enforce them or otherwise discourage bad behavior.  

What has not panned out is the assumption that ICANN would make
at least some serious effort to either persuade or induce
registries, even second-level entities under contracted TLD
registries, to follow those guidelines, especially with
regard to taking active responsibility for what they are doing.
But that doesn't make establishing the guidelines a bad idea.
I'm not aware of it having happened yet with IDNs, but there is
precedent for people who are harmed by someone else's bad
behavior to cite violation of widely-accepted guidelines as
evidence that the behavior was bad or negligent.

(3) The rules that are used by the operator/ administrator of a
particular registry to determine what can or will be allowed in
that zone.  Noting that the rules for the root (including the
LGR rules but not limited to them, at least yet) are just a
special case of this category, all that is necessary to
application and enforcement of the rules is that the registry
decide to do so... and that the rules that are chosen be
sufficiently acceptable to whatever community(ies) are in a
position to hold the registry accountable that they don't either
turn into a source of never-ending strife or change often enough
to cause perceived instability.  

I think some of our discussions have confused the first two and
last two categories and then gone on to confuse the third
(general guidelines for the second level and below) and fourth
(registry-specific rules), assuming, for example, that what is
suitable for one registry should be applicable, either as a
guideline or as a protocol changes, to the others.

best,
   john

---

From: "Patrik Fältström" <[email-masked]>
Date: Sat, 10 Mar 2018 09:06:09 -0400

On 9 Mar 2018, at 12:27, John C Klensin wrote:

> * Your explanation below is very helpful, at least to me.

Me too.

> I am
> confident that, if it was what we were given when we started the IDNA2008 design, some of the rules would have reflected it.   We were given a different explanation and answer to our questions, one much closer to "there is no problem and will be no problem in  the future, so don't worry - no special rules are needed".

Correct. Being on the receiving side of the messages, and one of the persons that asked the questions, yes, we asked, they responded...

> The question is what to do now.  If we were to decide that your discussion would made a good addition to the protocol, we would run into at least two problems.  One is that we promised the registry community that there would be no more disruptive
> outrage at our altering the way a handful of code points were treated would presumably be much greater if we were to make a change that would invalidate a large number of labels that might now be registered and in use somewhere in the tree.  Another is the problem that started this thread -- there appears to be no energy in the IETF to consider and process changes to IDNA, even fairly trivial clarifications.

I think we should do some scenario planning here. Remember that we do not have a world where IDNA2008 based on Unicode 6.x is what people use. People use all different kind of mix between IDNA2008, IDNA2003 and Unicode versions. I have myself worked with the curl library (that uses libidn) and to be honest, I do not think people KNOW what they use. Or they know, and they know they violate the rules. For the contracted parties, they have the LGR coming down the road anyways, so...

And *if* we go down this path, is it "enough" to do in the LGR (i.e. ICANN) or should IETF do some adoption (and W3C?), or should IETF say "we can move forward in a more safe way AS WE KNOW ICANN DO LGR"...

Or...

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Sat, 10 Mar 2018 08:51:34 -0500

--On Saturday, March 10, 2018 09:06 -0400 Patrik Fältström
<[email-masked]> wrote:

> I think we should do some scenario planning here. Remember
> that we do not have a world where IDNA2008 based on Unicode
> 6.x is what people use. People use all different kind of mix
> between IDNA2008, IDNA2003 and Unicode versions. I have myself
> worked with the curl library (that uses libidn) and to be
> honest, I do not think people KNOW what they use. Or they
> know, and they know they violate the rules. For the contracted
> parties, they have the LGR coming down the road anyways, so...
> 
> And *if* we go down this path, is it "enough" to do in the LGR
> (i.e. ICANN) or should IETF do some adoption (and W3C?), or
> should IETF say "we can move forward in a more safe way AS WE
> KNOW ICANN DO LGR"...

As I said at far more length in another note, the LGR is (at
least by its charter/ Procedure) designed for, and applicable
only to, the root.  While those code points would probably be
safe to use in any zone, trying to apply them globally would be
unduly restrictive, would violate the "administratively
distributed hierarchy" principle, and, IMO, just wouldn't fly..
The latter would probably quickly reach the point that attempts
to apply the LGR to labels at the second level and beyond would
encourage non-compliance and daring ICANN to do anything about
it.     

As an example of the overly restrictive part, I assume that the
LGR rules, like the ccTLD Fast Track Procedures, would prohibit
the use of "ур" (U+0443 U+0440) or other permutations of those
characters) in to root on the grounds of either intrinsic
confusability or the potential conflict with the 3166-1 alpha-2
list.  However, if we accept the logic that justifies IDN TLDs,
that sequuence would be acceptable in a subtree of a domain
whose TLD label was in Cyrillic and that published and applied a
policy that its subdomains were entirely Cyrillic.

> Or...

I think the above puts us well into the "or..." range.

best,
   john

---

From: Mark Davis ☕️ <[email-masked]>
Date: Sat, 10 Mar 2018 17:24:22 +0100

>> We were given a different explanation and answer to our questions, one
much closer to "there is no problem and will be no problem in  the future,
so don't worry - no special rules are needed".
> Correct. Being on the receiving side of the messages, and one of the
persons that asked the questions, yes, we asked, they responded...

I'm mystified as to who exactly you think said "there is no problem and
will be no problem in the future" (or words to that effect), since I can't
imagine any Unicode expert saying that NFC, NFKC, or the then-proposed IDNA
2008 solves the problem of visually visually indistinguishable or confusing
characters. In fact, there are many emails on
https://mailarchive.ietf.org/arch/search/?email_list=idna-update NFKC
making it clear that NFKC reduces, but does not close to solving that
issue. And the same for the then-proposed context rules.

So I'm guessing that there was a misunderstanding as to the problem being
discussed.

Can you point to the email(s) on this list from a Unicode expert that you
would characterize as responding that "there is no problem and will be no
problem in  the future, so don't worry - no special rules are needed" ? (Of
course, it needn't be those exact words — I just want to figure out what
was the foundation for your statement.)

Mark

On Sat, Mar 10, 2018 at 2:06 PM, Patrik Fältström <[email-masked]> wrote:

> IDNA2008 design, some of the rules would have reflected it.   We were given
> a different explanation and answer to our questions, one much closer to
> "there is no problem and will be no problem in  the future, so don't worry
> - no special rules are needed".
>
> Correct. Being on the receiving side of the messages, and one of the
> persons that asked the questions, yes, we asked, they responded...
>
> discussion would made a good addition to the protocol, we would run into at
> least two problems.  One is that we promised the registry community that
> there would be no more disruptive
> > outrage at our altering the way a handful of code points were treated
> would presumably be much greater if we were to make a change that would
> invalidate a large number of labels that might now be registered and in use
> somewhere in the tree.  Another is the problem that started this thread --
> there appears to be no energy in the IETF to consider and process changes
> to IDNA, even fairly trivial clarifications.
>
> I think we should do some scenario planning here. Remember that we do not
> have a world where IDNA2008 based on Unicode 6.x is what people use. People
> use all different kind of mix between IDNA2008, IDNA2003 and Unicode
> versions. I have myself worked with the curl library (that uses libidn) and
> to be honest, I do not think people KNOW what they use. Or they know, and
> they know they violate the rules. For the contracted parties, they have the
> LGR coming down the road anyways, so...
>
> And *if* we go down this path, is it "enough" to do in the LGR (i.e.
> ICANN) or should IETF do some adoption (and W3C?), or should IETF say "we
> can move forward in a more safe way AS WE KNOW ICANN DO LGR"...
>
> Or...
>
>    Patrik
>

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Sat, 10 Mar 2018 10:08:08 -0800

Patrik asked, where do we go from here.

Let me propose something, but first some response to John.

>> parties, they have the LGR coming down the road anyways, so...

Correct, and it doesn't stop there: some parties are off the reservation 
doing emoji...

> encourage non-compliance and daring ICANN to do anything about

Totally agree. There's no benefit in "imposing" the RZ-LGR on other zones.

However, the various script LGRs can serve both as examples and as starting
points. They do contain (or are about to contain) worked examples of
repertoire context rules and or variant rules applicable to modern users
of each script. They are also heavily documented, so that anybody can
understand the "why" behind their design. That makes them useful as
a starting point as well, to which you can add features as needed. One
thing almost anyone will want to add outside the root zone is digits
(and the hyphen).

That's why I have consistently argued that the RZ-LGR is a useful example
and starting point.

> list.

The LGR does not have to "prohibit" these. Instead, they are recognized as
so-called cross-script variants of the Latin letter.

Meaning, if you have a label containing Latin "yp" in the root, then you
do not get to use Cyrillic "ур"in some other label, if the two would 
otherwise
be (or look) the same.

However, you remain free to use Cyrillic "ур" in any label that does not
collide with some Latin label, either because no homograph label exists,
or because yours has some specific Cyrillic code point in it that would
distinguish it.

The ability to use the variant mechanism for such cases is  the main
difference between using LGRs (aka idn tables) compared to the protocol
in enforcing restrictions.

The cross-script variant mechanism is a more appropriate method to address
issues like that, because it doesn't blindly ban perfectly acceptable 
uses of code
points, but instead resolves any conflicts in favor of the first mover.

This reflects linguistic reality - in most cases, cross-script labels 
that are
homographs would look more than a bit "contrived", such the licence
plate with the Russian expletive AXY HEXO that some Unicoder had for
a while - it makes no sense in English, and passed the DMV.

At one point, we did some research in one of the scripts and found
that defining even a substantial numbers of such variants had little
impact on blocking legitimate words. Had a similar number of code
points been banned, the utility of the IDN labels would have been
seriously compromised.

For that reason alone, simply banning code points is more restrictive
than necessary.

> However, if we accept the logic that justifies IDN TLDs,

There are other RZ restrictions (no digits, no hyphen) that aren't
appropriate for zones lower down the tree.

And there are a few cases where it wasn't possible to cater to
all languages in a given script simultaneously, as some had
conflicting conventions (for complex scripts).

Such cases, while rare, are another good reason to use the Root Zone
as a starting point (and not the end-point) of LGR design for other Zones.

However,  the audience of 90%+ of existing zones would have been served
equally well had their idntables and policies been based on nothing more
than the RZ-LGR plus digits and hyphen. The stuff that the Root Zone
restricts is very often simply not in use.

> I think the above puts us well into the "or..." range.

I think we've come to that conclusion before - and we keep revisiting
ground that I thought had somewhat settled. We had been working
on a two-prong approach, of which your ID was one of the prongs:

1) to reiterate that the "raw" protocol doesn't by itself design
useful and secure LGRs (aka. idn tables/policies), but requires
conscious selection (and other decisions).

2) to give guidance as to what the issues are and how to address
them.

Now, we thought that the best approach for the second prong was
to create a registry of "troublesome" code points.

That turned out to be more challenging than anticipated. Not
because it is difficult to come up with a list. That's actually not
as difficult as it looks - for modern scripts.

No, the issue is that the code points, like your example of "yp"
are not problematic in isolation. What then should get listed? If we list
just the Cyrillic code points, then people creating a Cyrillic zone
will complain loudly that the entry is unjustified. If we list both the
Latin and the Cyrillic code point, on the basis that their interaction
is problematic, we get that reaction from two groups.

However, in the RZ effort, both groups were perfectly happy
defining mutually blocked variants - in fact, we had to stop
them from going overboard. . . .

Where to we need to go?

We need some recommendation that goes beyond "know what
you are doing" (while true, it's too vague to be helpful). We
need a set of recommendations that is more or less like the
example I wrote down for the combining marks. Something
that's specific enough that people can act on it, but also can
be adjusted to fit circumstances.

In writing recommendations for (more) "secure" IDNs, it's
important to make a distinction between modern-use scripts
and the vast number of historical scripts and technical (phonetic)
notations. The latter are really not well suited to secure use
in public zones. (The protocol must cater for non-public zones
as well).

Now, one option would be to create something like a "profile"
on IDNA2008 for "(more) secure IDNs for public zone."

Doing that would sidestep the issue of backwards compatibility
because you could be more restrictive; however, there's
the same issue that any enumerated restrictions would
run into the same issues of changes in usage (spelling
reforms) or changes in encoding.

However, qualitative descriptions wouldn't suffer as much
from that issue; I think they would add value, whether
as recommendation or as "profile".

A./

---

From: "John Levine" <[email-masked]>
Date: 10 Mar 2018 13:29:07 -0500

>encourage non-compliance and daring ICANN to do anything about

I'm confused.  Every gTLD and many ccTLDS have language and script
rules that limit what 2LDs they will accept.  When I was looking at
gTLD applications, the rules all looked pretty reasonable, give or
take what you think is a reasonable approach to handling variants.

What am I missing?  I realize that below the 2LDs anything goes.

R's,
John

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Sat, 10 Mar 2018 21:57:47 -0800

I was not centrally involved in these discussions, but knowing a bit 
about the different backgrounds and frames of reference of the 
participants I would not be surprised to learn that there was a certain 
amount of miscommunication leading to some fundamental disconnect in 
understanding....

However, independent of the details of what may have been exchanged a 
decade ago, I strongly  suspect that going over that ground is going to 
do little in terms of helping to make progress forward.

Instead, the sole question we should be asking is what can we do to 
effectively improve the experience involving IDNs.

A./

On 3/10/2018 8:24 AM, Mark Davis ☕️ wrote:
> >> We were given a different explanation and answer to our questions, 
> one much closer to "there is no problem and will be no problem in  the 
> future, so don't worry - no special rules are needed".
> I'm mystified as to who exactly you think said "there is no problem 
> and will be no problem in the future" (or words to that effect), since 
> I can't imagine any Unicode expert saying that NFC, NFKC, or the 
> then-proposed IDNA 2008 solves the problem of visually visually 
> indistinguishable or confusing characters. In fact, there are many 
> emails on 
> So I'm guessing that there was a misunderstanding as to the problem 
> being discussed.
>
> Can you point to the email(s) on this list from a Unicode expertthat 
> you would characterize as responding that "there is no problem and 
> will be no problem in  the future, so don't worry - no special rules 
> are needed" ? (Of course, it needn't be those exact words — I just 
> want to figure out what was the foundation for your statement.)
>
>
> Mark
> //
>
> On Sat, Mar 10, 2018 at 2:06 PM, Patrik Fältström <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     one much closer to "there is no problem and will be no problem in 
>     the future, so don't worry - no special rules are needed".
>
>     Correct. Being on the receiving side of the messages, and one of
>     the persons that asked the questions, yes, we asked, they responded...
>
>     I think we should do some scenario planning here. Remember that we
>     do not have a world where IDNA2008 based on Unicode 6.x is what
>     people use. People use all different kind of mix between IDNA2008,
>     IDNA2003 and Unicode versions. I have myself worked with the curl
>     library (that uses libidn) and to be honest, I do not think people
>     KNOW what they use. Or they know, and they know they violate the
>     rules. For the contracted parties, they have the LGR coming down
>     the road anyways, so...
>
>     (i.e. ICANN) or should IETF do some adoption (and W3C?), or should
>     IETF say "we can move forward in a more safe way AS WE KNOW ICANN
>     DO LGR"...
>
>     Or...
>
>        Patrik
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/idna-update>
>
>

---

From: Asmus Freytag <[email-masked]>
Date: Sat, 10 Mar 2018 23:28:51 -0800

On 3/9/2018 9:13 AM, John C Klensin wrote:
> policies.
... eliding discussion of elements of the protocol and stipulating for the
remainder of the discussion that protocol and its requirements are assumed
to be met.
> broad variety of domains.

May not need to define precisely what the limits of that variety is, but 
should
note that zones where all labels are owned by the same entity may already
be restrictive in other ways, whereas zones that not only are "public" but
also support users of more than one script (and by extension multiple 
languages
are perhaps most in need of these).

> Unless we are going to have chaos,
> under the protocol limitations above.
Agreed.
> For zones that are well
> allow exceptions.

These zones will take active steps to mitigate issues caused by the fact
that IDNs are necessarily reflective of f the writing systems involved -
including their comparative complexity vis a vis the basic Latin alphabet.

>   The meta-rule that zones are required to
> try.

Not my impression at all. For anything other than very simple scripts,
the IDNtables that can be easily inspected because they are available
online as tables are not reflective of a deeper understanding of the writing
system, beyond the most basic limitation of the repertoire to a given 
script.

> For those who are not inclined (for business or other

"willing" registries might benefit from more detailed guidelines that would
give them a metric of how successful their "attempts" turned out to be.
I think at least some of them might be surprised at the result.
> regard to taking active responsibility for what they are doing.

This is an empty letter as long as no guidelines exist that establish 
meaningful
metrics allowing to tell apart successful mitigation from simply copying 
some
basically insufficient IDN table from some other zone.
> evidence that the behavior was bad or negligent.

This would be a beneficial use of guidelines that actually cover meaningful
best-practice.
> that zone.

Note that for many zones what is allowed is not static, but depends on
what is already delegated. The rules effectively describe what is allowed
next, given a status quo.

This approach is something that is useful (or should be understood as
useful) for the majority of zones: in all zones registrants compete for
available labels, but in some zones, a delegated label does not only
prevent the same label, but also certain related labels from being 
registered.

Such related labels are called variants -- but the use of the term
variants does not necessary imply that more than one label gets to
be allocated. On the contrary, our experience in Root Zone has shown
that blocked variants are by far more common and useful tool.

>   Noting that the rules for the root (including the

Again, the Root Zone project is informative.

So far, a single proposed script LGR has proven contentious -- with the 
wider
community objecting to it being too permissive, rather than the other
way around. As this has not reached the stage where an LGR has been
submitted, the community may yet come to some agreement.

Generally, the drafting panels appear to have been well anchored in their
respective communities and their submitted proposals perform well when
tested against lists of putative labels while mitigating known issues.

> guideline or as a protocol changes, to the others.

Very cogent observation.

I'm certainly keenly aware of the difference between the RZ-LGR project
which effectively results in registry-specific rules (for the root) and 
general
guidelines that can be used to define registry-specific rules on other 
levels.

The RZ-LGR is purposefully limited to the modern-use subset of modern-
use scripts. (It also includes hyphen and digits). It further makes the 
assump-
tion that the zone is shared by users of multiple scripts - and users of
"all" languages - at least those that are widely written for everyday 
purposes.

The individual script LGRs are also accompanied by very detailed 
description
and references to source materials.

That allows them to serve as examples/starting point for defining registry-
specific rules for other zones. The closer these zones match the other
assumptions for the RZ project, the closer to the RZ-LGR their eventual
rules would be expected to end up - unless the RZ-LGR erred in how to
account for the risks inherent in multi-script/multi-zone domains.

This simplifies the task of the guideline writer, without reducing the task
to simply imposing the RZ-LGR unexamined everywhere.

If I were to be asked by a registry to develop policy for the second 
level, I
might proceed as follows:

(0) Start with the RZ-LGR s for all the scripts to be supported in the 
new zone.

(1) Add all digits and the hyphen
(1a) Mitigate the issue of homoglyph digits (Arabic) by making them blocked
         in-script variants
(1b) Mitigate the issue of some digits being homglyphs of letters by making
        these blocked in-script variants

(2) Retain cross-script variants from RZ-LGR for all scripts in the new zone

(3) Optional, if a feature (restriction) in the RZ-LGR is documented as 
being
       motivated by the need to be particularly restrictive for the root,
       investigate the cost/benefit of removing the restriction. (If in 
doubt
       keep the restriction).

(4) If the zone is limited to certain languages, remove features 
(restrictions)
       documented as being necessary in a multilingual zone
(4a) investigate the cost/benefit of adding language-specific support 
for any
        language that isn't well-supported due to RZ restrictions.

(5) Make sure LGR follows guidelines on combining marks; (like the ones
       we discussed under separate cover.

(6) Make sure LGR follows guidelines on repertoire (TBD).

(7) Make sure WLE rules and context rules continue to work if repertoire
      expanded
     (7a) It may be appropriate to relax / tighten some of these rules if
      the mix of languages to be supported requires that or benefits 
from it.

and so on.

The guidelines would effectively focus on well-understood modern-use script,
because that's what is addressed by the RZ-LGR. Because (most) historical
scripts are supremely ill-understood and have not had the benefit of any
deeper analysis for IDN purposes, a general guideline would strongly
recommend against including them.

Some lesser-used and/or emerging scripts (with modern user communities)
would benefit from their communities following the RZ-LGR procedure
with appropriate adaptations.

Some very limited zones, e.g. Cyrillic only, or Polynesian only, might come
under pressure to support code points for characters that look like
punctuation marks; these code points were not ruled out in IDNA, but
are considered deeply troublesome (not least by RFC 6912) -- actual
guidelines would have to be written so as to settle the question whether
"secure" LGRs should always eschew them, or whether limited exceptions
are to be seen as justified.

The issues facing the various scripts are diverse enough that trying to 
write
fully general guidelines gets either meaningless (too vague) or 
bewilderingly
complex in no time. That's the reason I keep coming back to the use of the
RZ-LGR as a non-binding starting point.

A./

---

From: "Patrik Fältström" <[email-masked]>
Date: Sun, 11 Mar 2018 06:57:32 -0400

On 11 Mar 2018, at 1:57, Asmus Freytag (c) wrote:

> However, independent of the details of what may have been exchanged a decade ago, I strongly  suspect that going over that ground is going to do little in terms of helping to make progress forward.
>
> Instead, the sole question we should be asking is what can we do to effectively improve the experience involving IDNs.

+1

Thanks Asmus.

   paf

---

From: John C Klensin <[email-masked]>
Date: Sun, 11 Mar 2018 09:43:19 -0400

--On Saturday, 10 March, 2018 21:57 -0800 "Asmus Freytag (c)"
<[email-masked]> wrote:

>...
> However, independent of the details of what may have been
> exchanged a decade ago, I strongly  suspect that going over
> that ground is going to do little in terms of helping to make
> progress forward.
> 
> Instead, the sole question we should be asking is what can we
> do to effectively improve the experience involving IDNs.

I concur.   The history is not, however, completely irrelevant.
It can be important to avoid repeating prior mistakes and,
perhaps, to calibrate some proposals going forward.  

In particular, there is some non-technical history that may
constrain what we can consider today.  In addition to a change
of model, IDNA2008 included a few changes that invalidated some
labels that were possibly in use ("possibly" because I'm talking
about the whole DNS now, not just the root zone or second-level
names) and changed the interpretation of others.  Those changes
were controversial and arguably remain so, but there was broad
consensus in the registry community that they were worth it if
(and perhaps only if) that type of change never occurred again.
If we take that as a constraint today, a guideline that says
"don't do so-and-so unless you have a very good reason" would
still allow registries to "grandfather" existing names or make
other rules about them as needed.  A protocol change, especially
one with lookup-time implications, would not have that property.
That is one of several reasons why the three-level distinction I
tried to make in my earlier note is important.

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Sun, 11 Mar 2018 10:44:56 -0400

--On Saturday, 10 March, 2018 13:29 -0500 John Levine
<[email-masked]> wrote:

>> The latter would probably quickly reach the point that
>> attempts to apply the LGR to labels at the second level and
>> beyond would encourage non-compliance and daring ICANN to do
>> anything about it.     
> 
> I'm confused.  Every gTLD and many ccTLDS have language and
> script rules that limit what 2LDs they will accept.  When I
> was looking at gTLD applications, the rules all looked pretty
> reasonable, give or take what you think is a reasonable
> approach to handling variants.
> 

This may not be worth beleaguering, but you are missing, or,
more likely, not looking hard enough, at least four things:

(1) Some of those rules have, at least historically, been no
more restrictive than "what IDNA2008 allows" and even (although
the new Board resolutions may gradually change that too) "what
IDNA2003 allows".    And, fwiw, lists of permitted code points
have been common; actual stated/specific rules much less so.
IIR, the latter were not even provided for in the initial
request for "character" (code point and code point sequence)
lists.

(2) Those tables started out as indications of intention, and
advice to others about what they might do or think about.  They
were not even intended as constraints on what the registering
zone was going to do.  Even though it was generally assumed that
a registry would not allow labels that did not conform to the
labels it registered, that was never a requirement that anyone
expected to be enforced at least up to the time those tables got
entangled with the new gTLD program.

(3) Perhaps in part because the tables that were registered with
IANA were optional, not "rules", there is a history of names
being registered as 2LDs that do not conform to the published
tables.  The distinction becomes important the moment the term
"variant" is mentioned, even as a blocking mechanism, because
even a simple FCFS rule such as "if you have 'google' registered
as an SLD, 'goog1e' is not allowed" is a rule with which an
allowed character list won't help.    That is actually an area
in which something could, in principle, be done immediately and
without any need to sort out the IETF review issues: ICANN could
update the general guidelines with advice to block additional
registrations in a given zone that reflect that sort of
relationship, using whatever information can be gleaned from the
LGR process, Asmus's efforts, and whatever else we know and not
excluding any scripts (as the example and dozens like it
indicate, this is not just an IDN problem).   I note that
variations on this idea have been proposed, several times, by
Andrew and others --I'm not inventing anything in this note.
For those of you who are spending the next several days in
Puerto Rico, that is probably a worthy discussion topic.  Going
into that discussion, you should, however, remember that rules
that are ultimately based on FCFS --preference for first party
to register a name from a set -- have had a rough history in
ICANN, setting off either claims of unfairness or unpleasant
"land rush" behavior or both.

(4) I don't know if you are "missing" it or not, but we need to
keep in mind that efforts to promote the use of DNAME records in
the root, at the second level, and elsewhere, have potential bad
side-effects.   If I have a zone "foo" at some level of the DNS
with an associated set of rules or policies, and nodes owned by
"bar" and "foobar" with DNAME records pointing at "foo", it is
not clear whether there is a reasonable expectation the foo's
rules will apply if bar and foobar can have their own rules or
no rules at all.  It is fairly clear, at least to me, what the
rules should be if the DNS RRSETs for foo, bar, and foobar all
exist at the same level in the same zone, e.g., no arrangements
like

   $ORIGIN zork.
   foo IN ...
   foobar.baz IN DNAME foo.zork.

but, as I trust everyone reading this understands, there is no
such requirement on DNAME -- there can, in principle, be a
record with RRTYPE DNAME pointing to "foo" from any zone and any
level of the DNS tree (and, btw, with any owner/registrant).
The rather unfortunate relationship between DNAME records in
arbitrary zone and DNSSEC doesn't help much here.  

That is, again, one of those "if we had known then what we know
now" bits of history: when DNAME was first designed and
deployed, it was intended for temporary, transitional, use and
DNSSEC was not even something that people were dreaming about.
Its evolution into a variant mechanism that is intended to be
stable (even if not "permanent") suggests the need for some
additional constraints but, even if we could introduce a new
RRTYPE with those constraints, we almost certainly could not
make DDNAME go away. 

best,
   john

---

From: "John R Levine" <[email-masked]>
Date: 12 Mar 2018 11:03:54 -0400

> Even though it was generally assumed that

I've tried some experiments and it appears that gTLDs now enforce the 
script rules.  It is certainly true that there is some grandfathered 
garbage from before the IDNA rules, and ccTLDs do what they do.

> side-effects.

No kidding.  DNAME at the second level was so bad that .cat stopped doing 
it.  There is one DNAME at the root, for the two versions of Taiwan in 
Chinese which as far as I can tell nobody has noticed.  But that seems 
like a separate can of worms.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 12 Mar 2018 11:22:14 -0700

> garbage from before the IDNA rules, and ccTLDs do what they do.

John, (L, not K)

Restricting repertoire is only the tip of the iceberg.

Any TLD that allows SEA and/or South Asian scripts, but does not place 
limits on the context in which certain code points may occur (those code 
points mostly being the combining marks) will allow the registration of 
"garbage" labels - even though each element in the label, if taken by 
itself may be well-vetted. In those scripts, the "unit" that readers 
perceive is is not the code point, but the syllable; sequences of code 
points that do not form valid syllables are expected neither by readers 
nor by rendering engines, making them ambiguous or unreliable for 
identifier purposes. (For those used to scripts like Latin or Chinese 
were "random" strings of code points are merely "not words" but 
otherwise valid, this constraint inherent to "complex" scripts takes 
some getting used to, but it is quite real).

A cursory glance at the IDN tables published for some of these scripts 
reveals them as not necessarily vetted well enough and definitely devoid 
of any contextual restrictions.

Before RFC 7940, the only way contextual restrictions could be expressed 
was in natural language or "pseudo code". The former was done in the 
registry policy documents or the preamble to a few exceptional idn 
tables, while you can find the latter in RFC 5892 for the few context rules.

With RFC 7940, it is finally possible to capture these rules in a 
machine readable form, and if you look at the second iteration of the 
Root Zone LGR (LGR-2) you will find these in place for Thai, Khmer and 
Lao. (The next iteration will likely bring the ones for all the Indic 
scripts).

Here's a link to the human readable version of the Thai LGR:
https://www.icann.org/sites/default/files/lgr/lgr-2-thai-script-01jun17-en.html

And here's the overview document for LGR-2:
https://www.icann.org/sites/default/files/lgr/lgr-2-overview-01jun17-en.pdf

For a full description of the rationale for, say, the constraints on the 
Thai script, you'd need to follow the link to the original submission:
https://www.icann.org/en/system/files/files/proposal-thai-lgr-25may17-en.pdf

You mentioned that you were "interested" but felt that you didn't have 
the background to understand the issues in other scripts without some 
background and explanation.

I would be curious, whether you can make sense out of the description 
for Thai in the Root Zone. I also invite you to compare that LGR to some 
of the gTLD IDN tables for Thai.

I'd be interested in your conclusion,

A./

---

From: "John Levine" <[email-masked]>
Date: 13 Mar 2018 17:47:45 -0400

>of the gTLD IDN tables for Thai.

I think I can follow them OK.  The characters each are characterized
as consonant, various types of vowels, tones, and diacritics.  The
ordering rules would make more sense to me as regular expressions.

R's,
John

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Tue, 13 Mar 2018 16:47:17 -0700

> ordering rules would make more sense to me as regular expressions.
Check out the bottom of the HTML version of each scripts LGR file.

Click on

and go to WLE Rules. You should see a table of regexes.

Let me know if you'd like to see any additional information.
A./

---

From: "John Levine" <[email-masked]>
Date: 15 Mar 2018 15:42:56 -0400

>and go to WLE Rules. You should see a table of regexes.

I was more thinking of regexes for an entire valid string, not for
each rule, but close enough.

R's,
John

---

From: "Asmus Freytag (c)" <[email-masked]>
Date: Thu, 15 Mar 2018 14:01:28 -0700

> each rule, but close enough.

John, the rules are designed this way for a purpose. The reason is that 
the most common scenario is the need to prevent some Y from following 
some X, where the Y is typically a combining mark.

Sometimes, script rendering will "ligate" certain combinations, but fail 
if incompatible ones are encountered; in that case you may see right had 
contexts or both left and right contexts.

We originally thought we could start for the Indic scripts with an 
Akshar (syllable) for which there exist BNF definitions. A label would 
then be a concatenation of valid Akshars. The attempt resulted in 
horrifically complex looking rules that were impossible to review and  
tended to be overly restrictive: there are many strings that are a bit 
unusual/uncommon, but not "broken". Sometimes the available Akshar rules 
were language-specific.

For all of these reasons, but mainly to be able to break down things 
into easily understood bits, we ended up with the system we have now.

Now, given that an XML per RFC 7940 is fully machine parsable, it could 
be an exercise to the reader to compile a regex for full labels. I think 
that should be possible, however, I've not attempted to prove this - 
there may be some syntax construction that is difficult/impossible to 
translate into a regex for the whole lable. All I know is that I can 
translate each of the context rules to regexes (that's how the tool I 
wrote evaluates labels based on an LGR), but I apply them iteratively at 
each character position in the string rather than using the regex mechanism.

A./

---

From: "John R Levine" <[email-masked]>
Date: 15 Mar 2018 17:30:12 -0400

> Now, given that an XML per RFC 7940 is fully machine parsable, it could be an 
> exercise to the reader to compile a regex for full labels. I think that 
> should be possible, however, I've not attempted to prove this - there may be 
> some syntax construction that is difficult/impossible to translate into a 
> regex for the whole label.

Interesting point, I'll take a look.

I'm not surprised that other people are less thrilled by regexes than I 
am, since the probably weren't teaching compiler courses and drawing state 
machines on the chalkboard 40 years ago.

Regards,