Subject: Re: Possible BofF question -- I18n (was: Re: Possible OBF question -- I18n)

From: John C Klensin <[email-masked]>
Date: Fri, 01 Jun 2018 17:14:08 -0400

Hi.

First a status report.  

Rather than wait until the last minute, I went ahead and pasted
a template description into the BOF wiki page.  The entry there
is a slightly shortened and rewritten version of my initial note
in this thread.
https://trac.tools.ietf.org/bof/trac/wiki/WikiStart

I've listed Patrik (without asking him) and myself as
"proponents" for this, but would be happy to add names (or have
others do so).   Spencer, I assumed it would be not be
appropriate to add you there, but you'd certainly be welcome if
I got that wrong.

Because I had to choose and because most of the i18n-specific
WGs have been in Apps/ART, I've put it into that area, but I
believe that IESG needs to figure out how to handle this.  I
could make a case for General (on the theory that many areas are
covered or that procedural changes might be needed) or for
Security (because the most recent major i18n interactions that I
know about have been with a WG there), but, again, the IESG
should decide.

   --------

Now a few comments on recent postings (after I went to work on
the template).  

* I think it would be a just dandy idea to require an "I18N
Considerations" section in every I-D or RFC.  I see only two
problems with it.  One is that we've had such a requirement for
documents that do have i18n issues for 20+ years and it hasn't
helped.  We've also had suggestions for including a statement
about i18n issues in Shepherd templates.   If the IESG were to
decide to enforce that requirement by changing the I-D checker
and refusing to consider any document that doesn't have at least
a clear "no issues" statement, we might make some progress, but
no one has yet convinced them it would be appropriate (that has
been tried several times, but mostly in individual or
directorate discussions).  The other is that, while I rather
like the idea of some sort of formal process or review by
experts, we explored that some years ago too.  It requires
committed experts and the difficulty in finding those is part of
the problem, not an obvious solution.   Speaking as a likely
stuckee, if the idea doesn't have a clear structure and a
commitment from the IESG and the community will be taken
seriously rather than as miscellaneous review input (i.e., more
like MIB Doctors input in their area of expertise rather than
notes from whomever drew the short straw, regardless of
expertise, on the Foo Area Review Team), staffing that effort
and keeping it staffed is likely a non-starter.   So, even if we
are agreed that a required  I-D/ RFC section would solve the
problem, we would still benefit from this BOF to start shorting
out those details.

* Neils, having spend months, probably years, of my life trying
to explain to assorted well-intentioned people in social
justice, human rights, and getting more people on the Internet
in a way that is fair and appropriate to their cultures spaces
why these are hard problems and that superficial solutions make
things worse, no, making this a human rights issue wouldn't
help.  I'm sorry to single out the documents to which you
pointed for special consideration, but RFC 8280 is an example of
a document that should have gotten serious i18n review and
apparently didn't.  The second question in Section 6.2.5 should
have set off alarms because character coding is only a tiny part
of the issues and not even the most important part.   Yes,
support for multiple "charsets" is a bad idea, but, because the
mappings from other CCSs and back are usually (but not always)
completely deterministic and reversible, a far less serious one
than a long series of other issues.  For a web-oriented
illustration of another set of important problems, I suggest
that you have a look at
https://w3c.github.io/typography/gap-analysis/language-matrix.html
.  I'm also very concerned that the last paragraph of that
section seems to argue for relaxing the "protocol identifier"
constrain while showing no evidence of understanding it beyond
what might be described as "multlingualism good".   If the Human
Rights Review Team were to start complaining about the absence
of Internationalization Consideration sections, I think that
would be positive, but the expertise needed to adequate consider
Human Rights issues and that needed to consider i18n ones is
very different and RFC 8280 may actually be proof of that.

* Crypto expert mindsets -- I agree with John Levine on this.
FWIW, a few of us have made the observation that this i18n area
is easy to learn about for one class of people.  Not impossible
for others, but seemingly hard to grasp.   That class of people
consists of those who had strong exposure to at least three
languages, using at least two scripts (ideally ones that are not
closely related, i.e., Greek, Latin, and Cyrillic may only count
as one... or not), before the age of about seven (connected to
the Chomsky hypothesis about roughly the point that most people
can no longer acquire languages and language intuition as a
child).  As Peter points out, "competence" is easier, but one
still has to get fairly deep into the characteristics of several
unrelated (at least in the last three or four thousand years or
so) writing systems to reach that point.  And even that ignores
issues with spoken language, which much of the linguistics
community would claim is the only thing that is important.

* Mentions of UTR#46 in general I18N contexts may be close to
mentioning Hitler under Godwin's Law.  It is strictly about IDNs
and not the general problem; its current version makes
assumptions equivalent to domain names not needing to be treated
as having important properties of identifiers; etc.  As I said,
I don't think this BOF effort is about particular protocol
issues.  However, as a non-protocol issue (and maybe the point
John Levine was trying to make) due in no small measure to
UTR#46 and the homebrew updates to IDNA2003 it has encouraged,
we have a large number (perhaps dozens) of interpretations about
what IDNA is "really" about and what the requirements "really"
are.   That is an operational and user predictability problem, a
security problem, etc.    

"No urgency"?  Only if one wants to see the IDN situation
deteriorate to the point that systems that are concerned about
privacy, malware, etc., will simply filter out all of them as a
good tradeoff against the risks.

And, again, I see IDNs as one issue here among many, not the
only or even the key issue.

   john

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 1 Jun 2018 18:42:45 -0500

Hi, John (Klensin),

> https://trac.tools.ietf.org/bof/trac/wiki/WikiStart

Thanks for doing this.

> appropriate to add you there, but you'd certainly be welcome if

I think this is an important enough issue that I want to be supportive.

I think that will be easier if I wasn't a proponent or responsible AD. I
went back and read the last IAB Internationalization statement (from 2015),
https://www.iab.org/2015/01/27/iab-posts-statement-on-identifiers-and-unicode-7-0-0/,
and don't think I'd do this topic justice in either role - I understand 90
percent of the words and 30 percent of the meaning.

I do look forward to being helpful.

Spencer

> ..  I'm also very concerned that the last paragraph of that

---

From: Nico Williams <[email-masked]>
Date: Fri, 1 Jun 2018 19:09:23 -0500

> helped.  [...]

Please be specific.  What failures have we had.

> is easy to learn about for one class of people.  Not impossible

The condescension is thick in this one.

> * Mentions of UTR#46 in general I18N contexts may be close to

Finally.  900 words and this is the one specific thing, but not that

> only or even the key issue.

Is this an elliptical way of alluding to confusables issues?

Can you boil things down to a small set of bullet items?

 - IDNs and confusables?

 - I18N and human rights?

   (what's that got to do with Internet protocols?  sure we have to make
   sure they internationalize, but we already knew that long before
   anyone made the connection to human rights)

 - something else?  what?

Please also list any high-profile failures of the IETF in this space.

Keep it brief out of consideration for your readers' time.

BTW, regarding IDNs and confusables, I'm going to restate my previous
position on that:

a) it _is_ possible to write code to detect confusables issues (lots of
   people are doing this; I've seen several repos in github implementing
   Unicode confusable detection)

   Of course, confusable detection is best done with UC standards (there
   is one, but is it complete?).  There is no role for the IETF in this
   one.

b) we can't make the registrars do the right thing

c) we can't make the registries make the registrars do the right thing

d) we can't make ICANN make the registries make the registrars do the
   right thing

e) we _can_ give advice to ICANN and the registries, and the advice
   should be that they make up registry-specific policies as to
   confusables, and we should make up some sample policies

f) besides giving ICANN and the registries advice, there is nothing for
   us to do here.

   It isn't possible to construct a one-size-fits-all-registries policy
   for IDNA -- the registries would reject such a thing.  Dictating such
   a policy in IDNA would be inappropriate and very unpopular -- don't
   do it.

g) the best default policy would be to construct a "canonicalized" form
   of any proposed domain and allow only one domain for each
   "canonicalized" form

   For example, "paypal" and "paypa1" could be considered equivalent in
   some policy, with "paypal" the canonical form, and only one of those
   two allowed in the given registry.  A registrant could be allowed to
   have all equivalent forms of a domain's canonical form.

   Note that such a policy can be implemented (see (a)), but also that
   the uniqueness check can only be made by the registry, therefore we
   cannot dictate this in IDNA.

Nico
--

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 1 Jun 2018 20:14:24 -0500

Ok, I own this part ... please see below.

> Can you boil things down to a small set of bullet items?

I pointed out that the Human Rights Review Team included consideration of
i18n when they reviewed an IAB draft I'm editing, within the past month. My
understanding was that people were looking for ways to get i18n reviews,
and that's the only formal mechanism I've encountered since the IAB
concluded their internationalization program over a year ago.

Perhaps that wasn't helpful. As far as I can see, it wasn't.

I'll wait for the BoF.

Spencer

---

From: "Patrik Fältström" <[email-masked]>
Date: Sat, 02 Jun 2018 12:51:47 +0200

> Keep it brief out of consideration for your readers' time.

Just the situation with definition of IDNA and difference between theory and practice there is enough. And the problem with having registrars actually implement IDNA2008 as explained for example by SSAC.

Now, the BOF is not on these issues but instead on why IETF is unsuccessful on addressing I18N issues and the RFC presented by Niels with issues John point out is a 2nd example.

That a summary is requested and needed is for me an example not even people being interested enough in a BOF on the issue might not be aware of the issues existing, which is sad by itself, but, of course "to get up to speed" a certain education is needed.

But I want to before I spend tons of time here see the BoF and a plan on how to move forward.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Sat, 02 Jun 2018 10:15:51 -0400

--On Friday, June 1, 2018 19:09 -0500 Nico Williams
<[email-masked]> wrote:

>...
>...

> Finally.  900 words and this is the one specific thing, but
> not that specific either:
>...

No.  "Confusables" are, IMO. at most a symptom.  See below for a
bit more.

>...
 
>    (what's that got to do with Internet protocols?  sure we
> have to make    sure they internationalize, but we already
> knew that long before    anyone made the connection to human
> rights)

And that is consistent with, and complementary to, what I said
in my note to Neils.

>...

I'm sorry.  You are asking for details and asking that I/we be
brief.  If we are brief, you denounce us for being non-specific.
It is hard to satisfy contradictory requests.

   ----

More generally, I'm sorry but I had assumed, given your
extensive comments and claim of having developed expertise, that
you had been following these activities.  Without exception,
every item on the list below has been identified on at least the
most relevant one of the IAB i18 list ([email-masked]),
the former IDNABIS, EAI, or PRECIS list, or this (IETF discuss)
lists. Many, if not most, have had attention drawn to them on
multiple lists in the hope of alerting people who are not
following all of them.  There have also been extensive
discussions on several of these issues, including lack of
progress, in the ICANN SSAC, Universal Acceptance, ccTLD/CCNSO,
and other contexts, and at IGF and other more political forums,
but those are not IETF venues.   

If you, as an expert, have been unaware of those issues and
discussions and there are a significant number of other experts
like you, then the hypothesis that we lack sufficient expertise
may be false and should be replaced by a hypothesis that we have
not been successful in getting a sufficient number of the
available experts to engage.  That would probably not change the
need to do something, but should be a much easier problem to
solve.

In any event, and with the understanding that this is almost
certainly not a complete list, we'd had:

* Identification of an incorrect assumption, made in both the
IDNA2003 and IDNA2008 designs, about the scope and effectiveness
of normalization in Unicode, creating security risks as well as
heightened risk for user-perceived errors in comparisons.  That
issue was initially identified differently, with the original
identification turning out to be a subset or symptom of the
larger one.   The problem generated two IAB statements, one of
which essentially acknowledges that no progress had been made, a
BOF, and a few iterations of an I-D which we have not been able
to progress.

* The freeze Patrik identified, which is, in some respects, just
another symptom of our inability to engage on the above issue.

* A draft clarifying the responsibilities of zone administrators
("registries") under IDNA2008 which we have been unable to
process.

* A draft proposing dealing with a number of issues by creating
a "troublesome characters" list, which we have not been able to
process, even to the extent of a serious discussion of whether
such a list under IETF auspices is desirable and can be
maintained.

* Difficulty bringing the EAI/SMTPUTF8 and PRECIS work to
conclusion with work about which everyone has high confidence
due to the WGs running out of energy and active, informed, and
contributing, participation.

* Difficulty getting adequate input and review for work being
done in LAMPS on non-ASCII characters in X.509 certificates.

* The "many IDNA standards" problem Patrik and I mentioned
earlier in the context of UTR#46.   Given the variety of
implementations and interpretations, it seems clear that the
IAB/IETF should be considering how to get the message out, but
it has not been possible to even start that discussion.  For
this issue and the next one, it is possible to claim that the
IETF's responsibility stops when a specification is put out
there and dealing with variations, alternate interpretations,
and plain non-conformance is someone else's problem.   I see
that as an abdication of responsibility, especially when IETF
may not have done a good enough job of explaining the reasons
for its decisions.  YMMD.

* Similarly the EAI WG, after running experiments after its
first-round work, came to the conclusion that attempts to
convert local parts in transit (similar to IDNA), especially
using Punycode encoding, was risky, unlikely to work well in
many important cases, and just plain a bad idea.  Yet we've seen
heavily-promoted implementations that do just that, claiming
that it works well because their clients in users interact well
with their servers and mail stores.   We could be doing a better
job of explaining why that is a bad idea.

* As another element of the UTR#46-related problems, IDNA2008's
rule structure prohibits the use of symbols in domain name
labels.   We've seen some top-level registries violate that rule
and sell names containing some of those code points,
particularly emoji, and UTR#46 specifically authorized emoji.
Whether they do so or not, ICANN can put some pressure on
second-level registrations but,  by design and their own
decisions, cannot say very much about zones further down the
tree.  Should IETF or IAB be following up some of the
explanations that have been made in ICAAN or assorted mailing
list with a document explaining more broadly why the IDNA2008
rules are the way they are (including the rules for lookup-time
checking, which UTR#46 has been interpreted as discouraging) and
why people should be paying attention to them?  I don't know,
but believe that the discussion has not occurred and that, if
work were done on developing such a description, there would be
no way to discuss and progress it is a problem.

Again, not a complete list.   I suggest that each of those is a
significant problem and that the combination is more so, at
least unless one doesn't believe that addressing i18n usability,
security, and interoperability issues is important.  I also
suggest that many of those issues should be considered important
enough to justify the discussion proposed for this BOF even if
none of the others were out there, which I think is Patrik's
point.   YMMD.

Patrik has addressed your category list question.  I think he is
right although the more detailed list above should fill in some
of the blanks on the protocols side.   I've explained my
position on the human rights side.   And, btw, I think the
confusion issue is actually two separate sets of problems
depending on whether the confusion is deliberate or not.

I've read your comments about confusion.  In the interest of not
making this note even longer and because I don't want to stray
into trying to solve specific problems, I'm not going to respond
except to suggest that there is ample evidence that parts of
your analysis are just incorrect.

    best,
      john

---

From: Nico Williams <[email-masked]>
Date: Sun, 3 Jun 2018 01:08:33 -0500

> No.  "Confusables" are, IMO. at most a symptom.  See below for a

Ok.

> It is hard to satisfy contradictory requests.

No, in the previous reply you provided no specifics but were very
long-winded.  Let's see how it goes this time.

> If you, as an expert, have been unaware of those issues and

Be careful.  I never claimed ot be an expert.  I said I'd learned
plenty, and I said this in the context of your and Levine's claim that
we have no experts: I explained that many of us can specialize as
needed.  You essentially deny this.  You seem to claim that you (or
someone else?) are a priest of I18N and that the IETF has no one else
who could help.

Needless to say I find that attitude offensive.  Not just because I
think I can specialize as needed, but because damn it, there had better
be lots of participants at the IETF who also can.

I had a boss once who said "we are generalists who can specialize".  I
already felt that way then, but that was a great synthesis, and I really
like that formulation.  Not every person is a generalist who can
specialize as needed, but I'm certain that many of the participants at
the IETF (and their colleagues at their places of employment) are.

> need to do something, but should be a much easier problem to

I've explained that I've not been following the IETF lately.  It's
absolutely fair to ask for a list of issues you think are urgent, and
it's not OK to play gotcha and say "aha! you don't know?  you know
nothing!1!!".

Do you even read your own posts?  They drip in condescention.

Anyways, let me give you a few I18N credentials, as it were, to
demonstrate my point that I learned something useful (not that I am an
expert) in this area of which I once knew nothing.

ZFS implements form-insensitive filename comparisons.  It does this
because I insisted on it at Sun way back when.  It was my response to
the mess that HFS+ made by normalizing to NFD (well, something very
close to NFD) while OS X's input methods generally produce precomposed
codepoints.  It took some doing to get there.  I did none of the
implementation work, mind you -- just some of the code reviews and
pointed out one important optimization in the core loop of the
u8_textprep() function in Solaris/Illumos -- but I did win the argument,
and ZFS does this because of that.

(I should add that ZFS also hashes filenames, and the hash function
normalizes as it goes.  The net effect is that hash table lookups are
also form-insensitive.)

I've pushed form-insensitive comparison in the IETF, though here no one
wants to hear about it.  IIRC the NFSv4 WG ignored my pleas on this.
I'm quite certain that you've read some such posts from me, and you
could easily find them if you wish.

Of course, form-insensitivity isn't always the right or even a workable
answer.  In some cases normalizing early is better, or the only option.
And often that's a good enough answer, but for filesystems it really is
better to do form-insensitive string comparison.

It's very annoying that we try to specify normalization in NFSv4,
WebDAV, ... when the functionality needs to be deep down in the
filesystem -- NFSv4, WebDAV, and many other protocols are just
protocols, and their implementations are almost never so closely bound
to a filesystem implementation that specifying what to do about
normalization in the protocol spec could possibly be the right thing to
do.  I think it is a serious design mistake to require normalization at
the remote filesystem protocol level rather than to do so at the
filesystem layer.

A second credential would be some old blog posts I've written about this
topic:

http://cryptonector.com/2006/12/filesystem-i18n/
http://cryptonector.com/2010/04/on-unicode-normalization-or-why-normalization-insensitivity-should-be-rule/

I won't bother digging up my many posts on IETF lists on these topics.
You can look for them yourself.

> certainly not a complete list, we'd had:

Yes, with that unnecessary, insulting, and pointless aside of yours out
of the way, let's get to the list!  I'm itching.

> BOF, and a few iterations of an I-D which we have not been able

OK.  Sounds important.  The BoF should link to these IAB statements.

> another symptom of our inability to engage on the above issue.

Slow review/publication process?

> ("registries") under IDNA2008 which we have been unable to

Yes, *this* I think is urgent.  But: who should do this?

ICANN can require that registries set out some policies.  Registries can
require that registrars follow them.  The IETF cannot do any of that --
we can only give them advice.

ISTR participating in threads here where you (I think) said that the
registries don't want to hear anything about confusables from us.  Is
that still the case?  Has there been any change?  Correct me if my
memory is wrong.

> such a list under IETF auspices is desirable and can be

Yeah, I don't think the IETF is the right place for this.  Why would it
be?  No, we can give the UC some feedback, but this is squarely their
job.  Even if they don't want to do it, we can't quite do it for them.

> done in LAMPS on non-ASCII characters in X.509 certificates.

ISTM that the way you get this done is by having shepherds and/or ADs go
look for experts they can pester into doing these reviews.

I've been asked before to review things like SASLprep (which I did).

This seems like a failure of the shepherds and ADs, not a systemic
failure.  But if it is systemic, then the obvious thing to do is to make
participants pay a bit of a cost of doing business.  Want your documents
reviewed?  Then review or find someone to review these others.  The IETF
already does this informally, and it should work here.

> may not have done a good enough job of explaining the reasons

We don't have a protocol police.  I'm not at all sure what you have in
mind.

Perhaps the ISOC should fund a para-trooper team to help out with fixing
[open source] implementation issues.  Maybe they can develop the cache
necessary to get non-open source implementors to pay them attention.  I
think that's a great idea.  But short of that, I don't see what we can
do here but cajole people.

> job of explaining why that is a bad idea.

Maybe they aren't wrong.  Maybe we are.

> particularly emoji, and UTR#46 specifically authorized emoji.

"Meh"

See comments above about protocol police and our lack of authority over
the registries, or, really, anything other than the RFC-Editor and IANA.

(Do I like the idea of emoji in domainnames?  Not really, but I'm no sure
I care.  What difference does it make to me?  None.  Sure, I can
recognize domainnames by their form, and that gets harder when people
use emojis in their domainnames -- but that's also true if they just use
Chinese characters.  So I don't care.  It's very likely that this is the
calculus that's leading to this rule breaking.)

> no way to discuss and progress it is a problem.

If the market is ignoring our rules, then simply complaining about that
isn't going to solve anything.  I think we may need to re-open the
specific issues in IDNA.

This doesn't strike me as urgent, honestly, but I suppose we can differ
on this.

> Again, not a complete list.   I suggest that each of those is a

Sure.  A small-but-incomplete list is a lot better than just some
handwaving.  And it's much better than saying that not knowing that list
makes one an ignoramus.

> significant problem and that the combination is more so, at

I thought the word urgent was used.  Now we're down to "significant"?

> security, and interoperability issues is important.  I also

Yea, I think so too.  But note that you've not given examples of
breakage.  You've only handwaved a lot.

> none of the others were out there, which I think is Patrik's

I don't object to the BoF.  I wanted a list of specific items to
discuss.

> your analysis are just incorrect.

No, please do respond to that.  Feel free to start a new thread.  That's
a very specific topic that is ripe for a deep-dive.

Nico
--

---

From: "Patrik Fältström" <[email-masked]>
Date: Sun, 03 Jun 2018 08:29:27 +0200

> Slow review/publication process?

Lack of responses on the IDNA WG mailing list when I as an expert asked questions. As I know we have more people in the IETF eco-system than myself and John, and as we definitely are no priests ;-) I decided to simply not move forward due to the lack of responses.

Now I am asked again by IAB to move things forward and that *that* have been slow is my fault only.

This discussion and the possible BOF have made me increase the priorities on this work item.

> ICANN can require that registries set out some policies.  Registries can require that registrars follow them.  The IETF cannot do any of that --

After lots of work in ICANN ICANN Board finally passed a resolution saying IDNA2008 should be followed (it is in the applicant guidebook, but that is instructions to the evaluation process). This based on (yet another) note from SSAC on how bad the situation is. ICANN Board asked ccNSO and gNSO to look into the issue and they are. The issue is though that ICANN only have some gTLDs as contracted parties, and not all of them have new enough contract (although exact contractual situation to me is unclear). ccNSO do not have contracts with ccTLDs although many ccTLDs have signed MOU-like documents with ICANN stating they should follow the ICANN policies developed. So we have from ICANN perspective a number of parties, not all contracted, that should follow IDNA2008. The problem is because of this *not* a contractual violation within the ICANN eco-system.

In reality we know we do not have any protocol police. So all policies are implemented by volunteers (open source and not) and the question is then what they use, and why they do IDNA2008 "in their own way".

You point at one excellent example, that people do not use the same normalization mechanism.

Another is that people just use whatever their library is using, and if we look at libCurl it can be compiled nowadays with strict IDNA2008 or relaxed (see below), and the programmer do not even know what is used. On top of this one might link with whatever libidn is in the operating system version one use, and that can also create some confusion. Specifically as one might use one or more of:

- IDNA2003
- IDNA2003 but home-brew application on top of newer versions of Unicode
- IDNA2008 with UTS#46 and some random version of Unicode
- IDNA2008 as IETF have approved the versions of Unicode
- IDNA2008 applied to later versions of Unicode

Not fun.

   paf