Subject: Re: Possible BofF question -- I18n (was: Re: Possible OBF question -- I18n)

From: Nico Williams <[email-masked]>
Date: Fri, 1 Jun 2018 10:10:55 -0500

On Fri, Jun 01, 2018 at 07:46:36AM +0200, Patrik Fältström wrote:
> On 1 Jun 2018, at 6:19, Nico Williams wrote:
> > PS: Really, I'm shocked at this thread.  I don't get it.  What series
> >     of events is causing the sad, utter despair on display here?
> >     I've not been following the IETF all that much, so it's possible
> >     I missed some disasters.  Can they have been so bad?  Please
> >     fill me in on them.
> 
> That we have for example:
> 
> - Had Unicode/IDNA discussions with comments from very very few
>   people, and I think three that "did send text"
> - Had IAB close a directorate on the topic
> - Had Precis "just about" get the review they needed to call consensus
> 
> This is what I have seen.
> 
> So, very very very few people actually "show up" in discussions.

Doesn't seem like the end of the world to me.  Maybe there is no sense
of urgency to IDNA improvements.

Require that documents have I18N Considerations sections, require review
by an I18N directorate, and you'll see how quickly participants who used
to not give a damn about I18 will come around, learn what they have to,
and get their I18N work done.  Suddenly the I18N directorate will be in
demand.

What's certainly not workable is to let other SDOs (which ones?  besides
the UC, which almost certainly doesn't have the Internet protocol
know-how/experience, which SDOs have more I18N clue and experience than
the IETF?!) do the I18N work on Internet protocols.  If they do so while
participating in the IETF process then that's just working within the
IETF.  If they do so without following IETF process then we might as
well close down the IETF.

These are [perhaps the only] choices we have:

 - do nothing about I18N for a while -- maybe we just don't and won't
   have the energy until the pain becomes palpable, and maybe waiting
   until then is OK

   (this is a distinct possibility -- maybe we've gotten good enough)

 - require Standards-Track (and Informational too, why not) RFCs have
   I18N Considerations sections, and require review by an I18N
   directorate -- or something along these lines

 - close down the IETF

I don't think anyone seriously thinks we should close down the IETF.
But I keep seeing that contradiction in terms of letting other SDOs do
our I18N work.  So who knows, maybe some of you think we should close
down the IETF!  If any of you do, please explain.

Participants already pay a Security Considerations and secdir review
cost of doing business "tax".  I18N is the same.  For many participants
I18N is actually critical, though most people/vendors who have critical
I18N business requirements are in the W3C / application space -- those
will gladly pay an I18N cost of doing business.

Nico
--

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 01 Jun 2018 17:13:53 +0200

On 1 Jun 2018, at 17:10, Nico Williams wrote:

> Doesn't seem like the end of the world to me.  Maybe there is no sense of urgency to IDNA improvements.
>
> Require that documents have I18N Considerations sections, require review by an I18N directorate, and you'll see how quickly participants who used to not give a damn about I18 will come around, learn what they have to, and get their I18N work done.  Suddenly the I18N directorate will be in demand.

Exactly this was what happened, which made IDNA2008 updates at IANA be blocked. IAB have now asked me as  the appointed expert to move things again, which I will do now actually(!) and we'll see how many will show up this time. Last time exactly three people. Out of which I was one.

   Patrik

---

From: Benjamin Kaduk <[email-masked]>
Date: Fri, 1 Jun 2018 10:59:37 -0500

On Fri, Jun 01, 2018 at 05:13:53PM +0200, Patrik Fältström wrote:
> Exactly this was what happened, which made IDNA2008 updates at

Er, exactly what is what happened?
I don't remember many drafts going by the IESG with I18N
Considerations sections...

-Ben

> IANA be blocked. IAB have now asked me as  the appointed expert to
> move things again, which I will do now actually(!) and we'll see
> how many will show up this time. Last time exactly three people.
> Out of which I was one.
> 
>    Patrik

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 1 Jun 2018 10:00:15 -0600

> and get their I18N work done.  Suddenly the I18N directorate will be in

This is worth considering...

Peter

---

From: Niels ten Oever <[email-masked]>
Date: Fri, 1 Jun 2018 18:15:25 +0200

> This is worth considering...

l18n is part of the Human Rights Considerations

https://tools.ietf.org/html/rfc8280#section-6.2.5

so you might want to consider joining the Human Rights Review Team and
scope for these issues in drafts:

https://www.ietf.org/mailman/listinfo/Hr-rt

Best,

Niels

-- 
Niels ten Oever

ARTICLE 19
www.article19.org

PGP fingerprint    8D9F C567 BEE4 A431 56C4
                     678B 08B5 A0F2 636D 68E9

---

From: "John R Levine" <[email-masked]>
Date: 1 Jun 2018 12:27:20 -0400

> This is worth considering...

I don't think it's a very good idea.  It'll just lead to useless pro-forma 
language.  It'd be OK if IESG review were to flag drafts and ask whether 
they should say something about I18N.

Like jck, I have to disagree with Nico's assertion that anyone can pick up 
I18N expertise quickly, and also jck's comment that if your experience is 
only with European alphabetic languages, you've barely scratched the 
surface.  It's remarkably subtle.  It needs a particular mindset, in much 
the way that being a crypto expert does.  Different mindset, though.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 1 Jun 2018 10:42:43 -0600

On 6/1/18 10:27 AM, John R Levine wrote:
> I don't think it's a very good idea.  It'll just lead to useless
> pro-forma language. 

Quite possibly.

>  It'd be OK if IESG review were to flag drafts and
> ask whether they should say something about I18N.

We might want authors to think about internationalization before IESG
review.

> Like jck, I have to disagree with Nico's assertion that anyone can pick
> up I18N expertise quickly, 

I don't think he said "quickly". :-)

> and also jck's comment that if your
> experience is only with European alphabetic languages, you've barely
> scratched the surface.  It's remarkably subtle.  It needs a particular
> mindset, in much the way that being a crypto expert does.  Different
> mindset, though.

All true.

To pursue the analogy, AFAIK (correct me if I'm wrong) we don't have
that many pure cryptography experts at the IETF. We do have folks who
know enough about crypto to make intelligent, well-informed
recommendations with regard to the use of cryptography in Internet
protocols. Perhaps that's mostly what we need for i18n, too. Whether we
call this "expertise" or "competence" doesn't especially matter. (FWIW,
although like Nico I've learned plenty about i18n and I've even authored
some RFCs on the topic, I do *not* consider myself an expert.)

Peter

---

From: "John R Levine" <[email-masked]>
Date: 1 Jun 2018 12:50:46 -0400

> of urgency to IDNA improvements.

It's hard to understand how anyone who's familiar with the tangled 
progress of UTS#46 would say that.

Regards,

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 1 Jun 2018 12:39:19 -0500

FWIW ...

> > ask whether they should say something about I18N.

This does happen now during IESG Evaluation. I believe Alexey is the most
likely AD to ask about i18n, but other ADs also ask, even ADs outside ART.

We typically don't see each other's AD Evaluations, so I don't know what
happens before IESG Evaluation.

> We might want authors to think about internationalization before IESG

Yes, please do.

There are very few things that the IESG wants to be the first people to
think about, and especially not when balloting on a document!

Spencer, speaking as 1/15th of the balloting IESG, most telechats

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 1 Jun 2018 12:49:26 -0500

What follows is semi-on topic ...

> https://www.ietf.org/mailman/listinfo/Hr-rt

To provide background on my +1, I recently received the first HR-RT review
I'd ever seen for a doc I'm editing.

The thread starts at
https://mailarchive.ietf.org/arch/msg/hr-rt/byJOPkEXzGlHcYISVEQfk_zkbug.

In particular, if you check the review, you'll see this:

 Human Rights Considerations
===========================

*Upon consideration of the draft report, no implications for*
*internationalization ([RFC8280], sec. 6.2.5), heterogeneity support*
*([RFC8280], sec. 6.2.8), accessibility ([RFC8280], sec. 6.2.11) or*
*localization ([RFC8280], sec. 6.2.12) have been found. *We will further
suppose that issues relating to Anonymity ([RFC8280], sec. 6.2.9),
Pseudonymity ([RFC8280], sec. 6.2.10) and Confidentiality ([RFC8280], sec.
6.2.16) have been sufficiently addressed in the draft as is.

In the below text, Security ([RFC8280], sec. 6.2.4), Integrity ([RFC8280],
sec. 6.2.16) and Authenticity ([RFC8280], sec. 6.2.17) are further
considered to have been addressed, or as in progress of being addressed by
the specific endeavours pointed out in the report. As such, this assessment
hopes to serve as a complement to remaining Human Rights Considerations
Guidelines which were not addressed by the MaRNEW draft.

## Privacy ([RFC8280], sec. 6.2.2)

We laud the general emphasis on considering user privacy concerns paramount
[Section 2.1.1, 2.3 etc].

In Section 3.1.1, para 1, it would be useful to emphasize that middleboxes
that are "authenticated and invoked explicitly" might not be very effective
at conserving user privacy if they lead to auto-click through. In practice,
"opted-in" middleboxes might not be much better than transparent
middleboxes. Additionally, this poses the danger of making matters worse if
users assume that because they haven't been asked for middlebox permission,
they are not going through a middlebox. For users accessing the Internet on
a browser client: if the connection is MITMed by a middlebox, the browser
would somehow need to convey this effectively and actionably to the user.
Whatever effort is expended on this should involve browser vendors.

## Connectivity ([RFC8280], sec. 6.2.1)

It is a priori obvious that a discussion with mobile network operators
would touch upon connectivity. As a general comment, the HRPC RG is not
concerned that encrypted internet traffic could create topological problems
in network deployment with respect to density of base stations. It is more
sensible to imagine that the density of base stations is kept low by
deployers to save money on infrastructure roll-out.

In the MarNEW draft report, middleboxes are brought up frequently as an
example of infrastructure rendered less useful by the ubiquitous adoption
of encryption. Middleboxes may enhance access to content for individuals,
and may also interfere with their ability to enjoy content free of
surveillance, censorship or content manipulation. The focus of the workshop
of exploring ways in which encrypted materials can benefit from the same
speed of deliverance as un-encrypted materials is therefore welcome.

## Content Agnosticism

Traffic prioritisation, whether based on technical or commercial grounds,
risks interfering with an individual's right to freedom of expression or
opinion, by restricting access or enjoy content of their own choosing. As
such, prioritisation should be made explicit and obvious, and subject to
consent from the individual, whether the motivations for deployment are
commercial or technical.

It is clear that internet filtering and blocking of content of specific
types, whether motivated by network management or regulatory concerns, does
not fulfill a content agnosticism criterion. It is the observation of HRPC
that [RFC7725] (An HTTP Status Code to Report Legal Obstacles) may assist
communication of specific filtering and blocking operations in action.

It the the impetus of the Human Rights Guidelines in RFC8280 that content
agnosticism be respected by protocols developed in the IETF, and in general
an accepted view in the global human rights community that content
agnosticism is desirable at all levels of technical infrastructure. As
such, the HRPC RG is concerned about ideas that traffic be classified
through additional headers or metadata, identifying traffic via 5-tuples,
trust models and frameworks, or heuristics.

So, with that as background and speaking only for myself, +1.

If people should be thinking about i18n for their work, HR-RT is a resource
that they might not know about.

(and thanks, again, for your review of my doc!)

Spencer

---

From: Nico Williams <[email-masked]>
Date: Fri, 1 Jun 2018 12:59:12 -0500

> the way that being a crypto expert does.  Different mindset, though.

Not everyone can pick up I18N expertise  quickly?  Sure, but we engineer
complex things, so i suspect many participants at the IETF can actually
pick up enough I18N expertise to be useful. I18N is not a dark art.

Can we get more training?  Will ISOC pay for it, perhaps?

Or are you saying the rest of us are not just good enough?  What are _your_
concrete proposals?

The negativity here makes me wonder.  With this sort of attitude nothing
good will result. Saying the sky is falling might get you some volunteers,
but saying none could help sure won't.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Fri, 1 Jun 2018 13:02:02 -0500

> progress of UTS#46 would say that.

Stop misreading.  "Maybe there is no sense of urgency" is clearly
speculation about the wider community's sense.  It's probably quite right
too.  If you think we should think it's urgent, then make the case.  Be
specific. What's broken, and why is fixing it urgent?

Nico
--

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 1 Jun 2018 19:30:09 +0100

Hiya,

> but saying none could help sure won't.

I do think Nico has a point in the above. I've personally
felt disinclined to get involved in this kind of work for
such reasons. ISTM that starting i18n discussions by saying
how little others know isn't a way to encourage folks to
do work, and does happy noticeably often.

Now it could be that it's better that someone who knows as
little as I about the topic is not involved, but in the
last couple of decades in the IETF we have I think seen a
lot of IETF participants learn enough about security to
do well, (or cause less accidental harm anyway:-), but I
don't think we can say the same about i18n.

S.

---

From: Nico Williams <[email-masked]>
Date: Fri, 1 Jun 2018 13:32:04 -0500

> > I don't think it's a very good idea.  It'll just lead to useless

We have the IESG and directorates to make sure these sections are not pro
forma.

> I don't think he said "quickly". :-)

I don't either :)

> some RFCs on the topic, I do *not* consider myself an expert.)

This is my point exactly.  We can make do with the participants that we
have.  And since they are all we have, making do is what wet must do.  Of
course, we may attract new participants, and we should educate the ones we
already have.  So let's do that.

Nico
--

---

From: John C Klensin <[email-masked]>
Date: Fri, 01 Jun 2018 15:17:21 -0400

One last comment and then I have to move on to other things
today....

--On Friday, June 1, 2018 13:32 -0500 Nico Williams
<[email-masked]> wrote:

>> To pursue the analogy, AFAIK (correct me if I'm wrong) we
>> don't have that many pure cryptography experts at the IETF.
>> We do have folks who know enough about crypto to make
>> intelligent, well-informed recommendations with regard to the
>> use of cryptography in Internet protocols. Perhaps that's
>> mostly what we need for i18n, too. Whether we call this
>> "expertise" or "competence" doesn't especially matter. (FWIW,
>> although like Nico I've learned plenty about i18n and I've
>> even authored some RFCs on the topic, I do *not* consider
>> myself an expert.)
> 
> This is my point exactly.  We can make do with the
> participants that we have.  And since they are all we have,
> making do is what wet must do.  Of course, we may attract new
> participants, and we should educate the ones we already have.
> So let's do that.

Let me pursue the crypto analogy a bit.   I contend that the
main problem with i18n expertise (or competence) in the IETF is
not what people know and have learned but that we've had a lot
of people participating in the debates who don't know what they
don't know.  As least when I was hanging around with security
experts, that was rarely a problem and people who had it were
rapidly educated or disabused about the level of knowledge.
That is not "negativity", it is trying to both educate and to
get the S/N ratio to a level at which serious discussions are
possible.  

Some of you know that I spend several years primarily working on
food composition data.  We used to have a saying that the number
of people in the world with strong opinions about food, food
nomenclature, and what was an was not good to eat (from either a
health or enjoyment standpoint) was roughly equal to the number
of people in the world who had the privilege of eating
regularly.  Those opinions and beliefs were sufficiently tied to
strong cultural norms that most of those people, when confronted
with science, facts, or counterexamples simply became more
insistent, and often belligerent, about their positions.  

The IETF I18n situation feels a lot like that, with the
condition of someone expressing strong opinions or behaving as
if they believe they are experts being that they speak and/or
read language.  I'm not aware of anyone who has contributed
significantly to I18n work in the IETF who is unwilling to
explain and educate as long as there is some evidence of
willingness to listen and learn.  The thing that drive people
off and contributes to a sense of hopeless frustration is the
noise from people who don't know what they don't know and who,
from their behavior patterns, aren't interested in finding out.  

I'd like to think that problem does not occur in
security-related work, at least to the same extent.  Maybe it is
because everything has to be learned and no one things they were
born with security knowledge.   Maybe because there are clearer
knowledge categories and boundaries.  Since the crypto expert
example was used, probably most of those who aren't don't think
they are and are swiftly (and I hope gentle) corrected.   I've
also observed that, when real crypto experts are needed, the
Security Area is able to find them, talk with them, and
understand and make use of the explanations.   The analogy of
that may not be the case for I18n and the N/S ratio doesn't help.

    john

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 01 Jun 2018 22:14:53 +0200

On 1 Jun 2018, at 20:02, Nico Williams wrote:

> Stop misreading.  "Maybe there is no sense of urgency" is clearly speculation about the wider community's sense.  It's probably quite right too.  If you think we should think it's urgent, then make the case.  Be specific. What's broken, and why is fixing it urgent?

IETF have not been able to approve IDNA2008 for later versions than 6.3.0.

   Patrik

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 1 Jun 2018 15:16:42 -0500

I'm confused about something here.

> We have the IESG and directorates to make sure these sections are not pro

I'm not seeing an obvious directorate that would review docs for i18n
considerations at https://datatracker.ietf.org/dir/.

To be fair, that's the list of active directorates, but searching for "ietf
internationalization directorate review" didn't turn up an obvious closed
directory.

What I do know, is that the IAB had an Internationalization program, but
that concluded in March 2017, and IAB programs don't have any particular
role in document approval beyond program members (who may or may not be IAB
members) responding to Last Call announcements as individuals. Details at
https://www.iab.org/activities/programs/concluded-programs/internationalization-program/
.

Transport guys don't internationalize a lot, because most of our headers
aren't intended for human consumption, and it's very possible that I'm
missing something - so, I'm just trying to stay synchronized with this
important conversation.

Thanks,

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 1 Jun 2018 14:48:31 -0600

On 6/1/18 2:16 PM, Spencer Dawkins at IETF wrote:
> On Fri, Jun 1, 2018 at 1:33 PM Nico Williams <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     On Fri, Jun 1, 2018 at 11:42 AM Peter Saint-Andre
>     <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     We have the IESG and directorates to make sure these sections are
>     not pro forma.
> 
> 
> To be fair, that's the list of active directorates, but searching for
> "ietf internationalization directorate review" didn't turn up an obvious
> closed directory.
> 
> role in document approval beyond program members (who may or may not be
> IAB members) responding to Last Call announcements as individuals.
> Details
> at https://www.iab.org/activities/programs/concluded-programs/internationalization-program/.
> 

AFAIK we've never had an i18n directorate. Some ADs (Alexey is great at
this) flag documents in IETF LC or IESG review that could use i18n help.
Sometimes the authors reach out and ask for early review / assistance,
either on their own or prodded by a WG chair or AD (this happens for
many things outside of i18n, of course). Formalizing this has potential
to improve outcomes without necessarily requiring an
Internationalization Considerations section in every RFC (although IMHO
we could include that during the I-D phase and remove it if there's
nothing actionable, as we do for IANA considerations). However, as you
note, there are plenty of protocols that don't include human-readable
text so the i18n considerations would be a no-op.

Peter

---

From: Nico Williams <[email-masked]>
Date: Fri, 1 Jun 2018 19:13:36 -0500

> text so the i18n considerations would be a no-op.

Sure, +1 to that.

But it'd be best if authors (and WGs) had to think about I18N before
IESG review.

One good place to push I18N is at WG (re)charter time.  That's because
it might then be obvious whether or not any of the WGs work items will
have I18N issues.  For example, TCP/IP has no I18N issues.  But SSHv2
does, as do NFSv4, HTTP, and many others.  Generally the protocols
needing I18N help are application protocols.

Nico
--

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 2 Jun 2018 15:54:54 +1200

I'm an ignoramus, but:
On 02/06/2018 05:59, Nico Williams wrote:
...
> pick up enough I18N expertise to be useful. I18N is not a dark art.

If a dark art is one that involves combinatorial degrees of complexity
mixed with human perception, judgment and emotion, then I fear that I18N
*is* a dark art. We can perhaps manage such complexity by limiting the
scope of what we try to do in our protocols, but I for one would very
much appreciate having an I18N directorate reviewing everything.

    Brian

---

From: "John R Levine" <[email-masked]>
Date: 2 Jun 2018 09:56:32 -0400

> much appreciate having an I18N directorate reviewing everything.

Thanks, that was what I was trying to get at.  It would be great if more 
people learned about I18n issues, but I worry about expertise at the level 
of security experts telling people to memorize all their passwords and 
change them every month.

"Confusables", different characters that look exactly or approximately the 
same is a good example.  I used to think that one could make sets of 
confusable characters and avoid security problems by disallowing strings 
that differed only in confusables.  Unfortunately, what is confusable is 
highly context dependent.  For example, an Arabic digit 5 looks a lot like 
a lower case letter o, so depending on who and where you are you might 
think it looks like o or 0 or you might think it looks like 5 or you might 
think it looks like both.  I didn't realize that until I talked to native 
Arabic speakers and tried to read speed limit signs in Abu Dhabi.  Don't 
get me started on composable emoji and skin tones.

I'm not saying it's hopeless, but we need to be careful assuming that some 
knowledge always leads to better analyses than none.  Remember all those 
passwords they force you to change every month.

Regards,

---

From: Nico Williams <[email-masked]>
Date: Sun, 3 Jun 2018 01:13:51 -0500

> much appreciate having an I18N directorate reviewing everything.

I disagree.  It's not a black art.  There are some corners where
reasonable people can and will disagree (should emoji be allowed in
domainnames?), and there will be some cases that require script-specific
expertise, and therefore a lot of time to sort out.  But I18N is not a
dark art at all.  If it were, then how would we get anything done in
that space?  The E in IETF stands for Engineering, not Dark Art.

---

From: Nico Williams <[email-masked]>
Date: Sun, 3 Jun 2018 01:22:19 -0500

On Sat, Jun 02, 2018 at 09:56:32AM -0400, John R Levine wrote:
> highly context dependent.  For example, an Arabic digit 5 looks a lot like a
> lower case letter o, so depending on who and where you are you might think
> it looks like o or 0 or you might think it looks like 5 or you might think
> it looks like both.  I didn't realize that until I talked to native Arabic
> speakers and tried to read speed limit signs in Abu Dhabi.  Don't get me
> started on composable emoji and skin tones.

Yes, confusability is in the eye of the beholder, but a first
approximation set of confusable mappings can be constructed (and has
been).  We don't need perfection, but we could apply AI techniques to
look for more confusables and get closer to perfection.  Registries
don't need us to tell them that any of this is a good idea (though we
should).

> passwords they force you to change every month.

It's hard to argue that more knowledge makes things worse.  Sure, it can
(e.g., by raising one's cognitive burdens), but mostly it's not the
case.  I'm not afraid of assuming that more knowledge will help.  But if
you are... then our supposed lack of I18N expertise should be a boon!

Nico
--

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 4 Jun 2018 08:56:36 +1200

> that space?  The E in IETF stands for Engineering, not Dark Art.

We're in a space where the evaluation of A==B depends on more than
the bit strings A and B. Your post about form-insensitive filename
comparisons is a case in point, although I don't pretend to understand
it. OK, we can argue whether that's a dark art or simply complicated
engineering, but really what I need is (a) some generally applicable
guidelines on protocol design in this area and (b) some people willing to
review any relevant design work.

    Brian

---

From: Nico Williams <[email-masked]>
Date: Mon, 4 Jun 2018 22:10:24 -0500

> it. OK, we can argue whether that's a dark art or simply complicated

  form_insensitive_strcmp(a, b) == memcmp(normalize(a), normalize(b))

Except that actually one can greatly optimize this to avoid most of the
compute and memory cost of normalization.

To see why consider comparing my first name as I usually write it
(Nicolas) vs.  how it should be written (Nicolás).  The two strings
should compare as not equivalent.  But the two ways to write the second
form (with the &acute; precomposed vs. decomposed) should compare as
equivalent (because they are).

Notionally one can iterate the codepoints in the two strings and compare
them, with a fast-path in the case where pairs of codepoints are
byte-wise equal, and a slow path where they are not.

In most cases all strings are in the same form (as input via whatever
input methods), in which case equal strings compare as equal without
having to use the slow path, and non-equivalent strings compare as
non-equivalent with only as many slow-path executions as needed before
the point where they differ.

The slow path basically collects one non-combining codepoint as a many
combining codepoints follow it, normalizes just that one character from
each string and memcmp()s them.  The slow path doesn't require
allocation either, since there is a limit to how many codepoints a
character can require.

For non-equivalent ASCII-mostly strings this is very fast.

Now, this optimization is pretty obvious -- it's the sort of thing
engineers do.  It's not a black art.

Now, of course, deciding what characters to allow or forbid in some
identifier... admits some subjectivity.  E.g., whether to allow emoji in
domainname labels.  I would submit to you that we already permit emoji
in domainname labels: what else are ideographs (Han/Kanji/whatever) if
not pictographs (emojis) [that have been in use for a long time]?  Is it
not snobbish/elitist to say that you can have any Kanji you want but not
a pictograph?  Have you seen how the cool kids write?  They are really
🆒, sometimes stringing along a sequence of emojis... much like one might
Kanji.

> review any relevant design work.

I agree.

For example, I've been saying for a long time that filesystem protocols
should not specify a normalization for filenames and such.  Instead the
filesystems (not the protocol implementations) should use form-
insensitive comparison.  For a protocol like Kerberos...  form-
insensitive comparison doesn't quite work as well as just normalizing as
soon as possible, so do that.  I mean, normalization is not really a
difficult thing anymore -- the code for it exists now.

As to what characters to allow/forbid in what contexts, I do think we're
better off getting the the Unicode Consortium (who are, arguably, the
real experts in this) to do the heavy lifting there, and in our
protocols to forbid as little as possible.  Similarly for mappings.

That's just for starters.  I hope I've illustrated that I18N is not that
much of a dark art.

Nico
--

---

From: Barry Leiba <[email-masked]>
Date: Tue, 5 Jun 2018 01:50:11 -0400

> equivalent (because they are).

But there's one of the things that makes this a complicated topic:

- we say that "nicolas" is not equivalent to "nicolás"
- but we say that "nicolás" *is* equivalent to "nicola´s", and we
handle this using normalization
- does that mean that it's OK to have "nicolas" and "nicolás" as two
different usernames assigned to two different users?
- if yes, how do we deal with the human interface issues involved?
What happens if the human identified as "nicolás" uses an input
mechanism that doesn't have a way to enter "á"?  How can he log in?
- if no, how do we make sure (in an automated way) that we don't make
that assignment?
- does the answer change if "nicolás" is a domain name instead of a username?
- does the answer change if "nicolás" is a *password*?
- and what about "nicolàs"?  and "nicolâs"?  and "nicoläs"?
- what about "nicolаs" (that's a Cyrillic character in the penultimate
position)?
- what about "nicolαs" (that's a Greek character in the penultimate position)?
- what about other Unicode characters that look like "a", either
exactly (as with Cyrillic) or closely (as with Greek)?
- what about handling of "ä" vs "ae"?  Do we want to avoid assigning
"käse" and "kaese" as distinct usernames?  Does the answer to this
differ depending upon whether the language is German (where using "ae"
to represent "ä" is common) or Swedish (where it is not)?

Now extend this to the many other characters that can look similar
(say, "n" vs "ñ" in Spanish).  Extend it to other language-related
issues ("i" vs "ı" vs "İ" vs "I" in Turkish; all the character
variants in Arabic).

These are only some of the reasons it's difficult.  And the number of
people who stand up and say, "oh, just <do this> and the problem is
solved," demonstrates that too too too many people *think* they
understand... and don't.

Barry

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 5 Jun 2018 02:14:28 -0400

> On Jun 5, 2018, at 1:50 AM, Barry Leiba <[email-masked]> wrote:
> 

The above is a simple statement about *equivalence*

> different usernames assigned to two different users?

They are not equivalent, whether assigning both is a good idea
is an entirely separate question.

> - what about handling of "ä" vs "ae"?

They are not equivalent. This is confusability question,
not an equivalence question.

>  Do we want to avoid assigning
> "käse" and "kaese" as distinct usernames?

Ditto.

> solved," demonstrates that too too too many people *think* they

Or do, but the participants are talking past each other...

Natural language issues are messy, and necessitate trade-offs,
which trade-offs to make can be the subject of much debate.

For example, in EAI, I find the decision to introduce non-identity
content-transfer-encodings of composite MIME parts to be far more
problematic than the problem it is intended to solve.  I wasn't
around for the discussion, and probably would not have been able
to change the outcome, but one way or the other someone would have
had to walk away unhappy...  (an alternative would introduce 
punycode encoding of localparts and break the sacrosanct rules about
local parts being only understood at the destination, pick which
axioms to violate, ... at least one).

-- 
	Viktor.

---

From: Christian Huitema <[email-masked]>
Date: Mon, 4 Jun 2018 23:33:46 -0700

On 6/4/2018 10:50 PM, Barry Leiba wrote:

.. a long list of questions, such as
> to represent "ä" is common) or Swedish (where it is not)?

When I look at these questions, I can't help thinking that we are trying
to deal with human interface issues at the wrong layer. Or rather, that
there are some layers at which the human interface issues are paramount,
and some layers at which it is much better to deal with binary strings.

For example, if I were writing a mail UI, I would be very concerned with
the representation of names and other strings. But then I would have
tools. I can consult with interaction designers, I can run the proposed
UI designs through user panels, I can design specific UI for specific
subsets of users, I can get feedback from beta users, I can analyze the
telemetry, I can push software updates to fix my inevitable mistakes.

On the other hand, I am writing an SMTP MTA, a DNS recursive resolver,
or a SIP server, I don't have any of those tools at my disposal. My
server is suppose to exactly implement the specified protocol. I will
only get indirect feedback from users who maybe are not even aware of
the server's presence. I will get telemetry about my server's
performance, but I won't be able to measure the level of befuddlement of
the users whose packets were processed.

Forty years ago, we started a path on a slippery slope with a basic
normalization process -- considering lower and upper case letters as
equivalent. That was probably justified by the hardware of the time,
when some devices could only produce upper case letters, something like
the Telex alphabet. But we slipped on the slope with enthusiasm,
embedding case insensitive comparisons in all kind of protocols, and
then attempting to extend the concept piecemeal to a variety of languages.

In hindsight, that was a bad idea. It leads to an expectation that
intermediaries not only can "normalize" character strings, but are
expected to do it. Barry gives some great examples of that silliness
with variations of European alphabets, but if I understand correctly the
same games can be played with Arabic/Persian letters or with variations
of the Chinese characters, and probably with quite a number of different
scripts.

Text comparison looks fundamentally like a human interaction engineering
issue, and a very hard one at that. I can't believe for a minute that
engineers writing code for message passing servers will deal with that
sort of problem without making a mess of it. Besides, it is not obvious
at all that there is one single right answer to these questions. So my
BofF question would not be "how to educate the engineers on the fine
points of normalizing Unicode strings", but rather, can we layer the
designs so that "the network" handles binary, and only specialized
systems handle the mapping from binary to "meaning"?

-- Christian Huitema

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Jun 2018 02:53:30 -0400

> But there's one of the things that makes this a complicated topic:

I was describing a specific primitive, but i do like your taking this
further:

> handle this using normalization

Right, that is simple enough.  For some value of simple.  You need
normalization code (which isn't trivial), then it is simple.

> different usernames assigned to two different users?

In filesystems there's also whether to be case-sensitive, and out can be a
per-filesystem opt-in.

As to usernames, principal names, and so on, well, it's a rather subjective
choice.  "Nicolas" is perfectly correct in French, and is distinct from
"Nicolás", though it can be confusing, especially if you have software that
cannot display accents...

Now, they obvious question is: is this something a protocol should address
by making &acute; equivalent to 'a' globally, or should this be policy
local to an appropriate administration domain?  Well, that's a bit of a
judgement call, but the best option is to give people the freedom to make
that choice where possible.  Thus, not globally considering any
combinations of 'a' equivalent to each other and 'a... is the better
approach.

In terms of DNS, consider a proposal to ban mixing of scripts in any one
label...  But in South Korea it is common to mix Hangul with -ing endings,
so why should .kr not be allowed to use at least that sort of actor
mixing?  There are almost certainly other similar cases, and more will
arise as culture evolves!

Who is in a better position than they registries to make such a decision?
Certainly NOT the IETF, not any one participant and not the IETF
collectively.

There is a big difference between form equivalence (same exact character,
two or more ways to represent it as the codepoint level) and confusables.
We can trivially (see above) deal with the former, but the latter is going
to need local policy.  I really don't see a better answer re: confusables,
and i know that's not a popular opinion, but i don't think it's wrong.

> mechanism that doesn't have a way to enter "á"?  How can he log in?

Answered above.

> that assignment?

This one is easy: IF you really want this (i don't think we should want
this globally) decompose (normalize to NFD) then drop combining codepoint.
This answer won't work for cross-script confusables, naturally, which is
partly why i wouldn't recommend this approach.

- does the answer change if "nicolás" is a domain name instead of a
> username?

Same answer!  The local authority (here: the registry) should decide this,
write a policy, and enforce it (by having the registrars implement it).
 (See comments below about user-agents as vessels for local policy as well.)

I don't think we can make such a policy globally that doesn't risk angering
some local communities.

It can.  Losing some entropy in a password might be safe, but this is
simpler as a global policy rather than as local policy.  It's even simpler
to tell users to only use characters they can reliably input on all devices
(this isn't as trivial as it should be, but by and large this approach
works).

- and what about "nicolàs"?  and "nicolâs"?  and "nicoläs?

> position)?

- what about "nicolαs" (that's a Greek character in the penultimate
> "käse" and "kaese" as distinct usernames?

Same answers as above.

Does the answer to this
> to represent "ä" is common) or Swedish (where it is not)?

Only if the context can let an end-user choose one (or more) language(s).
DNS, for example, cannot.  A filesystem cannot either.  Text documents /
word processors can (and might, especially in a search function).

> issues ("i" vs "ı" vs "İ" vs "I" in Turkish; all the character

Same answers as above.

The protocols should be permissive.  Local policies should be less so -
perhaps no more permissive than is absolutely necessary.

Note that a user-agent is also a place where local policy can be applied.
In fact, there exist browser extensions to deal with confusables.

> solved," demonstrates that too too too many people *think* they

It's difficult because our world culture has globalized while at the sane
tone we are not willing to unify confusable characters.  When i say "we"
here i mean mankind in all its local polities.  We've tried Han
unification, and that failed as a matter of politics.  We (the IETF) can
hate this all we like, but we cannot change it and should not even try.
We've talked about human rights and I18N.. some might say that getting
their characters drawn the way they want without needing a user context..
is a human right..  These are global political issues way beyond the IETF's
reach.

Nico
--

---

From: "tom p." <[email-masked]>
Date: Tue, 5 Jun 2018 09:49:03 +0100

----- Original Message -----
From: "Barry Leiba" <[email-masked]>
Sent: Tuesday, June 05, 2018 6:50 AM

<tp>

Barry,

This is a cut out and keep e-mail that I shall still be referring to in
10 years time because it summarises so beautifully the problems.  It
also is the kind of data that led me upthread to assert, contentiously,
that only Europeans were likely  to know what it was about since they
had been living with for decades in a way the most Americans had not.  I
mentioned CJK but John rightly pointed out that, worldwide, it was far
worse, with right-to-left, vertical and so on and so the most skills may
now lie further afield.

Two thoughts.  One is that your e-mail displayed superbly (the only
glitch being that my MUA did not differentiate the Cyrillic character)
so I looked at the encoding

Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

so that is something we got brilliantly right a long time ago.  (The bad
news is that I get an ever growing number of messy e-mails from some
e-mail ids who use some complicated Unicode characters instead of ASCII
punctuation; a sort of single quote being the commonest so the
technology, for me, gets misused).

My second thought is that much has been done in the IETF on security in
recent times but have we done enough to at least publicise, if not
eliminate, the scope for evil actors to exploit confusable and suchlike
characters by saying that they SHOULD NOT be used anywhere where it
matters for security - people SHOULD NOT be handed the rope with which
to hang themselves on a plate:-) - I suspect not.

Tom Petch

Barry

---

From: Barry Leiba <[email-masked]>
Date: Tue, 5 Jun 2018 05:10:59 -0400

> 10 years time because it summarises so beautifully the problems.

:-)  Thanks, Tom; glad to help.

> so that is something we got brilliantly right a long time ago.

Indeed; when it's used properly -- which, as you note, it isn't
always, we do have the representation mostly taken care of in email
and web.  Though note that bi-directional issues still come up all the
time, and that's a nasty one.

> to hang themselves on a plate:-) - I suspect not.

Security is a biggie, especially when we're looking at security
interoperability -- when different systems, or different parts of the
same system, answer some of these questions differently, we have a
problem.  This showed up recently in a way that didn't even have I18N
involved: mix one email system that ignores "." in mailbox names with
one that doesn't (for example, Gmail thinks that "barryleiba" and
"barry.leiba" refer to the same mailbox, and Yahoo! thinks they're
different mailboxes) and you have a recipe for a security exploit.
I18N issues can multiply that exposure many times.

And to be clear about my purpose in posting what I did:
I do not want answers in this forum to the questions I raised: the
point was not to do I18N design work on this list.  I brought up the
questions simply to show how difficult the problem can be, and how
there are many aspects to it that even experts may miss.  Let's not
get wrapped up in showing that any particular issue has solutions.
Rather, let's understand how difficult it is to "solve the problem" in
general, and how hard it is to find people with enough breadth of
experience to work on the big-picture issues effectively.

Barry

---

From: [email-masked]
Date: Tue, 05 Jun 2018 05:33:41 -0700 (PDT)

Viktor is spot-on here. A few addtitional comments...

> is an entirely separate question.

And one which is highly context dependent. The most the IETF could possibly
do in this space is document some of the tradeoffs. And I question whether
it's in the IETF's job description to undertake such work.

> Or do, but the participants are talking past each other...

More likely people are hearing what they want to - or have conditioned
themselves to - hear. A lot of times what people are actually saying is, "We
have no choice but to solve this problem, none of the solutions are ideal
because <reasons>, but I think the best we can do is to <do this>."

Somehow the first part of that is lost, leaving just the <do this> part, and
then the accusations of failing to understand - and worse - start flying.

This has been going on since I began participating in the IETF 29 years ago;
and frankly it got tiresome after the first month.

But there are three key differences between then and now.

The first is that back there was critical work to do: Internet email was
limited to ASCII text, charsets were a balkanized disaster, etc. Solving these
problems was difficult both technically and politically, but it was clear that
the benefits far outweighed the costs.

Fast forward to today, and most of those problems are solved: MIME/EAI/IDN,
Unicode/UTF-8, etc. The benefits of working on the remaining issues are far
smaller, and as for the costs ... a previous email in this thread actually had
the unbridled temerity to haul out the old "Americans think ASCII is
sufficient" line. The memories this calls forth are not good.

The second difference is that we now have decades of experience with how these
efforts actually go to look back on. And to be blunt, the track record of the
"experts" isn't exactly stellar.  For example, 25 years ago the expert
consensus was that any solution to the charset problem had to retain all of the
balkanization - and the specific proposal was to use a profile of ISO 2022 for
inline charset selection. As for Unicode, it was regarded as a hot mess
unworthy of any real consideration.

Back in 1992, Nathaniel Borenstein and I wrote in RFC 1341 that:

            It is our hope  that  ISO  10646  or  some  other
            effort  will  eventually define a single world character set
            which can then be specified for use in Internet mail, but in
            the  advance of that definition we cannot specify the use of
            ISO  10646,  Unicode,  or  any  other  character  set  whose
            definition is, as of this writing, incomplete.

This hardly seems like a contentious statement now, but it definitely was at
the time.

Of course anyone can be wrong. But there have been some real whoppers along the
way, and this along with the absolute certainty that accompanied them makes
trust somewhat difficult.

> which trade-offs to make can be the subject of much debate.

Debating trade-offs is fine, even laudable. But a significant number of these
discussions aren't debates, they start out as or devolve into little more than
a series of personal attacks.

And this brings me to the third difference. Back in 1990 my tolerance for
condescension in general and ad-homs in particular was pretty high. (Although
not infinite, as some participants no doubt recall.) These days, not so much,
in  part because I've gotten old and cranky, but also because of changes in our
broader societal discourse. (Enough said on this last point.)

As a result of these differences I'm, shall we say, considerably more selective
in regards to participation. And I strongly doubt I'm alone in this.

> had to walk away unhappy...

For me, EAI's end-to-end requirement is the far and away its most problematic
aspect, and one which in practice looks like it's going to  be ignored,
leading to egregiously incompliant implementations and all that implies.

Even so, I take your point. And I wonder to what extent support for
message/global is actually going to happen, although if just-send-UTF-8 comes
to rule the day you can just treat it as message/rfc822 and be done with it.

> (an alternative would introduce

First, having now done a full EAI implementation and seen how it interoperates,
it's quite clear to me that this is the approach we should have taken. (And
I say this as someone with their name on one of the core EAI documents.)

Second, I hope you realize that the local part interpretation rule is 
BS now and always has been. The problem is agents outside a given
ADMD at least have to be able to compare addresses for equality. (Mailing
lists managers are the obvious example of such an agent, but there are
countless others.)

And since most of these agents have elected to ignore ASCII case, anyone
who tried to take advantage of the supposed ability to have case-sensitive
local parts ends up in a world of hurt eventually.

This is going to be even more fun with EAI. Normalization of some sort is
pretty clearly needed, rule or no rule. And given the lack of specific advice
in the current standards, its not beyond possibility that case insensitivity
will again rule the day.

And yes, this is one of those "somethings" we should be discussing.

				Ned

---

From: [email-masked]
Date: Tue, 05 Jun 2018 10:29:15 -0700 (PDT)

> and some layers at which it is much better to deal with binary strings.

And some where they get conflated.

> or a SIP server, I don't have any of those tools at my disposal.

I can't speak to the DNS resolver or SIP server, but in the
case of an MTA, this is incorrect.

The fact is MTAs deal with quite a few i18n issues, especially if you implement
standards like EAI.

> My

I can assure you user bufuddlement is carefully monitored (support calls being
a cost center and all), and if those measurments point at a component like
an MTA being the cause, it gets communicated. Loudly.

> then attempting to extend the concept piecemeal to a variety of languages.

It's also justified by user expectations. Like it or not, people aren't
computers and aren't terribly good at retaining case.

As I said in my previous response, it's going to be really interesting to see
how this plays out with EAI addresses.

> at all that there is one single right answer to these questions.

Actually, it's quite clear that there is no single "right" answer. But it's
also cleat that there needs to be single answer. Because as I said before,
address comparison has to work for all addresses, even ones outside your
administrative domain.

> So my

If you include infrastructure components like MTAs in your definition of
"network", the answer is no, you can't.

				Ned