Subject: [xmpp] 6122bis: U-labels and A-labels in domainparts

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 19 Mar 2012 14:06:56 -0600

6122bis currently specifies that each A-label in a domainpart SHOULD
be converted to a U-label (for background, see RFC 5890). This means
that we prefer U-labels (i.e., labels that can natively include most
Unicode letters and digits), but we do not prohibit A-labels (i.e.,
labels that are not converted into the ASCII-compatible encoding via
Punycode [RFC 3492]). This is consistent with both RFC 3920 and RFC
6122. Now, in IETF-speak, "SHOULD" basically means "you must do X
unless you have a really good reason not to". So I wonder: what might
be a really good reason to use A-labels instead of U-labels? Would it
perhaps be better to tighten up the spec by saying that each
internationalized label MUST be a U-label? What do we gain by allowing
A-labels? How do existing implementations behave here?

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Joe Hildebrand <[email-masked]>
Date: Tue, 20 Mar 2012 13:39:07 -0600

Hat=individual

Do we specify how to compare U- and A- encodings of what, to humans,
probably mean the same thing?  If not, I'd argue we should just pick
U-labels and have a single mechanism for comparison.

On 3/19/12 2:06 PM, "Peter Saint-Andre" <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAk9nkeAACgkQNL8k5A2w/vwwpwCfcI5ynBICf/rqfTsQSP8QgPut
> zAsAn0mxEREe9BUhcjisDEvFhn7HBOey
> =vxj4
> -----END PGP SIGNATURE-----
> _______________________________________________
> xmpp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xmpp

-- 
Joe Hildebrand

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 20 Mar 2012 15:41:11 -0600

On 3/20/12 1:39 PM, Joe Hildebrand wrote:
> Hat=individual
> 
> Do we specify how to compare U- and A- encodings of what, to
> humans, probably mean the same thing?

Well, I think we'd need to compare U-to-U or A-to-A (if a machine has
labels in both forms it would need to translate them all to U or A
before performing its comparison operations). Or do you mean comparing
U to user expectations and A to user expectations? It's not clear to
me exactly what that means -- machines can compare U-to-U or A-to-A,
but they don't know about things that might have meaning to humans. We
could call that the inside-of-the-mind problem, which is the converse
of the side-of-the-bus problem. :)

> If not, I'd argue we should just pick U-labels and have a single
> mechanism for comparison.

Having a single mechanism does simplify matters.

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Joe Hildebrand <[email-masked]>
Date: Tue, 20 Mar 2012 15:44:11 -0600

I was thinking we could define a heuristic to determine that a label was a
punycoded A-label, and specify that it had to be canonicalized into a
U-label (then nameprep-equivalent canonicalized) before comparison with
another label.  However, that sounds like a PITA to specify and implement,
so I'd rather we just say they're always U-labels.

On 3/20/12 3:41 PM, "Peter Saint-Andre" <[email-masked]> wrote:

> iEYEARECAAYFAk9o+XYACgkQNL8k5A2w/vweGgCfbP3LOPzrYL1H66eG4oBt1rcu
> jq0AniAWPCgTDhMPx/nLpo6o416Jbjc4
> =EvGc

-- 
Joe Hildebrand

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 20 Mar 2012 15:46:34 -0600

On 3/20/12 3:44 PM, Joe Hildebrand wrote:
> I was thinking we could define a heuristic to determine that a
> label was a punycoded A-label, and specify that it had to be
> canonicalized into a U-label (then nameprep-equivalent
> canonicalized) before comparison with another label.  However, that
> sounds like a PITA to specify and implement, so I'd rather we just
> say they're always U-labels.

I think IDNA2008 might define such a heuristic, but I tend to agree
that if we can move from SHOULD to MUST on U-labels then we can avoid
that madness.

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Florian Zeitz <[email-masked]>
Date: Wed, 21 Mar 2012 02:25:46 +0100

Am 20.03.2012 22:46, schrieb Peter Saint-Andre:
>> canonicalized) before comparison with another label.  However,
>> that sounds like a PITA to specify and implement, so I'd rather
>> we just say they're always U-labels.
> 
> I think IDNA2008 might define such a heuristic, but I tend to
> agree that if we can move from SHOULD to MUST on U-labels then we
> can avoid that madness.
> 
Correct me if I'm wrong here, but I was under the impression we could
perform comparison on the result of the ToASCII operation. A-labels
would remain unaffected by this and U-labels would become A-labels.

That said I would definitely prefer if JIDs were only allowed to
contain U-labels.
I'm somewhat concerned about how to enforce this in user input though.
At least libidn apparently allows you to use ToUnicode on U-labels (as
a NOP), which means it's safe to use on all user input. However this
doesn't strike me as an elegant solution either.

Regards,
Florian Zeitz

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 20 Mar 2012 20:00:06 -0600

On 3/20/12 7:25 PM, Florian Zeitz wrote:
>>> that sounds like a PITA to specify and implement, so I'd
>>> rather we just say they're always U-labels.
> 
>> agree that if we can move from SHOULD to MUST on U-labels then
>> we can avoid that madness.
> 
> Correct me if I'm wrong here, but I was under the impression we
> could perform comparison on the result of the ToASCII operation.
> A-labels would remain unaffected by this and U-labels would become
> A-labels.

Sure, we can do that. The question is, what do we want to send over
the wire?

> That said I would definitely prefer if JIDs were only allowed to 

Actually, our specs might already say that. :)

Let's look at RFC 6122...

   A domainpart consisting of a fully qualified domain name MUST be an
   "internationalized domain name" as defined in [IDNA2003]; that is, it
   MUST be "a domain name in which every label is an internationalized
   label" and MUST follow the rules for construction of
   internationalized domain names specified in [IDNA2003].  When
   preparing a text label (consisting of a sequence of UTF-8 encoded
   Unicode code points) for representation as an internationalized label
   in the process of constructing an XMPP domainpart or comparing two
   XMPP domainparts, an application MUST ensure that for each text label
   it is possible to apply without failing the ToASCII operation
   specified in [IDNA2003] with the UseSTD3ASCIIRules flag set (thus
   forbidding ASCII code points other than letters, digits, and
   hyphens).  If the ToASCII operation can be applied without failing,
   then the label is an internationalized label.  (Note: The ToASCII
   operation includes application of the [NAMEPREP] profile of
   [STRINGPREP] and encoding using the algorithm specified in
   [PUNYCODE]; for details, see [IDNA2003].)  Although XMPP applications
   do not communicate the output of the ToASCII operation (called an
   "ACE label") over the wire, it MUST be possible to apply that
   operation without failing to each internationalized label.  If an
   XMPP application receives as input an ACE label, it SHOULD convert
   that ACE label to an internationalized label using the ToUnicode
   operation (see [IDNA2003]) before including the label in an XMPP
   domainpart that will be communicated over the wire on an XMPP network
   (however, instead of converting the label, there are legitimate
   reasons why an application might instead refuse the input altogether
   and return an error to the entity that provided the offending data).

So, we see that although it must be possible to apply the IDNA2003
ToASCII operation to a domainpart, the output of the ToASCII operation
(an ACE label, which in IDNA2008 we'd call an A-label) is not
communicated over the wire; if an XMPP application receives the
"offending data" of an ACE label, it SHOULD convert it into a U-label
for sending over the wire, but it's allowed to return an error instead.

I conclude that we've never allowed A-labels on the wire.

(IMHO the use of SHOULD was misleading in RFC 3920 / RFC 6122, because
it followed the kind of "SHOULD ... but MAY instead" construction that
we try to avoid these days.)

> I'm somewhat concerned about how to enforce this in user input
> though. At least libidn apparently allows you to use ToUnicode on
> U-labels (as a NOP), which means it's safe to use on all user
> input. However this doesn't strike me as an elegant solution
> either.

Are you concerned that users would enter A-labels? I see that as
exceedingly unlikely.

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Florian Zeitz <[email-masked]>
Date: Wed, 21 Mar 2012 03:31:10 +0100

On 2012-03-21 03:00, Peter Saint-Andre wrote:
>> A-labels would remain unaffected by this and U-labels would
>> become A-labels.
> 
> Sure, we can do that. The question is, what do we want to send
> over the wire?
> 
Indeed, or at least if the answer to that question is "only U-labels"
the comparison becomes less of an issue.

> A domainpart consisting of a fully qualified domain name MUST be
> an "internationalized domain name" as defined in [IDNA2003]; that
> is, it MUST be "a domain name in which every label is an
> internationalized label" and MUST follow the rules for construction
> of internationalized domain names specified in [IDNA2003].  When 
> Unicode code points) for representation as an internationalized
> label in the process of constructing an XMPP domainpart or
> comparing two XMPP domainparts, an application MUST ensure that for
> each text label it is possible to apply without failing the ToASCII
> operation specified in [IDNA2003] with the UseSTD3ASCIIRules flag
> set (thus forbidding ASCII code points other than letters, digits,
> and hyphens).  If the ToASCII operation can be applied without
> failing, then the label is an internationalized label.  (Note: The
> ToASCII operation includes application of the [NAMEPREP] profile
> of [STRINGPREP] and encoding using the algorithm specified in 
> [PUNYCODE]; for details, see [IDNA2003].)  Although XMPP
> applications do not communicate the output of the ToASCII operation
> (called an "ACE label") over the wire, it MUST be possible to apply
> that operation without failing to each internationalized label.  If
> an XMPP application receives as input an ACE label, it SHOULD
> convert that ACE label to an internationalized label using the
> ToUnicode operation (see [IDNA2003]) before including the label in
> an XMPP domainpart that will be communicated over the wire on an
> XMPP network (however, instead of converting the label, there are
> legitimate reasons why an application might instead refuse the
> input altogether and return an error to the entity that provided
> the offending data).
> 
> ToASCII operation to a domainpart, the output of the ToASCII
> operation (an ACE label, which in IDNA2008 we'd call an A-label) is
> not communicated over the wire; if an XMPP application receives
> the "offending data" of an ACE label, it SHOULD convert it into a
> U-label for sending over the wire, but it's allowed to return an
> error instead.
> 
> (IMHO the use of SHOULD was misleading in RFC 3920 / RFC 6122,
> because it followed the kind of "SHOULD ... but MAY instead"
> construction that we try to avoid these days.)
> 
I think this is actually underspecified in several ways.
Specifically we are not mandating that domainparts have to be U-labels
at all. An A-label is a valid "internationalized label", since (as
noted before) ToASCII applies to it just fine (or at least that is my
understanding). Also if we don't allow A-labels on the wire, why
SHOULD an application ever have to convert them?
Taking that text too literal it is also valid to send A-labels as long
as they were not received from somewhere else.
I think implementers usually did take this to mean that only U-labels
should be send over the wire, but I think there are several loop holes
to close here.

>> though. At least libidn apparently allows you to use ToUnicode
>> on U-labels (as a NOP), which means it's safe to use on all user 
I agree it's unlikely, in the spirit of Postel's law I do however at
least feel discomfort about this ;).

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 20 Mar 2012 21:36:13 -0600

On 3/20/12 8:31 PM, Florian Zeitz wrote:
>>>  could perform comparison on the result of the ToASCII
>>> operation. A-labels would remain unaffected by this and
>>> U-labels would become A-labels.
> 
> Indeed, or at least if the answer to that question is "only
> U-labels" the comparison becomes less of an issue.
> 
>>> That said I would definitely prefer if JIDs were only allowed
>>> to contain U-labels.
> 
>> an "internationalized domain name" as defined in [IDNA2003];
>> that is, it MUST be "a domain name in which every label is an 
>> internationalized label" and MUST follow the rules for
>> construction of internationalized domain names specified in
>> [IDNA2003].  When preparing a text label (consisting of a
>> sequence of UTF-8 encoded Unicode code points) for representation
>> as an internationalized label in the process of constructing an
>> XMPP domainpart or comparing two XMPP domainparts, an application
>> MUST ensure that for each text label it is possible to apply
>> without failing the ToASCII operation specified in [IDNA2003]
>> with the UseSTD3ASCIIRules flag set (thus forbidding ASCII code
>> points other than letters, digits, and hyphens).  If the ToASCII
>> operation can be applied without failing, then the label is an
>> internationalized label.  (Note: The ToASCII operation includes
>> application of the [NAMEPREP] profile of [STRINGPREP] and
>> encoding using the algorithm specified in [PUNYCODE]; for
>> details, see [IDNA2003].)  Although XMPP applications do not
>> communicate the output of the ToASCII operation (called an "ACE
>> label") over the wire, it MUST be possible to apply that
>> XMPP application receives as input an ACE label, it SHOULD 
>> ToUnicode operation (see [IDNA2003]) before including the label
>> in an XMPP domainpart that will be communicated over the wire on
>> an XMPP network (however, instead of converting the label, there
>> are legitimate reasons why an application might instead refuse
>> the input altogether and return an error to the entity that
>> provided the offending data).
> 
>> So, we see that although it must be possible to apply the
>> IDNA2003 ToASCII operation to a domainpart, the output of the
>> ToASCII operation (an ACE label, which in IDNA2008 we'd call an
>> A-label) is not communicated over the wire; if an XMPP
>> application receives the "offending data" of an ACE label, it
>> SHOULD convert it into a U-label for sending over the wire, but
>> it's allowed to return an error instead.
> 

It's underspecified in a lot of ways -- that's why we're working on
6122bis. :) Please understand that RFC 6122 is based on IDNA2003,
whereas 6122bis is based on IDNA20008.

> Specifically we are not mandating that domainparts have to be
> U-labels at all. An A-label is a valid "internationalized label",
> since (as noted before) ToASCII applies to it just fine (or at
> least that is my understanding). Also if we don't allow A-labels on
> the wire, why SHOULD an application ever have to convert them?

If we way that a server must enforce the rules but a client doesn't
have to, then a server might receive an A-label from the client.

> Taking that text too literal it is also valid to send A-labels as
> long as they were not received from somewhere else.

Too literal indeed. Anyway, it's better to look at 6122bis. I quoted
RFC 6122 only for historical purposes.

> I think implementers usually did take this to mean that only
> U-labels should be send over the wire, but I think there are
> several loop holes to close here.

Agreed. Please review draft-ietf-xmpp-6122bis and send your feedback. :)

>>>  though. At least libidn apparently allows you to use
>>> ToUnicode on U-labels (as a NOP), which means it's safe to use
>>> on all user input. However this doesn't strike me as an elegant
>>> solution either.
> 
> I agree it's unlikely, in the spirit of Postel's law I do however
> at least feel discomfort about this ;).

Let's see, would a Czech person type #2 instead of #1?

1. Pročprostěnemluvíčesky.cz

2. xn--Proprostnemluvesky-uyb24dma41a.cz

I rather doubt it, but I suppose it's possible.

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Florian Zeitz <[email-masked]>
Date: Wed, 21 Mar 2012 22:51:17 +0100

Am 21.03.2012 04:36, schrieb Peter Saint-Andre:
>> Taking that text too literal it is also valid to send A-labels
>> as long as they were not received from somewhere else.
> 
> Too literal indeed. Anyway, it's better to look at 6122bis. I
> quoted RFC 6122 only for historical purposes.
> 
Which is also my reason for getting into it's details. I think we need
to have a clear understanding of what the current RFCs say, to find a
(backwards compatible) way forward.
I disagree with your assessment that A-labels (ACE labels) were always
disallowed on the wire.

> Agreed. Please review draft-ietf-xmpp-6122bis and send your
I shall do a proper full review once I have more spare time.

Some notes concerning the issue discussed in this thread:
As it has been since RFC 3920 conversion of A-labels is only specified
in certain cases.
I.e. a JID's domainpart can contain an A-label, just if you want to do
$something specific with it SHOULD that label be converted to a U-label.

I personally don't see the compelling reason for that approach.
Is there anything wrong with saying that a JID's domainpart
SHOULD(/MUST?) be a internationalized domain name containing only U-
and NR-LDH-labels?

Particularly the current wording in 6122bis ("For the purposes of
communication over XMPP") seems incredibly vague to me.
Amongst other things it seems to exempt "for comparison", which is one
of the cases where we certainly need mapping, normalization, and some
coherent representation of labels.

Note that the point I'm trying to make here is specifically that I'd
want the domainpart of a JID to always follow certain rules, as
opposed to making arrangements for it to follow certain rules in
certain cases.

If worse comes to worse I'm willing to accept that a domainpart is a
general IDNA-aware domain name slot (i.e. can include A-labels).
Having rules that amount to "It's a general internationalized domain
name, except in 'to' and 'from' attributes and some other cases, where
the server converts A-labels to U-labels, except when there was a good
reason for the server not to" will almost certainly lead to problems.

>>> Are you concerned that users would enter A-labels? I see that
>>> as exceedingly unlikely.
> 
>> I agree it's unlikely, in the spirit of Postel's law I do
>> however at least feel discomfort about this ;).
> 
What I fear is that some users (unfortunately also/mostly/even
tech-savvy ones) are too convinced that internationalization is often
broken.
E.g. some people have apparently taught themselves to type ü, ä, ö and
ß as ue, ae, oe and ss for certain uses (usernames etc.) because
"everything else often causes problems". I suspect similar prejudice
might exist for labels too.
E.g. the reason the IDNA related testing at the XMPP Summit in
Brussels initially failed was that the server was configured using
A-labels and would not accept the U-label version thereof.

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 21 Mar 2012 16:45:31 -0600

On 3/21/12 3:51 PM, Florian Zeitz wrote:
> Which is also my reason for getting into it's details. I think we
> need to have a clear understanding of what the current RFCs say, to
> find a (backwards compatible) way forward. I disagree with your
> assessment that A-labels (ACE labels) were always disallowed on the
> wire.

What is your interpretation? I agree that RFC 3920 and RFC 6122 were
not worded very clearly, but it seems to me that ACE labels were not
permitted on the wire, e.g., "XMPP applications do not communicate the
output of the ToASCII operation (called an "ACE label") over the
wire". Because we want to be backward compatible, it's important to
know what we specified before (and how implementers understood the spec).

But in any case, I think the right thing to do now is to say that a
domainpart MUST NOT include A-labels.

> Some notes concerning the issue discussed in this thread: As it has
> been since RFC 3920 conversion of A-labels is only specified in
> certain cases. I.e. a JID's domainpart can contain an A-label, just
> if you want to do $something specific with it SHOULD that label be
> converted to a U-label.
> 
> I personally don't see the compelling reason for that approach. Is
> there anything wrong with saying that a JID's domainpart 
> SHOULD(/MUST?) be a internationalized domain name containing only
> U- and NR-LDH-labels?

That would be my preference.

> communication over XMPP") seems incredibly vague to me.

I am not sure that it is *incredibly* vague. The contrast was with
XMPP IRIs, i.e., representation of a JID in IRI form, not JIDs as
natively used on the XMPP network.

> Amongst other things it seems to exempt "for comparison", which is
> one of the cases where we certainly need mapping, normalization,
> and some coherent representation of labels.

That was not the intent.

> Note that the point I'm trying to make here is specifically that
> I'd want the domainpart of a JID to always follow certain rules,
> as opposed to making arrangements for it to follow certain rules
> in certain cases.

Agreed.

> If worse comes to worse I'm willing to accept that a domainpart is
> a general IDNA-aware domain name slot (i.e. can include A-labels). 
> Having rules that amount to "It's a general internationalized
> domain name, except in 'to' and 'from' attributes and some other
> cases, where the server converts A-labels to U-labels, except when
> there was a good reason for the server not to" will almost
> certainly lead to problems.

Right, that would lead to all sorts of trouble.

>>>> Are you concerned that users would enter A-labels? I see
>>>> that as exceedingly unlikely.
> 
> tech-savvy ones) are too convinced that internationalization is
> often broken. E.g. some people have apparently taught themselves to
> type ü, ä, ö and ß as ue, ae, oe and ss for certain uses (usernames
> etc.) because "everything else often causes problems". I suspect
> similar prejudice might exist for labels too. E.g. the reason the
> IDNA related testing at the XMPP Summit in Brussels initially
> failed was that the server was configured using A-labels and would
> not accept the U-label version thereof.

That's true, but I doubt that it would lead people to type A-labels,
which are horrendously ugly. See above. :)

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Florian Zeitz <[email-masked]>
Date: Thu, 22 Mar 2012 01:39:04 +0100

Am 21.03.2012 23:45, schrieb Peter Saint-Andre:
>>>> Taking that text too literal it is also valid to send
>>>> A-labels as long as they were not received from somewhere
>>>> else.
> 
>> Which is also my reason for getting into it's details. I think
>> we need to have a clear understanding of what the current RFCs
>> say, to find a (backwards compatible) way forward. I disagree
>> with your assessment that A-labels (ACE labels) were always
>> disallowed on the wire.
> 
> What is your interpretation? I agree that RFC 3920 and RFC 6122
> were not worded very clearly, but it seems to me that ACE labels
> were not permitted on the wire, e.g., "XMPP applications do not
> label") over the wire". Because we want to be backward compatible,
> it's important to know what we specified before (and how
> implementers understood the spec).
> 
"Remember that you asked for it" :)
I do believe that ACE label were/are not forbidden.
The over-shortened reasoning for this is that there is no "MUST NOT"
requirement to not send them.

The more/overly detailed version goes something like this:
There are (IMHO) just two places that could restrict ACE labels in
domain parts:
1. «Although XMPP applications do not communicate the output of the
    ToASCII operation (called an "ACE label") over the wire, it MUST be
    possible to apply that operation without failing to each
    internationalized label.»
First of all it says "do not" instead of "MUST NOT", which makes it
seem more like a side note.
That in mind it is easy to derive the following procedure from the text:
a) apply ToASCII to each label
b) verify it succeeded
b1) if it did, use your input to the ToASCII operation as domainpart
b2) otherwise, produce an error

The problem with that approach is that it does not catch ACE labels
that already existed in the input.
Also the ToASCII operation can have a LDH label as output (it passes
through unchanged). In that sense "output of the ToASCII operation"
and "ACE label" are in fact not equivalent. Hence taking this text to
mean that anything that looks like output of the ToASCII operation is
forbidden on the wire, would imply LDH labels are forbidden.

2. «If an XMPP application receives as input an ACE label, it SHOULD
    convert that ACE label to an internationalized label using the
    ToUnicode operation (see [IDNA2003]) before including the label in
    an XMPP domainpart that will be communicated over the wire on an
    XMPP network (however, instead of converting the label, there are
    legitimate reasons why an application might instead refuse the
    input altogether and return an error to the entity that provided
    the offending data).»
As I said before there is a point to be made about the precondition on
having received that label. It is certainly nitpicking, but this text
does not forbid an entity to produce ACE labels and send them.
Additionally this is only a SHOULD requirement, which means
implementations need to prepare for accepting domainparts from
implementations that had a good reason not to follow it. In particular
this is not a "MUST either ..., or ..." in combination with the
parenthetic remark.
The fact that there is text that defines what SHOULD happen when ACE
labels are received on the wire could be taken to imply that they
certainly were allowed (to be send) on the wire.

> But in any case, I think the right thing to do now is to say that
> a domainpart MUST NOT include A-labels.
> 
Agreed. While I do think they were allowed before, I'm also under the
impression that introducing this requirement will not break existing
deployments.
Feedback on whether that impression is correct, especially from
somebody who lives where IDNA domains are somewhat wider deployed,
would obviously be appreciated.

> natively used on the XMPP network.
Unfortunately it is to me, though that may partially be due to the
language barrier.
Especially in the context of XMPP I think "communication over XMPP"
could be interpreted as anything from "instant messaging" to "data
send over an XMPP stream" (where I'd assume the later was the intended
interpretation).

Regards,
Florian Zeitz