Subject: [xmpp] 6122bis: Unicode versions

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 15 Jul 2011 13:44:27 -0600

The good thing about the post-stringprep world is that we have agility
with regard to Unicode versions (a.k.a. "Unicode agility"). No more
being stuck at Unicode 3.2!

The bad thing is that we have Unicode agility. What if my client (or
your server) has Unicode 5.0 but my server has Unicode 6.0? The parties
might differ in their interpretation of certain code points, causing
problems with authentication, stanza routing, etc.

We might be able to mitigate these problems if we had a way to discover
which version of Unicode the other side supports. Using XEP-0030 or
stream features, we'd want a URI for each Unicode version, such as:

http://www.unicode.org/versions/Unicode6.0.0/ (the web page for 6.0)

or:

urn:unicode:versions:6.0.0

Two questions:

1. Do we think that a service discovery feature would be useful here?

2. If so, do we think the URLs from http://www.unicode.org/ are stable
enough, or would people like to have a URN? (I've asked someone from the
Unicode Consortium about a URN namespace for Unicode, but if we don't
think we'll need it then I won't pursue it further.)

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Matt Miller <[email-masked]>
Date: Fri, 15 Jul 2011 14:24:59 -0600

> 1. Do we think that a service discovery feature would be useful here?

I do think some form of announcing or indicating support is worthwhile, but I think a service discovery feature would be too late in the process.  A client (or initiating server) cannot reliably disco the (receiving) server until after it gets through TLS, SASL, and resource bind.  It's not exactly a stream feature, but maybe that's the best place for fit this?

I personally don't care either way, as long as we have *something* (-:

> 
> Peter
> 
> -- 
> Peter Saint-Andre
> https://stpeter.im/
> 
> 
> _______________________________________________
> xmpp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xmpp

- m&m

Matt Miller - <[email-masked]>
Collaboration Software Group - Cisco Systems, Inc.

---

From: Waqas Hussain <[email-masked]>
Date: Sun, 17 Jul 2011 00:54:14 +0500

On Sat, Jul 16, 2011 at 12:44 AM, Peter Saint-Andre <[email-masked]> wrote:
> which version of Unicode the other side supports.

That's the discussion I'm interested in. How can we mitigate Unicode
version incompatibility? And also, stringprep and post-stringprep
incompatability? Has there been anything written on this that I can
read?

I don't really see there being a good solution. The best we might
reasonably be able to do is handle <jid-malformed/> errors and just
accept that either two incompatible entities wont be able to
communicate at all, or one entity might see the other entity as
multiple JIDs. A recommendation that servers prep JIDs on all outgoing
stanzas might fix the latter.

--
Waqas Hussain

---

From: Jehan Pagès <[email-masked]>
Date: Sun, 17 Jul 2011 19:26:45 +0900

Hi,

> 1. Do we think that a service discovery feature would be useful here?

As said ny Matt, the service discovery might be too late. It would be
ok if the only problem was in exchanged stanza. But if the
incompatibility occurs in the JID, then having a service discovery is
no use.
So I guess that should indeed be a stream feature. We could consider
that a client would "negotiate" its known Unicode version with the one
of one's server.

Note also that probably the best way would be that servers would be
able to "re-encode" on the fly contents from one Unicode version to
another, if needed. We cannot expect any user's OS to embed several
versions of Unicode and might indeed end-up in broken link where 2
entities have errors in their communication, or worse cannot
communicate with each other at all (if the inconsistency occurs in the
JID in particular). That's definitely bad.
On the other side, it is easier to push server deployments into being
able to rencode from one version of unicode to another (are there
already libraries doing conversion from UTF-8 to UTF-8, but with
different Unicode version)?

That said, I think a good flow could be that:

(1) a client (let's call him [email-masked]) would see the "unicode
version" stream feature announced by example.net before authentication
(and after TLS); it negotiate this feature by indicating its own
unicode version (this feature negotiation part basically has no reason
to fail). That way the server knows the unicode version that this
client will use to authenticate, send its stanzas, and so on. From
there, this user wants to contact [email-masked].
(2) example.net contacts example.com. In s2s, the unicode version
feature takes another form: they agree on the unicode version they
will always use over the current s2s stream (whereas in c2s, the
client chooses, because we consider it may not be able to convert from
one version to another as easily).
(3) The stanza contents and the recipient jid ([email-masked]) sent by
[email-masked] is converted (if necessary) to the s2s-chosen unicode
version, then example.com receives the stanza.
(4) As example.com normally knows the unicode version used by
[email-masked] (when this one made the same step (1) as [email-masked]
before authenticating), it converts as well (if necessarily) stanza
contents and JID before finally sending to the finale recipient.

This way, most of the complexity is pushed on the servers (they have
to convert from one unicode version to another, only when necessary),
while the clients have nothing to do other than advertizing (before
authentication) the unicode version they will use during all the file
of the c2s stream. This is pretty good in the XMPP design and I think
it deals most of the issue.

What do you think of this flow? I could write this XEP (that would be
my first own XEP!).

> think we'll need it then I won't pursue it further.)

I think that we are not the one to ask the question about the URL
being stable. We should ask the Unicode consortium if they have any
politics on URL stability which allows us to consider them stable. If
they are not, then yes URN may be better solution.

Jehan

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 18 Jul 2011 16:16:56 -0600

On 7/16/11 1:54 PM, Waqas Hussain wrote:
> On Sat, Jul 16, 2011 at 12:44 AM, Peter Saint-Andre<[email-masked]>  wrote:

So far, I am not too concerned about incompatibilities between Unicode 
versions. See for example:

https://datatracker.ietf.org/doc/draft-faltstrom-5892bis/

As you can see from that document, only three rather obscure code points 
changed in backward-incompatible ways between Unicode 5.0 and Unicode 6.0.

Naturally, the changes between Unicode 3.2 (hardcoded into stringprep) 
and Unicode 6.0 were more substantial. Most of those changes were new 
code points, not code points that changed in backward-incompatible ways. 
During the transition from IDNA2003 to IDNA2008, in practice the most 
troublesome code points were:

    00DF (LATIN SMALL LETTER SHARP S)
    03C2 (GREEK SMALL LETTER FINAL SIGMA)

See http://tools.ietf.org/html/rfc5894#section-7.2 for details (there 
are other troublesome characters, but those were the worst because they 
were more widely deployed). Domain registrars know about those code 
points and probably have special processes for dealing with them.

> reasonably be able to do is handle<jid-malformed/>  errors and just
> communicate at all,

Unfortunate, but possible in a small number of cases.

> or one entity might see the other entity as
> multiple JIDs.

How so?

> A recommendation that servers prep JIDs on all outgoing

s/recommendation/requirement/ :)

But yes.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Waqas Hussain <[email-masked]>
Date: Tue, 19 Jul 2011 04:12:45 +0500

> changed in backward-incompatible ways between Unicode 5.0 and Unicode 6.0.

That's reassuring.

> Naturally, the changes between Unicode 3.2 (hardcoded into stringprep) and
> Unicode 6.0 were more substantial. Most of those changes were new code
> points, not code points that changed in backward-incompatible ways. During
> the transition from IDNA2003 to IDNA2008, in practice the most troublesome
> code points were:
>
> See http://tools.ietf.org/html/rfc5894#section-7.2 for details (there are
> other troublesome characters, but those were the worst because they were
> more widely deployed). Domain registrars know about those code points and
> probably have special processes for dealing with them.

I was mainly concerned about things like Jehan's suggestion of
're-encoding'. I don't think that's somewhere we want to go.

>> or one entity might see the other entity as

See below.

> s/recommendation/requirement/ :)

Note what I mean here is that some servers while verifying JIDs on
outgoing stanzas don't actually replace the to/from unprepped values
with the prepped ones in what gets sent over the wire. So the JID
[email-masked] is sent as [email-masked] over the wire, not as
[email-masked]. This will interact badly with IDNA2003 and IDNA2008
having different outputs for a given input.

Effectively, if given unprepped JID string X, which IDNA2008 preps to
string Y, but IDNA2003 accepts without prepping, and given the above
server behavior, a client could receive stanzas from both X and Y, and
treat them as the different JIDs when they are in fact the same
(that's just one example, others, e.g., the reverse could also be
possible). I haven't verified that this is actually possible, but IIRC
the two specs don't have the same transformations in many cases. How
compatible are the IDNA2003 vs IDNA2008 transformations?

--
Waqas Hussain

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 18 Jul 2011 20:52:52 -0600

On 7/18/11 5:12 PM, Waqas Hussain wrote:
> On Tue, Jul 19, 2011 at 3:16 AM, Peter Saint-Andre<[email-masked]>  wrote:

Agreed.

> having different outputs for a given input.

I see your point, and I agree that prepping all outbound JIDs would help 
to avoid this problem. Paradoxically, prepping inbound JIDs would hurt, 
not help (see below).

> compatible are the IDNA2003 vs IDNA2008 transformations?

As explained in RFC 5894, in fact there are few characters that are 
interpreted differently in IDNA2008 compared to IDNA2003: Eszett, Greek 
Final Sigma, Zero Width Joiner, and Zero Width Non-Joiner.

So, for instance, in IDNA2003 you could register fussball.de but not 
fußball.de because ß was mapped to "ss" (see Appendix B of RFC 3454, 
i.e. "Table B.2" as invoked by Nameprep in RFC 3491). In IDNA2008, ß is 
a distinct, allowable character, so you can now register fußball.de. 
Clearly the registrar for .de needs to know this when accepting 
registrations, because it might want to reserve fußball.de if 
fussball.de is already registered, automatically assign fußball.de to 
the registrant for fussball.de, or apply some other policy.

Now, the same is true for Nodeprep as used to stringprep the localpart 
of JIDs -- see Appendix A of RFC 3920. So in the current XMPP network 
(RFC 6122), you could register an account like [email-masked] but 
if you tried to register [email-masked] it would be stringprepped 
to [email-masked]. If we migrate to 6122bis, fußball would be 
allowed as a username. Therefore a 6122bis-compliant server might allow 
both accounts to be registered and might route stanzas from both 
[email-masked] and [email-masked] over an s2s link to your 
server. But if your 6122-compliant server stringpreps JIDs on incoming 
stanzas then it would consider both of those JIDs to be the same, since 
it doesn't consider [email-masked] to be valid (if your server 
doesn't stringprep JIDs on incoming stanzas then it would return a 
<jid-malformed/> error instead). Clearly this opens up the possibility 
of some attacks -- if I know you are subscribed to [email-masked] 
for the latest scores, I could register [email-masked] and send you 
bogus information.

As mentioned, this applies to four characters that are allowed in 
IDNA2008 and PRECIS (with the caveat that PRECIS isn't done yet!) but 
that are mapped to other characters (ß mapped to ss, ς mapped to σ) or 
to nothing (for Zero Width Joiner and Zero Width Non-Joiner) in IDNA2003 
and Nodeprep. In these four cases, the post-stringprep technologies are 
more inclusive and we could have problems of the kind I've outlined 
above (not "double JIDs" but certain new JIDs registered with 
6122bis-compliant servers that would be treated as equivalent to old 
JIDs by 6122-compliant software). Any migration plan we devise will need 
to provide guidelines for handling these cases.

All of this is a lot easier if existing servers reject JIDs that they 
consider malformed, instead of prepping them. However, note that RFCs 
3920 and 6120 don't say that a server MUST reject malformed JIDs, so 
some existing servers might be liberal in what they accept, which in 
this case leads to bad consequences.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Waqas Hussain <[email-masked]>
Date: Tue, 19 Jul 2011 10:49:28 +0500

On Tue, Jul 19, 2011 at 7:52 AM, Peter Saint-Andre <[email-masked]> wrote:
>>> and
> I see your point, and I agree that prepping all outbound JIDs would help to
> avoid this problem. Paradoxically, prepping inbound JIDs would hurt, not
> help (see below).

It helps too: I could create the JID [email-masked] on a
IDNA2003 server, and send stanzas as both [email-masked] and
[email-masked]. If my server passes the 'from' attribute as-is,
I can make myself seem like two entities to an IDNA2008 server/client.
Not too worrying a problem I suppose.

> fußball.de because ß was mapped to "ss" (see Appendix B of RFC 3454, i.e.
> "Table B.2" as invoked by Nameprep in RFC 3491). In IDNA2008, ß is a
> distinct, allowable character, so you can now register fußball.de. Clearly
> the registrar for .de needs to know this when accepting registrations,
> because it might want to reserve fußball.de if fussball.de is already
> registered, automatically assign fußball.de to the registrant for
> fussball.de, or apply some other policy.
>
> Now, the same is true for Nodeprep as used to stringprep the localpart of
> JIDs -- see Appendix A of RFC 3920. So in the current XMPP network (RFC
> 6122), you could register an account like [email-masked] but if you
> tried to register [email-masked] it would be stringprepped to
> [email-masked]. If we migrate to 6122bis, fußball would be allowed as
> a username. Therefore a 6122bis-compliant server might allow both accounts
> to be registered and might route stanzas from both [email-masked] and
> [email-masked] over an s2s link to your server. But if your
> 6122-compliant server stringpreps JIDs on incoming stanzas then it would
> consider both of those JIDs to be the same, since it doesn't consider
> [email-masked] to be valid (if your server doesn't stringprep JIDs on
> incoming stanzas then it would return a <jid-malformed/> error instead).
> Clearly this opens up the possibility of some attacks -- if I know you are
> subscribed to [email-masked] for the latest scores, I could register
> [email-masked] and send you bogus information.

This makes a bit nervous.

I assume this affects more than just XMPP? I'm interested in hearing
what non-XMPP folks might have to say on the matter.

> As mentioned, this applies to four characters that are allowed in IDNA2008
> and PRECIS (with the caveat that PRECIS isn't done yet!) but that are mapped
> to other characters (ß mapped to ss, ς mapped to σ) or to nothing (for Zero
> Width Joiner and Zero Width Non-Joiner) in IDNA2003 and Nodeprep. In these
> four cases, the post-stringprep technologies are more inclusive and we could
> have problems of the kind I've outlined above (not "double JIDs" but certain
> new JIDs registered with 6122bis-compliant servers that would be treated as
> equivalent to old JIDs by 6122-compliant software). Any migration plan we
> devise will need to provide guidelines for handling these cases.

+1.

> consider malformed, instead of prepping them. However, note that RFCs 3920
> and 6120 don't say that a server MUST reject malformed JIDs, so some
> existing servers might be liberal in what they accept, which in this case
> leads to bad consequences.

I think that's what existing servers probably do. We can check/ask
them. The ones which do stringprep anyway :)

--
Waqas Hussain

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 19 Jul 2011 10:22:23 -0600

On 7/18/11 11:49 PM, Waqas Hussain wrote:
> On Tue, Jul 19, 2011 at 7:52 AM, Peter Saint-Andre<[email-masked]>  wrote:

Sorry, I was not clear: I meant prepping on inbound s2s stanzas. The 
principle is that the first server processing a stanza must enforce the 
rules. I think your example of the example.com server allowing a client 
to send stanzas from either [email-masked] or [email-masked] 
is wrong, because an XMPP server that implements Nodeprep would have to 
prep the fußball username to fussball when the client sends the stanza.

>> incoming stanzas then it would return a<jid-malformed/>  error instead).

Me too. But it's unavoidable if we decide to migrate from stringprep to 
PRECIS, and we already have this issue for domain names (it's just that 
we assume the zone administrators will take care of it for us).

> what non-XMPP folks might have to say on the matter.

Well, I think we have more deployment of stringprep than any other 
non-IDN application technology -- I don't hear about much problems with 
non-ASCII characters in the context of IMAP or POP using SASLprep, or of 
LDAPprep, or of the iSCSI stringprep profile. However, this is something 
that definitely needs to be discussed in the PRECIS WG.

> them. The ones which do stringprep anyway :)

Yes, it would be good to complete a survey of existing server 
implementations on this point.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Florian Zeitz <[email-masked]>
Date: Wed, 20 Jul 2011 13:46:58 +0200

> problems with authentication, stanza routing, etc.
First I'd like to say that I expect some guidance on this from the work
of the PRECIS WG. I'd consider a lack of that at least as bad as being
stuck with Unicode 3.2. Therefore I think part of this discussion should
probably be held over there.

Another thing is that I think we first need to understand what the
possible problems are, before we try to avoid them.
I think we should safely be able to assume that a valid localpart stays
a valid localpart even with new Unicode versions (this should be
possible by adding entries to a list of exception. We would of course
hope/expect that this is not needed very often, if at all).
What I consider more problematic is normalizaion. I don't know how
probable it is, but there might be cases where
NFC(A, Unicode5) == NFC(B, Unicode5), but
NFC(A, Unicode6) != NFC(B, Unicode6), or
NFC(A, Unicode5) != NFC(B, Unicode6), etc.
And that is something we can't really fix. So we should look into
problems that this get's us into. This is possibly the same problem as
the stringprep vs. IDNA2008 discussion just in a different form.

One of the problems this would yield is that two accounts that are
different with Unicode 6 are registered on a server A. Another server B
using Unicode 5 considers both the same. Now if a user on server B tries
to send to either of the two accounts on server A messages will only end
up at one of them, potentially the wrong one.
Also in this case it's not usfull for server A to know that server B
uses Unicode 5, because he still can't tell which of the two accounts B
meant to send to, at best he could warn about this situation.

Now maybe there are situations where having this information can be
usfull, but I'd rather have them thought through before painting the
bikeshed by choosing a URI.

--
Florian Zeitz

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 20 Jul 2011 21:21:55 -0600

> probably be held over there.

I'm sure it will be.

> hope/expect that this is not needed very often, if at all).

The Unicode Consortium tries hard to ensure that the essential 
properties of code points don't change across versions, but it does happen.

> NFC(A, Unicode5) != NFC(B, Unicode6), etc.

Again, we need to look at specific characters (code points). I don't 
know of any cases where that kind of problem occurs.

> the stringprep vs. IDNA2008 discussion just in a different form.

The bigger challenge with normalization is the migration from NFKC to 
NFC. The IDNA2003 (stringrep) to IDNA2008 (non-stringprep) transition 
experienced this issue, because IDNA2003 used NFKC and IDNA2008 uses 
NFC. The IDNA folks solved this problem by prohibiting characters with 
compatibility equivalents in IDNA2008.

> using Unicode 5 considers both the same.

I know of no cases like the one you are worried about here, at least 
between recent Unicode versions. There are four cases to worry about 
between Unicode 3.2 and Unicode 5.0/6.0, especially eszett and Greek 
final sigma as we've already discussed.

> Now if a user on server B tries

This is theoretically possible, but highly unlikely as far as I know. If 
such cases occur in the future, both IDNA and PRECIS have something 
called a BackwardCompatibility list. But it has not yet been used in 
IDNA, see https://datatracker.ietf.org/doc/draft-faltstrom-5892bis/

> meant to send to, at best he could warn about this situation.

Warnings are better than nothing. :)

> bikeshed by choosing a URI.

I think that a URI will be useful when a Unicode X entity is talking to 
a Unicode X-1 entity, because the Unicode X entity can provide warnings 
or consult a BackwardCompatibility list. However, in general I think 
that the Unicode X server will need to use preventive measures to avoid 
such problems in the first place (e.g., controlling the registration of 
JIDs that contain code points that have compatibility issues across 
Unicode versions).

/psa

---

From: Florian Zeitz <[email-masked]>
Date: Thu, 21 Jul 2011 10:51:40 +0200

That's reassuring.

> compatibility equivalents in IDNA2008.
Right, they solved this by prohibiting compatibly decomposable
characters *cough*. However I consider that issue separate from Unicode
versions which this thread is about, or am I missing something?

> JIDs that contain code points that have compatibility issues across
I'm aware of the BackwardCompatibility list, however I was under the
impression that this is only used for classification/validity checking
and not for normalization. Am I mistaken here?
And again, we can talk about preventing "issues" all we want, but I have
not yet seen a single Unicode version issue (under the premisse that
mine is rather theoretical). I'd like to at least have an idea of what
could go wrong.

--
Florian Zeitz