Subject: [xmpp] Unicode Version Interop Concerns in JIDs

From: Ralph Meijer <[email-masked]>
Date: Tue, 10 Sep 2019 16:38:50 +0200

Hi,

Recently, there's been a discussion in the XSF Discussion room [1] about 
interop issues in the face of different Unicode versions used for 
processing XMPP Addresses, or JIDs. That particular discussion was 
mostly focused on nicknames in Multi-User Chat (MUC) rooms, which are 
encoded in the resourcepart of a JID, but is a concern for other address 
handling. As I suggested giving this topic a wider audience, I write on 
behalf of those involved in the initial discussion.

Ever since RFC 6122 was obsoleted by RFC 7622 [2], both titled “XMPP: 
Address Format”, resourceprep (which was fixed to Unicode 3.2) was 
replaced by PRÉCIS processing as discussed in section 3.4. This in turn 
the the resourcepart is a OpaqueString profile of the PRECIS 
FreeformClass as defined in RFC 7613 [3], section 4.2 and RFC 7564 [4], 
section 4.3 respectively. The idea is that in the face of newer Unicode 
versions, application can make use of the new codepoints therein.

RFC 7622 has extensive texts on JID handling, but there is uncertaintly 
over when servers, services like MUC, and clients, should be liberal or 
strict when checking JIDs. Different implementations perform their 
processing based on differing versions of Unicode, implementations have 
install bases still depending on older versions of the software and thus 
the Unicode version they check against, and finally, there are 
implementations and deployments performing the obsoleted stringprep.

A particular example is the following. Say a MUC service (including its 
server-to-server (s2s) handling) checks against Unicode version 12. One 
user, with a client and their server checking against Unicode >=9, 
chooses to use the nickname 'I♥🥓' (I love bacon). The MUC service 
assumes everything is fine, and the occupant JID becomes 
[email-masked]/I♥🥓. Both the BLACK HEART SUIT (U+2665) and 
BACON (U+1F953) are in the Symbols, Other (So) category, and thus valid 
for FreeformClass.

Now another user comes along, using a server that supports Unicode 6.3. 
Since BACON wasn't defined before Unicode 9, its code point is 
unassigned. When receiving presence from the other user, what should the 
receiving server do?

  a) It is liberal in what it accepts from other servers, it passes 
incoming remote stanzas on to the client.

  b) It is strict, and sends back a <jid-malformed/>, which likely boots 
the recipient from the room.

  c) In case a), if it wants to use private messaging towards the 
occupant JID, their own server might reject this with a similar 
<jid-malformed/> error.

The above is just an example. MIX [5] refers to RFC 7700 [6], obsoleted 
by RFC 8266, for preparing nicknames, which in turn also depends on 
FreeformClass, and thus exhibiting similar concerns, but not on the 
routing level.

Basically the question comes down to: how do we robustly handle 
different Unicode Versions in clients, services, and servers?

[1] <xmpp:[email-masked]>
[2] RFC 7622: XMPP: Address Format
     <https://tools.ietf.org/html/rfc7622>
[3] RFC 7613: PRECIS Representing Usernames and Passwords
     <https://tools.ietf.org/html/rfc7613>
[4] RFC 7564: PRECIS in Application Protocols
     <https://tools.ietf.org/html/rfc7564>
[5] XEP-0369: Mediated Information eXchange (MIX)
     <https://xmpp.org/extensions/xep-0369.html>
[6] <https://tools.ietf.org/html/rfc7700>
[7] RFC 8266: PRECIS Representing Nicknames
     <https://tools.ietf.org/html/rfc8266>

-- 
ralphm

---

From: Florian Schmaus <[email-masked]>
Date: Fri, 13 Sep 2019 09:35:46 +0200

> unassigned. When receiving presence from the other user, what should the

There is a fourth option/solution:

       agility regarding the supported Unicode standard.

PRECIS libraries typically use the Unicode/Character predicate and
property retrieval functions (.e.g., isISOControl(),
getDirectionality(), …) from their runtime environment. For example the
java.lang.Character API.

The problem is that most runtimes only update their supported Unicode
version with new (major) runtime release (e.g., Java 9). The key
observation here is: to support a new version of the Unicode standard,
no API changes or changes of the PRECIS library are necessary. What
changes is usually that a code point transitions from unassigned to
assigned, and so do its properties, and hence the return values of that
Unicode predicate and property retrieval functions.

So it would be great if runtimes would provide a way to load a new
Unicode Character Database [1] without updating the runtime environment.
Something like Character.loadUnicodeDb("unicode-12.1.0.dat")

Now I know that we may not have not much control over what functionality
runtime environments provide. However, nothing prevents you from using
(or implementing) a Unicode library with that such feature. I am
considering implementing something like that in [2]. The whole process
could eventually be automated, which, I believe, is solving the issue
Ralph describes.

- Florian

1: http://www.unicode.org/reports/tr44/tr44-24.html
2: https://bitbucket.org/sco0ter/precis

---

From: "Sam Whited" <[email-masked]>
Date: Fri, 13 Sep 2019 22:20:28 +0000

On Fri, Sep 13, 2019, at 07:35, Florian Schmaus wrote:
> Unicode Character Database [1] without updating the runtime
> environment. Something like Character.loadUnicodeDb("unicode-
> 12.1..dat")

The various Go Unicode libraries do this (more or less). They provide
build options that allow you to pick a Unicode version, so you could
always build with the latest one or possibly link in multiple (I think,
I haven't tried that but I don't see why it wouldn't work).

> Now I know that we may not have not much control over what
> functionality runtime environments provide. However, nothing prevents
> you from using (or implementing) a Unicode library with that such
> feature. I am considering implementing something like that in [2]. The
> whole process could eventually be automated, which, I believe, is
> solving the issue Ralph describes.

Sort of. Having libraries that link in UCD data or similar makes things
easier, but we still have to decide what's supported, what special cases
need to be worked around (I'm sure there will be some odd bugs and what
not that crop up with different versions), do we do negotiation or
support a specific range of versions, what if you're using an old
version of the XMPP client that doesn't have the new data files shipped
with it or linked in, etc.

I agree Unicode related libraries should support multiple versions, I
just don't think that really solves the problem of how we address
that in the XMPP specs and how we address compatibility between
clients and servers.

—Sam

-- 
Sam Whited

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 23 Sep 2019 16:52:12 +0100

Hi Ralph,

> different Unicode Versions in clients, services, and servers?

I agree this is a problem and as you demonstrate above there might not
be any simple solution for it. I think this issue deserve a document to
be written (an Internet Draft or an XSF document).

My personal inclination is that clients/servers should be liberal with
unassigned codepoints in JIDs that they don't control (i.e. servers can
be strict with local JIDs, but should be liberal with remote JIDs),
which I think is either case a) or an extension of it.

Best Regards,
Alexey

---

From: Waqas Hussain <[email-masked]>
Date: Mon, 23 Sep 2019 16:11:25 -0400

On Tue, Sep 10, 2019 at 10:39 AM Ralph Meijer <[email-masked]> wrote:

> _______________________________________________
> xmpp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xmpp

There's an old thread on this from 2011 on the IETF list. I don't believe
the core compatibility issue ever got resolved. See this message and the
connected thread:

https://mailarchive.ietf.org/arch/msg/xmpp/A2PT_EpDpR1swKbHC_te0NWwRgs

We also lack any form of advertisement of supported unicode version by a
remote entity, which is unfortunate. A stream feature and a caps hash may
be useful.

--
Waqas