Subject: T.61 -> Unicode conversion

From: Hallvard B Furuseth <[email-masked]>
Date: Sun, 7 Nov 2004 23:39:03 +0100

As far as I can tell, X.500/LDAP servers and X.500<->LDAP gateways may
use another mapping between T.61 and Unicode than the one in [Strprep]
Appendix A for transfer and storage of values, *except* for string
matching, for which [Strprep] section 2.1 says:

  TeletexString [X.680][T.61] values are transcoded to Unicode as
  described in Appendix A.

Though I don't quite understand when this sentence applies.  Only for
X.500/LDAP servers and when transferring values in binary format over
the protocol?

Can it happen that a Modify operation contains an UTF-8 value which
according to Appendix A does not match an existing value in the entry,
but which the server stores as a T.61 value which does match an existing
value?

Unless I'm confused about this, Appendix A shouldn't apply to just
string matching but to converting T.61 strings to Unicode in general.

Also,

I'm pretty sure that when I used X.500 and T.61 (10+ years ago?), at
least the T.61 Underline combining character could be combined with
other combining characters.  I don't know if Unicode allows that; the
strprep description of doing so at the end of Appendix A does not.

Also, I may remember wrong but there seems to be a lot more valid
character combinations (accent character + other character) than there
was in the T.61 I am used to.  Does the strprep algorithm accept any
T.61 character combination which can be mapped to Unicode, whether or
not the combination is valid in T.61?  If so, there are strings which
Stringprep will incorrectly will treat as valid.

I don't know where to find T.61 so I can check this, though.  At least
not on a Sunday:-)

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 09 Nov 2004 09:45:07 -0500

At 05:39 PM 11/7/2004, Hallvard B Furuseth wrote:
>Though I don't quite understand when this sentence applies.

It applies to preparation of values for matching using
any of the matching rules mentioned in the I-D.

>Only for

It used anytime the above matching involves an assertion value
and/or assertion value encoded as T.61.

>but which the server stores as a T.61 value which does match an existing

I believe [Models] prevents this through its (values must
be different, values must be preserved) requirements.

>string matching but to converting T.61 strings to Unicode in general.

I'll leave the remainder to others more familiar with T.61 details.

Kurt

---

From: Hallvard B Furuseth <[email-masked]>
Date: Tue, 9 Nov 2004 16:04:32 +0100

> be different, values must be preserved) requirements.

There have been quite a lot of disputes about whether characters in
different character sets are equivalent or not, though I don't know if
that applies to T.61->Unicode mapping.  I do remember that the meaning
of some character combinations were redefined for T.61 (not in relation
to Unicode, I think).  Something about an umlaut-like mark which should
or shouldn't mean umlaut?

Anyway, implementors who disagreed about this would get different
results, and maybe different results from the Appendix.  In particular
if they have implemented T.61->Unicode(LDAP) first, and add stringprep
later.

-- 
Hallvard

---

From: Hallvard B Furuseth <[email-masked]>
Date: Tue, 9 Nov 2004 16:42:53 +0100

Hallvard B Furuseth writes:
> that applies to T.61->Unicode mapping.

Come to think of it, I probably do know: Stringprep Appendix A says it's
based on RFC 1345 (an informational RFC, not a standard) rather than a
Unicode mapping or some X.500 work, and also that makes corrections to
the RFC 1345 mapping.  (If RFC 1345 is standard in the X.500 community,
please mention that in the appendix.  I'm assuming it's not.)

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 10 Nov 2004 16:02:50 -0500

>please mention that in the appendix.  I'm assuming it's not.)

I'm not aware of a standard T.61->Unicode mapping (nor a
standard Unicode->T.61 mapping).

However, once published, draft-ietf-ldapbis-strprep will
provide a Standard Track T.61->Unicode mapping.

Kurt

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 10 Nov 2004 16:24:05 -0500

At 10:04 AM 11/9/2004, Hallvard B Furuseth wrote:
>results, and maybe different results from the Appendix.

Implementors who follow the Appendix should produce identical
results.

>In particular

Okay, I think I see what your getting at.  Likely [Syntax]
should say that when a directory string is transcoded to
Unicode for transfer using UTF-8, that transcoding should
be done in a fashion consistent with LDAPprep transcode
step (and hence the appendix).

Kurt

---

From: Steven Legg <[email-masked]>
Date: Thu, 11 Nov 2004 09:52:47 +1100

> step (and hence the appendix).

I'll do this in the next revision of the syntaxes draft if I don't
hear any objections.

Steven

---

From: Howard Chu <[email-masked]>
Date: Wed, 10 Nov 2004 18:40:41 -0800

> strprep description of doing so at the end of Appendix A does not.

You're right, the notes I read state that the Underline can be combined 
with any other graphic character. I'll have to double-check how Unicode 
deals with this.

> Stringprep will incorrectly will treat as valid.

Appendix A doesn't specify that invalid combinations should be detected. 
Perhaps it should, but it seems unfriendly to break any software that 
might be using these sequences.

> I don't know where to find T.61 so I can check this, though.  At least

Considering that the CCITT withdrew this spec years ago, it's not 
surprising. It also has never been clear which version T.61 to use, as 
that spec evolved quite a bit before being withdrawn. (e.g., 
http://www.mail-archive.com/[email-masked]/msg04365.html

Appendix A only considers iso-ir-103 as listed in RFC1345 but there's 
really no way to know how many implementations out there did things 
differently.

-- 
   -- Howard Chu
   Chief Architect, Symas Corp.       Director, Highland Sun
   http://www.symas.com               http://highlandsun.com/hyc
   Symas: Premier OpenSource Development and Support

---

From: Howard Chu <[email-masked]>
Date: Wed, 10 Nov 2004 19:27:27 -0800

> with any other graphic character. I'll have to double-check how Unicode 

http://www.unicode.org/book/ch02.pdf page 25
   The Unicode Standard does not restrict the number of combining 
characters that may follow a base character.

And from http://www.itscj.ipsj.or.jp/ISO-IR/103.pdf
   The non-spacing character UNDERLINE shall never be used individually 
but always in combination with some other graphic character to represent 
the graphic rendition "underlined" for the associated character. The 
non-spacing character UNDERLINE can be used in combination with any 
graphic character of the repertoire, including an accented letter or an 
umlaut, or SPACE.

This note would imply that we need to add a check for this case to 
Appendix A: if the combining character is UNDERLINE and the next 
character is also a combining character, then the character after that 
is used as the base character and both of the combining characters are 
mapped and appended. E.g. "_'a" becomes "a_'"

---

From: Leif Johansson <[email-masked]>
Date: Thu, 11 Nov 2004 15:33:00 +0100

>   TeletexString [X.680][T.61] values are transcoded to Unicode as

This thread gives me the heeby-jeebies! We cannot possibly spend this
much time on T.61. I suggest that we say that usage of T.61 is a
SHOULD NOT and be done with it. The only situations where T.61 might
appear that I can think of are LDAPv2 (historic and therefore out of
scope) and Certificates with DirectoryString, but new certificates
MUST use only UTF8String (if I remember right) since about a year
ago so where exactly is the applicability of this?

	MVH leifj

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 11 Nov 2004 22:03:26 +0100

Leif Johansson writes:
> much time on T.61.

If we are going to publish a T.61->Unicode mapping, we should try to
avoid errors in it.  In particular when the strprep appendix with the
mapping is normative.  I have no idea if we need the appendix, or if
T.61 is as dead as you suggest.  The URL Howard posted:

recommends to use T.61 in some circumstances, but concludes with "In
2004, you can switch to UTF-8 strings and forget about this entire
section of the guide."  So maybe it's still in use but fading fast?

> I suggest that we say that usage of T.61 is a SHOULD NOT (...)

That sounds nice (from my completely-out-of-touch viewpoint:-), with or
without the appendix.  We should certainly not publish the appendix in a
form which might prolong the life of T.61.

Also, the above URL mentions shift states, Japanese and other encodings
which the appendix does not cover, disagreements on a level I never
dreamed of about which character sets a T.61 string is using by default.
So it looks like the appendix at least needs to mention a number of
caveats, and I'm unsure how much sense it makes that it is normative.

-- 
Hallvard

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 11 Nov 2004 22:39:27 +0100

I wrote:
> The URL Howard posted:

Note, that's from an X.509 certificate guide, not a general X.500 guide.

-- 
Hallvard

---

From: Steven Legg <[email-masked]>
Date: Fri, 12 Nov 2004 13:41:06 +1100

Hallvard,

> Note, that's from an X.509 certificate guide, not a general X.500 guide.

Also, worth noting - X.500 uses TeletexString, ASN.1 refers to ISO/IEC 2022,
ISO/IEC 2375 and the "ISO International Register of Coded Characters Sets"
for the BER encoding of TeletexString values. It doesn't depend on T.61.

P.S. I would like to stop using TeletexString, but there are X.500 servers
out there that barf if I try to use the uTF8String alternative of DirectoryString.

Regards,
Steven