Subject: Unicode string profiles for LDAP

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Mon, 13 May 2002 16:20:21 -0700

The LDAPv3 technical specification ambiguously defines matching
rules which acts upon values of Unicode-based syntaxes such as
DirectoryString.  For example, the specification does not state
which the direction characters should be folded when doing
case ignore matching.  Unlike IA5 matching, the direction matters
for Unicode strings.

draft-hoffman-stringprep-02.txt details three areas which we need
to profile Unicode string handling: input mapping, normalization,
and output prohibitions.  We also need to address transliteration
issues.  In this post, I've provided some thoughts in this area.

I think the specifications of Directory String and other
character string syntaxes should continue to allow transfer
of the full repertoire of characters (including unassigned
characters) and that the burden of preparation be placed upon
the application (e.g., the server) which is doing the comparison.
This ensures that LDAP can continue to be updated to support
additions to the character sets with minimal impact to existing
implementations.  It also allows introduction of matching rules
which have conflicting preparation requirements.

Transliteration:  We currently don't mandate transliteration
support.  For implementations which do transliteration, we need to detail
how transliteration is to be done (or say its implementation
specific).  I suggest we say that comparisons between different
CHOICEs are to performed after both strings transliterated to Unicode.
We then need to specify how each of the CHOICEs are to be transliterated
to Unicode.  And, as I think we should avoid per CHOICE preparation
requirements, we should state that comparisons between same CHOICEs
are to behave as if strings had been transliterated to Unicode.

Mapping of input:  select "invisible" characters should be
mapped out (removed) prior to normalization, including soft
hyphen and such.  For case insensitive matching rules, upper
case characters will be mapped to lower case.

Normalization: to Unicode KC form.

space: character string matching rules ignore "insignificant"
spaces.  We need to define which code points are considered spaces.
We likely should map all spaces to SPACE (U+00020) on input.

Unassigned characters:  The output cannot contain any unassigned
characters.  Hence, an assertion involving any unassigned character
will be Undefined.  The spec will list the set of unassigned characters
which all implementations MUST recognize as such.  When new characters
are assigned in Unicode, this specification can then be updated.  This
will cause some comparisons which previously evaluated to Undefined to
evaluate to True or False.

Currently ORDERING would remain implementation specific.

IMO, the clarifications of attribute/assertion value handling in
comparisions should be done in 2252bis where the syntaxes and matching
rules are specified.  It may also be appropriate to include in 2252bis
(non-normative appendix?) a discussion of the design choices made as
the choices may not obvious to the reader.

Kurt

---

From: Mark Wahl <[email-masked]>
Date: Mon, 13 May 2002 18:32:58 -0500

> We likely should map all spaces to SPACE (U+00020) on input.

I remember we looked at this issue several years ago when trying to come up
with conformance tests for LDAPv3.  Did ITU-T identify a list of 'space' 
characters in one of the updates or defect resolutions for X.500?   If not
I can track down my table... 

Mark Wahl
Sun Microsystems Inc.

---

From: David Chadwick <[email-masked]>
Date: Tue, 14 May 2002 07:27:33 +0100

"Kurt D. Zeilenga" wrote:

> will be Undefined.  

Kurt

if the matching rule is an approximate match, then it should be a local
matter as to whether the assertion value matches true, false or
undefined.

> rules are specified. 

The matching rules are currently defined in X.520, whereas 2252bis says
more about the LDAP syntax than the way assertion values are matched
against stored values. I would prefer it if the semantics of the
matching rules only occur in X.521, and that 22252bis points to X.520
(after all, the OIDs from X.520 are used, and we dont want two different
interpretations of how a particular matching rule should operate.) If
X.520 is ambiguous then we should submit a defect report on this (For
example, X.520 currently does not define a Fax matching rule - an
amazing oversight in my opinion - and at the last X.500 meeting we drew
up text for this new matching rule, and whilst doing so spotted several
ambiguities in telephone match, so these also will be fixed)

David

> It may also be appropriate to include in 2252bis

-- 
*****************************************************************

David W. Chadwick, BSc PhD
Professor of Information Systems Security
IS Institute, University of Salford, Salford M5 4WT
Tel: +44 161 295 5351  Fax +44 161 745 8169
Mobile: +44 77 96 44 7184
Email: [email-masked]
Home Page:  http://www.salford.ac.uk/its024/chadwick.htm
Research Projects: http://sec.isi.salford.ac.uk
Understanding X.500:  http://www.salford.ac.uk/its024/X500.htm
X.500/LDAP Seminars: http://www.salford.ac.uk/its024/seminars.htm
Entrust key validation string: MLJ9-DU5T-HV8J
PGP Key ID is 0xBC238DE5

*****************************************************************

---

From: David Chadwick <[email-masked]>
Date: Tue, 14 May 2002 08:07:23 +0100

> I can track down my table...

Unfortunately not. The only rules for the handling of space characters
are:

In the matching rules specified in 6.1.1 through 6.1.11, the following
spaces are regarded as not significant:
-       leading spaces (i.e. those preceding the first character that is
not a
space);
-       trailing spaces (i.e. those following the last character that is
not a
space);
-       multiple consecutive spaces (these are taken as equivalent to a
single
space character).
A string consisting entirely of spaces is equivalent to a string
containing exactly one space.
In the matching rules to which these apply, the strings to be matched
shall be matched as if the insignificant spaces were not present in
either string.

But no definition of the codes that equate to spaces.
David

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 14 May 2002 07:31:36 -0700

I agree that these matching rules are defined in X.520
and revising X.520 is the realm of the ITU.  I'd like
to continue our discussions here just to map out the
issues in this area.

Kurt

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 14 May 2002 07:47:18 -0700

>matter as to whether the assertion value matches true, false or

Yes, one could take the view that a string profile for
approximate matching should be implementation specific.

However, one could view the X.520 specifications as acting
upon some abstract character set.  What we are defining is
how code points of the Universal Character Set (UCS) relate
to this abstract character set.  That is, it can be argued
that there two levels of semantics and that approximate rule
semantics upon the abstract character set are implementation
specific but that the string profile isn't.  This can be
viewed as an additional constraint upon implementation
specific matching rules.

Kurt