Subject: Re: [Models] An attribute value should be equal to self

From: Hallvard B Furuseth <[email-masked]>
Date: Mon, 7 Mar 2005 12:41:01 +0100

At 16 Feb 2005, Kurt D. Zeilenga writes:
>I suggest adding the following statement to section 2.2 of
>[Models] (just before its last paragraph):
>
>   Additionally, no attribute is to have a value which is not
>   equivalent to itself.  For example, the 'givenName' attribute
>   cannot have as a value a directory string which includes the
>   REPLACEMENT CHARACTER (U+FFFD) code point as matching involving
>   that directory string is Undefined per this
>   attribute's equality matching rule.

I really dislike to be unable to store any valid UTF-8 string in a
Directory String, though I realize we have problems either way.  I
will have to review that discussion yet again soon...  I don't remember
if X.500 has restrictions like this, or anything like stringprep?

-- 
Hallvard

---

From: Steven Legg <[email-masked]>
Date: Tue, 08 Mar 2005 10:20:19 +1100

Hallvard,

Hallvard B Furuseth wrote:

> Directory String, though I realize we have problems either way.

I dislike it too. I would prefer that LDAPprep removes troublesome characters
instead of failing.

 > I will have to review that discussion yet again soon...

It was a short thread:
http://www.openldap.org/lists/ietf-ldapbis/200411/msg00190.html

 > I don't remember

No and not yet.

Regards,
Steven

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Mon, 07 Mar 2005 22:06:02 -0600

At 05:20 PM 3/7/2005, Steven Legg wrote:
>I dislike it too. I would prefer that LDAPprep removes troublesome characters instead of failing. 

It is not clear to me that X.500 requires for every equality
matching rule R and every value X of the applicable attribute
value syntax that:
        R( X, X ) == TRUE

In absence of a guarantee that R( X, X) is always TRUE,
the [Models] statement in necessary.

Kurt

---

From: Steven Legg <[email-masked]>
Date: Tue, 08 Mar 2005 15:58:00 +1100

Kurt,

> the [Models] statement in necessary.

It is only necessary in the case of the string matching rules because LDAPprep
can fail. If LDAPprep didn't fail then R(X, X) = TRUE would always be trivially
satisfied. No test would be required.

BTW, in the case where matching is not undefined, the X.500 requirement for
transitivity, i.e. a=b and b=c then a=c, and commutativity, i.e. a=b then b=a,
guarantees a=a.

If b=c then c=b, which means a=b and c=b implies a=c. If c is identical to a then
we have a=a.

Regards,
Steven

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 08 Mar 2005 00:19:18 -0600

At 10:58 PM 3/7/2005, Steven Legg wrote:

>can fail.

It's also necessary because X.500, I think, doesn't guarantee
that all equality rules ensure R( X, X ) == TRUE. 

And, in regards to LDAPprep, it is difficult to prevent LDAPprep
from failing.  For instance, step 3 (Normalize) will fail
on any input which cannot be processed by NKFC algorithm.

Hence, if X.500 requires all equality rules to ensure R(X,X),
not only would be have to eliminate all prohibitions, we'd
have to eliminate NKFC normalization.

>If LDAPprep didn't fail then R(X, X) = TRUE would always be trivially
>transitivity, i.e. a=b and b=c then a=c, and commutativity, i.e. a=b then b=a, guarantees a=a.  If b=c then c=b, which means a=b and c=b implies a=c. If c is identical to a then we have a=a.

If you can find values of b and c such that these statements
are true in respect to a, then, yes, a==a.  However, if
R(a,a) is Undefined, you shouldn't be able to find any such
b and c.

Kurt

---

From: Hallvard B Furuseth <[email-masked]>
Date: Tue, 8 Mar 2005 11:13:22 +0100

Steven Legg <[email-masked]>
> http://www.openldap.org/lists/ietf-ldapbis/200411/msg00190.html

Ah, thanks.

>> characters instead of failing.

I'm not sure that is right; it might be better to translate them to some
otherwise unused character or leave them alone or something.  Or let
EQUALITY match use a fallback which does not do LDAPprep if LDAPprep
fails, like Rici suggested.

> the [Models] statement in necessary.

Necessary for what?

In view of the above thread, security is the only remaining argument
I remember which I buy for letting LDAPprep fail AND letting mess up
everything else, and that's because I do not know Unicode nearly well
enough to judge the security issues.

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 08 Mar 2005 06:40:39 -0600

>> the [Models] statement in necessary.

To ensure that one can delete by value any and all values
added by value.  That is, if one can do:
        changetype: modify
        add: cn
        cn: X

one should be able to:
        changetype: modify
        delete: cn
        cn: X

and have the latter X match the former X.

Even though introduction of LDAPprep made apparent the need for
this, this is not just about LDAPprep,   It's about any rule where
there is any X where R(X,X) is Undefined.

Kurt

---

From: Hallvard B Furuseth <[email-masked]>
Date: Tue, 8 Mar 2005 14:24:55 +0100

Kurt D. Zeilenga writes:
> added by value.

That's useful, but not necessary.  One can delete the value by replacing
the attribute with all values but the one being deleted.  Just as being
able to store any UTF-8 string is useful but not necessary.  I think the
latter is far more important.

> there is any X where R(X,X) is Undefined.

Where?  It mentions interoperability and security in [Prep] 1.2.  I
don't see where it says that error returns from LDAPprep is necessary
for security, but for all I know it may be.  Nor does it mention why
being able to name individual values is more important than being able
to have any UTF-8 string in a Directory String.  I never agreed that
interoperability for values that fail LDAPprep, when interoperability
means "you can't use them at all", is more important than being able
to at least use them locally - and that was before we discovered just
how many problems there were with being able to store them at all.

I remember we discussed other workarounds for error returns (given that
there are error returns) than EQUALITY(X, X), but I don't remember if
any were acceptable.  Will be reading up on it, though.

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 08 Mar 2005 08:32:31 -0600

At 07:24 AM 3/8/2005, Hallvard B Furuseth wrote:
>the attribute with all values but the one being deleted.

That would require the user to know all those values.

Also, consider the implications in naming:

If a client adds:
  dn: cn=<REPLACEMENT CHARACTER>,dc=example,dc=com
  objectClass: object
  cn: <REPLACEMENT CHARACTER>

where <REPLACEMENT CHARACTER> is the replacement character, a
prohibited character.  How is a client to modify this entry?

>Just as being

Any UTF-8 string can be stored in the DIT.  One just has to use an
attribute type with an appropriate syntax and equality matching rule.

caseIgnoreMatch (or caseExactMatch) + Directory String is not an
appropriate combination where the application desires to store
strings of arbitrary Unicode code points.

That is, for this use, one should specify a "Unicode Code Point
Match" equality rule for use with Directory String.... or just
use octetStringMatch and octetStrings (and state a requirements
be constrained to UTF-8).

>> there is any X where R(X,X) is Undefined.

My point here is that LDAPprep-based equality rules are not
the only rules which may exhibit the behavior that for some
X, R(X,X) is Undefined.   That is, even if LDAPprep was
changed such that R(X,X) was TRUE for all X, the [Models]
statement would be appropriate to address the general issue.

The only other way to address the general issue is to require
that all equality rules have the property that R(X,X) is
TRUE for all possible values of X.  That is, IMO, a far more
a problematic requirement.

>Where?

anywhere.

>It mentions interoperability and security in [Prep] 1.2.  I

LDAPprep is still in development.  Maybe we'll reach consensus
that for all X, R(X,X) is to be TRUE.   But regardless of whether
we reach that consensus, X.500/LDAP allows (IMO) there to be
other rules where for some X, R(X,X) is Undefined.  And, because
of this, it is appropriate for [Models] to state the restriction
because allowing X as a value of an attribute where R(X,X) is
not TRUE is quite problematic.

Kurt

---

From: Hallvard B Furuseth <[email-masked]>
Date: Tue, 8 Mar 2005 16:05:49 +0100

[Skipping most of the reply for now - I disagree with some but
must read up on more]

> not TRUE is quite problematic.

That... sounds right.  I think.  Though I need to sort this all out
before I dare arrive at an opinion.

-- 
Hallvard

---

From: Steven Legg <[email-masked]>
Date: Thu, 10 Mar 2005 17:20:36 +1100

Hallvard,

Hallvard B Furuseth wrote:
> otherwise unused character or leave them alone or something.

You're right. Removing the troublesome characters means that a string
with "garbage" will match a string without "garbage". I'd suggest mapping
to something like the replacement character rather than leaving the characters
alone for the reason that two distinct bad sequences might one day be made
equivalent, which would then present a problem if an attribute already has both
as values.

 > Or let

That's fine for equality matching but could dramatically change the collation
order for ordering matches. If two values are the same except for some final
bad characters then it is desirable for them to still be close in the collation
order.

Regards,
Steven

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Thu, 10 Mar 2005 13:59:17 -0600

>equivalent, which would then present a problem if an attribute already has both

I don't see any value in mapping troublesome characters,
presently prohibited, to a character which itself remains
prohibited.  If instead, you meant to map the troublesome
charactes to some character which is not prohibited, then
I think this be problematic for the reason you note,
string+garbage would be equivalent to string.

Also, I note, that even if there were no "prohibited" characters,
LDAPprep can still suffer failures.  For instance, Unicode
normalization failures, unassigned Code point failures, etc..

>bad characters then it is desirable for them to still be close in the collation

Concur.

---

From: Steven Legg <[email-masked]>
Date: Fri, 11 Mar 2005 16:38:02 +1100

Kurt,

> that all equality rules ensure R( X, X ) == TRUE.

I not arguing against having a general requirement that R(X, X) == TRUE.
It could equally well be described as R(X, X) != Undefined, since X.500's
conditions on equality matching rules mean that R(X, X) == FALSE can never
happen. What I'm concerned about, and what I think Hallvard is concerned
about, is that LDAPprep is causing the string matching rules to evaluate
R(X, X) to Undefined for a wider range of inputs than we think is desirable.

The requirement for R(X, X) == TRUE is effectively imposing a restriction
on the string syntaxes, but it does not restrict arbitrary string components
of more complex syntaxes. Component matching can apply a string matching
rule to such components and I would much prefer that the result is definitively
TRUE or FALSE (as it currently is in my implementation), rather than Undefined
in some cases.

> on any input which cannot be processed by NKFC algorithm.

I did a quick scan of Unicode Standard Annex #15 and couldn't see any evidence
that the normalization procedure can fail. Can you point me at a definitive
statement ?

> have to eliminate NKFC normalization.

I don't see how that follows, especially since the prohibition step
comes after the normalization step. I'm asking that LDAPprep not fail.
Normalization is not incompatible with that.

Regards,
Steven

---

From: Steven Legg <[email-masked]>
Date: Fri, 11 Mar 2005 16:58:17 +1100

Kurt,

Kurt D. Zeilenga wrote:
> prohibited.

I'm saying that none of the characters should be prohibited, at least as
far as matching rules are concerned.

 > If instead, you meant to map the troublesome

By "troublesome characters" I meant all these sorts of things.

Regards,
Steven

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 11 Mar 2005 00:38:38 -0600

At 11:38 PM 3/10/2005, Steven Legg wrote:
>I'm asking that LDAPprep not fail. 

Given that unassigned code points are to be prohibited
[Section 7, Stringprep] (in at least one of the two involved
strings), and all revisions of Unicode have unassigned
code points, all proper profiles of Stringprep can fail
and will fail in face of unassigned code points.

That is, LDAPprep can fail.  There is no way to escape
this (and still have LDAPprep be a profile of the
StringPrep algorithm).

As you noted, troublesome characters should not be mapped
to nothing as that causes string+garbage to match string.
But mapping those troublesome characters to the replacement
character makes little sense.  If the replacement character
is prohibited, then the mapping has zero impact.  If the
replacement character is allowed, then string+garbage
would be equal to string+replacement.  I think this
is inappropriate as if we say the replacement character
is not a troublesome character, then no troublesome
character should match it.

The issue I was referring with Normalization is really
a code point v. character issue, and general assumption
that matching (of the prepared values) is done in
terms of characters not code points.  Ignore this for now.
(Also ignore the possibility that transcoding might fail.)

The prohibition on Unassigned code points will cause
StringPrep to fail.

Now, if you want Rule(X,X) != Undefined, then in [Syntaxes]
we could say that if LDAPrep fails the matching for either
the assertion value or the attribute value, the matching
is False.  I'd have to think about it a bit, but off-hand,
I think it might be better for assertions where an implementation
is unable to determine the abstract value represented (e.g.,
prepare the string) by either the assertion or attribute
value strings to return Undefined.  My rationale is that since
the implementation cannot determine what the abstract value
is and it cannot determine equality of that abstract value
to any other abstract value.  This behavior is similar to
the behavior with the string does not adhere to the
applicable syntax restrictions.  That is, I don't much
difference in (cn=garbage) and (timeStamp=20050231Z).  Both
should evaluate to Undefined.

Kurt

---

From: Steven Legg <[email-masked]>
Date: Fri, 18 Mar 2005 17:40:16 +1100

Kurt,

> and will fail in face of unassigned code points.

Not necessarily. Section 7 says that the "list of unassigned code
points MUST be given in a profile, and that list MUST be used by
implementations of the profile in step 3 (prohibit)". Step 3, i.e.
section 5, doesn't say anything about the unassigned code points
but does suggest that a profile has a degree of discretion as to
which characters to treat as prohibited. In the absence of anything
definitive, I think Section 7 is more relevant in that it says
"Stored strings using the profile MUST NOT contain any unassigned
code points. Queries for matching strings MAY contain unassigned
code points". This suggests to me that attribute values MUST NOT
contain unassigned code points, but assertion values MAY contain
unassigned code points.  Stringprep seems to be saying that there
are two different preparation algorithms. An algorithm that is
applied to stored strings to normalize them and to check for
prohibited characters, including unassigned code points, and another
algorithm that is applied to query strings to normalize them and
check for prohibited characters, but not including unassigned
code points. At the moment LDAPprep is not drawing any distinction
between a string in an attribute value versus a string in an
assertion value. However there are wider implications.

A requirement that R(X, X) == TRUE, with LDAPprep as it is currently
defined, will have the effect of preventing attribute values from
containing prohibited characters, but only for strings that are
examined by the equality matching rule for the attribute. There is
nothing that stops the DESC field in an attribute type description,
for example, from containing prohibited characters (including unassigned
code points). If we are to follow the spirit of stringprep then I
think that means we must prevent prohibited characters appearing in
any part of an attribute value, regardless of the equality matching rule
for the attribute. After all, with component matching any part of any
attribute value can be matched.

Incidentally, I wouldn't be adverse to a blanket ban on unassigned
code points in every part of every attribute value, as long as LDAPprep
doesn't fail if they appear in an assertion value. We have discretion
as to what other characters are to be prohibited from attribute values.

Something that isn't clear to me is how stringprep is supposed to
apply to stored values. I guess that the intent is that one should
apply the stringprep profile to each string that is about to be
stored and if that returns an error (because it contains prohibited
characters) then the string should be rejected. However Section 7.1
says "Stored strings MUST NOT contain any code points outside of AO
for the latest version of a profile. That is, they are forbidden to
contain code points from the MN, D, or U categories". I take this to
mean that what one is actually supposed to store is the output of
stringprep, i.e. the normalized string. We aren't doing that.

I think we are falling short of the intent of stringprep.

> character should match it.

If garbage is mapped then it has to be mapped to something.
The replacement character seems to be the most appropriate
because it already carries the sense of something that
cannot be properly represented.

> The prohibition on Unassigned code points will cause

For a stored string but not a query string.

> value strings to return Undefined.

I agree that in the face of failure a matching rule should
return Undefined. It's more a question of what causes failure.
I would be less concerned if LDAPprep only failed for characters
that are never allowed in stored strings anyway, but that currently
isn't generally the case. For the versioning reasons put forth
in stringprep we should allow unassigned code points in
assertion values. If attribute values were prevented from
containing unassigned code points by a blanket ban then LDAPprep
could simply ignore them. If they appear then it must be because
we are preparing an assertion value, and that's okay.

If we know that stored strings never contain prohibited characters
then we could sensibly map an LDAPprep failure to FALSE.

 > My rationale is that since

There's an important difference. The time stamp value violates
the syntax. The matching rule has nothing to do with it. The
value would still be illegal even if the attribute didn't have
an equality matching rule. The cn value satisfies the syntax
(though perhaps it shouldn't) but the equality matching rule
evaluates to undefined.

Regards,
Steven

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 18 Mar 2005 00:37:28 -0800

At 10:40 PM 3/17/2005, Steven Legg wrote:
>unassigned code points.

Correct.  While profiles may prohibit other code points, all profiles
are required to prohibit unassigned code points in "stored"
strings.  At least one of the two strings involved in the
comparison is to be categorized as a "stored" string.  It was my
assumption that we'd categorized, in general, the assertion value as
a "query" string and the attribute value as an "stored" string.

(I say in general because there are some special cases.)

The reason I prohibited them in LDAPprep in both strings (and had
LDAPprep failures lead to Undefined mapping) was that the presented
assertion value presented could contain an unassigned code (in the server)
X which the client thinks should be equivalent to some assigned code (in
both server and client) Y.  The server evaluate caseIgnoreMatch(prep(X),prep(Y))
to Undefined as has no way of knowing wether or not X and Y are equivalent
under some subsequent Unicode version.

I don't think it makes much sense for caseIgnoreMatch(prep(X),prep(Y))
to be FALSE on one server (implementing current LDAPprep) but TRUE on
another (implementing LDAPprep').

>Stringprep seems to be saying that there

Yes, there certainly are wider implications.

>code points in every part of every attribute value,

Okay.

>as long as LDAPprep doesn't fail if they appear in an assertion value.

That would require eliminating all other prohibitions.

>We have discretion

That we do.

>I think we are falling short of the intent of stringprep.

I had a discussion with Patrik last week regarding rationale behind
the Section 7.  Basically, I understand Patrik to say that they
wanted to preclude implementations from creating (for lack of a
better word) stored lookup keys (indices, hashes of values, etc.)
derived from values containing unassigned code points to prevent
problems upon subsequent assignment of code points and normalization
mappings.

One could argue that Stringprep is delving into implementation
details here.

One could argue that instead of LDAPprep, we should just require
the server to apply NFKC normalization to the assertion and
attribute values and then:
        if NFKC(assert) == NFKC(attribute) return TRUE,
        if NFKC(assert) contains unassigned code points return Undefined,
        if NFKC(attribute) contains unassigned code points return Undefined,
        else return FALSE

and then state various implementation consideration regarding lookup keys
involving unassigned code points, including specifically allowing
(if not recommending or mandating) implementations to reject attribute
values containing unassigned code points and noting that if they
don't that upgrade to LDAPprep' will requiring rebuilding lookup
keys for any value containing a previously unassigned code point.

(Note that I am not necessarily advocating this approach.  Just
food for thought.)

>cannot be properly represented.

My concern is that if the replacement character is
not garbage then no garbage should match the replacement
character.  If its allowed, its not garbage.

I rather allow garbage than to map them to something.

>For a stored string but not a query string.

but transcoding failure can happen in either string.  This
is another reason why I think LDAPprep failure should lead to
Undefined matching.

>then we could sensibly map an LDAPprep failure to FALSE.

I content that it more sensible for assertion involving an
Unassigned code point in the assertion value to evaluate to
Undefined.