Subject: LDAPprep: mapping of " " values

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Sun, 14 Nov 2004 22:50:13 -0800

Steven suggested that changing LDAPprep such that string
comprising only of whitespace would be mapped to "" instead
of " ".  I believe a poor approach for a number reasons.

While it can be argued (as Steven has) that such mapping
may make some assertions more intuitive, I argue that such
mapping will make various assertions less intuitive.

More importantly, the assertion (l=* *), which says "match a
significant space in values of l", would no longer behave
properly.  The " " ANY string would be mapped to "", leading
to (l=* *) matching any value instead of only those values
which contained a significant space.  This would likely
break a number of applications.

It's my view that the assertion (l= *) says  "match a
significant leading space in values of l".  These assertions
intuitively should only match strings which are all whitespace,
as leading whitespace is otherwise insignificant.  Likewise
for (l=* ).  Note that this behavior is actually useful.  One
can assert (!(l= *)) to match all values which are not
all whitespace.  Having (l= * * ) behave like (l=*)
substracts value (and likely will break applications, see
above).

Additionally, I believe it important that all outputs of
LDAPprep would not be valid per the syntax of the input.
If this is not so, then implementations must be very
careful not to apply LDAPprep to the output of LDAPprep.
Also, LDAPprep could not be used as a canonicalization
function if we were to adopt this mapping.

Kurt

---

From: Steven Legg <[email-masked]>
Date: Tue, 16 Nov 2004 17:38:37 +1100

Kurt,

> break a number of applications.

I agree that matching everything in such a case is excessive.

> as leading whitespace is otherwise insignificant.

Wouldn't it be easier to just say (l= ) ?

 > Likewise
> can assert (!(l= *))

or (!(l= ))

 > to match all values which are not

In the current specifications (l= * * ) will never match anything!
A value can only match (l= *) or (l=* ) if it is all whitespace.
If it is all whitespace then LDAPprep reduces it to a single space.
A single space cannot simultaneuously satisfy the initial, any and
final substrings.

What we are running into I think is the problem that whitespace in
different parts of an attribute value are treated differently, but the
whitespace in each substring of a substring assertion is treated the
same. Intuitively, one might expect that (l= * * ) should match a
value like "  foo  bar  ". It doesn't with the current specifications.
It would if whitespace were reduced to nothing, but it would match everything
else as well.

What we seem to need here is for leading whitespace in the initial substring
and trailing whitespace in the final substring to be reduced to nothing,
while every other sequence of whitespace characters, in the initial, any or
final substring, reduces to a single space.

It would be a modest change to LDAPprep to enable something like this.
We just need two parameters for each string handed to LDAPprep: a boolean
flag that indicates whether whitespace in the initial part of the string
is to be treated as leading whitespace, and a boolean flag that indicates
whether whitespace in the final part of the string is to be treated as
trailing whitespace. The syntaxes draft can then nominate values for the
flags for each string or substring it passes to LDAPprep. Alternatively,
LDAPprep can just reduce consecutive whitespace to a single space in every
case and leave the syntaxes draft to nominate the circumstances under
which a leading or trailing space is to be removed.

> function if we were to adopt this mapping.

In the wider context of component matching (and potentially even within the
framework of X.500) there are many ways that the output of LDAPprep could be
invalid with respect to the syntax, i.e. ASN.1 type, of the abtract value that
supplied the input string. It can change the length of the string such that it
is no longer an acceptable length - too short, too long (?) or an
explicitly disallowed length. It can introduce space characters where space
characters are disallowed. It can create a sequence of characters that
no longer satisfies a pattern constraint or value constraint. And so on.
And what exactly is the output syntax of LDAPprep in ASN.1 terms ?
A UTF8String ? A UniversalString ? That clearly doesn't line up with
an input that is a TeletexString.

LDAPprep is only used within the LDAP technical specification to prepare
character strings for a comparison routine. It is an internal part of
of a function that accepts two values and produces TRUE, FALSE or Undefined
as a result. If someone wants to use it for something else, like canonicalization
then they have to deal with the consequences, which are far more involved than
dealing with empty strings.

Regards,
Steven

---

From: Rici Lake <[email-masked]>
Date: Tue, 16 Nov 2004 02:01:53 -0500

On 16-Nov-04, at 1:38 AM, Steven Legg wrote:

> Alternatively,

This seems very sensible to me.

> A value can only match (l= *) or (l=* ) if it is all whitespace.

Not quite true. According to the insignificant space deletion rule,
a space is only a candidate for deletion if it is not followed by
a combining mark. Consequently, the sequence U+0041 U+0020 U+0020 U+0301
will be not be altered by ldapprep. (The string is LATIN CAPITAL LETTER 
A,
SPACE, SPACE, COMBINING ACUTE ACCENT.) One doesn't have to go out of 
one's
way to produce that sequence; the sequence U+0041 U+0020 U+00B4
(LATIN CAPITAL LETTER A, SPACE, ACUTE ACCENT) will be ldapprep'd into 
the
first sequence, as a result of the compatibility decomposition of U+00B4
into U+0020 U+0301. (That is, both strings render as "A ´" for anyone
with Unicode mail readers.)

Consequently, even (l= * *) could match something, but I believe that
(l= * * ) is truly impossible.

---

From: Rici Lake <[email-masked]>
Date: Tue, 16 Nov 2004 02:15:16 -0500

Rats, sending email too late again. The example should have *started*
with the string U+0020 U+0301, rather than including it as an interior
string. It is a demonstration that a non-empty string can *start* with
a space (although not end with one).

Combining marks, in general, create some interesting edge cases for
substring matching, which is the only place where the decision to
use NFKC rather than NFKD makes a significant difference. The most
significant difference will come with substring matches in Korean;
form NFKC requires Hangul Jamo to be composed into Hangul syllables,
which means that the individual Jamo are not available for substring
matching. I don't speak Korean so I cannot tell if a native speaker
would consider it desirable to, for example, do a substring match
on an initial consonant rather than having to create an "or" of the
several hundred syllables which start with that consonant.

On 16-Nov-04, at 2:01 AM, Rici Lake wrote:

> will be not be altered by ldapprep. (The string is LATIN CAPITAL 
> LETTER A,

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 16 Nov 2004 02:35:57 -0800

>Wouldn't it be easier to just say (l= ) ?

Yes, but X.520 allows (l= *) instead.  What does your
implementation do today?  In OpenLDAP, this assertion
will only match values which are composed entirely
of whitespace.  Others?

The logic here is that, except in one special case, that
leading and trailing spaces are insignificant.  One cannot
match on insignificant portions of the value without giving
them significance.  And giving leading and trailing spaces
significance changes the character of the rules in a major
way.

>In the current specifications (l= * * ) will never match anything!

I believe that this is correct.  As the old adage goes:
ask a stupid question, get a stupid answer.

>A value can only match (l= *) or (l=* ) if it is all whitespace.

I believe that this is correct.  As every string ("X") is equivalent
to some string which has insignificant leading and trailing
whitespace (" X "), these assertions would match the same
entries as (l=*).   The client should simply do (l=*) if that
what it wants. 

>A single space cannot simultaneuously satisfy the initial, any and

I believe that this is proper as there is only one significant
space and the assertion asked whether there is three significant
spaces.

>It would if whitespace were reduced to nothing, but it would match everything else as well.

If intuitively one might expect this, then they might also
expect (l=* * *) to match "x  x" (or (l=*  *) to match "x  x"
but not "x x").  If one can match insignificant leading and
trailing spaces, then it intuitively follows one can match
insignificant consecutive spaces.

I believe that this is nonsense and that we should redesign
matching to support matching of insignificant spaces.

>final substring, reduces to a single space.

If there is a need to match insignificant spaces, a rule which
is specifically design to support that matching should be used.
These rules were designed to ignore insignificant spaces.  We
should not change that.

>It would be a modest change to LDAPprep

What you ask for, IMO, is a change to matching rule to support
matching of insignificant spaces in certain cases.  I believe
that such a change is inappropriate and certainly should be
viewed as a new feature.

> to enable something like this.

---

From: Steven Legg <[email-masked]>
Date: Wed, 17 Nov 2004 11:36:56 +1100

> an input that is a TeletexString.

I missed an obvious case. One of the outputs of LDAPprep is failure.
Failure isn't a legal value of any input to LDAPprep.

It disturbs me that LDAPprep can fail on syntactically correct input as this
can give rise to ugly situations in the directory. If I add a new attribute with
a single value to an entry then there is no need to call on LDAPprep because
there is no need for any attribute value comparisons. If it so happens that
the single attribute value contains character sequences that cause LDAPprep
to fail then I cannot subsequently add other attribute values because the
comparison of a new value to the existing value will always fail. Likewise,
I cannot subsequently remove the single value because the comparison of the
value in the modify operation to the existing value will also fail.

There are implications for sorting as well.

Regards,
Steven

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 16 Nov 2004 17:20:28 -0800

In response to the empty IA5String discussion I noted my
thought that one could view the size restriction as a
artifact of the particular equality matching rule and
not of the syntax...

Because of the nature of Stringprep, I don't think we
can generally prevent preparation failures.  Even if we
could for LDAPprep, we couldn't for nameprep, saslprep,
and other independently developed profiles.  It is
desirable to allow introduction of LDAP matching rules
which utilize these Stringprep profiles, without having
to jump through too many hops.

One approach here is modify [models] to state that each
individual value of attribute with an equality rule
MUST be equivalent to itself per that rule.

I note that this is behavior of the OpenLDAP server.
Our server prepare values on input and if the preparation
failed, it returns invalidSyntax.

Kurt

---

From: Steven Legg <[email-masked]>
Date: Wed, 17 Nov 2004 16:11:19 +1100

Kurt,

Just so we're clear, I started out arguing that LDAPprep should allow an
empty string as output to avoid anomalous behaviour in substring matching.
I concede that this may produce more matches than the user was expecting.
It is clear to me now that treating all strings and substrings exactly
the same way is the problem, no matter what that way is. I am now arguing
for LDAPprep and/or syntaxes to be revised so that whitespace treatment is
dependent on the context of the (sub)string. Sometimes that means reducing
a string of all spaces to an empty string, and sometimes it doesn't.

More comments below.

Kurt D. Zeilenga wrote:
> implementation do today?

(l= ) matches any string composed entirely of spaces.
(l= *) is equivalent to a presence match.

 > In OpenLDAP, this assertion
> them significance.

The way I see it I am treating leading space as insignificant.
Treating (l= *) like a presence match is saying that leading space
is insignificant. One gets the same result whether or not attribute
values have leading space.

 > And giving leading and trailing spaces

I think I'm actually doing the opposite.

It is LDAPprep that is sometimes giving whitespace in substring assertions
the wrong significance because the context of the substrings is being ignored.

> ask a stupid question, get a stupid answer.

If it's a stupid question then why the concern about breaking applications ?

> expect (l=* * *) to match "x  x"

Given the explicit statement that multiple consecutive spaces are equivalent
to a single space I now think it is quite reasonable to regard (l=* * *) as a
request for something significant separated by whitespace from something
significant separated by whitespace from something significant, so "x  x"
doesn't match.

Following the same point of view, it would be reasonable to regard (l=* x *)
as a request to match something significant separated by whitespace from an
"x" separated by whitespace from something significant. With the current
specifications the request will actually match any string that has an "x" in
it somewhere, regardless of whether it is preceded or followed by whitespace.
This is because LDAPprep applied to the any substring treats the whitespace
as leading and trailing whitespace and removes it leaving only "x".
However, the fact that the user has put the string into an any substring
makes it highly likely the user intended the spaces to be significant.
If we change syntaxes + LDAPprep to conditionally strip leading and trailing
spaces then an any substring of " x " would not be further reduced and the
request would match only those values containing an "x" surrounded by whitespace.

 >(or (l=*  *) to match "x  x"
> but not "x x").

LDAP and X.520 are pretty clear that multiple consecutive spaces are equivalent
to a single space. Anyone who expects "x  x" to match but not "x x" hasn't
read the matching rule descriptions.

 > If one can match insignificant leading and

It doesn't automatically follow since it is clear from X.520 that the significance
of interior spaces is treated differently to leading and trailing spaces.

> matching to support matching of insignificant spaces.

I assume you left a "not" out of there.

I now think we should redesign substring matching so that the treatment of
whitespace in a substring is consistent with the part of the attribute value
that substring is expected to match. Thus, leading whitespace is removed from
an initial substring because the part of the attribute value it potentially
matches also has leading whitespace removed. Likewise, the trailing whitespace
is removed from a final substring because the part of the attribute value it
potentially matches also has trailing whitespace removed. All other consecutive
whitespace characters, including trailing whitespace in an initial substring
and leading whitespace in a final substring, are reduced to a single space because
they are expected to match significant whitespace within the attribute value.

> that such a change is inappropriate and certainly should be

I see it differently. It is clear from X.520 that some spaces are
significant and some are not. We have a problem in substring matching,
as it is currently specified, that it loses track of which is which,
leading to unintuitive results. I'm suggesting a way to fix that.

Regards,
Steven

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 16 Nov 2004 23:00:27 -0800

At 09:11 PM 11/16/2004, Steven Legg wrote:
>the same way is the problem, no matter what that way is.

I actually reached the same conclusion.  As it stands now,
(l=foo * bar) will match "foobar".  That seems counter to
the X.520/LDAP specification of these matching rules.

>I am now arguing for LDAPprep and/or syntaxes to be revised so that whitespace treatment is dependent on the context of the (sub)string. Sometimes that means reducing a string of all spaces to an empty string,
>and sometimes it doesn't.

Problem is that this is not sufficient in the above case.
(l=foo * bar) needs to match "foo bar" as well as "foo  bar"
as well as "foo X bar"  We cannot simply apply LDAPprep to
each substring and the value and match octet wise.  Also,
as Rici noted, we have to be careful not that (l=x*)
doesn't match "x'" where ' is a combining character.

Seems what we to do is remove the insignificant character
removal step from LDAPprep (or have it only collapse
adjacent spaces into one space) and then deal with
space issues in matching rule specification.

This implies a divergence from Stringprep which calls
for the output to compared code-point for code-point.

Kurt

---

From: Hallvard B Furuseth <[email-masked]>
Date: Wed, 17 Nov 2004 18:55:10 +0100

Let's see - attribute values that fail LDAPprep...

* cannot coexist with other values in the same attribute,

* cannot be removed by naming the value, only the entire attribute,

* cannot be (successfully) searched for or compared with,

* probably cannot be used in RDNs - the alternative would be a mess like
  they can exist in the RDN but one cannot look up the entry by its DN
  (since that involves comparing the supplied DN with the entry's DN),
  and maybe not add the entry if another entry exists with the same
  parent (since one must check if the other entry has the same name),

* can mess up a replication scheme which checks if a value is already
  present by EQUALITY-comparing the new value with the old value,

* cannot be named as individual values in access controls, if such
  controls use the EQUALITY rule to check if access is allowed,

* cannot be sorted by the ORDERING rule.

Anything else?

-- 
Hallvard

---

From: Rici Lake <[email-masked]>
Date: Wed, 17 Nov 2004 13:20:32 -0500

From: 	  [email-masked]
	Subject: 	Re: LDAPprep Failure was: Re: LDAPprep: mapping of " " values
	Date: 	November 17, 2004 1:19:32 PM GMT-05:00
	To: 	  [email-masked]

On 17-Nov-04, at 12:55 PM, Hallvard B Furuseth wrote:

 > ...

It seems to me that there are only two possible solutions:

1) Reject as syntactically invalid any value which would
    not compare equal to itself, as Kurt suggested

2) Specify the fall-back of either octet identity matching
    or NFD/NFC matching (i.e. not compatibility, just
    canonicalisation) for values which fail the LDAPprep
    conversion.

I would personally prefer the second option, although it is
not ideal either, since it would allow the use of characters
added to Unicode since Unicode 3.2. Some of these characters
are necessary for the representation of names in African and
Native American languages.

---

From: John McMeeking <[email-masked]>
Date: Wed, 17 Nov 2004 15:35:13 -0600

I've been trying to think through this and some of the other examples
presented regarding spaces.

I suggest that LDAPprep:
- optionally(?) collapse adjacent spaces into a single space
- state that empty string in = empty string out

I assume we can sort out the combining character issues, BiDi, etc. I am
way out of my depth there.

I further suggest that the matching rules define the handling of leading
and trailing spaces in both the entry values and the assertion values.

Preparation for the existing string matching rules would then be something
like:

Entry values are prepared as follows:
1) apply LDAPprep, collapsing adjacent spaces
2) remove all leading and trailing spaces.  This can result in an empty
string.

Filter substrings are prepared as follows:

Note: Its a bit unclear at the protocol level whether (ou=foo) should
result in an initial, any, or final filter component.  I assume all three
are valid and basically equivalent.  [filter] implies it should be an any
filter -- substring = attr EQUALS [initial] any [final].

1) apply LDAPprep.
2) if this is the first substring in a filter, remove leading spaces.  This
can result in an empty string.
3) if this is the last substring in a filter, remove trailing spaces.  This
can result in an empty string.

Using examples previously presented:
(ou= * ) matches everything (empty substring, *, empty substring)
(ou=* ) matches everything
(ou= *) matches everything

Its clear to me that not everyone will agree on this (a solution follows),
but the existing definitions of string matches imply that " foo " matches
"foo" and vice versa.  I don't see how to reconcile that with having " *"
match values with a leading space; having " foo*" match the value " foobar"
ought to imply that "foo*" should NOT match " foobar".

(l= * * ) matches values containing at least one space (empty substring, *,
" ", *, empty substring)

From the note below:
(l=foo * bar) does not match "foo bar" -- it matches "foo X bar" where X is
zero or more non-whitespace characters.  To match "foo bar" would require
preparing "foo " and " bar" differently depending on the contents of the
other substrings - and not just the next or previous substrings.  That
seems unwieldly.  The alternate filter (&(l=foo *bar)(l=foo* bar)) would
match as Kurt seemed to be suggesting "foo * bar" should act.

And, should we ever resolve the empty IA5 String discussion:
"" matches an empty string or values consisting solely of whitespace

If matching with leading and/or trailing spaces is desired, I suggest
defining new matching rules like caseIgnoreWithWhitespaceMatch.  That name
suggests all whitespace is significant.  Maybe LDAPprep should have a
parameter determining whether spaces should be collapsed at all -- or maybe
it should be left to the matching rules to do this.  But my intent is that
this hypothetical caseIgnoreWithWhitespaceMatch rule could behave like:

Entry values are prepared as follows:
1) apply LDAPprep

Filter substrings are prepared as follws:
1) apply LDAPprep
2) if this is the first substring in a filter, remove all but one leading
spaces
3) if this is the last substring in a filter, remove all but one trailing
spaces

Then:
- " foo " does not match "foo" or vice versa
- " *" matches only values with leading whitespace, etc.
- " * * " matches strings with a leading space, trailing space, and at
least one other embedded space surrounded by non-whitespace

John  McMeeking

[email-masked] wrote on 11/17/2004 01:00:27 AM:

> >I am now arguing for LDAPprep and/or syntaxes to be revised so that
> whitespace treatment is dependent on the context of the (sub)string.
> Sometimes that means reducing a string of all spaces to an empty string,

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 18 Nov 2004 07:44:06 +0100

I need to read up on X.520, but:  The behaviour which seems most
consistent with the current ldapbis documents seems to me to be:

- "foo * bar" does not match "foo bar" - the two spaces in the
  substrings do not match _disjoint_ portions of "foo bar".
  Nor does it match "foo  bar", which is equivalent to "foo bar".

- " *" matches all non-empty values.
  (Matches " " where the space is significant, matches other
   non-empty values where an initial space is insignificant.)

- "* *" matches all values containing a space.

- So " * * " matches <anything non-space spaces non-space anything>.
  (Collapse multiple spaces, then combine the two rules above.)

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Thu, 18 Nov 2004 07:15:52 -0800

>  Nor does it match "foo  bar", which is equivalent to "foo bar".

My view is that (l=foo * bar) since "foo X bar" without
consuming any portion of X, it should match "foo  bar"
(which passes your disjoint concern), and since "foo bar"
is equivalent, that too.

>   non-empty values where an initial space is insignificant.)

What about (l= * )?  Does it match " "?  If so, wouldn't the
initial and final be matching non-disjoint portions?

I note that by having " *" match "foo", you are saying that
because initial substring is matching leading spaces in an
equivalent value, it must also match "foo".

Why aren't you applying the same logic to the first case? 

>- "* *" matches all values containing a space.

If " *" matches "foo" then "* *" should match "foo" because
the ANY substring has even less restrictions on it than an
INITIAL (or FINAL) substring.  That is, basically by definition,
if (l=X*) matches, (l=*X*) MUST match for all X.  Likewise for
(l=*X).

>  (Collapse multiple spaces, then combine the two rules above.)

I disagree.  It should match "foo".  This because the INITIAL
substring can match a leading space in an equivalent value,
the ANY string can match a subsequent leading space in an
equivalent value, and the FINAL can match a trailing space
in the an equivalent value.

Kurt

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 18 Nov 2004 17:15:46 +0100

I wonder if I should pretend I never wrote that message...
OTOH, I still wonder if something like it does make the best sense.

>>consistent with the current ldapbis documents seems to me to be:

Yuck.  That was _not_ the introductory statement which should have been
there.  Add "...plus a reasonable way to treat spaces differently next
to the '*' in substrings, plus not treating empty strings as an error in
this context":-(

> (which passes your disjoint concern), and since "foo bar"

The _prepared_ substrings are matched with the _prepared_ attribute
value.  E.g. in [Syntaxes] 4.2.6 (caseExactSubstringsMatch).  That one,
at least, seems clear to me with the present drafts if one is to take
the spaces as significant at all.  It would certainly be useful to have
it match "foo bar" but not "foobar", but I can't justify such a result
in a way I can approve of.

Skipping a few now-irrelevant points - but it all made sense until I got
to...

> if (l=X*) matches, (l=*X*) MUST match for all X.  Likewise for

Oops.  Forgot that one.  I arrived by the other cases by similar logic.
E.g. (l= x*) matches "x", so (l= *) should also match "x".  (l= )
matches " ", so (l= *) should also match " ".  Oh well.

> in the an equivalent value.

That's one I disagree with.  And I don't get the explanation, either
in terms of the current draft nor in terms of changing spaces to be
different in substrings.

-- 
Hallvard

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 18 Nov 2004 17:30:08 +0100

Hallvard B Furuseth writes:
> value.  E.g. in [Syntaxes] 4.2.6 (caseExactSubstringsMatch).

I've lost track of this... Is each substring prepared separately?
Then the spaces in the filter disappear, and it matches "foobar".

Otherwise, if the substringAsssertion is prepared before splitting it
at the '*'s, then hex-escaped characters are going to be troublesome.

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Thu, 18 Nov 2004 11:41:59 -0800

>I've lost track of this... Is each substring prepared separately?

At presently specified, yes.

>Then the spaces in the filter disappear, and it matches "foobar".

Right. And that's a problem.  (l=foo *) and (l=* bar) should not
match "foobar", that's plain wrong IMO.

>at the '*'s, then hex-escaped characters are going to be troublesome.

Whether or not the assertion was presented via the string
representation or via some other means, is irrelevant to
the protocol and the matching.

Kurt

---

From: John McMeeking <[email-masked]>
Date: Fri, 19 Nov 2004 08:30:52 -0600

> (which passes your disjoint concern), and since "foo bar"

I don't see how (l=foo * bar) can match "foo bar" (single space).  Doesn't
that require that the trailing and leading spaces in the filter substring
match the same character?  The * is supposed to match zero or more
characters between two substrings, not allow the substrings to overlap.

I'd also argue that (l=foo * bar) does NOT match "foo  bar" because "foo
bar" is equivalent to "foo bar".  I know it seems like it ought to match,
and I wouldn't complain if it did match, but I think it is reasonable to
say that for matching purposes, the server compares filter substrings to
the equivalent prepared string (according to the matching rule) "foo bar",
which (l=foo * bar) does not match (or so I believe).  I think that would
be far easier to describe than, and has a consistency that I find lacking
in, these examples where sometimes a given space is insignificant, and
sometimes the same space in the same string (entry value or filter
substring) is significant.

John McMeeking

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 19 Nov 2004 07:01:57 -0800

>characters between two substrings, not allow the substrings to overlap.

It's matching the same trailing/space space in the same sense
that (l= * ) matches " ".

>substring) is significant.

Here's the question, can spaces in the asserted value match
insignificant spaces in the attribute value?  If we say no,
then (l= * ) must not match any string.  If we say yes,
then (l= * ) matches every string.  And, as I've shown above,
the choice impacts other matching behavior.

Kurt

---

From: Rici Lake <[email-masked]>
Date: Fri, 19 Nov 2004 10:18:21 -0500

I don't know how complex substring matching should get but it does seem 
to me that there is a use for being able to express word-aligned 
substring matches without jumping through a lot of hoops or thinking 
deeply about edge cases.

How about the following:

-- existing wording of substring matching rule

The rule evaluates to TRUE if and only the prepared substrings of the 
assertion value match disjoint portions of the prepared attribute value 
character string in the order of the substrings in the assertion value, 
and

an <initial> substring, if present, matches the beginning of the 
prepared attribute value character string, and

a <final> substring, if present, matches the end of the prepared 
attribute value character string

-- proposed addition:

, and

an <any> substring, if present and starting with an insignificant space 
as per [strprep] either matches the beginning of the prepared attribute 
value character string or matches the attribute value character string 
at a position following a "breaking space", and

an <any> substring, if present and ending with an insignificant space 
as per [strprep] either matches the end of the prepared attribute value 
character string or matches the attribute value character string at a 
position immediately preceding a "breaking space"

where "breaking space" is defined (similarly to [strprep]) as the SPACE 
(U+0020) code point followed by no combining marks.

-- end of proposed addition.

This would have the effect that initial and final spaces in substring 
matches would have the intuitive meaning of restricting the substring 
match to a word boundary. It would also deal with the issue of spaces 
used as base characters for freestanding combining characters, since 
these would not count as "breaking spaces" or "insignificant spaces".

This would allow, for example, the filter:
   (cn=* John * Ellsworth)
to match:
   John Ellsworth
   P. John Ellsworth
   John P. Ellsworth
which is probably what was intended.

It would also allow the filter:
   (|(description=* Angola *)(description=* Mozambique *))
to find references to lusophone African countries, whether or not the 
keywords appeared in initial or final positions.

I don't believe it would add much complexity to the matching algorithm. 
It might be desirable to add a rule to [strprep] which changed SPACE 
(U+0020) to NO-BREAK SPACE (U+00A0) if the following character is a 
combining character, just before the insignificant space removal step 
(at which point, there cannot be any NO-BREAK SPACEs in the string).

---

From: Hallvard B Furuseth <[email-masked]>
Date: Fri, 19 Nov 2004 16:41:05 +0100

> that (l= * ) matches " ".

I don't see in which sense (l= * ) matches " " because of trailing
space.  With the old semantics, both spaces in the filter are
significant.  If we make both spaces insignificant, (l= * ) matches " "
becuase (l=*) does.

Going back to "foo bar": If you mean that (l=foo * bar) also matches
"foobar", yes, it also matches "foo bar".  If we are talking about
fixing things so it does not match "foobar", then I don't see it.  You
have to treat one of the spaces in the filter as insignificant and one
as significant.  While that may indeed be preferable, it should be done
with the intent that there is something one can replace the '*' with to
turn it into a successful equality match.

Or we can explicitly decide that it would be very useful to have "word
matching" semantics, where " * " is to be treated as a "*" where the
substrings on both sides are to be delimited by space - even if it is
the same space.

[Skipping the rest for now, this is complex enough already...]

-- 
Hallvard

---

From: John McMeeking <[email-masked]>
Date: Fri, 19 Nov 2004 09:47:07 -0600

Buried in a previous post, was my view that (l= * ) matches everything
(presence filter the hard way), following these steps:
1) (l = * *) becomes
    initial " ", any "*", final " "
2) preparation of the filter substrings converts this to
   initial "", any "*", final "" (empty substrings)
which is basically a presence filter.

Or it matches nothing because the leading and trailing spaces in "l" values
are not significant.  I think it would not be acceptable to change the
existing string matching rules to state that leading and trailing spaces in
an entry value are significant (or treat consecutive spaces like anything
other than a single space).

And I suggested that if we want a filter like (l = * ) to match " foo " but
not "foo", we should define additional matching rules.

Furthermore, even with such additional matching rules, I'd expect that (l=
* ) would not match " " (single space), as that requires the same character
to be simultaneously matched by all three filter substrings (or at least
the initial and final substrings).

I think I've said enough on this now and will let the working group get on
with their business.

John  McMeeking

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 19 Nov 2004 08:47:03 -0800

At 07:41 AM 11/19/2004, Hallvard B Furuseth wrote:
>as significant.

Just as in (l= * ) matching " ". 

A single space is significant.  The assertion (l= * ) reads
"match values which start with a space and end with a space".
Can the space the match be same space?
I argue yes, because were there is one space there could be
many spaces.  Hence, each can be viewed as matching different
(disjoint) spaces.

If you take the view that the two spaces in (l= * ) can
only match significant spaces, then (l= * ) cannot matching
any value, not even " ".

>While that may indeed be preferable, it should be done

Er, yes, that is (l=foo * bar) should match "foo X bar" for any
and all X, including an empty X.  "foo " and " bar" are disjoint
parts of "foo X bar" regardless of X.  The insignificant space
rule applies not only the attribute value, but to assertion
value (a sequence of substrings) as a whole.  When matching
against "foo bar", one of the spaces in the assertion value
is insignificant to matching of this attribute value.

One cannot apply the insignificant space rule to individual
substrings as that simply breaks things, leading to oddities
like (l=foo * bar) matching "foobar".

>[Skipping the rest for now, this is complex enough already...]

BTW, I'm heading out for a nice weekend away... so have fun
bashing this out without me.

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 19 Nov 2004 09:05:35 -0800

>which is probably what was intended.

But won't match, as I believe was intended, "JohnEllsworth".

>to find references to lusophone African countries, whether or not the keywords appeared in initial or final positions.

But won't match, as I believe was intended, "Angola&Mozambique".

The problem we have in revised TS is that we're handling
spaces in each substrings in the same way we handle them
in attribute values. This leads to odd behavior.

What we need to do is treat spaces in substrings in
relationship to other substrings and the attribute
value.

-- Kurt

---

From: Rici Lake <[email-masked]>
Date: Fri, 19 Nov 2004 12:30:21 -0500

> But won't match, as I believe was intended, "JohnEllsworth".

Actually, I think the average person searching a directory would be 
surprised at a concatenated match like that. It is actually quite 
common for people to do searches which are effectively (with my 
proposal)

   (cn=* J* Ell*)

Which they would probably expect to match "John Ellsworth" and "Jim 
Ellsworth" but not "Jell-O Corporation of America". But maybe I'm 
wrong. (Had they wanted the more inclusive match, they could always 
have left out the second space, so it may be reasonable to assume they 
put it there for some reason.)

> But won't match, as I believe was intended, "Angola&Mozambique".

Ah, well, there you have me. It wouldn't, and that was perhaps not a 
great example. Nor, as written, would it match "Angola, Mozambique, and 
other southern african countries". Matching is an inexact science :) 
And in any case, it is reasonable to expect that Mozambique is not 
going to show up as part of a word, so the word-break comparison is 
maybe not so useful.

It would be possible to define a different substring matching rule 
based, for example, on the Unicode Technical Report which suggests a 
word break algorithm. However, that would be a lot more work for 
implementors, and in any event I don't believe that report is 
normative. It would certainly not be a good idea to attempt to encode 
in a formal specification something like the behaviour of Perl's 
zero-length word-break assertions.

The modest suggestion I made struck me as a reasonable sort of 
compromise between utility and ease of specification/implementation. 
But it was just a little suggestion; take it for what it's worth.

> relationship to other substrings and the attribute

Sure. That's more or less what I was getting at. It was just one way of 
working such a relationship.

I do think that the distinction between spaces (as per strprep, "not 
following by a combining character") and U+0020 is important, although 
not world-shattering. There will be U+0020 U+0301 sequences 
post-strprep, and they really should not cause false matches. Treating 
them as the same character in a (substring) search but handling them 
differently during insignificant space removal is one of those things 
that comes back to bite you.

R.