Subject: Re: A tricky matched values problem

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 01 Oct 1999 11:58:36 +0200

At 12:48 30.09.99 +0100, Anthony Hodson wrote:
>Hi, David! Erik!!
>
>The solution to this which is given in the FDAM says (to counter this
>kind of problem):
>
>"The matchedValuesOnly argument indicates that certain attribute values
>are to be omitted from the returned entry information. Specifically,
>where an attribute to be returned is multi-valued, and some but not all
>of the values of that attribute contributed to the search filter, in its
>last effective form (i.e. taking relaxed matching rules into account)
>returning TRUE via filter items other than present, then the values that
>did not so contribute are omitted from the returned entry information.
>An attribute value is said to contribute to the search filter if it is
>matched by any non-negated filter item in the filter (see 7.8.1),
>whether or not omission of that item would have led to a different
>outcome in the selection of the containing entry."
>
>Does this resolve the problem?

It seems to me that matchedValuesOnly is asked to perform tasks of 
intuiting the user's state of mind, which usually means that there's 
something wrong with the design.

What about redefining the thing as an attribute return filter:

INPUT:
   Attribute search filter

OUTPUT:
   Only those values of that attribute, for all selected entries, that match
   the filter.

If the intent of the

(OR(AND(mail=[email-masked])
        (telephoneNumber=47))
    (cn=Sean*))

was to return all persons named Sean, and point out the one who had this
email address, it could be written as

(OR(AND(mail=[email-masked])
        (telephoneNumber=47)
    (cn=Sean*)

RETURNFILTERED (mail=[email-masked])

this would return all people named Sean, and the email address for the
one who matched that.

If the intent was something else, one could state that.
Simpler? Better?

                      Harald

--
Harald Tveit Alvestrand, Maxware, Norway
[email-masked]

---

From: Anthony Hodson <[email-masked]>
Date: Fri, 1 Oct 1999 19:05:29 +0100

Dear Harald,

Thanks for this.

In message <[email-masked]>, Harald
Tveit Alvestrand <[email-masked]> writes
>something wrong with the design.

Of course, we started off with a broken design, but the one chosen
reflects the Molesworth principle, which is:

        If in doubt, send too much.

It's also a simple algorithm.

The problem with what you suggest is that, if you have two parts of a
filter, both of which match, which values do you send? 

All the best

Anthony

>If the intent was something else, one could state that.

Anthony Hodson <[email-masked]>      X   X    DDD
XdotD Associates                             X X  O  D  D
Spring Lanes House, Holly Spring Lane         X  OOO D   D  
Bracknell, Berks RG12 2JL, ENGLAND           X X  O  D  D
Tel: +44 1344 310665                        X   X    DDD
Fax: +44 870 056 8242

---

From: "Paul Dale" <[email-masked]>
Date: Mon, 04 Oct 1999 15:38:33 +0100

David Chadwick wrote:

> Thomas Salter providing the following filter:
>
> (OR(AND(mail=[email-masked])(telephoneNumber=47))(cn=
> Sean*))
>
> to an entry containing the following attributes.
>
> cn: Sean Mullan
> mail: [email-masked]
> mail: [email-masked]
> telephoneNumber: +1 781 442 0926
> telephoneNumber: 555-9999
>
> As you can see, the overall filter is true, the AND is false, but the
> mail attribute value matches true. The question is "does matched
> ValuesOnly apply to the mail attribute or not". ie. should one or two
> mail attribute values be returned.
>
> On the FOR side (one returned value), we have that the AVA was
> true,
> On the AGAINST side, we have that the AVA did not contribute to
> the overall filter matching true.
>
> My first reaction was to say that only one mail value should be
> returned, but whilst drafting the proposed textual addition to X.500 I
> have started to change my mind.

I would say "hmm" :-) I think the matchedValuesOnly will be extremely
useful but will also require clients to construct filters carefully. I
would rather have sound logic and careful filters than something that is
not intuitive. If the AND clause does not contribute to truth then the
matchedValuesOnly shouldn't apply. If the intent was to constrain a
telephonenumber search with a mail attribute and also get mail matches as
well then the filter should be,

(OR(AND(mail=[email-masked])(telephoneNumber=47))(mail=[email-masked])(cn=Sean*))

The premise here is that only attributes in a clause expression that
contributes to the overall (outer) truth should contribute. This would
mean that, although the mail part of the AND clause contributes true, the
AND clause evaluates false to the outer expression and therefore should
not contribute values.

> What does anyone else think.

IMHO but open to counter argument. I think a consistent and easily
reasoned scheme is needed.

Paul Dale
--
Product Development Manager
[email-masked]
www.intracus.com
All views expressed are my own and do not represent those of Intracus Ltd.

---

From: "David Chadwick" <[email-masked]>
Date: Sun, 17 Oct 1999 23:43:35 +0100

<color><param>0100,0100,0100</param>First let me apologise for taking so long to answer all the emails on 
this topic, but I have been up to the proverbials in work (I still am, its 
now 10,30pm on Sunday evening).

Firstly I do not believe that the Molesworth principle should apply for 
the matched values control. This is because the user, in specifying 
this control, is obviously trying to limit the size of the response, 
therefore the correct action by the server would be the Molesless 
principle if there were one i.e. if in doubt return less attribute values 
rather than more.

But I am not advocating either principle, but rather I am agreeing 
with Paul Dale that sound logic and a consistent and easily 
reasoned scheme is needed. And as Erik said "we need to be 
precise".

Harald has rightly pointed out that quote "</color>it might be better to have a 
control that tells the server something about what the client wants 
back rather than something about how he wants the search criteria 
treated." What matched values was trying to do was both with one 
simple boolean. ie. the matched values boolean was saying 
something about what the client wanted back (i.e. limit the values), 
but using the search filter to specify precisely which values are to 
be limited in the return. The trouble as we can see arose because 
the search filter can be a mixture of truths, unknowns and falses, 
and a simple matchedValuesOnly boolean was too blunt a tool (as 
currently specified) to differentiate between them. It therefore was 
not precise enough in its specification.

Therefore it seems that there are two sensible options that can be 
taken:

<bold>1) is to tighten up considerably on the definition of the current 
MVO boolean. </bold>There are three views of how to tighten up the 
definition. 1A) One is to say that only those attributes that 
contributed to the overall truth of the filter are governed by the MVO 
boolean. 1B) Another is to say that if the filter item  was true then 
MVO applies to the attributes in the filter item (even if the subfilter 
containing the filter item was false). 1C) And the final one suggested 
by Bruce was if the attribute was in the filter, even if the filter item 
was false, then apply MVO.

<bold>2) the other is to replace the boolean by a return filter </bold>(as 
suggested by both Harald and Bruce) that acts on the attributes to 
be returned and filters out some of the values. This works 
independently of, and after, the Search filter.

My preferred approach is the first, 1A) i.e.only those attributes that 
contributed to the overall truth of the filter are governed by the MVO 
boolean. My reasons are as follows:

I reject the current X.500 text (option 1B) for the following reason:

subfilters that evaluate false or undefined might as well be missing 
from the filter, and if they were there on their own then no entries 
would be returned. THerefore they have no claim to act on the 
returned values of any entries.

I reject option 1C) as it is too loose and it does not work properly for 
false filter items, as presumably no attribute values would be 
returned from these attributes.

Option 2) has its merits, and is clean. The reason for rejecting it is 
compatibility with X.500, and that option 1A can (as Mark points out) 
do the same task anyway, but perhaps not quite as efficiently. So 
option 2) might be more efficient than 1A) but at the expense of 
compatibility with existing implementations.

I am happy to update the ID was the proposed solution 1A unless 
anyone has strong objections to this.

David

<nofill>
***************************************************

David Chadwick
IS Institute, University of Salford, Salford M5 4WT
Tel +44 161 295 5351  Fax +44 161 745 8169
Mobile +44 790 167 0359
Email [email-masked]
Home Page  http://www.salford.ac.uk/its024/chadwick.htm
Understanding X.500  http://www.salford.ac.uk/its024/X500.htm
X.500/LDAP Seminars http://www.salford.ac.uk/its024/seminars.htm
Entrust key validation string MLJ9-DU5T-HV8J

***************************************************

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 18 Oct 1999 07:39:08 +0200

My objection is not particularly strong, but if X.500 does 1B and your 1A 
will in some cases return a different set of values, then X.500 
compatibility is broken anyway, and we should do what makes sense, rather 
than the smallest possible incompatible motification to X.500.

If a return filter is the Right Thing, then it is also the Right Thing for 
X.500, and X.500 should change.

                             Harald

---

From: Anthony Hodson <[email-masked]>
Date: Mon, 18 Oct 1999 11:09:34 +0100

Hi, David!

Good to have your analysis, which I *generally* agree with.  But ...

With 1A, consider a filter OR(A, B) where, with a particular entry, both
A and B are true; as a result of this (using 1A), assume that A would
cause a value x to be returned, and that B would cause a value y to be
returned.

Presumably you would return both x and y with 1A, even though neither A
nor B could be *completely* said to have contributed, since either could
be omitted without affecting the outcome.

It was consideration of this kind of issue that led to the simple and
easy-to-implement rough-justice rule: if a filter item is TRUE, send the
value anyway.  I don't defend it at all on philosophical grounds.

My view is that your solution 2 is *much* the better solution because of
its direct approach to solving the problem, since matched-values was
always a bit kloodgy for the reasons that you suggest.  I think that it
would be much better for FDAS/F.510, too.  Something has to change with
existing MVO implementations, and (2) presents no technical challenges.

Best wishes

Anthony

In message <[email-masked]>, David Chadwick
<[email-masked]> writes
>First let me apologise for taking so long to answer all the emails on  this 
>topic, but I have been up to the proverbials in work (I still am, its  now 
>10,30pm on Sunday evening).
>
>Firstly I do not believe that the Molesworth principle should apply for  the 
>matched values control. This is because the user, in specifying  this control, 
>is obviously trying to limit the size of the response,  therefore the correct 
>action by the server would be the Molesless  principle if there were one i.e. if 
>in doubt return less attribute values  rather than more.
>But I am not advocating either principle, but rather I am agreeing  with Paul 
>Dale that sound logic and a consistent and easily  reasoned scheme is needed. 
>And as Erik said "we need to be  precise".
>
>Harald has rightly pointed out that quote "it might be better to have a  control 
>that tells the server something about what the client wants  back rather than 
>something about how he wants the search criteria  treated." What matched values 
>was trying to do was both with one  simple boolean. ie. the matched values 
>boolean was saying  something about what the client wanted back (i.e. limit the 
>values),  but using the search filter to specify precisely which values are to  
>be limited in the return. The trouble as we can see arose because  the search 
>filter can be a mixture of truths, unknowns and falses,  and a simple 
>matchedValuesOnly boolean was too blunt a tool (as  currently specified) to 
>differentiate between them. It therefore was  not precise enough in its 
>specification.
>
>Therefore it seems that there are two sensible options that can be  taken:
>
>1) is to tighten up considerably on the definition of the current  MVO boolean. 
>There are three views of how to tighten up the  definition. 1A) One is to say 
>that only those attributes that  contributed to the overall truth of the filter 
>are governed by the MVO  boolean. 1B) Another is to say that if the filter item  
>was true then  MVO applies to the attributes in the filter item (even if the 
>subfilter  containing the filter item was false). 1C) And the final one 
>suggested  by Bruce was if the attribute was in the filter, even if the filter 
>item  was false, then apply MVO.
>
>2) the other is to replace the boolean by a return filter (as  suggested by both 
>Harald and Bruce) that acts on the attributes to  be returned and filters out 
>some of the values. This works  independently of, and after, the Search filter.
>
>contributed to the overall truth of the filter are governed by the MVO  boolean. 
>My reasons are as follows:
>
>subfilters that evaluate false or undefined might as well be missing  from the 
>filter, and if they were there on their own then no entries  would be returned. 
>THerefore they have no claim to act on the  returned values of any entries.
>
>I reject option 1C) as it is too loose and it does not work properly for  false 
>filter items, as presumably no attribute values would be  returned from these 
>attributes.
>
>compatibility with X.500, and that option 1A can (as Mark points out)  do the 
>same task anyway, but perhaps not quite as efficiently. So  option 2) might be 
>more efficient than 1A) but at the expense of  compatibility with existing 
>implementations.
>
>I am happy to update the ID was the proposed solution 1A unless  anyone has 
>strong objections to this.
> David
>
>David Chadwick IS Institute, University of Salford, Salford M5 4WT Tel +44 161 
>295 5351  Fax +44 161 745 8169 Mobile +44 790 167 0359 Email 
>[email-masked] Home Page  http://www.salford.ac.uk/its024/chadwick.h
>tm Understanding X.500  http://www.salford.ac.uk/its024/X500.htm X.500/LDAP 
>Seminars http://www.salford.ac.uk/its024/seminars.htm Entrust key validation 
>string MLJ9-DU5T-HV8J

Mob: +44 771 360 7086
Fax: +44 870 056 8242

---

From: Anthony Hodson <[email-masked]>
Date: Mon, 18 Oct 1999 11:12:47 +0100

Hi, David!

Best wishes

Anthony

---

From: Bruce Greenblatt <[email-masked]>
Date: Mon, 18 Oct 1999 22:35:38 -0700

I would definitely choose option 2.  I think that any version of option 1
overly (conf)uses the Search filter.  I further believe that for an
application to use option 1 that it already has to know the answer to the
answer to part of the search filter.  Further still, it will not generally
be the case that every attribute type mentioned in the search filter should
be used to contribute to the MVO control computation.  For example,
consider this pseudo-filter:

(&(email=*dtasi.com) (userCertificate = [certIssuedBy VeriSign Class2 CA]))

With the MVO control in the option 1 format, there is no way to indicate
that I want all of the email addresses back, and only the userCertificate
attribute values that were issued by the VeriSign Class 2 CA.  So, it
appears to me that the option 1 MVO control meets a sufficiently small
portion of application developer requirements to not be that useful.

Thus, I prefer option 2 which allows the application to cleanly separate
the two different requirements.

Bruce

>X.500, and X.500 should change.
==============================================
Bruce Greenblatt, Ph. D.
Directory Tools and Application Services, Inc.
http://www.directory-applications.com

---

From: "David Chadwick" <[email-masked]>
Date: Tue, 19 Oct 1999 14:25:37 +0100

Date forwarded: 	Sun, 17 Oct 1999 22:38:09 -0700 (PDT)
Date sent:      	Mon, 18 Oct 1999 07:39:08 +0200
To:             	[email-masked], [email-masked]
From:           	Harald Tveit Alvestrand <[email-masked]>
Subject:        	Re: A tricky matched values problem
Copies to:      	[email-masked]
Forwarded by:   	[email-masked]

> than the smallest possible incompatible motification to X.500.

X.500 at the moment is unclear and so we dont know what 
implementations actually will do. When I said X.500 does 1B, I 
should have been more precise and said that the proposed 
clarifying text for X.500 (2000) proposes 1B. I believe the sparse 
text in X.500(97) would tend you to plump for option 1A, but the text 
is so sparse that implementors could choose 1A or 1B. Hence we 
need to nail down precisely what X.500  implementations MUST do 
with this control. And of course, the input to the LDAP list will 
contribute to the decision made by the X.500 meeting next week in 
Copenhagen.

> X.500, and X.500 should change.

Yes, and I am starting to see support for this approach on the list 
too, so if we can get X.500(2000) to add this at the final editing 
meeting in Copenhagen next week, plus fix the text for MVO then we 
are in a strong position for getting alignment between X.500 and the 
new LDAP extension (which could be one or two extensions, 
depending upon the choice of the list)

David

David Chadwick