Subject: Returning single values from multivalued attributes

From: "David Chadwick" <[email-masked]>
Date: Thu, 5 Aug 1999 19:52:10 +0100

This topic has been briefly discussed on this list before (30 July), 
but no conclusions were reached. Briefly the situation is that X.500 
DAP allows a user to search an entry and only request that matched 
values are returned from a multi-valued attribute rather than all 
attributes. LDAP only allows all or no values to be returned.

There has also been a request in the PKIX group that LDAP should 
allow a single user certificate to be returned (the one that matches 
the users filter), rather than all the users certificates.

I believe that once clients start to retreive schema definitions they 
will also want matched values only to be returned.

There are a couple of approaches this group can take

i) say that this is not a significant problem and ignore it. Let the 
client sort out the value it wants

ii) say that it is a significant problem and try to fix it via a new 
matchedValuesOnly control ID. (I can volunteer to write the ID if 
people are interested in it)

What do people think about this?

David

***************************************************

David Chadwick
IS Institute, University of Salford, Salford M5 4WT
Tel +44 161 295 5351  Fax +44 161 745 8169
Mobile +44 790 167 0359
*NEW* Email [email-masked] *NEW*
Home Page  http://www.salford.ac.uk/its024/chadwick.htm
Understanding X.500  http://www.salford.ac.uk/its024/X500.htm
X.500/LDAP Seminars http://www.salford.ac.uk/its024/seminars.htm
Entrust key validation string MLJ9-DU5T-HV8J

***************************************************

---

From: Chris Apple <[email-masked]>
Date: Thu, 05 Aug 1999 15:44:36 -0400

I already have a need for such a feature. I was considering use of an
extensible matching rule to implement the same sort of thing.
Unfortunately, I don't have time to contribute to the document.

Chris.

--
------------------------------------------------------------------------
Chris Apple                     Business Site: AnyWho Directory Service
Internet Directory Group                       http://www.anywho.com
AT&T Labs
[email-masked]
+1 973 236 6470                 Tired of slow directories?  Try AnyWho.
------------------------------------------------------------------------

---

From: "David Chadwick" <[email-masked]>
Date: Thu, 5 Aug 1999 21:36:33 +0100

Date sent:      	Thu, 05 Aug 1999 15:44:36 -0400
From:           	Chris Apple <[email-masked]>
Send reply to:  	[email-masked]
Organization:   	AT&T Labs
To:             	[email-masked]
Copies to:      	[email-masked]
Subject:        	Re: Returning single values from multivalued attributes

> Unfortunately, I don't have time to contribute to the document.

This could be made to work of course, but a control would be better 
I think, since existing matching rules can then be used, and this 
maintains compatibility with X.500 which already has a boolean in 
the DAP to switch the feature on and off.

David

David Chadwick

---

From: [email-masked] (Mark C Smith)
Date: Thu, 05 Aug 1999 17:09:56 -0400

> the users filter), rather than all the users certificates.

Is it a valid assumption that most entries will contain only a few
certificates?  If so, the value of returning only the one that is
matched is reduced.  Of course certificates are typically fairly large,
so reducing the amount of data sent might be useful.  My guess is that
reducing the number of network round trips is more important than
reducing the data itself -- and a matchedValuesOnly control won't change
help there.

> ...

Some clients do this now.  I suspect many of them download all of the
schema in a subschemasubentry the first time they need to make use of
any of it and then cache it... which makes future use fast but involves
fetching many (potentially) unneeded values during the initial subschema
search.  Personally, I think it would be better to represent each schema
element as a separate entry, but a matchedValuesOnly control would help
here.  Apparently Microsoft's Active Directory does represent each
schema element as its own entry (they also support the standard "all
schema elements in one subentry" approach).

> What do people think about this?

I lean towards (i) but I'd like to have more discussion about this --
both about how people envision using this feature and how the existing
feature works in X.500 DAP.  Can you explain exactly how the
matchedValuesOnly argument works in DAP?  I find the text in X.511 to be
quite confusing.  Some basic questions I have:

a) Does matchedValuesOnly affect single-valued attributes or just
multivalued ones?

b) Can matchedValuesOnly be used with equality filters?

-- 
Mark Smith
iPlanet Directory Architect / Sun-Netscape Alliance
My words are my own, not my employer's.   Got LDAP?

---

From: Sean Mullan <[email-masked]>
Date: Thu, 05 Aug 1999 17:12:38 -0400

Hi David,

I'd like to work on an LDAP draft with you as I believe this is very
important. (Actually, I was planning to write a draft but I was a
bit discouraged by the lack of response to my message on 7/30). Please 
let me know how you want to coordinate this.

I think that 2 drafts are probably needed, one describing a control for the
matchedValuesOnly feature and another for describing the X.509 certificate
and CRL matching rules as new LDAP matching rules.

Thanks,
--Sean

-- 
Sean Mullan			Email: [email-masked]
Sun Microsystems Laboratories	Tel:   (781) 442-0926	
One Network Drive		Fax:   (781) 442-1692
Burlington, MA 01803-0902

---

From: [email-masked] (David Boreham)
Date: Thu, 05 Aug 1999 14:16:51 -0700

Also of interest in the context of this thread is
Jim Sermersheim's duplicate entry control,
which allows a client to
ask for a multi-valued attribute to be returned 
as multiple _entries_ (each with the same DN and
once single attribute). This is also usable with
VLV, allowing a client to page or scroll through
the values.

http://www.ietf.org/internet-drafts/draft-ietf-ldapext-ldapv3-dupent-01.txt

---

From: [email-masked] (Rob Weltman)
Date: Thu, 05 Aug 1999 14:23:35 -0700

> http://www.ietf.org/internet-drafts/draft-ietf-ldapext-ldapv3-dupent-01.txt

  I think they're somewhat orthogonal in usefulness. The duplicate entry
control will help a lot with displaying the contents of static groups (when
combined with a VLV control) in a browsing way. The matchedValuesOnly control
will help when searching (the certificate example is a great one, but it would
also be helpful for e.g. showing only the members of a group that matched
search criteria).

Rob

---

From: [email-masked] (David Boreham)
Date: Thu, 05 Aug 1999 14:30:00 -0700

> also be helpful for e.g. showing only the members of a group that matched

Indeed, but I wanted to make sure folk here knew about
the dupent control. For example, it might be more useful
in the context of retrieving schema definition entries
than the matchedValuesOnly control.

---

From: [email-masked] (Rob Weltman)
Date: Thu, 05 Aug 1999 14:31:42 -0700

> than the matchedValuesOnly control.

  Oh no! For schema I would take the matchedValuesOnly control any time! Most of the
time a client gets attributes back from a search and would like to easily know more
about them (syntax, multi-valued, ...). There are server management programs that
will want to browse the schema, but that is a much smaller niche.

Rob

---

From: [email-masked] (David Boreham)
Date: Thu, 05 Aug 1999 14:47:36 -0700

> will want to browse the schema, but that is a much smaller niche.

Yes, I was concerned with that niche.

---

From: Sean Mullan <[email-masked]>
Date: Thu, 05 Aug 1999 18:10:30 -0400

Mark C Smith wrote:
> certificates?  

A CA may store a potentially large number of cross certificates 
issued to or from other CAs. When building certification paths, it is 
useful to be able to search for a subset of certificates which satisfy
policy and name constraints, validity and other parameters required
for validating the certification path.

I can imagine an end-entity may also store many certificates, issued
by different CAs with different key usages and policies or certifying different
application specific identities. Searching for another user's S/MIME
certificate would involve constructing a certificate matching rule which
matched on the user's email address in the SubjectAltName component and minimally
an encryption bit in the keyUsage component. A rule such as this, when
combined with a matchedValuesOnly control could prevent the client from
downloading a large number of irrelevant certificates. 

> If so, the value of returning only the one that is

I believe the matchedValuesOnly control could be abused if not used
wisely, especially if you have to perform multiple searches to find the
right value. But I think it is very valuable in certain scenarios
such as certificate searching and certification path building.

--Sean

---

From: Chris Apple <[email-masked]>
Date: Thu, 05 Aug 1999 20:13:16 -0400

I agree that a control is a better way to implement it.

Chris.

---

From: Ella Paton Bassett <[email-masked]>
Date: Fri, 06 Aug 1999 12:28:49 -0400

David:

We would like to see these drafts written. Thanks for pointing out that
we need the matched values only feature in addition to the certificate
matching rule to select a particular value of a certificate using LDAP. 

We are also curious to know if client vendors intend to implement
attribute types other than userCertificate to retrieve certificates from
the directory using LDAP v3.

Ella

---

From: "David Chadwick" <[email-masked]>
Date: Sat, 7 Aug 1999 00:44:54 +0100

Mark Smith wrote

> I lean towards (i) but I'd like to have more discussion about this -- both
> about how people envision using this feature and how the existing feature
> works in X.500 DAP.  Can you explain exactly how the matchedValuesOnly
> argument works in DAP?  I find the text in X.511 to be quite confusing. 
> Some basic questions I have:
> 

Its effect on a single valued attribute is null.

> b) Can matchedValuesOnly be used with equality filters?

No, but the easy work around to this is to put the equality filter in the 
extended filter element of the search.

David

David Chadwick

---

From: "David Chadwick" <[email-masked]>
Date: Sat, 7 Aug 1999 00:44:54 +0100

Date sent:      	Thu, 05 Aug 1999 17:12:38 -0400
From:           	Sean Mullan <[email-masked]>
Organization:   	Sun Microsystems
To:             	[email-masked]
Copies to:      	[email-masked]
Subject:        	Re: Returning single values from multivalued attributes

> bit discouraged by the lack of response to my message on 7/30). Please let
> me know how you want to coordinate this.

Great, if you want to write the first version and send it to me, as I am 
going on holiday next week for some time so I wont be able to 
publish the draft myself until September.

> 
> I think that 2 drafts are probably needed, one describing a control for
> the matchedValuesOnly feature and another for describing the X.509
> certificate and CRL matching rules as new LDAP matching rules.

I have already made a start on this in the soon to be released 
update of LDAPv3 profile for PKIX. Mark Wahl has said he will wrap 
the new matching rules into the revision of RFC 2252 when he 
updates it.
So we probably dont need the second draft.

David

> 
> Thanks,
> --Sean
> 
> David Chadwick wrote:
> > 
> > This topic has been briefly discussed on this list before (30 July), but
> > no conclusions were reached. Briefly the situation is that X.500 DAP
> > allows a user to search an entry and only request that matched values
> > are returned from a multi-valued attribute rather than all attributes.
> > LDAP only allows all or no values to be returned.
> > 
> > allow a single user certificate to be returned (the one that matches the
> > users filter), rather than all the users certificates.
> > 
> > matchedValuesOnly control ID. (I can volunteer to write the ID if people
> > are interested in it)
> > 
> > What do people think about this?> > 

David Chadwick

---

From: [email-masked] (Mark C Smith)
Date: Mon, 09 Aug 1999 14:17:05 -0400

> Its effect on a single valued attribute is null.

I find it surprising and confusing that the behavior is different when
an attribute has multiple values vs. when it only has one.  Just so I
understand how this works, here's an example.  Suppose entry E1 has one
userCertificate value and entry E2 has two such values.  Further suppose
that a client wants to always retrieve the cn attribute from an entry
but wants to receive a userCertificate value only if the cert value
presented in a filter matches one of the certs present in the entry. 
For E1, this will require two search operations (base search to grab the
cn and another to test for the cert) but for E2 it can be done in one
(base search with a filter like
"(|(cn=*)(userCertificate;binary=<DER>))" with matchedValuesOnly set to
TRUE).  Correct?  Or by "single valued attribute" do you mean an
attribute type that allows multiple values?  Technically, the filter I
present above won't work because presence and equality filters don't
work with matchedValuesOnly (see below).

> extended filter element of the search.

I'd like to see us fix this problem when we define a matched values only
control for LDAPv3.  Presence filters are not allowed either.  This
seems like an unnecessary limitation to me.

---

From: Helmut Volpers <[email-masked]>
Date: Tue, 10 Aug 1999 09:46:35 +0200

Mark C Smith wrote:
> an attribute has multiple values vs. when it only has one. 

I didn't follow the complete discusion but what is surprising ?
When I have a single valued attribute I don't need a matchedValuesOnly
because if it
match you get it back and need no additional flag.

 Just so I
> TRUE).  Correct?  

Not correct. If you make this search and the userCertificate match that
in E1
you get the entry and userCertificate back I hope.
But is this the normal search to find somebody in the directory where
the client already know the complete userCertificate ? I think you have
some parts
of the userCertificate and use the special matching rules to find the 
complete userCertificates.

Or by "single valued attribute" do you mean an
> work with matchedValuesOnly (see below).

I see no reason at the moment why the matchedValuesOnly cannot be used 
with equality filters ? David is there a reason ?
> control for LDAPv3. 

I agree with you.

 Presence filters are not allowed either.  

How do you want to use matchedValuesOnly with the presence filter ?

This

---

From: [email-masked] (David Boreham)
Date: Tue, 10 Aug 1999 07:03:17 -0700

> match you get it back and need no additional flag.

I think the problem is that you might not know in advance whether it's
single or multi-valued.

---

From: Mark Smith <[email-masked]>
Date: Tue, 10 Aug 1999 10:06:23 -0400

> match you get it back and need no additional flag.

True.  But it is certainly possible to construct compound filters that
will always return the entry but may have one or more components that do
not match.  Simple examples can be constructed using OR, e.g.,
(|(cn=*)(member=<dn>))  Maybe no one wants to use such filters, but it
seems inconsistent for the matchedValuesOnly flag to have differing
behavior depending on how many values are present.

> you get the entry and userCertificate back I hope.

My example is contrived, but the idea is that I want to always get back
the entry's cn and also test for the presence of a particular
certificate at the same time.  For E1, if my filter does not match the
certificate, I don't want to get the certificate back (even though the
other cn=* component of the filter does cause the entry to be returned).

> complete userCertificates.

Sure, it might be that special matching rules are used.  But that
doesn't change the spirit of my example.

> How do you want to use matchedValuesOnly with the presence filter ?

Probably there is no good reason to do so.  The behavior of
matchedValuesOnly just strikes me as inconsistent.  Maybe David or
someone can explain what the correct server behavior is if a presence
filter component is submitted along with matchedValuesOnly == TRUE.  Is
an error returned to the client?

Anyway, I look forward to the draft on this topic.

---

From: Helmut Volpers <[email-masked]>
Date: Tue, 10 Aug 1999 17:17:36 +0200

> I think the problem is that you might not know in advance whether it's

I mean you must not know whether it's single or multivalued, you should
get it back if it matches if matchedValuesOnly is that, independant
whether its single or multivalued.

---

From: Bruce Greenblatt <[email-masked]>
Date: Tue, 10 Aug 1999 08:19:43 -0700

I'm a little confused by this.  My perspective is that the requirements for
this new functionality arise almost entirely from poor Schema/DIT design.
I certainly agree that if you put 1000s of certificates in a single
attribute, you will have difficulty retrieving a single certificate
instance from the entry inquestion.  The obvious solution to this problem
is NOT to enhance LDAP (or DAP for that matter).  The obvious solution IS
"don't put 1000s of certificates in a single attribute".

Bruce
==============================================
Bruce Greenblatt
Directory Tools and Application Services, Inc.
http://www.directory-applications.com

---

From: "David Chadwick" <[email-masked]>
Date: Wed, 18 Aug 1999 16:18:50 +0100

Date sent:      	Tue, 10 Aug 1999 10:06:23 -0400
From:           	Mark Smith <[email-masked]>

> behavior depending on how many values are present.

Mark
I dont quite follow you here. Firstly, the user chooses which 
attributes are to be returned. Secondly, if the chosen attributes did 
not contribute to the filter, then matchedValuesOnly has no effect on 
them. ONly if the chosen attributes were also in the filter and they 
contributed to the matching, are the matched values only returned. 
Therefore there is no differing or inconsistent behaviour dependent 
upon the number of values in the attribute. If the attribute did not 
contribute, matched values is ignored, if the attribute did contribute 
(irrelevant how many values it has) then only the ones that matched 
are returned.

> the entry's cn and also test for the presence of a particular certificate
> at the same time.  For E1, if my filter does not match the certificate, I
> don't want to get the certificate back (even though the other cn=*
> component of the filter does cause the entry to be returned).
> 
> 

This is where your misunderstanding comes in, I think. YOu decide 
whether you want the certificate attribute to be returned 
INDEPENDENT of matched values only. This is set in the attributes 
parameter of the Search. You cannot say "I only want this attribute 
if it contributed to the filter". Matched values says whether you want 
all certificates back or only some of them, but does not allow you to 
say none of them (which is what you wanted to do in your contrived 
example. For this you would need another control that has not been 
suggested yet by X.500 or LDAP

> > with equality filters ? David is there a reason ?

There was a reason originally proposed by Australia Telecom, but I 
cannot remember what it was. (I remember arguing against the 
restriction at the time, but lost, as it remained.)

> filter component is submitted along with matchedValuesOnly == TRUE.  Is an
> error returned to the client?
> 

Matched values does not make sense with a presence filter, since 
presence is only testing the attribute type and not any of the values, 
so how can you then ask for the matched values only? I would say 
that the matched values is simply ignored by the server.

David

David Chadwick

---

From: "David Chadwick" <[email-masked]>
Date: Wed, 18 Aug 1999 16:18:51 +0100

Date sent:      	Tue, 10 Aug 1999 07:03:17 -0700
From:           	[email-masked] (David Boreham)
To:             	Helmut Volpers <[email-masked]>
Copies to:      	Mark C Smith <[email-masked]>, [email-masked],
 	[email-masked]
Subject:        	Re: Returning single values from multivalued attributes

> > because if it match you get it back and need no additional flag.
> 

Irrelevant in my opinion. What the user is saying is "give me this 
value only and ignore the rest". If there is only a single value then 
the ignored values are none. But the user gets what he wanted 
independent of the number of actual values there are

David

David Chadwick

---

From: Mark Smith <[email-masked]>
Date: Wed, 18 Aug 1999 15:11:01 -0400

> (irrelevant how many values it has) then only the ones that matched

Thanks for the clarification.  So a more precise name for the
matchedValuesOnly flag is
"matchedValuesOnly-WhereAnyValuesWereMatchedAtAll."  In other words, if
no values for an attribute type are matched by a component of the search
filter, the matchedValuesOnly flag has no effect on the values returned
for that type.

---

From: "David Chadwick" <[email-masked]>
Date: Fri, 20 Aug 1999 17:15:26 +0100

Date sent:      	Wed, 18 Aug 1999 15:11:01 -0400
From:           	Mark Smith <[email-masked]>

> "matchedValuesOnly-WhereAnyValuesWereMatchedAtAll."  In other words, if no
> values for an attribute type are matched by a component of the search

Thats correct. Glad we've sorted this one out. Do you still have a 
requirement for your "only return this attribute if it matched, 
otherwise dont" flag

David

David Chadwick

---

From: Mark Smith <[email-masked]>
Date: Fri, 20 Aug 1999 13:32:47 -0400

> requirement for your "only return this attribute if it matched,

I don't think so.  I was just trying to understand how X.500's
matchedValuesOnly flag works.  I see that the I-D you just posted on
this topic is clearer than the X.500 specification -- thanks!