Subject: FW: Active Directory question

From: "Schleiff, Marty" <[email-masked]>
Date: Thu, 15 Apr 2004 10:50:06 -0700

Gentlemen,

Can you please let me know your impressions about the MS Active
Directory response  with ranges of multi-valued attribute values?
Also, using tools lke ldapsearch,  how could I retrieve subsequent
ranges?
 
Thx,

[email-masked];  CISSP 
Associate  Technical Fellow - Cyber Identity Specialist 
IT Access & Security  Services 
(425)  957-5667 
-----Original Message-----
From: Chris Harding  [mailto:[email-masked]] 
Sent: Wednesday, April 14, 2004  11:20 AM
To: Schleiff, Marty
Subject: RE: Active Directory  question

Hi, Marty -

Thanks - sounds like this is  definitely one for the IETF experts!

At 18:52 14/04/2004, you wrote:
Hi Dr. harding,

Thanks for your response. I'd like to point out that this issue is
not  about a server limiting the number of entries to return; instead
it's about  the number of values within a single multi-valued
attribute to return. The  entry gets returned, but not all its
attribute values.

[email-masked]; CISSP 
Associate  Technical Fellow - Cyber Identity Specialist 
IT Access & Security Services 
(425) 957-5667 
-----Original Message-----
From: Chris Harding [mailto:[email-masked]] 
Sent: Wednesday, April 14, 2004 9:32 AM
To: Schleiff, Marty
Subject: Re: Active Directory question

Hi, Marty -

Our Product Standard is based on the IETF RFCs, so I think this
would be  legal behavior for an LDAP Certified server only if it
is legal according to  RFC 2251. Now the RFC says that "Servers may
enforce a maximum number of  entries to return" (section 4.5.1 under
"sizelimit") so it looks to me as  though the behavior may be legal.
However, I have got my fingers burnt  before trying to interpret
this RFC, and I suggest you send mail to the  ldapbis list
([email-masked]) if you want to find out what the  IETF
experts think.

At 22:57 13/04/2004, you wrote:
Hi Dr. Harding,

Microsoft Active Directory responds to queries on groups having
more  than 1024 members with the first 1000 members, with the
'member' attribute  changed to 'member;range=0-999'. See:
http://www.hut.fi/cc/docs/kerberos/nss_ldap.html In TOG's efforts
to brand "ldap-compliant" servers and applications,  is this practice
condoned? So far I've not been able to figure out how to  get the
next batch of members; I'm not sure it's possible via  LDAP.

[email-masked]; CISSP
Associate Technical Fellow - Cyber Identity Specialist
IT Access & Security Services
(425) 957-5667

Regards,

Chris
+++++

===========================================================================
           Dr. Christopher J. Harding
  T H E    Executive Director for the Directory  Interoperability Forum
 O P E N   Apex Plaza, Forbury Road, Reading RG1 1AX,  UK
G R O U P  Mailto:[email-masked] Phone: +44 118 902  3018
           WWW: http://www.opengroup.org Mobile: +44 774 063 1520
===========================================================================
Boundaryless Information Flow: Managing the Flow
Brussels Hilton Hotel, Brussels, Belgium.  April 19-23, 2004
http://www.opengroup.org/brussels2004/
===========================================================================

Regards,

Chris
+++++

===========================================================================
            Dr. Christopher J. Harding
  T H E    Executive Director  for the Directory Interoperability Forum
 O P E N   Apex  Plaza, Forbury Road, Reading RG1 1AX, UK
G R O U P  Mailto:[email-masked] Phone: +44 118 902 3018
           WWW: http://www.opengroup.org  Mobile: +44 774 063  1520
======================= ==================================================== 
Boundaryless  Information Flow: Managing the Flow
Brussels Hilton Hotel, Brussels,  Belgium.  April 19-23, 2004
http://www.opengroup.org/brussels2004/
===========================================================================

---

From: "Ramsay, Ron" <[email-masked]>
Date: Tue, 20 Apr 2004 17:44:35 +1000

Hi Jim,

You seem to be completely bypassing the issue of whether this *option* can be parsed. With all the options I have seen, it is possible to find the value because none of them allow '=' in the tag. Having an '=' in the range tag means that you must be expecting it, for you cannot find the value otherwise.

Compare

  cn;lang-en=myname

with

  cn;range=001=myname

Normal options are terminated by ';' or '=' and can therefore be ignored. This option must actually be recognised to be ignored.

So, apart from your metaphysical argument below, there is a real, practical issue.

Ron

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Jim Sermersheim
Sent: Tuesday, 20 April 2004 16:55
To: [email-masked]
Cc: [email-masked]; [email-masked];
[email-masked]; [email-masked]
Subject: Re: Active Directory question

I'm not sure exactly what the meanings of 'truly optional' 'truly non-optional' are, or whether there is the notion of 'optional * but not truly'. I agree with adding something like you suggest as long as the terminology is defined. 
 
But to take the issue a bit further, regarding your first point; The fact that language tags and the binary options conform to RFC 2251 doesn't make me feel that they are any less flawed. The attribute options feature (in my mind) is clearly an extension mechanism. Without means for discovery or solicitation, I believe this extension mechanism is fundamentally flawed.
 
Second, I suspect that the introduction of any new option (especially those that are not 'tagging' options) is going to cause a number of applications to encounter problems unless there is a client solicitation mechanism for it. Currently, a client can only recognize those attribute options that are known to it and fail over to treating attributes with unknown options as either unrecognized attributes, or (worse in some cases, better in others) treat them like tagging options. Who knows what other types of options will be concocted in the future? As long as it's possible to invent an option which (like the range and binary options) can cause mandatory attributes to be treated as unrecognized by a client, I see problems.
 
Third, when you say things like: "While <the binary option> could technically be used with other attribute types (e.g., cn), it is recognized that returning such without solicitation would be problematic" I disagree. Obviously people who don't recognize this class of problems are writing extension documents and implementing them. Furthermore, I don't remember any discussion where anyone warned the authors of this draft against the problems they were creating. 

It's fine to say that LDAP was designed with attribute options in mind, and that there are no problems with language tags, but I suspect it's more the case not stumbling on one's first steps. It would have been just as easy for an early LDAP author to specify an unsolicited benign response control that carried diagnostics information.
 
It could be that I'm trying to smack the problem with too big of a hammer (though an extension mechanism with no means of solicitation still seems ugly and dangerous). If we're unwilling to recommend against the use of unsolicited options, then I don't believe there is sufficient guidance regarding how one is to define new ones in order to avoid interoperability problems. 
Maybe your suggestions below, along with something a little more obvious in [Models] (though what that would be escapes me right now) will have to be good enough. I note that the authors of the range option could technically follow the current instructions in [Models] and still claim (with a not very straight face) that they did due diligence in creating the new option. It might look something like:
- This option is not a tagging option.
- This option can be associated with any multi-valued attribute.
- This option does not affect the transfer of attribute values.
- This option is to be treated as if it were the attribute description without this option * it is not to be treated as a subtype of the attribute description minus this option.
 
Lame? Yes.  But it does follow the instructions in [Models].
 
Jim

>>> "Kurt D. Zeilenga" <[email-masked]> 4/19/04 6:54:14 PM >>>
At 04:01 PM 4/19/2004, Jim Sermersheim wrote:
>Regarding #1, My opinion is that servers should not return extension information like this unless it is solicited.

You and I like differ in what we consider 'like this'.

The problem with the range extension is that it using options
not to create subtypes. It using options (which don't conform
to the protocol specification) to alter the basic semantics
of the search operation in a fashion which is incompatible
with peers which don't support the extension. Assuming the
client doesn't puke, the client certainly won't treat these
values as being of the right type (it might consider them
to be of some oddly named subtype, but that's not quite the
same thing).

>Unfortunately, we have already set a precedent with language tags and the binary option which both return unsolicited attribute options.

But there are some significant differences. First, language
tag options and the binary option conform to RFC 2251, 4.1.5.
These range options don't.

Second, with language tags, the presumption is that values of
the subtypes (the tagged descriptions) are being returned in
addition to those of the untagged description, just like
values of 'cn' when asking for 'name'. With these range options,
the values of the ranged descriptions are sent INSTEAD of
unranged descriptions. This means, even clients which don't
choke on the protocol violation, still won't recognize the
values as being of the same description.

Third, with the binary option (as used in practice), only applies
to attribute types (e.g., userCertificate) which do not have
native string encodings and isn't a problem (the information
presented with ;binary is not available otherwise). While it
could technically be used with other attribute types (e.g., cn),
it is recognized that returning such without solicitation
would be problematic (because the client might treat it as
a subtype) (and was intended only be used where the server
couldn't make the information available without ;binary, e.g.,
server didn't support the native string encoding (or syntax
didn't have one)). I think Steven's ;binary revision proposal
adequately addresses this problem (e.g., don't return cn;binary
unless cn;binary was requested).

>The danger is exaclty the problem you've run into clients are given data that they don't understand and can't properly work with.

I disagree that this is the case with language tags and ;binary.
I believe LDAP was designed specifically with language tags
and ;binary in mind.

With range option is quite different. Not only are the options
invalid, the client not supporting this extension will interpret
the response as not containing values of the requested type
(but might interpret the response as containing values of a
subtype)... or interpret the response as a protocol error
(and disconnect).

This is why I say the range extension is truly non-optional,
which is far worse than being not truly optional. I think
language tags and ;binary, when properly implemented (or
as generally implemented in practice), are truly optional.

>Since we're talking about this on the LDAPBis list, I'll suggest that we add a recommendation to this effect:
> 
>Servers SHOULD NOT return attributes with options unless solicited to do so.

Since '*' does solicit all user attributes and 'cn;lang-x-foo'
is a user attribute, it should be returned. Since a requesting
'name' is a request for the attribute 'name' AND its subtypes,
'name;lang-x-foo' (as well as 'cn') is solicited. So, your
suggestion seems meaningless as worded.

But, if I take your suggestion as I think you intended it,
I would argue strongly against it.

Attribute options mechanism and subtypes created through this
mechanism IS a standard feature of LDAP. RFC 2596 defines
a set of option-based subtypes just as some schema document
might define a set of SUP-based subtypes. As long as these
subtypes are used consistently with core specification, the
are no significant interoperability issues.

Now, I note, the language tag specification needs to be more
clear that tagged values are provided IN ADDITION TO non-tagged
values. And ;binary specification needs to be address issues
previously raised on this list.

But, I think, the [Protocol] and [Models] are fine with
regards to attribute options.

However, [Roadmap] likely should say something in general
about extensions.
LDAP is an extensible protocol. Extensions are
expected to be truly optional.

There is little we can say beyond this to deter creation
of extensions which are not truly optional or extensions,
such as range options, which are truly non-optional.

Kurt

> 
>Jim
>
>>>> "Schleiff, Marty" < [email-masked] > 4/18/04 12:31:39 AM >>>
>Gentlemen,
>
>When using ldapsearch to lookup the membership in an Active Directory group, I got the following (abbreviated) response:
>
>CN=PC Backup Group 1,OU=Groups,OU=Accounts,DC=sw,DC=nos,DC=boeing,DC=com
>member;range=0-999=CN=Peterson\, Randy R,OU=End Users,OU=Accounts,DC=sw,DC=nos,DC=boeing,DC=com
>member;range=0-999=CN=Tatham\, William C,OU=End Users,OU=Accounts,DC=sw,DC=nos,DC=boeing,DC=com
>member;range=0-999=CN=Pham\, Steve,OU=End Users,OU=Accounts,DC=sw,DC=nos,DC=boeing,DC=com
>
>I'd never before encountered "range=0-999" and wondered what it meant. A little searching turned up the following URLs:
>< http://www.hut.fi/cc/docs/kerberos/draft-kashi-incremental-00.txt > http://www.hut.fi/cc/docs/kerberos/draft-kashi-incremental-00.txt and < http://www.hut.fi/cc/docs/kerberos/nss_ldap.html > http://www.hut.fi/cc/docs/kerberos/nss_ldap.html 
>
> From the second URL: "Active directory uses a method called "Incremental Retrieval of Multivalued Properties" which nss_ldap has no support for, at least not yet. Querying groups having more than 1000 members will return the first 1000 members, with the 'member' attribute changed to 'member;range=0-999'. This is not recognized by nss_ldap, so you will not see any members for that group. More information can be found in the (now expired) Internet Draft labeled Incremental Retrieval of Multivalued Attributes <draft-kashi-incremental-00.txt> ."
>
>I was also referred to < http://www.dfn-pca.de/bibliothek/standards/ietf/none/internet-drafts/draft-haripriya-partial-entry-00.txt > http://www.dfn-pca.de/bibliothek/standards/ietf/none/internet-drafts/draft-haripriya-partial-entry-00.txt ; however, I'm not sure that applies to my situation (it seems to describe how a client could request the server to return a partial set of values for a multi-valued attribute -- in my case I didn't request the range, Active Directory just gave it to me when I really wanted the complete set of values).
>I'd value the opinions of this list around the following questions:
>1) What do you think of this practice? Being a stranger on this list, I've withheld my editorial opinion of this practice; however, I'd like to hear your opinions.
>2) Even if I can get my ldap client apps to learn to deal with the ";range=0-999", I don't know how to teach them to obtain subsequent ranges. Also, as pointed out in section "5.3 Element Ordering", another client operating on the same entry as my client can add or remove values between my clients operations to retrieve multiple ranges so that my client's "requests may result in overlapping, duplicated, or skipped elements". 
>3) Does The Open Group assessment of "LDAP Ready" require a client to understand ranges?
>4) Does The Open Group assessment of "LDAP Certified" require a server to return ranges? Or if a server does return ranges (which probably would confuse many clients), can that server be "LDAP Certified".
>5) Will common tools like ldapsearch, PerLDAP, LDAP URLs on a browser location box, etc. ever be able to request ranges of multi-values attributes (or do they already have this capability and I'm just unaware)?
>Thx,
>
>< mailto:[email-masked] > [email-masked] ; CISSP

---

From: "Jim Sermersheim" <[email-masked]>
Date: Tue, 20 Apr 2004 07:29:18 -0600

Yes there are other problems with the range tags that I'm not even
talking about.

>>> "Ramsay, Ron" <[email-masked]> 4/20/04 1:44:35 AM >>>
Hi Jim,

You seem to be completely bypassing the issue of whether this *option*
can be parsed. With all the options I have seen, it is possible to find
the value because none of them allow '=' in the tag. Having an '=' in
the range tag means that you must be expecting it, for you cannot find
the value otherwise.

Compare

cn;lang-en=myname

with

cn;range=001=myname

Normal options are terminated by ';' or '=' and can therefore be
ignored. This option must actually be recognised to be ignored.

So, apart from your metaphysical argument below, there is a real,
practical issue.

Ron

-----Original Message-----
From: [email-masked] 
[mailto:[email-masked]]On Behalf Of Jim Sermersheim
Sent: Tuesday, 20 April 2004 16:55
To: [email-masked] 
Cc: [email-masked] ; [email-masked] ;
[email-masked] ; [email-masked] 
Subject: Re: Active Directory question

I'm not sure exactly what the meanings of 'truly optional' 'truly
non-optional' are, or whether there is the notion of 'optional * but not
truly'. I agree with adding something like you suggest as long as the
terminology is defined. 

But to take the issue a bit further, regarding your first point; The
fact that language tags and the binary options conform to RFC 2251
doesn't make me feel that they are any less flawed. The attribute
options feature (in my mind) is clearly an extension mechanism. Without
means for discovery or solicitation, I believe this extension mechanism
is fundamentally flawed.

Second, I suspect that the introduction of any new option (especially
those that are not 'tagging' options) is going to cause a number of
applications to encounter problems unless there is a client solicitation
mechanism for it. Currently, a client can only recognize those attribute
options that are known to it and fail over to treating attributes with
unknown options as either unrecognized attributes, or (worse in some
cases, better in others) treat them like tagging options. Who knows what
other types of options will be concocted in the future? As long as it's
possible to invent an option which (like the range and binary options)
can cause mandatory attributes to be treated as unrecognized by a
client, I see problems.

Third, when you say things like: "While <the binary option> could
technically be used with other attribute types (e.g., cn), it is
recognized that returning such without solicitation would be
problematic" I disagree. Obviously people who don't recognize this class
of problems are writing extension documents and implementing them.
Furthermore, I don't remember any discussion where anyone warned the
authors of this draft against the problems they were creating. 

It's fine to say that LDAP was designed with attribute options in mind,
and that there are no problems with language tags, but I suspect it's
more the case not stumbling on one's first steps. It would have been
just as easy for an early LDAP author to specify an unsolicited benign
response control that carried diagnostics information.

It could be that I'm trying to smack the problem with too big of a
hammer (though an extension mechanism with no means of solicitation
still seems ugly and dangerous). If we're unwilling to recommend against
the use of unsolicited options, then I don't believe there is sufficient
guidance regarding how one is to define new ones in order to avoid
interoperability problems. 
Maybe your suggestions below, along with something a little more
obvious in [Models] (though what that would be escapes me right now)
will have to be good enough. I note that the authors of the range option
could technically follow the current instructions in [Models] and still
claim (with a not very straight face) that they did due diligence in
creating the new option. It might look something like:
- This option is not a tagging option.
- This option can be associated with any multi-valued attribute.
- This option does not affect the transfer of attribute values.
- This option is to be treated as if it were the attribute description
without this option * it is not to be treated as a subtype of the
attribute description minus this option.

Jim

>>> "Kurt D. Zeilenga" < [email-masked] > 4/19/04 6:54:14 PM >>>
At 04:01 PM 4/19/2004, Jim Sermersheim wrote:
>Regarding #1, My opinion is that servers should not return extension
information like this unless it is solicited.

>Unfortunately, we have already set a precedent with language tags and
the binary option which both return unsolicited attribute options.

>The danger is exaclty the problem you've run into clients are given
data that they don't understand and can't properly work with.

Kurt

>When using ldapsearch to lookup the membership in an Active Directory
group, I got the following (abbreviated) response:
>CN=PC Backup Group
1,OU=Groups,OU=Accounts,DC=sw,DC=nos,DC=boeing,DC=com
>member;range=0-999=CN=Peterson\, Randy R,OU=End
Users,OU=Accounts,DC=sw,DC=nos,DC=boeing,DC=com
>member;range=0-999=CN=Tatham\, William C,OU=End
Users,OU=Accounts,DC=sw,DC=nos,DC=boeing,DC=com
>member;range=0-999=CN=Pham\, Steve,OU=End
Users,OU=Accounts,DC=sw,DC=nos,DC=boeing,DC=com
>
>I'd never before encountered "range=0-999" and wondered what it meant.
A little searching turned up the following URLs:
>< http://www.hut.fi/cc/docs/kerberos/draft-kashi-incremental-00.txt >
http://www.hut.fi/cc/docs/kerberos/draft-kashi-incremental-00.txt and
< http://www.hut.fi/cc/docs/kerberos/nss_ldap.html >
http://www.hut.fi/cc/docs/kerberos/nss_ldap.html 
>
> From the second URL: "Active directory uses a method called
"Incremental Retrieval of Multivalued Properties" which nss_ldap has no
support for, at least not yet. Querying groups having more than 1000
members will return the first 1000 members, with the 'member' attribute
changed to 'member;range=0-999'. This is not recognized by nss_ldap, so
you will not see any members for that group. More information can be
found in the (now expired) Internet Draft labeled Incremental Retrieval
of Multivalued Attributes <draft-kashi-incremental-00.txt> ."
>I was also referred to <
http://www.dfn-pca.de/bibliothek/standards/ietf/none/internet-drafts/draft-haripriya-partial-entry-00.txt
http://www.dfn-pca.de/bibliothek/standards/ietf/none/internet-drafts/draft-haripriya-partial-entry-00.txt
; however, I'm not sure that applies to my situation (it seems to
describe how a client could request the server to return a partial set
of values for a multi-valued attribute -- in my case I didn't request
the range, Active Directory just gave it to me when I really wanted the
complete set of values).
>1) What do you think of this practice? Being a stranger on this list,
I've withheld my editorial opinion of this practice; however, I'd like
to hear your opinions.
>2) Even if I can get my ldap client apps to learn to deal with the
";range=0-999", I don't know how to teach them to obtain subsequent
ranges. Also, as pointed out in section "5.3 Element Ordering", another
client operating on the same entry as my client can add or remove values
between my clients operations to retrieve multiple ranges so that my
client's "requests may result in overlapping, duplicated, or skipped
elements". 
>4) Does The Open Group assessment of "LDAP Certified" require a server
to return ranges? Or if a server does return ranges (which probably
would confuse many clients), can that server be "LDAP Certified".
>5) Will common tools like ldapsearch, PerLDAP, LDAP URLs on a browser
location box, etc. ever be able to request ranges of multi-values
attributes (or do they already have this capability and I'm just
unaware)?