Subject: Re: draft-ietf-ldapbis-protocol - controls

From: Howard Chu <[email-masked]>
Date: Tue, 29 Mar 2005 10:45:19 -0800

Jim Sermersheim wrote:
>  >>> Howard Chu [email-masked]> 3/27/05 1:09 PM >> 
> <mailto:[email-masked]> 3/27/05 1:09 PM >>>
> <snip>
>  >The first problem is that there is no way to indicate that a control is
>  >only supported within specific contexts, as opposed to server-wide.
>  >Since there is only the single rootDSE to use for advertising features,
>  >it would be reasonable for clients to assume those features are
>  >server-wide but in this case they are not.
> The same problem applies to servers which chain operations. To me, a 
> server with multiple backends could be viewed as a server chaining to 
> multiple other servers. Or a server front ending a lot of other 
> (backend) servers.
>  
> I don't think the advertisement of supportedControls, 
> supportedExtensions or supportedFeatures can ever be completely relied 
> on to represent the entire set of services when the DSA fronts multiple 
> backends or chains to other DSAs.

OK, that statement is clear, but it is not helpful for client implementors.

>  >The interesting points I draw from this are:
>  >3) contrapositive to (1), controls with Criticality FALSE may NOT
>  >cause an otherwise valid request to fail with an error response
>  
> Sure, if the server cannot service the control for whatever reason, it 
> will fail with an error response (see example below).
> 
>  >4) the Abandon request is also defined to have no response, and thus
>  >should have the same exception.
>  
> The outcome of Abandon is and always has been indeterminate. The outcome 
> of Unbind was intended to be deterministic. While I agree that adding a 
> critical control to an Abandon request seems to have very little use, it 
> could be argued that it gives the server an extra hint at how (or 
> whether) to apply the abandon request.
> 
>  >If (3) is true then the problem that prompted me to write this message
>  >is solved. Regardless, (4) probably ought to be clarified.
>  
> 3 is not true because the control itself may cause a number of problems 
> which can cause failure as applied to the operation. Note that some 
> implementations restrict access to some controls like the Virtual List 
> View control. In the case where the server recognizes the control, it's 
> appropriate for the operation, but the user doesn't have sufficient 
> rights to apply it, the operation fails.

This really doesn't make any sense. Again, in this same paragraph 
talking about criticality:
    A value of TRUE indicates that it is unacceptable to perform
    the operation without applying the semantics of the control.

The logical inference to draw from this is that a value of FALSE 
indicates that it *is* acceptable to perform the operation without 
applying the control.

Think about what an actual client is going to do with a non-critical 
control - if the request with the control fails, the client is just 
going to have to re-issue the request without the control anyway. The 
most logical behavior for the server is to ignore any non-critical 
control that cannot be applied to the operation regardless of reason. 
The behavior you're describing here just makes the use of controls more 
inefficient, requiring more transaction roundtrips to get something that 
the client writer thought "would be nice, but isn't essential."

Clearly if the control is marked Critical, all of these checks and error 
responses are essential, and the client explicitly wants to be told if 
the request cannot be processed exactly as requested. But when the 
control is not marked Critical, the client doesn't really care whether 
the control is applied so long as the request is processed.

-- 
   -- Howard Chu
   Chief Architect, Symas Corp.       Director, Highland Sun
   http://www.symas.com               http://highlandsun.com/hyc
   Symas: Premier OpenSource Development and Support

---

From: "Jim Sermersheim" <[email-masked]>
Date: Tue, 29 Mar 2005 17:08:37 -0700

Our implementations do this:
 
Is the control recognized and appropriate (as specified) for the operation?
   Yes, process the operation
   No, Is it marked critical?
      Yes, fail with unavailable critical extension
      No, Ignore the control and process the operation
 
This has been discussed at length in the past, and consensus was that this was the original intent and that this is what should be conveyed in the new document.
 
IIRC, there were interoperability side effects of allowing "appropriate" to mean anything the server wishes it to mean.
 
Jim

>>> "Kurt D. Zeilenga" <[email-masked]> 3/29/05 12:14:07 PM >>>
At 10:01 AM 3/29/2005, Jim Sermersheim wrote:
>Appropriate means that the control specification says it can be applied to that operation. I can add that clarification.

That seems what the WG has concluded, but I still for wonder
if that's best. It certainly seem counter to how many
implementor have interpreted the RFC 2251 language, as well
as quite counter-intuitive to me.

To test the waters, I implemented code in OpenLDAP 2.3beta
which behaves as the WG concluded. If a client asks for
paged results (regardless of critical), the server returns
unwillingToPerform if it cannot honor this control for
the particular search operation requested. However, this
is not the behavior client developers seem to expect. They
assume that if the client says paging is not critical
to them, that the server should simply ignore the control
if it not willing to honor the control for that operation.

It seems they expect this behavior, not only based on how
they RFC 2251, but how many current servers behave today.
I wonder how other servers which have different capabilities
in different contexts behave? I understand that many servers
support configuration and monitoring contexts, it is likely
that these contexts do not support all the same capabilities
of DIT contexts. So, are there cases in your server
(or other servers you are familiar with) ignores a
recognized non-critical control attached to an applicable
request? (applicable here meaning that the technical
specification indicated the control was appropriate to that
kind of request)

Personally, I think the logic that RFC 2251 was intending
was that a server should not apply non-critical controls to
portions of the operation. It must up front (before
initiating the operation) make a determination whether
to make use of the control or to ignore it. Once
it does choose to make use of the control in the operation,
there is no going back. This implies, for instance, that
if a client asks for non-critical FOO and the server
involves other servers in the evaluation of the operation,
that all the servers involved must either all provide
paging or none provide FOO. If the first server decides
to provide FOO, then it must indicate to the other servers
that FOO is now critical. If the first server decides
not to provide FOO, then it must indicate to the other
servers that FOO is not to be provided (e.g., remove the
control).

Kurt

---

From: "Jim Sermersheim" <[email-masked]>
Date: Tue, 29 Mar 2005 18:42:09 -0700

Also see 
http://www.openldap.org/lists/ietf-ldapbis/200408/msg00012.html
 
I believe that defining appropriate to mean that the control is attached to an appropriate operation as specified by the control specification will provide the most interoperability.
 
I do agree that as a feature goes, it's not that useful. However, I worry that changing its meaning to something that renders a more useful feature will not only cause interoperability problems, but also that it oversteps our charter.
 
I prefer to defer the invention of a more useful feature to a future specification, or to the extension guidelines (where we could also invent a way for the client to know whether a non-critical control was applied).

Kurt

---

From: "Jim Sermersheim" <[email-masked]>
Date: Tue, 29 Mar 2005 22:40:14 -0700

I believe this is what our chaining server does as well, though we have talked about changing chained controls to be critical in the past. 
 
I think your work around could be implemented and render (somewhat) expected effects:
 
Say a search operation has attached to it a non-critical control (which the DSA supports). When the local DSA chains, it forces it to be critical.
 
If the operation was chained before the control was used in processing the operation, then any resultCode returned from the chained-to DSA would be preserved.
 
If the operation was chained during operation evaluation after being partially supported by the local DSA, and if the chained-to DSA returned unavailableCriticalExtension, the operation could return a different error (which would indicate that the operation could not be completed as specified).

>>> Mark Ennis <[email-masked]> 3/29/05 6:11:00 PM >>>
As indicated in the message Jim referenced, our server will ignore the 
criticality flag of a control it recognises. However, this does not 
entirely address the issue Howard raised.

In a distributed environment, a server receiving a request may need to 
chain this request to a number of other servers, where the support of 
various controls in the different servers may be different. Where a 
given server receiving a chained request does not support a control, it 
will base its processing decision on the criticality of the control. So 
even if the server which received the request from the client can 
support a given control, the control will potentially be ignored for 
some part of the result set. This results in the effective behaviour of 
the server, which received the original request from the client, 
ignoring the non-critical control for some contexts, even though it does 
support the control. This is the way our server effectively behaves.

The only way around this, at present, would seem to be to force all the 
controls in a chained request to be critical, which would seem to be 
counter to the client's request that the processing of a non-critical 
control is optional.

Our policy is to handle controls, in general, in the way John McMeeking 
described, which is to ignore unsupported and inapropriate non-critical 
controls.

- Mark.

Jim Sermersheim wrote:
>>From another vendor:
> http://www.openldap.org/lists/ietf-ldapbis/200403/msg00055.html 
> 
> 
>>>>"Jim Sermersheim" < [email-masked] > 3/29/05 5:08:37 PM >>>
> 
> 
>>>>"Kurt D. Zeilenga" < [email-masked] > 3/29/05 12:14:07 PM >>>
>

---

From: Mark Ennis <[email-masked]>
Date: Wed, 30 Mar 2005 17:13:20 +1000

Jim Sermersheim wrote:
> I believe this is what our chaining server does as well, though we
> have talked about changing chained controls to be critical in the
> past.
> 
> Say a search operation has attached to it a non-critical control
> (which the DSA supports). When the local DSA chains, it forces it to
> be critical.
> 
> If the operation was chained before the control was used in
> processing the operation, then any resultCode returned from the
> chained-to DSA would be preserved.
Such as an unavailableCriticalExtension resultCode when the original 
request had no critical controls?

> 
> If the operation was chained during operation evaluation after being
> partially supported by the local DSA, and if the chained-to DSA
> returned unavailableCriticalExtension, the operation could return a
> different error (which would indicate that the operation could not be
> completed as specified).
Might work, although this strategy would lead to a difference in the 
results obtained depending on whether the server handles the chaining 
versus returning referrals and the client handling followup of the 
referrals.

Possibly the question of "what result would a client get if it handled 
requests to each individual context?" is worth exploring. I would be 
inclined to think a client would send the same request to each context 
with the control marked non-critical (or not bother to send it to the 
contexts which do not advertise support for the control). This leads me 
to be inclined to have a server replicate this behaviour rather than 
return errors for non-critical controls.

> As indicated in the message Jim referenced, our server will ignore
> the criticality flag of a control it recognises. However, this does
> not entirely address the issue Howard raised.
> 
> In a distributed environment, a server receiving a request may need
> to chain this request to a number of other servers, where the support
> of various controls in the different servers may be different. Where
> a given server receiving a chained request does not support a
> control, it will base its processing decision on the criticality of
> the control. So even if the server which received the request from
> the client can support a given control, the control will potentially
> be ignored for some part of the result set. This results in the
> effective behaviour of the server, which received the original
> request from the client, ignoring the non-critical control for some
> contexts, even though it does support the control. This is the way
> our server effectively behaves.
> 
> The only way around this, at present, would seem to be to force all
> the controls in a chained request to be critical, which would seem to
> be counter to the client's request that the processing of a
> non-critical control is optional.
> 
> Our policy is to handle controls, in general, in the way John
> McMeeking described, which is to ignore unsupported and inapropriate
> non-critical controls.
> 
> - Mark.
> 
> Jim Sermersheim wrote:
> 
>>> From another vendor:
>> Is the control recognized and appropriate (as specified) for the
>> operation? Yes, process the operation No, Is it marked critical? 
>> Yes, fail with unavailable critical extension No, Ignore the
>> control and process the operation
>> 
>> This has been discussed at length in the past, and consensus was
>> that this was the original intent and that this is what should be
>> conveyed in the new document.
>> 
>> IIRC, there were interoperability side effects of allowing
>> "appropriate" to mean anything the server wishes it to mean.
>> 
>> Jim
>> 
>> 
>>

---

From: "Jim Sermersheim" <[email-masked]>
Date: Thu, 31 Mar 2005 20:54:49 -0700

>>> Howard Chu <[email-masked]> 3/31/05 8:01:01 PM >>>
>Jim Sermersheim wrote:
>> Sorry, if the operation is chained before being applied locally, don't 
>> change criticality.
>> 
>> And yes, other options are to test the capabilities of the other 
>> agents/backends, and to perform the behavior locally when not supported 
>> (a server or frontend could do this with the sorting or vlv control, but 
>> would have to gather all entries first.
>
>The idea of testing the capabilities of the other servers/backends in 
>advance really doesn't scale well. If you have a large distributed set 
>of directory servers, where any server may be the frontend for yet 
>another arbitrarily large set of servers, you may spend more time 
>querying the capabilities (and getting unreliable answers) than actually 
>processing the client's original request. Nor is this sort of thing 
>properly cachable, since different servers with different capabilities 
>may enter and leave the network at arbitrary times.

Right, and you don't really know in advance which servers will even be chained to.

>Following this line of attack is futile. It imposes additional overhead 
>on the servers to collect a type of information in which the client has 
>absolutely no interest.

It is likely futile in a chaining environment, but may work where a server has multiple backends * I don't know. Note also that another option mentioned is to service the control locally when the remote DSA can't or won't service it. This could only be done for certain controls.

>There is nothing in RFC2251 that suggests the current LDAPbis 
>interpretation is either intended or correct. Call me crazy, but I'd say 
>a behavior that is fundamentally useless doesn't belong in the spec.

When I read RFC2251, I have to disagree. 

"If the server recognizes the control type and it is appropriate for
the operation, the server will make use of the control when
performing the operation."
 
I agree that the word "appropriate" is subject to interpretation. I interpret it as meaning that the control specification states that it's usable with the operation. Others (I think) extend this to also including other server policies (such as access controls). Regardless of how far people stretch the definition of appropriate, I think everyone agrees that (due to multiple backends and chaining), server advertisement of support of a control cannot always be relied on to provide an authoritative answer as to the servicability of the control.
 
"If the control is not appropriate for the operation and criticality
field is TRUE, the server MUST NOT perform the operation, and MUST
instead return the resultCode unsupportedCriticalExtension."
 
If "appropriate" means that the server found *any* reason in which the control causes the operation to fail, this says it should return unsupportedCriticalExtension (as opposed to something like insufficientAccessRights) for critical controls.
 
In your view, what would be the proper definition of "appropriate"? Or if that's too limiting, what would you like the criticality to mean?
 
Jim

---

From: "Jim Sermersheim" <[email-masked]>
Date: Thu, 31 Mar 2005 21:18:39 -0700

>>> Mark Ennis <[email-masked]> 3/31/05 8:21:43 PM >>>
>While I agree that the idea of testing the capabilities of the servers 
>is not practical, I do think the idea of defining procedures for 
>distributed operations which force controls to be critical and then 
>process the results of distributed requests to form an appropriate 
>response to the clients original request is potentially viable. This 
>assumes that a control specification is not permitted to mandate that a 
>control MUST be non-critical.
>
>That said, I do agree that non-critical controls should be ignored 
>rather then result in an error code for the operation. And this would 
>simplify procedures for distributed operations for LDAP.

If it is not immediately known whether a control would cause an error, then at what point can a non-critical control be ignored? Prior to name resolution? Prior to operation evaluation? During operation evaluation? Before sending any responses to the client? After sending responses to the client?
 
Similarly, when does one know whether to return an error versus discarding an transaction which has begun and re-starting the operation sans the control?
 
Also, some controls define error codes to be returned for certain conditions (see VLV for example). When these conditions are met when the control is non-critical, should it also be ignored and the operation be re-applied without the control?
 
There are cases where allowing non-critical controls to be ignored doesn't simplify the distributed operation problem. When a control is paired with a response control which indicates whether or not the control was successfully applied, then the chaining server is still faced with having to deal with different behaviors of chained-to servers. Also, any non-critical control on search operations may succeed on an initial server, and be ignored when chained to another server. Meaning the control was partially supported. I think most people would agree that this is undesirable * a control should either be applied or ignored unless that control specification allows it to be partially applied. 
 
Also consider the tree delete control. A client may send this as non-critical, knowing (well, expecting) that if it's recognized and appropriate, it will be applied, and the subtree will be deleted. If it can partially fail (due to being chained or sent to multiple backends), the subtree may be partially deleted and the tree left with a hole in it.
 
Jim

---

From: "Jim Sermersheim" <[email-masked]>
Date: Thu, 31 Mar 2005 21:49:38 -0700

I believe the broad interpretation of "appropriate" leads to far more interoperability problems than the narrow view. 
 
I'm glad to see that you don't believe a control can be partially applied.
 
The fact is, every implementation is going to behave quite differently when it is allowed to either fail-over to ignoring a control or return an error which was caused by the application of that control.
 
It will be extremely difficult to evaluate all error paths in code to determine whether the error could be allieviated by ignoring any non-critical controls that might be present. If one cooses not to do that, one could instead, upon encountering any error condition, ignore one non-critical control. If the operation still fails, one could ignore another non-critical error (and probably should un-ignore that first one). One could keep doing this until one has ignored enough non-critical controls (rolling back the transaction each time), until one determines that the operation is just going to fail no matter what, or until the the operation succeeds (one hopes one hasn't accidentally ignored a few non-critical controls that really wouldn't have caused the error)
 
If the implementation instead, just performs some arbitrary pre-checks to make sure it's fairly certain that the control won't cause the operation to fail, it will never perform the same as another implementation.
 
Maybe I'm over-complicating or over-simplifying or over-thinking this?
 
I rather doubt anyone is confused as to whether a control may be partially applied * that's just silly, so I don't think we need to add language to that effect.
 
I think the confusion has to do with what "appropriate" means. If the WG wants it to be more precicely defined, then let's come up with some wording. As it is, it just says appropriate. My only fear is that it will ultimately lead to worse interoperability.
 
Jim
 
>>> "Kurt D. Zeilenga" <[email-masked]> 3/31/05 9:09:46 PM >>>
I find this comment from Steve Kille interesting:
Some extensions can safely be ignored without loss of service.
Others will simply not work if they are not understood and
need to be rejected. The criticality mechanism allows this
to be communicated. This is particularly important in a
store and forward messaging environment, or where
directory requests are chained.
< http://www.openldap.org/lists/ietf-ldapext/199802/msg00088.html >

I think the key point I and Howard are making is that by not
ignoring a recognized and applicable (per the specification)
non-critical control one cannot service, one causes an
unnecessary and inappropriate loss of service.

The language in RFC 2251 was:
If the server recognizes the control type and it is appropriate for
the operation, the server will make use of the control when 
performing the operation.

I think it natural for implementors interpret "appropriate
for the operation" in a broad manner, and I think it clear that
many client and server implementors have interpreted "appropriate
for the operation" in a broad manner. Narrow interpretations,
such as viewing the language as a mandatory requirement
for servers to "make use" of applicable controls they recognize
leads to idiotic behavior.

For instance, I could code an LDAP server to "recognize"
every known request control but not support any of them.
Under the narrow interpretation taken by the WG, my server would
be obligated to turn unwillingToPerform (or some other
appropriate result code). I do not believe this is what was
intended by designers of LDAP. But regardless of their intent,
the interpretation taken by the WG will lead to significant
interoperability problems and hinder development/deployment
of distributed directory services.

What I think needs to be clarified is the above text is that
the server is to perform the operation, the whole operation,
as extended by the control. It cannot perform only part of
the operation as extended by the control. This, of course,
has some implications on distributed systems, but I argue
that the server ability to ignore applicable non-critical
controls is key to making such systems work in face of
extensions... which is what I think Steve was hinting at
in is 1998 ASID/LDAPEXT post.

-- Kurt

At 07:01 PM 3/31/2005, Howard Chu wrote:
>Jim Sermersheim wrote:
>>Sorry, if the operation is chained before being applied locally, don't change criticality.
>> 
>>And yes, other options are to test the capabilities of the other agents/backends, and to perform the behavior locally when not supported (a server or frontend could do this with the sorting or vlv control, but would have to gather all entries first.
>
>The idea of testing the capabilities of the other servers/backends in advance really doesn't scale well. If you have a large distributed set of directory servers, where any server may be the frontend for yet another arbitrarily large set of servers, you may spend more time querying the capabilities (and getting unreliable answers) than actually processing the client's original request. Nor is this sort of thing properly cachable, since different servers with different capabilities may enter and leave the network at arbitrary times.
>
>Following this line of attack is futile. It imposes additional overhead on the servers to collect a type of information in which the client has absolutely no interest.
>
>There is nothing in RFC2251 that suggests the current LDAPbis interpretation is either intended or correct. Call me crazy, but I'd say a behavior that is fundamentally useless doesn't belong in the spec.
>
>> >>> Mark Ennis < [email-masked] > 3/30/05 12:13:20 AM >>>
>>Jim Sermersheim wrote:
>> >>>> Mark Ennis <_ [email-masked] < mailto:[email-masked] >_ > 3/29/05 6:11:00 PM >>>
>> >
>> >> _http://www.openldap.org/lists/ietf-ldapbis/200403/msg00055.html_
>> >>
>> >>
>> >>
>> >>>>> "Jim Sermersheim" < [email-masked] < mailto:[email-masked] >_ > 3/29/05 5:08:37 PM >>>
>> >>>>>
>> >>
>> >>