Subject: StartTLS responseName became optional

From: Hallvard B Furuseth <[email-masked]>
Date: Fri, 22 Jul 2005 17:34:18 +0200

I've just learned that the StartTLS responseName changed from
mandatory to optional in Protocol-26, but I cannot find anything about
it in the LDAPbis archive.

I found an explanation from Kurt elsewhere:

> The specification here is considered to be in error here.
>
> An client which implements RFC 2830 must be capable of handling
> responses from servers which do not implement RFC 2830.  And such
> a server cannot be expected to know what responseName to specify,
> this MUST is flawed.
>
> (...) responseName (...) is really intended only for use with
> unsolicited notifications, not extended operation responses (as
> they are simply not needed).

I disagree.  The current spec is RFC 2830, which was written as an
LDAPv3 extension.  As such, it describes how LDAP implementations that
support this extension shall implement this extension.  The wording
may be clumsy, but the real bug was introduced when the wording was
"imported" into core LDAP and thus changed meaning.  (If it did change
meaning, I can read it another way:-)

RFC2251 section 4.12 requires protocolError in response to an
unrecognized extended request.  So I think a client implementing RFC
2830 may treat a StartTLS response with no responseName and result
code other than protocolError as a protocol error.

Thus, for backwards compatibility I think the spec should be that
servers still MUST send the responseName with any other result code,
though it wouldn't hurt to also say that clients SHOULD or MUST accept
responses without the responseName.

While it seems pointless of RFC 2830 to require the responseName,
cleaning that up isn't worth breaking clients that also require the
responseName, if any.

-- 
Hallvard
For sale: Parachute. Never opened, used once, slightly stained.

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 22 Jul 2005 19:42:29 +0200

>it in the LDAPbis archive.

I believe this issue was previously discussed on the list and,
IIRC, in face-to-face meetings.  I am on the road at present
so cannot provide URLs.

>responses without the responseName.

I disagree.  Adding a MUST here will lead to clients being
developed which are not accepting of the current practice
of some servers not to provide a responseName in this
case.  And if such clients did exist already, they likely
have other problems.

>While it seems pointless of RFC 2830 to require the responseName,

Not only is it pointless, it is very bad for clients to
require a responseName here.  Obviously there is the unrecongized
case where the server cannot provide a responseName.  But there
are likely other cases where a server cannot generate a
responseName for some recognized control, for instance when
the server fronts for a distributed system.

>cleaning that up isn't worth breaking clients that also require the

This is not merely protocol clean.  This is fixing existing
broken clients.  The MUST lead to development of clients
which absolutely required the responseName, even from
servers which don't implement the extension.  Client developers
MUST NOT expect a responseName (not only in response to
protocolError, but in general).

Another issue is that StartTLS technical specification,
because of its inclusion in the core TS, is used as a
template for future extended operation specifications.
Having it do something which future specifications should
not do is a bad thing.

Kurt

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 22 Jul 2005 21:05:03 +0200

If it was not clear in my previous response, there are other
cases where it is appropriate for a server not supporting an
extended operation to return a result code other than
protocolError.  For instance, it would be reasonable for
a server which is busy to return busy, or for a server which
believes another server is better able to response to
return a referral, or a server which has an internal
error to return other, or operationsError when the
request cannot be processed due to issues with other
operations.

A client which assumes that responseName is
absent only when the resultCode is protocolError is
broken.  We need to ensure the specification language
is clear to the fact that clients not to expect a
responseName in an extended operation response.

---

From: Hallvard B Furuseth <[email-masked]>
Date: Mon, 25 Jul 2005 00:26:45 +0200

Kurt D. Zeilenga writes:
>> I've just learned that the StartTLS responseName changed
>> from mandatory to optional in Protocol-26, but I cannot
>> find anything about it in the LDAPbis archive.
>
> I believe this issue was previously discussed on the list
> and, IIRC, in face-to-face meetings.  I am on the road at
> present so cannot provide URLs.

The closest I can find is a thread 'starttls result codes' after
protocol-26 was released, but that did not mention responseName.

Do anyone else remember?  The dates of the drafts seems to put it
near the Control Combination wars, so I'm not surprised if I lost
it.  Or even if I participated and then forgot it:-)

I'll wait some days before responding after this (if I remember
to wait:-), in case someone finds the old thread first.

BTW, this change is not listed in Appendix C, "Changes".

>> (...) Thus, for backwards compatibility I think the spec should
>> be that servers still MUST send the responseName with any other
>> result code, though it wouldn't hurt to also say that clients
>> SHOULD or MUST accept responses without the responseName.
>
> developed which are not accepting of the current practice of
> some servers not to provide a responseName in this case.

Eh?  That's why I suggested to add that clients SHOULD/MUST accept
responses without responseName.  And we wouldn't be _adding_ the
MUST for servers, we'd be restoring it after less than a year's
absence from a draft.  People have had five years to produce such
RFC 2830 clients.

> And if such clients did exist already, they likely have other
> problems.

Well, if common practice is to break RFC 2830 in this respect, I
suppose such clients might work poorly with servers that break the
standard in other common ways too.  Or do you mean something else?

> case where the server cannot provide a responseName.

In that case RFC 2251 says ExtendedResponse.responseName
should be absent, so clients that require responseName even
with protocolError are broken.  So far we agree.  But -

> But there are likely other cases where a server cannot
> generate a responseName for some recognized control, for
> instance when the server fronts for a distributed system.

I don't understand, even after your next message.  This is just a
matter of composing a correct response.  How does a control or a
distributed system make it difficult to insert the responseName?
Drop this if I'm responding to banalities instead of what you mean
to say, but anyway:

Do you mean a control could tell the server to change the StartTLS
responseName field to something else than the StartTLS spec says,
and a distributed system would make that difficult?  If so I don't
see how it makes a difference to this problem what the StartTLS
spec says, be it absent, present or rot13-encoded responseName.
Nor why it is difficult.  The possibilities listed in 4.1.11 are
- make use of the control when performing the operation.  So
  the server will know what to put in the responseName field.
- not apply the control (because it's recognized by a distributed
  system component which is down?).  So the server will still know
  what to put in responseName: whatever the StartTLS spec says.
  And in this case either return unavailableCriticalExtension or
  ignore the control, depending on its criticality.

[Pasting from your next message]

> protocolError.

Not according to [Protocol] 4.12:

   If the Extended operation associated with the requestName is not
   supported by the server, the server MUST NOT provide a responseName
   nor a responseValue and MUST return with resultCode set to
   protocolError.

I guess it depends on what you mean with "appropriate", and if you
mean something else with "support" than the draft does.  There are
times when it's appropriate for software to violate standards, but
it's not appropriate according to the standard.  And nothing
prevents software which violates the Extended Operation spec from
violating the StartTLS spec too, no matter what [Protocol] says.

> For instance, it would be reasonable for a server which is busy to
> return busy, or for a server which believes another server is better
> able to response to return a referral, or a server which has an
> internal error to return other, or operationsError when the request
> cannot be processed due to issues with other operations.

Sure.  But I don't see why any of that makes it difficult to
insert the responseName.  The server can't e.g. respond to an
extended operation with referral if it doesn't even (bother to)
recognize the extended operation.  Possibly that's a problem, but
if so the place to fix it is section 4.12 (Extended Operation).
If it does recognize the operation, it knows enough to compose a
correct response.

Well, except for severe internal errors.  But if the server is too
confused it can always drop the connection.

> A client which assumes that responseName is absent only when the
> resultCode is protocolError is broken.

Not according to RFC 2830.

[Back to previous message]

>> cleaning that up isn't worth breaking clients that also the
>> require responseName, if any.
>
> broken clients.

Well, it's telling and encouraging broken clients to get fixed.
But as you see, I think it can also break correct clients.
I think it's better to encourage both clients and servers to
address the problem, so they will work together as soon as at
least one of them is fixed.

> The MUST lead to development of clients which absolutely
> required the responseName, even from servers which don't
> implement the extension.

I suggested a fix which covered that:  The StartTLS section
can say clients MUST accept an absent responseName.

> Having it do something which future specifications should

RFC 2830 is a separate RFC detailing one extended operation, so it
can be used as a template for other such documents.  The LDAPbis
StartTLS spec won't be that.

If an extended operation spec should normally say the responseName
should be absent, that can be mentioned in Section 4.12 (Extended
Operation) or in the separate document you sometimes suggest which
could describe how to write extensions.  And the StartTLS spec
could mention that servers should include the responseName "for
the sake of backward compatibility".

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Mon, 25 Jul 2005 10:10:52 +0200

>to wait:-), in case someone finds the old thread first.

Note that I, as chair, have not nixed this thread.  Hence,
please feel free to discuss it.  I do note that the document
has already been approved by the IESG, there will be some
resistance to changes.   However, let's discuss the issues
and if the WG reaches a timely conclusion that a change
is necessary (not just appropriate), I will take the change
up with our AD.

In the remainder of this thread is with chair hat off.

>BTW, this change is not listed in Appendix C, "Changes".

Point noted.

Let's step back a bit and discuss extended operations in
general.

It should be clear that upon detection of a
problem with a request that the server may immediately
return a error response.  In generating this error
response, it is not required to further examine the
request PDU.

The RFC 2251 statement:
   If the server does not recognize the request name, it MUST return   
   only the response fields from LDAPResult, containing the
   protocolError result code.  The response field is
   absent.

means that during the examination of the request name
field, if it finds the value to be unrecognized, its
response should be a protocol error and the responseName
and response[Value] fields should be absent.

However, RFC 2251 obviously missed the point that a server
can detect an error BEFORE or DURING examination of the
request name field.  It should be obvious that in these
cases the server should return a response with an
appropriate resultCode but no responseName/response[Value]
field.

Less obvious is that the server could have a problem,
such as some internal fault, AFTER examination of the
request name field which disallows it from sending the
responseName/responseValue field.

It should be noted that to the protocol, it is irrelevant
whether the problem occurred before, during, or after
examination as the order of server processing is not
dicated.
 
The RFC 2830 statement
   A Start TLS extended response MUST contain a responseName field which
   MUST be set to the same string as that in the responseName field
   present in the Start TLS extended request.

is seriously flawed.  Even if we take this as applying to
servers implementations of RFC 2830, it should be obvious
that a server implementation of RFC 2830 could run into
situations where it needs to fall back to RFC 2251 allowed
behavior, such as returning other without responseName
on an internal error detected before or during examination
of the request name field.

This statement is also flawed in that it can be taken as a
client requirement to verify a responseName field is present
and contains a particular value.  I do not believe that was
ever intended.  And the few clients which did attempt to
verify this value did so in an incorrect manner.  They
required the field to be present when RFC 2251 allowed it
to be absent, hence causing interoperability problems.

These interoperability problems cannot be solved by requiring
the responseName in supporting servers, aa the problem spans
servers which do not support this extended operation.  We
need to make it clear that the responseName field in
in extended responses is optional.  If we mandate it any
case, clients will be developed which verify it unnecessarily,
but verify it improperly (as our operational experience
shows).

Secondly, we need to make LDAP easier to implement.
Mandating a responseName makes it difficult to implement
generic error handling of extended operations (as one
must special case the response to the request).

>absence from a draft.  People have had five years to produce such

Sorry, I was disagreeing with the "servers still MUST send
responseName" as clearly there are cases where they cannot
reasonably be expected to do so.  As far as the "clients
SHOUDL or MUST accept", yes, we should make it clear that
clients should expect the unexpected.

>standard in other common ways too.  Or do you mean something else?

My point here is that it likely that clients verifying
the requestName would not fully consider all the cases
where it reasonable and acceptable for a server to return
an extended response without a responseName.

>with protocolError are broken.  So far we agree.  But -

Clients which require responseName with various other
result codes are also broken.

>> instance when the server fronts for a distributed system.

s/control/extended operation/

>distributed system make it difficult to insert the responseName?

Because of lack of complete knowledge of what is or isn't
supported by others in the distributed system.

>   nor a responseValue and MUST return with resultCode set to

As with all other texts prescribing some error handling, this
text only applies if this is the error that's being handled.
If some other is being handled, such as an internal error,
this text simply doesn't apply.

Note that [protocol] else says that servers are to return an
result code that indicates the nature of the problem they
detected.  They should return busy when busy, other when
they have an internal error, etc.., irregardless of whether
they recognize the requestName.   If they don't recongnize
the requestName, then clearly they cannot provide a responseName.
And if they have yet to examine the requestName, they are
not obligated to do so merely to provide a responseName.
This should be obvious because if there was some obligation
to do so.  If not obvious to you, consider a condition that
prevent the server from determining the responseName to
return, such as an internal problem.  This condition should
not preclude the server from responding.  Hence, it must
be allowed to respond without a responseName in cases
where it is unable to determine what responseName to
provide.

>violating the StartTLS spec too, no matter what [Protocol] says.

You are, IMO, misinterpreting the text.  That is, the
specification allows a server not supporting an extended
operation to return a result code other than protocol error.

The text you quote means that when a server does examine
the requestName and finds a value it does not recognize,
it should respond with a protocolError with no responseName
and no responseValue.

This text (or any text detailing a particular error case)
should not be taken to mean that if a server were to
detect some other error that it is not free to return a
response indicative of the nature of that error.

>insert the responseName.

Maybe the internal error is that the internal data structure
in the server holding information about the extended operation
indicates that a responseName should be provided in responses
but does not include the response name to be provided.  It seems
quite difficult to provide the responseName in this case.

>The server can't e.g. respond to an
>recognize the extended operation.

The server most certainly can respond with a referral to
any extended operation, including those it does not recongize
the requestName of.

>Possibly that's a problem, but

It would be odd a server to drop the connection when it is
willing and able to accept further requests.

>Not according to RFC 2830.

A client cannot assume the server implements RFC 2830.
RFC 2830 doesn't trump the general LDAP error handling
semantics.

Upon detection of an error, the server returns an error
response which indicates the nature of the error.  It
should be obvious that if it has not yet examined the
requestName field, or the problem occurred during that
examination, that it won't be able to provide a responseName.

>But as you see, I think it can also break correct clients.

While certainly no check is needed, a client could check
responseName on success.  But I know of no client which only
checks on success.  A client which checks on any non-success
result code is broken.

>address the problem,

The problem is that StartTLS should never have specified
a responseName.  responseNames are extraneous in extended
operations.  Servers should not be obligated to provide them
(as clearly there are cases which they cannot).   Clients
certainly should not check that they exist (because of the
above).

>so they will work together as soon as at

We should do is make it clear that responseName in
extended operation responses are optional.

>can say clients MUST accept an absent responseName.

I suggest that this be generalized and stated for
all extended operations.  The problem is not just with
StartTLS.   However, I wouldn't have a problem with
reenforcing the general statement in StartTLS given
our experiences with StartTLS.

>StartTLS spec won't be that.

The form of the StartTLS specification may not be as usable as a
template for other extension specifications, but the
specified syntax and semantics for StartTLS are still likely
be used as a basis for future extended operations.

>could describe how to write extensions.

IIRC, draft-zeilenga-ldap-ext covers this.  BTW, this I-D is
ripe for discussion (on [email-masked] please).

>And the StartTLS spec

I rather not include such.