Subject: Re: i24: Requiring Allow in 405 responses

From: Mark Nottingham <[email-masked]>
Date: Mon, 3 Mar 2008 13:01:48 +1100

Exactly. We're not here to re-design Allow or come up with a better  
mechanism; just to clarify what it means today.

To reiterate, my proposal:

> "The actual set of allowed methods is defined by the origin server  
> at the time of each request."
>
> to
>
> "The actual set of allowed methods is defined by the origin server  
> at the time of each request, and may not necessarily include all (or  
> any) methods that the server would actually allow in a request if  
> presented."

Some will argue that that's loosening the requirements of 2616; I  
don't think I buy that, because there isn't a RFC2119-level  
requirement about the contents of the header.

Thinking about the subsequent discussion, I'm ambivalent about adding  
a SHOULD-level requirement on the server side WRT completeness; I  
think the text above stands on its own.

Cheers,

On 01/03/2008, at 7:04 AM, Roy T. Fielding wrote:

>
> There is no point in arguing this.  Look at what has been  
> implemented so far
> and remove the cases that have not.
>
> ....Roy
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Robert Siemer <[email-masked]>
Date: Mon, 3 Mar 2008 05:28:07 +0100

> >any) methods that the server would actually allow in a request if  

That reads like:

"The actual set of allowed methods ... may not include all methods 
actually allowed ... ."

I prefer the original.

Greetings,
		Robert

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Mar 2008 09:19:17 +0100

Mark Nottingham wrote:
>> "The actual set of allowed methods is defined by the origin server at 
>> the time of each request."
>>
>> to
>>
>> "The actual set of allowed methods is defined by the origin server at 
>> the time of each request, and may not necessarily include all (or any) 
>> methods that the server would actually allow in a request if presented."
> 
> Some will argue that that's loosening the requirements of 2616; I don't 
> think I buy that, because there isn't a RFC2119-level requirement about 
> the contents of the header.

I have to disagree. The absence of a RFC2119-level requirement does not 
mean that something isn't a requirement.

RFC 2616 states:

"The Allow entity-header field lists the set of methods supported by the 
resource identified by the Request-URI."

That's what it means, no matter whether it says or doesn't say "SHOULD" 
or "MUST".

So yes, we are changing a requirement. And yes, we may have to do that 
to reflect reality, but in this case, we'll have to consider the effect 
of that change.

Therefore, if we allow the server not to include all methods, we also 
have to allow a client not to believe what it got, thus:

"However, the indications given by the Allow header field value SHOULD 
be followed."

should be dropped then.

> Thinking about the subsequent discussion, I'm ambivalent about adding a 
> SHOULD-level requirement on the server side WRT completeness; I think 
> the text above stands on its own.
> ...

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 3 Mar 2008 05:23:24 -0800

> should be dropped then.

That needs to be dropped anyway. It cannot be implemented unless the
client caches the Allow headers it receives, but there is no
(SHOULD-level) requirement for any client-side caching in HTTP. Besides,
there is no freshness information associated with the Allow header to
control the caching. (Maybe that is why it was defined as an entity
header?)

Plus, besides wasting the client's time and a little bandwidth, there is
really no harm in ignoring the requirement, and no benefit to following
it. It might be different if extensions to HTTP are being used, but
those extensions can seperately document their extended usage of the
Allow header as they see fit. 

- Brian

---

From: John Kemp <[email-masked]>
Date: Mon, 03 Mar 2008 10:03:31 -0500

Hello,

>> methods that the server would actually allow in a request if presented."

Why is this additional text necessary? RFC 2616 says that

"The purpose of this field is strictly to inform the recipient of valid 
methods associated with the resource."

There is no requirement, stated or even seemingly implicit, that a 
server include ALL valid methods in its response. Only the implied 
requirement that a server does not include "disallowed" methods in the 
response.

And let's remember, this Allow header came in a 405 response - the 
client tried something that didn't get it what it was looking for. The 
Allow header says "you tried something I don't allow on that resource - 
I claim that I support these methods on it."

The client can then try again. If it keeps trying unsupported methods 
(which it is free to do of course) it will keep falling afoul of the 
server's (hopefully not merely capricious!) requirements, so it's 
probably best if the client follows the server's advice ("the 
indications given by the Allow header field value SHOULD be followed")

> the contents of the header.

Right - I don't see it as a loosening - but in my reading of 2616 it 
does seem an unnecessary addition.

- johnk

>> There is no point in arguing this.  Look at what has been implemented 
>> so far

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Mar 2008 16:56:40 +0100

> requirement that a server does not include "disallowed" methods in the 

Sorry????

I would say that "*the* set of methods" is clear enough; it doesn't 
allow a subset.

BR, Julian

---

From: John Kemp <[email-masked]>
Date: Mon, 03 Mar 2008 11:19:25 -0500

Julian Reschke wrote:
>> "The purpose of this field is strictly to inform the recipient of 
>> valid methods associated with the resource."
>> 
>>  requirement that a server does not include "disallowed" methods in
>> the response. ...
> 
> Sorry????
> 
> "The Allow entity-header field lists the set of methods supported by
> the resource identified by the Request-URI."
> 

You're right - so we could change the "the" to an "a" and be done?

"The Allow entity-header field lists a set of methods supported by the 
resource identified by the Request-URI."

- johnk

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Mar 2008 17:24:20 +0100

> resource identified by the Request-URI."

I think that would relax it too much; it's almost making it useless.

My preference would be to use Mark's text, and also relax the "SHOULD 
believe" requirement for the client.

BR, Julian

---

From: "Mark Baker" <[email-masked]>
Date: Mon, 3 Mar 2008 11:28:00 -0500

> resource identified by the Request-URI."

Do we have to bother?  Even with "the", there's nothing preventing a
server from changing the methods it supports from one minute to the
next, and therefore no guarantee that "*the* set" is actually *the*
set at any given point in time.

Mark.

---

From: John Kemp <[email-masked]>
Date: Mon, 03 Mar 2008 11:38:05 -0500

Julian Reschke wrote:
> 
> John Kemp wrote:
>> ... You're right - so we could change the "the" to an "a" and be
>> done?
>> 
>> "The Allow entity-header field lists a set of methods supported by
>> the resource identified by the Request-URI." ...
> 

 From my interpretation, I would say that the spec. already allows me to
return a subset of methods (even with "the"), but I thought you were 
arguing that the spec. is clearly NOT currently allowing this behaviour 
because if the "the"?

>  believe" requirement for the client.

I don't believe that either of these things makes the spec. any less
ambiguous than it is today. But that's probably just fine:

> this Allow header came in a 405 response - the client tried something
> that didn't get it what it was looking for. The Allow header says
> "you tried something I don't allow on that resource - I claim that I
> support these methods on it."
> 

It's in the server's best interest to be as helpful as possible to the 
client under whatever circumstances it discovers (unauthenticated 
client, inability to determine the full list of supported methods etc. etc.)

It's in the client's best interest to accept the server's advice.

- johnk

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 4 Mar 2008 13:18:39 +1100

I think that's the core of the issue, and primary source of confusion.

There is no "the" set of methods; a single request can only contain  
one method, so it's impossible to determine what "the" set of methods  
is; HTTP does not provide atomicity for multiple requests.

Therefore, some frame of reference is needed. Most people seem to read  
"the set of methods" as implying "...that you would accept on this  
resource" for some undefined, fuzzy period of time, and for some  
undefined, fuzzy period of request characteristics. The set may  
contain PROPFIND now, but it may not in three minutes. The set may  
contain POST if you present the right credentials, but may not if you  
don't. And so on.

Add to this the original concern of the issue; that some  
implementations may not be able to determine the complete set at  
runtime.

In short, the phrase does not include "complete", and the set of  
methods that it will accept is not necessarily the inversion of the  
set of methods it won't accept; there is a gray area in between.

One way to fix this is to make the definition of the set less fuzzy,  
but as Roy has pointed out, that's taking liberties.

The other is to document the fuzziness and move on, which is what I'm  
trying to do. If others have better ways of doing that, or a third way  
forward, please say so.

Cheers,

On 04/03/2008, at 2:56 AM, Julian Reschke wrote:

>> the response.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 4 Mar 2008 13:17:36 +1100

I'd note that the issue was raised because some people read the  
phrasing as requiring all possible methods to be sent, and certainly  
some implementations try to do this; e.g.,

   http://bugs.php.net/bug.php?id=32561
   http://oldsite.webdav.org/mod_dav/bugs/index.php3?id=134

IMO we need to clarify this text so it's unambiguous. I know people  
would *like* to depend upon the values in Allow as a complete set, but  
that's not what implementations do, and it's actually very hard to do  
in any case.

FWIW, I like "the" -> "a"; it's more elegant than my proposal. I'm  
less convinced that it's necessary / good to loosen the SHOULD on  
clients; this sort of thing is what SHOULD is for.

Cheers,

On 04/03/2008, at 3:28 AM, Mark Baker wrote:

>> the

---

From: Julian Reschke <[email-masked]>
Date: Tue, 04 Mar 2008 12:18:55 +0100

Mark Nottingham wrote:
> 
> I'd note that the issue was raised because some people read the phrasing 
> as requiring all possible methods to be sent, and certainly some 
> implementations try to do this; e.g.,
> 
> that's not what implementations do, and it's actually very hard to do in 
> any case.

I think we have the following issues:

- spec inconsistency: 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.7> says 
"The Allow entity-header field lists the set of methods supported by the 
resource identified by the Request-URI.", while 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.10.4.6> says 
"The response MUST include an Allow header containing a list of valid 
methods for the requested resource."

- wishful thinking: many servers do not get this right, but 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.7>, 
however spec requires clients to trust it: "However, the indications 
given by the Allow header field value SHOULD be followed. "

- confusion between 405 Not Allowed, 501 Not Implemented, and other 
conditions under which a request could be rejected.

> FWIW, I like "the" -> "a"; it's more elegant than my proposal. I'm less 
> convinced that it's necessary / good to loosen the SHOULD on clients; 
> this sort of thing is what SHOULD is for.

That does not work. If we explicitly allow a subset, the header will be 
almost useless. If we also require clients to trust it, it becomes 
totally useless.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 04 Mar 2008 12:29:52 +0100

Mark Nottingham wrote:
> There is no "the" set of methods; a single request can only contain one 
> method, so it's impossible to determine what "the" set of methods is; 
> HTTP does not provide atomicity for multiple requests.

That sounds like splitting hairs to me.

The allowed set is the set of methods that was allowed at the time when 
the response was generated.

Yes, there is no cacheability information attached to that, but then 
there's no reason for the client to believe that the set of methods 
*does* change frequently.

Keep in mind that the set of allowed methods is the one that doesn't 
generate a 405 response, not that's going to succeed.

> undefined, fuzzy period of request characteristics. The set may contain 
> PROPFIND now, but it may not in three minutes. The set may contain POST 

Why would that ever be the case, except if you reconfigure your server 
not to support PROPFIND? How frequently do these kinds of config changes 
happen?

> if you present the right credentials, but may not if you don't. And so on.

Nope. That's a completely different issue. If your credentials do not 
allow executing a particular method (such as PUT when no write access), 
the answer would be 403; so this has nothing to do with PUT being in the 
"Allow" set.

Yes, this is confusing; so we may want to clarify first what the 405 
status code means.

> Add to this the original concern of the issue; that some implementations 
> may not be able to determine the complete set at runtime.

The original concern seems to be about two things:

- the set of methods may be hard to compute -- a server that says "405" 
but really doesn't know what methods would not generate a 405 seems a 
bit strange to me, but it could still generate something else, like a 
generic 400, or 501.

- you may not want to expose methods to unauthorized clients -- that's a 
very valid concern, so I'd say we just allow the server to make the 
"allow" set depend on the credentials; I don't think this would affect 
any clients in practice.

> In short, the phrase does not include "complete", and the set of methods 
> that it will accept is not necessarily the inversion of the set of 
> methods it won't accept; there is a gray area in between.

Sorry? What would be in that gray area?

> One way to fix this is to make the definition of the set less fuzzy, but 
> as Roy has pointed out, that's taking liberties.
> 

It seems to me it would be a good idea to first come up with a common 
understanding when it is correct to return 405, as opposed to 501 or 403.

BR, Julian

---

From: John Kemp <[email-masked]>
Date: Tue, 04 Mar 2008 08:58:56 -0500

Hi,

> the response was generated.

As has already been mentioned, that set may vary with each response, 
depending on contextual issues, such as authorization.

> *does* change frequently.

There is no need for the client to "believe" anything at all is there? 
The client is interested in getting a result, often on behalf of a 
human, who probably doesn't want to see "HTTP/1.1 405 Method Not Allowed".

If the server tells the client that the allowed methods for that 
resource are currently GET and POST and the client gets what it was 
after by using one of those methods, what else is necessary?

[...]

> the answer would be 403; so this has nothing to do with PUT being in the 

In my understanding, 403 indicates that access to the resource is 
currently "Forbidden", not that the method used is not allowed. For 403, 
isn't it the case that the request was found to be formed correctly (ie. 
the resource exists, and the method used on that resource is allowed)?

I think Mark was saying that the server may want to tell the client that 
the method was not allowed, but should also be allowed to return either 
no list, or an incomplete list of allowed methods, depending on whether 
the request contained adequate authorization to receive even this 
additional information.

>> implementations may not be able to determine the complete set at runtime.
> 

Agreed; that is the case noted above. In other words, the set of ALLOWed 
methods returned might vary between requests to the same resource.

> understanding when it is correct to return 405, as opposed to 501 or 403.

I agree with Mark - "document the fuzziness" in the simplest possible 
way, and move on. In my mind, the spec. is "clearly ambiguous" enough 
already, but it seems that others don't see it that way.

The simplest possible change (adding the smallest amount of extra 
ambiguity ;) may be the one-word change I suggested in response to your 
earlier complaint, Julian. But I'm also just fine if we use Mark's 
suggested text.

The spec. is currently ambiguous (there are different interpretations 
possible based on the text, as proven by this discussion), and I think 
it should become no less so - but also the least amount more so.

Regards,

- johnk

---

From: Julian Reschke <[email-masked]>
Date: Tue, 04 Mar 2008 15:26:59 +0100

John Kemp wrote:
> ...
>> The allowed set is the set of methods that was allowed at the time 
>> when the response was generated.
> 

As long as the client sticks to the same credentials, there is little 
reason why it would vary.

> after by using one of those methods, what else is necessary?

Nothing.

But if the server does support PATCH, but doesn't include "PATCH" in 
"Allow:", and then the client even doesn't try PATCH because of that, 
that *is* a problem.

>> allow executing a particular method (such as PUT when no write 
>> access), the answer would be 403; so this has nothing to do with PUT 
>> being in the "Allow" set.
> 

Yes. That's what you send when the client was authorized, but the 
request is forbidden for the credentials that were given.

> the request contained adequate authorization to receive even this 

Not sure I understand this. Seems to be another hint that we need to 
clarify what "405 Not Allowed" means.

>> - the set of methods may be hard to compute -- a server that says 
>> "405" but really doesn't know what methods would not generate a 405 
>> seems a bit strange to me, but it could still generate something else, 
>> like a generic 400, or 501.
>>
>> - you may not want to expose methods to unauthorized clients -- that's 
>> a very valid concern, so I'd say we just allow the server to make the 

Possible, but extremely unlikely.

If the client is not authorized at all, and the resource requires 
authentication for the given method, then the right answer is 401 anyway.

If it is authorized, the set of allowed methods will *usually* not vary 
across requests with the same credentials. Why would it?

>>> trying to do. If others have better ways of doing that, or a third 
>>> way forward, please say so.
>>

The ambiguities that I see are:

- it's not really clear when to send 405 -- which really is a separate 
issue, and

- the spec only says "the set of methods" in one location.

> earlier complaint, Julian. But I'm also just fine if we use Mark's 

As I have explained (or tried to explain) several times, if we relax the 
server requirement, we *have to* relax the client requirement ("SHOULD 
believe the header") as well.

> it should become no less so - but also the least amount more so.

I disagree that there are multiple correct interpretations of what it 
currently says in 14.7. Some confusion is caused by the weaker 
requirement in 10.4.6, so we need to fix that.

Besides that, I'd argue to not break what's in use; thus just warn about 
certain servers not sending a complete list, and taking out the 
requirement on clients not to try anything else.

BR, Julian

---

From: John Kemp <[email-masked]>
Date: Tue, 04 Mar 2008 09:49:16 -0500

[...]

> "Allow:", and then the client even doesn't try PATCH because of that, 

If the client *wants* or *needs* to try PATCH, then it can, regardless 
of what the server has or has not told it. If it gets a 405 in return, 
then it probably wouldn't want to try that again.

[...]

>>
>> Agreed; that is the case noted above. In other words, the set of 
>> ALLOWed methods returned might vary between requests to the same 
>> resource.
> 

Possible, though.

> across requests with the same credentials.

If the client doesn't present credentials at all when making an initial 
request, the server may not be willing to present it with any additional 
status information in a 405. If the client later does acquire 
authorization credentials (perhaps during access to another resource) 
and then accesses the same resource for which it earlier received a 405, 
then the resulting response may now contain more information.

> Why would it?
>>>> The other is to document the fuzziness and move on, which is what 
>>>> I'm trying to do. If others have better ways of doing that, or a 
>>>> third way forward, please say so.
>>>
>> ambiguity ;) may be the one-word change I suggested in response to 
>> your earlier complaint, Julian. But I'm also just fine if we use 
>> Mark's suggested text.
> 

My understanding of SHOULD is that it is not MUST.

I imagine a client will often simply try a method that it needs to use 
anyway.

> requirement on clients not to try anything else.

OK - I can agree on the warning (I guess that's more or less Mark's text?)

But the SHOULD already doesn't prevent a client from trying something it 
needs to (and will not unless it became a MUST). And that SHOULD *does* 
give the impression that the client should listen to the server, and 
that the server should not deceive the client. That all seems fine and 
proper.

Cheers,

- johnk

---

From: Julian Reschke <[email-masked]>
Date: Tue, 04 Mar 2008 16:14:03 +0100

> then it probably wouldn't want to try that again.

Many clients try to present a UI to their user, disabling things that 
are known not to work. So being able to find out whether a resource 
supports a specific method is a useful feature.

Yes, this does not apply to all clients, but that doesn't mean we can 
just get rid of it.

>> Possible, but extremely unlikely.

But that makes a big difference. When, in general, the information is 
reliable, clients are going to use it.

When, in general, it is unreliable, clients will ignore it.

>> If it is authorized, the set of allowed methods will *usually* not 
>> vary across requests with the same credentials.
> 

My point was that in this case 405 is already a bad choice. And if the 
status code is not 405, the whole discussion is pointless.

> then the resulting response may now contain more information.

Yes. One more reason not to use 405 in the first case.

>> As I have explained (or tried to explain) several times, if we relax 
>> the server requirement, we *have to* relax the client requirement 
>> ("SHOULD believe the header") as well.
> 

That is correct but not really helpful.

A spec that says "servers MAY lie" but "clients SHOULD trust them" makes 
a very bad use of RFC2119 terminology.

If we want to *allow* servers to return incomplete lists, we simply can 
not say "SHOULD follow that list".

> I imagine a client will often simply try a method that it needs to use 

Yes, that's one way to things. Clients that work this way ignore 
"Allow:" anyway.

What I am interested in are clients that *do* use the value, for 
instance, in order to populate a UI.

> that the server should not deceive the client. That all seems fine and 

Doesn't work for me. We can't have a "SHOULD use" for information we 
allow to be broken.

If we make changes, let's do so in a consistent way, please.

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 4 Mar 2008 09:25:18 -0800

Let me reiterate.  What 2616 says on this issue is irrelevant because we
all know that no implementation behaves the way that 2616 suggests, no
matter how the sentence is parsed. Therefore, arguing over the exact
meaning of the existing phrasing is a waste of time.

Backwards compatibility is NOT about retaining consistency between
a document that nobody reads and a new document that we hope people do.
It is about specifying the protocol so that it corresponds to what
existing implementations accept and what new implementations should
implement in order to operate with both new and existing
implementations of HTTP.

After the definition of Allow is rephrased to correspond to existing
practice and the false requirements are removed, then we can talk about
the exact wording of the true requirements (if any).

The only current usage of Allow that I know of is in the response to
OPTIONS and in the response to 405, and in both cases it includes the
list of methods believed to be allowed at the time of that request by
the resource handler (which may or may not be aware of all possible
methods).  No client that I know of depends on the list being 100%
complete, though at least some webdav clients will look for their
favorite methods in the list.  Indeed, all that matters is that the
field contains the list of methods that the server wants to provide
to the user agent, regardless of why they want to provide it and
regardless of its completeness.

If anyone knows of specific examples that differ from the above,
please let us know.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 04 Mar 2008 18:32:23 +0100

> meaning of the existing phrasing is a waste of time.

(I wrote an implementation which I believe does what RFC2616 says; not 
that this would be relevant, but anyway).

> If anyone knows of specific examples that differ from the above,

No, I would think this is correct.

So how would this translate into specification text?

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 14 Mar 2008 00:58:51 +0100

On Thu, 2008-02-28 at 13:26 +0100, Julian Reschke wrote:

> It seems to me it would be unwise to say "clients SHOULD believe the 
> Allow header", but "servers MAY leave of methods".
> 
> If we relax the requirement for the production, we also need to relax 
> the requirement for the recipient.

What I have been saying all the time. If a server cannot make a
realistic list of methods it accepts for a given resource it's better it
stays silent on the subject than trying to guess.

BUT if a server can make a realistic list of methods it SHOULD indicate
this, to allow clients to effectively select the best methods for what
they want to do (i.e. enable WebDAV instead of querying the user for
what method to use for updating content).

Relaxing the meaning of Allow on the server side is the wrong way to
tackle the problem, just makes Allow useless.

If Allow is given by a server it SHOULD be trusted by clients. But it
may include methods THIS client can not perform due to other
restrictions (i.e. not yet having the credentials needed to perform the
request)

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 14 Mar 2008 01:07:20 +0100

> believe" requirement for the client.

What exacly is the problem with the client requirement "SHOULD be
followed"?

It's a recommendation that IF the server has indicated a list of
supported methods the client should not try to use methods outside this
set unless is has specific reasons to go against the servers
recommendation.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 14 Mar 2008 01:17:44 +0100

> requirement on clients not to try anything else.

It's already only a recommendation, not a requirement.

SHOULD, not a MUST.

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 13 Mar 2008 17:23:18 -0700

> What exacly is the problem with the client requirement "SHOULD be

It is not an interoperability requirement and there are no known
implementations.

> set unless is has specific reasons to go against the servers

In other words, it says nothing useful.  There are no interchangeable
methods in HTTP.  Allow only lists the methods that are known to
be allowed, not the set of all methods that might be allowed.
As such, "SHOULD be followed" is wrong.  Allow is useful for one
and only one purpose: to discover those methods that the server
is willing to advertise, usually in response to OPTIONS.

....Roy

---

From: Robert Sayre <[email-masked]>
Date: Thu, 13 Mar 2008 20:27:16 -0400

...

>
> In other words, it says nothing useful.

Fully agree.

- Rob