Subject: PROPOSAL: i24 Requiring Allow in 405 Responses

From: Mark Nottingham <[email-masked]>
Date: Mon, 17 Mar 2008 17:43:21 +1100

The proposal is to:

* In p2 10.1, change "The actual set of allowed methods is defined by  
the origin server at the time of each request."  to  "The actual set  
of allowed methods is defined by the origin server at the time of each  
request, and may not necessarily include all (or any) methods that the  
server would actually allow in a request if presented."  (with normal  
editorial discretion)

* In p2 10.1, remove "However, the indications given by the Allow  
header field value SHOULD be followed."

--
Mark Nottingham     http://www.mnot.net/

---

From: Stefan Eissing <[email-masked]>
Date: Mon, 17 Mar 2008 10:29:34 +0100

Am 17.03.2008 um 07:43 schrieb Mark Nottingham:

> The proposal is to:
>
> * In p2 10.1, change "The actual set of allowed methods is defined  
> by the origin server at the time of each request."  to  "The actual  
> set of allowed methods is defined by the origin server at the time  
> of each request, and may not necessarily include all (or any)  
> methods that the server would actually allow in a request if  
> presented."  (with normal editorial discretion)

How about:

"The published set of allowed methods is defined by the origin server  
at the time of each request. The absence of methods in this set has  
no defined semantics."

> header field value SHOULD be followed."
+1

--
<green/>bytes GmbH, Hafenweg 16, D-48155 Münster, Germany
Amtsgericht Münster: HRB5782

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 17 Mar 2008 20:54:59 +1100

I'm happy for that to be input to the editors -- or are you making a  
separate proposal?

Cheers,

On 17/03/2008, at 8:29 PM, Stefan Eissing wrote:

> "The published set of allowed methods is defined by the origin  
> server at the time of each request. The absence of methods in this  
> set has no defined semantics."
>
>
>>

---

From: Robert Siemer <[email-masked]>
Date: Mon, 17 Mar 2008 11:40:27 +0100

> server would actually allow in a request if presented."  (with normal  

Once again that reads "The actual set of allowed methods ... may not ... 
include all (or any) methods that the server would actually allow ..."

The actual set _is_ the actual set. - I understood the old sentence, but 
I don't understand the proposed change.

> header field value SHOULD be followed."

Okay.

Robert

---

From: Julian Reschke <[email-masked]>
Date: Mon, 17 Mar 2008 13:15:56 +0100

OK,

here's what I'd like to do, based on Mark's summary and Stefan's latest 
proposal:

In the description of the Allow header, replace

  This field cannot prevent a client from trying other methods.
  However, the indications given by the Allow header field value
  SHOULD be followed. The actual set of allowed methods is defined
  by the origin server at the time of each request.

by

  This field cannot prevent a client from trying other methods.
  The published set of allowed methods is defined by the origin server at
  the time of each request. The absence of methods in this set has no
  defined semantics.

Also, in "Changes from RCF2616", add

  Relax the server requirement on the contents of the Allow header and
  remove requirement on clients to always trust the header value.

BR, Julian

Stefan Eissing wrote:
> "The published set of allowed methods is defined by the origin server at 
> the time of each request. The absence of methods in this set has no 
> defined semantics."
> 
> 
>>

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 16:15:30 +0100

> server would actually allow in a request if presented."  (with normal  

Would be good to get something about "advertised as supported" in there.
The use of Allow is for servers to advertise that they do support a
given method (or indirectly HTTP extension) so clients MAY enable the
use of those methods, not to tell clients not to use certain methods.

> header field value SHOULD be followed."

I am fine with that with the above clarification of the use of Allow
keeping it on track with the intended use.

If not clarified I am afraid that implementers may loose sight on why
Allow should include a meaningful list of methods or why they should
send Allow at all.

Regards
Henrik

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 18 Mar 2008 09:18:00 -0700

Julian Reschke wrote:
>   The published set of allowed methods is defined by the 
>   origin server at the time of each request. The absence of
>   methods in this set has no defined semantics.

The first sentence "This field cannot prevent..." is meaningless and
thus confusing. The server cannot prevent the client from doing
anything. It should be removed.

"The absense of of methods in this set has no defined semantics" implies
that the presence of methods in the set does have (well-)defined
semantics, which is not really true.

How about:

"The methods listed in the Allow field of a response were allowed when
that response was produced; additional, unlisted, methods may have been
allowed at that time as well. Regardless of whether or not a method was
listed in a previously-returned Allow field, the server MAY allow or
refuse the use of any method in any (future) request. Servers are
RECOMMENDED to include an accurate and complete set of methods in any
Allow field returned in a response."

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2008 18:54:08 +0100

> anything. It should be removed.

Agreed.

> semantics, which is not really true.

I wouldn't go that far; the presence does have semantics, with some caveats.

> Allow field returned in a response."

- This emphasizes a lot that the Allow set may change over time; in 
general I think this is misleading. It should only change when either 
the implementation or the "type" of the resource changes (for instance 
in WebDAV: collection vs non-collection)

- "RECOMMENDED" re-adds the normative requirement that most people want 
to get rid of, as some servers do not get this right in practice 
(remember that RECOMMENDED == SHOULD).

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 20:21:58 +0100

> (remember that RECOMMENDED == SHOULD).

True, and which means something you SHOULD do unless you have a darn
good reason not to do so (such as not being able), and if you can't do
what you SHOULD then you MAY do things otherwise knowing that then you
MAY confuse others and any such confusion is entirely your own fault by
not being able to provide the correct information.

None of the proposed changes change this picture at all. All the
proposed changes do is to add to the confusion imho, and even supporting
it.

a) Encourage servers to not even attempt in providing valid information
by lowering the requirement level from a SHOULD or saying that the list
does not need to be correct.

b) Up front tells clients that they should not look into the header as
it's probably useless (which it isn't).

The issue was brought up because of servers having a filter before the
actual processing engine which blocks methods, without any knowledge of
what methods the actual processing engine can support. Imho such filters
have no business returning 405 in the first place and should return 403
instead. If they do insist on returning 405 it's better if such filters
do not return an Allow header at all.

Thats my view of this.

But if there is a consensus that it's meaningful to make Allow by
removing the conditions which gives it any value, only to make servers
who insist on returning 405 without even knowing what they do support
still fully compliant (not only conditionally compliant) when I won't
object, but it's a poor direction to take imho.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 22:04:01 +0100

> Allow field returned in a response."

Or the existing SHOULD.. it's a actually fine condition in terms of HTTP
functionality, and it being a SHOULD (not a MUST) means that server who
can't comply for some reason MAY do what they want but knowing that then
clients MAY not do entirely what they expect..

I don't strongly object the suggested changes, but I simply don't see
why they are needed.

Regards
Henrik

---

From: John Kemp <[email-masked]>
Date: Tue, 18 Mar 2008 15:30:01 -0700

Henrik Nordstrom wrote:

[...]

> I don't strongly object the suggested changes, but I simply don't see

+1. I don't see how they actually make the situation any clearer than it 
is today, and so I wonder why they are needed.

- johnk

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Mar 2008 11:54:44 +1100

I think some change is needed, because the text currently places  
unclear conformance requirements on servers and clients.

Revised proposal;

In p2 10.1, change
'''This field cannot prevent a client from trying other methods.  
However, the indications given by the Allow header field value SHOULD  
be followed. The actual set of allowed methods is defined by the  
origin server at the time of each request.'''
to
'''The advertised set of allowed methods may not necessarily include  
all (or any) methods that the server would actually allow in a  
presented request.'''

(keeping in mind that the start of the Allow's header definition is:
'''The Allow entity-header field lists the set of methods supported by  
the resource identified by the Request-URI. The purpose of this field  
is strictly to inform the recipient of valid methods associated with  
the resource.'''
)

On 19/03/2008, at 9:30 AM, John Kemp wrote:

> +1. I don't see how they actually make the situation any clearer  
> than it is today, and so I wonder why they are needed.
>
> - johnk
>
>

---

From: Stefan Eissing <[email-masked]>
Date: Wed, 19 Mar 2008 15:40:54 +0100

Am 19.03.2008 um 01:54 schrieb Mark Nottingham:

> SHOULD be followed. The actual set of allowed methods is defined by  
> the origin server at the time of each request.'''
> to
> '''The advertised set of allowed methods may not necessarily  
> include all (or any) methods that the server would actually allow  
> in a presented request.'''
>
> '''The Allow entity-header field lists the set of methods supported  
> by the resource identified by the Request-URI. The purpose of this  
> field is strictly to inform the recipient of valid methods  
> associated with the resource.'''
> )

I am fine with this.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 19 Mar 2008 15:50:20 +0100

Mark Nottingham wrote:
> 
> I think some change is needed, because the text currently places unclear 
> conformance requirements on servers and clients.
> 
> Revised proposal;
> 
> In p2 10.1, change
> However, the indications given by the Allow header field value SHOULD be 
> followed. The actual set of allowed methods is defined by the origin 
> server at the time of each request.'''
> to
> '''The advertised set of allowed methods may not necessarily include all 
> (or any) methods that the server would actually allow in a presented 
> request.'''
> 
> the resource identified by the Request-URI. The purpose of this field is 
> strictly to inform the recipient of valid methods associated with the 
> resource.'''
> )

+1

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 20 Mar 2008 00:20:10 +0100

> unclear conformance requirements on servers and clients.

Yes. The correct requirements for Allow should be

Servers SHOULD advertise a correct list of the methods the server want
clients to try to use on the resource.

Clients SHOULD honor what the server says when it says something.

Ofcourse there may be supported methods outside of the advertised set,
but servers SHOULD NOT count on clients probing for such methods.

And ofcourse clients MAY attempt other methods is they have exceptional
reason to believe those methods are in fact supported (i.e. by
out-of-band knowledge or manual override).

The rest of the fuzzy cruft discussed is then more or less automatic
from the requirements being just a SHOULD level requirement, allowing
implementations to diverge from correctness is they absolutely need to.

It does not need to be a MUST level requirement for servers to provide a
relevant list in 405 responses, which is what got this thread started.
Some servers can't. Some servers won't. But implementations not doing so
should carefully consider what this means to clients looking into the
contents of the Allow header and assume responsibility if their
implementation causes problems to clients trying to make use of the
information and therefore a SHOULD level requirement to provide a
relevant list.

So my proposed change is to just change "MUST include an Allow header"
to SHOULD in both "Allow" and "405 Method Not Allowed", and leave the
rest as it is.

While at it I would also recommend lowering the proxy rewrite
requirement in "Allow" to SHOULD NOT (currently a MUST NOT), or restrict
it to "A transparent proxy ...". That's another MUST level requirement
which doesn't really make sense in this context.

I would not go into the fuzzy land of "need not be complete" etc. Such
decisions is best left to implementers and only confuses matters further
I think. 

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Mar 2008 15:21:12 +0100

> clients to try to use on the resource.

RFC2616 says "MUST" advertise (in 405 responses). I think the concern is 
that if we relax this to "SHOULD", clients that rely on the presence of 
the header will break. That is, break in a more fatal way as if the 
header is there but incomplete.

Note this is already only a "SHOULD" for the OPTIONS responses. As far 
as I understand, the "Allow on OPTIONS" is the thing clients are really 
interested in, not "Allow on 405".

(I still have a really hard time understanding why a server ever would 
be able to state "405 Not Allowed", but not be able to say what would be 
allowed. Note that 405 != 501.)

Anyway: do we have any evidence of

1) clients failing in *absence* of Allow in a 405 response, and

2) servers failing to return Allow in a 405 response?

(Henrik, can you check logs...?)

If we have evidence for 1) (which I really doubt), we probably need to 
stick with the MUST.

If we have evidence for 2) (which wouldn't surprise me), we probably can 
relax it to SHOULD, as proposed by Henrik.

If we have evidence for both, I'd recommend that we put this issue back 
on the stack of open issues, and try to make progress somewhere else :-)

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 20 Mar 2008 07:32:34 -0700

Henrik,

I agree with your interpretation of what SHOULD means. I mostly agree
with you about the server side, but I disagree that the server should
return 403 instead of 405 if it cannot create an accurate Allow header
field. 403 should only be used when none of the more specific status
codes is applicable, or when one of the other status codes would
disclose too much information. If the server is rejecting the request
solely because of the method used, then 405 is the most appropriate. I
would say that the server SHOULD return an Allow header field in its 405
responses if and only if it can accurately list all of the allowed
methods.

I totally disagree with you regarding the client side behavior though:

* The requirements for client processing of 405/Allow do not need to be
stricter than the requirements for client processing of
401/WWW-Authenticate. When the server returns 401/WWW-Authenticate,
there is nothing that says the client cannot submit the request again
unchanged; we rely on common sense there, and we can (and should) rely
on common sense for 405/Allow.

* The SHOULD requirement implies that the client SHOULD maintain state.

* The SHOULD requirement doesn't have any benefit. The server doesn't
benefit because still has to be able to handle requests with methods it
doesn't support.

* Regardless of what the specification says, clients still have to deal
with servers that do not follow the specification. 

Regards,
Brian

---

From: David Morris <[email-masked]>
Date: Thu, 20 Mar 2008 07:12:16 -0800 (PST)

> allowed. Note that 405 != 501.)

It could be because obtaining a complete inventory is not practical OR
perhaps the server plug-in component which knows that this method isn't
allowed doesn't have permission to make the inventory. Requiring a
complete inventory would mean that server/container would have to
manage the collection when it notices the 405 AND each plug-in
would need a special "is this allowed" API which wouldn't actually
perform the work.

Not reasonable to expect.

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 20 Mar 2008 21:39:33 +0100

On Thu, 2008-03-20 at 15:21 +0100, Julian Reschke wrote:

> RFC2616 says "MUST" advertise (in 405 responses).

Yes.. but it's not entirely clear why this is a MUST.

> I think the concern is 

I very much doubt so for 405. It's not OPTIONS we are talking about
here..

> interested in, not "Allow on 405".

Exactly.

And it's valuable to be able to respond with 405 even if you don't qave
full knowledge of the allowed methods. For example in a frontend
intermediary server or web server internal security filter infront of
the actual web application stack.

> allowed. Note that 405 != 501.)

Yes, I have also raised this. There is several other status codes one
MAY use if access is denied due to the method used. But it's a SHOULD
level requirement that 405 SHOULD be used.

"Method"

An origin server SHOULD return the status code 405 (Method Not Allowed)
   if the method is known by the origin server but not allowed for the
   requested resource, and 501 (Not Implemented) if the method is
   unrecognized or not implemented by the origin server

> 1) clients failing in *absence* of Allow in a 405 response, and

Can't think of any. Most clients uses OPTIONS and should only get to 405
if they do not care about the Allow result in the first place, or if
there a server configuration error where it includes disallowed methods
in Allow.. (which most likely would bite Allow in 405 as well..)

> 2) servers failing to return Allow in a 405 response?

Quite likely.

> (Henrik, can you check logs...?)

405 is a very rare status code in normal traffic, and the logs I have
access to do not include headers..

But from a quick manual probe based on the handful unique 405's I have
there is some servers not sending Allow in 405 today, as expected.

HEAD http://articles.moneycentral.msn.com/Feeds/RSS/latestrss.aspx

is one..

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 20 Mar 2008 22:03:34 +0100

> responses if and only if it can accurately list all of the allowed

Same here actually, but it's not a big difference between 405 and 403
here, save for the Allow header requirement. But it is true that 403
does not make any direct claims on what it was with the request that was
unacceptable.

> on common sense for 405/Allow.

I am not sure the above is true.. it's pretty much implied clients
should stop on 4xx errors and not automatically retry that request. But
I see your point and it has some value.

Common sense here however is that the SHOULD should be followed for the
duration of this transaction or session, not that the client should keep
persistent state information.

> * The SHOULD requirement implies that the client SHOULD maintain state.

Yes.. to a certain degree at least.

> benefit because still has to be able to handle requests with methods it

It's there to indicate that clients should know better than going
against the servers stated intentions as it's most likely a waste to
attempt to do so.

> with servers that do not follow the specification. 

Irrelevant implementation detail. Clients do not have to interoperate
with servers not following specifications.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Mar 2008 08:44:44 +1100

I'm afraid I disagree (as much as I'd like to see this issue closed  
quickly).

Your suggestion clarifies when an Allow header must/should appear, but  
does not clarify what it must contain -- although there seems to be an  
implication that it be 'complete.'

As others and myself have pointed out before, that's not realistic.

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 20 Mar 2008 23:28:41 +0100

> As others and myself have pointed out before, that's not realistic.

I still haven't really got what about "lists the set of methods
supported by the resource" is unclear. A method is either supported or
it's not.

But a small suggestion is found below maybe addressing the concerns and
lining up the definition more with the intended use and emphasizing why
sending meaningful Allow headers is important.

What is currently said about the contents of Allow:

"Method"

   The list of methods allowed by a resource can be specified in an
   Allow header field (section "Allow"). The return code of the response

"OPTIONS"

   A 200 response SHOULD include any header fields that indicate
   optional features implemented by the server and applicable to that
   resource (e.g., Allow), possibly including extensions not defined by
   this specification.

"405 Method not allowed"

   [...] include an
   Allow header containing a list of valid methods for the requested
   resource.

"Allow"

      [...] lists the set of methods supported
      by the resource identified by the Request-URI. The purpose of this
      field is strictly to inform the recipient of valid methods
      associated with the resource.

[...]

My reading of the above: It's a header telling the client "These are the
methods I currently support on this resource and which you MAY use to
perform actions on this resource. I recommend you not to try to use any
other methods as I most likely do not support them and will reject any
such attempts with an appropriate error so doing so would be a waste of
both yours and my time and resources."

For security reasons the following may also be implied "Things may
change in the future if my configuration or the resource state is
changed to allow other methods or significant properties of your request
changes making you gain more access rights than you have today.", but I
am not convinced the specs really means the latter. But makes sense from
a security point of view and is commonly implemented today.  The first
(configuration & state) is kind of pointless to mention as it's a
constant fact about anything, but is what the "at the time of each
request" is about.

So here is my suggestion about resolving the contents of Accept:

Does it help to get readers more aligned if "Allow" is rewritten to
include advertised in the definition?

     [...] lists the set of methods advertised as supported ...

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Mar 2008 22:13:33 +1100

Yes, I think that's helpful; it's a more explicit version of the "the"- 
 >"a" suggestion, IMO.

Is your suggestion that the language be rewritten throughout in this  
style (e.g., "The list of methods allowed by a resource can be  
specified" -> "A list of methods allowed by a resource can be  
advertised")?

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 25 Mar 2008 13:50:33 +0100

> specified" -> "A list of methods allowed by a resource can be  

Yes, or reduced to references to the "Allow" definition.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Mar 2008 14:00:55 +0100

Henrik Nordstrom wrote:
> ...
> ...

I don't think so. If the authorized principal lacks permissions, the 
right response is 403 (Forbidden) or 404 (Not found, if the server 
doesn't want to reveal whether the resource exists). I don't see why it 
would ever be 405, except if the author of the server didn't understand 
the difference between 403 and 405 (and I think this is something where 
our spec could be clearer).

>      [...] lists the set of methods advertised as supported ...

Works for me. Would we still remove the SHOULD-level requirement for the 
client?

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 25 Mar 2008 15:18:38 +0100

> Works for me. Would we still remove the SHOULD-level requirement for the 

I don't see why, and have never seen why, but I don't strongly object to
removing the client requirement.

The only problem brought up in this thread that I can acknowledge as a
real problem is with Allow being a MUST in 405 combined with the fact
that there is value for an intermediary layer be able to indicate 405
without knowing the actual list of supported methods (which would call
for a 405 without any Allow header). 403 is a more detailed indication
of the error than 403 even without Allow.

The client requirement is a SHOULD (not a MUST), and ignoring servers
who advertise wrongly (i.e. going against a MUST/SHOULD requirement)
there is not really a problem with the requirement as it is today save
for the slight issue of kind of implying that clients need to keep state
about the Allow header. Yes, it's true that most clients using the Allow
header do not follow Allow to 100% but only looks for a small subset of
the methods they use, but that's an implementation detail well within
the SHOULD level requirement. And I also consider clients not keeping
state or follow the Allow header partially or at all as compliant as
long as they are prepared with handling the outcome of their actions and
assume responsibility for them.

But with the change in language to use advertised I think everyone
looking into this aspect of the specs will get on track so it doesn't
matter much what is said about the client. Common sense is that you
should not try to use methods outside of the advertised supported set,
and that you are on your own if you do.

Regards
Henrik