Subject: #230: Considerations for registering new methods

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Oct 2010 17:04:24 +1100

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/230>

I've incorporated the proposal below as a starting point:

--->8---

When it is necessary to express new semantics for a HTTP request that aren't specific to a single application or media type, and currently defined methods are inadequate, it may be appropriate to register a new method [ref].

New methods SHOULD be potentially applicable to any resource. I.e., they should not be specific to any particular media type, "type" of resource, or application.

New methods MUST NOT prohibit a message-body on either the request or the response message; however they MAY specify that only a zero-length body is allowed.

New methods MUST define whether they are safe [ref] and whether they are idempotent [ref]. They MUST also state whether they can be cached [ref to p6]; in particular what conditions a cache may store the response, and under what conditions such a stored response may be used to satisfy a subsequent request.

New methods SHOULD explain how conditional request headers [ref] affect the response (if there is any effect).

HTTP methods SHOULD be registered in a document that isn't specific to an application or other use of HTTP, so that it's clear that they are not specific to that application or extension.

---8<---

I expect we'll find more here, but wanted to get something in. Feedback appreciated, as always.

--
Mark Nottingham   http://www.mnot.net/

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 19 Oct 2010 00:39:20 -0600

Mark Nottingham wrote:
> 
> New methods MUST NOT prohibit a message-body on either the request or
> the response message;
>

Why not?  I'm tinkering with a new method, for which a request message-
body, even zero-length, makes no sense.

-Eric

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Oct 2010 17:44:08 +1100

Because that makes it a special case for parsers, and you'll have interop issues with existing implementations.

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 19 Oct 2010 09:39:06 +0200

> body, even zero-length, makes no sense.

Even if it makes no sense, it must not be an exception to how the
messaging is designed to work. So even if the body makes no sense,
it should be specified as a "must be skipped and ignored".

Regards,
Willy

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 19 Oct 2010 10:17:23 +0200

On Tue, 19 Oct 2010 08:44:08 +0200, Mark Nottingham <[email-masked]> wrote:
> Because that makes it a special case for parsers, and you'll have  
> interop issues with existing implementations.

Yeah, for XMLHttpRequest we had to special case GET/HEAD to omit any  
passed request entity bodies. We do not want to add more methods there.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 19 Oct 2010 03:07:56 -0600

> passed request entity bodies. We do not want to add more methods

OK, I understand that.  But does this mean that the no-entity-body
requirement for GET/HEAD is a historical mistake, or was there some
reason for parsing these requests differently?  What I'm experimenting
with is an IDLE method (IMAP has one) very similar to GET, so I'm
trying to understand why I can't just copy the definition of GET as a
starting point.  I'm convinced by the responses *not* to do that, but
wondering what gotcha may be lurking.

-Eric

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 19 Oct 2010 11:12:06 +0200

> wondering what gotcha may be lurking.

IMHO, it is fine to say that the method MUST NOT include a body, but it
is still required to indicate what to do with that body if any is found.
Otherwise, you'd get the same mistake as GET/HEAD consisting in some
implementations not looking at the content-length at all and being
vulnerable to request smuggling attacks.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Oct 2010 12:11:10 +0200

> I've incorporated the proposal below as a starting point:

Thanks a lot.

I have some comments, both editorial and content-wise; almost the same 
apply to #229 (status codes).

> New methods SHOULD be potentially applicable to any resource. I.e., they should not be specific to any particular media type, "type" of resource, or application.

General question: is the use of RFC2119 keywords appropriate here? What 
about the "should" in the 2nd sentence?

> New methods MUST NOT prohibit a message-body on either the request or the response message; however they MAY specify that only a zero-length body is allowed.

I think it would be good if we rephrased that as a general statement 
about HTTP methods, and then pointed out that new methods need to follow 
that.

Such as:

"Due to the parsing rules defined in ..., HTTP methods cannot prohibit 
the presence of a message-body on either the request or the response 
message (with the single exception of HEAD responses). Definitions of 
new methods cannot change this rule, but they can specify that only 
zero-length bodies (as opposed to absent bodies) are allowed."

> New methods MUST define whether they are safe [ref] and whether they are idempotent [ref]. They MUST also state whether they can be cached [ref to p6]; in particular what conditions a cache may sto…

Agreed. We'd better make sure that our method definitions give a good 
example how to do this ;-)

> New methods SHOULD explain how conditional request headers [ref] affect the response (if there is any effect).

Wow. I always thought they apply to any method; thus extensions methods 
can't be special. Maybe this deserves a separate issue?

> HTTP methods SHOULD be registered in a document that isn't specific to an application or other use of HTTP, so that it's clear that they are not specific to that application or extension.

That's very vague (what does "other use" mean).

Maybe we should start with examples, and then come up with guidelines?

- PATCH (RFC 5789) is certainly a good example for a generic definition.

- MKCALENDAR (RFC 4791) is certainly a good example for something that 
shouldn't have been a separate method.

I think most of us agree on the above. But what about things between 
these extremes, such as WebDAV?

> ...

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 20 Oct 2010 10:08:48 +1100

Re GET, see
  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/19

The only exception method is HEAD, as per
  http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-11#section-3.3

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 20 Oct 2010 11:32:14 +1100

On 19/10/2010, at 9:11 PM, Julian Reschke wrote:
> General question: is the use of RFC2119 keywords appropriate here? What about the "should" in the 2nd sentence?

In as much as they describe requirements for the registration of new methods, and what is considered a conformant application, I think it's appropriate. Using the keywords has the advantage of unambiguously communicating the requirements.

However, I'm not lie-down-in-the-road about using them.

> I think it would be good if we rephrased that as a general statement about HTTP methods, and then pointed out that new methods need to follow that.
> 
> Such as:
> 
> "Due to the parsing rules defined in ..., HTTP methods cannot prohibit the presence of a message-body on either the request or the response message (with the single exception of HEAD responses). Definitions of new methods cannot change this rule, but they can specify that only zero-length bodies (as opposed to absent bodies) are allowed."

Seems reasonable.

> Agreed. We'd better make sure that our method definitions give a good example how to do this ;-)

Yup :)

> Wow. I always thought they apply to any method; thus extensions methods can't be special. Maybe this deserves a separate issue?

Well, we could say that 304 is specific to GET/HEAD, and all other methods use 412 for failed conditions, but that bit of glue seems to be missing ATM. 

We should probably also warn that clients can't take it for granted that a resource will support conditional requests (if we don't already); this can be surprising if you're expecting it on POST/PUT, etc.

It sounds like we should remove this sentence and open two new issues.

> That's very vague (what does "other use" mean).

That can be dropped.

> - MKCALENDAR (RFC 4791) is certainly a good example for something that shouldn't have been a separate method.
> 
> I think most of us agree on the above. But what about things between these extremes, such as WebDAV?

What this is saying is that the WebDAV methods should have been registered in a non-WebDAV specific document, to clarify that they can be used in other contexts.

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 19 Oct 2010 22:34:02 -0600

>   http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-11#section-3.3

Thanks for the reference, it led me to this, which is the explanation I
was looking for:

http://lists.w3.org/Archives/Public/ietf-http-wg/2002JulSep/0036.html

*Now* I get it (I'm not dense, just a little slow).  Also, that last
sentence is particularly helpful to me, as I was defining my method
based on my intended use, instead of thinking in terms of extensibility
i.e. re-use...

> 
> >> HTTP methods SHOULD be registered in a document that isn't
> >> specific to an application or other use of HTTP, so that it's
> >> clear that they are not specific to that application or extension.
> > 

Or changed; if the spirit and intent is to think of extensibility.  That
language describes what I was already doing -- defining a new method as
a standalone document -- without alerting me to the pitfall of defining
my method as a reflection of my application, instead of as something of
general use.  Whereas Roy's last sentence caused a forehead-slap moment
for me, that I shouldn't even require an empty entity-body.

IOW, I just took my first shot at defining an HTTP method, and got it
completely wrong right out of the gate; a little guidance would have
made for a better start IMHO.

-Eric

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 19 Oct 2010 22:48:33 -0600

Mark Nottingham wrote:
> 
> >> New methods SHOULD explain how conditional request headers [ref]
> >> affect the response (if there is any effect).
> > 
> > Wow. I always thought they apply to any method; thus extensions
> > methods can't be special. Maybe this deserves a separate issue?
> 
> Well, we could say that 304 is specific to GET/HEAD, and all other
> methods use 412 for failed conditions, but that bit of glue seems to
> be missing ATM. 
> 

This sounds like good advice.  Perhaps I'm speaking out-of-turn because
I'm just getting started, but what I'm thinking of for IDLE would make
must-revalidate meaningless because it's implied by the method; a match
isn't 304 or 412, but instead results in an open connection.  While the
semantics of conditional-request headers remain unchanged, the results
vary from those of other methods, and should therefore be documented.

-Eric

---

From: Julian Reschke <[email-masked]>
Date: Wed, 20 Oct 2010 18:12:03 +0200

> However, I'm not lie-down-in-the-road about using them.

My preference is to reserve those keywords for instructions on 
implementations, not processes.

>> "Due to the parsing rules defined in ..., HTTP methods cannot prohibit the presence of a message-body on either the request or the response message (with the single exception of HEAD responses). D…

See <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1040>:

"Due to the parsing rules defined in &message-body;, definitions of HTTP
methods cannot prohibit the presence of a message-body on either the 
request or the response message (with responses to HEAD requests being 
the single exception). Definitions of new methods cannot change this 
rule, but they can specify that only zero-length bodies (as opposed to 
absent bodies) are allowed."

> It sounds like we should remove this sentence and open two new issues.

Hm.

Are we aware of any implementations that do PUT but do not handle 
If-Match? That would be surprising.

>> That's very vague (what does "other use" mean).

Maybe we should think of applications whether extensions? 
Authoring/Querying calendar data (CalDAV) would qualify as application, 
while things like namespace operations (WebDAV COPY/MOVE) or partial 
updates (PATCH) would be considered extensions.

> What this is saying is that the WebDAV methods should have been registered in a non-WebDAV specific document, to clarify that they can be used in other contexts.

If you take out the message definitions from WebDAV, little will be 
left. Really.

COPY/MOVE does require some understanding of collections. 
PROPFIND/PROPPATCH does require some understanding of metadata. 
LOCK/UNLOCK does require some understanding of locking semantics.

That's why I'm very nervous about making this a hard requirement; what 
it should be is a recommendation to make methods re-usable, and a 
reminder for the Expert Reviewers to check this.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 21 Oct 2010 09:13:16 +1100

On 21/10/2010, at 3:12 AM, Julian Reschke wrote:
> Are we aware of any implementations that do PUT but do not handle If-Match? That would be surprising.

First one I looked at (and used, many moons ago):
  <http://perso.ec-lyon.fr/lyonel.vincent/apache/mod_put.html>

AFAICT nothing requires implementation of If-Match et al when using PUT. Likewise, conditional requests are almost universally UNsupported for POST.

> Maybe we should think of applications whether extensions? Authoring/Querying calendar data (CalDAV) would qualify as application, while things like namespace operations (WebDAV COPY/MOVE) or partial updates (PATCH) would be considered extensions.

Right. I think that's what we're trying to encourage.

> If you take out the message definitions from WebDAV, little will be left. Really.
> 
> COPY/MOVE does require some understanding of collections. PROPFIND/PROPPATCH does require some understanding of metadata. LOCK/UNLOCK does require some understanding of locking semantics.

COPY and MOVE don't necessarily require understanding of collections, if you're not using WebDAV. PROPFIND and PROPATCH require understanding of a non-HTTP metadata system, which is probably why they feel so wrong. LOCK and UNLOCK don't necessarily require understand of locking semantics that are specific to WebDAV, do they?

> That's why I'm very nervous about making this a hard requirement; what it should be is a recommendation to make methods re-usable, and a reminder for the Expert Reviewers to check this.

I think that's the direction we're headed in here...

---

From: Julian Reschke <[email-masked]>
Date: Thu, 21 Oct 2010 18:59:48 +0200

> AFAICT nothing requires implementation of If-Match et al when using PUT. Likewise, conditional requests are almost universally UNsupported for POST.

<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.24>:

"The If-Match request-header field is used with a method to make it 
conditional...

...If none of the entity tags match, or if "*" is given and no current 
entity exists, the server MUST NOT perform the requested method, and 
MUST return a 412 (Precondition Failed) response...."

That doesn't sound optional to me, but I agree that it's likely many 
people get this wrong.

We need to clarify this, and maybe also think about how to discover this 
(do a test request with a known not-to-match Etag?).

> Right. I think that's what we're trying to encourage.

"HTTP methods SHOULD be registered in a document that isn't specific to 
an application (for instance, synchronization of browser settings), so 
that it's clear that they are of generic use."

?

 > COPY and MOVE don't necessarily require understanding of collections,
 > if you're not using WebDAV.

That's true. So COPY/MOVE could be defined for "plain" resources (only 
affecting the resource they are applied to), and WebDAV could describe 
how they word when applied to non-leaf nodes in the WebDAV collection 
model. (This is a good discussion to have!)

 > PROPFIND and PROPATCH require understanding of a non-HTTP metadata
 > system, which is probably why they feel so wrong.

Can you defined what a "HTTP metadata" system is?

We can argue that header fields are sufficient, in which case there'd be 
no need for PROPPATCH/PROPFIND, unless we want overwrite individual 
properties, and retrieve only a subset.

On the other hand, if header fields are *not* sufficient, then a 
different format is needed, and a way to link resources to their metadata.

The format used by WebDAV is DAV:prop (no media type though, sigh). The 
WebDAV way to like metadata to resources is PROPFIND (no link, no 
adressability through GET).

So yes, WebDAV got several aspects wrong, but *if* there's a method with 
PROPFIND-like semantics, it also needs a format for marshaling to be useful.

 > LOCK and UNLOCK don't necessarily require understand of locking
 > semantics that are specific to WebDAV, do they?

Why would you want to LOCK a resource if you don't know what effect it has?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 22 Oct 2010 13:06:34 +1100

Updated for conversation to date; I removed the language about conditional requests (will open a ticket shortly).

On 22/10/2010, at 3:59 AM, Julian Reschke wrote:

> "The If-Match request-header field is used with a method to make it conditional...
> 
> ...If none of the entity tags match, or if "*" is given and no current entity exists, the server MUST NOT perform the requested method, and MUST return a 412 (Precondition Failed) response...."
> 
> That doesn't sound optional to me, but I agree that it's likely many people get this wrong.
> 
> We need to clarify this, and maybe also think about how to discover this (do a test request with a known not-to-match Etag?).
> 
> "HTTP methods SHOULD be registered in a document that isn't specific to an application (for instance, synchronization of browser settings), so that it's clear that they are of generic use."
> 
> ?
> 
> 
> That's true. So COPY/MOVE could be defined for "plain" resources (only affecting the resource they are applied to), and WebDAV could describe how they word when applied to non-leaf nodes in the WebDAV collection model. (This is a good discussion to have!)
> 
> We can argue that header fields are sufficient, in which case there'd be no need for PROPPATCH/PROPFIND, unless we want overwrite individual properties, and retrieve only a subset.
> 
> On the other hand, if header fields are *not* sufficient, then a different format is needed, and a way to link resources to their metadata.
> 
> The format used by WebDAV is DAV:prop (no media type though, sigh). The WebDAV way to like metadata to resources is PROPFIND (no link, no adressability through GET).
> 
> So yes, WebDAV got several aspects wrong, but *if* there's a method with PROPFIND-like semantics, it also needs a format for marshaling to be useful.
>