Subject: RE: Content negotiation for request bodies

From: "Brian Smith" <[email-masked]>
Date: Sun, 24 Feb 2008 11:02:24 -0800

Adrien de Croy wrote:
> Brian Smith wrote:
> > One question I have frequently seen asked is "How do I know if the 
> > server supports compressed request bodies in POST and PUT requests?" 
> > Often the answer is "don't even try" or "try it, and if it fails, 
> > try without the compression." A better answer may be "Send the 
> > request, with the headers
> > 'Content-Encoding: deflate' and 'Expect: 100-continue', and if you 
> > get a 100-continue response, send the compressed request body; 
> > otherwise send a new request without content-coding."

(quoted out of order)

> The problem with Expects, is it's only specified in HTTP 1.1, and 
> every link in the chain must support it for it to work.  In the 
> special case of
> Expects: 100-continue, this has grave problems with proxies that I've 
> raised before, and a heuristic timeout waiting for a 100-continue 
> which may never arrive causes further problems.

There is a lot of software generating "Expect: 100-continue" now. In particular, the .NET client libraries seem to add this by default. Curl uses it by default. Apache supports it well. If there is any problem, it would seem to lie in with proxies. Is "Expect: 100-continue" really that problematic for commonly deployed proxies?

> I don't see much difference between "try it, and if it fails..." and 
> sending an Expects header.  Don't they amount to the same thing?

> I guess the difference is in the amount of resource that may have 
> already been sent before a failure comes back

The other difference is that the client can implement a policy "if the server is too dumb to understand Expect: 100-continue, then it is also probably too dumb to understand Content-Encoding or other advanced features I would like to use in this request, so let's back off and use something simpler." In particular, the client might assume that the lack of a 100-continue response means that there is an ancient (HTTP 1.0) proxy or server involved somewhere.

> Balance that against the time
> you'd have to wait before giving up on getting a 100-continue back if 
> the server were only HTTP/1.0, keeping in mind connecting to a proxy 
> may be a lot quicker than the proxy connecting upstream.
>
> A lot of work has gone into HTTP to try and cut down on round-trips - 
> requests back to the server for whatever reason.  This has meant some 
> compromises have been made especially when it comes to clients sending 
> message bodies to servers.

A smart client can use the information it learns from an initial unsuccessful request in future, similar requests. For example, if a deflated request fails, but the same request succeeds without the content-encoding, then the client can avoid using deflated request bodies and/or "Expect: 100-continue" in similar requests for some time period. 

> The result is that HTTP does not contemplate negotiating content 
> transfers from clients to servers, only the other way around.  This is 
> not such a problem for direct client-server comms, but when you 
> introduce proxies in the chain, the problem is exascerbated.
> 
> Keeping in mind that there are many proxies potentially in a chain you 
> don't know about.  Company intercepting proxies, ISP intercepting 
> proxies, and a reverse proxy through to an origin server.  It's not 
> uncommon to have about 3 proxies in a chain.

There are plenty of places in the protocol where we can help with negotiation. For example, when the server responds with a "415 Unsupported Media Type", right now the client can only keep trying to resend different types (e.g. first image/jpg, then image/png, then image/gif; text/html;charset=TIS-620, then text/html;charset=UTF-8). But, if the server could add an "Accept: image/gif" or "Accept: text/html;charset=UTF-8" to its 4xx responses, then we can avoid some roundtrips.

If the client is a little clever it can avoid such performance problems by remembering the server's limitations, and skipping right to the content type/charset/content-encoding that the server accepted in previous, similar requests.

- Brian

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 25 Feb 2008 06:57:10 +0000

Brian Smith wrote:
> The other difference is that the client can implement a policy "if
> the server is too dumb to understand Expect: 100-continue, then it is
> also probably too dumb to understand Content-Encoding or other
> advanced features I would like to use in this request, so let's back
> off and use something simpler." In particular, the client might assume
> that the lack of a 100-continue response means that there is an
> ancient (HTTP 1.0) proxy or server involved somewhere.

Is "Expect: deflate" of any use here?

I'm thinking if you send a compressed request body (for example), in
the hope that a server which doesn't understand it will reject it
safely, there is a possibility of unintended side effects as the
server might interpret the body in a way that's not intended.

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Feb 2008 22:38:28 +0100

Brian Smith wrote:
> ...

One problem is that server framework implementations (such as servlet 
implementations) frequently answer with "100 Continue", without actually 
doing what RCF2616 says they should do:

"The purpose of the 100 (Continue) status (see Section 10.1.1) is to 
allow a client that is sending a request message with a request body to 
determine if the origin server is willing to accept the request (based 
on the request headers) before the client sends the request body. In 
some cases, it might either be inappropriate or highly inefficient for 
the client to send the body if the server will reject the message 
without looking at the body." -- 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.8.2.3>

Today servlet containers just say "100 Continue" without actually 
consulting the servlet. That defeats part of what it's for.

There have been proposals around for many years to improve this, but as 
far as I know, nothing has happened so far.

BR, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 27 Feb 2008 12:16:44 +1300

Just floating an idea.

GET has its good friend HEAD, which allows you to do the equivalent of a 
GET command without transferring any body.  In this case you can test 
for all sorts of things, including whether you will need to auth, 
whether the request might be accepted etc etc.

No other methods have such a helper method.

When discussing issues the other day, I came up with the concept of a 
potential new method that could address many of these issues.  For lack 
of a better name, I've called the method PROBE.

e.g.

PROBE someURI HTTP/1.1
Probe-Headers: Method=POST; Content-Length=1000000000; Content-Type="..."
... other headers that aren't being tested.
Content-Length:0

This defines a message with no body, but which can be used to test a 
request that the client really would like to make, but be able to make 
efficiently.

A server understanding PROBE could use the Probe-Headers fields to 
determine if and under what conditions, such a request would be 
processed (or not), and respond accordingly.  Any auth at a proxy or 
server could be set up in the realm of the POROBE command before the 
actual request was made on the same connection.  Therefore you can auth 
your way through any number of intermediaries before you need to 
transfer any request body, and check if the request will be allowed 
anyway, all without breaking any semantics.

HTTP proxies that pass unknown methods should already be able to forward 
this as well.

Adrien

Julian Reschke wrote:
>>> Expects: 100-continue, this has grave problems with proxies that 
>>> I've raised before, and a heuristic timeout waiting for a 
>>> 100-continue which may never arrive causes further problems.
>>
>> There is a lot of software generating "Expect: 100-continue" now. In 
>> particular, the .NET client libraries seem to add this by default. 
>> Curl uses it by default. Apache supports it well. If there is any 
>> problem, it would seem to lie in with proxies. Is "Expect: 
>> 100-continue" really that problematic for commonly deployed proxies?
>> ...
>
> implementations) frequently answer with "100 Continue", without 
> actually doing what RCF2616 says they should do:
>
> allow a client that is sending a request message with a request body 
> to determine if the origin server is willing to accept the request 
> (based on the request headers) before the client sends the request 
> body. In some cases, it might either be inappropriate or highly 
> inefficient for the client to send the body if the server will reject 
> the message without looking at the body." -- 
> There have been proposals around for many years to improve this, but 
> as far as I know, nothing has happened so far.
>
> BR, Julian
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 26 Feb 2008 16:26:26 -0800

Adrien de Croy wrote:
> When discussing issues the other day, I came up with the 
> concept of a potential new method that could address many of 
> these issues.  For lack of a better name, I've called the 
> method PROBE.
> 
> e.g.
> 
> PROBE someURI HTTP/1.1

The idea seems reasonable. It is basically implementing "Expect: 100-continue" in a way that has no side effects when the server doesn't understand it. But, you can't cram all the headers into a "Probe-Headers" header because of implementations that limit the size of headers. It would make more sense to have the headers included in the body, using a Content-Type that is a cross between message/http and message/external-body. 

Many servers/proxies are likely to reject PROBE even if they would accept the request it describes. So, there has to be a clear way of distinguishing between a failed PROBE, and a successful PROBE that indicates that the described request would fail.

The main problem is that, if the PROBE indicates that the described request would fail, then it still doesn't tell you what changes you would need to make to get it to succeed.

The more I look into this, the more I agree with Henrik Nordström when he said that it makes more sense to leave this kup to the application protocol than to encourage users to depend on 100-continue (especially due to the fact that 100-continue is so poorly implemented, as Julian pointed out). I no longer think it makes sense for the server to advocate the use of "Expect: 100-continue" for anything, really.

One thing I would like to avoid is stuff like "Accept-Patch" from the PATCH draft. This is a header that is returned in a response (usually OPTIONS) that lists content types that the server accepts in PATCH requests. Continuing with that leads to Accept-Post, Accept-Put, Accept-Encoding-Patch, Accept-Encoding-Put--basically N*M new headers, where N is the number of methods and M is the number of request/entity headers to use for negotiation. 

I do agree that Accept-Patch is definitely useful, but having it without an analogous mechanism for the fundemantal methods like POST and PUT doesn't make sense.

- Brian