Subject: Content negotiation for request bodies

From: "Brian Smith" <[email-masked]>
Date: Thu, 14 Feb 2008 14:58:04 -0800

One question I have frequently seen asked is "How do I know if the server supports compressed request bodies in POST and PUT requests?" Often the answer is "don't even try" or "try it, and if it fails, try without the compression." A better answer may be "Send the request, with the headers 'Content-Encoding: deflate' and 'Expect: 100-continue', and if you get a 100-continue response, send the compressed request body; otherwise send a new request without content-coding."

However, this is very dependent on the server correctly handling request Content-* headers, defined in Section 9.6 (PUT): "The recipient of the entity MUST NOT ignore any Content-* (e.g. Content-Range) headers that it does not understand or implement and MUST return a 501 (Not Implemented) response in such cases."

I would like to see the following changes made to the spec:

1. The requirement "The recipient of the entity MUST NOT ignore any Content-* headers" does not apply only to PUT, it also applies to POST any any other method where a request body may be sent. This requirement should be moved from Section 9.6 into another section that is not specific to PUT. This section should include Additionally, a note that says this is a common implementation error should be added to this section. 

2. A new worked example of the above (using "Expect: 100-continue" to send the server a compressed response if and only if the server responds with a "100 Continue") should be added to the section that describes the 100-continue mechanism. References to this new example should be added to the "Content Negotiation" section and to the "recipient...MUST NOT ignore any Content-* headers" section.

Thoughts?

- Brian

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 15 Feb 2008 12:43:11 +1300

> One question I have frequently seen asked is "How do I know if the server supports compressed request bodies in POST and PUT requests?" Often the answer is "don't even try" or "try it, and if it fa…

I don't see much difference between "try it, and if it fails..." and 
sending an Expects header.  Don't they amount to the same thing?  The 
problem with Expects, is it's only specified in HTTP 1.1, and every link 
in the chain must support it for it to work.  In the special case of 
Expects: 100-continue, this has grave problems with proxies that I've 
raised before, and a heuristic timeout waiting for a 100-continue which 
may never arrive causes further problems.

I guess the difference is in the amount of resource that may have 
already been sent before a failure comes back, and the failure code (501 
vs ...).  Balance that against the time you'd have to wait before giving 
up on getting a 100-continue back if the server were only HTTP/1.0, 
keeping in mind connecting to a proxy may be a lot quicker than the 
proxy connecting upstream.

A lot of work has gone into HTTP to try and cut down on round-trips - 
requests back to the server for whatever reason.  This has meant some 
compromises have been made especially when it comes to clients sending 
message bodies to servers. 

Simply because the first agent to send a message is the client (rather 
than the server), the server can know advertised client capabilities 
(i.e. Accept- headers) before choosing to send a resource that 
complies.  The client however cannot know what the server supports, 
because the server has not had an opportunity to advertise any 
capability.  There's no server welcome message.  So clients send message 
bodies blindly, whereas servers send message bodies with knowledge.

The result is that HTTP does not contemplate negotiating content 
transfers from clients to servers, only the other way around.  This is 
not such a problem for direct client-server comms, but when you 
introduce proxies in the chain, the problem is exascerbated.

Keeping in mind that there are many proxies potentially in a chain you 
don't know about.  Company intercepting proxies, ISP intercepting 
proxies, and a reverse proxy through to an origin server.  It's not 
uncommon to have about 3 proxies in a chain.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: David Morris <[email-masked]>
Date: Sat, 23 Feb 2008 18:04:55 -0800 (PST)

> bodies blindly, whereas servers send message bodies with knowledge.

Well it would be rare for a client to send a significantly sized request
body without a previous interaction with the server or one if its peers
in the web based application. It is not uncommon for a web page to direct
an upload or post transaction to another server, but the html page source
should be aware of target's capabilities. A browser should be able to
infer server capabilities from previous responses. If needed, we can
perhaps add some kind of hint of server preferences along with
implementation 'advice' which suggests that clients remember prior
experience with a server for a modest interval.

This should reduce error case to a few stupid servers which don't provide
proper clues.  Tring what is likely to succeed based on history with the
server or its application peers will actually have a very low error rate.

Even non-browser clients can be expected to have or acquire knowledge
as to a server's capabilities.

It would also help if HTML Forms, etc. had the ability to express server
preferences.

Dave Morris

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 24 Feb 2008 19:02:19 +1300

> server or its application peers will actually have a very low error rate.
I can't even begin to imagine the administrative nightmare of trying to 
provide proper hints in a redirect about capabilities of some other server.

Why put all that burden (support in software for the feature, admin 
burden to configure it, and ongoing maintenance burden) onto millions of 
people when you can solve it in the protocol?

As a gross generalisation, heuristic solutions to problems when a 
deterministic solution exists are generally more trouble and effort than 
they are worth.

Adrien

---

From: Henrik Nordström <[email-masked]>
Date: Sun, 24 Feb 2008 17:00:27 +0100

tor 2008-02-14 klockan 14:58 -0800 skrev Brian Smith:
> One question I have frequently seen asked is "How do I know if the
> server supports compressed request bodies in POST and PUT requests?"
> Often the answer is "don't even try" or "try it, and if it fails, try
> without the compression." A better answer may be "Send the request,
> with the headers 'Content-Encoding: deflate' and 'Expect:
> 100-continue', and if you get a 100-continue response, send the
> compressed request body; otherwise send a new request without
> content-coding."

Sounds like a reasonable approach to me, and is infact already in the
specifications in "Use of the 100 (Continue) Status". See also below.

Adding full Accept-* negotiation would be a bit overkill imho. It's a
pretty rare case, and in most cases the application protocol defines
what entity types is acceptable. Content-Encoding is part of the
HTTP/MIME entity type specification, not a "dynamic compression". I.e. a
PUT with Content-Encoding: deflate means store a deflated entity on the
server, not that the server should inflate the entity and store the
"original". The server obviosly MAY inflate the entity or perform
whatever processing on it it likes, but that's not the primary
expectations of HTTP for PUT.

A more valid question for HTTP specification is server ability to
advertise transfer-encoding support beyond chunked. The TE header is
today declared as a request header, but should probably be generalized
to be included in OPTIONS responses and 501 responses.

> However, this is very dependent on the server correctly handling
> request Content-* headers, defined in Section 9.6 (PUT): "The
> recipient of the entity MUST NOT ignore any Content-* (e.g.
> Content-Range) headers that it does not understand or implement and
> MUST return a 501 (Not Implemented) response in such cases."

Correct.

> 1. The requirement "The recipient of the entity MUST NOT ignore any
> Content-* headers" does not apply only to PUT, it also applies to POST
> any any other method where a request body may be sent. This
> requirement should be moved from Section 9.6 into another section that
> is not specific to PUT. This section should include Additionally, a
> note that says this is a common implementation error should be added
> to this section.

Agreed.

> 2. A new worked example of the above (using "Expect: 100-continue" to
> send the server a compressed response if and only if the server
> responds with a "100 Continue") should be added to the section that
> describes the 100-continue mechanism. References to this new example
> should be added to the "Content Negotiation" section and to the
> "recipient...MUST NOT ignore any Content-* headers" section.

If really needed I think a better approach would be to push a little
more for 100 Continue and the benefits of it in general. Pretty much the
exact same problem of backing off nicely applies to any "request not
satisfiable" condition when trying to send a request entity.

   - Malformed requests
   - Unacceptable request entityt types
   - Unacceptable transfer encoding
   - Authentication required
   - Not authorized
   - Server error
   - Redirection
   - etc etc..

Regards
Henrik

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 24 Feb 2008 10:24:17 -0800

Henrik Nordström wrote:
> Brian Smith wrote:
> > A better answer may be "Send the request, 
> Adding full Accept-* negotiation would be a bit overkill 
> imho. It's a pretty rare case, and in most cases the 
> application protocol defines what entity types is acceptable. 

Besides Content-Encoding, Content-Type is important, especially with
charset parameters. 

> Content-Encoding is part of the HTTP/MIME entity type 
> specification, not a "dynamic compression". I.e. a PUT with 
> Content-Encoding: deflate means store a deflated entity on 
> the server, not that the server should inflate the entity and 
> store the "original".

I disagree. It just means "I'm sending you something and it is
compressed." Nothing more or less.

> The server obviosly MAY inflate the 
> entity or perform whatever processing on it it likes

Right.

> but that's not the primary expectations of HTTP for PUT.

There aren't any agreed-upon expectations for how compressed request
bodies are processed; that is the problem.

> A more valid question for HTTP specification is server 
> ability to advertise transfer-encoding support beyond 
> chunked. The TE header is today declared as a request header, 
> but should probably be generalized to be included in OPTIONS 
> responses and 501 responses.

A TE in a response to OPTIONS implies that the listed encodings are
valid for any method. The "Expect: 100-continue" can handle this on a
per-method (actually per-request) basis. I think both mechanisms are
useful. If you allow TE in OPTIONS responses, you might as well allow
Accept-* for OPTIONS too.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 24 Feb 2008 11:02:00 -0800

Adrien de Croy wrote:
> David Morris wrote:
> > This should reduce error case to a few stupid servers which don't 
> > provide proper clues.  Tring what is likely to succeed based on 
> > history with the server or its application peers will 
> > actually have a very low error rate.
>
> I can't even begin to imagine the administrative nightmare of 
> trying to provide proper hints in a redirect about 
> capabilities of some other server.

Then don't provide any hints. That is the current state of things
anyway.

> Why put all that burden (support in software for the feature, 
> admin burden to configure it, and ongoing maintenance burden) 
> onto millions of people when you can solve it in the protocol?
> 
> As a gross generalisation, heuristic solutions to problems 
> when a deterministic solution exists are generally more 
> trouble and effort than they are worth.

I agree, but what is the deterministic solution?

As long as a client's assumptions are pessimistic (ex: "compressed
request bodies didn't work when posting to http://example.org/foo the
first time I tried today, so I'm not even going to try to compress any
requests when posting to that URL for the rest of the day"), then no
harm is done.

- Brian

---

From: David Morris <[email-masked]>
Date: Sun, 24 Feb 2008 17:09:52 -0800 (PST)

> provide proper hints in a redirect about capabilities of some other server.

Not really an administrative problem ... there is no way with a commonly
used modern browser to initiate a large request without the existance of
an HTML form (or active content included in HTML). Such forms are
generally going to target the same administrative space as the
server sending the original form. If the client has no information as to
what form of a request is acceptable, it can try the safe but not optimum
approach or try something more optimum and be prepared to recover from an
error. Remembering what happened on the current request will provide
optimization opportunities for the next.

In terms of non-browser clients, it seems very likely that they will have
apriori knowledge of the unique servers they will be interacting with. If
not, something like OPTIONS or a special GET can be used to acquire the
needed list of choices.

Dave Morris

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 25 Feb 2008 06:54:02 +0000

David Morris wrote:
> server sending the original form.

In practice this is already done to some extent.  Form HTML implicitly
encodes knowledge of the server receiving the subsequent request.
Forms handling international text may have browser-specific code to
manage the character encoding that's used to send the subseqent request -
whether it's POSTed or URL-encoded with GET.

-- Jamie