Subject: Re: estimated Content-Length with chunked encoding

From: Adrien de Croy <[email-masked]>
Date: Fri, 14 Nov 2008 13:21:13 +1300

this brings up another issue, since chunked requests was the suggested 
solution to the problem of coping with authentication (possibly of 
proxies and/or origin servers) for POST and/or PUT with large entities.

It's easy for a server to make the decision whether to use chunking or 
not, because it has the HTTP version of the connected client from the 
request.

In the case of a client sending however, unless it has prior knowledge, 
it doesn't know whether the server supports chunking or not.

An HTTP1.0 server receiving a chunked request could break in a very bad 
way, since it will most likely

a) ignore the Transfer-encoding field
b) treat the chunk headers and trailers as part of the content.

leading to breaking the content on the server, and the client may have 
no way of undoing this.

Therefore before sending any chunked request, the client would need to 
establish a-priori that the request path was capable of successfully 
processing a chunked request.

It's not necessarily trivial to do this reliably.

Daniel Stenberg wrote:
>
> On Tue, 28 Oct 2008, Henrik Nordstrom wrote:
>
>>> Interesting enough, the MacOS X WebDAV Client already seems to use
>>> something like that:
>>>
>>>> C-68-#000000 -> [PUT /content/dam/STROMBERG_2_2.mp4 HTTP/1.1 ]
>>>> C-68-#000280 -> [Transfer-Encoding: Chunked ]
>>
>> Oh, cool. A client daring to use chunked encoding in requests!
>
> Is that really so unusual? The "HTTP Implementations" spreadsheet 
> lists at least 4 clients supporting it, but it seems the info is 
> lacking for most of them:
>
> http://spreadsheets.google.com/pub?key=pAZLaupBqGqXrtsaexTjPYg&gid=0
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 14 Nov 2008 02:12:20 +0000

> It's not necessarily trivial to do this reliably.

What about sending "Expect: 100-continue" in the request headers, and
waiting for a "100 Continue" response.  If you get one, you _ought_ to
be able to assume it's a chunked-request-capable HTTP/1.1 server or
proxy, and if you don't, you time out, abort that connection (because
you don't know if it will interpret Transfer-Encoding), and try again
with a non-chunked request.

Does that work in principle, disregarding broken implementations?

Does it work in practice?

-- Jamie

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 14 Nov 2008 07:46:27 +0100 (CET)

On Fri, 14 Nov 2008, Jamie Lokier wrote:

> waiting for a "100 Continue" response.  If you get one, you _ought_ to be 
> able to assume it's a chunked-request-capable HTTP/1.1 server or proxy, and 
> if you don't, you time out, abort that connection (because you don't know if 
> it will interpret Transfer-Encoding), and try again with a non-chunked 
> request.
>

I think this is a case that would work if things worked the way we understand 
them and read the RFC, but in practise I believe the 100-continue support in 
the wild is not implemented this cleverly. I think a vast amount of 
100-continue responses are just not doing any checks at all but simply respond 
OK-continue without consideration. And then there's the opposite - servers 
that don't like 100-continue at all but would support a chunked-request.

I say "think" here because this is just my gut feeling, I haven't actually 
tested the theory.

-- 

  / daniel.haxx.se

---

From: "Robert Brewer" <[email-masked]>
Date: Thu, 13 Nov 2008 23:34:33 -0800

Daniel Stenberg wrote:
> I think this is a case that would work if things worked the way
> we understand them and read the RFC, but in practise I believe
> the 100-continue support in the wild is not implemented this
> cleverly. I think a vast amount of 100-continue responses are
> just not doing any checks at all but simply respond OK-continue
> without consideration. And then there's the opposite - servers
> I say "think" here because this is just my gut feeling, I haven't
> actually tested the theory.

Of the half-dozen servers with data in Mark's http-implementations
Google doc, only one said 'no' in the 'chunked bodies' column.

While we're on the subject, why are the highest-profile servers, like
Apache, lighttpd, and nginx, still unrepresented? Doesn't *anyone* know
how they work anymore?

Robert Brewer
[email-masked]

---

From: [email-masked]
Date: Fri, 14 Nov 2008 08:57:12 +0100 (CET)

> I say "think" here because this is just my gut feeling, I haven't actually

In my experience, chunked requests are nowadays much better supported than
proper 100-continue behaviour. So it will probably be safe to deduce the
former from the latter, but one misses out on some servers.

The reason for this is that chunked requests can be implemented often in
some quite isolated server code while proper 100-continue needs to reach
up through all API layers (until it crashes into the servlet api for
example which does not support it).

I say "proper" because some server just send out 100-continue whenever
they see an Expect header without consulting authentication or even the
application layer itself.

Cheers, Stefan

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 14 Nov 2008 13:31:39 +0000

> application layer itself.

Servers sending "100 Continue" without further checks isn't actually
_wrong_, it's just sub-optimal performance.

The only "missed server" opportunities are those which _don't_ send
"100 Continue", which both Stefan and Daniel think is not the likely
issue - the likely issue being they they _do_ send without checking.

If that's the case, the remaining problem is servers which are too
slow to respond.  You could wait a bit longer before sending, if you
have such a large request body that chunking is worth doing, or if
prior behaviour suggests the server normally sends "100 Continue";
it's not perfect but a heuristic.

I think a bigger issue would be proxies accepting chunked requests,
and not properly converting them to unchunked when the upstream server
does not accept chunked.  Good proxies would convert, of course.

-- Jamie