Subject: 100 Continue and Expects

From: Adrien de Croy <[email-masked]>
Date: Sat, 18 Jul 2009 16:58:01 +1200

Hi All, I'm just reading through p1-messaging.

there are a few problems with S 7.2.3

First para regarding the purpose of the status is misleading.  It 
implies one can avoid sending a message body.  It doesn't go into the 
grisly details of how one would actually avoid sending the message body 
and the associated problems, which is one of

a) try your POST with a Content-Length: 0 like IE does (which causes 
other problems)
b) use chunking for your request bodies and prematurely abort the send 
with a 0 chunk.

Both of these methods have problems.

If you've made the awful mistate of sending a Content-Length with your 
POST, you're doomed to disconnect and try again with one of the other 
methods or send the whole message.  If you're trying to use 
connection-oriented auth (like many millions of people are every day) 
then you're basically hosed, or have to use option a or just bite the 
bullet and send the whole message body each time (which most browsers 
other than IE do).  I've already complained about the 0 length POST, 
which breaks conditional auth (where auth may not always be challenged 
for by a proxy / site).  We've already had to code a nasty hack into 
WinGate to cope with this (if method=POST and Content-Length = 0 and 
haven't authed, then send auth challenge regardless of policy).  And 
about the use of chunked requests (which discovers many broken 
proxies/servers).  I think the best solution from an engineering 
standpoint is an extension of HEAD to allow it to test other methods 
rather than just GET, or another method which does the same thing (e.g. 
a TEST method).  This is a major version change to the protocol though. 

Anyway, in terms of a proposal for the section, I'd like to see some 
mention of the options / problems, because they are far from 
obvious/intuitive.

Under "Requirements for HTTP/1.1 origin servers"

clause 3 contradicts the MUST requirement in clause 1.  e.g.

   o  An origin server MAY omit a 100 (Continue) response if it has
      already received some or all of the request body for the
      corresponding request.

conflicts with

   o  Upon receiving a request which includes an Expect request-header
      field with the "100-continue" expectation, an origin server MUST
      either respond with 100 (Continue) status and continue to read
      from the input stream, or respond with a final status code.

the way I read the first clause is that the server must immediately 
reply either with 100 continue or the final status (e.g an auth 
challenge etc).  That's the whole point of 100 continue.  However the 
3rd clause implies if you've already read something it's ok to wait 
around for the final status.  Propose striking clause 3, or adding "in 
which case it must immediately send the final status", or limit the 
applicability of this clause to the case where there was no Expects: 
100-continue sent.

Under "Requirements for HTTP/1.1 proxies"

o Proxies SHOULD maintain a cache recording the HTTP version numbers 
received from recently-referenced next-hop servers.

this is problematic.  I've seen sites that respond with varying HTTP 
versions in the responses (even with same server).  Presumably due to 
versions being set in different scripts.  It's impossible to cache 
version vs site in these cases.  Sure, can cache the rest, but the 
overhead of keeping a cache for this purpose seems overly heavy.

I know that the issue of Expects has been thrashed on this list 
already.  I understand the theory of why such a mechanism is required, 
but in the case of 100 continue (the only one specified) it's so 
problematic I'd be very surprised if any client ever implements it.

Does anyone know of any (IMO suicidal) client that uses Expects? 

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 19 Jul 2009 10:16:49 +1200

Henrik Nordstrom wrote:
> lör 2009-07-18 klockan 16:58 +1200 skrev Adrien de Croy:
>
>   
> Not in this context.
I guess I'm mixing auth with 100-continue and expects.  The reason for 
this is that a frequent reason why it may be inappropriate to send the 
full body is because of auth.

So in my mind they are strongly linked.  In fact apart from redirects, 
auth is the only case I can think of.

> c) Closing the connection when it's determined that sending the message
> body is undesireable.
>
>   
Sure, but that's not an option for connection-oriented auth.

>> then you're basically hosed
>>     
>
> Which is why you should not use connection-oriented auth with requests
> having Content-Length. I don't see how that's related to 100 Continue or
> Expects. Connection-oriented auth is plain broken in the message
> oriented context of HTTP.
>
>   
easier said than done.  when people want to use NTLM, try getting them 
to use something else.

Obviously digest is the other option.

> True. But the intentions is quite obvious if one reads both.
>
>   
>> challenge etc). 
> Not my reading.
>>  That's the whole point of 100 continue.  However the 
>> around for the final status.
>>     
>
> iff the client has aready sent you some part of the request body, in
> which case the server knows the client has already left it's expectation
> of a 100 Continue response.
>
>   
OK, that makes sense thanks.  But does that mean the client already gave 
up waiting?

>> versions in the responses (even with same server).
>>     
>
> Which is broken servers.
>
>   
>> Presumably due to versions being set in different scripts.
>>     
>
> and servers not implementing HTTP for those scripts..
>   
OK.  Some server architectures make this difficult for them, I imagine 
it's pretty wide-spread, and depends on the scripts.

A server needs to parse and fix script output to fix this.

> There is quite many using it today. And many of those isn't prepared to
> deal with expectation failures.. (learnt the hard way when implementing
> Expect...)
>   
that puts the proxy in a difficult position.  Is it allowed to send the 
100 continue itself?

Or are there other expectations now than only 100-continue which these 
clients rely on?

I can't really see the point of relying on 100 continue.

> On a side note I have also been thinking about this version cache topic
> lately in another topic. Will post about that later today..
>   
sure, the other obvious application (where you need apriori knowledge of 
the server protocol version support)
that comes to mind is chunked request bodies.

Regards

Adrien

> Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 18 Jul 2009 22:56:22 +0200

lör 2009-07-18 klockan 16:58 +1200 skrev Adrien de Croy:

> implies one can avoid sending a message body.

Yes, it means indirectly from the requirement of monitoring the
connection for errors etc.

>   It doesn't go into the 
> and the associated problems

Because this is specified elsewhere.

Not in this context.

Not my reading.

Which is broken servers.

> It's impossible to cache  version vs site in these cases.

No, but the outcome won't be the most favorable for those servers, but
won't break things in a general sense.

> Sure, can cache the rest, but the 

Partially agreed. But in case of Expect: 100-continue it's just an
optimization to fail the expectation early if it's known within
reasonable doubt that the expectation will fail. Additionally nothing in
the specs says what constitutes a "recently-referenced next-hop" and is
left as an implementation detail.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 19 Jul 2009 02:03:28 +0200

> auth is the only case I can think of.

connection-oriented auth is the only when request bodies with
Content-Length is a significant problem, as it can not abort the request
and continue the auth process.

For all other cases it's always possible to abort the request and
continue on another connection. Redirections, auth, errors, whatever.

If chunked encoding is used the request can always be aborted mid-flight
without closing the connection and is not a problem.

> Obviously digest is the other option.

Digest is surely one option.

Another is having another secure message oriented HTTP auth scheme
deployed. Sure it will take some time, but certainly not impossible.

A suggestion for such auth scheme if someone wan't to work on fixing
HTTP authentication is something session key oriented with pluggable
auth method, allowing for NTLM/Kerberos/plain/whatever when initiating
the session. I.e. in some way similar to Digest MD5-sess but somewhat
decoupled from the actual authentication.. (and also using more secure
hashing).

> OK, that makes sense thanks.  But does that mean the client already gave 

Yes, as it otherwise is not allowed to send any of the request body.

> it's pretty wide-spread, and depends on the scripts.

I imagine those are getting more and more rare actually. A server which
runs scripts without parsing the response and framing it as an HTTP
response is easily a target for response-splitting attacks, with it's
associated cache poisoning effects.

> A server needs to parse and fix script output to fix this.

Yes.

And a server using CGI always does as the CGI script output is not HTTP.

> that puts the proxy in a difficult position.  Is it allowed to send the 

Not really no.

> Or are there other expectations now than only 100-continue which these 

Haven't seen any.

> I can't really see the point of relying on 100 continue.

Funnily those clients I encountered doesn't... as they also implement
the timeout as well. What they failed on was to retry the request
without the expectation when getting a 417 Expectation Failed
response...

Relying on seeing 100 Continue is quite fine if you know the next hop
and origin is both HTTP/1.1, you can then use a fairly large timeout
only to catch the occasional unexpected forwarding path change which
happens to route the request via a HTTP/1.0 proxy where it did not
before...

Waiting for 100 Continue has benefits in
  - Less bandwidth waste
  - Errors actually reaching the client and not getting lost in a TCP
Reset.
  - Less demanding on servers as well as they don't need to linger so
much on "aborted" requests after sending an error or redirect status.

Regards
Henrik

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 20 Jul 2009 23:26:13 +0200 (CEST)

> Does anyone know of any (IMO suicidal) client that uses Expects?

The "http implementations" spreadsheet[*] lists three, and since they're all 
client libraries/frameworks one would assume that there are more than three 
actual clients. Most clients don't have this field filled-in so I expect there 
are actually more with support for this.

[*] = http://spreadsheets.google.com/pub?key=pAZLaupBqGqXrtsaexTjPYg&gid=0

-- 

  / daniel.haxx.se

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Jul 2009 10:09:51 +1200

thanks - that's very interesting, esp the pipelining column under 
intermediaries.  Not a single implementation on that list.  Admittedly 
there are a few gaps on the list, but it reinforces my earlier point 
about the difficulty of implementing pipelining through a proxy.

Regards

Adrien

Daniel Stenberg wrote:
> The "http implementations" spreadsheet[*] lists three, and since 
> they're all client libraries/frameworks one would assume that there 
> are more than three actual clients. Most clients don't have this field 
> filled-in so I expect there are actually more with support for this.
>
> [*] = 
> http://spreadsheets.google.com/pub?key=pAZLaupBqGqXrtsaexTjPYg&gid=0
>

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Jul 2009 10:18:49 +1200

ignore that - maybe I should have read the readme first...

Adrien de Croy wrote:
>> are more than three actual clients. Most clients don't have this 
>> field filled-in so I expect there are actually more with support for 
>> this.
>>
>> [*] =