Subject: p2: Expect: 100-continue and "final" status codes

From: Mark Nottingham <[email-masked]>
Date: Tue, 23 Apr 2013 17:22:37 +1000

p2 5.1.1.1 explains the semantics of 100-continue: "If the origin server responds with a final status code, it must not have performed the request method and may either close the connection or continue to read and discard the rest of the request."  

In my (admittedly quick) testing, pretty much nobody does this, at least by default; i.e., if I send a GET to a server with Expect: 100-continue, it's going to give me a 200 or 30x, not a 417. Sure, they might send 417 for a request with a body, but as written pretty much no one is conformant.

One thing we could do would be to only place requirements upon proxies and servers when Expect: 100-continue is on a request with a body. 

Stepping back, though, I have to wonder if it's reasonable to only allow 100 (Continue) or, effectively, an error (since the request can't be "applied") in the presence of Expect: 100-continue. I've seen many implementations that purposefully ignore Expect: 100-continue and send back 200 (OK) responses to avoid the interop problems that expect/continue brings.

OTOH if we do maintain the notion that a final response to an Expect: 100-continue request needs to NOT be applied on the server, we should use more specific terminology (i.e., say that it needs to be a 4xx or 5xx response status, not just a "final response.").

Thoughts?

--
Mark Nottingham   http://www.mnot.net/

---

From: Ken Murchison <[email-masked]>
Date: Tue, 23 Apr 2013 09:39:29 -0400

On Tue, 23 Apr 2013 17:22:37 +1000, Mark Nottingham wrote:
> p2 5.1.1.1 explains the semantics of 100-continue: "If the origin 
> server responds with a final status code, it must not have performed 
> the request method and may either close the connection or continue to 
> read and discard the rest of the request."  
>
> In my (admittedly quick) testing, pretty much nobody does this, at 
> least by default; i.e., if I send a GET to a server with Expect: 
> 100-continue, it's going to give me a 200 or 30x, not a 417. Sure, 
> they might send 417 for a request with a body, but as written pretty 
> much no one is conformant.
This bring up a question:  For those methods where a body isn't 
currently defined and/or typically isn't expected, is a server supposed 
to distinguish between cases when the request includes a body vs the 
request includes Expect:100-continue?  Are the correct responses 415 and 
417 respectively?

And what about TRACE?  Is 400 the correct response in both cases?

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

---

From: James M Snell <[email-masked]>
Date: Tue, 23 Apr 2013 08:50:44 -0700

For 1.x, I would say just leave this alone as is. I doubt there's
anything we can do that'll bring the non-conforming implementations
into line and it's not a critical enough of an issue to deserve
pushing on too much.

For 2.x, We ought to update the definition of Expect so that, for any
request method, an 100-Continue expectation MUST result in either a
RST_STREAM (if unsatisfied) or a 100 (Continue) as you suggest.

That said, I would also prefer that we discourage any further
definition of new expectations that dictate specific status response
codes from the server. The server ought to be free to return any
status it determines to be appropriate, and we now have the Prefer
header for allowing the client to communicate it's preferences while
still leaving the decision in the servers hands.

- James

---

From: Zhong Yu <[email-masked]>
Date: Tue, 23 Apr 2013 12:42:02 -0500

#Expect 100-continue without a body

If a client sends a request without a body yet with a "Expect:
100-continue" header, the request ought to be considered a bad request,
because of a previous requirement:

   o  A client MUST NOT send an Expect header field with the "100-
      continue" expectation if it does not intend to send a payload
      body.

However a server may not be able to detect all bad requests. If it sees a
request without a body, it may simply assume that there's no 100-continue
expectation.

The spec may add a leeway for server

   o  Upon receiving a request with "100-continue" expectation yet without
a message body (i.e. both Transfer-Encoding and Content-Length are
missing), an origin server SHOULD either respond with 400 (Bad Request), or
ignore the expectation.

#Performing request method without reading request body

I don't understand why the origin server must not perform the request
method if it opts not to read the request body. Even though it's probably
the right thing to do in almost all cases, do we have to make it an
absolute requirement?

#Managing connection

The last requirement for the origin server talks about requests *without* a
"100-continue" expectation. Therefore it should not be in this section. The
subject is covered very well in p1#6.6 already, I think we can simply
delete it from this section.

On the other hand, the spec does not address connection management
adequately in the most important use case of the section. It currently says
(paraphrasing)

    upon receiving a request that includes the 100-continue expectation, if
the origin server responds with a final status code instead of 100
(Continue), after sending the response, it may either close the connection
or continue to read and discard the rest of the request.

I think we can give better advice than that. If a server responds with a
final status code instead of 100 (Continue)

1. The response must be the last response on the connection. The response
should contain "Connection: close" header. After the response is written,
the server must initiate a lingering close of the connection (p1#6.6).

2. If the client receives a final status code instead of 100 (Continue), it
should stop sending request body if it is doing so; it must close the
connection after the response is received.

This is to avoid the RST problem if the client decides to start to write
request body before receiving any response from the server.

Zhong Yu

On Tue, Apr 23, 2013 at 2:22 AM, Mark Nottingham <[email-masked]> wrote:

> p2 5.1.1.1 explains the semantics of 100-continue: "If the origin server
> responds with a final status code, it must not have performed the request
> method and may either close the connection or continue to read and discard
> the rest of the request."
>
> In my (admittedly quick) testing, pretty much nobody does this, at least
> by default; i.e., if I send a GET to a server with Expect: 100-continue,
> it's going to give me a 200 or 30x, not a 417. Sure, they might send 417
> for a request with a body, but as written pretty much no one is conformant.
>
> One thing we could do would be to only place requirements upon proxies and
> servers when Expect: 100-continue is on a request with a body.
>
> Stepping back, though, I have to wonder if it's reasonable to only allow
> 100 (Continue) or, effectively, an error (since the request can't be
> "applied") in the presence of Expect: 100-continue. I've seen many
> implementations that purposefully ignore Expect: 100-continue and send back
> 200 (OK) responses to avoid the interop problems that expect/continue
> brings.
>
> OTOH if we do maintain the notion that a final response to an Expect:
> 100-continue request needs to NOT be applied on the server, we should use
> more specific terminology (i.e., say that it needs to be a 4xx or 5xx
> response status, not just a "final response.").
>
> Thoughts?
>
> --

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Apr 2013 12:19:59 +1000

On 24/04/2013, at 3:42 AM, Zhong Yu <[email-masked]> wrote:

> If a client sends a request without a body yet with a "Expect: 100-continue" header, the request ought to be considered a bad request, because of a previous requirement:
> 
> However a server may not be able to detect all bad requests. If it sees a request without a body, it may simply assume that there's no 100-continue expectation.
> 
>    o  Upon receiving a request with "100-continue" expectation yet without a message body (i.e. both Transfer-Encoding and Content-Length are missing), an origin server SHOULD either respond with 400 (Bad Request), or ignore the expectation.

+1

> I don't understand why the origin server must not perform the request method if it opts not to read the request body. Even though it's probably the right thing to do in almost all cases, do we have to make it an absolute requirement? 

See below about recovery.

> #Managing connection
> 
> The last requirement for the origin server talks about requests *without* a "100-continue" expectation. Therefore it should not be in this section. The subject is covered very well in p1#6.6 already, I think we can simply delete it from this section.

+0.5 - I tend to agree, but don't mind too much if it stays.

> On the other hand, the spec does not address connection management adequately in the most important use case of the section. It currently says (paraphrasing)
> 
>     upon receiving a request that includes the 100-continue expectation, if the origin server responds with a final status code instead of 100 (Continue), after sending the response, it may either close the connection or continue to read and discard the rest of the request.
> 
> I think we can give better advice than that. If a server responds with a final status code instead of 100 (Continue)
> 
> 1. The response must be the last response on the connection. The response should contain "Connection: close" header. After the response is written, the server must initiate a lingering close of the connection (p1#6.6).

That seems too restrictive; as long as the server reads the rest of the request properly (discarding it), it should be able to recover and reuse the connection.

> 2. If the client receives a final status code instead of 100 (Continue), it should stop sending request body if it is doing so; it must close the connection after the response is received.

Well, it has a choice; it can either close the connection, or continue sending and recover the connection. If it only has a few more bytes to write, and some requests queued, that might be preferable.

OTOH, given the general level of interop around Expect/Continue, maybe not.

> This is to avoid the RST problem if the client decides to start to write request body before receiving any response from the server.

I agree that the current language (esp. the use of MAY) is too ambiguous, and doesn't highlight why we're taking this approach (because the client might decide to start sending the body).

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 24 Apr 2013 14:41:20 +1200

> That seems too restrictive; as long as the server reads the rest of the request properly (discarding it), it should be able to recover and reuse the connection.

The problem comes with intermediaries. How are they to know the bytes 
following were the original advertised payload or not? the status from 
server has no guarantee of arriving after the client payload starts 
arriving.
The only way to guarantee safety on the connection is to close it or 
always send payload.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Apr 2013 13:32:16 +1000

On 24/04/2013, at 12:41 PM, Amos Jeffries <[email-masked]> wrote:
> The problem comes with intermediaries. How are they to know the bytes following were the original advertised payload or not? the status from server has no guarantee of arriving after the client payload starts arriving.
> The only way to guarantee safety on the connection is to close it or always send payload.

True.

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 24 Apr 2013 04:39:16 +0000

------ Original Message ------
From: "Mark Nottingham" <[email-masked]>
>>>>  I think we can give better advice than that. If a server responds 
>>>>with a final status code instead of 100 (Continue)
>>>>
>>>>  1. The response must be the last response on the connection. The 
>>>>response should contain "Connection: close" header. After the 
>>>>response is written, the server must initiate a lingering close of 
>>>>the connection (p1#6.6).
>>>  That seems too restrictive; as long as the server reads the rest of 
>>>the request properly (discarding it), it should be able to recover 
>>>and reuse the connection.
>>

I'm really struggling to see what benefit can be derived by a client in 
knowing whether a server supports 100 continue or not.  So to me 
Expects: 100-continue is a complete waste of space.  I've never seen one 
so I guess implementors by and large agree.

Regardless of 100 continue being transmitted, the client has to send the 
payload if it wants to reuse the connection.  The only early-out options 
involve closing the connection.

There was quite a lot of discussion about this in the past, and my 
understanding was that 100 continue couldn't be used to negotiate 
whether or not the payload would be sent.  The outcome of this 
discussion was not satisfactory IMO, since the "answer" was for the 
client to send request bodies always chunked, and send a 0 chunk if it 
needed to abort early.

This IMO is unsatisfactory because it does not indicate that the client 
didn't send the payload, and a whole heap of intermediary agents may act 
on that as if it were complete.

So for me therefore there's still a hole in the spec around this - 
chunking doesn't have a way to indicate aborting the body.  And there's 
no way to pre-authorization transmission of a request body.

I don't see how a server can return a success status code to a message 
it didn't even receive yet.  Returning a 417 due to expectation not met 
is just extra noise and RTT, and the connection needs to be closed 
anyway or the payload sent.

So, what would we really lose if 100-continue were deprecated?  and what 
would we gain.

Some servers send it on every POST, regardless.  This has some 
significant cost.  What's the real benefit?

Adrien

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 24 Apr 2013 18:10:23 +1200

On 24/04/2013 4:39 p.m., Adrien W. de Croy wrote:
>>>>  That seems too restrictive; as long as the server reads the rest 
>>>> of the request properly (discarding it), it should be able to 
>>>> recover and reuse the connection.
>>>
>>>  The problem comes with intermediaries. How are they to know the 
>>> bytes following were the original advertised payload or not? the 
>>> status from server has no guarantee of arriving after the client 
>>> payload starts arriving.
> I'm really struggling to see what benefit can be derived by a client 
> in knowing whether a server supports 100 continue or not. So to me 
> Expects: 100-continue is a complete waste of space.  I've never seen 
> one so I guess implementors by and large agree.

I guess you have never tried uploading a video to the YouTube through an 
old intermediary which requires authentication. At best (Basic) it 
doubles the upload time and can cause the whole transaction to abort 
with a timeout. At worst (NTLM) it can do the same while consuming up to 
3x the total size of the uncompressed video in bandwidth. This exact 
use-case is why we pushed HTTP/1.1 experiments into Squid-2.7.

>
> Regardless of 100 continue being transmitted, the client has to send 
> the payload if it wants to reuse the connection.  The only early-out 
> options involve closing the connection.

Regarding pipelining sure. The benefits come from avoiding the above 
mentioned resource waste and timeouts. Adding a whole TCP setup overhead 
and RTT (few hundred ms) is far faster and cheaper that transferring >10 
MB of data multiple times (from whole seconds to whole minutes).

> client to send request bodies always chunked, and send a 0 chunk if it 

Agred. I too am unhappy with that. It does work however.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Apr 2013 08:21:33 +0200

> > The only way to guarantee safety on the connection is to close it or always send payload.

+1 as well

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Apr 2013 08:31:22 +0200

Hi Adrien,

> so I guess implementors by and large agree.

The first place I saw lots of them (100% of the requests) were between
applications using web services. All the requests were POST and all of
them were using 100-continue. That's how I discovered that it was a non
final status code and that haproxy didn't handle it properly at this
time...

> involve closing the connection.

... or using chunked-encoding.

> whether or not the payload would be sent.

But this can be quite useful with a webmail for example, where you don't
want to upload your mail with attached documents to discover that your
session has expired and that you must upload again!

> The outcome of this 

Yes indeed, this is the only reliable way of using it.

> no way to pre-authorization transmission of a request body.

It's not a big problem because if the server says it rejects the request, it
will just drop the payload and it can safely be transmitted and truncated.

> it didn't even receive yet.

It will only base its decision on credentials or everything found in headers
(eg: auth, cookies, advertised content-length, ...).

> Returning a 417 due to expectation not met 

Except it's sometimes hard for the client to stop uploading something
that was already sent.

> So, what would we really lose if 100-continue were deprecated?  and what 

First, it's the only way for the client to send non-idempotent requests
over existing connections without the risk that they expire during the
upload and that they don't know if the server could process them. If you
want to use a connection pool, you have no other choice.

Second, it's true that it's annoying in high latency networks as it adds
an RTT. I think that clients could have a threshold on the amount of data
below which they don't use it (unless they're reusing an existing
connection).

Regards,
Willy

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 24 Apr 2013 07:46:53 +0000

------ Original Message ------
From: "Amos Jeffries" <[email-masked]>
>I guess you have never tried uploading a video to the YouTube through 
>an old intermediary which requires authentication. At best (Basic) it 
>with a timeout. At worst (NTLM) it can do the same while consuming up 
>to 3x the total size of the uncompressed video in bandwidth. This exact 
similar issue with webmail uploading attachments.  that's why I wrote 
http://tools.ietf.org/id/draft-decroy-http-progress-00.txt

I removed the discussion about flow-control after the aforementioned 
discussion about using chunked transfers for requests.

But I don't see how 100 continue makes any difference in this case.  The 
client needs to either

a) close and retry.  This won't work for any connection-oriented auth 
mechanism.
b) send the whole thing
c) abort the thing with a chunked transfter terminated early (and risk 
horrible side-effects upstream).

All the 100 does is give the client some pause to reflect and spin CPU 
cycles.

Or is the point of this just to provide an easy work-around for clients 
that don't notice server transmissions whilst they themselves are 
sending?  If so, I'd suggest the protocol isn't the place to solve that, 
especially not at the costs involved in 100 continue.

>mentioned resource waste and timeouts. Adding a whole TCP setup 
>overhead and RTT (few hundred ms) is far faster and cheaper that 
>transferring >10 MB of data multiple times (from whole seconds to whole 
>minutes).
sure.  But only if that's an option.

>Agred. I too am unhappy with that. It does work however.
I don't think anyone has tried it yet?

Is it tested?  I'd be surprised if it didn't come with a raft of 
side-effects/problems.

Adrien

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 24 Apr 2013 08:34:30 +0000

------ Original Message ------

>server has no guarantee of arriving after the client payload starts 

Any bytes after the request + headers but not including advertised 
payload must be payload.  There's no other option apart from closing 
(e.g. another request is not an option, since the first message has not 
been sent until any advertised payload is sent, or if it's chunked a 0 
chunk has been sent).

So there's no ambiguity and no need for an intermediary to have 
previously received the server response (initial or otherwise).

Adrien

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 25 Apr 2013 01:28:38 +1200

On 24/04/2013 7:46 p.m., Adrien W. de Croy wrote:
>>>>>>>  I think we can give better advice than that. If a server 
>>>>>>> responds with a final status code instead of 100 (Continue)
>>>>>>>
>>>>>>>  1. The response must be the last response on the connection. 
>>>>>>> The response should contain "Connection: close" header. After 
>>>>>>> the response is written, the server must initiate a lingering 
>>>>>>> close of the connection (p1#6.6).
>> to 3x the total size of the uncompressed video in bandwidth. This 
>> exact use-case is why we pushed HTTP/1.1 experiments into Squid-2.7.
> But I don't see how 100 continue makes any difference in this case.  
> The client needs to either
>

On the contrary the connection can safely be closed after the first 
request/response and the initial challenge with any mechanism, even 
connection-oriented ones. The credentials state does not exist until the 
followup client request with Authorization: header attached. That is the 
point where closure is a probem for connection-oriented auth, BUT also 
by that point Expect has already taken place and capability is known to 
be available or not.

Squid administrators have been using exactly this challenge+close method 
for some years now to avoid MSIE bugs in NTLM. So we have evidence of 
success outside of Expect.

Amos

---

From: Zhong Yu <[email-masked]>
Date: Wed, 24 Apr 2013 11:27:40 -0500

On Tue, Apr 23, 2013 at 9:19 PM, Mark Nottingham <[email-masked]> wrote:

> > However a server may not be able to detect all bad requests. If it sees
> a request without a body, it may simply assume that there's no 100-continue
> expectation.
> >
> >    o  Upon receiving a request with "100-continue" expectation yet
> without a message body (i.e. both Transfer-Encoding and Content-Length are
> > The last requirement for the origin server talks about requests
> *without* a "100-continue" expectation. Therefore it should not be in this
> section. The subject is covered very well in p1#6.6 already, I think we can
> simply delete it from this section.
>
> >     upon receiving a request that includes the 100-continue expectation,
> if the origin server responds with a final status code instead of 100
> response should contain "Connection: close" header. After the response is
> written, the server must initiate a lingering close of the connection
> (p1#6.6).
>
> That seems too restrictive; as long as the server reads the rest of the
> request properly (discarding it), it should be able to recover and reuse
> the connection.
>
>
You are right, scratch my proposal.

>
> > 2. If the client receives a final status code instead of 100 (Continue),
> it should stop sending request body if it is doing so; it must close the
> Well, it has a choice; it can either close the connection, or continue
> sending and recover the connection. If it only has a few more bytes to
> write, and some requests queued, that might be preferable.
>
> I agree that the current language (esp. the use of MAY) is too ambiguous,
> and doesn't highlight why we're taking this approach (because the client
> might decide to start sending the body).
>
>
> --

---

From: Ken Murchison <[email-masked]>
Date: Wed, 24 Apr 2013 13:00:42 -0400

> connection after the response is received.

I don't understand point #2.  If the client submits a request with 
Expect:100-continue, I would assume that the client MUST NOT send any 
part of the body until it receives 100 (Continue) from the server.  If 
the server rejects the request based on the headers (with 412, 415, 417, 
etc) there should be no body data in the pipe for either the client or 
server to worry about, correct?

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Apr 2013 19:06:38 +0200

> server to worry about, correct?

In fact the client can decide that it's been waiting too long for 100
and decides to send anyway (because some old servers or intermediaries
do not know about Expect and will wait).

So what is generally done is that the client sends the headers, waits a
bit then starts to send data if the server does not respond.

Implementation of expect+100 seems to be a real mess at some places,
but when it works it prove to be quite useful.

Willy

---

From: "Ken Murchison" <[email-masked]>
Date: Wed, 24 Apr 2013 13:34:51 -0400

> bit then starts to send data if the server does not respond.

Fair enough. But I would expect that a compliant 1.1 server would respond with 100 (Continue) or failure pretty quickly -- well within the client's "wait" interval.     Given that RFC 2616 is over a decade old, I would like to think that any 1.1 implementation would be compliant with the Expect behavior or should be deprecated. 

Unless we are worried about Expect:100-continue being sent to a 1.0 server, allowing a client to start sending a body in the absence of 100 (Continue) seems like a bad idea to me.  But if this behavior IS needed a client should at least wait several seconds or something longer than the expected roundtrip time.

Am I way off base here?  I'm not privy to all of the history of HTTP.  I only started developing our CalDAV server a couple of years ago.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Apr 2013 19:47:16 +0200

On Wed, Apr 24, 2013 at 01:34:51PM -0400, Ken Murchison wrote:
> Fair enough. But I would expect that a compliant 1.1 server would respond
> with 100 (Continue) or failure pretty quickly -- well within the client's
> "wait" interval.     Given that RFC 2616 is over a decade old, I would like
> to think that any 1.1 implementation would be compliant with the Expect
> behavior or should be deprecated. 
> 
> Unless we are worried about Expect:100-continue being sent to a 1.0 server,
> allowing a client to start sending a body in the absence of 100 (Continue)
> seems like a bad idea to me.  But if this behavior IS needed a client should
> at least wait several seconds or something longer than the expected roundtrip
> time.
> 
> Am I way off base here?  I'm not privy to all of the history of HTTP.  I only
> started developing our CalDAV server a couple of years ago.

No you're perfectly right, but you forgot about the messy case where the 1.1
client sends via a 1.0 gateway to an 1.1 server and the 1.0 gateway needs to
validate the contents before passing the request :-)

Willy

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 24 Apr 2013 19:51:25 +0200 (CEST)

On Wed, 24 Apr 2013, Ken Murchison wrote:

> "wait" interval.  Given that RFC 2616 is over a decade old, I would like to 
> think that any 1.1 implementation would be compliant with the Expect 
> at least wait several seconds or something longer than the expected 
> roundtrip time.

curl/libcurl uses "Expect: 100-continue" by default when it sends larger 
POSTs. As a result, it is probably one of the most commonly disabled headers 
in curl requests. A significant amount of servers have problems with it, 
although I can't back this statement up with any numbers.

The nature of the problems can vary from blatantly refusing to respond when 
the header is present (or just return an error), to just not being aware of it 
and thus the wait-time is just wasted time that makes the operation much 
slower.

curl waits one second for a response before it continues anyway, and it is of 
course possibly too short in some edge cases but I don't think I've ever seen 
any problems due to that.

-- 

  / daniel.haxx.se

---

From: Ken Murchison <[email-masked]>
Date: Wed, 24 Apr 2013 15:20:48 -0400

> validate the contents before passing the request :-)

Yes, ignorance on my part.  If the client-side special handling needs to 
stay because of 1.0 intermediaries, then so be it.  But I don't think 
there should be any wiggle room for 1.1 servers at this point in the game.