Subject: #467: Expect: 100-continue and "final" status codes

From: Mark Nottingham <[email-masked]>
Date: Fri, 17 May 2013 12:47:27 +1000

Looking at this again, I think one of the problems is that people misunderstand what e/c is for.

The current definition of 100-continue in requests doesn't help:

> 100-continue
> 
> 	• The request includes a payload body and the client will wait for a 100 (Continue) response after sending the request header section but before sending the payload body. The 100-continue expectation does not use any expect-params.

<https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#header.expect>

"will wait for" is misleading here; the client might send the body before getting the 100 response. This should really say something like:

"""
100-continue

* The request includes a payload body and, after sending the request header section, the client will wait before some period of time before sending it, to give the server an opportunity to reject the request with a final status code. The server can shorten the wait time by sending a 100 (Continue) response.
"""

It then goes on:

> The primary purpose of the 100 (Continue) status code (Section 6.2.1) is to allow a client that is sending a request message with a payload to determine if the origin server is willing to accept the request (based on the request header fields) before the client sends the payload body. In some cases, it might either be inappropriate or highly inefficient for the client to send the payload body if the server will reject the message without looking at the body.

Again, I think this is misleading. It should say something like:

"""
The 100-continue expectation and 100 (Continue) status code (Section 6.2.1) are useful when a request that has a large body might be rejected by the server; for example, if the request requires authorization (ref to p7). In these situations, clients will often pause between sending the request headers and its body, to give the server an opportunity to refuse the request. 

In cases where the request is successful, this can cause a needless delay, as the client waits to time out (a typical period is one second). If the client has send the 100-continue expectation, the server can use the 100 (Continue) status code to indicate that the request is not going to be rejected, thereby avoiding the remainder of this delay period.

Note that this mechanism does not change the request message parsing algorithm; in particular, whether or not a final response status code is sent, the client still needs to send a complete request message. As such, if a final status code is received, clients will often choose to close the connection, rather than send a complete request (e.g., if it is length-delimited).
"""

If we can agree on that, I think it'll help guide the rest of the discussion here and in the other E/C related issues:
  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/458
  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/468

Am I on track?

Cheers,

On 25/04/2013, at 3:06 AM, Willy Tarreau <[email-masked]> wrote:

> On Wed, Apr 24, 2013 at 01:00:42PM -0400, Ken Murchison wrote:
>>> 2. If the client receives a final status code instead of 100 
>>> (Continue), it
>>> should stop sending request body if it is doing so; it must close the
>>> connection after the response is received.
>> 
>> I don't understand point #2.  If the client submits a request with 
>> Expect:100-continue, I would assume that the client MUST NOT send any 
>> part of the body until it receives 100 (Continue) from the server.  If 
>> the server rejects the request based on the headers (with 412, 415, 417, 
>> etc) there should be no body data in the pipe for either the client or 
>> server to worry about, correct?
> 
> In fact the client can decide that it's been waiting too long for 100
> and decides to send anyway (because some old servers or intermediaries
> do not know about Expect and will wait).
> 
> So what is generally done is that the client sends the headers, waits a
> bit then starts to send data if the server does not respond.
> 
> Implementation of expect+100 seems to be a real mess at some places,
> but when it works it prove to be quite useful.
> 
> Willy
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 17 May 2013 13:12:24 +1000

On 17/05/2013, at 12:47 PM, Mark Nottingham <[email-masked]> wrote:

> the client will wait before some period of time before sending it

...the client will wait for some period of time before sending it...

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 17 May 2013 07:36:23 +0200

Hi Mark,

On Fri, May 17, 2013 at 12:47:27PM +1000, Mark Nottingham wrote:
> "will wait for" is misleading here; the client might send the body before
> getting the 100 response.

I think you made a good point here.

> This should really say something like:
> 
> """
> 100-continue
> 
> * The request includes a payload body and, after sending the request header
> section, the client will wait before some period of time before sending it,
> to give the server an opportunity to reject the request with a final status
> code. The server can shorten the wait time by sending a 100 (Continue)
> response.
> """

However, I wouldn't want server implementations to consider that a client
that is waiting too long before sending (or which does not send at all) is
not compliant, because I'm sure it's already used to send non-idempotent
requests over established connections.

Maybe we should try a variation of your text above, approximately like this
(I don't like my wording but it's just for the general idea) :

 * The request includes a payload body and, after sending the request header
 section, the client is willing to give the server an opportunity to reject
 the request with a final status code based on header inspection alone. The
 the client will then wait for the server to respond with an intermediary
 100 (Continue) status code before sending the payload, and may decide to
 send it anyway after a period of time which only depends on the client
 (typically one second).

> In cases where the request is successful, this can cause a needless delay, as the client waits to time out (a typical period is one second). If the client has send the 100-continue expectation, the…

Here we introduce a problem that will make one think that 100-continue
implies acceptation of the request, which is not true. I'd rather say :

  "... 100 status code to indicate that it needs to receive the full
   request to decide if it will accept it, ..."

> Note that this mechanism does not change the request message parsing algorithm; in particular, whether or not a final response status code is sent, the client still needs to send a complete request…

This makes me notice something, shouldn't we suggest that 417 should be
accompanied with "connection: close" since there's always a risk that
the client started sending anyway ? Note that this would be a small
"should", not a normative one, which would just be a recommendation
to improve the communication between the two ends.

>   http://trac.tools.ietf.org/wg/httpbis/trac/ticket/468

Yes I think so :-)

Willy

---

From: Ken Murchison <[email-masked]>
Date: Fri, 17 May 2013 07:28:33 -0400

> Note that this mechanism does not change the request message parsing algorithm; in particular, whether or not a final response status code is sent, the client still needs to send a complete request…

This is an important point that I completely missed from any reading of 
2616 and the current httpbis.  I assumed that if the server responded 
with a final status code before sending 100-continue, that the client 
would not send the body.

If I read your proposed text properly, the only way for a client to 
avoid sending the body of a failed conditional request while maintaining 
a persistent connection is to send the request using e/c and te/chunked, 
and just send a zero-length chunk after receiving the final response 
code.  Correct?

Having the ability to not send a large body if its going to be rejected 
seems like a useful feature to me, but maybe its not actually 
implemented in the wild.

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

---

From: Zhong Yu <[email-masked]>
Date: Fri, 17 May 2013 12:45:30 -0500

Hi Mark, I disagree with the change.

The old text describes the intended use case - the client should wait for a
100 response before sending the request body. The text then explains the
exceptional case - if the server is not known to be 1.1 compliant, the
client should not wait forever.

Your proposed text emphasizes the exceptional case, making the intended
case rather hidden. I personally prefer the old text.

One can argue that both texts have the same implication for
implementations, which must handle both the intended and exceptional cases
anyway.

However there is one difference, which I think is critical. According to
the old text, if a server is 1.1 compliant, and the client *knows* that the
server is 1.1 compliant, client should wait indefinitely for a response
from server; it should not unilaterally decide to send the request body.
According to the proposed text, the client must send the request body after
very a short timeout. This is a new requirement on clients, which is
probably not damaging, but unjustified nevertheless.

Zhong Yu

On Thu, May 16, 2013 at 9:47 PM, Mark Nottingham <[email-masked]> wrote:

> >       • The request includes a payload body and the client will wait for
> a 100 (Continue) response after sending the request header section but
> before sending the payload body. The 100-continue expectation does not use
> any expect-params.
>
>
> <
> https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#header.expect
> >
>
> getting the 100 response. This should really say something like:
>
> """
> 100-continue
>
> * The request includes a payload body and, after sending the request
> header section, the client will wait before some period of time before
> sending it, to give the server an opportunity to reject the request with a
> final status code. The server can shorten the wait time by sending a 100
> (Continue) response.
> """
>
> It then goes on:
>
> > The primary purpose of the 100 (Continue) status code (Section 6.2.1) is
> to allow a client that is sending a request message with a payload to
> determine if the origin server is willing to accept the request (based on
> the request header fields) before the client sends the payload body. In
> some cases, it might either be inappropriate or highly inefficient for the
> client to send the payload body if the server will reject the message
> without looking at the body.
>
> The 100-continue expectation and 100 (Continue) status code (Section
> 6.2.1) are useful when a request that has a large body might be rejected by
> the server; for example, if the request requires authorization (ref to p7).
> In these situations, clients will often pause between sending the request
> headers and its body, to give the server an opportunity to refuse the
> request.
>
> In cases where the request is successful, this can cause a needless delay,
> as the client waits to time out (a typical period is one second). If the
> client has send the 100-continue expectation, the server can use the 100
> (Continue) status code to indicate that the request is not going to be
> rejected, thereby avoiding the remainder of this delay period.
>
> Note that this mechanism does not change the request message parsing
> algorithm; in particular, whether or not a final response status code is
> sent, the client still needs to send a complete request message. As such,
> if a final status code is received, clients will often choose to close the
> connection, rather than send a complete request (e.g., if it is
> length-delimited).
> """
>
> If we can agree on that, I think it'll help guide the rest of the
> discussion here and in the other E/C related issues:

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 20 May 2013 10:44:19 -0600

On 05/17/2013 05:28 AM, Ken Murchison wrote:
>> algorithm; in particular, whether or not a final response status code
>> is sent, the client still needs to send a complete request message. As
>> such, if a final status code is received, clients will often choose to
>> close the connection, rather than send a complete request (e.g., if it
>> is length-delimited). 

"the client needs to send a complete message" is perhaps too strong.
"the client needs to send a complete message if the client wants to keep
the connection open and the server response allows for a persistent
connection" may be more appropriate.

> with a final status code before sending 100-continue, that the client

Keep in mind that by the time the origin server sends a response (any
status code), the client may have already started sending the request.
And intermediaries may see a different order of those two events.

> and just send a zero-length chunk after receiving the final response

I do not think sending last-chunk prematurely is a good idea because
there will be no way for the intermediaries (and the origin server) to
distinguish an incomplete request from a complete one. The origin server
MUST NOT perform the requested method after sending 417, but
intermediaries may not know that the server has sent the final status
code and, hence, may "perform" the requested method. Misrepresenting the
end of the request body may have nasty side effects in real world.

Terminating the connection is probably the only safe course of action
for the client that does not want to send the whole body.

Cheers,

Alex.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 28 May 2013 15:37:37 +1000

> connection" may be more appropriate.

Agreed.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 28 May 2013 16:03:36 +1000

> Hi Mark, I disagree with the change.

OK.

> The old text describes the intended use case - the client should wait for a 100 response before sending the request body. The text then explains the exceptional case - if the server is not known to be 1.1 compliant, the client should not wait forever.
> 
> Your proposed text emphasizes the exceptional case, making the intended case rather hidden. I personally prefer the old text.

I'd say that what's usable in practice is pretty far from the intent, and furthermore, the intent isn't terribly clear. 

> One can argue that both texts have the same implication for implementations, which must handle both the intended and exceptional cases anyway.

Right.

> However there is one difference, which I think is critical. According to the old text, if a server is 1.1 compliant, and the client *knows* that the server is 1.1 compliant, client should wait indefinitely for a response from server; it should not unilaterally decide to send the request body.

Perhaps, but that's a very limited case; in practice, the only way it can be sure of this knowledge is if it has previously received a HTTP/1.1 response on the *same* connection. Since implementations often use a new connection for a request with a body (since they're often unsafe, and the connection could idle out in transit), this isn't terribly common, AIUI (YMMV).

> According to the proposed text, the client must send the request body after very a short timeout. This is a new requirement on clients, which is probably not damaging, but unjustified nevertheless.

There aren't any new requirements in there; all of this text is only describing semantics and giving implementation advice. What if we modify the text slightly, e.g.,

"""
100-continue

* The request includes a payload body and, after sending the request header section, the client will wait before some (possibly unbounded) period of time before sending it, to give the server an opportunity to reject the request with a final status code. The server can shorten the wait time by sending a 100 (Continue) response.
"""

With similar modifications as appropriate elsewhere?

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 28 May 2013 08:22:52 +0200

Hi Mark,

On Tue, May 28, 2013 at 04:03:36PM +1000, Mark Nottingham wrote:
> There aren't any new requirements in there; all of this text is only
> describing semantics and giving implementation advice. What if we modify the
> text slightly, e.g.,
> 
> """
> 100-continue
> 
> section, the client will wait before some (possibly unbounded) period of time
> before sending it, to give the server an opportunity to reject the request
> with a final status code. The server can shorten the wait time by sending a
> 100 (Continue) response.
> """

I personally think the "possibly unbounded" part solves the issue. However,
we should ass "or a final status code" at the end of the last sentence.

Best regards,
Willy

---

From: Zhong Yu <[email-masked]>
Date: Tue, 28 May 2013 10:57:46 -0500

Sounds good.

I wonder whether this mutual distrust between the client and the server is
still valid today. RFC 2616 wrote in 1999 that some servers did not
understand 100-continue. Is that still the case? I suspect that most
deployed servers today do support 100-continue properly.

Zhong Yu

On Tue, May 28, 2013 at 1:03 AM, Mark Nottingham <[email-masked]> wrote:

> > The old text describes the intended use case - the client should wait
> for a 100 response before sending the request body. The text then explains
> the exceptional case - if the server is not known to be 1.1 compliant, the
> I'd say that what's usable in practice is pretty far from the intent, and
> furthermore, the intent isn't terribly clear.
>
> Perhaps, but that's a very limited case; in practice, the only way it can
> be sure of this knowledge is if it has previously received a HTTP/1.1
> response on the *same* connection. Since implementations often use a new
> connection for a request with a body (since they're often unsafe, and the
> connection could idle out in transit), this isn't terribly common, AIUI
> (YMMV).
>
> > According to the proposed text, the client must send the request body
> after very a short timeout. This is a new requirement on clients, which is
> describing semantics and giving implementation advice. What if we modify
> the text slightly, e.g.,
>
> """
> 100-continue
>
> header section, the client will wait before some (possibly unbounded)
> period of time before sending it, to give the server an opportunity to
> reject the request with a final status code. The server can shorten the
> wait time by sending a 100 (Continue) response.
> """
>
> > >       • The request includes a payload body and the client will wait
> for a 100 (Continue) response after sending the request header section but
> > "will wait for" is misleading here; the client might send the body
> before getting the 100 response. This should really say something like:
> >
> > """
> > 100-continue
> >
> > > The primary purpose of the 100 (Continue) status code (Section 6.2.1)
> is to allow a client that is sending a request message with a payload to
> > In cases where the request is successful, this can cause a needless
> delay, as the client waits to time out (a typical period is one second). If
> the client has send the 100-continue expectation, the server can use the
> 100 (Continue) status code to indicate that the request is not going to be

---

From: Michael Sweet <[email-masked]>
Date: Tue, 28 May 2013 12:04:16 -0400

You'll still find many embedded HTTP servers that do not honor the Expect header.  CUPS has explicit timeout code on the client side for this since it affects (far too) many IPP printers... :(

But things are improving compared to 5 years ago...

On 2013-05-28, at 11:57 AM, Zhong Yu <[email-masked]> wrote:

> Sounds good. 
> 
> I wonder whether this mutual distrust between the client and the server is still valid today. RFC 2616 wrote in 1999 that some servers did not understand 100-continue. Is that still the case? I suspect that most deployed servers today do support 100-continue properly.
> 
> Zhong Yu
> 
> 
> 

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: "Peter Occil" <[email-masked]>
Date: Tue, 28 May 2013 21:51:23 -0400

To make it a bit clearer, change the proposed text “the client will wait before some (possibly unbounded) period of time before sending it” to “the client will wait before some (possibly unbounded) period of time sending the payload body”.

--Peter

From: Zhong Yu 
Sent: Tuesday, May 28, 2013 11:57 AM
To: Mark Nottingham 
Cc: Willy Tarreau ; Ken Murchison ; HTTP Working Group 
Subject: Re: #467: Expect: 100-continue and "final" status codes

Sounds good. 

Zhong Yu

  OK.

  Right.

  """
  100-continue

  Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 29 May 2013 07:15:15 +0200

On Tue, May 28, 2013 at 12:04:16PM -0400, Michael Sweet wrote:
> You'll still find many embedded HTTP servers that do not honor the Expect
> header.  CUPS has explicit timeout code on the client side for this since it
> affects (far too) many IPP printers... :(

Not to mention transparent 1.0 proxies which want to analyse the payload
before passing the request.

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 29 May 2013 16:21:02 +1000

It sounds like we've got agreement on this. I'm marking the issue for incorporation with the (modified) suggested text.

Cheers and thanks,