Subject: Re: 100 Continue and Expects

From: Greg Wilkins <[email-masked]>
Date: Sat, 27 Mar 2010 10:24:28 -0700

Hi,

I'd like to wake up the discussion about 100 Continues in the specification.

In my experience with the Jetty server, one of the main users of Expect
continues is .net clients doing webservices.   Unfortunately there appears
to be a some clients that although they send expect 100 continues, they
always send the body regardless if the server has sent 100 continues
or not.

This behaviour is kind of allowed by RFC2616 as it says:

  "Because of the presence of older implementations, the protocol allows
  ambiguous situations in which a client may send "Expect: 100- continue"
  without receiving either a 417 (Expectation Failed) status or a 100
  (Continue) status. Therefore, when a client sends this header field
  to an origin server (possibly via a proxy) from which it has never
  seen a 100 (Continue) status, the client SHOULD NOT wait for an
  indefinite period before sending the request body"

The issue with this, is what happens if the server does not
send a 100 continues on the very first request it receives
from a client with the Expect: 100 header.

The spec allows container to not-send a 100 continues if there
is already content available to be read:

  "  - An origin server MAY omit a 100 (Continue) response if it has
        already received some or all of the request body for the
        corresponding request."

This is fine if the container wants to read and handle that content.

However, the case that is not covered by the spec, is what if
the server does not want to read the body (eg it wants to send
a 401 or 302), but there is already content available to be read.

It should not be possible for this content to be a pipelined
request - as using pipelining with expect 100 cannot possibly
work (and maybe spec should explicitly say they can't be used
together).    So the data must be the body of the request
that was sent regardless by the client (maybe after NOT
waiting for an indefinite period).

So it would be great if the spec could give guidance to
tell servers what to do with data that is received
before any response is sent to an expect 100 continues.

Perhaps something like:

    An origin server MUST read the request body if it has
        already received some or all of the request body for the
        corresponding request before sending any response

There is still an implicit race here... but if clients are
going to send bodies regardless of the 100 continue mechanisms
then servers need to be able to read those bodies regardless
as well.

cheers

---

From: David Morris <[email-masked]>
Date: Sat, 27 Mar 2010 12:33:46 -0700 (PDT)

> always send the body regardless if the server has sent 100 continues
...
>         corresponding request before sending any response

I don't think so ... the server MUST be prepared to deal with the
unexpected input ... could be a connection reset ... I don't think
it is our place to dictate error handling. 

A new MUST also isn't possible as that would change the protocol.

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 27 Mar 2010 14:23:47 -0700

> it is our place to dictate error handling. 

David,

I'm not sure what you are saying here, as this is not really error
handling.   In the circumstances I'm describing, the spec
allows a client to either not send the body (because it never
got a 100 continues) or it can send a body (because it didn't
delay an unreasonable time).

But the spec is not clear on how the server should handle
the later case, when the server is actually a RFC2616 compliant
server.

> A new MUST also isn't possible as that would change the protocol.

Oops I admit I've not read the bis charter, so I didn't know that
new MUSTs  are not possible.    However, I don't really think this
is a change in the protocol, just some guidance on how to handle
a little ambiguous corner.  A MAY or SHOULD would be equally
helpful in reminding servers that clients may send bodies
even if 100 is not sent.

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 28 Mar 2010 00:31:31 +0000

> helpful in reminding servers that clients may send bodies

The client may send a request body, or close the connection.  What it
can't do is omit the body and send another request on the same
connection.  That said, if it's sending chunked, the client could
truncate the body it's sending if it believes the server doesn't care
about the body, and then reuse the connection.  Because nobody does
chunked requests, though, I'd be hesitant about assuming it works.

I found the section in RFC2616 about the client not having to send the
request body when it receives an error a bit confusing, but after
asking this list a couple of years ago, it was clarified that the
client may send it or close the connection, but it can't simply not
send a body and reuse the connection for another request (unfortunately).

The only benefit of 100 Continue is that the client can read the
response and close, without saturating the link with an unused request
body.

  => RFC2616 does refer to an "error" response in this case, which
     is also a bit confusing - because this behaviour should be fine with
     *any* response code, and it's useful for 3xx.  Perhaps this could
     be relaxed in HTTPbis?

I did experiment with streaming a response at the same time as sending
the request body - which should be fine with the abort rule as long as
the server sends 100 Continue first - but found most clients and
servers didn't behave well with mutual streaming like this.  A few
were ok, which would have been a nice way to reduce connections for
some ajaxy/comety applications.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 27 Mar 2010 21:20:38 -0700

Jamie,

thanks for that info.  I had not realized that the client had
not choice but to not reuse the connection if a non-100 response
is received.

Then perhaps the bis spec can enhanced to make this point clearer.

It might also be good to suggest that servers send Connection: close
with non 100 responses if expect 100 is received.  This would
make clients that don't know they should close, do the close.

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 28 Mar 2010 15:20:01 +0100

> not choice but to not reuse the connection if a non-100 response

I believe it's ok to reuse the connection if it posts a request body

> Then perhaps the bis spec can enhanced to make this point clearer.

I found this part of RFC2616 a little confusing:

    8.2.2 Monitoring Connections for Error Status Messages

       An HTTP/1.1 (or later) client sending a message-body SHOULD
       monitor the network connection for an error status while it is
       transmitting the request. If the client sees an error status,
       it SHOULD immediately cease transmitting the body. If the body
       is being sent using a "chunked" encoding (section 3.6), a zero
       length chunk and empty trailer MAY be used to prematurely mark
       the end of the message.  If the body was preceded by a
       Content-Length header, the client MUST close the connection.

As far I know, the client does not have to close the connection, if
it continues to transmit the rest of the body.

And if it does close, of course it shouldn't do so until it's seen the
whole response, not just the "error status".

Moreover, I wasn't sure what _exactly_ counts as an "error status", as
it's not a defined term, and 3xx (not an error) would seem to need the
same behaviour.  Or even 2xx, if the request body isn't relevant.
(Although I really like the idea of simultaneously streaming
request and response bodies).

> make clients that don't know they should close, do the close.

I don't think it's necessary for them to close, despite the MUST above
;-) In other words, I parse the above RFC2616 extract with the MUST
clause conditional on the client performing the SHOULD and ceasing to
transmit the body - if it hasn't already finished that.

Actually I go a bit further:

With certain types (Microsoft) of authentication, the client must not
close the connection...

Here's a proposed alternative text.  It could be tidied a bit:

    8.2.2 Monitoring Connections and Aborting Request Bodies

       An HTTP/1.1 (or later) client sending a message-body SHOULD
       monitor the network connection for a response while it is
       transmitting the request.  If the client sees a 3xx, 4xx or 5xx
       status, it SHOULD immediately cease transmitting the body (by
       which we also mean not starting to transmit if it has not
       started), although it MAY continue transmitting the body if
       there is not much remaining to transmit, the latter being an
       implementation-defined heuristic.  All references to ceasing to
       transmit the body shall

       The client SHOULD also cease transmitting the body, subject to
       the above option to continue transmitting, if it sees a 2xx
       status which is not preceded by 100 (Continue) after it had
       sent the Expect request header field with the "100-continue"
       expectation.  This allows a server to indicate that it is still
       consuming the request body while it sends the response, or to
       indicate that it does not require the request body.

       When ceasing to transmit the body, if the body is being sent
       using a "chunked" encoding (section 3.6), a zero length chunk
       and empty trailer MAY be used to prematurely mark the end of
       the message, and connection reuse is possible.

       When ceasing to transmit a body which was preceded by a
       Content-Length header, the client SHOULD halt transmission and
       wait until it has received the response, and it MUST close the
       connection without sending another request.  If the client
       requires the whole response, it MUST NOT signal end of
       transmission prior to receiving the whole response by
       "half-closing" the connection, in the sense described as
       simplex close in [RFC793] section 3.5, as that typically causes
       the server to cease transmitting the response.

An idea: When aborting with a zero length trunk, could we append an
"aborted" chunk-extension, to indicate clearly that it was truncated,
not ended cleanly?

-- Jamie

---

From: Henrik Nordström <[email-masked]>
Date: Sun, 28 Mar 2010 21:16:33 +0200

> from a client with the Expect: 100 header.

The server should send an 100 Continue when it accepts to receive the
body. Exception being if part of the body has already received by the
server.

> is already content available to be read:

Yes as it then knows the client is no longer waiting for 100 Continue.

> This is fine if the container wants to read and handle that content.

Irrelevant.

> a 401 or 302), but there is already content available to be read.

That case is the same as when there is no Expect: 100-continue. The
server then responds with the final status, and eats the request body if
it wants to keep the connection persistent. Or if it wants to close the
connection then waits until the response data has been ACK:ed at the TCP
level plus some more to give the client some chance to pick up the
response before connection is closed.

The last part is one of the reasons why 100 Continue was added, adding
structure in how servers can reject request bodies by closing the
connection in controlled manners.

But it's true the wording in RFC2616 leaves much of the detail out or
hides it scattered over many different and seemingly unrelated sections.

> waiting for an indefinite period).

Correct.

>         corresponding request before sending any response

No.

>From 100 Continue server requirements:

server handling of request with 100-continue expectation:

        If it responds with a final status
        code, it MAY close the transport connection or it MAY continue
	to read and discard the rest of the request.

server handling when there is no such expectation:

	request includes a request body, and the server responds
        with a final status code before reading the entire request body
        from the transport connection, then the server SHOULD NOT close
        the transport connection until it has read the entire request,
        or until the client closes the connection.

The race window you worry about is covered in the section following this
"Client Behavior if Server Prematurely Closes Connection", which
squarely puts the burden on the client if it's not being careful when
sending request bodies..

But in real-life and experience servers need to be careful as well, and
is reflected in some other sections of the specifications requiring
servers to perform a graceful close and/or wait for TCP ACK of the
response data.

Regards
Henrik

---

From: Henrik Nordström <[email-masked]>
Date: Sun, 28 Mar 2010 21:35:18 +0200

sön 2010-03-28 klockan 15:20 +0100 skrev Jamie Lokier:
> 
> With certain types (Microsoft) of authentication

You forgot to add "which is not true HTTP authentication schemes". That
family of auth schemes makes many assumptions which is opposite the
intentions of the HTTP specifications so using them as an example when
trying to understand the wording of the specification text is not valid.

Still interesting when talking about what should be said as they are a
reality and something HTTP has to deal with today, but not when trying
to understand why RFC2616 is written in a certain manner.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 29 Mar 2010 09:51:45 -0700

> request and response bodies).

See <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/108>.

Cheers,

--
Mark Nottingham       [email-masked]

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 30 Mar 2010 00:20:22 +0100

> servers to perform a graceful close and/or wait for TCP ACK of the

I think it's a good idea to combine graceful close with TCP ACK when
possible, and otherwise to do graceful close.  But not TCP ACK alone,
as suggested by your "and/or".

I would caution against depending on TCP ACK *alone*, because it fails
in three scenarios:

  1. Sometimes it's HTTPS relayed over a proxy using CONNECT.
     Then TCP ACK does not indicate the client TCP has received the data.
 
  2. Many client TCPs will discard received data on reception of the
     TCP RST from the closed server TCP, before the client application
     receives the data from its TCP.

  3. Sometimes data is relayed over a TCP retransmitter.
     Then TCP ACK does not indicate the client TCP has received the data.

     I've read that some networks do this for all connections, to
     compensate for channel characteristics (radio).  But I've only
     seen it used for certain kinds of tunnelling, and to bypass firewalls.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 30 Mar 2010 00:22:08 +0100

> to understand why RFC2616 is written in a certain manner.

Agreed, Microsoft's version is not standard HTTP.

Regardless of why, it's important to recognise that clients have the
option to send the whole request body if they decide to, despite one
possible reading of that section of RFC2616 that they must abort.

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 30 Mar 2010 13:42:16 +1300

I did a review of what browsers do in some cases like this (e.g POST + 
auth/NTLM) a while back.

Their behaviour varies.  For instance IE (even current) if it thinks it 
will get an auth challenge, will submit initial POSTs with a 
content-length 0 (this breaks a heap of stuff).

FF and Chrome will send the whole POST body as many times as required to 
get through auth.

Server behaviour varies also.  IIS5 for instance will send a 100 
continue on all POST requests regardless of whether they contain 
Expects.  In fact I've never seen a browser send Expects.

I wrote a I-D partly about this a while back, but it was somewhat 
misguided, based on my misunderstanding of the Expects mechanism.

In the end, I think that's the biggest problem with Expects - people 
expect (pardon the pun) more from it than it delivers.  The reason the 
history of this list contains numerous discussions about Expects, is 
because of common misinterpretation.

The wording in the RFC implies that Expects and 100 Continue can be used 
to avoid sending the body of a request.  It omits to mention that if 
indeed you do want to avoid this (e.g. when you get a 3xx or 4xx), you 
either need to disconnect and re-try, or complete the message some other 
way (e.g. use chunking and prematurely complete it with a 0 chunk).

I would really hate to see anyone adopt the 0 chunk approach - it's just 
as bad as sending Content-Length: 0 when the browser thinks the proxy or 
server will send an auth challenge.  Browsers second-guessing proxies / 
servers is not a good option, and when the browser gets it wrong, things 
get ugly.  We had to write a hack into WinGate to cope with the IE 
misbehaviour (WinGate only sends an auth challenge when policy dictates 
auth is required - the browser cannot possibly predict this, and a 0 
length post is valid, but breaks websites).  Sending a 0 length chunk 
tells the proxy the message is complete.  It could then go through many 
stages of processing that is really undesirable.  There's no way to 
signal an abort and maintain a connection.

In the end, the best option IMO is to send the whole thing each time (as 
per FF and Chrome - sorry don't know what Opera does).  It can be 
hideously painful with chained connection-oriented auth over a slow link.

So actually there's no clean solution to this problem.  I proposed a 
while back a kind of HEAD command for POST to establish a path with 
credentials before posting a body.  But in the end, any proper solution 
to this problem will be a significant protocol change to HTTP (since 
HTTP is designed for intermediaries to be able to work with unknown 
methods), which means we'll be living with this problem for a long time yet.

As for NTLM, it's not going away either.  So much as people don't like 
it because it's not HTTP compliant auth - it's the reality we are stuck 
with.

Cheers

Adrien

---

From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Date: Tue, 30 Mar 2010 02:54:05 +0300

On Tue, 30 Mar 2010 03:42:16 +0300, Adrien de Croy <[email-masked]> wrote:

> In fact I've never seen a browser send Expects.

Opera will send "Expect: 100-continue" if the body is >=10 KB, and will in  
such cases wait 1.5s before sending the body.

-- 
Sincerely,
Yngve N. Pettersen

********************************************************************
Senior Developer                     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
********************************************************************

---

From: Mark Pauley <[email-masked]>
Date: Wed, 31 Mar 2010 15:29:07 -0700

CFNetwork on Mac OSX will send a zero-length post as the first leg of the 3-way auth for NTLM after it has received the initial 4xx response.

CFNetwork ought to soon allow clients to specify Expects and handle the 100 Continue properly to reduce bandwidth on any requests with a large payload, though I can't say when 'soon' will be.  Unfortunately, the NTLM case really is dictated by Microsoft.  Their own behavior shows that they believe it's okay to send a POST with a zero length for the initial NTLM transaction, and since Microsoft is by far the largest producer of NTLM-using server software, CFNetwork just ape's IE.  What about the zero-length post breaks HTTP?  It seems like a POST that both has length 0 and Expects: 100-continue is silly, and the client ought to be able to handle either a 100-continue followed by standard HTTP response or simply just a standard HTTP response.  Garbage in, garbage out.

On Mar 29, 2010, at 5:42 PM, Adrien de Croy wrote:

> 
> I did a review of what browsers do in some cases like this (e.g POST + auth/NTLM) a while back.
> 
> Their behaviour varies.  For instance IE (even current) if it thinks it will get an auth challenge, will submit initial POSTs with a content-length 0 (this breaks a heap of stuff).
> 
> FF and Chrome will send the whole POST body as many times as required to get through auth.
> 
> Server behaviour varies also.  IIS5 for instance will send a 100 continue on all POST requests regardless of whether they contain Expects.  In fact I've never seen a browser send Expects.
> 
> I wrote a I-D partly about this a while back, but it was somewhat misguided, based on my misunderstanding of the Expects mechanism.
> 
> In the end, I think that's the biggest problem with Expects - people expect (pardon the pun) more from it than it delivers.  The reason the history of this list contains numerous discussions about Expects, is because of common misinterpretation.
> 
> The wording in the RFC implies that Expects and 100 Continue can be used to avoid sending the body of a request.  It omits to mention that if indeed you do want to avoid this (e.g. when you get a 3xx or 4xx), you either need to disconnect and re-try, or complete the message some other way (e.g. use chunking and prematurely complete it with a 0 chunk).
> 
> I would really hate to see anyone adopt the 0 chunk approach - it's just as bad as sending Content-Length: 0 when the browser thinks the proxy or server will send an auth challenge.  Browsers second-guessing proxies / servers is not a good option, and when the browser gets it wrong, things get ugly.  We had to write a hack into WinGate to cope with the IE misbehaviour (WinGate only sends an auth challenge when policy dictates auth is required - the browser cannot possibly predict this, and a 0 length post is valid, but breaks websites).  Sending a 0 length chunk tells the proxy the message is complete.  It could then go through many stages of processing that is really undesirable.  There's no way to signal an abort and maintain a connection.
> 
> In the end, the best option IMO is to send the whole thing each time (as per FF and Chrome - sorry don't know what Opera does).  It can be hideously painful with chained connection-oriented auth over a slow link.
> 
> So actually there's no clean solution to this problem.  I proposed a while back a kind of HEAD command for POST to establish a path with credentials before posting a body.  But in the end, any proper solution to this problem will be a significant protocol change to HTTP (since HTTP is designed for intermediaries to be able to work with unknown methods), which means we'll be living with this problem for a long time yet.
> 
> As for NTLM, it's not going away either.  So much as people don't like it because it's not HTTP compliant auth - it's the reality we are stuck with.
> 
> Cheers
> 
> Adrien
> 
>

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 01 Apr 2010 11:48:34 +1300

we saw the issue with IE + WinGate Proxy + NTLM going through to a 
webmail site.

WinGate was configured to require auth, but credentials get inherited, 
and policy can be used to specify which sites need auth or not.  So not 
every connection was being asked to auth (407).

IE was obviously working on the assumption that it would ALWAYS be 
challenged with a 407 on new connections.

Customers complained that they couldn't reliably post messages, the 
webmail would randomly break.  We tracked it down to the 0 length POST 
on a new connection.  IE thought it would get a 407, but it didn't, and 
the proxy passed the 0 length post through to the webmail server, which 
barfed on it, with the side-effect of the browser losing the email the 
customer had just written.

I complained about this to MS, but I think they still think it's a good 
idea.

Our hack fix was to check UA, 0 length etc, and if we get one, 
arbitrarily send a 407 back.  It's really ugly and I wish to death that 
MS would pull this broken behaviour.

Part of our selling point is that people can have policy which allows 
for access to some sites to be anonymous, other sites need auth etc - 
the admin chooses when and what needs auth.  There's no way the browser 
can predict it will get a 407 or not, so to send a 0-length POST is 
suicidal.

---

From: Mark Pauley <[email-masked]>
Date: Wed, 31 Mar 2010 15:59:53 -0700

You're right, and we've identified these sort of issues as well.  CFNetwork (and any HTTP client in general) can't assume that they'll be given a 407, even if we know that this host has sent us one before.  Moreover, using a HEAD to 'prime' the auth also doesn't work, nor does anything similar work.  Sometimes you have actually try the full request, because HTTP allows for servers to be as pernicious as they please.

In the end, the only time CFNetwork zeroes out the POST's length is in the case that we've already gotten a 4xx, and we are starting the NTLM auth dance, because there is no way that NTLM can just decide to not give us a 4xx once we give it the NTLM initiation in response to the initial 4xx.

On Mar 31, 2010, at 3:48 PM, Adrien de Croy wrote:

> 
> we saw the issue with IE + WinGate Proxy + NTLM going through to a webmail site.
> 
> WinGate was configured to require auth, but credentials get inherited, and policy can be used to specify which sites need auth or not.  So not every connection was being asked to auth (407).
> 
> IE was obviously working on the assumption that it would ALWAYS be challenged with a 407 on new connections.
> 
> Customers complained that they couldn't reliably post messages, the webmail would randomly break.  We tracked it down to the 0 length POST on a new connection.  IE thought it would get a 407, but it didn't, and the proxy passed the 0 length post through to the webmail server, which barfed on it, with the side-effect of the browser losing the email the customer had just written.
> 
> I complained about this to MS, but I think they still think it's a good idea.
> 
> Our hack fix was to check UA, 0 length etc, and if we get one, arbitrarily send a 407 back.  It's really ugly and I wish to death that MS would pull this broken behaviour.
> 
> Part of our selling point is that people can have policy which allows for access to some sites to be anonymous, other sites need auth etc - the admin chooses when and what needs auth.  There's no way the browser can predict it will get a 407 or not, so to send a 0-length POST is suicidal.
>