Subject: #468 p2: Expectation extensions (was: Expect header 'understand' vs 'meet')

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 10 Aug 2013 01:40:45 -0700

I spent a few days trying to wrangle the issues around the
Expect header field in p2 (#466, #467, #458, #468) and have
run into a brick wall.  I think we should reopen #468,
remove everything except 100-continue, and then specify the
100-continue mechanism as it is implemented in practice.

I tried to rewrite the section so that the existing text
would make sense as a must-understand-or-fail feature for
HTTP/1.1.  I was able to do that, based on the fantasy that
HTTP/1.1 servers had at least deployed that much.  Apparently,
I had already forgotten the testing I did two years ago, and
nothing has changed since then.

In short, Apache httpd is the only server that I could find
which sent 417 Expectation Failed in response to an expectation
extension.  Apache does not actually parse the syntax -- it
simply rejects anything that isn't 100-continue.  There exists
a patched version of nginx that does the same, but only for
file upload requests that are about to succeed.  IIS does not
appear to support Expect at all, though it may for some resources
that I could not find.

Then, I stopped writing, since there is no point in
specifying a must-understand feature that still hasn't been
implemented consistently 16 years after it was originally defined.

What we can do is define how "Expect: 100-continue" is implemented
in practice as an indication that the client will wait some
implementation-dependent amount of time to see if it gets an
error before continuing to send the request body.  We can specify
that sensibly, including its shortcomings, if we discard the
must-understand semantics and the protocol versioning bit
about responding with 417 if the next hop is HTTP/1.0.

I know that #468 was closed due to lack of interest, but the
alternative is to specify something that hasn't been implemented
and will never be interoperable outside a closed environment
(where both client and server are controlled by the same org),
and we don't need Expect extensions in a closed environment.

....Roy

On Sep 8, 2011, at 11:00 AM, Roy T. Fielding wrote:

> On Sep 8, 2011, at 4:43 AM, Amos Jeffries wrote:
> 
>> On 08/09/11 09:09, Roy T. Fielding wrote:
>>> On Sep 7, 2011, at 1:24 PM, Julian Reschke wrote:
>>> 
>>>> On 2011-09-07 22:18, Jan Algermissen wrote:
>>>>> ..
>>>>>> No, that's unfortunately correct. Maybe we need to mention it.
>>>>> 
>>>>> Sorry to keep bugging - do you know the rationale behind this?
>>>> 
>>>> That predates my interest in HTTP.
>>>> 
>>>> Roy?
>>> 
>>> "Expect" is not part of my original design.  It is in the archives, somewhere.
>>> 
>>> http://lists.w3.org/Archives/Public/ietf-http-wg-old/1997MayAug/0263.html
>>> http://lists.w3.org/Archives/Public/ietf-http-wg-old/1997MayAug/0296.html
>>> http://lists.w3.org/Archives/Public/ietf-http-wg-old/1997MayAug/0316.html
>>> http://lists.w3.org/Archives/Public/ietf-http-wg-old/1998MayAug/0165.html
>>> 
>>> IIRC, we had a discussion in the editorial group wherein I pointed
>>> out that the MUST respond to 417 on extensions was a fatal flaw --
>>> we might as well have made it non-extensible.
>>> 
>>> ....Roy
>> 
>> I don't think it is entirely fatal.
>> 
>> Expect: is cleanly restricted to probing features of the underlying pathway or requirements for success of the particular request+entity. Where things either work or fail hop-by-hop, no middle ground for almost worked, but room for middleware recovery.
>> 
>> Perhapse that usability scope needs to be re-worded and made clear.
> 
> Except for the minor detail that Expect was added after RFC2068, and hence
> anyone implementing to the older RFC (or anyone just failing to implement
> every single aspect of 2616) will not fail as defined.
> 
>> Here are some extensions where I believe using it could benefit HTTP implementations today:
>> 
>> 
>> * testing for next-hop pipeline support
>> 
>> As I already suggested in the pipeline thread. Sending "Expect: pipeline" or similar would allow intermediaries pipeline support to be tested and alert them to the fact that extra efforts may need to be made to keep that connection alive.
>> (I know the round-trip arguments, just pointing out it could be a useful extension of Expect: which is up to the client to decide whether to use or not.)
> 
> It is a performance extension.  We can't deploy a performance extension
> using a field that drastically kills performance.
> 
>> * testing for working NTLM auth
>> 
>> Sending "Expect: connection-auth" during the auth logins over the Internet could be used by browsers to boost NTLM/Negotiate reliability. Intermediaries who cannot support or don't understand the requirements for NTLM can send 417 and break the potential infinite loop of auth, lost credentials, connection re-use mistakes, etc, etc.
>> Yes the intermediaries can (and do) today detect the auth headers going through. This still breaks when transiting the intermediaries which lack proper support.
> 
> True.  OTOH, such a client would break on every working install
> of NTLM as well, which means it can't ever be deployed because
> the main reason to use NTLM is to support intranet services.
> 
>> * testing for secure communications
>> 
>> Sending "Expect: tls" during a request to indicate that it must only transit over encrypted connections. Intermediaries who don't comply or understand, send 417 and the connection fails to safety.
> 
> The client already controls whether or not it uses TLS.
> Expect cannot be used to force an intermediary to do something
> on the next hop, since the client can't trust the intermediary
> to respect Expect.
> 
>> I'm sure thats not the end of the possibilities. But it goes to show that Expect: is useful in its current form because of the MUST.
> 
> I don't think so.  Even the 100-continue support would have failed
> if Apache hadn't deployed it early.  Expect is only useful for that
> and for private extensions within corporate networks where one can
> ensure that all of the internal servers support Expect in the same
> way.
> 
> ....Roy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 10 Aug 2013 09:00:12 +0000

In message <[email-masked]>, "Roy T. Fielding" w
rites:

>extension. [...]
>Apache does not actually parse the syntax -- it
>simply rejects anything that isn't 100-continue.

Same for Varnish:

        /* XXX: Expect headers are a mess */
        if (req->err_code == 0 && http_GetHdr(req->http, H_Expect, &p)) {
                if (strcasecmp(p, "100-continue")) {
                        wrk->stats.client_req_417++;
                        req->err_code = 417;
		} [...]

>implemented consistently 16 years after it was originally defined.

Expect ran foul of Gettys 3rd rule from the start:

	The only thing worse than generalizing from one example is
	generalizing from no example at all.

It was neither needed nor useful, and should never have been added.

There's a lesson HTTP/2.0 could have learned, but we clearly don't
have time for that.

>What we can do is define how "Expect: 100-continue" is implemented [...]

I think we should deprecate Expect.

Document the 100 and 417 responses for back- & bug-wards compatibility
and write that clients SHOULD NOT send Expect.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 10 Aug 2013 11:28:04 +0200

> and write that clients SHOULD NOT send Expect.

I disagree with deprecating it, it's the only way to send non-idempotent
request over a reused connection. However I agree that it's non-sense to
try to make it handle everything that can happen in the world. The use
case was only to get the server's ACK for sending data and nothing else,
so let's simplify it to only match the original intent (which is the only
one in use) and forget every useless overdesign that might have been
thought about.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 10 Aug 2013 09:37:53 +0000

In message <[email-masked]>, Willy Tarreau writes:

>request over a reused connection. 

Not really.

Send the headers, wait a RTT to see if you get an ACK, if not abandon.

It's much faster than using Expect and at least as realiable.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 10 Aug 2013 12:01:45 +0200

> It's much faster than using Expect and at least as realiable.

No, for two reasons :

  - this is not really doable at the application level (can only be
    OS dependant, as the application manipulates connected sockets,
    not packets).

  - the fact that the TCP stack in front of you has ACKed your data
    only means they were received in socket buffers, not that they
    were delivered to the application. And when the keep-alive
    timeout strikes and the server closes, there is no way to ensure
    that pending data are always taken care of (since they can come
    in between the moment you check and the close).

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 10 Aug 2013 11:25:48 +0000

In message <[email-masked]>, Willy Tarreau writes:

>    only means they were received in socket buffers, [...]

The fact that the proxy in front of you have given you a 100-Continue
is of approximately the same value...

Expect is useless.  It should at least be mentioned in the document.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 10 Aug 2013 14:49:00 +0200

> is of approximately the same value...

No because if it sends this, it has received the request, so it can
postpone its timeout.

Willy

---

From: Michael Sweet <[email-masked]>
Date: Sat, 10 Aug 2013 19:03:29 -0400

> It's much faster than using Expect and at least as realiable.

Not really, no.  Many server implementations, in the absence of an Expect header, will simply consume the entire POST/PUT and *then* respond.

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Michael Sweet <[email-masked]>
Date: Sat, 10 Aug 2013 19:01:49 -0400

Hi,

On 2013-08-10, at 5:00 AM, Poul-Henning Kamp <[email-masked]> wrote:
> ...

Note that Expect is widely used for printing, specifically to avoid sending gigabytes of print data only to discover that you need to authenticate the request.

So I would be -1 on deprecating Expect.  It might not see consistent usage among generic web browsers, but it *is* consistently used for things like IPP.

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 11 Aug 2013 07:16:51 +0200

Hi Michael,

On Sat, Aug 10, 2013 at 07:01:49PM -0400, Michael Sweet wrote:
> Note that Expect is widely used for printing, specifically to avoid sending
> gigabytes of print data only to discover that you need to authenticate the
> request.
> 
> So I would be -1 on deprecating Expect.  It might not see consistent usage
> among generic web browsers, but it *is* consistently used for things like
> IPP.

And web services which is where I first saw it heavily used.

Willy

---

From: Julian Reschke <[email-masked]>
Date: Mon, 12 Aug 2013 09:22:26 +0200

> implemented consistently 16 years after it was originally defined.

Is there a bug report about nginx not implementing it? If there is it 
would be interesting to understand why it wasn't fixed.

Same for IIS?

Best regards, Julian