Subject: Re: #468 p2: Expectation extensions (was: Expect header 'understand' vs 'meet')

From: Mark Nottingham <[email-masked]>
Date: Thu, 12 Sep 2013 12:17:50 +1000

On 10/08/2013, at 6:40 PM, Roy T. Fielding <[email-masked]> wrote:

> I spent a few days trying to wrangle the issues around the
> Expect header field in p2 (#466, #467, #458, #468) and have
> run into a brick wall.  I think we should reopen #468,
> remove everything except 100-continue, and then specify the
> 100-continue mechanism as it is implemented in practice.
> 
> I tried to rewrite the section so that the existing text
> would make sense as a must-understand-or-fail feature for
> HTTP/1.1.  I was able to do that, based on the fantasy that
> HTTP/1.1 servers had at least deployed that much.  Apparently,
> I had already forgotten the testing I did two years ago, and
> nothing has changed since then.
> 
> In short, Apache httpd is the only server that I could find
> which sent 417 Expectation Failed in response to an expectation
> extension.  Apache does not actually parse the syntax -- it
> simply rejects anything that isn't 100-continue.  There exists
> a patched version of nginx that does the same, but only for
> file upload requests that are about to succeed.  IIS does not
> appear to support Expect at all, though it may for some resources
> that I could not find.
> 
> Then, I stopped writing, since there is no point in
> specifying a must-understand feature that still hasn't been
> implemented consistently 16 years after it was originally defined.
> 
> What we can do is define how "Expect: 100-continue" is implemented
> in practice as an indication that the client will wait some
> implementation-dependent amount of time to see if it gets an
> error before continuing to send the request body.  We can specify
> that sensibly, including its shortcomings, if we discard the
> must-understand semantics and the protocol versioning bit
> about responding with 417 if the next hop is HTTP/1.0.
> 
> I know that #468 was closed due to lack of interest, but the
> alternative is to specify something that hasn't been implemented
> and will never be interoperable outside a closed environment
> (where both client and server are controlled by the same org),
> and we don't need Expect extensions in a closed environment.

Based on discussion so far, that seems like a reasonable path forward. Personally, I'm +1 on this (as should have been clear by all of the Expect-related issues I raised).

Roy, are you taking a stab at this?

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Thu, 12 Sep 2013 09:46:50 +0200

On 2013-09-12 04:17, Mark Nottingham wrote:
> ...

I remain unconvinced.

I just tried "expect: foobar" with a set of sites I use, and many of 
them (facebook.com, twitter.com, google.com, spiegel.de...) indeed 
return 417 although they do not appear to run apache (it's not always 
easy to tell).

My preference would be leave things as they are, add a warning about 
incomplete support, and file issues against the servers that are known 
to be broken.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 12 Sep 2013 10:36:31 +0200

On 2013-09-12 09:46, Julian Reschke wrote:
>> Based on discussion so far, that seems like a reasonable path forward.
>> Personally, I'm +1 on this (as should have been clear by all of the
>> Expect-related issues I raised).
>> ...
>
> I remain unconvinced.
>

(google.com does not, sorry for the inaccuracy).

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 12 Sep 2013 09:58:41 -0700

On Sep 12, 2013, at 12:46 AM, Julian Reschke wrote:
> I just tried "expect: foobar" with a set of sites I use, and many of them (facebook.com, twitter.com, google.com, spiegel.de...) indeed return 417 although they do not appear to run apache (it's not always easy to tell).

They all run Apache code -- they just tweaked it a bit (or a lot).
Akamai would be interesting, since it is (or was) based on a
different source.

In any case, the extension feature is entirely dependent on
universal support for interop.  If it doesn't work, and nobody
has used in in 15 years, then it is safer to deprecate it.

I have been trying to get the last issues done all week, but
other people keep interrupting me.  Maybe today.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 12 Sep 2013 19:46:29 +0200

> They all run Apache code -- they just tweaked it a bit (or a lot).

spiegel.de claims to use something squid-ish.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 12 Sep 2013 11:36:01 -0700

> spiegel.de claims to use something squid-ish.

Must be a recent version.

www.akamai.com fails on curl with a 403 (apparently attempting
to redirect it to an internal server with access control).

http://s.a.ak6i.net/a1/ ignores Expect even though it claims
to be Apache (it is probably Zeus)

http://statse.webtrendslive.com/dcsd5z3icpifwzr6ntaprqwib_7q9t/dcs.gif
ignores Expect (MS IIS/6.0)

http://www.google-analytics.com/__utm.gif?
ignores Expect (Golfe2) 

http://api.demandbase.com/api/v2/ip.json
handles Expect correctly and doesn't look like Apache:

   HTTP/1.1 417 EXPECTATION_FAILED
   Content-Length: 0
   Connection: Close

http://adobe.tt.omtrdc.net/m2/adobe/mbox/standard
ignores Expect (Adobe Target, probably nginx-based)

http://p.typekit.net/p.gif
handles Expect correctly (ECSF, probably Apache-based)

https://1295336.fls.doubleclick.net/activityi
ignores Expect (Floodlight server)

Note that the examples above are analytics and CDNs, which means
extensively referred to by most commercial sites.

My suggestion is that we remove the extension syntax and leave a
requirement that anything other than "100-continue" SHOULD be
given a response of 417 unless it indicates a private extension
recognized and implemented by the recipient.  The extension syntax
is problematic because it doesn't look at all like 100-continue
and implies that extensions will actually work.  They won't.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 12 Sep 2013 21:25:51 +0200

> Must be a recent version.

http://redbot.org/?uri=http%3A%2F%2Fspiegel.de&req_hdr=expect%3Afoobar

says "3.1.4".

> to redirect it to an internal server with access control).

Indeed: 
http://redbot.org/?uri=http%3A%2F%2Fwww.akamai.com&req_hdr=expect%3Afoobar

> requirement that anything other than "100-continue" SHOULD be

But the servers that ignore Expect today will continue to be broken, right?

> and implies that extensions will actually work.  They won't.

What part of the syntax are you referring to? Anything beyond "expect-name"?

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 12 Sep 2013 14:03:44 -0700

> But the servers that ignore Expect today will continue to be broken, right?

Yes.  My earlier suggestion was to remove the MUST-417 requirement
along with the extension syntax, and leave it with a MAY-417 if
the field contains anything other than "100-continue".

> What part of the syntax are you referring to? Anything beyond "expect-name"?

Anything beyond "100-continue".

The problem I am dealing with is that the non-interoperable
discussion of expectation-extension parameters and requirements
on forwarding Expect completely obscures the only real purpose
remaining for this field and prevents intermediaries from
handling 100-continue on behalf of downstream 1.0 recipients.

My preferred solution is to remove the garbage, define
100-continue, allow it to be processed by intermediaries,
and leave a Note behind to explain the failed extension
mechanism and usage of 417.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 13 Sep 2013 10:51:10 +1000

> mechanism and usage of 417.

+1

---

From: Jeff Pinner <[email-masked]>
Date: Fri, 13 Sep 2013 08:41:42 -0700

FWIW, twitter.com runs custom (non-apache code) that does the following:

We send a 417 to any HTTP/1.0 request with a Expect header.
We send a 417 to any HTTP/1.1 request with a Expect header that is not
case-insensitive 100-continue (this includes empty Expect headers but we do
allow some ridiculousness like Expect: , , 100-continue)
We then remove the Expect header for further processing, responding with
100 Continue if necessary.