Subject: Expect header 'understand' vs 'meet'

From: Jan Algermissen <[email-masked]>
Date: Wed, 7 Sep 2011 21:38:23 +0200

Dear WG,

I have a question on the Expect header section 9.2 in HTTPbis 16[1]

Is my understanding correct:

1. A proxy that finds an expectation extension in the Expect header that it does understand but cannot meet must respond 417
2. A proxy that finds an expectation extension in the Expect header that it does not understand ignores the extension

Jan

[1] http://www.ietf.org/id/draft-ietf-httpbis-p2-semantics-16.txt

---

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Sep 2011 21:58:36 +0200

> [1] http://www.ietf.org/id/draft-ietf-httpbis-p2-semantics-16.txt

"The Expect mechanism is hop-by-hop: that is, an HTTP/1.1 proxy MUST 
return a 417 (Expectation Failed) status code if it receives a request 
with an expectation that it cannot meet."

I think that implies 417 for unknown extensions; how would the proxy 
"meet" the expectation when it doesn't know what it is?

Best regards, Julian

---

From: Jan Algermissen <[email-masked]>
Date: Wed, 7 Sep 2011 22:03:16 +0200

Hi Julian,

On Sep 7, 2011, at 9:58 PM, Julian Reschke wrote:

> "The Expect mechanism is hop-by-hop: that is, an HTTP/1.1 proxy MUST return a 417 (Expectation Failed) status code if it receives a request with an expectation that it cannot meet."
> 
> I think that implies 417 for unknown extensions; how would the proxy "meet" the expectation when it doesn't know what it is?

fair enough.

But this means an extension can only be successfully defined and used when all intermediaries understand it - which is: never.

Or am I getting this wrong?

Jan

---

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Sep 2011 22:09:00 +0200

On 2011-09-07 22:03, Jan Algermissen wrote:
> ...

No, that's unfortunately correct. Maybe we need to mention it.

Best regards, Julian

---

From: Jan Algermissen <[email-masked]>
Date: Wed, 7 Sep 2011 22:18:26 +0200

> No, that's unfortunately correct. Maybe we need to mention it.

Sorry to keep bugging - do you know the rationale behind this?

Can't I have a mechanism that allows the client to ask specifically the origin server to only work on my request if it exhibits a certain behavior? And isn't that (also) the idea behind 'Expect'?

Jan

---

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Sep 2011 22:24:21 +0200

On 2011-09-07 22:18, Jan Algermissen wrote:
> ..

That predates my interest in HTTP.

Roy?

> Can't I have a mechanism that allows the client to ask specifically the origin server to only work on my request if it exhibits a certain behavior? And isn't that (also) the idea behind 'Expect'?

I think lots of people do not want "must understand" extensibility.

See also the various proposals for the "Prefer" header, which defaults 
the other way...

---

From: Jan Algermissen <[email-masked]>
Date: Wed, 7 Sep 2011 22:35:05 +0200

> I think lots of people do not want "must understand" extensibility.

Consequently, the only way then to ensure[1] the client that some resource exhibits behavior X is to provide an appropriate 'link', yes?

So, instead of

POST /foo
Expect: exhibits-feature-x

one must do sth like

>> GET /start

<< 200 Ok
<< Link: </foo>; rel=exhibits-feature-x

POST /foo

....

All fine, just try to understand.

Jan

[1] Ignoring the time gap between the client learning and the client doing the request

> See also the various proposals for the "Prefer" header, which defaults the other way...

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 7 Sep 2011 14:09:49 -0700

> That predates my interest in HTTP.

"Expect" is not part of my original design.  It is in the archives, somewhere.

http://lists.w3.org/Archives/Public/ietf-http-wg-old/1997MayAug/0263.html
http://lists.w3.org/Archives/Public/ietf-http-wg-old/1997MayAug/0296.html
http://lists.w3.org/Archives/Public/ietf-http-wg-old/1997MayAug/0316.html
http://lists.w3.org/Archives/Public/ietf-http-wg-old/1998MayAug/0165.html

IIRC, we had a discussion in the editorial group wherein I pointed
out that the MUST respond to 417 on extensions was a fatal flaw --
we might as well have made it non-extensible.

....Roy

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 08 Sep 2011 23:43:53 +1200

> we might as well have made it non-extensible.

I don't think it is entirely fatal.

Expect: is cleanly restricted to probing features of the underlying 
pathway or requirements for success of the particular request+entity. 
Where things either work or fail hop-by-hop, no middle ground for almost 
worked, but room for middleware recovery.

Perhapse that usability scope needs to be re-worded and made clear.

Here are some extensions where I believe using it could benefit HTTP 
implementations today:

  * testing for next-hop pipeline support

  As I already suggested in the pipeline thread. Sending "Expect: 
pipeline" or similar would allow intermediaries pipeline support to be 
tested and alert them to the fact that extra efforts may need to be made 
to keep that connection alive.
  (I know the round-trip arguments, just pointing out it could be a 
useful extension of Expect: which is up to the client to decide whether 
to use or not.)

  * testing for working NTLM auth

  Sending "Expect: connection-auth" during the auth logins over the 
Internet could be used by browsers to boost NTLM/Negotiate reliability. 
Intermediaries who cannot support or don't understand the requirements 
for NTLM can send 417 and break the potential infinite loop of auth, 
lost credentials, connection re-use mistakes, etc, etc.
  Yes the intermediaries can (and do) today detect the auth headers 
going through. This still breaks when transiting the intermediaries 
which lack proper support.

  * testing for secure communications

  Sending "Expect: tls" during a request to indicate that it must only 
transit over encrypted connections. Intermediaries who don't comply or 
understand, send 417 and the connection fails to safety.

I'm sure thats not the end of the possibilities. But it goes to show 
that Expect: is useful in its current form because of the MUST.

AYJ

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 8 Sep 2011 11:00:36 -0700

On Sep 8, 2011, at 4:43 AM, Amos Jeffries wrote:

> Expect: is cleanly restricted to probing features of the underlying pathway or requirements for success of the particular request+entity. Where things either work or fail hop-by-hop, no middle ground for almost worked, but room for middleware recovery.
> 

Except for the minor detail that Expect was added after RFC2068, and hence
anyone implementing to the older RFC (or anyone just failing to implement
every single aspect of 2616) will not fail as defined.

> Here are some extensions where I believe using it could benefit HTTP implementations today:
> 
> 
> As I already suggested in the pipeline thread. Sending "Expect: pipeline" or similar would allow intermediaries pipeline support to be tested and alert them to the fact that extra efforts may need to be made to keep that connection alive.
> (I know the round-trip arguments, just pointing out it could be a useful extension of Expect: which is up to the client to decide whether to use or not.)

It is a performance extension.  We can't deploy a performance extension
using a field that drastically kills performance.

> Sending "Expect: connection-auth" during the auth logins over the Internet could be used by browsers to boost NTLM/Negotiate reliability. Intermediaries who cannot support or don't understand the requirements for NTLM can send 417 and break the potential infinite loop of auth, lost credentials, connection re-use mistakes, etc, etc.
> Yes the intermediaries can (and do) today detect the auth headers going through. This still breaks when transiting the intermediaries which lack proper support.

True.  OTOH, such a client would break on every working install
of NTLM as well, which means it can't ever be deployed because
the main reason to use NTLM is to support intranet services.

> Sending "Expect: tls" during a request to indicate that it must only transit over encrypted connections. Intermediaries who don't comply or understand, send 417 and the connection fails to safety.

The client already controls whether or not it uses TLS.
Expect cannot be used to force an intermediary to do something
on the next hop, since the client can't trust the intermediary
to respect Expect.

> I'm sure thats not the end of the possibilities. But it goes to show that Expect: is useful in its current form because of the MUST.

I don't think so.  Even the 100-continue support would have failed
if Apache hadn't deployed it early.  Expect is only useful for that
and for private extensions within corporate networks where one can
ensure that all of the internal servers support Expect in the same
way.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 12 Sep 2011 14:51:31 +1000

> every single aspect of 2616) will not fail as defined.

Right. For (lots) more detail, see Bestevros, et al's work:
  http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.4734

I'm all for adding some language that points out further use of Expect as an attractive nuisance. 

--
Mark Nottingham   http://www.mnot.net/