Subject: Re: Expect: 100-continue and proxies

From: Henrik Nordström <[email-masked]>
Date: Fri, 14 May 2010 11:52:24 +0200

mån 2010-02-22 klockan 09:36 +0000 skrev Joe Orton:
'
> Ah, I wasn't clear here: httpd/mod_proxy will, given a request 
> containing "Expect: 100-continue":
> 
> 1) send the request including the Expect header to the next hop
> 2) generate and send a "100 Continue" to the client

2 there is a clear violation. Intermediaries MUST NOT generate 1xx
responses.

> 3) read the request body from the client and forward it to the next hop
> 4) read the response from the next hop and forward it to the client
> 
> So I guess this seems to comply with the letter of 2616 ("MUST forward 
> the request, including the Expect header field") but not the spirit: 
> that Expect is handled end-to-end, not hop-by-hop.

Err.

Expect is supposed to be handled hop-by-hop but sent end-to-end. There
is severaly MUST NOT forward the request clauses attached to Expect.

1xx responses is end-to-end.

417 responses is used at hop-by-hop Expect processing to signal that the
expectation can not be met if needed.

Regards
Henrik

---

From: Joe Orton <[email-masked]>
Date: Fri, 14 May 2010 11:11:18 +0100

> 2 there is a clear violation. Intermediaries MUST NOT generate 1xx

I can't find that explicit requirement in 2616; am I missing it, or 
should it simply be implied from the MUST requirement to forward Expect?

Regards, Joe

---

From: Henrik Nordström <[email-masked]>
Date: Fri, 14 May 2010 13:02:32 +0200

> should it simply be implied from the MUST requirement to forward Expect?

It's implied form

- Only mention of 100 in relation to proxies is MUST forward, except
when it MUST NOT or MAY NOT forward.

- The only entity mentioned in creating 100 responses is explicitly
origin servers which either MUST, MUST NOT or MAY generate 100 responses
depending on the conditions, not servers in general.

Plus the general function of Expect + 417, See below.

The main reason why proxies should not normally generate 100 responses
is the same reason why 100 responses were added, the need of providing a
clean forwarding path where the client knows the server is willing to
accept the request without obvious waste of network resources should the
server (or proxy) want to reject the request. A proxy responding with
100 Continue breaks this as it does not yet know the request body can or
should be forwarded. And if you apply this recursively in a chain of
proxies then the breakage becomes more apparent.

The proxy MAY however add Expect: 100-continue provided it's then
prepared to deal with 417 responses without bothering the client, but
doing so is not trivial as it requires storing a copy of the request
until a response of some kind is seen, allowing the proxy to reforward
the request without Expect: 100-continue should a 417 response be seen.

For similar reasons proxies MUST NOT respond with synthesized 100
Continue responses to Expect: 100 Continue, as this breaks any handling
of 417 responses to the Expect. It's not valid to send or forward 417
after sending 100 Continue. So if a proxy synthesizes 100 Continue in
response to Expect: 100-continue it must also save a copy of the request
data until a response is seen and then reforward without Expect:
100-continue if a 417 is seen, BUT this is incompatible with the
requirement that proxies MUST forward Expect headers which leaves the
proxy in the unfortunate situation that it should not forward the 417
and MUST NOT reforward the request without Expect..

All in all, attempting to deal with adding 100 Continue or
adding/removing Expect: 100-continue in a proxy is a complex matter with
quite subtle cases you have to deal with to not break the protocol
expectations of the requesting client. The only case where it's
reasonably safe is after a considerable delay in response to 2068
requests without Expect: 100-continue. And then only to work around old
clients who don't implement Expect: 100-continue and which also do not
implement a timer themselves when seeing that the proxy is HTTP/1.1.

Proxies MAY however drop 100 responses if either the request was without
Expect: 100-continue or of the client have already started to send the
request body. This is a harmless optimization of the protocol.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 14 May 2010 23:20:34 +1200

> 2 there is a clear violation. Intermediaries MUST NOT generate 1xx

does this really apply to all 1xx responses, or just 100-continue (which 
is the only one defined for Expects)?

Just thinking about my I-D about progress notifications, which relies on 
intermediaries potentially inserting 102 or other (as yet undefined 1xx 
code) to indicate progress state.

On that note, is  there any interest in resuscitating this I-D?  It's 
still a major problem (AV scanning at an HTTP gateway).  In fact the 
problem just keeps getting worse.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Henrik Nordström <[email-masked]>
Date: Fri, 14 May 2010 13:57:06 +0200

> is the only one defined for Expects)?

I would assume that new 1xx codes can be specified to be generated by
any server, proxies included. I do not see any reason why not.

> problem just keeps getting worse.

I think it's a good idea, and this is a good place as any to discuss it
even if outside httpbis charter.

What remains to be done in the spec as such?

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 14 May 2010 12:31:37 -0700

> should it simply be implied from the MUST requirement to forward Expect?

There is no such requirement.  Upgrade and 101 is hop-by-hop.
A 100 message may be hop-by-hop if the intermediary has request
storage capability -- requirements placed only on origin servers
imply that all other servers are unconstrained, not that all
other servers are MUST NOT.

....Roy

---

From: Henrik Nordström <[email-masked]>
Date: Fri, 14 May 2010 22:12:45 +0200

> other servers are MUST NOT.

Ok. One follow up question on that. Is it OK to respond with 417
Expecttion Failed after 100 Continue have been sent?

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 14 May 2010 13:41:31 -0700

> Expecttion Failed after 100 Continue have been sent?

Sure, it is a valid status code.

....Roy

---

From: Henrik Nordström <[email-masked]>
Date: Sat, 15 May 2010 10:14:00 +0200

> Sure, it is a valid status code.

Right. Just me putting it in the "clients won't expect this" category
based on the poor experience regarding 417 handling in general.

Not sure if I mentioned this before, but specifications probably should
clarify that if a proxy adds Expect: 100-continue to the request then it
must also be prepared to deal with the 417 response this may result in.
A proxy adding Expect and then forwarding the resulting 417 response to
the client would break things badly. Now this is already covered by the
specs in the sense that a proxy is both a client & server, but it's
still very easy to overlook.

Regards
Henrik

---

From: Joe Orton <[email-masked]>
Date: Sun, 16 May 2010 22:57:47 +0100

> other servers are MUST NOT.

If it is valid behaviour for the proxy to generate the "100" message, is 
the MUST condition for such a proxy to forward the Expect header 
unnecessary?  This was my original question in this thread; 8.2.3 says:

      - If a proxy receives a request that includes an Expect request-
        header field with the "100-continue" expectation, and the proxy
        either knows that the next-hop server complies with HTTP/1.1 or
        higher, or does not know the HTTP version of the next-hop
        server, it MUST forward the request, including the Expect header
        field.

Regards, Joe

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 17 May 2010 10:53:49 +1200

Hi

with reference to 
http://tools.ietf.org/id/draft-decroy-http-progress-04.txt which was the 
last iteration of the proposal, the issues I see needing some further 
discussion are:

1. what form a client advertisement of support for Progress should 
take.  The I-D currently proposes a request header called Progress.  
Another option could be a Connection token.  E.g. Connection: 
Keep-alive, Progress.  This would (should) then be stripped from 
requests by proxies, which could be safer, in that the proxy handling 
Progress would therefore also be sure to handle multiple 1xx responses 
from upstream. A compliant proxy could re-add such a header, although it 
makes no sense to add the header if the client did not initially provide it.

2. Whether or not it makes any sense for a client to suggest relevant 
timeframes for progress.

3. Whether it makes any sense to really limit rate of interim 
responses.  For instance for a local proxy well-connected to the client, 
why limit updates to max 1/s

4. How best to transfer messages for humans.  One suggestion was to use 
a set of well-known tokens that would be universally understood, and 
could therefore be appropriately localised by the browser.

There are also potential issues with what to do about streamed 
(unending) media content.  This is currently not contemplated by the 
I-D, and potentiall falls outside its scope, although it's another issue 
related to gateway scanning.  My feeling is that progress notifications 
provide benefit in a number of situations so perhaps it's best to keep 
it simple so it can proceed, I don't think it will resolve issues around 
pending / spooling of media content, and in fact I think the only way 
around that would be some other mechanism to explicitly address that 
issue (e.g. Origin servers mark content as non-ending somehow).

In terms of progressing, it would be useful if some browser vendor(s) 
were interested, to get some prototype implementation going for testing 
and evaluating the best way forward.  I can provide a proxy for testing 
fairly quickly.

Cheers

Adrien