Subject: RE: [Fwd: I-D ACTION:draft-decroy-http-progress-00.txt]

From: Adrien de Croy <[email-masked]>
Date: Tue, 13 Feb 2007 08:51:40 +1300

S. Mike Dierken wrote:
 >
 > Couple questions about the draft:
 >
 >"Frequently problems occur where users of client agents give up 
waiting for
 >visible progress."
 >When I read this, I first thought the issue was not displaying 
progress of
 >the upload itself - but it seems that the issue is with waiting for an
 >indication of a response, or an indication that a response is being 
worked
 >on. Is that correct?

Yes that's correct.  The problem here is largely a human one - people don't
like waiting, and especially if they don't know why or have any clue as 
to how
long.  So this is purely to provide some visible feedback to the person at
the screen about what is going on, and why the delay.

It's quite a different issue to the one of flow-control also discussed 
in the
draft.

 >"There is a clear need for upstream agents to be able to provide timely
 >progress notifications to downstream users [...]"
 >Later in the document, an example of multiple back-and-forth messages is
 >given - could the user agent indicate that this back-and-forth 
messaging is
 >actually a sign of progress?

That example was an example of the flow-control problem rather than the
lack of visual progress problem.  The progress notifications header was
intended to solve user timeouts (giving up) in scenarios where a response
to the user will be a long time coming, e.g:

a. logging into a slow IMAP webmail server with many folders can take a long
time to see any response (my webmail takes 4min).
b. generation of large reports, where generation takes a long time before
any data can be sent
c. proxies scanning content.
d. SOAP requests that take a long time to process
e. etc

 >
 >Much of the document describes a single use-case dealing with 
authentication
 >and NTLM specifically. Are there other specific situations other than
 >authentication that can be described? Are there other styles of
 >authentication other than NTLM that would benefit from a solution?

It's easy to demonstrate with Basic auth as well.  The numbers of retries
are slightly better, but still bad.

 >
 >What other approaches have been considered?

Progress indication
-------------------

The approaches depend on the exact situation.  For example for AV scanning
at a gateway, there are 2 common approaches taken by proxy vendors:

1. Drip-feed a proportion of the resource through to the client whilst it
is being downloaded, and abort the rest of the resource if the scanning 
shows
a virus.

2. Redirect the browser to a "patience page", with a refresh set which shows
updates and progress, and finally a link to download the resource.

Problems with the first approach are:

a. you can't guarantee it is even safe to send even say 75% of the resource
through to the client
b. clients have extreme problems diagnosing what went wrong with their 
download
(need to view resources with binary editors to see appended error messages),
so many clients simply retry many times before giving up.

problems with the second approach.

a. works ok for downloads which aren't to be presented in a browser, but not
good for scanning of large and complex HTML pages.
b. synchronisation issues - proxy can't guarantee the redirect will come 
down
the same connection as the original request
c. difficulties for automated clients doing downloads, as this is a non-
protocol-level handling for the issue.

For flow-control
----------------

The 100 continue is the intended solution to the problem, and whilst I 
can see
how it could be effective in direct client-server communications, there are
issues with it when there are intermediaries or delays.

The alternative proposed of sending chunked resource from the client has 
other
problems, notably:

a. issues when a proxy is connecting to an HTTP/1.0 server.  Unless it 
knows
apriori that the server is HTTP/1.1 compatible it can't send chunked 
resource anyway.
Clients are in the same boat, and that's why I think there aren't any 
(that I have
found) that send chunked data, since they would need to maintain a 
database of
internet webservers to keep track of what server supported chunking or 
not - a
fairly low-return on investment.

b. loss of information on which to base policy decisions.  Unless you 
can set the
content-length field as well?

c. implementation complexity -> compatibility issues with non-compliant 
clients
servers and intermediaries.  An additional status code for a client to 
see is fairly
low-impact, compared to servers and proxies suddenly seeing chunked 
resource
from a client.

Protocols have had 2 signals for flow control since year dot.  RS232 had 
RTS/CTS
Xmodem had X-on/X-off.   The one notable exception is ethernet, which 
uses carrier
sense, and has associated issues not dissimilar to those of HTTP: e.g. 
when you
get a collision, you have to resort to waiting tactics.
Ethernet only really works well like this because it has large bandwidth 
and low
latency, so the collision retry wait times are very small.  Something 
which the
internet unfortunately cannot guarantee.

 >Regarding a connection oriented authentication approach, if the client
 >submitted a request with no body (like a GET) would that be sufficient to
 >establish the credentials that could be used with a larger request 
later on?

not guaranteed to work.

 >Regarding large message bodies, would using chunked transfer-encoding 
with a
 >small initial chunk be useful to quickly 'probe' for these chained
 >authentication routes?
 >

Regards

Adrien de Croy

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 12 Feb 2007 23:25:15 +0100

> issues with it when there are intermediaries or delays.

Not really. 100 Continue is end-to-end, not hop-by-hop. Just as your
proposed "Defer" status.

Your draft adds the new 1xx response not for flow control but for
terminating flows without terminating the connection. This is soft
abortions of request entities, not flow control.

The proposed approach in the draft have at least two major flaws making
it unsuitable:

a) There is no guarantee the client will actually wait for the 100
Continue response before sending the request body, especially not when
there is intermediaries involved which may significantly delay the
request or response, so the server may get a response body even after
sending the "abort" signal and this may then get misread as a different
request (i.e. a PUT of a object containing an HTTP request).

b) And since you are significantly changing the message formatting rules
based on end-to-end communication and not hop-by-hop you are guaranteed
to cause problems when there is intermediaries involved. For example
proxies MUST forward unknown 1xx responses, and this would change the
message format under the feets of the proxy without it knowing causing a
real mess for the proxy.

Both the above problems is related, but from different aspects of HTTP
and at different locations in the request forwarding path.

To guard from this you could in theory add a new "Expect: 1xx Defer"
condition with the added side effect that the client guarantees that it
will not transmit the request body until an 100 Continue is seen and
will instead close the connection and send the request again if 100
Continue is not seen in a timely fashion.

In my eyes, so far the chunked encoding approach looks the most
promising way to solve the authentication issue in a reasonable manner,
even if it means loosing the information on the request size.. And
loosing the information about the request size is frankly the only real
drawback of the approach (broken implementations set aside).

> apriori that the server is HTTP/1.1 compatible it can't send chunked 

Correct. But not really a big problem. 100 Continue solves the ugly part
of this and for authentication as it allows the client to reasonably
probe the server without sending the request body before it knows it
will get accepted by the server.

This whole thing is only a significant problem for NTLM (and Negotiate)
authentication as it can not close the connection on authentication
challenges and therefore MUST transmit the request body which will be
dumped to the bitbucked by the server..

On the initial request when the HTTP level of the next-hop is unknown
the client must use Content-Length, and will from the response learn if
the path is HTTP/1.1, if not neither approach of short-circuiting the
request body can be used and the client MUST resend the request body as
is done today.

To avoid transmitting the request body on the initial request the client
has to close the connection if seeing an authentication challenge or
other error.

In the next attempt in sending the request (possibly with updated
credentials) the client should connect to the same next-hop and assume
an HTTP/1.1 capable path if the last response indicated it's an HTTP/1.1
path. In these conditions it knows for certain that the next-hop is
HTTP/1.1, and that quite likely the whole request path is.

As long it knows the next-hop is HTTP/1.1 it's always safe to send
chunked encoding. In worst case the request may get aborted with a 4xx
forcing the client to fall back to Content-Length and resending the
request body in each roundtrip.. And chunked encoding is a requirement
for the client to clearly indicate to the next-hop that the body has
been terminated avoiding the problems indicated above.

To avoid frequent 411 responses it's probably best for the client to
first send the request with Content-Length and Expect: 100-continue
unless it's known the last path used to that server is fully HTTP/1.1
and that authentication is quite likely needed to finish the request.
Yes, this costs one TCP connection for the initial probe, but for most
uses the probe will not be needed as the server and next-hop is both
already known to the client.

> (that I have found) that send chunked data, since they would need to
> maintain a database of internet webservers to keep track of what
> server supported chunking or  not - a fairly low-return on investment.

Client's dont use chunked encoding today as there hasn't been much
benefit for them to do so.

The NTLM authentication mess is a good reason why to use chunked, and to
care to implement the HTTP/1.1 probing/ needed to do it safely..

Just because the "Defer" status code seemingly looks like it may
initially be less lines of code to implement in existing products does
not make a broken approach a good approach. In the end you'll end up
with about the same amount of code plus much stricter requirements on
when it may be used as it needs all intermediaries to support the new
feature for it to be used reliably and it's also plagued by exactly the
same "next-hop status unknown" issues.

> content-length field as well?

You can't.

> low-impact, compared to servers and proxies suddenly seeing chunked 

True. As chunked encoding is rarely used in requests it hasn't been
tested much and there quite likely is broken implementations out there.
But I think it's safe to say that most if not all HTTP/1.1
implementations will simply abort the request with a 411, at least if
it's a PUT/POST request.

> Xmodem had X-on/X-off.

I don't see how this compares to your proposed extension. The proposed
"Defer" status is not a flow control, it's an abort condition.

X-on/X-off is in HTTP "Expect: 100-continue" and "100 Continue", plus
all the transport flow control on top of which HTTP runs.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 13 Feb 2007 12:49:51 +1300

hi

thanks for your comments, I have a few more below.  It's interesting to go
through the issues.

> Not really. 100 Continue is end-to-end, not hop-by-hop. Just as your
That wasn't my intention.  100 continue in practice can be hop by hop,
as a proxy can send it back
to a client (or is this prohibited?).

> abortions of request entities, not flow control.
Actually it's not intended to abort it but make the client wait for a
100 before sending, so it's a
"please wait for 100 before continuing" response rather than an abort.

This is purely to solve the issue of the client not knowing otherwise
how long to wait for a 100 continue.

Once a client receives the "please wait for 100 before continuing" then
it knows not to send request body
until it receives a 100 continue or completion (i.e denied, auth
required etc)

> request (i.e. a PUT of a object containing an HTTP request).
any 1xx signal should be interpreted as an interim result?  So it
wouldn't matter if a client ignored it as
part of the initial request, or ignored it as part of a subsequent request?

the whole problem is that there is no guarantee that a client will wait
for 100 continue.  If we could guarantee that
I wouldn't have needed to write this spec.

The purpose of the spec is to force compliant clients to wait  for a 100
continue.
A client receiving the defer response then KNOWS that it WILL receive a
100 continue or a final result
instead of having to guess at how long to wait.

> message format under the feets of the proxy without it knowing causing a
Actually where I am coming from is proxy development, and the reason I
wrote the spec was to deal with
issues in proxy implementations relating to flow control.

The proxy that doesn't understand a 10x defer response and passes it
through won't notice anything different from
any other 1xx response.  It is the behaviour of the client that this is
intended to modify

So I'm not sure I understand your point here - if the client treats any
1xx response as possibly multiple (which it
already should), and the proxy forwards everything, then I don't see the
problem.

> Continue is not seen in a timely fashion.

The point is that whatever sends the 1xx defer must guarantee to either

a. send a 100 continue; or
b. send a completion / auth challenge / denial; or
c. close the connection.

so the client MUST wait forever or give up.

As for Expects, I've never seen an "Expects" header.  I think no-one
uses them because the concept is broken.

Sending an Expects header is like putting your head on the block, and
saying "please don't chop my head off"

Because if the server doesn't support it, your request is toast, and you
are going to have to:

a) resubmit the request
b) store information about that server in an database so you don't try
and send an Expects header back
to that server again etc. etc etc.

There's no mechanism in HTTP where a client can apriori evaluate the
capabilities of a server.

So any client developer is going to look for alternatives before
resorting to the Expects header, and will only
add it once there is wide-spread support in it from a majority of
deployed servers (years away).  Even then,
use of the Expects header in a browser is likely to result in only one
thing: tech support calls to your tech support
saying:

"why does your browser have to make each request twice when connecting
to xxxxx?"

> drawback of the approach (broken implementations set aside).
I think most software developers or support staff may disagree.
Implementation difficulty places a large
burden on

a) developers
b) customers (dealing with inevitable bugs).

First day in engineering school they teach the KISS principle.
> dumped to the bitbucked by the server..
actually the mechanism that triggers the problem is the delay in the TCP
connection between the
proxy and the upstream server.  Often we see the client sending the
message body before the
proxy has even connected.  There's no chance for 100 continue to do
anything useful there.

If a proxy could send a 1xx defer however, there could be something
useful done, and this would
benefit all auth schemes.
> request body can be used and the client MUST resend the request body as
yes, problem is maintaining a cache of server behaviour is costly.

> has to close the connection if seeing an authentication challenge or
yes, which breaks session-based auth.

Some may not care about NTLM, since it is MS and non-standard and all,
but there are several dozen
million HTTP users who do care about it, and they have to bear the brunt
of design decisions made in
this protocol.

> HTTP/1.1, and that quite likely the whole request path is.
that depends on the proxy.  If the proxy always responds with the HTTP
version of the request, then
the upstream path capability remains unknown to the client.

having to make a next attempt is also costly.

> chunked encoding. 
I'd argue in general since there's no tried and tested real-life
browsers out there that this isn't safe at all.

> In worst case the request may get aborted with a 4xx
see comment above about server capabilities being masked by the proxy.
In the end in many cases
the only thing the client is sure about is the capability of the proxy,
which doesn't help.

> not make a broken approach a good approach. 
I think the broken approach was to try and create flow control with only
half the minimum signals
required.  This is an attempt to fix that.  I've been watching problems
like this with HTTP for
12 years now.

any layman would laugh at you if you proposed installing traffic lights
at intersections which could
be either green or not there, and you never knew whether it would ever
turn green.

That's what HTTP 100 continue is though.

> In the end you'll end up
next-hop unknown is one hop less to worry about however, since the proxy
that supports this
can already provide major benefit to the end user.

I know from implementing RFC extensions to a myriad of protocols that
I'm much more likely to
implement something
a) at all
b) reliably

if it is simple.  Requiring a client to produce chunked resource data
when the developer knows it's
most likely going to encounter a broken proxy or server is going to
encounter some stiff resistance.

>> content-length field as well?
thought so - thanks for the confirmation.

> implementations will simply abort the request with a 411, at least if
sure, which leaves us no worse off than we are now I guess.

> "Defer" status is not a flow control, it's an abort condition.
no, it's a defer - a X-off see initial comments, the connection or
transaction
isn't aborted at all, it's deferred.

> all the transport flow control on top of which HTTP runs.
X-on is in there, X-off is not.

Regards

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 13 Feb 2007 12:58:17 +1300

PS

I think the issue of NTLM and proxy auth has been a bit of a side-track.

if you consider the issue of

client -> proxy (no auth) -> server (basic auth)

then the problem can still be clearly shown.

1. Client wants to post to server.
2. client connects to proxy, sends initial request, but not body
3. client waits for 100 continue.
4. proxy evaluates request, DNS and policy lookups etc. some time elapses.
5. Proxy Initiates connection to sever.. some more time elapses.
6. Client stops waiting for 100 continue and starts sending body
7. proxy achieves connection, server sends 401 auth required
8. proxy passes back 401 auth required
9. client has to terminate connection and start again.

At least with basic auth, the credentials can be submitted the second time.

If the intermediary is an intercepting proxy however, and requires auth 
(even basic)
then breaking the connection will lose the auth either to the server or 
proxy,
since there can't be 2 sets of Authorization tags, and so this situation 
prevents
operation completely unless the client sends chunked data as you've 
suggested.

I am thinking more about your point about using TCP flow control.  We 
are able to
send a 0 window size, but that won't guarantee that the client won't 
enter into the
state where it thinks it is sending the body (and would therefore 
terminate on
any 4xx response).

regards

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 13 Feb 2007 20:44:15 +1300

> request (i.e. a PUT of a object containing an HTTP request).
sorry, I just re-read this.  I presume you mean a server/proxy may get a
request body after sending a 1xx defer
before sending a 100 continue.  In this case, it's a fairly unambiguous
indication that the client doesn't support this
spec, and the proxy/server should continue as per the current situation.

If a proxy or server sends a 1xx defer to a client, there are several
things that may possibly happen:

1. the client doesn't recognise it and fails in some way (e.g closes the
connection or goes into a spin).  This doesn't comply with section 10.1
of RFC2616.
2. the client ignores it
3. the client treats it as a green light and sends the content immediately
4. the client understands it, and waits for a 100 continue.

each of these should be unambiguous for the server.  Actually para 3 of
RFC2616 10.1 implies that 100 continue and Expects: 100-continue are
hop-by-hop negotiation.

Maybe a status code isn't the best way to achieve flow control, and it
could/should be done with Connection tags?  But in the end if you want
to actively stop a client from sending the body (presuming this is
deemed desirable), then some sort of  message needs to be actively sent
to the client.

for example, a Client which understands flow-control semantics could
send a request such as (just plucking tag-names out of thin air here)

POST http://www.wingate.com/testform.php HTTP/1.1
Host: www.wingate.com
Connection: Keep-alive; Flow-Control=on
Content-Length: 512
Progress: 1 en-nz
etc

then any provisional response except for 100 continue (which is a green
light which is undesired in this case) could send a Connection tag back e.g.

HTTP/1.1 102 Processing
Connection: Flow-Control=Wait; Keep-alive
Progress: %0; Connecting to upstream server
etc

then

HTTP/1.1 100 Continue
Connection: Flow-Control=Proceed; Keep-alive
Progress: %0; Connected to upstream server

this at least allows a client to advertise support for flow-control
without jeopardising the request (as it would be with an Expects
header).  It also allows servers / Proxies to deal with it even if they
don't understand it.

Other problems with the chunked approach.

1. the proxy doesn't know if it has received the whole resource or not,
and what should it do with it.  It may already have started passing it
up to the server.  If the server is HTTP/1.0, then the proxy will have
to be able to handle a rejection of the initial request, and in that
case send unchunked data.

Makes life difficult for proxy vendors, which I why I'm here.  We
already get complaints about file uploads with auth etc.  It would be
nice to actually solve our customers' problems in that regard, and this
should benefit all web users, not just ours.

In the end though, we are supporting chunked data from a client with our
next major version (although we will need to write a client to test
it!), and should be able to resynch this scenario.  I really can't vouch
for anyone else though.

Cheers

Adrien

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 13 Feb 2007 15:21:58 +0000

Adrien de Croy wrote:
> I am thinking more about your point about using TCP flow control.
> We are able to send a 0 window size, but that won't guarantee that
> the client won't enter into the state where it thinks it is sending
> the body (and would therefore terminate on any 4xx response).

It's not permitted to reduce the TCP advertised window, so you can't
send a 0 window size after you've already opened the window to receive
the request headers of unknown size.

RFC 1122, section 4.2.2.16:

    "A TCP receiver SHOULD NOT shrink the window"

    "Many TCP implementations become confused if the window shrinks
    from the right after data has been sent into a larger window.
    Note that TCP has a heuristic to select the latest window update
    despite possible datagram reordering; as a result, it may ignore a
    window update with a smaller window than previously offered if
    neither the sequence number nor the acknowledgment number is
    increased."

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 13 Feb 2007 22:17:24 +0100

> 9. client has to terminate connection and start again.

Well.. no matter what the client only has the choice of either closing
the connection or sending the request body. That's the only two
available options once the request headers have been sent indicating
there will be a request entity body.

Adding a new "Defer" status code does not change this, and is why I got
confused by your draft into thinking that you somehow proposed that the
client should change how it sends the request body.

> At least with basic auth, the credentials can be submitted the second time.

Yes. And the same for Digest or any other message based authentication
scheme following the HTTP/1.1 protocol requirements.

> operation completely unless the client sends chunked data as you've 

An intercepting proxy can not use HTTP auth as HTTP proxy auth is only
possible when following specifications.

Not a all. It's just that if any hop uses NTLM or other connection
oriented aurh sceme then closing the connection is not an option so the
request body MUST be transmitted in order for the authentication to
work.

The new "defer" status code only avoids situations where the body is
unneededly transmitted before closing the connection. It does not at all
solve the problems of NTLM or other connection oriented authentication
methods.

> state where it thinks it is sending the body (and would therefore 

Terminating the connection on 4xx is completely unrelated to if the
client has begun sending the request body or not. That's a property of
the server response and the preferences of the client just as any other
request/response pair.

The only way of not having to send the complete request body without
closing the connection is to use chunked encoding. If Content-Length is
used then the client MUST either send the indicated amount of data or
close the connection no matter what response is received.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 13 Feb 2007 22:41:33 +0100

[reposted to the mailinglist]

tis 2007-02-13 klockan 12:16 +1300 skrev Adrien de Croy:

> 100 before sending, so it's a "please wait for 100 before continuing"
> response rather than an abort.

Ah, right. I misunderstood you completely. Sorry. Been working a bit too
much with the problems of NTLM lately... So we have been speaking of two
completely different problems.

The flaws I pointed out in my last message is obviously false and from
my own misunderstanding of the draft. Must have been very tired when I
first read it.

Thinking. Makes sense in the origin server case. But for an intermediary
to send this it must also implement the timer and fake a 100 Continue
which it is not allowed to do, or know already that the path is HTTP/1.1
before it responds with 102. 

Also if clients follows the specs correctly then in many cases the
client already should wait indefinitely without an additional flow
control. The timer in theory only applies when the client hasn't already
seen a 100 Continue from the same server in response to another request.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 13 Feb 2007 22:44:02 +0100

tis 2007-02-13 klockan 12:49 +1300 skrev Adrien de Croy:

> as a proxy can send it back to a client (or is this prohibited?).

My understanding:

RFC2616 does not intend for proxies to send 100 Continue on their own,
only forwarding of server generated 100 Continue.

Proxies generating 100 Continue on their own messes up the HTTP version
cache of the client and may falsely fool the client into thinking that
the server has accepted the request and is ready to receive the
response.

100 Continue and it's related delay in sending the request body has two
purposes

a) Avoiding wasting bandwidth on large request bodies which will be
ignored by the server anyway, allowing the client to abort the request
early withoutsending the body (by closing the connection or terminating
chunked encoding).

b) To a client receiving 100 Continue this is supposed to mean the
request as such has been accepted by the origin server and it's now
ready to process the body.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 14 Feb 2007 11:41:47 +1300

> there will be a request entity body.

There is another option.  Waiting for a 100 Continue before sending 
message body.

> client should change how it sends the request body.

At the moment, a client wishing to submit a POST or PUT with a request 
body has 3 options after
sending the request headers.

1. immediately send the request body (in contradiction to RC2616, which 
says it should wait for a 100 continue)
2. wait for a period of time for a 100 continue, if it doesn't receive 
one soon enough, send the body.
3. wait indefinitely for a 100 continue before sending the request body.

1 is in contradiction to RFC2616.
At the moment, no clients choose 3, because that would be pointless if 
the server was never going to send 100 continue

The aim of the defer status is to remove option 2.  So there is then 
ONLY option 3.  The client must wait
for a 100 continue (whilst still keeping the connection open) before 
sending the request body.

So, if you have a client that honours a 1xx defer status.

case 1, proxy requires auth, server does not.

1. Client sends request headers only (as per current RFC2616 
suggestions) and goes into a short wait state
2. proxy sees that there will be a request body, so immediately sends 
"HTTP/1.1 1xx wait please"
3. client terminates timer for when it would otherwise send request 
body.  It does not send request body
4 proxy then sends "HTTP/1.1 407 auth required"
5 client establishes auth credentials with proxy, still not sending 
request body6 proxy connects to upstream server. 
7. server sends 100 continue
8 proxy sends server response back through to client
9 client sends request body
10 proxy relays request body.
11. server sends 200 ok.
12. proxy relays 200 ok.
13 done.

case 2, proxy requires auth, and server also requires auth.

1. Client sends request headers only (as per current RFC2616 
suggestions) and goes into a short wait state
2. proxy sees that there will be a request body, so immediately sends 
"HTTP/1.1 1xx wait please"
3. client terminates timer for when it would otherwise send request 
body.  It does not send request body
4 proxy then sends "HTTP/1.1 407 auth required"
5 client establishes auth credentials with proxy, still not sending 
request body
6 proxy connects to upstream server. 
7. server sends 401 auth required
8 proxy sends server response back through to client
9 client resends request with server credentials  Still no request body
10 proxy relays request to server
11. server sends 100 continue
12. proxy relays 100 continue
13. client sends request body
14 proxy relays request body
15 server sends 200 ok
16 proxy relays 200 ok.
17 done.

case 3, only server requires auth

1. Client sends request headers only (as per current RFC2616 
suggestions) and goes into a short wait state
2. proxy sees that there will be a request body, so immediately sends 
"HTTP/1.1 1xx wait please"
3. client terminates timer for when it would otherwise send request 
body.  It does not send request body
4 proxy connects to upstream server. 
5. server sends 401 auth required
6 proxy sends server response back through to client
7 client resends request with server credentials  Still no request body
8 proxy relays request to server
9. server sends 100 continue
10. proxy relays 100 continue
11. client sends request body
12 proxy relays request body
13 server sends 200 ok
14 proxy relays 200 ok.
15 done.

You can see that all three cases, steps 1 - 3 are the same, and the 
steps for server auth, and proxy auth are the same.

Also, step 1 is that proposed by RFC2616. Send and wait before sending 
the request body.

 From the client's point of view, once it has received a 102 wait 
please, it won't send any request body unless it receives
a 100 continue.  If it receives anything else, it will process that.

All this can be done on the same connection between client and proxy, 
proxy and server.

I should really draw up a flow chart of all this.  from the client's 
point of view, it sends a request, and if it receives a 102
wait please, it will then receive any number of:

a. a 100 continue
b. an auth challenge
c. some failure code indicating the request is denied.
d. connection closed (some failure condition)..

until it has received a 100 continue however, it won't send the request 
body.

This is the only way to guarantee that the request body will only ever 
be transmitted once, no matter how many steps
must be taken before it is "safe" for the client to transmit the body.

This method scales to any number of chained elements and any number of 
auth steps, redirects etc etc etc.

> scheme following the HTTP/1.1 protocol requirements.

I think with the way security and encrytion are moving, we are moving 
more towards negotiation of credentials
on a per-connection basis.  Reuse of any crypto derived information is 
normally seen as a vulnerability, so being
able to reuse nonces etc is arguably a security hole.  I see pressure to 
move towards a nonce per connection, or in
other words session-based credentials like NTLM.  Therefore I see this 
"problem" growing rather than shrinking.

> possible when following specifications.
There are many that still do it, and customers wish it.  Are we to deny 
the customers wishes?

> unneededly transmitted before closing the connection. 
Not just this case "where the body is unneededly transmitted before 
closing the connection" but in
all cases, whether the connection is closed or not, and it is in fact 
the intention of this spec that the
connection remain open.

> It does not at all
yes it does.

> close the connection no matter what response is received.
I think there's still some confusion about what I'm proposing.  
Hopefully that's cleared up with the
3 cases I outlined above.

I'm actually working on building Firefox at the moment.  If it would be 
helpful I can make available
a windows proxy and a windows web browser to actually demonstrate this 
in operation.

Will take me a while though.

Regards

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 14 Feb 2007 12:04:13 +1300

> only forwarding of server generated 100 Continue.
fair enough - it's not particularly useful for a proxy to send a 100 
continue anyway.

> the server has accepted the request and is ready to receive the
understood.

I am a bit worried about the HTTP version cache though.  Do clients keep 
a record of HTTP version
responses of servers?

In this case, must the proxy always respond with the same HTTP version 
number that the server
responded with?

We breach this all the time, since we upgrade everything to HTTP/1.1 for 
the client.

For instance, we (in WinGate's HTTP proxy) do everything we can to keep 
the connection to the client
open (if client submits HTTP/1.1 request) in all cases, so

1. if we connect to a HTTP/1.0 server which sends a response with no 
content-length then we send
the data through to the client chunked, so we can keep the client-side 
connection open.  To
do this we must send it back as HTTP/1.1.  When the server closes, we 
send a final chunk.

2. if we are asked for an FTP URL, and have to act as an FTP client, we 
send data through to the
client chunked as well, this lets us maintain a session to the FTP 
server as well, which has great
performance benefits (don't need to keep connecting and logging into the 
FTP server).

we also re-chunk received chunked data from upstream (so we don't need 
to assemble entire chunks,
which would make us vulnerable to resource problems with arbitrarily big 
chunks).  this allows us to
handle chunks received up to any size.

On that note, some guidelines to web client developers on issues of 
which connection to send what
request down would be really useful.  For instance we see browsers 
sending http then ftp then CONNECT
requests down the same connection.  We see them when browsing an FTP 
server, splitting the requests
between multiple client sessions.  It would be useful if client 
developers were guided to keep the FTP
requests for the same FTP server to keep coming down the same connection 
to the proxy.

> ready to process the body.
ok, this was my understanding too.  Thanks

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 14 Feb 2007 02:33:00 +0100

> I am a bit worried about the HTTP version cache though.  Do clients keep 

Clients:

MAY maintain a cache of servers they have seen 100 Continue responses
from. This to avoid the timer fallback when it's known the server
supports 100 Continue.

MAY maintain a HTTP Version cache of the requested server. It's unclear
if this is just the next-hop server, or the whole request path defined
by response HTTP version + Via header version trace. This to determine
if it's reasonably safe to use chunked encoding in requests as it's not
allowed otherwise.

Proxies:

SHOULD maintain a HTTP version cache of resently used next-hop servers
(both proxies and origin servers). Needed to fulfill Expect
requirements.

> In this case, must the proxy always respond with the same HTTP version 

No. Proxies MUST respond with the HTTP version they conform to.

> We breach this all the time, since we upgrade everything to HTTP/1.1 for 

Good.

> requests down the same connection.

And what's the problem with that?

> We see them when browsing an FTP 

Patly agreed.

Specs focuses on minimizing the amount of connections per hop.

As indirect result proxies should if possible maintain a shared pool of
upstream connections shared by all client connections to fulfill their
part of the deal, but it's not a strict requirement.

To minimize the risk of excessive amount of connections HTTP/1.1 clients
SHOULD limit themselves to 2 connections.

But yes, some recommendation that the client should if possible prefer
reusing the same proxy connection for additional requests to the same
origin server would improve things considerably when the proxy do not
maintain a shared upstream connection pool.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 14 Feb 2007 03:05:16 +0100

> There is another option.  Waiting for a 100 Continue before sending 

Well, what I am trying to say is that this does not change the available
options for finishing this request.

> sending the request body.

You still have the same window where the client must guess, just
hopefully a bit smaller.. 

And in case of 102 generated by a proxy you also run into the problem of
how to guarantee there finally will be a 100 Continue delivered. And as
result the proxy must also implement the timer and "fake" a 100 Continue
if no 1xx is seen from the next-hop within a reasonable timeframe.

> 4 proxy then sends "HTTP/1.1 407 auth required"

* client closes the connection to avoid transmitting the request body

> 4 proxy then sends "HTTP/1.1 407 auth required"

* client closes the connection to avoid transmitting the request body.

> 12 proxy relays request body

case 4, server does not implement 100 Continue

1. Client sends request headers only (as per current RFC2616
suggestions) and goes into a short wait state
2. proxy sees that there will be a request body, so immediately sends "HTTP/1.1 1xx wait please"
3. client terminates timer for when it would otherwise send request body.  It does not send request body
4. proxy connects to upstream server. 
5. server sends 401 auth required
6. proxy sends server response back through to client
7*. client closes the connection to avoid transmitting the request body
8. client resends request headers with server credentials
8. proxy relays request headers to server
9. server doesn't implement 100 Continue and none gets sent..

> until it has received a 100 continue however, it won't send the request 

Yes, but you do understand that for the connection to be reused to
complete authentication etc the first request must first be completed
which requires either sending the request body or closing the
connection.

> "problem" growing rather than shrinking.

To use connection oriented schemes effectively HTTP needs a bit or
redesign. The protocol is not designed for connection oriented
authentication, only message oriented authentication.

> the intention of this spec that the

For the connection to remain open without redefining the HTTP message
format or use chunked the request body MUST be transmitted and
discarded.

> > solve the problems of NTLM or other connection oriented authentication

Only if its short-circuiting the request skipping the request-body where
HTTP message format requires one toe be sent, in which case the
objections in the first message again applies and I did not misread what
you wrote..

> 3 cases I outlined above.

I think I understood what you are proposing the first time, even if it's
not what you have written.

You propose changing the basic HTTP message format telling the client
that if it received a 1xx Defer then it should not only wait
indefinitely before sending the request body but also skip sending the
request body and still keep the connection open if it receives a final
response (i.e. auth challenge) before the body has been sent.

Imho to do this you MUST use chunked encoding, and have the client
terminate the request with a end chunk. Redefining the HTTP message
format to allow skipping transmission of a Content-Length indicated
request body is not possible within HTTP/1.1 as it requires the basic
message format to be redefined to allow the body to be omitted.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 14 Feb 2007 16:09:49 +1300

> options for finishing this request.
OK

> hopefully a bit smaller.. 
understood.  If the proxy checks this first, it should be able to be a 
very small window.

If we went with a tag approach, where the client advertises support for 
flow control with e.g. a connection
tag, then it would be more deterministic.

Part of the spec would also need to mandate a minimum amount of time 
that the client MUST wait, e.g. 100ms.
before sending the body, rather than using RTT, which is often only 1 or 
2 ms for a connection to a proxy.

Or the client may already know that the proxy uses this method.

> if no 1xx is seen from the next-hop within a reasonable timeframe.
true. That's what I was also planning on having to do anyway.  The 
entity that sends a "102 wait" MUST
guarantee that a 100 continue will be sent to the client.  So the proxy 
will need to act as a client for
the next-hop server, which is what it needs to do anyway.  If the 
next-hop server doesn't support this
which it wont for some time even if we get the spec through, then the 
proxy needs to manage this.

> * client closes the connection to avoid transmitting the request body

Why would it though?   there's no actual need to close here.  Just 
simply not send the request body,
rather than closing, and pop up an auth dialog box.  It makes no sense 
to waste resource by sending
something the client KNOWS will be dumped.  The Greens will be after us !

clients I've seen are really good at keeping the connection to a proxy 
open, even when connecting to
multiple different sites requiring different auth.

> 9. server doesn't implement 100 Continue and none gets sent..

If it's a new connection from step 8, then it's a new session.

if the client closes at step 7 above then it continues as

0. client makes new connection to proxy
1. Client sends request headers only including server credentials 
2. proxy sees that there will be a request body, so immediately sends "HTTP/1.1 1xx wait please"
3. client terminates timer for when it would otherwise send request body.  It does not send request body
4. proxy connects to upstream server. 
5. proxy sends client request including credentials to server (presumably the credentials are valid)
6. proxy waits for 100 continue from server.  If it doesn't receive one within some time, it sends 100 continue to client.
if it does get one from the server within that time, it sends it on to the client.
7. client sends request body.
8. proxy relays request body
9. server sends 200 OK
10 proxy relays server response
11 done.

clients that support NTLM don't close when they get a 401 or 407, but I guess they may if they haven't
sent a request body.  Part of this support in the client for a wait response would be changes to ensure
that the client did not close the connection, since that's no longer necessary, and breaks connection-based auth

> which requires either sending the request body or closing the

OK, I see where you're coming from.  I was assuming we would change the 
client to not disconnect.

I should not assume :)

the request would have effectively been "completed", (i.e. there was a 
non-interim response given to the
client) although the request body would not have been sent. 

> authentication, only message oriented authentication.

understood.  It has been hacked a bit by MS to allow for NTLM, and all 
windows browsers
now support NTLM (because it's commercial suicide not to).  So it's 
actually well supported out there.

> format or use chunked the request body MUST be transmitted and

I guess more changes to the spec would need to be made to allow for the 
request to be deemed complete
without having to send the request body.

So, yes, it then is a change to the semantics of client requests.

> message format to be redefined to allow the body to be omitted.

Thanks for your patience with me!

OK, so you have what I am proposing there. 

It looks like I omitted some things from the spec.  But, since the spec 
would require changes
to HTTP, and clients and servers and proxies, then is it too much of a 
divergence to allow for
the spec to also then cover a relaxation in strict cases (clearly 
indicated) of the definition of completeness
of a request?

I foresee problems if the client arbitrarily terminates the request body 
by sending a final chunk.  The
proxy then has something that it no longer knows is complete or not.  It 
may have already sent
most of this to the server.  Who knows what the server will have done 
with it.

If the client supports the proposed spec though, we can avoid this 
altogether.

if the client doesn't support this proposal, we are where we are now.

If we are going to change the protocol, and change clients, then we may 
as well change them to provide
the best solution, not one that has known problems, simply to fall 
within the current letter of the current
HTTP spec which we would be changing anyway.  There should be no 
ambiguities where a client supporting this
encounters proxies and servers that dont either, so in all cases, the 
step to take should be clear.

So, I guess I should go away and make a second draft of the proposal 
which covers these aspects.

Anyone know of a decent I-D formatting client?  I may have to write 
one.  Took longer to manually format
pages etc than to write the content (maybe that shows...).

Adrien

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 14 Feb 2007 15:51:35 +0000

> request body6 proxy connects to upstream server. 

But, if I've understood that, it does not work.

This is what I think happens in your scheme:

 1. Client sends request headers only.
 2. Proxy sends "HTTP/1.1 1xx wait please".
 3. Client receives "HTTP/1.1 1xx wait please" and does not send request body.
 4. Some time later, proxy sends "HTTP/1.1 407 auth requiered"
 5. Client sends new request headers to establish auth credentials...

But there is a problem.  What happens if the "HTTP/1.1 1xx wait
please" is not received quickly enough by the client?  Current clients
will time out and send the request body:

 1. Client sends request headers only.
 2. Proxy sends "HTTP/1.1 1xx wait please", but this takes a few
    seconds to transit the network due to delays.
 3. Client's "100 continue" timer times out, and it begins sending the
    request body.
 4. Some time later, client receives "HTTP/1.1 1xx wait please".

The problem is, how shall the proxy parse the data stream from the
client?  In the first case, the data stream consists of request
headers indicating a request body, but no body, followed by more
request headers.  In the second case, the data stream consists of
request headers indicating a request body and at least part of a
request body.

How can the proxy detect the start of the second request in each case?
It's not possible to distinguish request headers from the start of a
request body.

This is the reason why a client is only allowed to send a request
body, or abort the connection.  And this is why it is said the only
solutions are to use chunked request body (which is always sent, but
can be truncated), or for the client to send a request header which
means "I will definitely not send a request body unless I receive 100
Continue".  But both of those will fail with some servers and some proxies.

> until it has received a 100 continue however, it won't send the request 

But what about the case when "102 wait please" is sent by the proxy or
server, but isn't received by the client fast enough to make it wait?

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 15 Feb 2007 05:07:02 +1300

> It's not possible to distinguish request headers from the start of a
hmm, good point.  There is a very small chance that the request body 
could look like a normal request.

Normal scenario is proxy on same LAN as client, so delays would be very 
small.  If the proxy is
across the internet, then delay would be in line with RTT to connect to 
the proxy.  But still it's always
better to have a completely deterministic method - with no room for 
guessing.

The problem you raise is easily fixed though, if we require the client 
to advertise support for flow control
with a tag in say the Connect header (or maybe another tag).

In that case, the proxy only sends the 102 if the client advertised 
support for it.  I wouldn't suggest using
the expects header, since the proxy or server would then be required to 
reject the request if it didn't understand
rather than just ignore the tag (no graceful fall-back).  The 
advertisement could also contain info about how
long the client will wait before sending the message body if there is no 
100 continue or 102 please wait.

It needs to be an advertisement that is safe to send to any server 
without causing a rejection.

> Continue".  But both of those will fail with some servers and some proxies.
yes, I think the expects header has some resistance to implementation.  
Due mainly to

1. difficulty in clients maintaining knowledge about whether servers 
support 100 continue or not
2. possibility that any request sent with a expects header will be 
rejected purely because of that header.
3. lack of support from HTTP/1.0 servers

If all the web clients I've investigated this for actually used an 
Expects header, this would be a moot point
but I've never seen one sent... ever.

Maybe the IE and Firefox teams are just a bit paranoid :)

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 14 Feb 2007 16:20:55 +0000

> something the client KNOWS will be dumped.  The Greens will be after us !

The connection has to be closed because the proxy cannot tell if the
client received "102 wait" in time*, so the proxy cannot tell if the
next request is really another request, or a body which looks
identical to a request.

* You cannot assume that the round trip from client to proxy is small,
and you cannot assume that it's consistent from one request to
another.  There might be surprise network delays, there might be
scheduling issues at the client or the proxy, and also HTTP must work
over slow networks as well as fast ones.

> 6. proxy waits for 100 continue from server.  If it doesn't receive one 
> within some time, it sends 100 continue to client.
> that the client did not close the connection, since that's no longer 
> necessary, and breaks connection-based auth

The problem is that the client must send the request body (though it
could be chunked), or close the connection.  Anything else results in
an ambiguous data stream at the proxy, because the start of a request
is (in general) not distinguishable from the start of a request body.

Omitting the request body while continuing to use the connection is
only possible when the proxy is _certain_ the client is not sending
any request body.  But your scheme doesn't guarantee that, because (a)
the "102 wait" message might not be delivered fast enough to stop
transmission of the request body, and (b) the client might not support
it anyway (although a request header could indicate this support).

For this to work without the message parsing ambiguity, the client
would have to indicate, in a request header, that it will not send a
request body unless "100 continue" is recieved, with no timer.  But
such a client wouldn't work well with other servers.

> So, yes, it then is a change to the semantics of client requests.

A change would be needed which is interoperable with existing clients
and existing servers/proxies.  Unfortunately that's rather difficult.

> indicated) of the definition of completeness

The completeness is required so that clients interoperate with
existing proxies and servers, and vice versa.

> If we are going to change the protocol, and change clients, then we
> may as well change them to provide the best solution, not one that
> has known problems, simply to fall within the current letter of the
> current HTTP spec which we would be changing anyway.  There should
> be no ambiguities where a client supporting this encounters proxies
> and servers that dont either, so in all cases, the step to take
> should be clear.

I think there are several ambiguities which make your change
incompatible with existing proxies and clients.

   1. A client supporting this with an old proxy will work fine.

   2. A proxy supporting this with an old client will not work: it
      will send "102 wait", but it will receive a request body anyway.
      How can it tell it's receiving a request body, and not the start
      of another request?  After the request headers, a proxy cannot
      tell if the next data is the start of the next request, or the
      start of the request body.  They are indistinguishable (a body
      can legimitably begin with "GET /foo HTTP/1.1...").

   3. A client supporting this which does not receive "102 wait"
      quickly enough will still timeout and send at least part of the
      request body.  Causing the same problem as 2.

      This occurs when both client and proxy implement your spec!

   4. To fix 3, you could say a client will wait until it receives a
      1xx message before sending a request body, with no timeout.
      (Then you wouldn't bother with "102 wait".)  But this would then
      break 1, compatibility with existing proxies/servers which don't
      send 1xx.

I think the main problem with your proposal is the assumption that a
proxy can detect the difference between a request body, and the
start of another request with no intervening body.  But it
can't, because a body can look identical to a request.

And the other problem is that "102 wait" will be received by the
client quickly enough to stop it sending any request body; this cannot
be assumed.

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 15 Feb 2007 11:27:44 +1300

Actually there could still exist a race condition that needs considering 
as Jamie pointed out, although the chances are very small.

There could be a race condition where the proxy sends anything back to 
the client other than a 100 continue after having
sent a 102 wait, and the client has given up waiting and started sending 
the request body.  So the proxy has sent an interim AND another response 
eliciting a response from the client, and the client has started to send 
the request body.

Until the proxy has sent a "100 continue" to a client that advertised 
flow control, it knows that anything it receives
from the client after the initial request headers is request message 
body.  This would only occur when the client gives up waiting
for a response from the proxy, which could occur over high-latency slow 
links.  As Jamie says, HTTP must work over
all sorts of links, not just fast low-latency ones.

So there could be a race condition where the proxy gets any other sort 
of response from upstream, or deems it
necessary to send anything other than a "100 continue" (after having 
sent a "102 wait please").  The timer proposed in
the request tag is of little use, since the proxy doesn't know how long 
the client has been waiting already, it doesn't
even know how long it's taken for it to receive notification of the 
incoming connect from its TCP stack (highly loaded
server can take a while between SYN packet, SYN ACK and FD_ACCEPT on a 
socket).

e.g

1. Client connects to proxy, sends request advertising flow control
2. proxy responds with 102 wait please, but it takes a while.
3. proxy connects up-stream and gets a 401 from the server
4. proxy sends 401 back to client.

At this stage, the client has received a 102, and a 401.  However, if it 
took too long to get  the 102, then
it may have started to send the request body.

If it starts sending the request body AND the proxy starts receiving 
from the client before the proxy sends the 401 back, we're fine because 
the proxy knows it hasn't sent anything to the client that warrants the 
client sending anything, and therefore the client must have timed out 
and therefore the data being received is request body.

If the proxy has sent a 401, or any other non interim result, then there 
is some possible doubt.

So there is a small overlap of time between the proxy sending the (102 
and 401) where if the client gives up waiting
there could be a problem.

OK, so what does this mean?

In order for this to happen, the latency between client and proxy must 
be high, and proxy and server must be low.  This happens
in common reverse-proxy scenarios.  E.g. where it takes at least twice 
as long for the 102 to get back to the client, than for the connection 
to be made up-stream, and that server to respond with an auth challenge.

The bigger the gap between a proxy sending a 102, and any subsequent 
response (such as 401, or 302), the less the possibility of this 
occurring.  Obviously we don't want to introduce arbitrary delays, 
although the section in RFC2616 for 100 continue does just that.

OK, so I think that the race condition is a very low likelihood, but a 
non-zero possibility.  I think the chances of such a race
condition coupled with the request body looking like the same request as 
before is worse than your chance of being hit by lightning twice.  But 
anyway.

The entity that can reliably detect the race condition is the client.  
If it receives a 102 wait please after having advertised flow control 
and after it has already started sending the message body because of 
timeout, it would need to abort the connection.

Considering we are talking about a scenario where the client is trying 
to send message body, then an abort and retry (but now in the knowledge 
that the proxy supports flow control), is still a great deal better than 
the current situation.

Regards

Adrien

Adrien de Croy wrote:
> Normal scenario is proxy on same LAN as client, so delays would be 
> very small.  If the proxy is
> across the internet, then delay would be in line with RTT to connect 
> to the proxy.  But still it's always
> the expects header, since the proxy or server would then be required 
> to reject the request if it didn't understand
> long the client will wait before sending the message body if there is 
> no 100 continue or 102 please wait.
>
> yes, I think the expects header has some resistance to 
> implementation.  Due mainly to
>
>>> until it has received a 100 continue however, it won't send the 
>>> request body.
>>>     
>>

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Feb 2007 00:12:19 +0100

> something the client KNOWS will be dumped.  The Greens will be after us !

Because you then make fundamental changes of the HTTP message format and
run into all the issues of my initial objection.

> multiple different sites requiring different auth.

Yes, and it's the intention of HTTP/1.1. But it's all hop-by-hop
transport connections not end-to-end connections.

> client) although the request body would not have been sent. 

The request has not yet been completed. Only the response.

> actually well supported out there.

Yes. And it works out relatively well as long as the first request is
not a very large request and the client has a direct connection to the
server..

> So, yes, it then is a change to the semantics of client requests.

Unless you use chunked encoding, in which case the client can
immediately finish the request at any time.

> indicated) of the definition of completeness

I still think you would have greater success looking into using chunked
encoding, which doesn't require any changes to the specs, just a usage
profile how it should be used.

> most of this to the server.  Who knows what the server will have done 

Thats the thing. It's complete. Chunked encoding works that way.

> If the client supports the proposed spec though, we can avoid this 

And instead getting bitten by the major flaws I outlined before, and
resulting in a highly unstable migration path as it won't work at all as
soon as there is any HTTP/1.1 compliant proxy in the path.

> HTTP spec which we would be changing anyway.

The chunked encoding proposal has very few problems in terms of the
protocol specifications. The only question mark there is how well
supported chunked encoding is in existing proxies having support for
proxying of connection oriented authentication (RFC4559).

> which covers these aspects.

Please do. And I again encourage you to evaluate the digest alternative.

> Anyone know of a decent I-D formatting client?

http://tools.ietf.org/inventory/author-tools

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Feb 2007 00:38:05 +0100

> Continue".  But both of those will fail with some servers and some proxies.

As the root of this discussion is the NTLM authentication scheme the
situation isn't that bad for chunked. There is quite many exchanges
taking place, well sufficient to teach the client that the path is
HTTP/1.1 capable.

An NTLM authentication sequence typically looks like follows

1. Request sent without any credentials
2. Server responds with "WWW-Authenticate: NTLM"
3. Client sends it's initial challenge wrapped up as an HTTP NTLM
authenticication request.
4. Server responds with it's challenge wrapped up as an 401 auth
challenge. From here on the connection MUST be persistent end-to-end.
RFC4559 defines how this capability is negotiated via proxies.
5. Client responds with the user credentials.

Closing the connection after '2' if the server (or forwarding path)
isn't priorly known to support chunked is no big deal imho, and after
'2' the client has reasonably good indication of the capabilities of the
forwarding path to be able to determine if it's OK to use chunked
encoding.

The only significant trouble (ignoring non-conforming implementations)
is if the client connects via a mesh of HTTP proxies and only some of
them is HTTP/1.1 + RFC4559 capable and that the initial auth challenge
is delivered via a HTTP/1.1 + RFC4559 capable path but another path may
get selected if the client disconnects and retries the request on a new
connection.

This is btw the same problem which bites the client "server is 100
Continue capable" cache (8.2.3, where it talks about 100 Continue and
older implementations).

If proxies doing authentication is involved then the situation is quite
similar, except that the client may need to switch back to
Content-Length when finishing the proxy authentication in case the
complete path is not yet known..

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 15 Feb 2007 12:39:00 +1300

> Thats the thing. It's complete. Chunked encoding works that way.
the content body wasn't complete though - but anyway, I think you hit
the nail on the head with your comment below
about any HTTP/1.1 proxy in the path.

Such a proxy will still expect that after headers indicating request
body, there will be request body.  therein lies the problem

We may have to leave proper (end to end) flow control for HTTP/2.0 then

Unless we ban it if there's a via header, but many proxies do not insert
these for security reasons.

darn, it coulda been so good!  Would have worked well in the case where
the proxy supporting it was the next hop
for a client that supported it.

So if we go back to chunking then, so that any intermediaries can be
satisified as to request being completed, then
we could still possibly do something about the issue of
connection-oriented auth?  Shame about losing the Content-length tag is all.

Ditching NTLM isn't an option for me, as many of my customers insist on
MS Active Directory integration.

> http://tools.ietf.org/inventory/author-tools
thanks I looked at this before - i'll take another look.

regards

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Feb 2007 01:01:51 +0100

> with a tag in say the Connect header (or maybe another tag).

Expect: 1xx-defer

Why Expect? Because the client need to be told if any step along the
path does not support the needed extension forcing it to fall back to
current behavior.

Problem: Ignored by HTTP/1.0 proxies and servers, but it can probably be
dealt with as the extension must be supported by each hop to be used and
you already concluded each proxy hop must implement the 100 Continue
timer if the next hop is not known to support the extension.

> support 100 continue or not

The problem is knowing if the server supports Expect, not if 100
Continue is supported.

> rejected purely because of that header.

Which isn't a bad thing.

> 3. lack of support from HTTP/1.0 servers

Pretty much the same as '1'.

> Maybe the IE and Firefox teams are just a bit paranoid :)

Or maybe they haven't seen a point in using one as most servers assumes
any HTTP/1.1 client understands 100 Continue anyway..

The only case where conforming implementations will reject an "Expect:
100-continue" is proxies when the next hop is known to be HTTP/1.0. All
HTTP/1.1 conforming implementations MUST support 100 Continue.

Regards
Henrik

---

From: Jeffrey Mogul <[email-masked]>
Date: Wed, 14 Feb 2007 17:12:35 -0800

I confess that I have not had (and probably won't have) the time
to read this thread very carefully.

But I should point out that the "100-continue" mechanism, and
especially how different versions of that spec interoperate, has
been fairly carefully analyzed in a conference paper, and it
might be a good idea to read that paper before taking any new steps.

Since Alex Rousskov actually pointed this out here 4.5 years ago,

   <http://lists.w3.org/Archives/Public/ietf-http-wg/2002JulSep/0032.html>

I will just quote from his message:

   FYI: There was an interesting paper presented at recent Web Caching
   Workshop[1]: "Safe Composition of Web Communication Protocols for
   Extensible Edge Services" by Adam D. Bradley, Azer Bestavros, and
   Assaf J. Kfoury (Boston University). The title is scary but does not
   say much about the study. You can get a PDF version of the paper [2].

[In fact, you would never guess from the title that it analyzes this
aspect of HTTP! --Jeff]

   The authors used a formal model to find problems with
   Expect/100-continue implementations. If their results are correct,
   there are a few more-or-less realistic cases where a combination of
   compliant and semi-compliant HTTP devices may lead to a communication
   deadlocks or other bad things.

   [1] http://2002.iwcw.org/
   [2] http://2002.iwcw.org/papers/18500001.pdf

As I recall, the paper concludes that RFC2616 more or less gets
the design right, whereas the design in RFC2068 was flawed, and
even so, there are "legal" ways of implementing RFC2616 that could
lead to deadlocks.  In mixed environment (2068 + 2616 implementations
trying to interoperate), deadlock is possible.  (So I would be
very careful about trying to design a new protocol mechanism that is
intended to interoperate with existing RFC2616 implementations.)

The main point of this paper, however, is that it's nearly
impossible to get this kind of protocol "right" (with respect
to certain desirable criteria, such as liveness) unless you
do a formal analysis.  Thinking hard about a few example cases
is a good start, but it doesn't necessarily avoid the bad cases
you hadn't thought of.

My guess is that we were able to get the RFC2616 version "almost
right" mostly because we were lucky.  And even though I had a
lot to do with the final design, I no longer remember clearly how
it works or how we got there.  (And sorry, I certainly couldn't
do this kind of formal analysis myself.  Maybe some grad student
is looking for a project.)

Also: this thread has touched briefly on how proxies should
deal with HTTP version numbers.  Please see RFC2145 -
"Use and Interpretation of HTTP Version Numbers."  It's not
just a good idea, it's the law.

-Jeff

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Feb 2007 04:06:05 +0100

> We may have to leave proper (end to end) flow control for HTTP/2.0 then

And stop calling it flow control. Abandoning the body transfer when one
is indicated is not flow control.

> these for security reasons.

Well.. specs says they MUST. Nothing except for the HTTP version numbers
contained therein needs to mean anything however..

> connection-oriented auth?

Yes. The scheme I outlined earlier should work in nearly all situations.
The exception being if the capability of the forwarding path changes
drastically if the client disconnects and tries again after the initial
dummy auth challenge. But even then at least for PUT/POST it should fail
gracefully. For other methods which MAY have a request-body (for example
GET) the analysis is not so easy, but thankfully a chunked body can not
be misread as a valid HTTP request so the likelyhood for bad things to
happen as result of a chunked request being sent to a HTTP/1.0
proxy/server is quite distant..

The problem with NTLM auth is not so much the 100 Continue timer, but
the fact that the connection MUST be kept persistent once the client
"challenge"/capability-negotiation packet has been sent and therefore
the request body MUST be transmitted. RFC2616 only allows this when
using chunked (8.2.2 Monitoring Connections for Error Status Messages)
so to comply with RFC2616 transmission requirements any session using
NTLM auth MUST use chunked during the handshake until the final
credentials is sent.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Feb 2007 04:14:29 +0100

> just a good idea, it's the law.

If this based is my comment earlier in this thread then it was not about
how to deal with HTTP version numbers but how to determine the HTTP
version level of the complete request path, and what RFC 2616 really
means with "unless the server is known to be HTTP/1.1 compliant" when
there is proxies involved..

Knowing and understanding the HTTP version level of a next-hop server is
possible and reasonable in most cases, but reliably knowing what HTTP
version levels will be used in the complete request path when making a
request is non-trivial when there is multiple levels of proxies involved
or non-conforming proxies not adding Via headers.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 15 Feb 2007 16:23:22 +1300

> is indicated is not flow control.
I guess what I'm talking about when I say flow control, is fully 
bilaterally negotiated agreement on sending
of message bodies in either direction.  So separating out the 
transmission of headers, from transmission of body, and
negotiating agreement on whether a body may be transmitted.

when a client makes a request for a server resource, and a server 
responds, that's bilateral.  The client chose to make
the request, and the server can choose not to send a body in the response.

when a client makes a request containing a resource, there is no 
opportunity for negotiation or agreement about
WHEN the transfer of the message body from the client will occur.

Since transmission of the message body may be costly, being able to 
negotiate its transfer is what I'm talking about.

There is normally no negotiation phase in HTTP of course, for 
performance reasons.  A negotiation phase in the case of
clients submitting large request bodies could be useful however.  There 
are mechanisms already to deal with this
which noone uses, such as sending an OPTIONS command first to evaluate 
the server capabilities could be one way to
probe likely outcomes before attempting to submit a large body.  But 
it's not clear how this would be used in this case.

understood.

> proxy/server is quite distant..

understood
.

> NTLM auth MUST use chunked during the handshake until the final
that would solve message / control  synchronisation issues for sure, 
since you could always be sure
of message boundaries.

So maybe this just really needs to be a clarification note to 
implementors on what to do.

e.g use chunking until you get a 100 continue, then send Content-Length 
would be a preferred
way, then the server can still make policy decisions using the entity 
length, and proxies can still
pump through to an upstream HTTP 1.0 entity knowing apriori the length.

I think we're getting somewhere here?

Are there any comments on the proposal about status messages back to the 
client?

there is still the issue about what status is best to be used for this - 
really a purely informational
message is best one which does not induce a state-change in the client.

Regards

Adrien