Subject: Connection:Keep-Alive and Proxies

From: Diwakar Shetty <[email-masked]>
Date: Tue, 26 Nov 2002 21:28:26 +0530

Following is a para which i read in one book.
I could not understand it.
Could somebody please elaborate

---------------------------------------------------

An HTTP/1.0 client could send a "Keep-Alive" header to a HTTP/1.1 proxy that did not understand "Connection" but might mistakenly forward it. If
the downstream connection also maintained a "Keep-Alive" connection, the proxy in the middle would never receive the closing of the response
---------------------------------------------------

My question, is why does it matter. The proxy will just relay whatever comes from actual server to client.

Thanks
diwakar

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 26 Nov 2002 09:46:54 -0700 (MST)

On Tue, 26 Nov 2002, Diwakar Shetty wrote:

> Following is a para which i read in one book. I could not understand
> it. Could somebody please elaborate
>
> An HTTP/1.0 client could send a "Keep-Alive" header to a HTTP/1.1
> proxy that did not understand "Connection" but might mistakenly
> forward it. If the downstream connection also maintained a
> "Keep-Alive" connection, the proxy in the middle would never receive
> the closing of the response

You may be slightly misquoting the book or the book may be slightly
misquoting the protocol -- ``HTTP/1.1 proxy that did not understand
Connection'' is an oxymoron. An exact quote from RFC 2616 (section
19.6.2) reads:

   The problem was that some existing 1.0 clients may be
   sending Keep-Alive to a proxy server that doesn't understand
   Connection, which would then erroneously forward it to the next
   inbound server, which would establish the Keep-Alive connection and
   result in a hung HTTP/1.0 proxy waiting for the close on the
   response.

> My question, is why does it matter. The proxy will just relay
> whatever comes from actual server to client.

The problem is not with relaying bytes. That works just fine. The
problem is with relaying the "end of message" information.  Pure
HTTP/1.0 applications rely on the [TCP] connection close to detect the
end of a message. If the connection happens to be persistent, a pure
HTTP/1.0 client will not detect the end of message for quite a while
(until the connection is closed, which may happen minutes or even days
after the original transaction). Such a delay in end of message
detection is bad for at least three reasons:

	- various resources are wasted for nothing
	  (it takes some CPU cycles, RAM, TPC ports, and possibly even
	   network bandwidth to keep an idle connection open)

	- the end-user software may be indicating that
	  "download" is still in progress; the user may be tempted
	  to reload the page because it "got stuck", etc.

	- HTTP/1.0 clients used by an automated software package
	  (e.g., web crawler or rsync) will block the caller
	  preventing scheduled updates and such from happening
	  [on time]

HTH,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

---

From: Scott Lawrence <[email-masked]>
Date: 26 Nov 2002 11:53:57 -0500

Not at all clear - I can see why you are confused.

I don't think that the case described above is actually one of the
problem cases (there are others).  If by the 'downstream' connection,
they mean the proxy-origin connection, then it isn't a problem case.
If the origin server honored the Keep-Alive it would send a
Content-Length header in the response, and thus the end would not be
ambiguous.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 26 Nov 2002 10:33:42 -0700 (MST)

> Content-Length header in the response, and thus the end would not be

As far as I can see, the following real-world problems make the above
difficult to rely on:

	- many old HTTP/1.0 clients ignore Content-Length header
	  (because they do not really need it for anything other
	  than double checking the content validity)

	- some HTTP/1.0 servers include incorrect Content-Length
	  headers and, hence, 1.0 clients SHOULD NOT depend on the
	  Content-Length value being correct (RFC 1945, section 7.2.2)

	- under certain conditions, the origin server may not include
	  a Content-Length header and a buggy proxy may not append
	  it when downgrading to HTTP/1.0 (or it would not know that
	  it needs to downgrade because the proxy that is going to
	  get stuck has tunneled HTTP/1.1 request version or other
	  headers implying it can handle persistent connections)

Alex.

---

From: Scott Lawrence <[email-masked]>
Date: 26 Nov 2002 13:38:12 -0500

> 	  headers implying it can handle persistent connections)

All of the above are only problems when you have 2 or more buggy
implementations in the same transaction - at some point, you just have
to throw up your hands and give up.

At this point, there is little exuse for deploying any new 1.0 system,
and proxies in particular should be using only 1.1.  I certainly hope
that people who install proxies are checking to see whether or not
they really do the right thing, but I have no practical way of
checking that.  What ever became of the W3C Web Characterization
activity?  It was originally going to try to answer questions like
that.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 26 Nov 2002 13:17:46 -0700 (MST)

On 26 Nov 2002, Scott Lawrence wrote:

> implementations in the same transaction - at some point, you just
> have to throw up your hands and give up.

Let's not forget the context of this thread. Diwakar Shetty asked to
explain the rationale behind the RFC wording.  Whether the problem RFC
2616 tried to address still exists is pretty much irrelevant. The
problem was very real at the time persistent connection handling was
redesigned!

> At this point, there is little exuse for deploying any new 1.0
> system, and proxies in particular should be using only 1.1.  I
> certainly hope that people who install proxies are checking to see
> whether or not they really do the right thing, but I have no
> practical way of checking that.

Many existing systems violate major HTTP/1.1 MUSTs. One of the most
popular proxies (Squid) is still HTTP/1.0. It is questionable whether
it is better to install a "simple mostly working HTTP/1.0 system" or a
"complex mostly working HTTP/1.1 system". There is little excuse for
selling seriously broken HTTP/1.1 implementations with known bugs, but
that is not the subject of this thread :-).

Alex.

---

From: Diwakar Shetty <[email-masked]>
Date: Wed, 27 Nov 2002 09:47:35 +0530

My replies inline below

>    result in a hung HTTP/1.0 proxy waiting for the close on the

The book is referring to HTTP/1.1 proxy and not HTTP/1.0 proxy
If interested, one can refer the book "Web Protocols and Practice" by Balachander Krishnamurthy and Jennifer Rexford, Addison Wesley, May 2001
ISBN 0-201-71088-9
Page no. 289 in the para for "Semantic requirements on an HTTP/1.1 proxy

It concludes at the end...."HTTP/1.1 proxies are not permitted to establish a persistent connection with HTTP/1.0 clients

> (until the connection is closed, which may happen minutes or even days

But it was the HTTP/1.0 client which sent the "Keep-alive" in the first place
Hence it is not pure.
It intentionally wants to keep the Connection alive
Hence, the client can detect the end of message using "Content-length" sent by the origin server, as mentioned by Scott

Perhaps another issue to ponder over would be the different use of "Connection" in HTTP/1.0 and HTTP/1.1

Thanks
Diwakar

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 26 Nov 2002 22:47:20 -0700 (MST)

> The book is referring to HTTP/1.1 proxy and not HTTP/1.0 proxy

I think the book is misquoting the RFC. Again, there is no such thing
as HTTP/1.1 proxy that does not understand Connection header because
Connection: support is a MUST. But the terminology is not important
here. I suspect you are more interested in a real-world problem the
RFC talks about rather than in trying to interpret the book :-).

> It concludes at the end...."HTTP/1.1 proxies are not permitted to
> establish a persistent connection with HTTP/1.0 clients

I hope that's not the end! RFC 2616 comes to the same
conclusion, of course, but then continues:

   The result is that HTTP/1.0 clients must be prevented from
   using Keep-Alive when talking to proxies.

   However, talking to proxies is the most important use of persistent
   connections, so that prohibition is clearly unacceptable. Therefore,
   we need some other mechanism for indicating a persistent connection
   is desired, which is safe to use even when talking to an old proxy
   that ignores Connection. Persistent connections are the default for
   HTTP/1.1 messages; we introduce a new keyword (Connection: close) for
   declaring non-persistence. See section 14.10.

   The original HTTP/1.0 form of persistent connections (the Connection:
   Keep-Alive and Keep-Alive header) is documented in RFC 2068. [33]

As you can see, the problem is solved as long as the client is
HTTP/1.1.

> But it was the HTTP/1.0 client which sent the "Keep-alive" in the
> first place Hence it is not pure.

Proxy is a client in the above context. You need to draw a picture:

	client C (end-client or proxy)
	(issues Connection: keep-alive)
	       |
	       |
	proxy P
	(forwards Connection: keep-alive,
	 does not understand persistency and works as a tunnel)
	       |
	       |
	server S (origin or proxy)
	(establishes persistent connection: technically with proxy A,
	 but, essentially, with client C, which is wrong)

Now, server S will not close the connection after the response is
sent.  End-client C may be fine because it knows how to detect the end
of the response. Proxy P will get stuck though because it will wait
for connection close from server S.

> It intentionally wants to keep the Connection alive Hence, the
> client can detect the end of message using "Content-length" sent by
> the origin server, as mentioned by Scott

True, but the problem is not with client C. The problem is with proxy
P. See above. Now imagine that proxy P does not forward small
responses until they are "completely received"; in this case, client C
gets stuck as well!

HTH,

Alex.

---

From: Jeffrey Mogul <[email-masked]>
Date: Wed, 27 Nov 2002 11:20:16 -0800

Alex wrote:

   I think the book is misquoting the RFC.

Actually, I think Diwakar Shetty's original message misquoted the book.
(Alex guess this right.)  He wrote:

    An HTTP/1.0 client could send a "Keep-Alive" header to a HTTP/1.1
    proxy that did not understand "Connection" but might mistakenly
    forward it. If the downstream connection also maintained a
    "Keep-Alive" connection, the proxy in the middle would never
    receive the closing of the response

He later identified this as from page 289 in Krishnamurthy & Rexford.
The actual paragraph is much longer (the quote above is not the
whole paragraph!) and what the paragraph in the book actually says is:

	[2 sentences I'm not quoting]
	However, interaction between the Connection header and
	Keep-Alive header could result in a hung connection.  This
	occured because an HTTP/1.0 client could send a "Keep-Alive"
	header to a proxy that did not understand "Connection" but
	might mistakenly forward it.  If the downstream connection also
	maintained a "Keep-Alive" connection, the proxy in the middle
	would never receive the closing of the response.  To avoid such
	problems, HTTP/1.1 proxies are not permitted to establish a
	persistent connection with HTTP/1.0 clients.

So the discussion on this mailing list has been misguided because
the book never mentioned "a HTTP/1.1 proxy that did not understand
'Connection'".  

I'm sure the Krishnamurthy & Rexford book does have bugs, and
I suspect this paragraph could have been clearer if the phrase
"to a proxy" had been "to an HTTP/1.0 proxy".  But from now on,
let's insist on accurate quotes before discussing whether some
publication got the story right.

-Jeff

---

From: Diwakar Shetty <[email-masked]>
Date: Thu, 28 Nov 2002 15:45:13 +0530

The original para was a very long one.
So to reduce it, I added the important sentences only.

Also the whole para was in the section for "Semantic requirements on an HTTP/1.1 proxy."
Hence, it made sense to replace the "Proxy" in the para with "HTTP/1.1 proxy"
In fact, thoughout that para all mention was to HTTP/1.1 proxy with the exception of only one place where "only" proxy was mentioned

So I believe, I did not misquote the book
And niether did the book misquote the RFC because the author never said that it was as per the RFC in the first place

In fact, I am more concerned with how things work (or rather does not work in the current discussion topic)  practically rather than what the
RFC says

But anyway, will henceforth send the quotes on "as-is" basis :-)

Thanks
Diwakar

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 28 Nov 2002 13:21:29 -0700 (MST)

On Thu, 28 Nov 2002, Diwakar Shetty wrote:

> In fact, I am more concerned with how things work (or rather does
> not work in the current discussion topic)  practically rather than
> what the RFC says

In this particular case, the RFC actually explains how things do not
work [well] with HTTP/1.0 and how they work [well] with HTTP/1.1,
which is what you seem to be after. Have you received good-enough
answers? Or are there still some doubts you need the group to qualify?
If there are, what are they?

Thanks,

Alex.