Subject: HTTP/1.1 pconns to 1.0 servers

From: Mark Nottingham <[email-masked]>
Date: Wed, 20 Sep 2006 10:08:20 -0700

HTTP/1.0 persistent connections are documented in RFC2068. One case  
that's not explicitly covered is when a HTTP/1.1 client sends a  
request to a HTTP/1.0 server without a Connection: keep-alive header.

My reading of 2068, 2616 and 2145 is that the fact that the client  
indicates HTTP/1.1 in the request advertises their support for  
persistent connections, and a HTTP/1.0 server (whether origin or  
proxy) may safely use a Content-Length delimited persistent response.

In my testing, two popular clients that advertise themselves as HTTP/ 
1.1 devices without sending a Connection token -- curl and Apple's  
Safari -- handle persistent connections from HTTP/1.0 servers very well.

However, the widely deployed Squid proxy cache <http://www.squid- 
cache.org/> does not behave in this manner; it requires a Connection:  
keep-alive header (or Proxy-Connection, but that's another  
discussion) in requests in order for them to persist.

In discussion on their mailing list*, it seems they believe that some  
HTTP/1.1 clients may not be able to handle a persistent connection to  
a HTTP/1.0 server that they didn't explicitly "ask for" with a  
Connection token.

What was the intent here? I can't imagine that it was the intent of  
the WG to require HTTP/1.1 clients to send Connection tokens in  
perpetuity for backwards compatibility.

Cheers,

* Starting at <http://www.squid-cache.org/mail-archive/squid-users/ 
200609/0546.html>; responses not yet in the archive.

--
Mark Nottingham
[email-masked]

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 20 Sep 2006 13:29:23 -0700

> proxy) may safely use a Content-Length delimited persistent response.

What is an HTTP/1.0 server?

> In my testing, two popular clients that advertise themselves as  
> HTTP/1.1 devices without sending a Connection token -- curl and  
> Apple's Safari -- handle persistent connections from HTTP/1.0  
> servers very well.
>
> cache.org/> does not behave in this manner; it requires a  
> Connection: keep-alive header (or Proxy-Connection, but that's  
> another discussion) in requests in order for them to persist.
>
> In discussion on their mailing list*, it seems they believe that  
> some HTTP/1.1 clients may not be able to handle a persistent  
> connection to a HTTP/1.0 server that they didn't explicitly "ask  
> for" with a Connection token.

Maybe, though it is more likely that the client will adapt its behavior
based on the contents of the response regardless of the request.  A  
better
question to ask is what happens to the connection if the client does not
understand keep-alive? If the content-length is accurate, the HTTP/1.1
client should close the connection and everyone will be happy. However,
the HTTP/1.1 client might be programmed or configured to assume that all
HTTP/1.0 servers do a lousy job of setting content-length (which was  
true
before keep-alive was deployed), so the client might just hold the
connection open waiting for more content until the server times-out.

I don't know of any clients that would do such a thing, just as I don't
know of any HTTP/1.1 clients that don't implement HTTP/1.0 keep-alive.
However, no clients should accept HTTP/1.0 keep-alive from a proxy
because of the well-known problem with dumb-forwarding proxies.

> perpetuity for backwards compatibility.

It was the intent of the WG that folks who claim to be using HTTP on
the Internet should be doing so with HTTP/1.1, instead of making
silly excuses not to.  Anyone still claiming to use HTTP/1.0 will have
to fend for themselves and test against every deployed system, since
the only standard is for HTTP/1.1.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 20 Sep 2006 14:16:09 -0700

On 2006/09/20, at 1:29 PM, Roy T. Fielding wrote:

>> HTTP/1.0 persistent connections are documented in RFC2068. One  
>> case that's not explicitly covered is when a HTTP/1.1 client sends  
>> a request to a HTTP/1.0 server without a Connection: keep-alive  
>> header.
>>

OK, I'll bite. One that claims to conform to RFC1945 by sending the  
version string "HTTP/1.0" in responses.

> silly excuses not to.

If the WG really believed that, why bother with making 1.1. backwards- 
compatible at all?

> Anyone still claiming to use HTTP/1.0 will have

Well, that would be nice, but anecdotal evidence says that Squid is  
one of the most widely-deployed proxies out there, and it's still  
advertising HTTP/1.0. Saying that Squid has to fend for itself  
ignores the fact that everyone else has to deal with Squid (as we see  
with the prevalence of HTTP/1.1 clients sending Connection: keep-alive).

--
Mark Nottingham
[email-masked]

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 20 Sep 2006 15:17:56 -0700

On Sep 20, 2006, at 2:16 PM, Mark Nottingham wrote:

>>> case that's not explicitly covered is when a HTTP/1.1 client  
>>> sends a request to a HTTP/1.0 server without a Connection: keep- 
>>> alive header.
>>>
>>> My reading of 2068, 2616 and 2145 is that the fact that the  
>>> client indicates HTTP/1.1 in the request advertises their support  
>>> for persistent connections, and a HTTP/1.0 server (whether origin  
>>> or proxy) may safely use a Content-Length delimited persistent  
>>> response.
>>

Nothing can claim to conform to an informational RFC by sending a  
protocol
version that was in wide use long before the RFC was published.  RFC  
1945
documents (the best I could) the most common and interoperable  
variations
on what people were claiming (at the time) as HTTP/1.0.

> If the WG really believed that, why bother with making 1.1.  
> backwards-compatible at all?

To deploy HTTP/1.1.  That was over eleven years ago.

> ignores the fact that everyone else has to deal with Squid (as we  
> see with the prevalence of HTTP/1.1 clients sending Connection:  
> keep-alive).

Squid is open source.  If someone thinks it shouldn't be ignored, then
they should update its ancient protocol stack to be compliant with
Internet standards.  In any case, Squid as a proxy that sends HTTP/1.0
responses is irrelevant to the question you asked, because HTTP/1.1
clients are required to ignore keep-alive from HTTP/1.0 proxy servers.
It is only when Squid is used as a gateway (reverse proxy) that the
question might apply here, and in that case the owner of that Squid
instance can fix it themselves.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 20 Sep 2006 16:04:30 -0700

That's the case I'm interested in.

Thanks,

--
Mark Nottingham
[email-masked]

---

From: Stefan Eissing <[email-masked]>
Date: Thu, 21 Sep 2006 09:21:28 +0200

Mark Nottingham schrieb:
> Apple's Safari -- handle persistent connections from HTTP/1.0 servers 
> very well.
>
> However, the widely deployed Squid proxy cache 
> <http://www.squid-cache.org/> does not behave in this manner; it 
> requires a Connection: keep-alive header (or Proxy-Connection, but 
> that's another discussion) in requests in order for them to persist.
>
I struggled with exactly the same issue when implementing a HTTP stack. 
Someone forced me with money to make it work with squid as it is widely 
deployed. Those were the days...

If you ask for intent, I think at the time of writing RFC 2068, HTTP/1.1 
clients could not assume that 1.0 servers derive persistent connections 
from the 1.1 version number alone, since server implementation might 
precede protocol definition.

If you ask for the best way to fix it, heroically lifting squid to 1.1 
seems like the best technical solution.

A patch might be much more easier and achieve world domination much more 
quickly. But still HTTP stacks like mine would need to inter operate 
with unpatched squids (and other servers, I continue to be amazed what 
beasts lurk in corporate <strike>closets</strike> intranets).

Cheers,

Stefan

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 21 Sep 2006 14:06:12 +0100

> deployed. Those were the days...

What is the difficulty with writing a client to work with Squid?

If you omit the "Connection: keep-alive", you'll just get a
non-keepalive connection, but it'll work fine.  Isn't that right?

-- Jamie

---

From: Stefan Eissing <[email-masked]>
Date: Thu, 21 Sep 2006 16:23:02 +0200

> non-keepalive connection, but it'll work fine.  Isn't that right?

With "make it work" I meant "working well". You are right that a 
correctly implemented client will just see the missing "keep-alive" and 
rightly assume that the connection will be closed. I was struggling with 
the "correctly implemented" and the "rightly assume" at that time and 
the variety of server behaviours did not exactly help.

It would be "nice" for an HTTP/1.1 client to achieve persistent 
connections without sending "keep-alive" as default - so I understood 
Mark's initial post. But, as I wrote, this is not the case nor do I see 
anything which can be done about it.

Cheers,

Stefan

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 22 Sep 2006 03:59:21 +0200

> anything which can be done about it.

The reason why Squid does not assume a HTTP/1.1 request also implies
persistent connection on a HTTP/1.0 response is because HTTP/1.0
persistent proxy connections is not defined in any standard and we are
not sure the client will do the right thing if it hasn't advertised
support for it.

When we move to HTTP/1.1 things will clear up consideably I think. Being
HTTP/1.0 is a bit of a mess. And the number of broken HTTP/1.1 servers
out there responding with chunked encoding to a HTTP/1.0 request or
broken clients capable of handling content-encoding: gzip but failing to
do so on HTTP/1.0 responses may actually finally force this to happen,
just not sure how to fit trailers into Squid in a sane manner.

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 21 Sep 2006 20:43:36 -0700

> just not sure how to fit trailers into Squid in a sane manner.

That is why trailers are optional -- just drop them for now.  They can
be useful for later responses from the cache, but I agree that it is
tricky to handle them within an intermediary.  It is better to release
an HTTP/1.1 handler now that simply drops trailers than it is to wait
for a more complicated solution.

....Roy

---

From: Jeffrey Mogul <[email-masked]>
Date: Fri, 22 Sep 2006 14:15:00 -0700

Well, actually, yes.  Which is to say, from what I can recall after so
many years, the key point is that it is far more important for clients
to get the right answers than to have all connections persistent (and I
say this as one of the people who most forcefully pushed the idea of
persistent connections).  We also (probably correctly) assumed that
HTTP/1.0 implementations would never entirely disappear.

If you want to break compatibility with HTTP/1.0, it's time to switch
to HTTP/2.x.  (Please don't invite me to join the HTTP/2.0 working
group.)  Beyond that, it might be worth re-reading RFC2145 ("Use and
Interpretation of HTTP Version Numbers") rather than re-deriving the
arguments captured there.

-Jeff

---

From: Paul Leach <[email-masked]>
Date: Fri, 22 Sep 2006 22:36:20 +0000

In HTTP/1.0, persistent connection was not the default, and didn't even
need to be supported.

Hence, unless an HTTP/1.1 client somehow knew a priori that it was
speaking to an HTTP/1.1 server, I don't see any way to escape the need
for HTTP/1.1 clients to send "Connection: keep-alive" if it wants to
support persistent connections whenever possible. And they will have to
keep doing so until it makes sense for them to believe that support for
HTTP/1.0 servers can be dropped.

(Well, almost. An HTTP/1.1 client doesn't have to send Connection header
-- if it doesn't, it just needs to be prepared for the server to
indicate the end of the entity-body by dropping the connection. However,
this would only be good engineering when the number of HTTP/1.0 servers
drops to a small enough level that not using persistent connections has
no significant impact on Internet load.)

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Jeffrey Mogul
Sent: Friday, September 22, 2006 2:15 PM
To: Mark Nottingham
Cc: [email-masked]
Subject: Re: HTTP/1.1 pconns to 1.0 servers

---

From: Stewart Brodie <[email-masked]>
Date: Tue, 26 Sep 2006 11:32:21 +0100

> With "make it work" I meant "working well". You are right that a correctly
> implemented client will just see the missing "keep-alive" and rightly
> assume that the connection will be closed. I was struggling with the
> "correctly implemented" and the "rightly assume" at that time and the
> variety of server behaviours did not exactly help.

My clients have always used a conservative approach to persistent
connections to work with origin servers and proxies, in an attempt to work
as well as possible.  If it's not talked to the proxy before (or recently
enough), it sends it an HTTP/1.0 request with a Connection:close header and
sees what happens; for non-proxies, it sends an HTTP/1.1 request with a
Connection:keep-alive header.  On the basis of that response, it will then
use HTTP/1.1 requests with or without connection headers, as appropriate (in
the case of the prehistoric NCSA 1.4.3 server, it reverts to HTTP/1.0
requests all the time, because that server sends faulty responses to
HTTP/1.1 requests)

It sounds like this would but it sounds like it'd be defeated by squid,
though?

-- 
Stewart Brodie
Software Engineer
ANT Software Limited

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 26 Sep 2006 11:19:30 -0700

Well, it depends; Squid (and most other proxies, for that matter) can  
be configured to be a normal (client-specified) proxy, a  
"transparent" a.k.a. interception proxy, or as a "reverse" proxy,  
a.k.a. "surrogate" or gateway.

In the first case, the behaviour you describe will cause Squid to  
close the connection.

But, in the latter cases, Squid will see the Connection: keep-alive  
and try to keep up.

Cheers,

On 2006/09/26, at 3:32 AM, Stewart Brodie wrote:

>> correctly

--
Mark Nottingham
[email-masked]