Subject: Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers

From: "Carl Kugler/Boulder/IBM" <[email-masked]>
Date: Thu, 2 Nov 2000 12:07:15 -0700

I stand corrected.  But then, why was Content-Length added in HTTP/1.0 and
Tranfer-Encoding: chunked in HTTP/1.1?

     -Carl

"Fred Bohle" <[email-masked]> on 11/02/2000 11:27:24 AM

To:   Carl Kugler/Boulder/IBM@IBMUS
cc:   James Lacey <[email-masked]>, http-wg
      <[email-masked]>
Subject:  Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers

Fred Bohle@NEON
11/02/2000 12:27 PM

     We seem to be diverging into TCP coding.  A read will return zero
length
when the other end has issued a normal close (and all the data has been
read).
A read will return -1 when the connection is ReSeT, or there is a
connection time-out
of any sort.  So the server can too tell the difference between end of data
and
a connection failure.

Fred

From: Carl Kugler/Boulder/IBM <[email-masked]> on 11/02/2000 11:59 AM

To:   James Lacey <[email-masked]>
cc:   http-wg <[email-masked]>

Subject:  Re: Of HTTP/1.1 persistent connections and TCP Keepalive timers

...
>
>For example, suppose the  server sends back an HTTP response to the client
>that does not have a Content-Length: header field and that it is not
>chunked.
>
>Then the only way the client knows that it has read the
>entire response off of the pipe is when the server closes the connection.
>When the server closes the connection the client will receive a
>zero-byte read which is socket layer's indication that the pipe
>is broken.
>
This is not good.  If this is the server's normal behavior, the client has
no way to distinguish a dropped connection from end of file.  So the client
can never be sure it received an entire message.

If this is the server's behavior for error conditions, this is still not
good unless the server either waits for the entire request (could be a
humongous POST and/or a very slow connection), or only closes one half of
the connection (which, BTW, is impossible in Java, except maybe in the
latest releases).  Otherwise, the client might get a RST while transmitting
the request, and will then never see the error response.

...
>
>I've done some snoops on browsers that are GETting HTML pages with
>lots of embedded links that are in the same realm as the original HTML
>page.
>
>Despite this obvious opportunity to take advantage of persistent
connections
>the browser opens  a connection for each subsequent GET.
>
I was looking at IE's (version 5 somthing) traffic yesterday, and it seems
to send two requests per connection.

     -Carl

---

From: James Lacey <[email-masked]>
Date: Thu, 02 Nov 2000 13:13:13 -0600

> Tranfer-Encoding: chunked in HTTP/1.1?

Content-Length was added so that persistent connections would be possible.

Without a content length, neither the client or the server would know
where the message ended. You have to realize that TCP is a stream and
it is up to the users of the stream to impose some kind of framing protocol
onto the stream.

Transfer-Encoding: chunked was added so that servers could return
possibly large dynamic content in chunks, as it was generated, so that
the server did not have to buffer up all of the dynamic content before
sending it back. With chunking you can also have persistent connections
w/o the need of a Content-Length header field.

One curious think about chunking that I do not understand is that
HTTP proxies are required to move the chunked (any?) encoding before
forwarding the response.

---

From: Patrick McManus <[email-masked]>
Date: Thu, 2 Nov 2000 15:17:14 -0500

> HTTP proxies are required to move the chunked (any?) encoding before

transfer encodings (of which chunk is one) are hop to hop.. so they
have to be theoretically removed.. but if the outgoing encoding is the
same as the incoming encoding that phsyical operation can certainly be
optimized away. The requirement is that the outgoing encoding meets
the requirements of the request received by the proxy.. so if the
proxy gets a 1.0 request it will need to remove 'chunked' and
terminate the message in some other fashion. 

That other way is probably to buffer it and add a CL, or more likely
terminate it by closing the connection.

which leads me to a pet peeve of mine.. 2616 prohibits the sending of
CL when a non identity transfer encoding is used.. consider the case I
just used of a 1.1 proxy with a 1.1 origin-server and a 1.0
user-agent.. the proxy receives a 1.1 chunked (and something else non
identity) response but it's only choice to send it downstream is to
buffer it in a store and fowrard fashion to figure out the content
length or close the connection w/ EOF which would rob of it of the
chance to use 1.0 style keepalives... 

admittedly some folks think 1.0 keepalives ought to be burned.. but by
that argument all responses to 1.0 reqs should lack CL's and be TCP
terminated, right?

if the content length is known by the original sender it seems a
little odd to lose this information if for no reason other than use in
resource allocation or UI thermometer widgets. Certainly with dynamic
content the server probably doesn't know the CL anyhow at header time,
but what if it's just sending say trasnfer-encoding: deflate, chunked
??

-P

---

From: [email-masked] (Julien Pierre)
Date: Fri, 03 Nov 2000 15:05:58 -0800

Hi,

> w/o the need of a Content-Length header field.

"Transfer-Encoding chunked" also works on the client side, and would allow a
client to send dynamic content in its request. If both the client and the server
are doing chunking, it makes streaming applications possible over a standard HTTP
connection (eg: VPN, multimedia, etc).

I'd like to point out that with iPlanet Web Server 4.1, you can easily write such
applications in NSAPI and all the chunking issues are transparently taken care of
for the application developer. The only requirement is of course that the client
must be HTTP/1.1 compliant and support chunking. The client counterpart of such a
server-side streaming application could be a Java applet or browser plugin.

> HTTP proxies are required to move the chunked (any?) encoding before

It seems that such a proxy would break the type of application mentioned above
... Full duplex would be required for such an application. This is something that
is not very well addressed in current HTTP specs (hint: this is just a trick to
get immediate attention from the RFC2616 owners :)).

Traditionally, with the content-length indicated in requests and responses, many
browsers and servers have used a synchronous model, where clients try to submit
the whole request, then only after it's complete, begin read the server response.
Many servers operate similarly and read the entire request before starting the
reponse. This is generally true for small GET requests, which the server can
afford to buffer. However, for POST, the server will not buffer since that would
make the memory footprint of the server very high when multiplied by the number
of connections. So the HTTP server will just act as a sort of intermediate and
use a relatively small buffer for the POST data, then pass it gradually to the
consumer of the request - whether it's a CGI, servlet, or NSAPI application ...

I have found that HTTP clients typically break if the server tries to respond too
early to a request, before it is complete. For example, this can happen with a
large POST request to a non-existing object. The server detects early on that the
URI doesn't point to a valid object on the server, and will try to respond
immediately with a 404 response. This is done before it reads the entire amount
of POST data ; since that can be very large and slow to read.

At that point however, the client is still submitting the large POST data ; and
the connection ends up being reset since both sides are sending and the client
isn't trying to read until its write loop is over. In iWS, we have some
workarounds that I consider ugly, including trying to read more POST data on such
erroneous large requests and just discarding it, as well as doing shutdown before
close, just to make sure the client doesn't see a "connection reset" error. This
doesn't work 100% of the time since we set some limits, but obviously we aren't
going to read megabytes of data if the request is too large, or wait a long time
on that socket, since that could be a denial of service attack - eating up socket
descriptors and in some cases also holding up a thread in the server. It would be
really good if the HTTP spec would address such cases and require both client and
servers to try both sides of the socket before closing/shutting the connection
down ... Such workarounds would not be necessary and the servers could just
peacefully shut down the connection after reporting an error, and assume that the
client in fact was really able to receive that error - not causing a "connection
reset".

--
The network is the computer, and it's down

---

From: [email-masked] (Ruslan Belkin)
Date: Fri, 03 Nov 2000 15:16:25 -0800

> early to a request, before it is complete. For example, this can happen with a

that's cuz the right thing to do would be to send expect continue along with POST
then wait for continue then send the data. But many servers are broken 2 with respect
to this,

Ruslan

---

From: "Joris Dobbelsteen" <[email-masked]>
Date: Sun, 5 Nov 2000 17:46:10 +0100

Content-length in HTTP/1.0 is not directly needed, except when using the addional RFC purposing persistent connections. It gave the client a indication of the length of the file. This is nice when downloading a large file, you know that you have all the data received (in case of line failures)...

HTTP/1.1 added chunked encoding in order to let HTTP implementations benefit when sending content where the length is not known (like dynamic content or an real-time video/measurement) and benefit from the persistent connections. The next request (e.g. a picture) can be requested without waisting time to open a new connection.

That looks like the most logic explaination for me....

- Joris

> -----Original Message-----
> From: Carl Kugler/Boulder/IBM [mailto:[email-masked]]
> Sent: Thursday, 02 November 2000 20:07
> To: Fred Bohle
> Cc: James Lacey; http-wg