Subject: CONNECT message including tunneled data

From: Robert Siemer <[email-masked]>
Date: Tue, 29 Jan 2008 18:16:50 +0100

Hello!

Some time ago we had the discussion about the CONNECT responses and that 
they should carry a Content-Length header if the result code is 200.

We had the example of "Content-Length: 0" but what about letting the
tunneled data be part of the request/response?

That raises the question: May a successful response overtake the 
request? The spec has only a POST-rejecting example which does that. And 
what about if the response measures up to the request? For example an 
"echo POST", which can answer as far as the request is. - If the request 
terminates, the response will, too.

I see no restrictions in HTTP for that behaviour, does anybody else?

CONNECT could be such a "catch up" method.

Regards,
		Robert

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 30 Jan 2008 07:55:41 +1300

> they should carry a Content-Length header if the result code is 200.
rules for whether there should be a Content-Length field are based 
around whether there is an entity body or not.  No entity body, no header. 

With CONNECT, the purpose is to establish a TCP connection.  There's no 
guarantee what protocol will be used over the connection. 

I don't see how having an entity body on request or response can help 
the goal of setting up a connection.  It is obviously subsequent data.  
In no other place in HTTP do we consider sending entity data for a 
subsequent request on an initial message, and I would be opposed to 
adding that capability now.

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 31 Jan 2008 08:36:44 +0000

> I see no restrictions in HTTP for that behaviour, does anybody else?

I see no restrictions in HTTP, provided nobody minds being unable to
send an error response if a later part of the request causes one.

I investigate actual client behaviour a few years ago, because I
was interested in making a server which passes the request data
through a filter and streams the filtered result back as a response.
Also, for a server which returns a progressive HTML page showing
"upload progress".

What I found is many different client behaviours:

  1. A few clients can handle overlapping non-error response and requests.

  2. Many clients will not read the response until they have
     transmitted the whole request - at least enough that the request is in
     the TCP/IP system.  (Therefore the difference is only detected with
     large enough requests).

  3. Some clients will abort the request on receiving part of the
     response.  This is recommended for error responses, but these
     clients do it for "200 OK" as well.  My notes say Mozilla 1.2 but
     I didn't test it myself, plus some version of cURL.

  4. Most clients distinguish error responses: abort if it's an error
     response, and either behaviour 1 or 2 for non errors.

This means a general purpose server should not opportunistically
transmit the response early, as some clients will abort the request.
However, if it can determine that the client won't abort - or if such
clients are long gone - servers can transmit the response but must be
prepared to avoid deadlock by reading the whole request while the
response transmission is blocked.

For "upload progress" in a web browser, I found that people are using
a different method: separate IFRAMEs for the upload and progress bar,
to put the streamed response on a different HTTP connection than the
upload.  Even then, some web browsers will not display anything while
an upload is in progress (very annoying for large uploads).

> CONNECT could be such a "catch up" method.

I agree with you in principle: I always thought that would be cleanest
semantic for CONNECT, and remove a special case in the message
boundary rules.  And it would mean a CONNECT could end without closing
the connection.

Provided the client and server agree, I don't see a problem with
overlapping request and response in the HTTP specs, and potentially
unbounded streaming.  You can already do it today with POST.

(In fact I already do something like that in the two-way + multiplex
extension of HTTP used in my own application.)

However, keep in mind it requires the client and server to handle
chunked and other transfer-encodings (unless they agree not to use
any), which means they cannot simply set up the CONNECT and then step
out of the way, passing the socket directly to an application (plus
the initially buffered data).

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 31 Jan 2008 08:52:33 +0000

Adrien de Croy wrote:
> subsequent request on an initial message,

I don't agree that it's "obviously subsequent data".  I think that
depends on the application and on your point of view.

Consider an app which uses CONNECT to establish a temporary Telnet
session.  Following CONNECT, the client transmits a single unix shell
command.  The response is the output of that command, and then the
connection closes.

Similar patterns are used with other request/response protocols like
rsync-over-CONNECT (sends a command to fetch a file, gets the file
back, or sends a command to get a file listing, gets the listing
back).

>From one point of view, the CONNECT is a separate operation.

But it's equally reasonable to see the CONNECT as just the initial
part of an application request, which happens to be wrapped in a
CONNECT as its mechanism.

The operational consequences of CONNECT that I see are:

   - One extra TCP/IP round trip, because you have to wait for the 2xx
     response before transmitting the subsequent request data.

   - Cannot re-use the HTTP connection after the application protocol
     has finished with it.

   - Combination of the above: cannot pipeline multiple application
     requests, if they need to use separate connections.  (See this
     already with rsync-over-CONNECT).

The logical consequence of CONNECT that I see is:

   - It's one more special case in the HTTP message boundary rules,
     which was never necessary but it's history is quite understandable.

When the operation effects are significant, it's possible to use POST
with overlapping request/response to get better performance, which
seems to be allowed, but only works with known client/proxy/server
combinations, and of course isn't a standard method.

> and I would be opposed to adding that capability now.

Certainly I would be opposed to changing CONNECT now - it would break
everything :-)

[ However, if there's any interest in developing "next generation"
HTTP (which ought to have gracefully degrading long message
multiplexing, response reordering, and two-way requests), I would
suggest that two-way streaming _inside_ messages would be quite a
natural fit for that. ]

-- Jamie