Subject: NEW ISSUE: message-body in CONNECT response

From: Dan Winship <[email-masked]>
Date: Mon, 26 Nov 2007 14:21:58 -0500

As implemented in the real world, a successful response to a CONNECT 
request does not include a message-body. But this isn't stated in RFC 
2817, and is actually forbidden by RFC 2616.

draft-luotonen-web-proxy-tunneling said:

     Example of a response:

           HTTP/1.0 200 Connection established
           Proxy-agent: Netscape-Proxy/1.1

           ...data tunnelled from the server...

     After the empty line [that follows the message-headers], the proxy
     will start passing data from the client connection to the remote
     server connection

but RFC 2817 (5.3) removes the lack-of-message-body implication:

     Any successful (2xx) response to a CONNECT request indicates that
     the proxy has established a connection to the requested host and
     port, and has switched to tunneling the current connection to that
     server connection.

And RFC 2616 (4.3) requires a 200 response to a CONNECT to have a 
message-body anyway:

     All responses to the HEAD request method MUST NOT include a
     message-body, even though the presence of entity-header fields might
     lead one to believe they do. All 1xx (informational), 204 (no
     content), and 304 (not modified) responses MUST NOT include a
     message-body. All other responses do include a message-body,
     although it MAY be of zero length.

So to fix things, RFC 2616 4.3 should be updated to include "A 
successful (2xx) response to a CONNECT request MUST NOT include a 
message-body." And if 2817 is in-scope for 2616bis then the fact should 
probably be reiterated there too.

-- Dan

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 26 Nov 2007 20:29:39 +0100

>           ...data tunnelled from the server...

Do you have any information on how clients treat the response if it has
a Transfer-Encoding or Content-Length header? What if the response is
not a 2xx one and includes (or lacks) these headers?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 27 Nov 2007 08:47:13 +1300

Actually there are (I'm fairly sure) cases where a 200 response comes 
back to a CONNECT request with an HTML body.

For instances proxies that return a challenge page for authentication.

In this case, surely the message that is the response to the CONNECT 
command is not part of the tunneled data, and so any body on that 
message should likewise not be counted as tunneled data, but rather part 
of the response.

Maybe the requirement should be that there must be a Content-Length set 
(regardless of whether there is a non-zero length body), or other more 
normal methods of message delineation before entering the tunneling phase.

Regards

Adrien

Dan Winship wrote:
> message-body." And if 2817 is in-scope for 2616bis then the fact 
> should probably be reiterated there too.
>
> -- Dan
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Dan Winship <[email-masked]>
Date: Mon, 26 Nov 2007 15:56:41 -0500

> For instances proxies that return a challenge page for authentication.

That won't work. (Just tried it with Firefox.) I think you're thinking 
of the case where the proxy returns a 3xx response, to *redirect* the 
client to a login page.

> normal methods of message delineation before entering the tunneling phase.

That would be a change to the way CONNECT works though. My suggestion 
was just updating the spec to match how things already work.

-- Dan

---

From: Dan Winship <[email-masked]>
Date: Mon, 26 Nov 2007 16:19:40 -0500

> not a 2xx one and includes (or lacks) these headers?

I've looked more at what proxies do than what clients do, but given that 
some (most? all?) proxies return message bodies on 407, 403, etc 
responses to CONNECT, I assume that clients must DTRT in that case.

Some clients might use the rule "a CONNECT response has a body if and 
only if it has a Transfer-Encoding or Content-Length, etc" rather than 
"a CONNECT response has a body if and only if it has a non-2xx status 
code", but this wouldn't cause much in the way of problems; they'd just 
fail to see the response body if the server used the 
delimited-by-closing-the-socket type of response.

-- Dan

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 27 Nov 2007 04:55:45 +0100

> probably be reiterated there too.

Agreed. It's too late to fix this method in any other way.

Recommending the use of 204 would be clearer, but I suspect there is
many implementations failing seeing anything else than 200.. (Squid
being one of them.. now fixed)

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 27 Nov 2007 05:00:04 +0100

> being one of them.. now fixed)

Correction: Squid did not fail this.. was just a little tired..

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 27 Nov 2007 12:32:41 +0000

> not a 2xx one and includes (or lacks) these headers?

I can say for sure that some clients* using CONNECT just check the
response code, and if it's 2xx they read until the first blank line,
then assume what follows is the tunnelled data.  Such implementations
don't parse the headers at all.

* - Not HTTP clients as such, but clients of other protocols which
    have an option to connect through a HTTP proxy using CONNECT.

-- Jamie

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 27 Nov 2007 11:03:04 -0800

>     have an option to connect through a HTTP proxy using CONNECT.

The standard requires an empty body on a non-closed connection to be
indicated by one of the two message length indications (CL or TE  
chunked).
In this case, the obvious solution is to require "Content-Length: 0" be
included in the header fields of the 200 response.  It doesn't matter
if some clients ignore that field.  What matters is that we don't add
more method-specific parsing of response bodies.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 28 Nov 2007 10:08:05 +1300

I agree with Roy about maintaining correctness of message semantics.  We 
can't have the CONNECT method being the only one where the response 
doesn't need to be a well-formed, _complete_ HTTP message.

However, we do use in some cases a message body on a 2xx response to a 
CONNECT to present a Java based login page.  Some clients handle this 
better than others.  We've been doing this for years and had few issues 
with it. I'd actually love to deprecate this behaviour, but our 
customers want it.

The current spec in my view allows a body, we're talking about removing 
that functionality.

---

From: Eric Lawrence <[email-masked]>
Date: Tue, 27 Nov 2007 13:35:40 -0800

Wouldn't "Connection: Close" be more appropriate than "Content-Length: 0", as the latter implies that no content follows, while clearly that's not correct if the response headers precede blobs of HTTPS traffic?

A proxy that supports HTTPS tunneling is going to use Connection: close semantics anyway, right?

Vis-à-vis the idea of responding to a CONNECT request with a HTML 2xx "login" page, it may be worth mentioning that this does not work in IE6 or IE7. A user-agent which does support such responses must be very careful to ensure that the security context of the returned content is corrected to reflect its insecure nature.

Eric Lawrence
Program Manager - IE

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Roy T. Fielding
Sent: Tuesday, November 27, 2007 11:03 AM
To: Jamie Lokier
Cc: Bjoern Hoehrmann; Dan Winship; [email-masked]
Subject: Re: NEW ISSUE: message-body in CONNECT response

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 27 Nov 2007 14:21:52 -0800

On Nov 27, 2007, at 1:35 PM, Eric Lawrence wrote:

> Wouldn't "Connection: Close" be more appropriate than "Content- 
> Length: 0", as the latter implies that no content follows, while  
> clearly that's not correct if the response headers precede blobs of  
> HTTPS traffic?

No, close tells the client to close the connection after the message.
"Content-Length: 0" tells the client that this HTTP message ends
after the CRLFCRLF.  What follows after that message is not HTTP,
but it is still the same connection.

> close semantics anyway, right?

Maybe. It is going to drop into relay mode, which may be an entirely
different piece of logic than the bit that decides when to close at
the end of a message.  It depends on how the proxy forwards messages.

> Vis-à-vis the idea of responding to a CONNECT request with a HTML  
> 2xx "login" page, it may be worth mentioning that this does not  
> work in IE6 or IE7. A user-agent which does support such responses  
> must be very careful to ensure that the security context of the  
> returned content is corrected to reflect its insecure nature.

Yep, that is why the message-body is required to be empty.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 28 Nov 2007 12:44:13 +1300

Roy T. Fielding wrote:
>> Vis-à-vis the idea of responding to a CONNECT request with a HTML 2xx 
>> "login" page, it may be worth mentioning that this does not work in 
>> IE6 or IE7. A user-agent which does support such responses must be 
>> very careful to ensure that the security context of the returned 
>> content is corrected to reflect its insecure nature.
>
I'm not sure about the significance of "security context" in this case, 
since this is a message from the proxy to the UA.

In any case, there are numerous security issues with CONNECT. The main 
one being it's very commonly used to bypass restrictions or scanning.  
But that's a proxy policy issue (albeit non-trivial to solve).

So I guess then a proxy wishing to prevent a CONNECT command from 
succeeding without prior established credentials, but which doesn't have 
the option of using an HTTP auth mechanism can only really send back a 
401, and the user must establish credentials some other way first.

Adrien

---

From: Dan Winship <[email-masked]>
Date: Tue, 27 Nov 2007 19:46:42 -0500

Roy T. Fielding wrote:
> indicated by one of the two message length indications (CL or TE chunked).

I agree that it's stupid to have a special-case for CONNECT parsing, but
that's *already* how implementations are required to behave. A client
that tries to parse a CONNECT response according to the rules currently
given in RFC 2616 will be unable to create a tunnel. Requiring proxies
to add "Content-Length: 0" won't help, because clients will still want
to be able to deal with pre-2616bis proxies, which will require the
special case.

-- Dan

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 27 Nov 2007 17:06:33 -0800

> to be able to deal with pre-2616bis proxies, which will require the

No, the current 2616 standard requires the CL:0, as did 2068.
I don't care if some implementations don't bother to check --
I know what the requirement is for HTTP.

....Roy

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 28 Nov 2007 04:17:54 +0100

* Robert Siemer wrote:
>As of RFC2616 a 200 does not mean the response is going to have a 
>body, so why enforce that for CONNECT? The proxy has to be aware of that 
>method anyway for it to work.

It is not possible, under RFC 2616, for a 200 response to have no body,
unless it is a response to a HEAD request (see the last part of 4.3).
You can only have a zero-length body and need to indicate that using
Content-Length or Transfer-Encoding.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Robert Siemer <[email-masked]>
Date: Wed, 28 Nov 2007 02:57:01 +0000

> more method-specific parsing of response bodies.

I can tell you, the RFC2616 is definitively not written to make 
implementators life easy, so why draw the "method-specific parsing" 
argument?

Robert

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 28 Nov 2007 11:09:11 +0000

> more method-specific parsing of response bodies.

For CONNECT I don't think it is really necessary.  Without CL or TE
chunked, the following received data will be seen by some message
parsers as a body of indeterminate length (until the connection
closes), which is fine, they can pass it to applications, through
proxies etc. without a special case.

-- Jamie

---

From: Robert Siemer <[email-masked]>
Date: Thu, 29 Nov 2007 00:20:45 +0100

> Content-Length or Transfer-Encoding.

4.3. Message Body

   The message-body (if any) ...

   [three other paragraphs]

   For response messages, whether or not a message-body is included with
   a message is dependent on both the request method and the response
   status code (Section 6.1.1).  All responses to the HEAD request
   method MUST NOT include a message-body, ... All 1xx (informational),
   204 (no content), and 304 (not modified) responses MUST NOT include
   a message-body.  All other responses do include a message-body, ...

I read the last sentence as "all other respones - defined in this spec - do 
include a message-body..." Otherwise I wouldn't call the presence of a 
message-body beeing "dependent on both the request method and the response 
status code". In that case it would be better to say that it depends only on 
the status code and HEAD being the body-less GET exception.

Especially if the spec aims to include general "unknown method, what to do" 
information.

Robert

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 28 Nov 2007 15:58:21 -0800

On Nov 28, 2007, at 3:20 PM, Robert Siemer wrote:
> include a message-body..."

You read it wrong.  The only possible way that a proxy can forward
the response to a method it does not understand is if no method
other than HEAD (for legacy reasons ONLY) is allowed to change the
message delimiting rules.  This applies to anything that uses or
extends HTTP and will not be changed.

As it happens, CONNECT can't be forwarded by proxies without having
understood the method (because it uses a unique request format).
That allows non-compliant behavior to be ignored, but it is still
non-compliant.

....Roy