Subject: CONNECT command with message body

From: Adrien de Croy <[email-masked]>
Date: Tue, 05 May 2009 19:21:53 +1200

Hi all

I've come across a problem with an HTTP client which sends message 
bodies and Content-Length when using the CONNECT method.

Is it even legal for a CONNECT command to have a body?  If it does, what 
should the proxy do?  The client (a panasonic blue-ray player) breaks if 
the body is forwarded, however RFC2817 Sec 5.2 last para implies it is 
legal to have a body here.

Does anyone know of any other client that sends a body on CONNECT and 
relies on it?  It seems very odd to have a body there, since you don't 
even have a connection necessarily to the end server yet.  I'm loath to 
break the proxy to support a broken client.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 05 May 2009 19:30:52 +1200

sorry - correction, wasn't a panasonic player.  Apologies!

Adrien de Croy wrote:
> Is it even legal for a CONNECT command to have a body?  If it does, 
> what should the proxy do?  The client (a panasonic blue-ray player) 
> breaks if the body is forwarded, however RFC2817 Sec 5.2 last para 
> implies it is legal to have a body here.
>
> even have a connection necessarily to the end server yet.  I'm loath 
> to break the proxy to support a broken client.
>
> Regards
>
> Adrien
>

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 5 May 2009 09:36:51 +0200 (CEST)

On Tue, 5 May 2009, Adrien de Croy wrote:

> should the proxy do?  The client (a panasonic blue-ray player) breaks if the 
> body is forwarded, however RFC2817 Sec 5.2 last para implies it is legal to 
> have a body here.

Sorry, but can you clarify how you would read that paragraph to get that 
meaning out of it? (I'm not suggesting you think this, I just don't see how it 
can be intrepreted that way.)

Here's that last paragraph of RFC2817 5.2 with my comments:

    Like any other pipelined HTTP/1.1 request, data to be tunneled may be
    sent immediately after the blank line.

This seems to be like it's saying that everything after the blank line is 
tunneled. And it also mentions pipelining (in a funny way if you ask me), 
which only works for requests with no request-bodies.

    The usual caveats also apply: data may be discarded if the eventual
    response is negative, and the connection may be reset with no response if
    more than one TCP segment is outstanding.

Is it that "data may be discarded" part? It's not clear to me what data this 
refers to.

-- 

  / daniel.haxx.se

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 05 May 2009 21:49:45 +1200

For me S 5.2 is a bit vague.  It talks about "Other HTTP mechanisms can 
be used normally"... whatever they may be.  Would this other mechanisms 
include sending a message body?

The pipelining clause is very odd as well.  If tunnelling is used for 
any protocol other than HTTP (e.g. SSL/TLS which is an extremely common 
use-case) then pipelining doesn't make any sense. 

It talks about "data to be tunneled may be sent immediately after the 
blank line".

"the blank line" could be considered to be the separator between request 
headers and request body, and therefore anything in the request after 
the headers should be forwarded through to the target after the 
connection is made, or discarded if the connection fails.

It would be great if it explicitly mentioned whether a body is legal or 
not.  IMHO it makes no sense to have a body on CONNECT, all it does is 
save a small amount of time (the amount of time it takes the proxy to 
signal successful connection and the client to then send the subsequent 
data to the proxy).  Client <-> Proxy comms is usually at local LAN 
speeds for CONNECT.

Another point is that the CONNECT command has been around much longer 
than RFC2817.  Even longer than RFC2616, in which the method name was 
reserved.  It's a highly important and heavily used method, probably 2nd 
after GET for most web browsers configured to use a proxy.  Maybe even 
more used than GET if you count the spammers.  It therefore deserves a 
decent spec which covers the basics like message bodies.

I'm coming to the conclusion that message bodies on CONNECT are de-facto 
illegal though which helps my client with the blu-ray player.

Thanks

Adrien

Daniel Stenberg wrote:
> Sorry, but can you clarify how you would read that paragraph to get 
> that meaning out of it? (I'm not suggesting you think this, I just 
> don't see how it can be intrepreted that way.)
>
> This seems to be like it's saying that everything after the blank line 
> is tunneled. And it also mentions pipelining (in a funny way if you 
> ask me), which only works for requests with no request-bodies.
>
> Is it that "data may be discarded" part? It's not clear to me what 
> data this refers to.
>

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 5 May 2009 11:51:51 +0200 (CEST)

On Tue, 5 May 2009, Adrien de Croy wrote:

> Another point is that the CONNECT command has been around much longer than 
> RFC2817.  Even longer than RFC2616, in which the method name was reserved. 
> It's a highly important and heavily used method, probably 2nd after GET for 
> most web browsers configured to use a proxy.  Maybe even more used than GET 
> if you count the spammers.  It therefore deserves a decent spec which covers 
> the basics like message bodies.

Oh yes, I remember having to base all my initial CONNECT work on the early 
draft-luotonen-web-proxy-tunneling draft.

> illegal though which helps my client with the blu-ray player.

I agree with this conclusion.

-- 

  / daniel.haxx.se

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 5 May 2009 12:52:35 +0100

Daniel Stenberg wrote:
> meaning out of it? (I'm not suggesting you think this, I just don't see how 
> it can be intrepreted that way.)
> 

Pipelining works (in principle) for requests with request bodies too.

They just need to be idempotent requests, because you might not know
if the request succeeded when the connection is dropped.  Which is
probably why pipelining is never used with anything other than HEAD
and GET in practice.

The spec reads to me like a client can send CONNECT followed by data
to be tunnelled immediately if the CONNECT is successful.

> Is it that "data may be discarded" part? It's not clear to me what data 
> this refers to.

See RFC2616 8.2.2 "Monitoring Connections for Error Status Messages".

If you have a request with a body, and the response is an "error" (I
could never decide if 3xx counts as an error for this), the server is
allowed to send the error response before receiving the whole request,
and the client is allowed to abort the request when it sees the error
response.  If it does abort and it isn't a chunked request, the client
must close the connection at that point because the client and
server's request parsing is no longer synchronised.

If the intention is that you can send CONNECT followed by data to
tunnel immediately, and the CONNECT response is an error, it would be
just like other pipelined requests with body: because it isn't
chunked, the client must close the connection.

Note the server should _not_ close the connection immediately after
sending the error response, because the client will often not see the
error response if there is data in flight from the client to the
server, because that triggers a TCP reset.  The server could
half-close the TCP connection, but HTTP doesn't talk about half-closed
TCPs (it probably should).

-- Jamie

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 5 May 2009 14:00:44 +0200 (CEST)

> allowed to send the error response before receiving the whole request,

Thanks.

With your help I think I now lean towards this interpretation. The 'data may 
be discarded' only makes sense to me if data can be sent in the request-body 
for CONNECT. Without that, there is no data to discard.

I guess this only confirms that I am useless for trying to interpret that 
paragraph. I'll go back to my corner now.

-- 

  / daniel.haxx.se

---

From: Julian Reschke <[email-masked]>
Date: Tue, 05 May 2009 14:14:39 +0200

> probably why pipelining is never used with anything other than HEAD

I think Subversion uses it with REPORT and PROPFIND as well.

 > ...

BR, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 06 May 2009 00:28:46 +1200

> to be tunnelled immediately if the CONNECT is successful.
That's how I read it too, but it's difficult to know whether the 
implication is about tunneling raw data that follows the HTTP message 
that sets up the connection, vs the message which sets up the connection 
containing an entity-body.

If you have an entity-body on the CONNECT message, you could interpret 
this 2 ways.

1. ignore it:

a) The entity-body is part of the message
b) the purpose of the message is to establish a connection
c) therefore the entity-body is part of the establishment of the 
connection, not data to be relayed

2. forward it.

the language used seems to refer more to tunneled data being sent along 
with the message.  But including a Content-Length header makes the data 
an entity-body in this bluray player's case.

It's interesting, I've also been looking over a relatively recent 
discussion on this list about entity-body on responses to CONNECT.

There are so many bits of HTTP that really make no sense at all for the 
CONNECT method.  Such as chunking, transfer-encoding, content-encoding, 
entity headers etc.  It really bears little resemblance to HTTP at all.  
In fact once the connection is established, it is no longer necessarily 
HTTP.

I think some of the wording of RFC2817 was contemplating HTTP being used 
over the connection, which would explain some of the comments.  However 
I don't know of any cases of this (raw HTTP over CONNECT, rather than 
over TLS over CONNECT), and it's pretty much pointless since the normal 
proxy semantics would achieve the same thing.  You therefore use CONNECT 
when you want to use some protocol other than HTTP (e.g. SSL/TLS) over 
the connection.

So all in all, beefing up 2817 could be useful, or some sort of 
clarification somewhere.  I'm presuming CONNECT is out of scope for HTTPbis.

Adrien

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 5 May 2009 14:38:45 +0200 (CEST)

On Wed, 6 May 2009, Adrien de Croy wrote:

> over the connection, which would explain some of the comments.  However I 
> don't know of any cases of this (raw HTTP over CONNECT, rather than over TLS 
> over CONNECT), and it's pretty much pointless since the normal proxy 
> semantics would achieve the same thing.  You therefore use CONNECT when you 
> want to use some protocol other than HTTP (e.g. SSL/TLS) over the 
> connection.

Pointless if the behavior is identical to everyone, yes.

I can think of many situations where you'd either just (A) CONNECT and send 
plain HTTP GET through, or the more complicated setup: (B) CONNECT, tunnel ssh 
over to a remote proxy, and then send HTTP GET over that tunnel, to avoid 
local proxy enforcements/filters/logs/prying eyes.

Of course I'm just blindly guessing here that proxies will treat the case (A) 
differently than a normal GET in such aspects. I know case (B) is widely used 
though.

-- 

  / daniel.haxx.se

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 May 2009 13:09:08 +1200

Did we get anywhere with this issue?

I just made some mods to the proxy to ignore any entity body on CONNECT 
(e.g. not forward it to the server).  This then leads to issues around 
understanding when the request message is complete.

Fundamentally, if a message may have an entity, then the message is no 
longer delineated by the empty line that terminates the headers.

If no Content-Length field is present, and no Transfer-Encoding 
specified, then presumably for any request there will be no entity, 
since the client would need to disconnect to signal the end of the 
entity body.

Is this a fair assumption to make?

Adrien

Daniel Stenberg wrote:
>> I think some of the wording of RFC2817 was contemplating HTTP being 
>> used over the connection, which would explain some of the comments.  
>> However I don't know of any cases of this (raw HTTP over CONNECT, 
>> rather than over TLS over CONNECT), and it's pretty much pointless 
>> since the normal proxy semantics would achieve the same thing.  You 
>> therefore use CONNECT when you want to use some protocol other than 
>> HTTP (e.g. SSL/TLS) over the connection.
>
> I can think of many situations where you'd either just (A) CONNECT and 
> send plain HTTP GET through, or the more complicated setup: (B) 
> CONNECT, tunnel ssh over to a remote proxy, and then send HTTP GET 
> over that tunnel, to avoid local proxy 
> enforcements/filters/logs/prying eyes.
>
> Of course I'm just blindly guessing here that proxies will treat the 
> case (A) differently than a normal GET in such aspects. I know case 
> (B) is widely used though.
>

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 May 2009 16:15:32 +1200

p.s.  more information on this.

As part of the general theory that CONNECT or its response message may 
contain an entity-body, I did some testing with including a 
Content-Length header in the response message.

This caused a lot of problems for some browsers.

1. Goole chrome crashed immediately.
2. Firefox reported that it was still waiting for transfers to complete 
even though everything had been sent through. 
3. IE 7, ok.

in the case of Chrome and Firefox, the problems went away as soon as the 
Content-Length header was removed from the connect response.

I view this as a bug in FF and Chrome, however it's odd that these 
browser at least consider that it's illegal to have an entity body on a 
CONNECT reponse, and it makes me therefore highly reluctant to put the 
header in.

Adrien

Adrien de Croy wrote:
> I just made some mods to the proxy to ignore any entity body on 
> CONNECT (e.g. not forward it to the server).  This then leads to 
> issues around understanding when the request message is complete.
>
>> I can think of many situations where you'd either just (A) CONNECT 
>> and send plain HTTP GET through, or the more complicated setup: (B)

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 18 May 2009 05:16:37 +0100

> CONNECT reponse, and it makes me therefore highly reluctant to put the 

Does that include a "Content-Length: 0"?
Not that there's any point in it.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 18 May 2009 05:17:40 +0100

> since the client would need to disconnect to signal the end of the 

Disconnecting to signal the end of the body is only allowed for
responses.  For requests, it would need to be a TCP half-close and
that is not mentioned at all in HTTP, and probably not supported by
most/all servers.

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 May 2009 16:24:18 +1200

yes - Content-Length: 0

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 May 2009 16:26:11 +1200

> that is not mentioned at all in HTTP, and probably not supported by
that's the basis I've been operating under.  Was just wondering if it 
were explictly prohibited by the spec or not.

Since I didn't want to presume there can be no application where it may 
be valid to send an entity body in a request and then disconnect (i.e. 
if not interested in the response).

Adrien

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 18 May 2009 05:34:48 +0100

> if not interested in the response).

Which reminds me.

Real HTTP servers like Apache must not close the connection to signal
the end of a response.  They *half-close* it using the shutdown()
socket function, and then sit for a while discarding any subsequent
request which might have been pipelined.

This is necessary, because if any further data is received by a
*fully*-closed TCP socket on the server, the servers's TCP will reset
the connection and send a RST packet, which causes the server to abort
*sending* any data which it hasn't yet transmitted successfully, and
cause most clients to abort *receiving* the response, and sometimes
abort sending data it has received up to the receiving application.

In other words, HTTP pipelining requires servers to half-close instead
of fully-closing and then do a "lingering close" dance, if they send a
response whose end is signalled by closing the connection.

If they don't do that, responses can be lost depending on transient
network conditions.

This is necessary even if the reponse has "Connection: close", because
the client may still have sent further request bytes before it sees
any part of the response.

Since this is required for reliable HTTP pipelining over TCP, it
should be mentioned somewhere in an HTTP specification.  It isn't in
RFC 2616 anywhere that I could see.

-- Jamie

---

From: Adam Barth <[email-masked]>
Date: Sun, 17 May 2009 21:37:32 -0700

On Sun, May 17, 2009 at 9:15 PM, Adrien de Croy <[email-masked]> wrote:
> contain an entity-body, I did some testing with including a Content-Length
> header in the response message.
>

I was not able to reproduce this crash.  How did you get Google Chrome to crash?

Thanks,
Adam