Subject: HTTP/1.1 CONNECT request without Host header

From: Adrien de Croy <[email-masked]>
Date: Wed, 17 Jun 2009 10:26:57 +1200

I recently updated our proxy to reject all HTTP/1.1 messages from 
clients that lack a Host header.

This has been found to break a number of clients which use the CONNECT 
method.

So it's going to be a necessity to relax this particular requirement (in 
our proxy) at least for CONNECT.

Strictly speaking this is a violation of RFC2616 correct?  I'm sure it 
states all 1.1 messages MUST include a Host header, and a server 
receiving such a message MUST respond with 400.

May need to make reference to CONNECT for this, since a number of (non 
browser) clients seem to be affected.  In fact maybe in HTTPbis it would 
pay to put a bit more about CONNECT in, even if only a reference to the 
RFC for it.  Currently its status in RFC2616 is limited to only 
reserving the method name.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: David Morris <[email-masked]>
Date: Tue, 16 Jun 2009 16:47:51 -0700 (PDT)

I would note that in a proxy request, the host header is redundant. My
recollection was that host would only be required for direct origin server 
requests.

I would have objected loudly if I had noticed this as a requirement when 
the host header was introduced.

I don't have time to play RFC lawyer, but if this is the actual 
requirement, we should consider relaxing it for HTTPbis's output.

Dave Morris

On Wed, 17 Jun 2009, Adrien de Croy wrote:

>
> I recently updated our proxy to reject all HTTP/1.1 messages from clients 
> that lack a Host header.
>
> So it's going to be a necessity to relax this particular requirement (in our 
> proxy) at least for CONNECT.
>
> Strictly speaking this is a violation of RFC2616 correct?  I'm sure it states 
> all 1.1 messages MUST include a Host header, and a server receiving such a 
> message MUST respond with 400.
>
> browser) clients seem to be affected.  In fact maybe in HTTPbis it would pay 
> to put a bit more about CONNECT in, even if only a reference to the RFC for 
> it.  Currently its status in RFC2616 is limited to only reserving the method 
> name.
>
> Regards
>
> Adrien
>
> --

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 16 Jun 2009 17:07:29 -0700

On Jun 16, 2009, at 3:26 PM, Adrien de Croy wrote:

> This has been found to break a number of clients which use the  
> CONNECT method.

What clients?

> So it's going to be a necessity to relax this particular  
> requirement (in our proxy) at least for CONNECT.

Unlikely.  Apache always responds to such a request with 400.
AFAIK, all the other origin servers do as well, so even if the
request is passed by your proxy it is going to fail on the origin.

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 16 Jun 2009 17:13:47 -0700

On Jun 16, 2009, at 4:47 PM, David Morris wrote:

> recollection was that host would only be required for direct origin  
> server requests.

It is not redundant because the Host will be passed through
by an old 1.0 proxy that did not know (at the time) about Host.

> I would have objected loudly if I had noticed this as a requirement  
> when the host header was introduced.
>

The requirement was imposed by the IESG via the apps area AD
at that time.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 17 Jun 2009 12:28:58 +1200

Hi

>> This has been found to break a number of clients which use the 
I only have a couple of reports since this wasn't in general release - a 
mail.ru client and FTP commander using CONNECT.

However, on further reading of RFC2616, it appears David Morris is 
correct, that a Host header is only a MUST requirement if the 
Request-URI is not an AbsoluteURI, and MUST be ignored if it is.  This 
doesn't cover the case where Request-URI is authority (as per 
CONNECT)... so a strict interpretation is that CONNECT requires host, 
although I imagine this wasn't the intent, since authority contains the 
required information.

>
>> So it's going to be a necessity to relax this particular requirement 
>> (in our proxy) at least for CONNECT.
>
we don't have this problem in normal circumstances, since we don't pass 
the CONNECT method through to the O-S, only an upstream proxy, and we 
insert a Host header if there is none.

I think probably the intention was that in RFC2616 S 5.2 where it reads

1. If Request-URI is an absoluteURI, the host is part of the 
Request-URI. Any Host header field value in the request MUST be ignored.

2. If the Request-URI is not an absoluteURI, and the request includes a 
Host header field, the host is determined by the Host header field value.

should read

1. If Request-URI is an absoluteURI or authority, the host is part of 
the Request-URI. Any Host header field value in the request MUST be 
ignored.

2. If the Request-URI is neither an absoluteURI nor an authority, and 
the request includes a Host header field, the host is determined by the 
Host header field value.

This leaves the question of the final of the 4 cases for Request-URI, 
being "*".  Is a host header required or not for this?  My assumption 
would be not.

although it looks like S5.2 is intended for Origin Servers, rather than 
proxies.

S 9 states clearly "The Host request-header field (section 14.23) MUST 
accompany all HTTP/1.1 requests" which conflicts with 5.2 (otherwise 
must be present but must be ignored in case of Request-URI = 
AbsoluteURI...?)  5.2. implies (by saying "Any host header... ") that 
the Host header is optional, otherwise it should read "The host header 
MUST be ignored".

S 14.2.3 States "A client MUST include a Host header field in all 
HTTP/1.1 request messages .If the requested URI does not include an 
Internet host name for the service being requested, then the Host header 
field MUST be given with an empty value. An HTTP/1.1 proxy MUST ensure 
that any request message it forwards does contain an appropriate Host 
header field that identifies the service being requested by the proxy. 
All Internet-based HTTP/1.1 servers MUST respond with a 400 (Bad 
Request) status code to any HTTP/1.1 request message which lacks a Host 
header field. "

I've never seen an empty Host header, so I presume the up-take of that 
requirement is rare if any.

Regards

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 17 Jun 2009 13:16:51 +1200

Actually, apologies, I should have revised the last email before sending.

Sections 9, 14.2.3, and 19.6.1.1 are all in agreement that ALL http 1.1 
request messages MUST contain a Host header.

Section 5.2 implies in some cases there may not be one but that this 
seems to be OK.

So the problem section appears to be 5.2.

However, the sentence in 14.2.3 last para "If the requested URI does not 
include an Internet host name for the service being requested, then the 
Host header field MUST be given with an empty value" is plain wrong.  It 
is implying if there's no host in the request line, the host tag must be 
empty.  this for obvious reasons cannot work, and should read something 
like "If the requested URI includes an Internet host name for the 
service being requested, then the Host header field MUST be given with 
an empty value"

I'd actually also be keen to know what a proxy or O-S should do if there 
is a Host header and a host in the request URI which don't match.

5.2 states the Host header must be ignored, however isn't it an 
indication of something nasty, which should be bounced?  If the Host 
header is passed through unchanged by a proxy after taking the host out 
of the Request-URI (converting an AbsoluteURI to a abs_path), then the 
O-S will get a different host tag to the host specified in the request 
URI.  This is surely highly problematic.  I see 3 options for how a 
proxy should deal with this case.

1. Do nothing
2. Bounce the request
3. re-write the host tag.  Set it to the host part that was parsed out 
of AbsoluteURI. 

I'd be leaning towards 2, since it's an indication of malicious intent 
surely?

Regards

Adrien

Adrien de Croy wrote:
> I only have a couple of reports since this wasn't in general release - 
> a mail.ru client and FTP commander using CONNECT.
>
> although I imagine this wasn't the intent, since authority contains 
> the required information.
>
>>
> we don't have this problem in normal circumstances, since we don't 
> pass the CONNECT method through to the O-S, only an upstream proxy, 
> and we insert a Host header if there is none.
>
> 2. If the Request-URI is not an absoluteURI, and the request includes 
> a Host header field, the host is determined by the Host header field 
> value.
>
> should read
>
> the request includes a Host header field, the host is determined by 
> the Host header field value.
>
> although it looks like S5.2 is intended for Origin Servers, rather 
> than proxies.
>
> Internet host name for the service being requested, then the Host 
> header field MUST be given with an empty value. An HTTP/1.1 proxy MUST 
> ensure that any request message it forwards does contain an 
> appropriate Host header field that identifies the service being 
> requested by the proxy. All Internet-based HTTP/1.1 servers MUST 
> respond with a 400 (Bad Request) status code to any HTTP/1.1 request 
> message which lacks a Host header field. "
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 16 Jun 2009 18:17:07 -0700

On Jun 16, 2009, at 5:28 PM, Adrien de Croy wrote:
> I only have a couple of reports since this wasn't in general  
> release - a mail.ru client and FTP commander using CONNECT.

Then break them.  They are sending invalid requests, most likely
because they are trying to trigger a security hole.

> Request-URI is not an AbsoluteURI, and MUST be ignored if it is.   
> This doesn't cover the case where Request-URI is authority (as per  
> CONNECT)... so a strict interpretation is that CONNECT requires  
> host, although I imagine this wasn't the intent, since authority  
> contains the required information.

No, the intention was that Host is required on all HTTP/1.1 requests
for policy reasons (ensuring deployment of Host was a high priority).
Its value might be empty because not all URIs have host portions.

Apache will always respond 400 if HTTP/1.1 and no Host header field.
It will also respond 400 if HTTP >= 1.1 and the hostname target cannot
be obtained via either the request-uri or the Host field.
A "CONNECT host:port HTTP/1.1" request without a Host field will fail
the first test even if it passes the second.

....Roy

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 17 Jun 2009 09:22:52 +0200 (CEST)

> This has been found to break a number of clients which use the CONNECT 

Let me also mention that I remember that back in the days when I wrote my 
first CONNECT requests, they were being ignored or not at least not treated 
well by many proxies when we did them without Host: header.

I haven't checked this situation in a long time, but I would assume that this 
is still at least partly the case. Thus, doing CONNECTs host-less is already 
non functional in the real world.

-- 

  / daniel.haxx.se

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 17 Jun 2009 22:29:10 +1200

the interesting thing is that for the CONNECT method, RFC2616 doesn't 
say explicitly what part of the request provides the host information 
for the connection.  It only does this for AbsoluteURI and abs_path, not 
authority or "*" ( since Request-URI := "*" | AbsoluteURI | abs_path | 
authority )

The RFC for tunnelling does however.  So basically we're saying the Host 
header must be there, but it must be ignored.

I guess there are still enough proxies out there willing to honour an 
HTTP/1.1 CONNECT request without one, else the vendors of this software 
would have fixed it by now.

Cheers

Adrien

Daniel Stenberg wrote:
> Let me also mention that I remember that back in the days when I wrote 
> my first CONNECT requests, they were being ignored or not at least not 
> treated well by many proxies when we did them without Host: header.
>
> I haven't checked this situation in a long time, but I would assume 
> that this is still at least partly the case. Thus, doing CONNECTs 
> host-less is already non functional in the real world.
>

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 18 Jun 2009 12:26:11 +0200

> receiving such a message MUST respond with 400.

Correct, but at the same time a very stupid requirement when it comes to
proxy request as proxies also SHOULD discard the Host header...
(request-URI is authoritative, and outgoing request MUST match
request-URI)

I have already voiced my opinion that this no longer needs to be a MUST
requirement, and should be downgraded to a SHOULD (or in case of
requests sent to proxies perhaps even a MAY as it's completely redundant
there).

The MUST requirements has played it's role in getting major acceptance
for the use of Host...

Regards
Henrik