Subject: Handling multiple headers when only one is allowed

From: Bil Corry <[email-masked]>
Date: Tue, 08 Jun 2010 17:18:00 -0700

Michal Zalewski's excellent "Browser Security Handbook" points out that different browsers handle multiple headers differently when only one header is suppose to be present (scroll down to "First HTTP header of the same name takes precedence?"):

	http://code.google.com/p/browsersec/wiki/Part1#Hypertext_Transfer_Protocol

Essentially, the first header takes precedence for Internet Explorer and Safari while Firefox, Opera and Chrome use the last header.

In a similar thread I brought up on another list[1], Michal Zalewski explained the security implications of this issue[2], Julian Reschke pointed out that there's already a similar issue open[3] and Mark Nottingham suggested I bring it up here[4].

To summarize the issue, when a user-agent encounters multiple headers of the same name when only one is allowed, it must decide which header, if any, will be used.  The argument for using the first header centers on the premise that an attacker most likely will be injecting headers below the real header.  The argument for using the last header centers on the premise that sometimes web developers do not control the entire server, and thus can not control headers added by the server, but are able to add additional headers.

Given the mixed implementations among user-agents and the security implications therein, is it possible for this to be defined?

- Bil

[1] http://lists.w3.org/Archives/Public/public-web-security/2009Dec/0221.html
[2] http://lists.w3.org/Archives/Public/public-web-security/2009Dec/0223.html
[3] http://trac.tools.ietf.org/wg/httpbis/trac/ticket/95
[4] http://lists.w3.org/Archives/Public/public-web-security/2009Dec/0226.html

---

From: Nathan <[email-masked]>
Date: Wed, 09 Jun 2010 01:32:23 +0100

Does this (or should this) take in to account multiple values in header 
when only one is allowed? for instance if the following header is set:

   Content-Location: foo.html, bar.html

Best,

Nathan

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 09 Jun 2010 12:36:47 +1200

I don't think you can set a blanket rule for this, since it will depend 
on the header what the appropriate action is.

For instance if you receive 2 content-length headers with different 
lengths, it's probably unsafe to do anything other than reject the message.

Or if you get a Transfer-Encoding: chunked and a Transfer-Encoding: 
identity.

Or 2 or more different Content-Type headers, or even conflicting 
charsets.  Or multiple Content-Encoding headers.

Personally I don't think it's wise to "rely" on any particular 
hypothetical benefit (from a security standpoint) of order.  Any 
malicious inserter of headers should be considered to be able to 
potentially add headers anywhere in the header block.  So intrinsically 
I don't believe it's any safer to choose any particular one (first or 
last), and in fact it could be argued that picking a random one (if any) 
is more secure since it is harder for a malicious inserter of headers to 
predict where it must insert.

Are there any particular cases you have in mind of headers?  It might be 
safest to have a rule more like:

a) if the headers is optional, but can only exist once, then presence of 
more than one instance of the header should be treated as if the header 
were not present at all
b) if the header is required, reject the message.

In fact there are many ways to have conflicting headers, not just 
multiple instances of an unrepeatable header.

The spec for obvious reasons doesn't cover many of these (the number of 
combinations are very large).  It covers things like a message 
containing Content-Length and Transfer-encoding.

Cases which affect message delimiting are particularly problematic.  E.g 
a response which contains Connection: Keep-alive without a 
Content-Length nor Transfer-Encoding header, it's hard to tell what to 
do when you receive this, since the only way you can delimit the message 
boundary is if the server closes, which it indicated it wouldn't do.

Cheers

Adrien

---

From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Date: Wed, 09 Jun 2010 02:48:20 +0200

On Wed, 09 Jun 2010 02:18:00 +0200, Bil Corry <[email-masked]> wrote:

> To summarize the issue, when a user-agent encounters multiple headers of  
> the same name when only one is allowed, it must decide which header, if  
> any, will be used.  The argument for using the first header centers on  
> the premise that an attacker most likely will be injecting headers below  
> the real header.  The argument for using the last header centers on the  
> premise that sometimes web developers do not control the entire server,  
> and thus can not control headers added by the server, but are able to  
> add additional headers.
>
> Given the mixed implementations among user-agents and the security  
> implications therein, is it possible for this to be defined?

I'd say it would be difficult, because whatever is decided it will be the  
wrong choice in many cases (when the server sends bad headers). I suspect  
that most clients picked their current method because some important site  
(e.g. a bank) broke because of the old way of doing things, which was  
opposite of what the (at the time) major competing client did at the time.

As far as I can see there are three choices for changing the current  
status:

   - Always pick the first
   - Always pick the last
   - Ignore all such double headers, and in the case of Content-Type,  
handle the document as an application/octet-stream and offer to save it to  
disk (no guessing allowed), or just discard the body (which might be the  
safest option) and tell the user "Could not display content due to  
conflicting instructions from the site".

In all three cases a site will break if it sends duplicate headers and  
rely on some specific manner (opposite of what is selected) for the  
content to be handled, but in the last case the result will be predictable.

Well, there is actually a fourth choice: Ask the user (Yes, I know, the  
user will most likely know just as little as the client about what those  
header were intended to mean, and the opportunities for social engineering  
attacks will be legion).

-- 
Sincerely,
Yngve N. Pettersen

********************************************************************
Senior Developer                     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
********************************************************************

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 09 Jun 2010 08:31:09 +0200

On Wed, 09 Jun 2010 02:48:20 +0200, Yngve N. Pettersen (Developer Opera  
Software ASA) <[email-masked]> wrote:
> header were intended to mean, and the opportunities for social  
> engineering attacks will be legion).

There's also a fifth. Based on implementation experience we can probably  
figure out what the scenario for headers should be. You might end up with  
special cases, but at least you know it can be implemented and you can  
give advice for future clients so they will no longer have to reverse  
engineer the market leader.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jun 2010 14:24:31 +0200

On 09.06.2010 02:18, Bil Corry wrote:
> ...

Interesting.

That text mentions the test

   "Content-Length header value overrides actual content length?"

I have trouble understanding what this means... Unless the connection is 
closed, or chunked encoding is in place, or the message is by definition 
not having a body (HEAD response), there *is* no other signal than 
Content-Length to find out the actual content length.

Michal, could you clarify what this test is about?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jun 2010 14:34:10 +0200

On 09.06.2010 08:31, Anne van Kesteren wrote:
>> Well, there is actually a fourth choice: Ask the user (Yes, I know,
>> the user will most likely know just as little as the client about what
>> those header were intended to mean, and the opportunities for social
> figure out what the scenario for headers should be. You might end up
> with special cases, but at least you know it can be implemented and you
> can give advice for future clients so they will no longer have to
> reverse engineer the market leader.

As far as I can tell from the data Bill provided, there is no general 
agreement on how this needs to be done.

That being said; a document explaining that widely-used UAs do, and also 
pointing out potential security problems would be a good thing to have. 
I just do not see why this would need to be part of the protocol definition.

Best regards, Julian

---

From: Dan Winship <[email-masked]>
Date: Fri, 11 Jun 2010 08:51:31 -0400

> Content-Length to find out the actual content length.

It's about the case when you know the connection will be closed at the
end of the response, but the actual number of bytes sent by the server
before closing the connection doesn't match the Content-Length header.

Eg, if you GET http://sourceforge.net/apps/wordpress/sourceforge, you'll
get back a 302 response with "Content-Length: 338" and "Connection:
close", followed by 8 bytes of body data before the server closes the
connection. A client should consider that to be a truncated response,
but as noted by the page above, most web browsers instead just assume
that the Content-Length header was wrong. (I assume this was originally
another case of bug-compatibility with IE to support broken web sites.)

-- Dan

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 12 Jun 2010 01:24:23 +1200

I raised this problem a while back.

All the browsers except Opera (limited case) make no complaint when a 
download is truncated.  This is whether it's chunked and doesn't receive 
a final 0 chunk, or whether there's a content length and the connection 
is closed (whether or not the server indicated it would close) prior to 
that many bytes being transferred.

I personally view this as highly problematic, and it's tied in with the 
work I've been doing recently with scanning at a proxy.

The reason it's problematic, is because every single proxy I've tested 
(TMG/ISA, WinRoute, WinGate, Webmarshall - admittedly there are many 
more) does something called either "drip-feeding" or "trickling".  If 
you're downloading a file through one of these proxies, they will send 
you a portion of the resource as it's coming down to the proxy.  When 
the proxy has received the whole file, it scans it and sends the rest if 
it's ok, but if it's not ok, it has 1 option only - abort the connection.

Since the browsers ignore the connection having been aborted, and 
present the downloaded file as if nothing was wrong, then any malware 
purveyor need only pad their malware out, so that the executable part 
will fall within the drip-feeding window.  It basically renders AV at 
gateway potentially useless.

If OTOH the browsers were to act on the fact that the download was 
aborted, this wouldn't be nearly as big a security risk.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Julian Reschke <[email-masked]>
Date: Fri, 11 Jun 2010 15:32:53 +0200

On 11.06.2010 15:24, Adrien de Croy wrote:
> you a portion of the resource as it's coming down to the proxy. When the
> proxy has received the whole file, it scans it and sends the rest if

+1 to all of this (the problem also applies to cases where the server 
breaks while sending the content).

Do we have a test case for this? For the browsers that get this wrong, 
are there bug reports?

Best regards, Julian

---

From: "Yngve Nysaeter Pettersen" <[email-masked]>
Date: Fri, 11 Jun 2010 15:38:59 +0200

> download is truncated.  This is whether it's chunked and doesn't receive

When accessing the server directly, that is correct.

When accessing through a proxy Opera will assume a broken connection means  
all was loaded anyway. The proxy special case is due to how a number of  
adbuster proxies edited content without removing the content-length  
header, causing the resend request heuristic to spin out of control.

> gateway potentially useless.

There is only one realistic way to avoid that: Not forward a byte of  
content until it has been checked. The recent discussion of keep alive 1xx  
responses might be useful here, but OTOH Opera (at least) will not shut  
down a normal connection while waiting for content; the only case we do  
that is for special requests with a timeout, like requests for CRLs  
(considering that some requests can take minutes or  more we can't tell  
what is the expected time to complete the operation).

-- 
Sincerely,
Yngve N. Pettersen
********************************************************************
Senior Developer		     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 23 69 32 60              Fax:    +47 23 69 24 01
********************************************************************

---

From: Adam Barth <[email-masked]>
Date: Thu, 17 Jun 2010 22:29:27 -0700

On Tue, Jun 8, 2010 at 5:36 PM, Adrien de Croy <[email-masked]> wrote:
> Or 2 or more different Content-Type headers, or even conflicting charsets.
>  Or multiple Content-Encoding headers.

For multiple Content-Type headers, the most compatible behavior is to
use the last header.  I've seen several examples where using the last
header is required to process the response as the sender intended.
This information is document in draft-abarth-mime-sniff.

Adam