Subject: #282: Recommend minimum sizes for protocol elements

From: Mark Nottingham <[email-masked]>
Date: Mon, 2 May 2011 11:46:42 +1000

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/282>

We talked about this briefly in Prague. Since then I've put together a straw-man proposal:

Insert at end of p1 3.2:

> HTTP does not place a pre-defined limit on the length of header fields, either in isolation or as a set. A server MUST be prepared to receive request headers of unbounded length and respond with the 413 (Request Entity Too Large) status code if the received header(s) would be longer than the server wishes to handle (see Section 8.4.14 of [Part2]).
> 
> A client that receives response headers that are longer than it wishes to handle can only treat it as a server error.
> 
> Various ad-hoc limitations on header length are found in practice. It is RECOMMENDED that all HTTP senders and recipients support messages whose combined headers have 20,000 or more octets.

Add section to p1 Security Considerations:

> 11.5 Protocol Element Size Overflows
> 
> Because HTTP uses mostly textual, character-delimited fields, attackers can overflow buffers in implementations, and/or perform a Denial of Service against implementations that accept fields with unlimited lengths.
> 
> To promote interoperability, this specification makes specific recommendations for size limits on request-targets [ref] and blocks of header fields [ref]. These are minimum recommendations, chosen to be supportable even by implementations with limited resources; it is expected that most implementations will choose substantially higher limits.
> 
> This specification also provides a way for servers to reject messages that have request-targets that are too long [ref] or request entities that are too large [ref].
> 
> Other fields (including but not limited to request methods, response status phrases, header field-names, and body chunks) SHOULD be limited by implementations carefully, so as to not impede interoperability.

Thoughts? We can go further than this, of course, but IME request-target and headers are the big ones.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 2 May 2011 08:13:10 +0200

> Thoughts? We can go further than this, of course, but IME request-target and headers are the big ones.

Some servers have a per-header size limit, and developers sometimes have to
allocate a string to store one header, or to limit their abilities to send
too large headers (eg: Location, Set-Cookie). Maybe we could emit suggestions
on this point too ? From what I recall, Apache has a 8kB per header limit,
which is plenty for all uses. Also we could remind that if some headers are
too large (eg: Set-Cookie), it's likely that the client will experience a
very poor performance when uploading this header to fetch objects.

Another point concerning the arbitrary 20kB size. Many hardware-based
equipments have much lower limits (1-3 MSS = 1.5-4.5 kB). I used to run
at 2kB in haproxy, which revealed too small after about 5 years of service.
Now it's configurable and defaults to 15kB from a 16kB buffer, and I never
received any complain with such a size. I was once notified by a user
running at 7kB who had a bug in his application (generating headers in
loops), and haproxy was the first one to trigger the limit. I always run
at 7kB everywhere without any issue.

>From my experience, it seems like the developers who care about performance
don't abuse headers nor cookies, and the dirty ones who don't care about
anything don't know how to manipulate headers.

It's important to give rough figures because allocating space requires
a lot of RAM. For instance, one million concurrent sessions at 20kB are
20GB of RAM. Some applications need to keep those headers for all the
duration of the request (logging, data manipulation, ...).

Thus I'd prefer that we suggest that we'd say something like "at least 4kB
in space-constrained system, and at least 20kB for safer interoperability".

Cheers,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 02 May 2011 06:29:42 +0000

In message <[email-masked]>, Willy Tarreau writes:

>From what I recall, Apache has a 8kB per header limit,
>which is plenty for all uses.

One would think, but I have seen 36kB headers in live traffic, but
that was a private cache-informing header between the backend and
a server-side cache, so it may not count in this context.

Poul-Henning

PS: Is it just me or are large pointless HTTP-headers an indicator of
mobile devices where users pay for bandwidth ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 2 May 2011 08:40:55 +0200

> a server-side cache, so it may not count in this context.

Similarly in private environments I have observed insane things which
required recompiling some products but that does not count.

> mobile devices where users pay for bandwidth ?

It may be just you :-)

The operator has little control over the terminal, and terminal makers
tend to emit limited headers, and from what I observed, they try to make
everything fit into one MSS. After that, the headers might be enriched
by proxies before going to the net.

BTW this makes me think that there are 3 request headers which are
responsible for something like 80% of the upstream bandwidth :
  - User-Agent
  - Cookie
  - Referer

And sometimes Accept adds a bit to the list. I think that a bit of civism
on the net would be nice and appreciated by the end user. Browsers are 100%
responsible for UA and Accept. Servers are 100% responsible for Cookie and
Referer, though we might suggest that browsers could trim the Referer when
too large.

When a mobile phone has to upload 2 MSS of data for every request and it
loads a page full of images, it can wait for a long time until its hundreds
of kB are uploaded.

Cheers,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 02 May 2011 07:12:43 +0000

>> mobile devices where users pay for bandwidth ?

Check the headers from a device rhyming with "Jack&Terry" :-)

It may be that the headers are added in a land-side proxy, but
they are somewhat obese still.

>responsible for something like 80% of the upstream bandwidth :

One of the more amusing ways to bust caches is to include "Vary:
User-agent" in a response.

One would think that there would be two identical browsers on the
planet but I guess we're all individualists in that respect.

I think that we should recommend that clients not send
a U-A header at all, to enourage servers to DTRT with respect
to content portablity.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 2 May 2011 17:14:59 +1000

> a U-A header at all, to enourage servers to DTRT with respect

Agreed, but from discussing this (at length) with browser vendors, they're unwilling to change UA and other headers much, because of potential interop problems with sites that depend upon them.

There has been some headway in recent browsers; IE9 and FF4 have somewhat shorter UA headers than before.

I'm currently cooking up a workaround for this; should have a draft soon.

Cheers,

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 2 May 2011 09:28:28 +0200 (CEST)

On Mon, 2 May 2011, Mark Nottingham wrote:

>> I think that we should recommend that clients not send a U-A header at all, 
>> to enourage servers to DTRT with respect to content portablity.
>
> Agreed, but from discussing this (at length) with browser vendors, they're 
> unwilling to change UA and other headers much, because of potential interop 
> problems with sites that depend upon them.

I can confirm that not using a user-agent header at all gives a certain amount 
of problems. I can't give any specific numbers, but libcurl does its default 
requests without such a header and that is known to cause grief every now and 
then.

I recall that even Apache has (had?) some set of UA-based browser checks that 
check for known faulty browsers to be able to do the necessary work-arounds.

-- 

  / daniel.haxx.se

---

From: Julian Reschke <[email-masked]>
Date: Mon, 02 May 2011 09:36:10 +0200

On 02.05.2011 09:28, Daniel Stenberg wrote:
> ...
> I can confirm that not using a user-agent header at all gives a certain
> amount of problems. I can't give any specific numbers, but libcurl does
> its default requests without such a header and that is known to cause
> grief every now and then.
>
> I recall that even Apache has (had?) some set of UA-based browser checks
> that check for known faulty browsers to be able to do the necessary
> work-arounds.
> ...

mod_dav has a "redirect carefully" option that suppresses 302 redirects 
on WebDAV collections, because a number of client doesn't properly deal 
with them.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 2 May 2011 10:34:02 +0200

> > User-agent" in a response.

Amusing indeed.

> > planet but I guess we're all individualists in that respect.

I've designed a very simple UA logging algorithm for a customer. The
principle is simply to log an ID when the UA has already been seen,
and otherwise to log both the UA and its new assigned ID. And doing
so revealed that we got something like 6000 IDs in a week among about
10M different visitors. For sure servers can't rely on the full UA.
 
> > a U-A header at all, to enourage servers to DTRT with respect

I know some customers who make the UA mandatory to serve requests, not for
stats, but because they know that (right now) 100% of requests without a
valid UA come from stupid bots.

> problems with sites that depend upon them.

Indeed, some applications need to identify small sets of known devices,
especially the ones with limited capabilities (IE6 announcing support for
compression which is not completely true, or other devices with small
screens). But I think the main reason is that browser vendors like to
provide a way to see their browsers reported in stats. BTW, one of the
reasons that Opera is not much represented is that it lets you pretend
to be another browser, and I've seen many people set it to MSIE :-)

> There has been some headway in recent browsers; IE9 and FF4 have somewhat shorter UA headers than before.

Indeed, just the browser name, its version and optionally the device type
should be plenty, as right now developers don't mention other criteria
when targetting a specific combination.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 02 May 2011 08:37:53 +0000

In message <[email-masked]>, Willy Tarreau writes:

>especially the ones with limited capabilities [...]

Yes, and this is exactly why I think why U-A should be banned: It makes
it possible for browser-writers to shift the trouble to the web-servers.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 2 May 2011 10:57:52 +0200

> it possible for browser-writers to shift the trouble to the web-servers.

Well, I get your point and agree in general. Still I think there are valid
uses ; it's not necessarily stupid to present a different virtual keyboard
to a 320x240 device than to a 1600x1200 one. Just as you'd probably not ask
for the same inputs on an iphone and FF4 running on a desktop.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Mon, 02 May 2011 11:28:24 +0200

On 02.05.2011 10:57, Willy Tarreau wrote:
>> In message<[email-masked]>, Willy Tarreau writes:
>>

But at least for HTML content, CSS offers much better support for this 
through media queries.

Best regards, Julian