Subject: Use of 1*DIGIT

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Thu, 21 Dec 2006 12:22:00 -0800

There are many instances in the spec that have 1*DIGIT (and one instance of 1*HEX), where the number needs to be parsed for, e.g., a length. This practically forces an implementation to choose between using bignums everywhere (in case, e.g., they start talking to an implementation with a bigger native word size), or to assume that everybody can and will be able to use a word size of some fixed amount.

If I were the betting type, I'd say that implementations probably operate on the latter more than they do the former. In either case, how an implementation is supposed to deal with too-large numbers is not specified. Content-Length, Content-Range, Byte-Ranges, Age, Cache-Control, Retry-After, and the chunked transfer encoding are probably all susceptible to mischief if you place values past the 4Gi mark into them (assuming a 32-bit world and a general lack of oversize checks, of course).

Anyone know how Apache handles this, off the top of their heads? Squid? Other servers? Clients?

Thanks,

-- Travis

---

From: Paul Leach <[email-masked]>
Date: Thu, 21 Dec 2006 15:21:34 -0800

There will be lots of cases where the syntax says something is legal,
but the semantics or an implementation say it isn't. This seems like one
of those, and unless there isn't an appropriate 5xx series error code
for any particular case, I don't think said case would be a problem.

In this case, a 64 bit implementation could handle lengths that a 32
versions couldn't. 

I don't see that we need to note every place in the syntax where this
problem could arise, just like we don't need to be explicit that
implementers shouldn't code buffer overflows.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Travis Snoozy (Volt)
Sent: Thursday, December 21, 2006 12:22 PM
To: [email-masked]
Subject: Use of 1*DIGIT

There are many instances in the spec that have 1*DIGIT (and one instance
of 1*HEX), where the number needs to be parsed for, e.g., a length. This
practically forces an implementation to choose between using bignums
everywhere (in case, e.g., they start talking to an implementation with
a bigger native word size), or to assume that everybody can and will be
able to use a word size of some fixed amount.

If I were the betting type, I'd say that implementations probably
operate on the latter more than they do the former. In either case, how
an implementation is supposed to deal with too-large numbers is not
specified. Content-Length, Content-Range, Byte-Ranges, Age,
Cache-Control, Retry-After, and the chunked transfer encoding are
probably all susceptible to mischief if you place values past the 4Gi
mark into them (assuming a 32-bit world and a general lack of oversize
checks, of course).

Anyone know how Apache handles this, off the top of their heads? Squid?
Other servers? Clients?

Thanks,

-- Travis

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 21 Dec 2006 16:10:01 -0800

> implementers shouldn't code buffer overflows.

Right.  It is actually more dangerous for implementers to have
required size limitations in the protocol, since then they often
assume the value is going to remain conformant to the standard
(and we all know that isn't a constraint on attackers).

Implementions need to handle large numeric strings no matter how
large they might be, regardless of what the protocol says they
should be, and generally do so by returning an error if the
number is larger than the maximum for the internal representation
used for the value.  This will change over time (as data get bigger)
and may be much larger for specialized implementations than it
would be for general-purpose implementations.  10 years ago almost
everyone thought that 4GB would be a reasonable limit for an
implementation of Content-Length -- now that is clearly not the
case for the video-on-demand folks.

....Roy

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Thu, 21 Dec 2006 16:18:42 -0800

Paul Leach said:
> There will be lots of cases where the syntax says something is legal, but
> the semantics or an implementation say it isn't. This seems like one of
> those, and unless there isn't an appropriate 5xx series error code for any
> particular case, I don't think said case would be a problem.

A 5xx would work fine for dealing with a Request that the server can't handle (given that the implementation even realizes the error is there), but it doesn't help in the case when a client gets a Response it can't handle.

> In this case, a 64 bit implementation could handle lengths that a 32

Yep. And a 64 bit server could give a 32 bit client a headache if that client happens to request a file over 4GiB (like a DVD image) and the server tries to cram it down the pipe with a single Content-Length. Same deal down the line for 16 or 8 bit processors. Again, this means that everyone either has to use bignums (which is unlikely), or attempt interop in some unspecified manner (also unlikely), or abort gracefully (reasonable), or be broken but work in most cases (likely).

Ideally, the transfer problem would be solved by allowing the client and server to negotiate word sizes. This would allow any oversized to be sent with a chunked transfer encoding (with each chunk at or below the word size for the receiver) instead of with Content-Length. It would also be useful in ensuring that chunked entities have chunk sizes that are easily digestible by the recipient. It's also forwards compatible, so if we ever decide we need 100EiB files on 128-bit computers, everything will still play nice with an old, clunky 32-bit machine. (Note that this does _not_ address the security issue with maliciously constructed messages, it merely facilitates interop in the non-malicious cases.)

Clearly that's not going to happen in HTTP/1.1, But I _do_ think that the spec should offer some guidance on the best way to fail or recover, instead of just being mum on the matter.

> implementers shouldn't code buffer overflows.

Well, given the number of buffer overflows and integer overflows in the world, I'd say that it sure might be helpful.

Tossing a note in section 15 would be good enough for me. There's a risk in parsing numbers that are bigger than the application wants to deal with, clients SHOULD do this (probably abort, and user-agents give a message), and servers SHOULD do that (probably return a 5xx code).

-- Travis

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Thu, 21 Dec 2006 16:56:30 -0800

Roy T. Fielding said:
<snip>
> used for the value.

That's fine and dandy at the API level. I'm asking how that error should be
represented at the protocol level. Should the client close the stream?
Should the client try again? Should the server send a 5xx? Or a 4xx, since
the request needs to be modified? Should the server close the stream?

> This will change over time (as data get bigger)
> would be for general-purpose implementations.

Regardless, the two should be able to interoperate, since they're
purportedly both speaking HTTP/1.1 -- but that won't happen unless the spec
is implemented to the T, using a representation that can hold an arbitrarily
large value (which I'd argue that most implementations won't do, because in
general it is an overly-burdensome requirement).

> 10 years ago almost

Yeah. I remember wondering how I was _ever_ going to fill that shiny new
100MB hard drive.

Thanks,

-- Travis

---

From: "Justin Erenkrantz" <[email-masked]>
Date: Fri, 29 Dec 2006 11:58:20 -0800

> Anyone know how Apache handles this, off the top of their heads? Squid? Other servers? Clients?

httpd 2.2 and newer parses incoming Content-Length as an off_t which
if LFS is enabled is 64-bit, otherwise, it'd be 32-bit.  (Almost all
modern platforms are LFS-enabled these days.)  1.3 and 2.0 use 'long'.

http://svn.apache.org/repos/asf/httpd/httpd/trunk/modules/http/http_filters.c

See the call to apr_strtoff() there.

HTH.  -- justin

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 29 Dec 2006 13:04:16 -0800

> See the call to apr_strtoff() there.

Ah; thank you! So, half of my guess was right, and the other half was wrong: the most widely-used server on the 'net does not use bignums here, but the code DOES consider the overflow condition (in the case of content-length, at least). The underlying functions (strtol/strtoi) are also specified to saturate, as opposed to wrap, so that's a plus, too.

Can anyone clue me in on Firefox?

Thanks again,

-- Travis

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 29 Dec 2006 13:41:36 -0800

Travis Snoozy said:
> Ah; thank you! So, half of my guess was right, and the other half was
> wrong: the most widely-used server on the 'net does not use bignums here,
> but the code DOES consider the overflow condition (in the case of content-
> length, at least). The underlying functions (strtol/strtoi) are also
> specified to saturate, as opposed to wrap, so that's a plus, too.
>

I've answered my own question through a little empirical testing. Here's a response to "HEAD http://cdimage.debian.org/debian-cd/3.1_r4/i386/iso-dvd/debian-31r4-i386-binary-1.iso HTTP/1.1" (no headers):

HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 4683952128
Age: 4683
Expires: Sat, 30 Dec 2006 19:49:30 GMT
Date: Fri, 29 Dec 2006 21:07:33 GMT
Content-Type: application/octet-stream
ETag: "19987f7-1172f7000-4225dbbfc0000"
Server: Apache/2.2.3 (Unix)
Last-Modified: Thu, 16 Nov 2006 14:04:48 GMT
Accept-Ranges: bytes
Keep-Alive: timeout=5, max=1000

(I stripped some proxy info. I promise it was valid :)

IE7 (!) on 32-bit Windows XP chokes (I don't have the equipment to try the 64-bit version). It will not download this file, very probably because of the Content-Length. The error message is:

Internet Explorer cannot download [...the file...] from cdimage.debian.org

The supplied HTTP header is invalid

Note: invalid. There is nothing actually invalid about the above headers. It's just that content-length is bigger than 32 bits can handle. Firefox (2.0.0.1), however, gets it right and downloads the file.

So, this is a problem. It's very, very, very unfortunate that the IE7 folks let this one slip through. However, this is _exactly_ what I expected to happen, because the behavior is unspecified, and clients tend to not want to work on things bigger than a native word.

I'd be interested to hear if this problem exists on earlier versions of IE, because that has ramifications about how servers should go about serving huge files (mostly because IE has such a large installed base). If this happens with other clients, that would be good to know, too.

-- Travis

---

From: David Morris <[email-masked]>
Date: Fri, 29 Dec 2006 14:30:42 -0800 (PST)

Look at the bright side ... you got an error message from IE7 ....
in the long established tradition of clear messages, the message
even related to to problem.

IE6 is totally confused and ends up downloaning 451,140 bytes and
declares success ...

Dave Morris

On Fri, 29 Dec 2006, Travis Snoozy (Volt) wrote:

> I've answered my own question through a little empirical testing.
> Here's a response to
> "HEAD http://cdimage.debian.org/debian-cd/3.1_r4/i386/iso-dvd/debian-31r4-i386-binary-1.iso

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 29 Dec 2006 15:58:31 -0800

> IE6 is totally confused and ends up downloaning 451,140 bytes and

Well, there's a good 60+% of users on the WWW that won't interoperate with this particular server using this particular file, then. IIS appears to have an integer overflow (!) when replying to a HEAD method on a file greater than 4GiB:

HEAD http://localhost/testdir/test.gbg HTTP/1.1
Content-Length: 0
Host: localhost

HTTP/1.1 200 OK
Server: Microsoft-IIS/5.1
X-Powered-By: ASP.NET
Date: Fri, 29 Dec 2006 23:30:51 GMT
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 29 Dec 2006 23:21:54 GMT
ETag: "c9226120a02bc71:909"
Content-Length: 73314544

For reference, the size of the file is 4,368,281,840 bytes, 4GiB is 4,294,967,295 bytes, and the difference is 73,314,545 bytes (the value of Content-Length + 1). The actual GET returns a 501/Not Supported, but the erroneous HEAD reply is still Bad and Wrong.

Not that it's a surprise; these are the _exact_ problems that I predicted would show up, based solely on what the spec said. Go figure. More digging in more products will very likely uncover similar issues (and not just in Content-Length, but anywhere where 1*DIGIT is present).

-- Travis

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 30 Dec 2006 00:40:55 +0000

Travis Snoozy (Volt) wrote:
> For reference, the size of the file is 4,368,281,840 bytes, 4GiB is
> 4,294,967,295 bytes, and the difference is 73,314,545 bytes (the value
> of Content-Length + 1). The actual GET returns a 501/Not Supported,
> but the erroneous HEAD reply is still Bad and Wrong.

Just a small note: 4GiB is 4,294,967,29_6_, and the difference is
equal to the reported Content-Length.  Overflow is as expected for a
32-bit integer.

I wonder if it ever reports negative values like Content-Length: -1?

-- Jamie

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 29 Dec 2006 17:13:33 -0800

> equal to the reported Content-Length.  Overflow is as expected for a

That's what I get for using "calculator" without thinking. Glad to see
you're paying attention, though ;).

> I wonder if it ever reports negative values like Content-Length: -1?

Nope; it appears to be an honest-to-goodness unsigned value (as it darn well
should be).

HEAD http://localhost/testdir/test2.gbg HTTP/1.1
Content-Length: 0
Host: localhost
Connection: close

HTTP/1.1 200 OK
Server: Microsoft-IIS/5.1
X-Powered-By: ASP.NET
Connection: close
Date: Sat, 30 Dec 2006 00:52:28 GMT
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Sat, 30 Dec 2006 00:50:54 GMT
ETag: "9183988fac2bc71:909"
Content-Length: 2346074112

-- Travis

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 30 Dec 2006 20:52:34 -0800

On Dec 29, 2006, at 3:58 PM, Travis Snoozy (Volt) wrote:
> Well, there's a good 60+% of users on the WWW that won't  
> interoperate with this particular server using this particular  
> file, then. IIS appears to have an integer overflow (!) when  
> replying to a HEAD method on a file greater than 4GiB:

Implementations can have bugs.  That doesn't change the standard.

> 4,294,967,295 bytes, and the difference is 73,314,545 bytes (the  
> value of Content-Length + 1). The actual GET returns a 501/Not  
> Supported, but the erroneous HEAD reply is still Bad and Wrong.

You should file it with the other protocol errors in IIS.

> Not that it's a surprise; these are the _exact_ problems that I  
> predicted would show up, based solely on what the spec said. Go  
> figure. More digging in more products will very likely uncover  
> similar issues (and not just in Content-Length, but anywhere where  
> 1*DIGIT is present).

That is complete nonsense.  The spec does not say "Fail to use any  
common
sense or valid software engineering techniques while reading untrusted
network input." Nor does it say "Failure to recognize and handle integer
field values larger than the expected integer size is okay."

Professional software developers are expected to know better and be
able to use their own judgement. They don't need a standard to tell
them it is a bug.

....Roy