Subject: OT: Effective Spec Writing [was RE: Large content size value]

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Tue, 2 Jan 2007 10:44:36 -0800

Roy T. Fielding said:

<snip>

> > For reference, the size of the file is 4,368,281,840 bytes, 4GiB is
> > 4,294,967,295 bytes, and the difference is 73,314,545 bytes (the
> > value of Content-Length + 1). The actual GET returns a 501/Not
> > Supported, but the erroneous HEAD reply is still Bad and Wrong.
>
> You should file it with the other protocol errors in IIS.
>

Trust me, I'm trying to figure out how to do that :).

> > Not that it's a surprise; these are the _exact_ problems that I
> > predicted would show up, based solely on what the spec said. Go
> > figure. More digging in more products will very likely uncover
> > similar issues (and not just in Content-Length, but anywhere where
> > 1*DIGIT is present).
>
> That is complete nonsense.  The spec does not say "Fail to use any
> common sense or valid software engineering techniques while reading
> untrusted network input." Nor does it say "Failure to recognize and handle
> integer field values larger than the expected integer size is okay."
>
> Professional software developers are expected to know better and be
> able to use their own judgement. They don't need a standard to tell
> them it is a bug.

This isn't actually my original problem -- I want the spec to tell me how
the error case should be handled, esp. on the client, and esp. in regard to
connection handling. However, I've got more than $0.02 on the topic of
quality spec writing, and I've already hovered off-topic onto it this far,
so forgive me if I indulge a bit in my response :).

The bar for "professional" software developers is not exactly high (which
leads to a completely different and much lengthier rant). Expecting that
*every person who reads the spec* can make competent interpretations and
judgments at every opportunity the spec gives them (and there are a *lot* of
opportunities) is ignoring the reality of the end-users.

Let me ask, because I honestly don't know: how many HTTP/1.1 features can we
not use because implementations are just too broken? Are -all- the
implementers in these cases (if they exist) inept or incompetent? Or is the
spec not being tight enough? Are these not the same types of issues that
required the HTTP/1.1 respin after HTTP/1.0?

It's not just in this spec, either: examining *any* spec is an excellent way
to determine what implementers are likely to get wrong, and in turn to
figure out how to break them. This is as great a technique for the black
hats as it is for the white: if we can read and re-read the spec until we
can't find anything that could easily be mis-implemented, we've improved the
life not only for the implementers (the end-users of the spec), but also for
the ultimate end-users of the protocol.

Finally, there are the "political" and "business" reasons why it has to be
the spec that is overtly uptight. Businesses are interested in getting stuff
out the door fast -- quality too often winds up as an afterthought. In order
to get good, solid implementations out of this situation, a specification
needs to do many things:

1. Be correct. Implementers are blind, and will implement bugs as spec'd.

2. Be consistent. Implementers have not gotten any more sighted since (1).

3. Be understandable. Implementers that don't understand will guess, and
   they tend to guess wrong.

4. Be simple (KISS). Implementers have more places to screw up if either the
   document itself is difficult to read (too many cross-references,
   unlabeled cross-references, baroque verbiage, etc.), or the algorithms
   the document is specifying are "tricky" in some way.

5. Be implementation-oriented. The spec should make it possible for one
   person to read-and-implement the spec in sequence, and for many people to
   be assigned discreet sections to implement in parallel. Requiring that
   everyone read the entire spec (possibly multiple times) to get enough of
   a grasp to start implementing is unrealistic.

If our goal as spec writers is to promote interoperability, we need to take
these constraints into consideration when we're writing the spec, otherwise
we're just ensuring interop on paper at best. IMHO, HTTP/1.1 fails on points
3, 4, and 5 (with 1 and 2 type problems being about average for a spec this
size), and I think that unduly limits the effectiveness of the spec in
practice.

To conclude, my point is that we shouldn't be writing this spec for
ourselves, for academics, for people who enjoy reading specs, for competent
and well-reasoned engineers, or for people who have infinite time and
patience. It should be written for the people who really have to implement
this -- people who have never seen the spec before, who may be mediocre at
engineering, and who just want to get the implementation over with as soon
as humanly possible (if not sooner).

Thanks,

-- Travis

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Thu, 4 Jan 2007 12:59:45 -0800

All right; to rebut all of the below arguments (sorry Roy, I'm going to pick on you a bit ;):

Roy T. Fielding said:
> On Dec 21, 2006, at 3:21 PM, Paul Leach wrote:
> > In this case, a 64 bit implementation could handle lengths that a 32
> > versions couldn't.
> >
> > I don't see that we need to note every place in the syntax where this
> > problem could arise, just like we don't need to be explicit that
> > implementers shouldn't code buffer overflows.
>
> Right.  It is actually more dangerous for implementers to have
> required size limitations in the protocol, since then they often
> assume the value is going to remain conformant to the standard
> (and we all know that isn't a constraint on attackers).
>
> Implementions need to handle large numeric strings no matter how
> large they might be, regardless of what the protocol says they
> should be, and generally do so by returning an error if the
> number is larger than the maximum for the internal representation
> used for the value.  This will change over time (as data get bigger)
> and may be much larger for specialized implementations than it
> would be for general-purpose implementations.  10 years ago almost
> everyone thought that 4GB would be a reasonable limit for an
> implementation of Content-Length -- now that is clearly not the
> case for the video-on-demand folks.

Roy T. Fielding said:
<snip>
> Implementations can have bugs.  That doesn't change the standard.
<snip>

> able to use their own judgement. They don't need a standard to tell

Then why is following clause in Section 14.6 (Age), page 106?

   If a cache receives a value larger than the largest positive integer it
   can represent, or if any of its age calculations overflows, it MUST
   transmit an Age header with a value of 2147483648 (2^31). [...] Caches
   SHOULD use an arithmetic type of at least 31 bits of range.

This deals explicitly with the overflow condition, and tells implementers exactly how to handle it. Why are "professional software developers" expected to "know better" in regards to Content-Length & co., but need explicit guidance in regards to Age? And if the answer is "because caches have the potential to not interoperate otherwise," what about the issue Content-Length has been shown have? Is that not considered an "interoperation" problem?

Or is the clause redundant/unnecessary, and in need of removal? ;)

Thanks,

-- Travis

---

From: Paul Leach <[email-masked]>
Date: Thu, 4 Jan 2007 13:14:08 -0800

The following already existing errors would be appropriate when a server
gets an integer larger than it can handle:

413 Request Entity Too Large
501 Not Implemented

I think these can handle any interop problems with same.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Travis Snoozy (Volt)
Sent: Thursday, January 04, 2007 1:00 PM
To: Roy T. Fielding
Cc: [email-masked] Group
Subject: RE: Large content size value

All right; to rebut all of the below arguments (sorry Roy, I'm going to
pick on you a bit ;):

Roy T. Fielding said:
<snip>
<snip>

   If a cache receives a value larger than the largest positive integer
it
   can represent, or if any of its age calculations overflows, it MUST
   transmit an Age header with a value of 2147483648 (2^31). [...]
Caches
   SHOULD use an arithmetic type of at least 31 bits of range.

This deals explicitly with the overflow condition, and tells
implementers exactly how to handle it. Why are "professional software
developers" expected to "know better" in regards to Content-Length &
co., but need explicit guidance in regards to Age? And if the answer is
"because caches have the potential to not interoperate otherwise," what
about the issue Content-Length has been shown have? Is that not
considered an "interoperation" problem?

Thanks,

-- Travis

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Thu, 4 Jan 2007 14:01:29 -0800

> I think these can handle any interop problems with same.

Deja vous; you forgot the client again (behavior is unspec'd).

Also, IHMO, failure isn't interop; it's the last resort when you *can't*
interop. A Windows+NTFS system can totally handle a 4+GiB file; the fact
that a client that (should) be HTTP/1.1 compatible can't handle a file of
that size says "interoperability problem" to me. Granted, it's not
implementing the spec, but the spec says you need to have infinitely long
numbers or fail (which is flat-out unreasonable).

A better spec would enforce what Apache does: the server/client needs to
support transferring files as large as the OS's filesystem will support
("infinitely long with a reasonable cap," if you will). This is sane,
because if a client can't download (or a server can't serve) a file bigger
than the FS can handle, it doesn't really matter -- the file wouldn't be
able to exist on the filesystem anyway. That's an OS-level interop problem,
then, as opposed to a protocol-level one. Making a friendly warning a SHOULD
level requirement would be kind of nice, too, so that the end-users
understand *why* they can't download certain files.

A cool spec (but potentially one that's too fancy) might allow for word-size
negotiation, so an 8-bit computer on a 32-bit filesystem could be more
efficient with its buffer management code (and have to do 32-bit ops only
when it came time to flush the buffers). This would allow for a whole other
host of problems (e.g., chunked transfers that are larger than the
filesystem can handle, with the error showing up -very- late in the game),
but would solve this particular issue.

<snip>

I admit and concur that the best we can probably do right now is fail;
however, the spec needs to give specifics on *how* to fail (especially for
clients, and especially regarding connection handling). Guidance (not
necessarily requirements) regarding sane max lengths (e.g., using an integer
size that can represent the maximum size supported by the underlying OS/FS)
would be nice, too.

Thanks,

-- Travis

---

From: "Larry Masinter" <[email-masked]>
Date: Thu, 4 Jan 2007 14:14:42 -0800

> if a client can't download (or a server can't serve) a
> file bigger than the FS can handle

It is possible for a client to use range retrieval
to get parts of a large file, even if the client couldn't
store the whole thing because of file size limitations.
(This can happen with JPEG2000 image files, for example.)

It's quite possible for a server to serve a dynamically
generated resource that is bigger than can fit into a
single file on the file system.

So I don't think the protocol limits and the underlying
operating system file size limits should be linked
in any way.

Larry

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 4 Jan 2007 15:28:27 -0800

> going to pick on you a bit ;):

Are you going to fix the bugs in those implementations?  If so, feel
free to try to "pick on me".  If not, then go to

   http://diveintomark.org/archives/2004/08/16/specs

>    SHOULD use an arithmetic type of at least 31 bits of range.

Because: 1) Jeff wanted it in there, not me; 2) Age does have a  
reasonable
maximum value that is way below that number (somewhere around one day
between refreshes is high); and 3) the entire section on Caching is  
written
as a tutorial, not a specification.

> implementers exactly how to handle it. Why are "professional  
> software developers" expected to "know better" in regards to  
> Content-Length & co., but need explicit guidance in regards to Age?  
> And if the answer is "because caches have the potential to not  
> interoperate otherwise," what about the issue Content-Length has  
> been shown have? Is that not considered an "interoperation" problem?

It is unnecessary garbage that makes real specification harder to find.
When an Age calculation comes out anywhere near that high, the value
is wrong.  The requirement ensures that it is considered "really old",
but it is still a wrong value.  It would be better for real-world
interoperability if the party simply ignores Age at that point and
makes the error visible, but doing so would not be as "transparent"
as marking it as stale.

What do you expect the specification to say regarding content-length?
Something like "If the server's storage interface contains numeric
offsets that are really big numbers, maybe using a really big integer
to store that offset would be a good idea."?  "If the indicated
Content-Length is larger than the user agent's capacity to read and
store, then choose any one of a hundred different alternative solutions
to viewing partial representations."  Or maybe just
"If the content-length is wrong, the recipient may not know it?"
Or maybe you think I should have specified a 4GB limit back in 1994?

Just fix the friggin bugs.  They aren't going to be any more or less
of a bug regardless of what is placed in the specification, and the
majority of implementations are obviously capable of implementing
that feature without being buggy.  The spec does not need any more.

IETF specifications are supposed to specify the interface.  That is
only part of the total system implementation problem -- namely, the
part that we can all agree is a standard.  If you fill the standard
with a bunch of useless cruft, nobody will agree to implementing it
at all.  Which is exactly where we are today with HTTP/1.1 caching.

What you want is a book like APUE by Stevens.  Sure, it's valuable,
but that isn't a standard.  We don't have to get agreement from
hundreds of vendors on an implementation guide.

> Or is the clause redundant/unnecessary, and in need of removal? ;)

Read the archives.

....Roy

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Thu, 4 Jan 2007 15:50:22 -0800

> operating system file size limits should be linked

Excellent point! I completely ignored dynamic content (though I would think that would almost always be served with a chunked encoding in practice). You've also brought up a great solution.

Since it's possible for the client to detect when a Content-Length or a chunk-length is too long, SHOULD the client then attempt a series of byte-range requests instead? This would solve all the prior problems I've mentioned, assuming the server implements that part of the protocol (anyone know which servers do, in practice?).

Also, in regard to connection handling: as far as I can tell, the client is going to have to close the connection if an oversized Content-Length shows up, since the client won't be able to read through to the next request reliably. If this is the case, is it specified? It might make for a nice suggestion (not a requirement).

Thanks,

-- Travis

---

From: David Morris <[email-masked]>
Date: Thu, 4 Jan 2007 17:26:19 -0800 (PST)

The bottom line is that if the content length is too long for the client
to represent, it surely isn't going to do byte-ranges or anything else to
retrieve the pieces which it then isn't going to have anyway to deliver to
the end user anyway. A total waste of time to send a 4G file w/o
content-length to a client which can't use it.

Either the server can correctly represent the content length or it can't
and whether or not it can correctly serve the response.
I think it reasonable that any server be expected to detect when it can't
and report an error. I think it reasonable to expect a client to know when
it can't convert the content length ascii string to a value it understands
and reject the response as being in error. Furthermore, it is reasonable
for a client to reject a response it knows is too large, as soon as it
knows that a size constraint has been exceeded.

I don't see how these expectations can be added to the protocol in a
meaningful way w/o bumping the version.

It would have been useful 10 years ago if we'd had the foresight to add
resource negotiation to the protocol in a similar fashion to language,
etc.

Working from memory, seems like we had some text regarding precision in
what is send and tolerance in what is accepted. Perhaps adding text there
to highlight the additional burden of modern systems and native object
sizes in terms of fields like content-length AND chunked encoding length
segments.

We can happily talk about good programming standards, but unfortunately
the C library includes a number of commonly used conversion functions
which don't behave well in the presense of invalid input. They set a bad
example for the average software engineer and probably generate a
requirement for cautionary language where it shouldn't be required.

Dave Morris

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 05 Jan 2007 06:56:37 +0100

tor 2007-01-04 klockan 15:50 -0800 skrev Travis Snoozy (Volt):

> Since it's possible for the client to detect when a Content-Length or
> a chunk-length is too long,

To be precise, what most programming langauges tells you (when used
correctly) is that Content-Length could not be converted into the
word-sise the application developer had selected for storing the said
value internally in the application, not necessarily why it could not be
converted.

It is not realistic for the HTTP specification to expect that all
implementations uses bignum for every integer which may be transmitted
in the protocol. All that can be expected is that application developers
recognize that failure to handle >2GB files is a bug if their users
expects it to work, and that all parties who agree on handling files
>2GB do it in the same manner at the protocol level and this is
fulfilled fine by the specs as it is.

The range-retreival question is purely hypothetical. A client which can
not handle large integer values for content length won't be able to
split it up in ranges either as the range specifications need numbers
larger than the client can represent. And how clients store downloaded
content is completely outside the concerns of the specification. A
client is free to split downloaded files in many OS:level files if
required, protocol specs do not care and must not care.

>  SHOULD the client then attempt a series of byte-range requests
> instead?

Why on earth should a client do that under these conditions? It most
likely won't be able to reassemble the result, or even compose the range
requests..

> Also, in regard to connection handling: as far as I can tell, the
> client is going to have to close the connection if an oversized
> Content-Length shows up, since the client won't be able to read
> through to the next request reliably.

Yes, unless it's seen acceptable to waste the network bandwidth sending
the data to the bitbucket..

>  If this is the case, is it specified?

Does it need to? I think not.

Regards
Henrik

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 5 Jan 2007 11:09:29 -0800

Short version:
        S4.4  : No mandate for clients to actually use determined length.
              Implies with a MUST that user agents can use non-length ways
              to find the end of a message.
        S14.13: No mandate for clients to use length specified in Content-
              Length.
        S8.2.2: Very specific about clients closing the connection in
              exceptional situations when sending entities to the server.
              Nothing similar for clients receiving entities is spec'd.

That's enough rope to hang oneself with.

Long version:

> value internally in the application, not necessarily why it could not be

Programmers always have the option of writing this routine themselves, or
(at the very least) writing a verification routine that will give an error
as to why a given value can't be parsed. In any case, it is *possible* (and,
IMHO, not too hard) for the client to detect when Content-Length et. al. are
too long. Whether or not the implementer actually does their own checking is
another question altogether (but that's the practical question, and to that
end, you're probably right).

> in the protocol.

No argument here.

> All that can be expected is that application developers recognize that
> failure to handle >2GB files

(4GiB, but whatever; each end has its own cap that can be arbitrary.)

> is a bug if their users expects it to work, and that all parties who agree
> on handling files >2GB do it in the same manner at the protocol level and
> this is fulfilled fine by the specs as it is.

Ideally. I'm still working on poking holes in that one.

> larger than the client can represent.

[later]

> And how clients store downloaded content is completely outside the
> concerns of the specification.

I agree, though our definitions of "clients storing downloaded content"
might be slightly different. Storage proper is a secondary consideration,
and I agree that it's not something the spec has business with (aside from
no-store).

> A client is free to split downloaded files in many OS:level files if

No argument here.

> likely won't be able to reassemble the result, or even compose the range

Yep; that was half-baked of me, and I should've re-read that section before
asking. But to play devil's advocate for a moment, dealing with bignums in
one small, contained spot (composing the next range request, which would
presumably be <= the maximum int size the system wants to handle) is better
than having to deal with bignums at every step of the way.

> the data to the bitbucket..

An implementer that thinks he's particularly clever might decide to try and
read as much as the message as he can, then spin through the rest of the
input until he hits a new response line (or, if no pipelining took place,
until the socket appears to be empty). Now, I'd consider this broken, a bad
idea, and otherwise unwise -- however, it might work "good enough," and it
isn't against the spec.

Section 4.4 does not mandate how clients receive a message with a Content-
Length, nor does section 14.13; nowhere does it say that the client _can't_
butcher the message it's receiving in an attempt to get to the next message.
One would hope that implementers don't do stupid things (like treat the
lengths determined from section 4.4 strategies as "advisory"), but Murphy's
Law dictates that _someone_ will. If anyone can find something that would
prevent me from using *ahem* "alternate methods" for finding the start of
the next message, please point it out.

Now, with that said, section 4.4 *does* say something about lengths that I
missed earlier:

   HTTP/1.1 user agents MUST notify the user when an invalid length is
   received and detected.

Now, first off, this is just user agents. Proxies are out in the cold
(they'd probably just send back some 5xx status saying that the upstream
server is being a pill). But this _does_ seem to imply that user agents (at
least; clients perhaps in general) are allowed to use whatever means they
deem necessary to find the end of a message *so long as they notify the
user*.

> Does it need to? I think not.

It's a matter of consistency, and specified versus unspecified behavior.
Other parts of the spec are very explicit about when the client/server
should close the connection. Nowhere does it say that closing the connection
is the general way in which the client should deal with failure, just that
anyone MAY close their connections at any time. Case in point: section 8.2.2
does a fine job of indicating how the client should behave when _sending_ an
entity to the server, but there is no analogue for how the client should
behave when _receiving_ an entity.

Since the behavior is (strictly speaking) unspecified, I can assume that
closing the connection is the right course of action (like the spec probably
wants, but doesn't say), or I can assume that it's OK to grope out the end
of the message (and have the potential to make everyone downstream break).
I'm sure there are other things I could assume, too; that's why life tends
to be a whole lot easier when things are specified :).

Thanks,

-- Travis

---

From: "Larry Masinter" <[email-masked]>
Date: Fri, 5 Jan 2007 11:22:44 -0800

One use case for large content size files that aren't downloaded
in entirety are JPEG2000 image files using only range retrieval,
e.g.:

http://www.jpeg.org/jpeg2000/j2kpart9.html

 "They are also intended, however, to enable random access to JPEG 2000
 files in the absence of JPIP. For example, the byte-range requests
 built into an unmodified HTTP (version 1.1) server could be used
 for this purpose."

Some JPEG2000 files are very large, but you never retrieve
the whole thing; for example, a satellite image of California.

"It is not realistic for the HTTP specification to expect that all
implementations uses bignum for every integer which may be transmitted
in the protocol."

is hyperbole. It's realistic to expect implementations to
use 64-bit integers for quantities that reasonably exceed
32-bit representations, and it's realistic to expect
implementations to check (and fail gracefully) when
any received protocol value exceeds its representation
capacity.

Larry

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 05 Jan 2007 20:34:05 +0100

fre 2007-01-05 klockan 11:22 -0800 skrev Larry Masinter:

> 32-bit representations

If they want to be able to deal with such object yes, but only required
then. Having this as a general requirement in the specs is total
overkill.

> and it's realistic to expect

Yes, absolutely.

Regards
Henrik

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 5 Jan 2007 11:34:53 -0800

> content-length to a client which can't use it.

Well, from that perspective, it's an even bigger waste for a server that
generates a 4+GiB file (say, dynamically) and transmits it with the chunked
encoding (with chunks being <4GiB each). That's another case I'd like to
test, but it would require a slightly-more-than-trivial amount of setup.

> and reject the response as being in error.

It may be splitting hairs, but the response isn't in error; the client's
implementer made a design decision (conscious or not). There is no error,
only a refusal to accept on the client's part.

> Furthermore, it is reasonable for a client to reject a response it knows
> is too large, as soon as it knows that a size constraint has been
> exceeded.

> meaningful way w/o bumping the version.

Well, aside from providing strong recommendations that aren't real
requirements, perhaps, and making explicit things that might have been
heavily implied or intended. It's certainly tricky. :\

> resource negotiation to the protocol in a similar fashion to language,

Hindsight is 20/20, and it's not like the spec doesn't consider a slew of
obscure-but-important contingencies already. The big questions are, how do
we fix it, and how do we avoid making the same mistake twice.

> sizes in terms of fields like content-length AND chunked encoding length

I'm all for a mention in section 15 and/or 19 -- something is better than
nothing.

<snip>

Well, IIRC (I don't have my man pages with me), strtol et. al. are spec'd to
do the Right Thing: saturate, and set an error code. Now, whether or not C
library implementations adhere to the spec, or programmers check for the
error, I can't say. But ANSI C shouldn't steer folks wrong here. Unless you had other function(s) in mind...?

Thanks,

-- Travis

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 5 Jan 2007 12:31:26 -0800

Larry Masinter said:
<snip>
(Quoting Henrik Nordstrom)

> use 64-bit integers for quantities that reasonably exceed

1. I agree that "every integer" is an overstatement. I restrict my argument
   to only those fields containing 1*DIGIT (or 1*HEX).

2. The spec says 1*DIGIT. That means "any number, 0-positive infinity" not
   "the biggest number you think is reasonable". Other parts of the spec
   imply that (servers, at least) can impose an arbitrary limit, and reject
   messages with (certain) field-values outside those limits.

3. Given that implementers DO choose the biggest number they think is
   reasonable (and they do), it can still cause preventable interop
   problems.

4. Embedded computers and old software are <del>people</del> clients too.
   It's realistic to expect *modern* implementations on *medium-to-high-end
   hardware* to use 64- and 32-bit integers. Yesterday and tomorrow are a
   problem, though.

5. (4) is an interesting problem, but we can't really fix it in HTTP/1.1.

6. (5) means both servers AND clients should have part of the specification
   dedicated to how each should fail. Servers have this, clients don't.

More on 4 (in refutation to the assertion of reasonableness, despite the
fact we can't fix it):

On a sub-32-bit processor, I certainly wouldn't want to be stuck doing 32-
or 64-bit math unless -absolutely- necessary. I'll grant that it's much less
likely for a sub-32-bit processor to have to deal with a >4GiB file.
However, having to deal with all 32-bit numbers just because the average
user will hit some (substantial, but minority) number of files >64KiB (on a
16-bit proc) is just cruel. Workaround logic to do a 16/32 switch isn't
exactly a cakewalk, either (though if performance is a big concern, and 32-
bit numbers are slow enough, it might happen). Likewise, an 8-bit proc would
be stuck emulating 16 bits enough of the time already -- requiring 32-bits
everywhere (especially when it's not _really_ necessary) is wasteful.

I'd be interested to hear any comments from folks who run HTTP on (sub-32-
bit) embedded processors.

Regarding 5:

> and it's realistic to expect implementations to check (and fail
> gracefully) when any received protocol value exceeds its representation
> capacity.

Yes, but I think we have failed to define "graceful failure". I don't think
that we'll be able to solve the transfer problem in an ideal way, but we
also haven't defined any client behavior when the client gets a message it
doesn't want to deal with.

Thanks,

-- Travis

---

From: "Scott Lawrence" <[email-masked]>
Date: Fri, 5 Jan 2007 16:26:12 -0500 (EST)

> bit) embedded processors.

That would be me... (or used to be).  Such systems are very unlikely to be
able to (or interested in) handle such large body sizes, but they don't
need to do big-integer math to discover that fact.  The parser can (and
should) recognize that the ascii value is larger than it can cope with and
cause the request or response to fail (the sooner the better).  Whether or
not they use any given existing library routine to discover this is not
important and need not be addressed by the protocol spec.

> doesn't want to deal with.

Yes we have.   For a large body in a request, the server can return an
error; the client is required to abort sending the body if an error is
received.  For a large body in a response, it's not lovely, but it's well
understood - the client just closes the connection.  Since there is no
3-way handshake, that's the best that can be done.

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Fri, 05 Jan 2007 16:32:25 -0600

> 3-way handshake, that's the best that can be done.

+1, this is well understood and reasonably well documented.  There is no
need, and significant arguments against defining a range for DIGIT*1 as
anything less than infinity, and it is up to the implementor to decide
what it can parse, what it cannot parse yet still work around, or what it
simply cannot work around.  Disconnecting, in the third case, is the only
remaining alternative...

<snip>
8.1.4 Practical Considerations
[...]
   A client, server, or proxy MAY close the transport connection at any
   time. For example, a client might have started to send a new request
   at the same time that the server has decided to close the "idle"
   connection. From the server's point of view, the connection is being
   closed while it was idle, but from the client's point of view, a
   request is in progress.

   This means that clients, servers, and proxies MUST be able to recover
   from asynchronous close events. Client software SHOULD reopen the
   transport connection and retransmit the aborted sequence of requests
   without user interaction so long as the request sequence is
   idempotent (see section 9.1.2). Non-idempotent methods or sequences
   MUST NOT be automatically retried, although user agents MAY offer a
   human operator the choice of retrying the request(s). Confirmation by
   user-agent software with semantic understanding of the application
   MAY substitute for user confirmation. The automatic retry SHOULD NOT
   be repeated if the second sequence of requests fails.

   Servers SHOULD always respond to at least one request per connection,
   if at all possible. Servers SHOULD NOT close a connection in the
   middle of transmitting a response, unless a network or client failure
   is suspected.

   Clients that use persistent connections SHOULD limit the number of
   simultaneous connections that they maintain to a given server. A
   single-user client SHOULD NOT maintain more than 2 connections with
   any server or proxy. A proxy SHOULD use up to 2*N connections to
   another server or proxy, where N is the number of simultaneously
   active users. These guidelines are intended to improve HTTP response
   times and avoid congestion.

[...]

8.2.2 Monitoring Connections for Error Status Messages

   An HTTP/1.1 (or later) client sending a message-body SHOULD monitor
   the network connection for an error status while it is transmitting
   the request. If the client sees an error status, it SHOULD
   immediately cease transmitting the body. If the body is being sent
   using a "chunked" encoding (section 3.6), a zero length chunk and
   empty trailer MAY be used to prematurely mark the end of the message.
   If the body was preceded by a Content-Length header, the client MUST
   close the connection.

[...]

8.2.4 Client Behavior if Server Prematurely Closes Connection
</snip>

(Note: no corresponding description of "Server Behavior if Client Prematurely
Closes Connection").

I'm searching for any evidence of a behavioral requirement for Server Response
that the Client "does not understand" and I'm coming up empty.

So (s. 8.1.4 Practical Considerations) "A client, server, or proxy MAY close
the transport connection at any time" is the final word.

Unless I'm mistaken, if the client is unwilling to handle the server's response,
the client MUST (gracefully? abruptly?) close the transport connection.

And that's the beginning and end of the story for ALL RESPONSES AND RESPONSE
HEADERS THAT THE CLIENT IS UNWILLING/UNABLE TO HANDLE. (*)

(*) distinct from s7.1 "Unrecognized header fields SHOULD be ignored by the
recipient"

Of course, other means to avoid the condition in the first place; initially
issuing a HEAD request against the desired resource can avoid this quandary
but that's the choice of the user-agent's implementor, and doesn't avoid
the situation for C-E: chunked.

There is no need to proceed with this discussion.  There's an opportunity
to expand upon 8.1.4 offering that the client may disconnect with example
reasons; or an opportunity to expand upon 8.2 for the converse of 8.2.4.
The participants here, and the host of implementors to date, seem to have
no problem understanding this from the existing spec.

As 8.1.4 says, for whatever reason.  You don't need one.  If you aren't
going to use the response, disconnect already.  It's up to the implementor
not to ignore overflows of ANY response header, and that includes DIGIT*1.

Can we please move on?

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 5 Jan 2007 15:51:07 -0800

William A. Rowe, Jr. said:
> > received.  For a large body in a response, it's not lovely, but it's
> > well understood - the client just closes the connection.  Since there is
> > no 3-way handshake, that's the best that can be done.
>
> +1, this is well understood and reasonably well documented.

(Send me links?)

> There is no need, and significant arguments against defining a range for
> DIGIT*1 as anything less than infinity, and it is up to the implementor to
> decide what it can parse, what it cannot parse yet still work around, or
> what it simply cannot work around.  Disconnecting, in the third case, is
> the only remaining alternative...

Yeah, but the second case can be pretty ugly, given what the spec allows.

<snip>

> So (s. 8.1.4 Practical Considerations) "A client, server, or proxy MAY
> close the transport connection at any time" is the final word.

That's always an *option*, yes (but it's not the *only* option).

> Unless I'm mistaken, if the client is unwilling to handle the server's
> response, the client MUST (gracefully? abruptly?) close the transport
> connection.

I don't see any MUST. As you pointed out, there are three basic scenarios:
parse it right, parse it fuzzy, and fail. Parse it right is out, so that
leaves parse it fuzzy and fail. The spec shows no preference for one over
the other, BUT "parse it fuzzy" can cause some serious problems (see my
argument earlier in this thread). I totally agree that fail is the right
choice.

<snip>

> no problem understanding this from the existing spec.

I think the opportunity should be taken; "it's never been a problem before"
doesn't strike me as very compelling. Cost/benefit wise, I'd say it's a win,
but I'm not an editor.

<snip>

> Can we please move on?

Sure. I have my answer, there are no new points here, and there's enough
discussion for interested parties to draw an informed conclusion.

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Fri, 05 Jan 2007 18:24:22 -0600

> Yeah, but the second case can be pretty ugly, given what the spec allows.

The second case, misparsing DIGIT*1 in this case, is your fault as an
implementor, and not permitted by spec.  DIGIT*1 is well defined.

> That's always an *option*, yes (but it's not the *only* option).

The other option is to leave the connection open and swallow the results.
Irrespective of how you [mis]parsed the headers, is there any other option
besides disconnect now, disconnect when you surrender (exceed some arbitary
number of bytes or whatever) or finish reading the response from the wire?
All three options are allowed by spec - it's your choice as an implementor.
The spec has no need or reason to dictate to you which you choose.

> discussion for interested parties to draw an informed conclusion.

Excellent.

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 6 Jan 2007 17:40:17 +0000

> in entirety are JPEG2000 image files using only range retrieval,

I've also seen ranges used to stream video in a home video display
system.  It actually fetches about a frame's worth of data with each
request.  (Why they don't just stream it starting from a particular
position, I've no idea).

The streamed files are vastly larger than the client could possibly
store.

> It's realistic to expect implementations to use 64-bit integers for
> quantities that reasonably exceed 32-bit representations,

I don't agree; many implementations couldn't possibly store anything
that big (e.g. a mobile phone), and if they have no reason to do range
requests from a large resource, there's no point in them handling
64-bit size values.

But,

> it's realistic to expect implementations to check (and fail
> gracefully) when any received protocol value exceeds its
> representation capacity.

on that I wholeheartedly agree.  It's not hard; there's no good
excuse.  I'm astonished and disappointed that Microsoft screwed that up.

-- JAmie

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Mon, 8 Jan 2007 10:32:07 -0800

Jamie Lokier said:
<snip>
> excuse.  I'm astonished and disappointed that Microsoft screwed that up.

For the record, neither IE7/XP SP2 nor IIS 5.1 "screwed up" (except for the
HEAD request in the latter case). Both failed "gracefully," just like you
want them to.

(I can't vouch for versions <7 or OSs other than XP SP2, since I don't have
any other configurations available to test on. David Morris reports that IE6
[on an unspecified OS] gives an erroneous success message. Try it at home
and see!)

(Note that "standalone" versions of the IE browsers may not give accurate results; for me, standalone IE6 fails exactly the same way the installed IE7 fails.)

Thanks,

--
Travis

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 10 Jan 2007 23:52:34 +0000

> HEAD request in the latter case). Both failed "gracefully," just like you

Putting the wrong number in a HEAD response is still screwing up,

> [on an unspecified OS] gives an erroneous success message. Try it at home

In this thread David Morris reports that IE6 downloads the wrong
number of bytes in addition to declaring it a success.  I call that
screwing up majorly.  (Also it seems likely it would get the message
boundary of the next persistent response wrong, as a consequence):

David Morris:
> IE6 is totally confused and ends up downloaning 451,140 bytes and
> declares success ...

-- Jamie