Subject: Re: must a partial response range be exact?

From: Julian Reschke <[email-masked]>
Date: Wed, 01 Oct 2008 17:58:53 +0200

A. Rothman wrote:
> If a client requests partial range(s) using the Range header, is it 
> legal for the server to return a partial result which is larger than the 
> requested range(s) but fully contains it? This would allow a server to 
> satisfy a request for multiple ranges by returning a Content-Range 
> response with a single range that contains all of the requested data, 
> thus allowing a light-weight server to be able to support partial 
> content retrieval without having to implement the added complexity of 
> multipart/byteranges handling and range-skipping.

I would say it's always legal, as the response is self-descriptive.

> It is not apparent from the spec whether this is allowed or not - 
> perhaps it should be stated explicitly to avoid confusion (by both the 
> sender and recipient of the response).
> 
> 
> If it is not allowed, the alternative for such a server would be to 
> support single-range requests using Content-Range partial responses, and 
> ignore any multiple-range requests by always responding to them with 
> full content (200) - this is implied to be legal according to the spec 
> since range support is optional in the first place - but would be much 
> less efficient.
> 
> 
> So - must a partial response range be exact or not?

Do you have a specific use case in mind? The only real-world scenario I 
have seen where multiple parts were requested was with Adobe Acrobat 
(which, as far as I recall, asks for content from the beginning and from 
the end (the toc?))? That case doesn't seem to benefit a lot from the 
proposal you made...

Are there other clients you're thinking of, or is this a purely 
theoretical question?

BR, Julian

---

From: "A. Rothman" <[email-masked]>
Date: Thu, 02 Oct 2008 10:44:42 +0000

It's a theoretical question - I've never seen this feature in use 
personally (and wasn't aware of Acrobat's use either). I'm just working 
on a small very-lightweight embeddable HTTP server, and I want it to be 
RFC compliant - so I'm raising those points in the spec which are 
ambiguous/undefined/self-conflicting, in order both to clarify how my 
server implementation should behave, and hopefully have the RFC wording 
itself modified to be clearer, thus making sure everyone who follows it 
understands the same thing (that's what specs are for :-) ).

Specifically, I will currently only support the simpler Content-Range 
responses (no multiparts), for both single-range and multiple-range 
requests, and just wanted to make sure that doing so is legal - I agree 
that it makes sense that it should be legal, but making sense is often 
not good enough ;-)

Thanks,

Amichai

Julian Reschke wrote:

>> legal for the server to return a partial result which is larger than 
>> the requested range(s) but fully contains it? This would allow a 
>> server to satisfy a request for multiple ranges by returning a 
>> Content-Range response with a single range that contains all of the 
>> requested data, thus allowing a light-weight server to be able to 
>> support partial content retrieval without having to implement the 
>> added complexity of multipart/byteranges handling and range-skipping.
>
>> perhaps it should be stated explicitly to avoid confusion (by both 
>> the sender and recipient of the response).
>>
>>
>> support single-range requests using Content-Range partial responses, 
>> and ignore any multiple-range requests by always responding to them 
>> with full content (200) - this is implied to be legal according to 
>> the spec since range support is optional in the first place - but 
>> would be much less efficient.
>>
>>
> Do you have a specific use case in mind? The only real-world scenario 
> I have seen where multiple parts were requested was with Adobe Acrobat 
> (which, as far as I recall, asks for content from the beginning and 
> from the end (the toc?))? That case doesn't seem to benefit a lot from 
> the proposal you made...
>
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - http://www.avg.com 
> Version: 8.0.173 / Virus Database: 270.7.5/1702 - Release Date: 01/10/2008 09:05
>
>

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 08 Oct 2008 23:38:57 +0200

On tor, 2008-10-02 at 10:44 +0000, A. Rothman wrote:

> requests, and just wanted to make sure that doing so is legal - I
> agree that it makes sense that it should be legal, but making sense is
> often not good enough ;-)

There is at least one SHOULD that will bite you

   When a client requests multiple byte-ranges in one request, the
   server SHOULD return them in the order that they appeared in the
   request.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 09 Oct 2008 23:08:06 +0200

On tor, 2008-10-09 at 22:00 +0200, A. Rothman wrote:
> Thanks for pointing that out!
> 
> However, if that is so, then given that "A response to a request for
> multiple ranges, whose result is a single range, MAY be sent as a
> multipart/byteranges media type with one part", the spec MAY have this
> SHOULD clause bite itself as well (in exactly the same sense as
> returning a single Content-Range response)...

There is no contradiction. The MAY only applies if the SHOULD is
fulfilled. A MAY never has higher priority than SHOULD.

The specifications is a sum of MUST and SHOULD requirements, and MAY
hints on possible alternative ways of doing things as long as the MUST
and SHOULDs is also fulfilled.

But implementations MAY disregard SHOULD level requirements if they
insist, provided they understand the implications. Only MUST level
requirements is strictly required.

It's not possible to expand each clause to express the full conditions,
as this would result in the specifications being repeated all over, with
tons of opportunity for contradictions.

Regards
Henrik

---

From: "A. Rothman" <[email-masked]>
Date: Thu, 09 Oct 2008 19:59:19 +0000

Is this another minor self-contradiction?

Actually, I understood this SHOULD clause as referring to the 
possibility of returning several ranges out of order (which it 
discourages), rather than to combining ranges into a single range (but 
retaining the overall order).

What do u think?

Thanks,

Amichai

---

From: "A. Rothman" <[email-masked]>
Date: Thu, 09 Oct 2008 22:08:02 +0000

Henrik,

I'm confused about what you mean by 'The MAY only applies if the SHOULD 
is fulfilled'. In this case, if the SHOULD is fulfilled the way u 
intended in your original reply, i.e. a multiple-range request SHOULD be 
fulfilled with a multiple-range response, then the MAY clause can never 
be used, since replying to such a request with a single-part response 
violates this SHOULD requirement. In other words, to my understanding, 
if the SHOULD is fulfilled then the MAY can never apply (and may as well 
be dropped from the spec). Hence the confusion :-)

If I misunderstood your interpretation of this SHOULD clause and how 
exactly it might bite if a multiple-range request is fulfilled using a 
single-range response (returning a superset of all requested ranges), 
I'd appreciate a clarification...

Thanks,

Amichai

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 10 Oct 2008 02:13:39 +0200

> I'd appreciate a clarification...

It would bite you if responding with a single range returns the data in
a different order than requested.

Consider the following:

Range: bytes=100-200,0-99

If you merge the ranges you get 0-200, but that is not the order
requested by the client.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 10 Oct 2008 21:02:50 +0200

> returning a single Content-Range response)...

My reading:

A linear sequence of in-order ranges MAY be merged in the response. If
there is only a single range left after such merges then it SHOULD be
sent as a Content-Range response instead of multipart/byteranges, but
MAY be sent as multipart/byteranges is the sender prefer..The recipient
MUST accept both as valid responses, in addition to the 1-1 mapped
multipart/byteranges response or any other response that do cover the
requested ranges including (but not limited to) a full non-ranged
response.

Regards
Henrik

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 10 Oct 2008 15:26:41 -0500

Henrik Nordstrom wrote:
> A linear sequence of in-order ranges MAY be merged in the 
> response. If there is only a single range left after such 
> merges then it SHOULD be sent as a Content-Range response 
> instead of multipart/byteranges, but MAY be sent as 
> multipart/byteranges is the sender prefer..The recipient MUST 
> accept both as valid responses, in addition to the 1-1 mapped 
> multipart/byteranges response or any other response that do 
> cover the requested ranges including (but not limited to) a 
> full non-ranged response.

This is how I read it too, except for the "but MAY be sent as
multipart/byteranges." multipart/byteranges requires two parts.

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 11 Oct 2008 12:29:29 +0200

> multipart/byteranges." multipart/byteranges requires two parts.

Brian, do you disagree with the proposed resolution for issue 133, or 
did you miss it?

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 11 Oct 2008 15:01:21 +0200

On lör, 2008-10-11 at 12:08 +0200, A. Rothman wrote:

> btw do u think it is worth the added complexity to the spec and server
> implementation? is there some sort of usage/support statistics on
> clients that make use of out-of-order mutliple-ranges requests which
> show it to be worthwhile?

As already mentioned in this thread the Acrobar Reader browser plugin is
the most frequent user of out-of-order ranges. It first downloads the
ToC at the end of the document, then the rest of the document,

Regards
Henrik

---

From: David Morris <[email-masked]>
Date: Sat, 11 Oct 2008 19:16:10 -0700 (PDT)

On Sat, 11 Oct 2008, Henrik Nordstrom wrote:

> On lÃ¶r, 2008-10-11 at 12:08 +0200, A. Rothman wrote:
>
> As already mentioned in this thread the Acrobat Reader browser plugin is

Isn't that done with multiple requests? At one time their plan was to
actually request document segments. At least for large documents. That
doesn't happen either?

Dave Morris

---

From: "A. Rothman" <[email-masked]>
Date: Sat, 11 Oct 2008 10:10:35 +0000

This brings us back to 
http://trac.tools.ietf.org/wg/httpbis/trac/ticket/133 ...

Amichai

Brian Smith wrote:

> Version: 8.0.173 / Virus Database: 270.8.0/1718 - Release Date: 10/10/2008 07:07
>
>

---

From: "A. Rothman" <[email-masked]>
Date: Sat, 11 Oct 2008 10:07:45 +0000

ok, this is much clearer wording than the spec and the added 'recipient 
MUST' paragraph makes it explicit that clients MUST properly handle any 
form of (appropriate) response which includes the requested data, which 
from the existing spec is unclear - the reason I raised this question in 
the first place... any chance we'll see such improved wording in the new 
spec?

Also, where does the 'ordering' SHOULD clause at hand fit here? "When a 
client requests multiple out-of-order byte-ranges in one request , the 
server SHOULD return them as multipart/byteranges in the order that they 
appeared in the request" ?

Regards,

Amichai

---

From: Julian Reschke <[email-masked]>
Date: Sun, 12 Oct 2008 11:43:26 +0200

> actually request document segments. At least for large documents. That

It definitely requests ranges sometimes. As far as I recall, it varies 
with Acrobat Reader versions a lot.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 12 Oct 2008 11:04:05 -0500

> Brian, do you disagree with the proposed resolution for issue 133, or 

* Clients should accept single-part multipart/byterange responses, but
servers shouldn't generate them since RFC 2616 says that every
multipart/byterange request has at least two parts.

* Clients should not send range requests that can be trivially combined
(e.g. 0-2, 3-5), but they should accept any combination that the server
does. Server implementers shouldn't waste time trying to combine ranges; if
a client split contiguous ranges into separate ranges, it probably had a
reason for doing so.

* Clients should send ranges in the order they want the ranges to be
returned; servers shouldn't reorder out-of-order ranges (3-5, 0-2), even if
they are contiguous. Reordering and combining these ranges defeats the
purpose of splitting the request into ranges in the first place (e.g. Adobe
Reader's behavior).

* Clients shouldn't send obviously overlapping ranges (ranges that overlap
for any reason other than the combination of a regular range and a suffix
range).

* The server is free to reject any request (overlapping range requests or
otherwise) where the request would exhaust too many server resources.

Regards,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 12 Oct 2008 19:59:32 +0200

> otherwise) where the request would exhaust too many server resources.

I think I agree with all these statements.

What I still want to know is: do we need to make any additional changes 
to the spec? Is this all just "common sense", or should something of 
this become a normative requirement? If not, should it be added as guidance?

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 12 Oct 2008 20:32:57 -0500

Julian Reschke wrote:
> What I still want to know is: do we need to make any 
> additional changes to the spec? Is this all just
> "common sense", or should something of this become a
> normative requirement? If not, should it be 
> added as guidance?

Well, I derived all of that from RFC 2616 + common sense, so I don't think
any specification changes are necessary. 

I don't think the multipart/byteranges feature is valuable, because it
duplicates functionality that you can get with pipelined single-range
requests. So, the less effort spent on it, the better. Especially, I think
any clarifications here are much less important than other open issues. 

Regards,
Brian

---

From: "A. Rothman" <[email-masked]>
Date: Mon, 13 Oct 2008 00:18:37 +0000

If this is agreed upon, the spec has to be changed at least to some 
degree, since its current wording conflicts with these guidelines both 
explicitly and implicitly in various places. Also note that these 
guidelines do break existing implementations, since the RFC wording all 
but encourages server-side range recombinations. Some possible fixes:

"When an HTTP message includes the content of a single range (for
  example, a response to a request for a single range, or to a request
  for a set of ranges that overlap without any holes)"

=> the sentence in parentheses should be only "(a response to a request 
for a single range)", or dropped altogether

"When an HTTP message includes the content of multiple ranges (for
   example, a response to a request for multiple non-overlapping
   ranges), these are transmitted as a multipart message."

=> 'for example' and 'non-overlapping', or the entire sentence in 
parentheses, should be dropped

"A response to a request for a single range MUST NOT be sent using the
   multipart/byteranges media type."

=> maybe add explicitly that "Clients should accept single-part 
multipart/byterange responses"

"A response to a request for
   multiple ranges, whose result is a single range, MAY be sent as a
   multipart/byteranges media type with one part."

=> entire sentence must be dropped (resolving issue 133 in accordance to 
these guidelines - I like this solution better than the other proposed 
one since it reduces the complixity and ambiguity of the spec)

"When a client requests multiple byte-ranges in one request, the
   server SHOULD return them in the order that they appeared in the
   request"

=> maybe add that the ranges should be returned exactly as requested 
(full correspondence of both number and order of ranges), and further 
state explicitly that clients should accept anything the server throws 
at them

Finally, possibly add a paragraph with what the client should/shouldn't 
send (trivially combinable ranges, meaningful order, obviously 
overlapping ranges), though with this new set of guidelines the rules 
are simpler and less ambiguous, so this paragraph might be redundant and 
common sense will suffice. Maybe still warn that if the client does 
something too wasteful, the server can reject it, and if so, with what 
error code (and/or full 200 response).

Regards,

Amichai

Julian Reschke wrote:
> Brian Smith wrote:
>>> Brian, do you disagree with the proposed resolution for issue 133, 
>>> or did you miss it?
>>
> What I still want to know is: do we need to make any additional 
> changes to the spec? Is this all just "common sense", or should 
> something of this become a normative requirement? If not, should it be 
> added as guidance?
>
> BR, Julian
> Version: 8.0.173 / Virus Database: 270.8.0/1720 - Release Date: 11/10/2008 15:59
>
>