Subject: Coalescing ranges

From: "A. Rothman" <[email-masked]>
Date: Thu, 23 Jul 2015 17:56:50 +0000

Hi,

I'd like to raise an issue with the HTTP 1.1 RFCs regarding when the server can coalesce byte ranges requested by the client.

In RFC 2616, there's section that says:

"When an HTTP message includes the content of a single range (for
example, a response to a request for a single range, or to a request
for a set of ranges that overlap without any holes), ..."

which implies that there may be other legitimate cases in which the server may choose to coalesce the ranges, and it's all good.

However, RFC 7233 says:

"When multiple ranges are requested, a server MAY coalesce any of the
ranges that overlap, or that are separated by a gap that is smaller
than the overhead of sending multiple parts, regardless of the order
in which the corresponding byte-range-spec appeared in the received
Range header field.  Since the typical overhead between parts of a
multipart/byteranges payload is around 80 bytes, depending on the
selected representation's media type and the chosen boundary
parameter length, it can be less efficient to transfer many small
disjoint parts than it is to transfer the entire selected
representation."

Notably, it changed from a "for example" to a MAY with very detailed, very specific cases. Now it sounds to me like coalescing for other reasons is off the table.

I'd like to suggest that there may be other legitimate reasons why a server would want to coalesce the ranges, even if the exact efficiency calculations described in detail above do not hold. Regardless of the reasons for doing so, I think this should be more explicitly allowed by the RFC, or at least implicitly allowed with a 'for example' disclaimer like it used to be, for the simple reason that it would always still be more efficient to return a range (even if it's a single large range) than it would be to ignore the range header and return the whole content with a 200 status, which is explicitly allowed by the spec as an alternative. Disallowing it (even if implicitly, as the current wording might suggest) would prevent such simple but efficient optimizations from being valid, thus making the detailed byte-counting optimization detailed above somewhat... silly (for lack of a better word at the moment :-) ).

This of course would not affect existing servers, or new servers that want to implement any sort of complex calculations or heuristics on byte range responses (e.g. what is mentioned in the security considerations section), nor would it affect clients in any way since they must check the returned data ranges and handle any of these types of responses anyway.

If I've misinterpreted the RFC or missed some other section relevant to this issue, I'd be happy if someone can point it out :-)

Thanks,

Amichai

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 25 Jul 2015 12:29:58 +0200

Hi,

The thing to keep in mind here is that just because the spec says you MAY do something, it doesn't imply that you MUST NOT do other things. I.e., we do not operate under "anything which is not expressly allowed is forbidden" rules. 

I agree that this could be written a bit more clearly; the normative MAY is somewhat spurious here.

Cheers,

> On 23 Jul 2015, at 7:56 pm, A. Rothman <[email-masked]> wrote:
> 
> Hi,
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 24 Jul 2015 12:09:20 -0700

> On Jul 23, 2015, at 10:56 AM, A. Rothman <[email-masked]> wrote:
> 
> Hi,
> 

It doesn't imply that there, but it doesn't matter.

> Notably, it changed from a "for example" to a MAY with very detailed, very specific cases. Now it sounds to me like coalescing for other reasons is off the table.

No.  That interpretation would require a MUST NOT (or at least a SHOULD NOT).

> I'd like to suggest that there may be other legitimate reasons why a server would want to coalesce the ranges, even if the exact efficiency calculations described in detail above do not hold. Regar…

It isn't a byte counting optimization -- it is suggesting when coalescing might be a good idea.

> If I've misinterpreted the RFC or missed some other section relevant to this issue, I'd be happy if someone can point it out :-)

Simply that MAY never implies a requirement. Such a requirement would have to be
stated as a "MUST NOT coalesce unless ...".

....Roy

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 25 Jul 2015 23:48:24 +1200

> I agree that this could be written a bit more clearly; the normative MAY is somewhat spurious here.

That was to avoid problems like the interpretation we came up with for
Squid years back. That the coalescing was not expressly permitted, so we
could reject it. That led to problems with some software using Squid and
oter clients following the same line of reasoning.

With MAY, we can clearly see that clients need to be able to handle
combined ranges back from the server. Clients that don't can have the
RFC thrown at their devs.

I was and still am in preference for inverting the requirement so it
becomes a "MUST be able to handle" type requirement on client instead of
a MAY on servers. But what we have is what was compromised.

Amos

---

From: Mike Bishop <[email-masked]>
Date: Sat, 25 Jul 2015 17:22:11 +0000

I can add that we implemented range coalescing in http.sys several releases back.  We found that a major application could no longer open files hosted on IIS servers, and had to back it out.  Don't know if current versions of that application are as demanding about getting back exactly the ranges it asked for....  We do a little bit of coalescing still, but it's much less aggressive than I'd like it to be for compat reasons.

I do like that we added something normative here; I think a server-MAY implies client-MUST-handle, so I don't see the need to have a normative statement on both sides, though I also agree with the original note that it would be nice if that had been more general.  (Servers MAY return the entire file or any set of ranges that contains the bytes the client requested.)  We actually do have a code path, not hit in the common case, where we'll return a larger range than the client asked for.

-----Original Message-----
From: Amos Jeffries [mailto:[email-masked]] 
Sent: Saturday, July 25, 2015 1:48 PM
To: [email-masked]
Subject: Re: Coalescing ranges

> I agree that this could be written a bit more clearly; the normative MAY is somewhat spurious here.

That was to avoid problems like the interpretation we came up with for Squid years back. That the coalescing was not expressly permitted, so we could reject it. That led to problems with some software using Squid and oter clients following the same line of reasoning.

With MAY, we can clearly see that clients need to be able to handle combined ranges back from the server. Clients that don't can have the RFC thrown at their devs.

I was and still am in preference for inverting the requirement so it becomes a "MUST be able to handle" type requirement on client instead of a MAY on servers. But what we have is what was compromised.

Amos