Subject: Re: Range Responses of Indeterminate Length: Draft

From: Mark Nottingham <[email-masked]>
Date: Fri, 2 Oct 2015 13:15:45 +1000

Hi Rodger,

As discussed before, I think this is pretty much a non-starter, because an intermediary that doesn't understand this extension will receive Content-Range headers that it will (rightfully) considered malformed. We can't make breaking changes in the syntax or semantics of widely-deployed header fields.

Cheers,

> On 2 Oct 2015, at 10:27 am, Plex <[email-masked]> wrote:
> 
> I've updated https://datatracker.ietf.org/doc/draft-combs-http-indeterminate-range/ to address some minor formatting issues, and complete the IANA Considerations section. Does anyone have any input on this?
> 
>> On Jun 29, 2015, at 20:28, Plex <[email-masked]> wrote:
>> 
>> It's been a while since this thread has seen any movement; anyone have any suggestions on how to move forward on this?
>> 
>>> On Apr 8, 2015, at 04:43, Mark Nottingham <[email-masked]> wrote:
>>> 
>>> I did a fairly quick cut-n-paste; feel free to correct/elaborate in the bugs.
>>> 
>>> Cheers,
>>> 
>>> 
>>>> On 8 Apr 2015, at 5:25 pm, Rodger Combs <[email-masked]> wrote:
>>>> 
>>>> Some of the ticket descriptions there are incorrect. libavformat's strncmp() approach only matches if the _first_ 5 characters match "bytes", WebKit's g_ascii_strcasecmp only matches if the entire header is exactly "none", Gecko's nsAutoCString::EqualsLiteral only matches if the entire header is exactly "bytes", and Chrome's std::string::find approach works as you described. So they'd all be potentially buggy in different circumstances.
>>>> 
>>>>> On Apr 8, 2015, at 00:12, Mark Nottingham <[email-masked]> wrote:
>>>>> 
>>>>> Thanks! CC:ing a couple of relevant people re: the tickets below.
>>>>> 
>>>>>> On 7 Apr 2015, at 6:28 pm, Rodger Combs <[email-masked]> wrote:
>>>>>> 
>>>>>> All links from the mirror most convenient for me:
>>>>>> 
>>>>>> libavformat: https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/http.c#L583
>>>>> 
>>>>> https://trac.ffmpeg.org/ticket/4451
>>>>> 
>>>>>> Chromium: https://github.com/ChromiumWebApps/chromium/blob/c7361d39be8abd1574e6ce8957c8dbddd4c6ccf7/content/renderer/media/buffered_resource_loader.cc#L399
>>>>> 
>>>>> https://code.google.com/p/chromium/issues/detail?id=474864
>>>>> 
>>>>>> WebKit: https://github.com/WebKit/webkit/blob/4633eb6dd73f925b45244a7f716e571a94f65628/Source/WebCore/platform/graphics/gstreamer/WebKitWebSourceGStreamer.cpp#L860
>>>>> 
>>>>> https://bugs.webkit.org/show_bug.cgi?id=143512
>>>>> 
>>>>>> Firefox: https://github.com/mozilla/gecko-dev/blob/174ec4f7153479c41f6cae2cf58f0a8ac927db0e/dom/media/MediaResource.cpp#L234
>>>>> 
>>>>> https://bugzilla.mozilla.org/show_bug.cgi?id=1152162
>>>>> 
>>>>>> 
>>>>>>> On Apr 7, 2015, at 02:54, Mark Nottingham <[email-masked]> wrote:
>>>>>>> 
>>>>>>> Do you happen to have references to the specific code in each? Regardless of what we do here, it’d be good to file bugs.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> 
>>>>>>>> On 7 Apr 2015, at 5:45 pm, Rodger Combs <[email-masked]> wrote:
>>>>>>>> 
>>>>>>>> I strongly considered this (instead of adding a header), but upon looking into existing implementations, it's clear it wouldn't work with existing software. No implementation I checked actually parses `Accept-Ranges` as per the RFC:
>>>>>>>> 
>>>>>>>> 	• libavformat checks if the first 5 characters of the header are "bytes"
>>>>>>>> 	• Chromium checks if the string "bytes" appears at all
>>>>>>>> 	• WebKit just checks if it's not equal to "none"
>>>>>>>> 	• Firefox checks if it's exactly equal to "bytes"
>>>>>>>> 
>>>>>>>> That last one makes it impossible to change the header's value without breaking backwards-compatibility with older Firefox versions, and the others are differing degrees of problematic.
>>>>>>>> 
>>>>>>>>> On Apr 7, 2015, at 02:39, Mark Nottingham <[email-masked]> wrote:
>>>>>>>>> 
>>>>>>>>> That’s what I was thinking. Having said that, I’d want to test some deployed implementations before saying so for sure.
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On 7 Apr 2015, at 5:38 pm, Martin J. Dürst <[email-masked]> wrote:
>>>>>>>>>> 
>>>>>>>>>> I haven't studied all of the syntax details, but couldn't using a different unit name, such as "newbytes" (even if it's the same unit as bytes) be made to work? That would avoid the need to deploy a totally new mechanism.
>>>>>>>>>> 
>>>>>>>>>> Regards,   Martin.
>>>>>>>>>> 
>>>>>>>>>> On 2015/04/07 14:24, Mark Nottingham wrote:
>>>>>>>>>>> Hi Roger,
>>>>>>>>>>> 
>>>>>>>>>>> There are a bunch of folks talking about how to do streaming (especially video) over HTTP, and byte ranges often come up as a mechanism. This reminds me of a discussion we briefly had at the end of the Dallas meeting, where some folks wanted to use byte ranges for sparse content in MPEG DASH.
>>>>>>>>>>> 
>>>>>>>>>>> I think the concerns I had there apply here too; by overloading/changing the partial content mechanism, you’re risking interoperability problems with deployed infrastructure — especially proxy/caches, which can and do cache partial responses, generate partial requests, etc.
>>>>>>>>>>> 
>>>>>>>>>>> For example, if there’s a caching proxy in the middle that doesn’t understand Accept-Indefinite-Ranges, it’ll pass it through unmodified, and the server will then potentially generate 206 responses that are malformed.
>>>>>>>>>>> 
>>>>>>>>>>> This and many similar discussions I’ve had recently makes me wonder whether it’d be interesting to define a genuinely new partial content mechanism that’s tailored for media streaming; while it would depend on intermediaries rolling out support for the new mechanism to get any benefit from them, at least we could design it so that it doesn’t interact badly with deployed intermediaries.
>>>>>>>>>>> 
>>>>>>>>>>> What do people (especially intermediary implementers) think?
>>>>>>>>>>> 
>>>>>>>>>>> Cheers,
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On 6 Apr 2015, at 10:17 pm, Rodger Combs <[email-masked]> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hello!
>>>>>>>>>>>> 
>>>>>>>>>>>> I've written up a draft standard for an HTTP extension that allows range requests to work more effectively with resources of indeterminate length. I've submitted it as an Internet-Draft, and wondered if anyone had any suggestions as to how to improve it, or move it towards standardization?
>>>>>>>>>>>> https://tools.ietf.org/html/draft-combs-http-indeterminate-range-01
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> --Rodger
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Mark Nottingham   https://www.mnot.net/
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> .
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Mark Nottingham   https://www.mnot.net/
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Mark Nottingham   https://www.mnot.net/
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> --
>>>>> Mark Nottingham   https://www.mnot.net/
>>>> 
>>> 
>>> --
>>> Mark Nottingham   https://www.mnot.net/
>>> 
>> 
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Rodger Combs <[email-masked]>
Date: Fri, 02 Oct 2015 06:39:59 +0000

That's a perfectly valid concern about my current spec, but I'd hope that it would be possible to implement this feature in a way that works around that problem. I'm open to any and all suggestions on the topic, and am not at all attached to the current draft's methods. Could a different signaling mechanism help? How do intermediaries handle range requests today?

> On Oct 1, 2015, at 22:15, Mark Nottingham <[email-masked]> wrote:
> 
> Hi Rodger,
>

---

From: Plex <[email-masked]>
Date: Fri, 02 Oct 2015 00:28:25 +0000

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 13 Oct 2015 15:39:46 +1100

Range was designed for very specific use cases -- mostly, AIUI, PDF files. Retrofitting it for streaming media is likely to cause problems. I'd recommend looking at other approaches, such as a manifest-driven one. 

Cheers,

> On 2 Oct 2015, at 4:39 pm, Rodger Combs <[email-masked]> wrote:
>

---

From: <[email-masked]>
Date: Tue, 13 Oct 2015 08:57:25 +0000

On Friday,02 October 2015 02:28, [email-masked]<mailto:[email-masked]> wrote:

> I've updated https://datatracker.ietf.org/doc/draft-combs-http-indeterminate-range/

> to address some minor formatting issues, and complete the IANA Considerations section.

> Does anyone have any input on this?

Sorry for the late and possibly ignorant response. Is what you’re trying to accomplish already possible with the use of multipart/byteranges?

e.g.

- Client requests stream starting from byte 0 of an unknown length.

- Server responds in N-KByte buffers (e.g. 1K) of data from a stream of unknown length (chunking not shown for brevity)

- Response ends at the end of stream or when the client closes the connection to seek to a new location

GET /path/file HTTP/1.1

Host: example.com

Range: bytes=0-

HTTP/1.1 206 Partial Content

Date: Wed, 15 Nov 1995 06:25:24 GMT

Last-Modified: Wed, 15 Nov 1995 04:58:08 GMT

Transfer-Encoding: chunked

Content-Type: multipart/byteranges; boundary=THIS_STRING_SEPARATES

--THIS_STRING_SEPARATES

Content-Type: file/mimetype

Content-Range: bytes 0-1023/*

...the first range...

--THIS_STRING_SEPARATES

Content-Type: file/mimetype

Content-Range: bytes 1024-2047/*

...the second range...

--THIS_STRING_SEPARATES—

...

-keith

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 13 Oct 2015 22:36:12 +1300

On 7/04/2015 11:39 p.m., Rodger Combs wrote:
> Yup; we use that for some platforms, but not all devices support it,
> and it limits us to the codecs supported by MPEG-TS (or that the
> platform supports in it). We use MPEG-DASH on some platforms, but it
> has similar issues. Both also require either knowledge of the segment
> durations beforehand (here, before they're generated), or the ability
> to force them to be consistent (which is impractical if remuxing
> streams a user's existing video files), though HLS implementations
> tend to be a fair bit more forgiving on this point than DASH ones.
> When it's possible, we've found that a single live-generated MKV file
> streamed over chunked HTTP is both compatible with a very wide
> variety of codecs, and very flexible about the sizes of a stream's
> GoPs and the like.
> 
> One thing I've had success in some experiments with lately is writing
> the MKV header to a single file, writing individual segments to
> separate files, and keeping track of their start timestamps. Then,
> when a request comes in for time T, I can return the header
> concatenated to the segment with the greatest start time <= T, and
> all segments after it. This approach is somewhat similar to how
> MPEG-DASH works, but with the work of preparing a fully-valid MKV
> file handled by the server instead of the client. The downside is
> that implementations often refuse to make any kind of seek in a
> resource that doesn't support range requests (and you can't properly
> support them when the content length is unknown), so I end up limited
> to seeks to the nearest on-server chunk boundary.

Please do not do that to generate payload streams without some
indicators in the headers or request-URL to distinguish the different
responses. It screws cacheability both in proxies and browsers which are
unaware of how the body is generated.

YouTube partial .swf responses used to do this when the user selected a
custom offset to start playing. There were a lot of problems and
complaints about truncated videos and even wrong codecs being delivered
to users, where one user selecting a late start would make other users
receive truncated videos for a while.

Nasty as they are multipart payloads would be better for HTTP as a whole
to deliver this type of service.

Or dare say it; chunked extensions would have been a perfect solution to
present a kind of multipart payload where the chunk extension headers
are teh multipart prefix. But that ship has sailed for 1.1. We might be
able to get something defined with HTTP/2 extensions or perhapse
PUSH_PROMISE that serves the use case nicer.

Amos