Subject: Issue with "bytes" Range Unit and live streaming

From: Craig Pratt <[email-masked]>
Date: Thu, 14 Apr 2016 17:26:16 -0600

Hello all,

We have a use case where we are "live streaming" media over
HTTP. As with other (non-segmented) HTTP streaming applications,
our live streaming solution has the HTTP client perform a GET on a
URI representing the live audio or video content stream and with
the server progressively returning data using chunked transfer
coding. The data returned to the server is "live" so long as the
client reads data as quickly as it's generated - with TCP flow
control allowing the client's reading of the response data to be
paced to the live chunk generation of the HTTP server.

More recently, use cases have appeared that require a
combination of random access and live streaming. For example, an
HTTP media server may want to support streaming of an in-progress
recording - with the HTTP client using HTTP byte Range requests to
access any point within the already-recorded content. But we've
run into issues supporting both live content transfer and HTTP
byte Range requests on the same resource. There's no good way that
we've found to get to this "live point" and also support random
access. The use case requires a byte range that ends at an
indeterminate point, e.g.

Range: bytes=1234567-*

However, the ABNF for the bytes Range Unit doesn't allow for
this form of request.

Rather than coming up with a proprietary solution, we are
proposing a new Range Unit called "bytes-live" as a potential
solution to this issue. Support for this new "bytes-live" Range
Unit could be exposed by servers (using the Accept-Ranges header)
for content that support both byte-wise random access and live
streaming. e.g. A server that supports both the standard HTTP/1.1
bytes Range Unit and bytes-live would include the header:

Accept-Ranges: bytes bytes-live

An HTTP client can use Range requests with the standard "bytes"
Range Unit to transfer stored content from a representation. And
when the bytes-live Range Unit is supported, an HTTP client can
request a byte range that starts or ends at the live point of the
content representation. For example, a client could request the
transfer of bytes starting at byte 1234567 and transitioning to
the live point once all stored content has been transferred using:

Range: bytes-live=1234567-*

We believe the bytes-live Range Unit could have multiple
applications outside of our immediate use cases. For instance,
internet-connected video cameras with on-board storage could use
bytes-live to support streaming of continuously recording content
with lower latencies than can be achieved using HLS, MPEG-DASH, or
other segmented video streaming techniques. And there are also
applications with on-demand transcoding.

The draft RFC can be found here https://tools.ietf.org/html/draft-pratt-httpbis-bytes-live-range-unit-00.
Please let me know what you think - and if there are other
solutions we haven't considered.

Regards,

craig

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Remy Lebeau <[email-masked]>
Date: Thu, 14 Apr 2016 18:20:32 -0700

AFAICT, the existing ABNF for byte ranges *does* already support what 
you are looking for, without having to introduce a new unit type:

     byte-range-spec = first-byte-pos "-" [ last-byte-pos ]

Simply omit the * and leave the second value empty:

Range: bytes=1234567-

RFC 2616 Section 14.35.1 says:

"If the last-byte-pos value is absent, or if the value is greater than 
or equal to the current length of the entity-body, last-byte-pos is 
taken to be equal to one less than the current length of the entity- 
body in bytes."

RFC 7233 Section 2.1 says:

"If the last-byte-pos value is absent, or if the value is greater than 
or equal to the current length of the representation data, the byte 
range is interpreted as the remainder of the representation (i.e., the 
server replaces the value of last-byte-pos with a value that is one less 
than the current length of the selected representation)."

Since the media is being recorded while streaming, the client can 
randomly jump around within the recorded portion of the media all it 
wants. Once jumped, it would simply be receiving media from the 
recording, not the live feed.  To jump back to the live feed, simply 
omit the "Range" header.

Remy Lebeau
Lebeau Software

On 4/14/2016 4:26 PM, Craig Pratt wrote:
> Hello all,
>
> We have a use case where we are "live streaming" media over HTTP. As 
> with other (non-segmented) HTTP streaming applications, our live 
> streaming solution has the HTTP client perform a GET on a URI 
> representing the live audio or video content stream and with the 
> server progressively returning data using chunked transfer coding. The 
> data returned to the server is "live" so long as the client reads data 
> as quickly as it's generated - with TCP flow control allowing the 
> client's reading of the response data to be paced to the live chunk 
> generation of the HTTP server.
>
> More recently, use cases have appeared that require a combination of 
> random access and live streaming. For example, an HTTP media server 
> may want to support streaming of an in-progress recording - with the 
> HTTP client using HTTP byte Range requests to access any point within 
> the already-recorded content. But we've run into issues supporting 
> both live content transfer and HTTP byte Range requests on the same 
> resource. There's no good way that we've found to get to this "live 
> point" and also support random access. The use case requires a byte 
> range that ends at an indeterminate point, e.g.
>
> Range: bytes=1234567-*
>
> However, the ABNF for the bytes Range Unit doesn't allow for this form 
> of request.
>
> Rather than coming up with a proprietary solution, we are proposing a 
> new Range Unit called "bytes-live" as a potential solution to this 
> issue. Support for this new "bytes-live" Range Unit could be exposed 
> by servers (using the Accept-Ranges header) for content that support 
> both byte-wise random access and live streaming. e.g. A server that 
> supports both the standard HTTP/1.1 bytes Range Unit and bytes-live 
> would include the header:
>
> An HTTP client can use Range requests with the standard "bytes" Range 
> Unit to transfer stored content from a representation. And when the 
> bytes-live Range Unit is supported, an HTTP client can request a byte 
> range that starts or ends at the live point of the content 
> representation. For example, a client could request the transfer of 
> bytes starting at byte 1234567 and transitioning to the live point 
> once all stored content has been transferred using:
>
> We believe the bytes-live Range Unit could have multiple applications 
> outside of our immediate use cases. For instance, internet-connected 
> video cameras with on-board storage could use bytes-live to support 
> streaming of continuously recording content with lower latencies than 
> can be achieved using HLS, MPEG-DASH, or other segmented video 
> streaming techniques. And there are also applications with on-demand 
> transcoding.
>
> The draft RFC can be found here 
> https://tools.ietf.org/html/draft-pratt-httpbis-bytes-live-range-unit-00. 
> Please let me know what you think - and if there are other solutions 
> we haven't considered.
>
> Regards,
>
> craig
> -- 
>
> craig pratt
>
> Caspia Consulting
>
> [email-masked]
>
> 503.746.8008
>
> 	
>
>
> 	
>
>
>

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 16 Apr 2016 00:36:19 +1200

On 15/04/2016 11:26 a.m., Craig Pratt wrote:
> Hello all,
> 
> We have a use case where we are "live streaming" media over HTTP. As with other 
> (non-segmented) HTTP streaming applications, our live streaming solution has the 
> HTTP client perform a GET on a URI representing the live audio or video content 
> stream and with the server progressively returning data using chunked transfer 
> coding. The data returned to the server is "live" so long as the client reads 
> data as quickly as it's generated - with TCP flow control allowing the client's 
> reading of the response data to be paced to the live chunk generation of the 
> HTTP server.

Please be careful not to rely on the chunks arriving at the client in
the same sizes as were sent.

The chunked encoding is a hop-by-hop feature so any HTTP intermediary
which they travel through may re-code the chunk sizes according to its
own output needs.

Cheers
Amos

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 15 Apr 2016 13:15:28 +0000

AFAICT, the existing ABNF for byte ranges *does* already support what you are looking for, without having to introduce a new unit type:

"If the last-byte-pos value is absent, or if the value is greater than or equal to the current length of the entity-body, last-byte-pos is taken to be equal to one less than the current length of the entity- body in bytes."

<bhs> Yes the client can make an open-ended request. The problem is the server has no means of providing an open-ended response.

"If the last-byte-pos value is absent, or if the value is greater than or equal to the current length of the representation data, the byte range is interpreted as the remainder of the representation (i.e., the server replaces the value of last-byte-pos with a value that is one less than the current length of the selected representation)."

<bhs> Correct. The server is required to provide an actual value for last-byte-pos. And that value is based on the current length. If the server is recording the content from a live stream, whatever value the server provides will no longer be the last byte almost as soon as the value is sent. The actual endpoint is constantly increasing.

Since the media is being recorded while streaming, the client can randomly jump around within the recorded portion of the media all it wants. Once jumped, it would simply be receiving media from the recording, not the live feed.  To jump back to the live feed, simply omit the "Range" header.

<bhs> The use case is not one of the client recording the stream. It is of the server recording the stream. Let me see if I can explain the use case better.
The user is watching content on a client that is being streamed from a server. The server is recording the content from a live stream. The client has limited memory and can only buffer about 15 seconds worth of content.
The user pauses watching the content and comes back after 1 minute to resume. The client sends the server a request to provide content from the paused byte position to “-“.
The server is required to provide a response that has an actual last byte value. The only last byte value the server has is the one for the precise moment of the request. After a minute, the server has a minute more of content but has provided the client with the content up to that responded last byte. Shouldn’t the server stop sending bytes once it gets to the value of its responded last-byte-pos? Even though it has now a whole lot more bytes and we know that the user just wants to keep watching? Should the client be expected to make continual requests to keep up with the ever-increasing endpoint?
The purpose of this proposal is to allow the server to provide an open-ended response, so the server can send bytes that it didn’t have at the time of the original request.

Barbara

---

From: Craig Pratt <[email-masked]>
Date: Fri, 15 Apr 2016 11:56:13 -0700

On 4/15/16 5:36 AM, Amos Jeffries
wrote:

Hello all,

Cheers
Amos

Most definitely - the client cannot make any presumption about the
framing. In the renderers I've worked with, the chunk framing is
removed before the AV renderer get the content.

The chunk size can affect the performance of the streaming, however.
A single large chunk can cause a client to stall and introduce
latency that persists for the duration of the streaming session. But
clients are typically able to handle these "late" arrivals since
they can also be caused by congestion.

What we're trying to prevent with this EC is requiring the client to
continuously perform GETs to receive new chunks if/when the live
content also supports random access. We want the client to simply be
able to block on the socket while waiting for the data to arrive in
the same way it would for a non-random-access live stream.

cp

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Göran Eriksson AP <[email-masked]>
Date: Fri, 15 Apr 2016 19:03:54 +0000

Hi,

Have a similar use case in a slightly different context. I find the draft
and discussion useful.

A few questions inline.

>AFAICT, the existing ABNF for byte ranges *does* already support what you
>are looking for, without having to introduce a new unit type:
>
>"If the last-byte-pos value is absent, or if the value is greater than or
>equal to the current length of the entity-body, last-byte-pos is taken to
>be equal to one less than the current length of the entity- body in
>bytes."
> 
><bhs> Yes the client can make an open-ended request. The problem is the
>server has no means of providing an open-ended response.

You mean response must contain content length?

>"If the last-byte-pos value is absent, or if the value is greater than or
>equal to the current length of the representation data, the byte range is
>interpreted as the remainder of the representation (i.e., the server
>replaces the value of last-byte-pos with
> a value that is one less than the current length of the selected
>representation)."
> 
><bhs> Correct. The server is required to provide an actual value for
>last-byte-pos. And that value is based on the current length. If the
>server is recording
> the content from a live stream, whatever value the server provides will
>no longer be the last byte almost as soon as the value is sent. The
>actual endpoint is constantly increasing.
>
>recording, not the live feed.  To jump back to the live feed,
> simply omit the "Range" header.
> 
><bhs> The use case is not one of the client recording the stream. It is
>of the server recording the stream. Let me see if I can explain the use
>case better.
>The user is watching content on a client that is being streamed from a
>server. The server is recording the content from a live stream.
>The client has limited
> memory and can only buffer about 15 seconds worth of content.
>The user pauses watching the content and comes back after 1 minute to
>resume. The client sends the server a request to provide content from the
>paused byte position
> to “-“. 
>The server is required to provide a response that has an actual last byte
>value. The only last byte value the server has is the one for the precise
>moment of
> the request. After a minute, the server has a minute more of content but
>has provided the client with the content up to that responded last byte.

Isn’t the server providing the client with a new manifest with the latest
‘last byte’ every 15 seconds or so? Or are you after a solution where this
approach is not used, instead with an “open request” in the request
template?

Would be good to know what kind of media player behaviour is assumed here,
especially the manifest part of it.

>Shouldn’t the server stop sending bytes once it gets to the value of its
>responded last-byte-pos? Even though
> it has now a whole lot more bytes and we know that the user just wants
>to keep watching? Should the client be expected to make continual
>requests to keep up with the ever-increasing endpoint?
>The purpose of this proposal is to allow the server to provide an
>open-ended response, so the server can send bytes that it didn’t have at
>the time of the original
> request.
> 
>Barbara
>

---

From: Remy Lebeau <[email-masked]>
Date: Fri, 15 Apr 2016 12:21:50 -0700

> You mean response must contain content length?

Not a Content-Length header, no.  A "Transfer-Encoding: chunked" header 
can be used to represent an open-ended response.  But apparently the 
Content-Range response header can't be open-ended for a requested 
open-ended range.  It provides the requested first-byte-pos and the 
current last-byte-pos.  However, it can report an instance length of * 
to indicate the server does not know the full length of the entity being 
requested.

Remy Lebeau
Lebeau Software

---

From: Craig Pratt <[email-masked]>
Date: Fri, 15 Apr 2016 12:36:21 -0700

Barbara did a good job providing some
needed context. But I want to respond to the "jump to live" aspect
(below).

cp

On 4/14/16 6:20 PM, Remy Lebeau wrote:

AFAICT, the existing ABNF for byte ranges *does* already support
what you are looking for, without having to introduce a new unit
type:

Range: bytes=1234567-

"If the last-byte-pos value is absent, or if the value is greater
than or equal to the current length of the entity-body,
last-byte-pos is taken to be equal to one less than the current
length of the entity- body in bytes."

"If the last-byte-pos value is absent, or if the value is greater
than or equal to the current length of the representation data,
the byte range is interpreted as the remainder of the
representation (i.e., the server replaces the value of
last-byte-pos with a value that is one less than the current
length of the selected representation)."

Since the media is being recorded while streaming, the client can
randomly jump around within the recorded portion of the media all
it wants. Once jumped, it would simply be receiving media from the
recording, not the live feed.  To jump back to the live feed,
simply omit the "Range" header.

Remy Lebeau
Lebeau Software

[cp] There are a couple of issues with the above usage model (which
we definitely considered and discussed)

(1) The GET request (without a Range header) is
typically expected to transfer the entire representation. In the
kind of in-progress recording example, this means the transfer
would start from the first recorded byte (and not the live point).
While there might be some room for various interpretations, our
use case is constrained by the DLNA specifications here (and
implementations based on it). The spec and endpoints expect
Range-less transfers to start at the beginning of the
representation. And really, if there is ambiguity on the GET
response, I'd rather not make assumptions one way or the other.

(2) Even if a server returned live data from a Range-less GET, you
can end up with data discontinuity. For instance, say you want to
jump ahead in the stream - through the random access portion and
into the live data. Under this model, I might perform a Range
request with "bytes=1234567-", process the data, and then jump to
the live point once I'd consumed that response. Well, no matter
how small I make that Range request, data could have been added
between the time I did the "bytes=1234567-" request and I jump to
the live point. Since the client can't know/detect this
discontinuity, it has to be implemented assuming the worst -
flushing the framebuffer, resetting the rendering state, and
waiting for the next keyframe - which can be seconds away.

In fact, and in general, jumping to live on a video stream like this
is best accomplished by performing a request that precedes
the end of the stream. This is due to this dependance on a key frame
to initialize rendering. Being able to capture the most recent key
frame can make a difference of seconds in initial presentation of a
live stream. But it definitely requires continuity - which is what
the bytes-live range unit is intended to provide.

I hadn't made much of this "discontinuity" aspect in the RFC. Please
let me know if you think it deserves more attention/explanation. And
thanks much for the thoughts.

Happy Friday,

cp

On 4/14/2016 4:26 PM, Craig Pratt
wrote:

Hello all,

We have a use case where we are "live streaming" media over
HTTP. As with other (non-segmented) HTTP streaming
applications, our live streaming solution has the HTTP client
perform a GET on a URI representing the live audio or video
content stream and with the server progressively returning
data using chunked transfer coding. The data returned to the
server is "live" so long as the client reads data as quickly
as it's generated - with TCP flow control allowing the
client's reading of the response data to be paced to the live
chunk generation of the HTTP server.

More recently, use cases have appeared that require a
combination of random access and live streaming. For example,
an HTTP media server may want to support streaming of an
in-progress recording - with the HTTP client using HTTP byte
Range requests to access any point within the already-recorded
content. But we've run into issues supporting both live
content transfer and HTTP byte Range requests on the same
resource. There's no good way that we've found to get to this
"live point" and also support random access. The use case
requires a byte range that ends at an indeterminate point,
e.g.

Range: bytes=1234567-*

However, the ABNF for the bytes Range Unit doesn't allow
for this form of request.

Rather than coming up with a proprietary solution, we are
proposing a new Range Unit called "bytes-live" as a potential
solution to this issue. Support for this new "bytes-live"
Range Unit could be exposed by servers (using the
Accept-Ranges header) for content that support both byte-wise
random access and live streaming. e.g. A server that supports
both the standard HTTP/1.1 bytes Range Unit and bytes-live
would include the header:

An HTTP client can use Range requests with the standard
"bytes" Range Unit to transfer stored content from a
representation. And when the bytes-live Range Unit is
supported, an HTTP client can request a byte range that starts
or ends at the live point of the content representation. For
example, a client could request the transfer of bytes starting
at byte 1234567 and transitioning to the live point once all
stored content has been transferred using:

We believe the bytes-live Range Unit could have multiple
applications outside of our immediate use cases. For instance,
internet-connected video cameras with on-board storage could
use bytes-live to support streaming of continuously recording
content with lower latencies than can be achieved using HLS,
MPEG-DASH, or other segmented video streaming techniques. And
there are also applications with on-demand transcoding.

Regards,

craig

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Göran Eriksson AP <[email-masked]>
Date: Fri, 15 Apr 2016 19:37:48 +0000

Tnx for clarification.

>Not a Content-Length header, no.

---

From: Göran Eriksson AP <[email-masked]>
Date: Fri, 15 Apr 2016 19:53:49 +0000

>especially the manifest part of it.

The draft pretty much covered that.

Nice work.

---

From: Craig Pratt <[email-masked]>
Date: Fri, 15 Apr 2016 13:00:56 -0700

Thanks Göran.

Clarifications in-line.

cp

On 4/15/16 12:03 PM, Göran Eriksson AP wrote:

Hi,

A few questions inline.

[cp] Remy answered this: Content-Length is not expressly required
(since that would prevent chunked and other transfer encodings). But
the Content-Range header can only communicate a fixed response.

[cp] The media player in this case doesn't assume the presence of a
manifest or any other kind of segmented content representation.
Think of a single transport stream or fragmented MP4 (iso bmff)
(represented by a single URI) that grows over time.

Barbara

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 15 Apr 2016 13:24:36 -0700

> On Apr 15, 2016, at 1:00 PM, Craig Pratt <[email-masked]> wrote:

> [cp] Remy answered this: Content-Length is not expressly required (since that would prevent chunked and other transfer encodings). But the Content-Range header can only communicate a fixed response.

That is incorrect.  See RFC7233, Sec 4.2:

   For byte ranges, a sender SHOULD indicate the complete length of the
   representation from which the range has been extracted, unless the
   complete length is unknown or difficult to determine.  An asterisk
   character ("*") in place of the complete-length indicates that the
   representation length was unknown when the header field was
   generated.

   The following example illustrates when the complete length of the
   selected representation is known by the sender to be 1234 bytes:

     Content-Range: bytes 42-1233/1234

   and this second example illustrates when the complete length is
   unknown:

     Content-Range: bytes 42-1233/*

There is no need for another range unit.

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 15 Apr 2016 13:42:41 -0700

On Apr 15, 2016, at 1:24 PM, Roy T. Fielding <[email-masked]> wrote:
> 
>> On Apr 15, 2016, at 1:00 PM, Craig Pratt <[email-masked] <mailto:[email-masked]>> wrote:
> 

Oh, never mind, now I see that you are referring to the second number being fixed.

I think I would prefer that be solved by allowing last-byte-pos to be empty, just like
it is for the Range request.  I think such a fix is just as likely to be interoperable as
introducing a special range type (same failure cases).

....Roy

---

From: Darshak Thakore <[email-masked]>
Date: Fri, 15 Apr 2016 20:59:42 +0000

On 4/15/16, 2:24 PM, "Roy T. Fielding" <[email-masked]<mailto:[email-masked]>> wrote:

On Apr 15, 2016, at 1:00 PM, Craig Pratt <[email-masked]<mailto:[email-masked]>> wrote:

DT>> The issue we have is with specifying 1233 as the last-byte-pos since that will change as soon as it is set.

....Roy

---

From: Remy Lebeau <[email-masked]>
Date: Fri, 15 Apr 2016 14:04:55 -0700

What you say is true, but you are only focusing on the *second half* of 
the Content-Range value, ignoring the *first half*.  Here is the full 
definition:

Content-Range = byte-content-range / other-content-range

byte-content-range = bytes-unit SP ( byte-range-resp / unsatisfied-range )

byte-range-resp = byte-range "/" ( complete-length / "*" )

byte-range = first-byte-pos "-" last-byte-pos

unsatisfied-range = "*/" complete-length

complete-length = 1*DIGIT

other-content-range = other-range-unit SP other-range-resp

other-range-resp = *CHAR

If the range is satisfied, the *complete-length* can be "*", but the 
*byte-range-resp* cannot be "*".

If the range is unsatisfied, the *byte-range-resp* must be "*", and the 
*complete-length* cannot be "*".

Remy Lebeau
Lebeau Software

---

From: Darshak Thakore <[email-masked]>
Date: Fri, 15 Apr 2016 21:09:45 +0000

Oops, my earlier email got criss-crossed with this one.

DT>> We would be ok with that approach too. Our initial thought was that it would be more invasive (less acceptable) so we opted for suggesting an alternate unit. That way we don’t break existing clients and any client that desires the “open-ended last-byte-pos” feature could support the new unit.

....Roy

---

From: <[email-masked]>
Date: Mon, 18 Apr 2016 07:34:06 +0000

On Friday,15 April 2016 22:43, [email-masked] wrote:
> I think I would prefer that be solved by allowing last-byte-pos to be empty, just
> like it is for the Range request.  I think such a fix is just as likely to be
> interoperable as introducing a special range type (same failure cases).
>
> ....Roy

+1000

A very similar idea was proposed before [1] as an I-D [2] by Rodger Coombs. We've also brought this up informally with other members of the WG.

Alas, in our experience range requests don't seem to be a high priority :(  For example, the problem of combining gzip with range requests is still unsolved [3].

[1] https://lists.w3.org/Archives/Public/ietf-http-wg/2015AprJun/0122.html
[2] https://tools.ietf.org/html/draft-combs-http-indeterminate-range-01
[3] https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/1327.html

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Craig Pratt <[email-masked]>
Date: Mon, 18 Apr 2016 02:04:22 -0700

On 4/18/16 12:34 AM, [email-masked]
wrote:

Oh, never mind, now I see that you are referring to the second number being
fixed.

....Roy

+1000

[cp] Yeah, it's unfortunate that no solutions have moved forward for
this widely-desired feature. I can only assume that people just
started defining proprietary solutions - which is unfortunate. I'll
try to be "persistent"... ;^J

[cp] As was mentioned, the issue with just arbitrarily allowing an
open-ended Content-Range response (omitting last-byte-pos) is that
there's no good way for a client to indicate it can support
reception of a Content-Range without a last-byte-pos. So I would
fully expect many clients to fail in "unpredictable ways"
(disconnecting, crashing, etc).

[cp] I see that the indeterminate length proposal you referenced in
your first citation introduces a "Accept-Indefinite-Ranges" header
to prevent this issue. But I think this brings with it some other
questions. e.g. Would this apply to any/all Range Units which may be
introduced in the future? How can a Client issue a request that
starts at the "live point"? It feels like it has one hand tied
behind its back.

[cp] If I could, I would prefer to go back in time and advocate for
an alternate ABNF for the bytes Range Unit. Seeing as that's not an
option, I think using this well- and long-defined Range Unit
extension mechanism seems like a good path forward as it should not
create interoperability issues between clients and servers.

[cp] And I would hope adding a Range Unit would have a low/lower bar
for acceptance. e.g. If a Range Unit fills a useful role, is
well-defined, and isn't redundant, it seems reasonable that it
should be accepted as it shouldn't impact existing HTTP/1.1
semantics. In fact, the gzip case (referenced in your third
citation) seems like a perfect application of the Range Unit (better
than bytes-live). If there's interest, I'll write up an RFC to
demonstrate...

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Thorsten Lohmar <[email-masked]>
Date: Mon, 18 Apr 2016 15:10:49 +0000

Hi Craig, all,

My colleague Göran asked me some question around the problem and I would like to raise these questions directly to you. Of course, there are some alternative solutions available, where the client can work out the different things from a manifest. But you seem to look for a simple solution, which works with non-segmented media on a single HTTP session.

When I understood it correctly, an HTTP server is making a live stream available using HTTP. A normal live stream can be opened with a single HTTP request and the server can serve data "from the live point" either with or without HTTP chunked delivery. The server cannot give a Content-Length, since this is an ongoing live stream of unknown size.

Your use-case seem to be about recording of content. Client should access content from the recorded part, but should be able to jump to the live-point. I assume that you are not looking into sliding window recordings (i.e. timeshift). I assume that the a single program is continuous recording and the HTTP object is growing until the end of the live session, correct?

In any case, how does the client know "good" byte range offsets (i.e. service access points) to tune into the recording? Or is the assumption, that the client can synchronize to the media stream from any byte range?

How should the client know, which byte ranges are already available on the server? When the client is playing back from the recorded part and would like to skip 5min forward, how does the client know, whether a normal range request is needed or whether the client should as for the live point? What type of HTTP Status code should be provided, when the range request is not yet available of the server?

Can you please clarify the questions?

BR,
Thorsten

From: Craig Pratt [mailto:[email-masked]]
Sent: Monday, April 18, 2016 11:04 AM
To: [email-masked]; [email-masked]
Cc: Göran Eriksson AP; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Issue with "bytes" Range Unit and live streaming

On 4/18/16 12:34 AM, [email-masked]<mailto:[email-masked]> wrote:

On Friday,15 April 2016 22:43, [email-masked]<mailto:[email-masked]> wrote:

fixed.

....Roy

+1000

[3] https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/1327.html
[cp] Yeah, it's unfortunate that no solutions have moved forward for this widely-desired feature. I can only assume that people just started defining proprietary solutions - which is unfortunate. I'll try to be "persistent"... ;^J

[cp] As was mentioned, the issue with just arbitrarily allowing an open-ended Content-Range response (omitting last-byte-pos) is that there's no good way for a client to indicate it can support reception of a Content-Range without a last-byte-pos. So I would fully expect many clients to fail in "unpredictable ways" (disconnecting, crashing, etc).

[cp] I see that the indeterminate length proposal you referenced in your first citation introduces a "Accept-Indefinite-Ranges" header to prevent this issue. But I think this brings with it some other questions. e.g. Would this apply to any/all Range Units which may be introduced in the future? How can a Client issue a request that starts at the "live point"? It feels like it has one hand tied behind its back.

[cp] If I could, I would prefer to go back in time and advocate for an alternate ABNF for the bytes Range Unit. Seeing as that's not an option, I think using this well- and long-defined Range Unit extension mechanism seems like a good path forward as it should not create interoperability issues between clients and servers.

[cp] And I would hope adding a Range Unit would have a low/lower bar for acceptance. e.g. If a Range Unit fills a useful role, is well-defined, and isn't redundant, it seems reasonable that it should be accepted as it shouldn't impact existing HTTP/1.1 semantics. In fact, the gzip case (referenced in your third citation) seems like a perfect application of the Range Unit (better than bytes-live). If there's interest, I'll write up an RFC to demonstrate...

--

craig pratt

Caspia Consulting

[email-masked]<mailto:[email-masked]>

503.746.8008

---

From: Craig Pratt <[email-masked]>
Date: Mon, 18 Apr 2016 13:29:03 -0700

[cc-ing the co-authors]

Hi Thorsten,

I'm happy to help provide whatever answers I can.

Reply in-line.

cp

On 4/18/16 8:10 AM, Thorsten Lohmar wrote:

Hi
Craig, all,

My
colleague Göran asked me some question around the problem
and I would like to raise these questions directly to you.
Of course, there are some alternative solutions available,
where the client can work out the different things from a
manifest. But you seem to look for a simple solution, which
works with non-segmented media on a single HTTP session.

When
I understood it correctly, an HTTP server is making a live
stream available using HTTP. A normal live stream can be
opened with a single HTTP request and the server can serve
data “from the live point” either with or without HTTP
chunked delivery. The server cannot give a Content-Length,
since this is an ongoing live stream of unknown size.

[cp] all correct.

Your
use-case seem to be about recording of content. Client
should access content from the recorded part, but should be
able to jump to the live-point. I assume that you are not
looking into sliding window recordings (i.e. timeshift). I
assume that the a single program is continuous recording and
the HTTP object is growing until the end of the live
session, correct?

[cp] I didn't spell it out in the draft, but I would like to
consider adding clarifications for the time-shift cases. This should
just be a matter of a Client requesting one thing and getting
another. e.g. "Range: bytes-live=0-*" results in "Content-Range:
bytes-live 123456-*". In either case, you're correct: the end of the
representation is moving forward in time until the end of the live
session.

In
any case, how does the client know “good” byte range offsets
(i.e. service access points) to tune into the recording? Or
is the assumption, that the client can synchronize to the
media stream from any byte range?

[cp] For byte-level access, random access implementation is up to
the client. For some containers this is easier than others. e.g. For
MP4, the random access points can be established by reading the
movie and fragment header(s). For something like MP2, it's trickier
of course.

[cp] One major feature this draft allows is for retrieval of bytes
just preceding the live point. So for example, a client can do a
Range head request like "Range: bytes=0-", get a "Content-Range:
bytes 0-1234567/*", then perform something like a "Range:
bytes-live=1200000-*", and prime its framebuffer with 34567 bytes of
data that precede the live point - allowing for the client to find
an access point (e.g. mpeg2 start codes) and to allow live
presentation to display much sooner than it would from the live
point (without random access).

How
should the client know, which byte ranges are already
available on the server? When the client is playing back
from the recorded part and would like to skip 5min forward,
how does the client know, whether a normal range request is
needed or whether the client should as for the live point?
What type of HTTP Status code should be provided, when the
range request is not yet available of the server?

[cp] We're not trying to come up with a universal solution for
performing time-based seek on all media formats with this draft. So
some of this is out of scope. But let me see if I can fill in some
of the blanks.

[cp] Some applications of media streaming have time-based indexing
facilities built-in. e.g. MP4 (ISO BMFF) containers allow time and
data to be associated using the various internal, mandatory metadata
"boxes". In other cases, applications may provide a separate
resource that contains time-to-byte mappings (e.g. content index
files). In either case, there's a facility for mapping time offsets
to byte offsets - or sometimes the client incorporates heuristics to
perform time skips (e.g. VLC will do this on some file formats).

[cp] In all these cases, there's some mechanism that maps time
offsets to byte offsets.

[cp] When it comes to the available byte range, a client can know
what data range is available by utilizing a HEAD request with a
"Range: bytes=0-". The "Content-Range" response can contain
something like "Content-Range: bytes 0-1234567/*" which tells the
client both the current randomly accessible content range (via the
"0-1234567") and that the content is of indeterminate length (via
the "*").

[cp] Putting this all together, a client would implement a 5-minute
skip by:

(1) Adding 5 minutes to your current play time,

(2) determining the byte offset for that given time using the
appropriate index/heuristic (e.g. "3456789"),

(3) if the time is outside the index, jump to the live point and
update the time to the last-index time or other means (e.g. using
"Range: bytes-live=340000-*" to pre-buffer/pre-prime the
frame/sample buffer),

(4) if the time is inside the index, either perform a standard
bytes Range request to retrieve an implementation-specific quantum
of time or data (e.g. "Range: bytes=3456789-3556789") and render.

[cp] Again, some of this is out of scope, but I hope that clarifies
a common use case.

[cp] Regarding the status code, RFC7233 (section 4.4) indicates that
code 416 (Range Not Satisfiable) must be returned when "the current
extent of the selected resource or that the set of ranges requested
has been rejected due to invalid ranges or an excessive request of
small or overlapping ranges." This part of 4.4 applies to *all*
Range requests - regardless of the Range Unit.

[cp] The bytes-live draft then goes on to say that "A
bytes-live-range-specifier is considered unsatisfiable if the
first-byte-pos is larger than the current length of the
representation". This could probably be elaborated on a bit. But
this is supposed to be the "hook" into the 4.4 language.

Can
you please clarify the questions?

[cp] I hope I succeeded (at least partially). Apologies for the long
response. I wanted to make sure I was answering your questions.

BR,

Thorsten

From:
Craig Pratt [mailto:[email-masked]]

Cc: Göran Eriksson AP; [email-masked];
[email-masked]; [email-masked];
[email-masked]; [email-masked];
[email-masked]

Subject: Re: Issue with "bytes" Range Unit and
live streaming

fixed.

....Roy

+1000

[cp] Yeah, it's unfortunate that no
solutions have moved forward for this widely-desired
feature. I can only assume that people just started defining
proprietary solutions - which is unfortunate. I'll try to be
"persistent"... ;^J

[cp] As was mentioned, the issue with just arbitrarily
allowing an open-ended Content-Range response (omitting
last-byte-pos) is that there's no good way for a client to
indicate it can support reception of a Content-Range without
a last-byte-pos. So I would fully expect many clients to
fail in "unpredictable ways" (disconnecting, crashing, etc).

[cp] I see that the indeterminate length proposal you
referenced in your first citation introduces a
"Accept-Indefinite-Ranges" header to prevent this issue. But
I think this brings with it some other questions. e.g. Would
this apply to any/all Range Units which may be introduced in
the future? How can a Client issue a request that starts at
the "live point"? It feels like it has one hand tied behind
its back.

[cp] If I could, I would prefer to go back in time and
advocate for an alternate ABNF for the bytes Range Unit.
Seeing as that's not an option, I think using this well- and
long-defined Range Unit extension mechanism seems like a
good path forward as it should not create interoperability
issues between clients and servers.

[cp] And I would hope adding a Range Unit would have a
low/lower bar for acceptance. e.g. If a Range Unit fills a
useful role, is well-defined, and isn't redundant, it seems
reasonable that it should be accepted as it shouldn't impact
existing HTTP/1.1 semantics. In fact, the gzip case
(referenced in your third citation) seems like a perfect
application of the Range Unit (better than bytes-live). If
there's interest, I'll write up an RFC to demonstrate...

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Thorsten Lohmar <[email-masked]>
Date: Mon, 18 Apr 2016 22:50:13 +0000

Hi Craig, all,

Thanks for the clarification. Some further question inline

BR,
Thorsten

From: Craig Pratt [mailto:[email-masked]]
Sent: Monday, April 18, 2016 10:29 PM
To: Thorsten Lohmar; [email-masked]; [email-masked]
Cc: Göran Eriksson AP; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Darshak Thakore; STARK, BARBARA H
Subject: Re: Issue with "bytes" Range Unit and live streaming

[cc-ing the co-authors]

Hi Thorsten,

Reply in-line.

cp

Your use-case seem to be about recording of content. Client should access content from the recorded part, but should be able to jump to the live-point. I assume that you are not looking into sliding window recordings (i.e. timeshift). I assume that the a single program is continuous recording and the HTTP object is growing until the end of the live session, correct?
[cp] I didn't spell it out in the draft, but I would like to consider adding clarifications for the time-shift cases. This should just be a matter of a Client requesting one thing and getting another. e.g. "Range: bytes-live=0-*" results in "Content-Range: bytes-live 123456-*". In either case, you're correct: the end of the representation is moving forward in time until the end of the live session.

[TL] The "Range: bytes-live=0-*" case is not clear to me. Your ID says "All bytes currently in the representation and those appended to the end of the representation after the request is processed". I get the impression, that the server is deleting all data before a certain timepoint (aka, behavior of a slighting window timeshift).  So, the client seems to request all data from the beginning of the timeshift buffer. Why does the server need to change the byte offset from 0 to 123456?
I can understand, that the server must signal "growing resource" in the response.

In any case, how does the client know "good" byte range offsets (i.e. service access points) to tune into the recording? Or is the assumption, that the client can synchronize to the media stream from any byte range?
[cp] For byte-level access, random access implementation is up to the client. For some containers this is easier than others. e.g. For MP4, the random access points can be established by reading the movie and fragment header(s). For something like MP2, it's trickier of course.
[TL] Well, in case of fMP4, the client needs to get the Movie Header for initialization. Then, proper access point are fragment boundaries. There are various ways to signal time to byte-offsets.
In case of TS, the client needs a PAT, PMT and PES starts for tune-in. It is a bit more tricky, but also here are solutions.
But the email talks about "none-segmented" media. The draft talks about "mimicking segmented media". fMP4 is actually the way to create ISO-BMFF segments. So, it is for segmented media, but without a separate manifest?

[cp] One major feature this draft allows is for retrieval of bytes just preceding the live point. So for example, a client can do a Range head request like "Range: bytes=0-", get a "Content-Range: bytes 0-1234567/*", then perform something like a "Range: bytes-live=1200000-*", and prime its framebuffer with 34567 bytes of data that precede the live point - allowing for the client to find an access point (e.g. mpeg2 start codes) and to allow live presentation to display much sooner than it would from the live point (without random access).
[TL] So, how does the client know, that the proper fragment boundary is at byte position 120000? Do you assume that the client first fetches a time-to-byte offset file, which tells the client that a access point (e.g. a fragment boundary) is at byte pos 120000? If yes, why does the client need the HEAD request, when it already has the byte position?

How should the client know, which byte ranges are already available on the server? When the client is playing back from the recorded part and would like to skip 5min forward, how does the client know, whether a normal range request is needed or whether the client should as for the live point? What type of HTTP Status code should be provided, when the range request is not yet available of the server?
[cp] We're not trying to come up with a universal solution for performing time-based seek on all media formats with this draft. So some of this is out of scope. But let me see if I can fill in some of the blanks.
[TL] Ok, not everything needs to be in-scope. But an essential assumption should be, whether the client has a time-to-byteoffset table or whether the client can determine precisely the fragment boundary positions.

[cp] Some applications of media streaming have time-based indexing facilities built-in. e.g. MP4 (ISO BMFF) containers allow time and data to be associated using the various internal, mandatory metadata "boxes". In other cases, applications may provide a separate resource that contains time-to-byte mappings (e.g. content index files). In either case, there's a facility for mapping time offsets to byte offsets - or sometimes the client incorporates heuristics to perform time skips (e.g. VLC will do this on some file formats).

[TL] Yes. fMP4 supports this and MPEG DASH is leveraging this. But the live-point is not described in the fragments. The client determines the livepoint from the manifest.

[cp] In all these cases, there's some mechanism that maps time offsets to byte offsets.
[TL] Yes

[cp] When it comes to the available byte range, a client can know what data range is available by utilizing a HEAD request with a "Range: bytes=0-". The "Content-Range" response can contain something like "Content-Range: bytes 0-1234567/*" which tells the client both the current randomly accessible content range (via the "0-1234567") and that the content is of indeterminate length (via the "*").
[TL] So, that is the existing Content-Range response, but with an '*' to indicate the unknown content-length, correct?

[cp] Putting this all together, a client would implement a 5-minute skip by:
    (1) Adding 5 minutes to your current play time,
    (2) determining the byte offset for that given time using the appropriate index/heuristic (e.g. "3456789"),
    (3) if the time is outside the index, jump to the live point and update the time to the last-index time or other means (e.g. using "Range: bytes-live=340000-*" to pre-buffer/pre-prime the frame/sample buffer),
    (4) if the time is inside the index, either perform a standard bytes Range request to retrieve an implementation-specific quantum of time or data (e.g. "Range: bytes=3456789-3556789") and render.
[TL] In (2), How does the client determine the byte offset? fMP4 requires precise byteoffset, In case of TS, the client can sync to the stream by first searching for 0x47 sync bytes. In (3), how does the client determine "outside of the index"? Seems that some sort of manifest is implicitly needed, which allows the client to understand the latest byte pos.

[cp] Again, some of this is out of scope, but I hope that clarifies a common use case.
[TL] Would be good to clarify, what information the client needs to get in order to do the operations. How the client gets the info can be left out-of-scope.

[cp] Regarding the status code, RFC7233 (section 4.4) indicates that code 416 (Range Not Satisfiable) must be returned when "the current extent of the selected resource or that the set of ranges requested has been rejected due to invalid ranges or an excessive request of small or overlapping ranges." This part of 4.4 applies to *all* Range requests - regardless of the Range Unit.
[TL] ok.

[cp] The bytes-live draft then goes on to say that "A bytes-live-range-specifier is considered unsatisfiable if the first-byte-pos is larger than the current length of the representation". This could probably be elaborated on a bit. But this is supposed to be the "hook" into the 4.4 language.

Can you please clarify the questions?
[cp] I hope I succeeded (at least partially). Apologies for the long response. I wanted to make sure I was answering your questions.
[TL] Gets a bit clearer, but I still don't understand the "mimic HLS or DASH". DASH / HLS focuses on CDN optimization by creating a sequence of individual files. The client can work out the live-point URL from the manifest. Each segment is a "good" access point (in DASH always box boundaries and in HLS always TS boundaries even with PAT / PMT). So, the key issue here is to clarify, how the client gets the byte offsets of the fragment boundaries for range requests.

BR,
Thorsten

From: Craig Pratt [mailto:[email-masked]]
Sent: Monday, April 18, 2016 11:04 AM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: Göran Eriksson AP; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: Issue with "bytes" Range Unit and live streaming

fixed.

....Roy

+1000

--

craig pratt

Caspia Consulting

503.746.8008

--

craig pratt

Caspia Consulting

503.746.8008

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Apr 2016 17:19:52 +1000

On 16 Apr 2016, at 6:42 AM, Roy T. Fielding <[email-masked]> wrote:
> 
> I think I would prefer that be solved by allowing last-byte-pos to be empty, just like it is for the Range request.  I think such a fix is just as likely to be interoperable as introducing a special range type (same failure cases).

IIRC the issue is that the failure case for a cache -- especially in an intermediary -- is not so simple; if Content-Range parsing fails and something gets stored, the results are unpredictable, and *could* be a lot worse than an unsupported new range-unit (which would just get ignored).

Historically, we've been shy about changing the semantics or syntax of widely-deployed protocol elements, for the excellent reason that we don't -- and can't -- know all of the implementations, or consequences.

On the other hand, introducing a new range-unit is expensive; it won't be recognised and taken advantage of in things like proxy caches until they're updated, which we know can take a very long time. Furthermore, Rodger showed us a variety of bugs in existing client implementations that assume that the only range-unit that will ever exist is 'bytes'.

It would be helpful if we had some data about implementation behaviours regarding these two things (open-ended Content-Range and new range-unit); even if we don't know all of the implementations, we might be able to rule out one (or both) of them based upon practicality. 

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Apr 2016 17:19:18 +1000

Hi Craig,

Thanks very much for that. As others have noted, this has come up before; having a concrete proposal helps move the discussion forward.

I've recorded the draft on our "Potential Future Work" page:
  https://github.com/httpwg/wiki/wiki/WatchList

I'll respond in other parts of the discussion, but one quick bit of feedback; it's not a good idea to use the substring "bytes" in a new range-unit, because some implementations will be scanning for that anywhere in the header (Rodger Combs gave us a heads-up about those bugs in the previous discussion, and it's an unfortunately common pattern with other parts of the stack too).

Cheers,

> On 15 Apr 2016, at 9:26 AM, Craig Pratt <[email-masked]> wrote:
> 
> Hello all,
> 
> We have a use case where we are "live streaming" media over HTTP. As with other (non-segmented) HTTP streaming applications, our live streaming solution has the HTTP client perform a GET on a URI representing the live audio or video content stream and with the server progressively returning data using chunked transfer coding. The data returned to the server is "live" so long as the client reads data as quickly as it's generated - with TCP flow control allowing the client's reading of the response data to be paced to the live chunk generation of the HTTP server.
> 
> More recently, use cases have appeared that require a combination of random access and live streaming. For example, an HTTP media server may want to support streaming of an in-progress recording - with the HTTP client using HTTP byte Range requests to access any point within the already-recorded content. But we've run into issues supporting both live content transfer and HTTP byte Range requests on the same resource. There's no good way that we've found to get to this "live point" and also support random access. The use case requires a byte range that ends at an indeterminate point, e.g.
> 
>     Range: bytes=1234567-*
> 
> However, the ABNF for the bytes Range Unit doesn't allow for this form of request.
> 
> Rather than coming up with a proprietary solution, we are proposing a new Range Unit called "bytes-live" as a potential solution to this issue. Support for this new "bytes-live" Range Unit could be exposed by servers (using the Accept-Ranges header) for content that support both byte-wise random access and live streaming. e.g. A server that supports both the standard HTTP/1.1 bytes Range Unit and bytes-live would include the header:
> 
> An HTTP client can use Range requests with the standard "bytes" Range Unit to transfer stored content from a representation. And when the bytes-live Range Unit is supported, an HTTP client can request a byte range that starts or ends at the live point of the content representation. For example, a client could request the transfer of bytes starting at byte 1234567 and transitioning to the live point once all stored content has been transferred using:
> 
> We believe the bytes-live Range Unit could have multiple applications outside of our immediate use cases. For instance, internet-connected video cameras with on-board storage could use bytes-live to support streaming of continuously recording content with lower latencies than can be achieved using HLS, MPEG-DASH, or other segmented video streaming techniques. And there are also applications with on-demand transcoding.
> 
> The draft RFC can be found here https://tools.ietf.org/html/draft-pratt-httpbis-bytes-live-range-unit-00. Please let me know what you think - and if there are other solutions we haven't considered.
> 
> Regards,
> 
> craig
> -- 
> craig pratt
> Caspia Consulting
> [email-masked]
> 503.746.8008
> 
> 
> 
>

---

From: Craig Pratt <[email-masked]>
Date: Tue, 19 Apr 2016 02:55:13 -0700

Hey Thorsten,

I'll try to reply in-line.

cp

On 4/18/16 3:50 PM, Thorsten Lohmar wrote:

Hi
Craig, all,

Thanks
for the clarification. Some further question inline

BR,

Thorsten

From:

To: Thorsten Lohmar; [email-masked];
[email-masked]

Cc: Göran Eriksson AP; [email-masked];
[email-masked]; [email-masked];
[email-masked]; [email-masked];
[email-masked]; Darshak Thakore; STARK, BARBARA
H

[cc-ing the co-authors]

Hi Thorsten,

Reply in-line.

cp

Hi
Craig, all,

My
colleague Göran asked me some question around the
problem and I would like to raise these questions
directly to you. Of course, there are some alternative
solutions available, where the client can work out the
different things from a manifest. But you seem to look
for a simple solution, which works with non-segmented
media on a single HTTP session.

When
I understood it correctly, an HTTP server is making a
live stream available using HTTP. A normal live stream
can be opened with a single HTTP request and the server
can serve data “from the live point” either with or
without HTTP chunked delivery. The server cannot give a
Content-Length, since this is an ongoing live stream of
unknown size.

[cp] all correct.

Your
use-case seem to be about recording of content. Client
should access content from the recorded part, but should
be able to jump to the live-point. I assume that you are
not looking into sliding window recordings (i.e.
timeshift). I assume that the a single program is
continuous recording and the HTTP object is growing until
the end of the live session, correct?

[cp] I didn't spell it out in the draft,
but I would like to consider adding clarifications for the
time-shift cases. This should just be a matter of a Client
requesting one thing and getting another. e.g. "Range:
bytes-live=0-*" results in "Content-Range: bytes-live
123456-*". In either case, you're correct: the end of the
representation is moving forward in time until the end of
the live session.

[TL]
The "Range: bytes-live=0-*" case is not clear to me.
Your ID says “All bytes currently in the
representation and those appended to the end of the
representation after the request is processed”. I get
the impression, that the server is deleting all data
before a certain timepoint (aka, behavior of a
slighting window timeshift).  So, the client seems to
request all data from the beginning of the timeshift
buffer. Why does the server need to change the byte
offset from 0 to 123456?

I
can understand, that the server must signal “growing
resource” in the response.

[cp] I was trying to illustrate a case where the server had trimmed
off bytes 0-123456 (the TSB model). So in this case, it's signalling
to the client "you're getting bytes starting at 123456 (not 0)".
e.g. If a client requests "Range: bytes-live=0-*" on an in-progress
recording, one might expect:

Content-Range: bytes-live 0-*/*

[cp] Basically saying (as described in the ID) that all bytes
currently in the representation and those appended to the end of the
representation after the request is processed will be returned. But
on a TSB, one might expect:

Content-Range: bytes-live 123456-*/*

[cp] Basically saying that bytes starting at byte 123456 in
the representation and those appended to the end of the
representation after the request is processed will be returned.

[cp] While I'm thinking about TSB use cases in the back of my mind,
this is really not the primary use case I was considering for the ID
(but I would hope it can be covered).

In
any case, how does the client know “good” byte range
offsets (i.e. service access points) to tune into the
recording? Or is the assumption, that the client can
synchronize to the media stream from any byte range?

[cp] For byte-level access, random access
implementation is up to the client. For some containers this
is easier than others. e.g. For MP4, the random access
points can be established by reading the movie and fragment
header(s). For something like MP2, it's trickier of course.

[TL]
Well, in case of fMP4, the client needs to get the
Movie Header for initialization. Then, proper access
point are fragment boundaries. There are various ways
to signal time to byte-offsets.

[cp] Fragments can actually have multiple access points - implicit
(per sample) and explicit (random access points). But yeah, it seems
common for fragments to have one random access point (and often
correlate to a GOP) - and that there's a huge variety of ways to lay
out the samples.

In
case of TS, the client needs a PAT, PMT and PES starts
for tune-in. It is a bit more tricky, but also here
are solutions.

But
the email talks about “none-segmented” media. The
draft talks about “mimicking segmented media”. fMP4 is
actually the way to create ISO-BMFF segments. So, it
is for segmented media, but without a separate
manifest?

[cp] It's important to differentiate between *fragmented* and
*segmented* MP4/ISO BMFF representations. bytes-live is most
applicable to fragmented files - where you have one representation
being used for the entire streaming session - with this
representation being appended to periodically (usually one fragment
at a time).

[cp] I really need to revise my description in the draft to help
avoid confusion. What I was trying to describe was how a solution
using just byte-Range requests would always be slightly behind the
live point - as is the case with rendering "live" segmented streams.
While bytes-live could be used for fragmented (MP4/ISO BMFF) content
or segmented content, the primary use case is for non-segmented
representations.

[cp] One major feature this draft allows is for retrieval of
bytes just preceding the live point. So for example, a
client can do a Range head request like "Range: bytes=0-",
get a "Content-Range: bytes 0-1234567/*", then perform
something like a "Range: bytes-live=1200000-*", and prime
its framebuffer with 34567 bytes of data that precede the
live point - allowing for the client to find an access point
(e.g. mpeg2 start codes) and to allow live presentation to
display much sooner than it would from the live point
(without random access).

[TL]
So, how does the client know, that the proper fragment
boundary is at byte position 120000? Do you assume
that the client first fetches a time-to-byte offset
file, which tells the client that a access point (e.g.
a fragment boundary) is at byte pos 120000? If yes,
why does the client need the HEAD request, when it
already has the byte position?

[cp] How a client know the amount to pre-fetch before the live point
would depend upon the media format. For an MP4/ISO BMFF file, 120000
could represent the random access point most immediately preceding
the live point. It would be similar for an indexed MP2. And for
unindexed MP2 representations, it's not uncommon for a client to
prebuffer a fixed amount of content in the hopes of capturing a
keyframe (really a heuristic).

[cp] The HEAD request is necessary in this case to know where the
live point is at the time the request is made so the HTTP client
would know if it can jump into already-stored content or if it
should just acquire the live point.

[cp] The important point is that all common video formats need a
discontinuity-free number of bytes before the live point to provide
a quality user experience.

How
should the client know, which byte ranges are already
available on the server? When the client is playing back
from the recorded part and would like to skip 5min
forward, how does the client know, whether a normal range
request is needed or whether the client should as for the
live point? What type of HTTP Status code should be
provided, when the range request is not yet available of
the server?

[cp] We're not trying to come up with a
universal solution for performing time-based seek on all
media formats with this draft. So some of this is out of
scope. But let me see if I can fill in some of the blanks.

[TL]
Ok, not everything needs to be in-scope. But an
essential assumption should be, whether the client has
a time-to-byteoffset table or whether the client can
determine precisely the fragment boundary positions.

[cp] Optimally, time-to-byte indexes would be used. But even without
this, clients can often manage with heuristics. e.g. VLC can perform
a reasonable job of providing time-seek on unindexed MP2 files.

[cp] Some applications of media streaming have time-based
indexing facilities built-in. e.g. MP4 (ISO BMFF) containers
allow time and data to be associated using the various
internal, mandatory metadata "boxes". In other cases,
applications may provide a separate resource that contains
time-to-byte mappings (e.g. content index files). In either
case, there's a facility for mapping time offsets to byte
offsets - or sometimes the client incorporates heuristics to
perform time skips (e.g. VLC will do this on some file
formats).

[TL]
Yes. fMP4 supports this and MPEG DASH is leveraging
this. But the live-point is not described in the
fragments. The client determines the livepoint from
the manifest.

[cp] Correct. In fragmented content, the time-to-segment map tells
you which representation to fetch (via GET). While I'd say that
bytes-live can also improve segmented rendering (by reducing the
latency of rendering), the primary focus of our draft is for
non-segmented representations.

[cp] In all these cases, there's some mechanism that maps
time offsets to byte offsets.

[TL]
Yes

[cp] When it comes to the available byte range, a client can
know what data range is available by utilizing a HEAD
request with a "Range: bytes=0-". The "Content-Range"
response can contain something like "Content-Range: bytes
0-1234567/*" which tells the client both the current
randomly accessible content range (via the "0-1234567") and
that the content is of indeterminate length (via the "*").

[TL]
So, that is the existing Content-Range response, but
with an ‘*’ to indicate the unknown content-length,
correct?

[cp] Yeah, the "*" in place of the last-byte-pos indicates an
indetermine-length response body.

[cp] Putting this all together, a client would implement a
5-minute skip by:

(2) determining the byte offset for that given time
using the appropriate index/heuristic (e.g. "3456789"),

(3) if the time is outside the index, jump to the live
point and update the time to the last-index time or other
means (e.g. using "Range: bytes-live=340000-*" to
pre-buffer/pre-prime the frame/sample buffer),

(4) if the time is inside the index, either perform a
standard bytes Range request to retrieve an
implementation-specific quantum of time or data (e.g.
"Range: bytes=3456789-3556789") and render.

[TL]
In (2), How does the client determine the byte offset?
fMP4 requires precise byteoffset, In case of TS, the
client can sync to the stream by first searching for
0x47 sync bytes. In (3), how does the client determine
“outside of the index”? Seems that some sort of
manifest is implicitly needed, which allows the client
to understand the latest byte pos.

[cp] (2) is media-format-specific. For MP4/ISO BMFF, it would use
the built-in metadata, for MP2, it would either use an index file or
a heuristic.

[cp] For (3), if the current live point (in byte terms) is greater
than the last byte offset in the index, then the live point is
"outside the index". That is, the time the client is trying to
access isn't randomly accessible, and the client should just jump to
the live point.

[cp] Again, some of this is out of scope, but I hope that
clarifies a common use case.

[TL]
Would be good to clarify, what information the client
needs to get in order to do the operations. How the
client gets the info can be left out-of-scope.

[cp] ok - I hope I'm filling in more of the blanks...

[cp] Regarding the status code, RFC7233 (section 4.4)
indicates that code 416 (Range Not Satisfiable) must be
returned when "the current extent of the selected resource
or that the set of ranges requested has been rejected due to
invalid ranges or an excessive request of small or
overlapping ranges." This part of 4.4 applies to *all* Range
requests - regardless of the Range Unit.

[TL]
ok.

[cp] The bytes-live draft then goes on to say that "A
bytes-live-range-specifier is considered unsatisfiable if
the first-byte-pos is larger than the current length of the
representation". This could probably be elaborated on a bit.
But this is supposed to be the "hook" into the 4.4 language.

Can

[cp] I hope I succeeded (at least
partially). Apologies for the long response. I wanted to
make sure I was answering your questions.

[TL]
Gets a bit clearer, but I still don’t understand the
“mimic HLS or DASH”. DASH / HLS focuses on CDN
optimization by creating a sequence of individual
files. The client can work out the live-point URL from
the manifest. Each segment is a “good” access point
(in DASH always box boundaries and in HLS always TS
boundaries even with PAT / PMT). So, the key issue
here is to clarify, how the client gets the byte
offsets of the fragment boundaries for range requests.

[cp] If it's still a bit unclear how this is performed, I can go
into more detail. But like I say, I should really reword that
section of the draft since I think I've created some confusion. The
point I was trying to make was that *polling* a non-segmented
representation would - other than being inefficient - have the kind
of multi-second latency that segmented live streaming would have.

[cp] But the difficulty of expressing this (secondary) benefit in
the bytes-live is probably not worth the trouble. I'll see if I can
reword the draft to make it less confusing. I don't think this point
is necessary to "sell" the concept of bytes-live (or a
bytes-live-like feature).

[cp] BTW, if you're really interested in the details of mapping time
to offsets in a ISO BMFF container, have a look at
odid_mp4_parser.vala:get_random_access_points() and
get_random_access_point_for_time() at
https://github.com/cablelabs/rygel/tree/cablelabs/master/src/media-engines/odid/.
I can probable even get you instructions for printing RAPs for MP4
files using the test program.

hth - cp

BR,

Thorsten

From:

To: [email-masked];

[email-masked]

[email-masked];

[email-masked]; [email-masked];
[email-masked];

[email-masked]

Subject: Re: Issue with "bytes" Range Unit
and live streaming

fixed.

....Roy

+1000

[cp] Yeah, it's unfortunate that no
solutions have moved forward for this widely-desired
feature. I can only assume that people just started
defining proprietary solutions - which is unfortunate.
I'll try to be "persistent"... ;^J

[cp] As was mentioned, the issue with just arbitrarily
allowing an open-ended Content-Range response (omitting
last-byte-pos) is that there's no good way for a client to
indicate it can support reception of a Content-Range
without a last-byte-pos. So I would fully expect many
clients to fail in "unpredictable ways" (disconnecting,
crashing, etc).

[cp] I see that the indeterminate length proposal you
referenced in your first citation introduces a
"Accept-Indefinite-Ranges" header to prevent this issue.
But I think this brings with it some other questions. e.g.
Would this apply to any/all Range Units which may be
introduced in the future? How can a Client issue a request
that starts at the "live point"? It feels like it has one
hand tied behind its back.

[cp] If I could, I would prefer to go back in time and
advocate for an alternate ABNF for the bytes Range Unit.
Seeing as that's not an option, I think using this well-
and long-defined Range Unit extension mechanism seems like
a good path forward as it should not create
interoperability issues between clients and servers.

[cp] And I would hope adding a Range Unit would have a
low/lower bar for acceptance. e.g. If a Range Unit fills a
useful role, is well-defined, and isn't redundant, it
seems reasonable that it should be accepted as it
shouldn't impact existing HTTP/1.1 semantics. In fact, the
gzip case (referenced in your third citation) seems like a
perfect application of the Range Unit (better than
bytes-live). If there's interest, I'll write up an RFC to
demonstrate...

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Craig Pratt <[email-masked]>
Date: Tue, 19 Apr 2016 02:59:55 -0700

On 4/19/16 12:19 AM, Mark Nottingham
wrote:

Hi Craig,

Cheers,

[cp] Yeah, I can see where that could be a problem. I'll come up
with another name for the Range Unit. How about "b-live"?

Thx,

cp

Hello all,

Range: bytes=1234567-*

Regards,

craig
--
craig pratt
Caspia Consulting
[email-masked]
503.746.8008

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Thorsten Lohmar <[email-masked]>
Date: Tue, 19 Apr 2016 14:02:14 +0000

Hi Craig,

Thanks for sharing the github link. That certainly clarifies the use-case even further.

Maybe we should focus the discussion on the fMP4 format for some time, since tune-in into fMP4 requires random access to fragment boundaries. Compared to ts or mp3, fMP4 does not support synchronization to the stream from any received byte. The client must start processing an fMP4 stream from fragment boundaries (or box boundaries). Tuning to selected samples inside of the fMP4 file happens at a later stage.

So, when I understand your proposal around the byte-live request correctly, then the client is NOT asking for the precise byte range, but the next possible random access points in close proximity of the requested range. So, the client sends a request containing "Range: bytes-live=0-*" and gets a response with "Content-Range: bytes-live 123456-*": So, the server is providing the client with a HTTP resource, starting in case of fMP4 with the fragment boundary (which is at byte-offset 123456 of the resource).

Do I understand it correctly then: When the client wants to tune-in e.g. 5min in the TSB, then the client measures the bitrate of the media stream and calculates a rough byte offset (i.e. 5min x estimated bitrate, lets say byte pos 654321) [all out of scope of the ID] and creates a bytes-live range request of form "Range: bytes-live=654321-*". The server looksup a good fragment boundary (lets say 654420)  and responds with "Content-Range: bytes-live: 654420-*". Do I understand the proposal correctly?

If yes, should the solution be limited to live cases?  If I understand it correctly, then you are looking for a solution where the client indicates a rough byte range in the request and the server responds with another range, which is fulfilling some condition. In case of a live session with fMP4, the server looks for a random access point into the stream. The random access point must be a fragment boundary in case of fMP4 and can be PAT, PMT, PES in case of TS.

Nothing inline.

BR,
Thorsten

From: Craig Pratt [mailto:[email-masked]]
Sent: Tuesday, April 19, 2016 11:55 AM
To: Thorsten Lohmar; [email-masked]; [email-masked]
Cc: Göran Eriksson AP; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Darshak Thakore
Subject: Re: Issue with "bytes" Range Unit and live streaming

Hey Thorsten,

cp

BR,
Thorsten

From: Craig Pratt [mailto:[email-masked]]
Sent: Monday, April 18, 2016 10:29 PM
To: Thorsten Lohmar; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: Göran Eriksson AP; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Darshak Thakore; STARK, BARBARA H
Subject: Re: Issue with "bytes" Range Unit and live streaming

[cc-ing the co-authors]

Hi Thorsten,

Reply in-line.

cp

[TL] The "Range: bytes-live=0-*" case is not clear to me. Your ID says "All bytes currently in the representation and those appended to the end of the representation after the request is processed". I get the impression, that the server is deleting all data before a certain timepoint (aka, behavior of a slighting window timeshift).  So, the client seems to request all data from the beginning of the timeshift buffer. Why does the server need to change the byte offset from 0 to 123456?
I can understand, that the server must signal "growing resource" in the response.
[cp] I was trying to illustrate a case where the server had trimmed off bytes 0-123456 (the TSB model). So in this case, it's signalling to the client "you're getting bytes starting at 123456 (not 0)". e.g. If a client requests "Range: bytes-live=0-*" on an in-progress recording, one might expect:

[cp] Basically saying (as described in the ID) that all bytes currently in the representation and those appended to the end of the representation after the request is processed will be returned. But on a TSB, one might expect:

[cp] Basically saying that bytes starting at byte 123456 in the representation and those appended to the end of the representation after the request is processed will be returned.

[cp] While I'm thinking about TSB use cases in the back of my mind, this is really not the primary use case I was considering for the ID (but I would hope it can be covered).

In any case, how does the client know "good" byte range offsets (i.e. service access points) to tune into the recording? Or is the assumption, that the client can synchronize to the media stream from any byte range?
[cp] For byte-level access, random access implementation is up to the client. For some containers this is easier than others. e.g. For MP4, the random access points can be established by reading the movie and fragment header(s). For something like MP2, it's trickier of course.
[TL] Well, in case of fMP4, the client needs to get the Movie Header for initialization. Then, proper access point are fragment boundaries. There are various ways to signal time to byte-offsets.
[cp] Fragments can actually have multiple access points - implicit (per sample) and explicit (random access points). But yeah, it seems common for fragments to have one random access point (and often correlate to a GOP) - and that there's a huge variety of ways to lay out the samples.

In case of TS, the client needs a PAT, PMT and PES starts for tune-in. It is a bit more tricky, but also here are solutions.
But the email talks about "none-segmented" media. The draft talks about "mimicking segmented media". fMP4 is actually the way to create ISO-BMFF segments. So, it is for segmented media, but without a separate manifest?
[cp] It's important to differentiate between *fragmented* and *segmented* MP4/ISO BMFF representations. bytes-live is most applicable to fragmented files - where you have one representation being used for the entire streaming session - with this representation being appended to periodically (usually one fragment at a time).

[cp] I really need to revise my description in the draft to help avoid confusion. What I was trying to describe was how a solution using just byte-Range requests would always be slightly behind the live point - as is the case with rendering "live" segmented streams. While bytes-live could be used for fragmented (MP4/ISO BMFF) content or segmented content, the primary use case is for non-segmented representations.

[cp] One major feature this draft allows is for retrieval of bytes just preceding the live point. So for example, a client can do a Range head request like "Range: bytes=0-", get a "Content-Range: bytes 0-1234567/*", then perform something like a "Range: bytes-live=1200000-*", and prime its framebuffer with 34567 bytes of data that precede the live point - allowing for the client to find an access point (e.g. mpeg2 start codes) and to allow live presentation to display much sooner than it would from the live point (without random access).
[TL] So, how does the client know, that the proper fragment boundary is at byte position 120000? Do you assume that the client first fetches a time-to-byte offset file, which tells the client that a access point (e.g. a fragment boundary) is at byte pos 120000? If yes, why does the client need the HEAD request, when it already has the byte position?
[cp] How a client know the amount to pre-fetch before the live point would depend upon the media format. For an MP4/ISO BMFF file, 120000 could represent the random access point most immediately preceding the live point. It would be similar for an indexed MP2. And for unindexed MP2 representations, it's not uncommon for a client to prebuffer a fixed amount of content in the hopes of capturing a keyframe (really a heuristic).

[cp] The HEAD request is necessary in this case to know where the live point is at the time the request is made so the HTTP client would know if it can jump into already-stored content or if it should just acquire the live point.

[cp] The important point is that all common video formats need a discontinuity-free number of bytes before the live point to provide a quality user experience.

How should the client know, which byte ranges are already available on the server? When the client is playing back from the recorded part and would like to skip 5min forward, how does the client know, whether a normal range request is needed or whether the client should as for the live point? What type of HTTP Status code should be provided, when the range request is not yet available of the server?
[cp] We're not trying to come up with a universal solution for performing time-based seek on all media formats with this draft. So some of this is out of scope. But let me see if I can fill in some of the blanks.
[TL] Ok, not everything needs to be in-scope. But an essential assumption should be, whether the client has a time-to-byteoffset table or whether the client can determine precisely the fragment boundary positions.
[cp] Optimally, time-to-byte indexes would be used. But even without this, clients can often manage with heuristics. e.g. VLC can perform a reasonable job of providing time-seek on unindexed MP2 files.

[TL] Yes. fMP4 supports this and MPEG DASH is leveraging this. But the live-point is not described in the fragments. The client determines the livepoint from the manifest.
[cp] Correct. In fragmented content, the time-to-segment map tells you which representation to fetch (via GET). While I'd say that bytes-live can also improve segmented rendering (by reducing the latency of rendering), the primary focus of our draft is for non-segmented representations.

[cp] When it comes to the available byte range, a client can know what data range is available by utilizing a HEAD request with a "Range: bytes=0-". The "Content-Range" response can contain something like "Content-Range: bytes 0-1234567/*" which tells the client both the current randomly accessible content range (via the "0-1234567") and that the content is of indeterminate length (via the "*").
[TL] So, that is the existing Content-Range response, but with an '*' to indicate the unknown content-length, correct?
[cp] Yeah, the "*" in place of the last-byte-pos indicates an indetermine-length response body.

[cp] (2) is media-format-specific. For MP4/ISO BMFF, it would use the built-in metadata, for MP2, it would either use an index file or a heuristic.

[cp] For (3), if the current live point (in byte terms) is greater than the last byte offset in the index, then the live point is "outside the index". That is, the time the client is trying to access isn't randomly accessible, and the client should just jump to the live point.

Can you please clarify the questions?
[cp] I hope I succeeded (at least partially). Apologies for the long response. I wanted to make sure I was answering your questions.
[TL] Gets a bit clearer, but I still don't understand the "mimic HLS or DASH". DASH / HLS focuses on CDN optimization by creating a sequence of individual files. The client can work out the live-point URL from the manifest. Each segment is a "good" access point (in DASH always box boundaries and in HLS always TS boundaries even with PAT / PMT). So, the key issue here is to clarify, how the client gets the byte offsets of the fragment boundaries for range requests.
[cp] If it's still a bit unclear how this is performed, I can go into more detail. But like I say, I should really reword that section of the draft since I think I've created some confusion. The point I was trying to make was that *polling* a non-segmented representation would - other than being inefficient - have the kind of multi-second latency that segmented live streaming would have.

[cp] But the difficulty of expressing this (secondary) benefit in the bytes-live is probably not worth the trouble. I'll see if I can reword the draft to make it less confusing. I don't think this point is necessary to "sell" the concept of bytes-live (or a bytes-live-like feature).

[cp] BTW, if you're really interested in the details of mapping time to offsets in a ISO BMFF container, have a look at odid_mp4_parser.vala:get_random_access_points() and get_random_access_point_for_time() at https://github.com/cablelabs/rygel/tree/cablelabs/master/src/media-engines/odid/. I can probable even get you instructions for printing RAPs for MP4 files using the test program.

hth - cp

BR,
Thorsten

fixed.

....Roy

+1000

--

craig pratt

Caspia Consulting

503.746.8008

--

craig pratt

Caspia Consulting

503.746.8008

--

craig pratt

Caspia Consulting

503.746.8008

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 19 Apr 2016 14:16:45 -0700

Yes, in general, though I was considering the specific use cases for which a live stream is a viable response to a range request. For example, I would expect a live stream to be marked as non-cacheable, or at the very least correctly respond to if-match and if-range preconditions. That would prevent any misunderstanding by unaware caches. A live stream request would omit those preconditions.

Likewise, it sounds to me like this bit of protocol would only be used when all components on the chain are aware of the nature of the content and desiring such streaming, since there are just too many non-HTTP factors needed to make it work. IOW, I would not expect the client to allow the request to be proxied unless it already knew the proxy could cooperate.

Nevertheless, there is no hurry in adopting one proposal or another. It should be tested in practice first. 

....Roy

> On Apr 19, 2016, at 12:19 AM, Mark Nottingham <[email-masked]> wrote:
>

---

From: Craig Pratt <[email-masked]>
Date: Tue, 19 Apr 2016 14:54:00 -0700

Reply in-line.

cp

On 4/19/16 7:02 AM, Thorsten Lohmar wrote:
> Maybe we should focus the discussion on the fMP4 format for some time, 
> since tune-in into fMP4 requires random access to fragment boundaries. 
> Compared to ts or mp3, fMP4 does not support synchronization to the 
> stream from any received byte. The client must start processing an 
> fMP4 stream from fragment boundaries (or box boundaries). Tuning to 
> selected samples inside of the fMP4 file happens at a later stage.
>
> So, when I understand your proposal around the byte-live request 
> correctly, then the client is NOT asking for the precise byte range, 
> but the next possible random access points in close proximity of the 
> requested range. So, the client sends a request containing "Range: 
> bytes-live=0-*" and gets a response with "Content-Range: bytes-live 
> 123456-*": So, the server is providing the client with a HTTP 
> resource, starting in case of fMP4 with the fragment boundary (which 
> is at byte-offset 123456 of the resource).
>
> Do I understand it correctly then: When the client wants to tune-in 
> e.g. 5min in the TSB, then the client measures the bitrate of the 
> media stream and calculates a rough byte offset (i.e. 5min x estimated 
> bitrate, lets say byte pos 654321) [all out of scope of the ID] and 
> creates a bytes-live range request of form “Range: 
> bytes-live=654321-*”. The server looksup a good fragment boundary 
> (lets say 654420)  and responds with “Content-Range: bytes-live: 
> 654420-*”. Do I understand the proposal correctly?
>
TSBs and MP4s are a nasty combination. ISO BMFF containers accommodate 
amended content (via fragments). But it doesn't have facilities for 
front-end truncation. It's possible, just not easy. So this will go down 
the rabbit hole quickly.

One would need to define the operation of the server for providing a 
time-shifted MP4 representation. Basically the idea is that the server 
would maintain a list of valid fragments and would have to maintain a 
valid movie box at the front of the representation as fragments were 
added and removed (if a client's expected to consume a spec-compliant 
ISO BMFF container).

So I'd say you're mostly right, but let me paraphrase: When an MP4 
client wants to come in at a precise time on a MP4/ISO BMFF (e.g. 5 
minutes), it typically would:

1) get the movie box from the front of the representation (a couple 
byte-Range requests);
2) access the movie fragment header boxes until the requested time 
offset is determined (multiple byte-Range requests);
3) If the time offset is found in the currently-available fragments, 
perform a Range request to get the fragment containing the target time 
and start rendering it (one or two "bytes" Range request), and start 
fetching the next Fragment if/when desired. e.g. If the nearest random 
access point for 5 minutes is at the start of a fragment with offset 
4444444 with length 30001:

     Range: bytes=4444444-4474444

4) If the time offset is not found to be after the currently-available 
fragments, jump to the live point by grabbing the last fragment and any 
fragments that are added on (one "bytes-live" range request). e.g. if 
the last fragment's offset is 5555555 with length 40001:

     Range: bytes-live=5555555-*

The client would quickly render the last fragment (to prime the frame 
buffer), put the framebuffer on-screen, and then block on the socket (or 
wait for async notification) and render the new fragment(s) as they come in.

* Note that one might use MP4 timelines in here to communicate the fact 
that sample time 0 is not media time 0. But I'd rather not get too far 
into that.

** As with live content, TSBs are non-cachable. (but in-progress 
recordings *are* cache-able)
>
> If yes, should the solution be limited to live cases?  If I understand 
> it correctly, then you are looking for a solution where the client 
> indicates a rough byte range in the request and the server responds 
> with another range, which is fulfilling some condition. In case of a 
> live session with fMP4, the server looks for a random access point 
> into the stream. The random access point must be a fragment boundary 
> in case of fMP4 and can be PAT, PMT, PES in case of TS.
>
Technically, regular bytes-Range request could have this "fencing" 
behavior (a Client should be driven off the Content-Range response 
header). But I think this is somewhat disingenuous and might be 
inconsistent with RFC7233 (I'd have to look closely). And I think this 
is another example of where a different Range Unit would make sense.

e.g. DLNA specifications defined a TimeSeekRange.dlna.org header that 
carries this contract (of providing the most-immediately-preceding 
"decoder-friendly position") in the returned media. Now, this was 
defined to work with HTTP/1.0, so defining a new Range Unit wasn't an 
option. But if so, TimeSeekRange.dlna.org could be easily replaced with 
a "npt" Range Unit (normal play time). And what you're talking about 
would be similar but with byte offsets. Both carry a similar assumption: 
The server has some knowledge of the content structure.

Aside: Both use cases could be covered with something like a "mso" 
(media-sensitive offset) Range Unit which could take a time offset or a 
byte offset (and incorporate live semantics).

The bytes-live draft (or whatever we end up calling it) is intended to 
assume all knowledge is client-side - just as the case with the bytes 
Range Unit. I think you're assuming some of this server-side structure 
knowledge that I was not intending. But "mso" would make another great 
Range Unit.

There are many other potential applications of something like bytes-live 
I believe - which is why we're bringing this to the IETF. But really 
it's all about getting to the "live point" in a couple different ways. 
But it assumes the *client* knows where the random access points are 
located (and tailors the byte offsets accordingly).
> Nothing inline.
Good - that was getting crazy... ;^J

> BR,
>
> Thorsten
>
> *From:*Craig Pratt [mailto:[email-masked]]
> *Sent:* Tuesday, April 19, 2016 11:55 AM
> *To:* Thorsten Lohmar; [email-masked]; [email-masked]
> *Cc:* Göran Eriksson AP; [email-masked]; [email-masked]; 
> [email-masked]; [email-masked]; [email-masked]; 
> [email-masked]; Darshak Thakore
> *Subject:* Re: Issue with "bytes" Range Unit and live streaming
>
> Hey Thorsten,
>
>     *From:*Craig Pratt [mailto:[email-masked]]
>     *Sent:* Monday, April 18, 2016 10:29 PM
>     *To:* Thorsten Lohmar; [email-masked]
>     <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>
>     *Cc:* Göran Eriksson AP; [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>; Darshak
>     Thakore; STARK, BARBARA H
>     *Subject:* Re: Issue with "bytes" Range Unit and live streaming
>
>     [cc-ing the co-authors]
>
>     Hi Thorsten,
>
>         My colleague Göran asked me some question around the problem
>         and I would like to raise these questions directly to you. Of
>         course, there are some alternative solutions available, where
>         the client can work out the different things from a manifest.
>         But you seem to look for a simple solution, which works with
>         non-segmented media on a single HTTP session.
>
>         When I understood it correctly, an HTTP server is making a
>         live stream available using HTTP. A normal live stream can be
>         data “from the live point” either with or without HTTP chunked
>         delivery. The server cannot give a Content-Length, since this
>         is an ongoing live stream of unknown size.
>
>     [cp] all correct.
>
>
>     Your use-case seem to be about recording of content. Client should
>     access content from the recorded part, but should be able to jump
>     to the live-point. I assume that you are not looking into sliding
>     window recordings (i.e. timeshift). I assume that the a single
>     program is continuous recording and the HTTP object is growing
>     until the end of the live session, correct?
>
>     consider adding clarifications for the time-shift cases. This
>     should just be a matter of a Client requesting one thing and
>     getting another. e.g. "Range: bytes-live=0-*" results in
>     "Content-Range: bytes-live 123456-*". In either case, you're
>     correct: the end of the representation is moving forward in time
>     until the end of the live session.
>
>
>     */[TL] The "Range: bytes-live=0-*" case is not clear to me. Your
>     ID says “All bytes currently in the representation and those
>     appended to the end of the representation after the request is
>     processed”. I get the impression, that the server is deleting all
>     data before a certain timepoint (aka, behavior of a slighting
>     window timeshift).  So, the client seems to request all data from
>     the beginning of the timeshift buffer. Why does the server need to
>     change the byte offset from 0 to 123456? /*
>
>     *I can understand, that the server must signal “growing resource”
>     in the response.*
>
> to the client "you're getting bytes starting at 123456 (not 0)". e.g. 
> If a client requests "Range: bytes-live=0-*" on an in-progress 
> representation after the request is processed will be returned. But on 
> a TSB, one might expect:
>
> [cp] Basically saying that bytes starting at byte 123456 in the 
> representation and those appended to the end of the representation 
> after the request is processed will be returned.
>
> In any case, how does the client know “good” byte range offsets (i.e. 
> service access points) to tune into the recording? Or is the 
> assumption, that the client can synchronize to the media stream from 
> any byte range?
>
> [cp] For byte-level access, random access implementation is up to the 
> client. For some containers this is easier than others. e.g. For MP4, 
> the random access points can be established by reading the movie and 
> fragment header(s). For something like MP2, it's trickier of course.
>
> */[TL] Well, in case of fMP4, the client needs to get the Movie Header 
> for initialization. Then, proper access point are fragment boundaries. 
> There are various ways to signal time to byte-offsets. /*
>
> */In case of TS, the client needs a PAT, PMT and PES starts for 
> tune-in. It is a bit more tricky, but also here are solutions. /*
>
> */But the email talks about “none-segmented” media. The draft talks 
> about “mimicking segmented media”. fMP4 is actually the way to create 
> ISO-BMFF segments. So, it is for segmented media, but without a 
> separate manifest?/*
>
> being used for the entire streaming session - with this representation 
> being appended to periodically (usually one fragment at a time).
>
> [cp] I really need to revise my description in the draft to help avoid 
> confusion. What I was trying to describe was how a solution using just 
> byte-Range requests would always be slightly behind the live point - 
> as is the case with rendering "live" segmented streams. While 
> bytes-live could be used for fragmented (MP4/ISO BMFF) content or 
> segmented content, the primary use case is for non-segmented 
> representations.
>
>
>
> just preceding the live point. So for example, a client can do a Range 
> head request like "Range: bytes=0-", get a "Content-Range: bytes 
> 0-1234567/*", then perform something like a "Range: 
> data that precede the live point - allowing for the client to find an 
> access point (e.g. mpeg2 start codes) and to allow live presentation 
> to display much sooner than it would from the live point (without 
> random access).
>
> */[TL] So, how does the client know, that the proper fragment boundary 
> is at byte position 120000? Do you assume that the client first 
> fetches a time-to-byte offset file, which tells the client that a 
> access point (e.g. a fragment boundary) is at byte pos 120000? If yes, 
> why does the client need the HEAD request, when it already has the 
> byte position?/*
>
> could represent the random access point most immediately preceding the 
> live point. It would be similar for an indexed MP2. And for unindexed 
> MP2 representations, it's not uncommon for a client to prebuffer a 
> fixed amount of content in the hopes of capturing a keyframe (really a 
> heuristic).
>
> [cp] The HEAD request is necessary in this case to know where the live 
> point is at the time the request is made so the HTTP client would know 
> if it can jump into already-stored content or if it should just 
> acquire the live point.
>
> discontinuity-free number of bytes before the live point to provide a 
> quality user experience.
>
>
> How should the client know, which byte ranges are already available on 
> the server? When the client is playing back from the recorded part and 
> would like to skip 5min forward, how does the client know, whether a 
> normal range request is needed or whether the client should as for the 
> live point? What type of HTTP Status code should be provided, when the 
> some of this is out of scope. But let me see if I can fill in some of 
> the blanks.
>
> */[TL] Ok, not everything needs to be in-scope. But an essential 
> assumption should be, whether the client has a time-to-byteoffset 
> table or whether the client can determine precisely the fragment 
> boundary positions. /*
>
> this, clients can often manage with heuristics. e.g. VLC can perform a 
> reasonable job of providing time-seek on unindexed MP2 files.
>
>
>
> "boxes". In other cases, applications may provide a separate resource 
> that contains time-to-byte mappings (e.g. content index files). In 
> either case, there's a facility for mapping time offsets to byte 
> offsets - or sometimes the client incorporates heuristics to perform 
> time skips (e.g. VLC will do this on some file formats).
>
>
> */[TL] Yes. fMP4 supports this and MPEG DASH is leveraging this. But 
> the live-point is not described in the fragments. The client 
> determines the livepoint from the manifest. /*
>
> [cp] Correct. In fragmented content, the time-to-segment map tells you 
> which representation to fetch (via GET). While I'd say that bytes-live 
> can also improve segmented rendering (by reducing the latency of 
> rendering), the primary focus of our draft is for non-segmented 
> representations.
>
>
> */[TL] Yes/*
> [cp] When it comes to the available byte range, a client can know what 
> data range is available by utilizing a HEAD request with a "Range: 
> bytes=0-". The "Content-Range" response can contain something like 
> "Content-Range: bytes 0-1234567/*" which tells the client both the 
> current randomly accessible content range (via the "0-1234567") and 
> */[TL] So, that is the existing Content-Range response, but with an 
> ‘*’ to indicate the unknown content-length, correct? /*
>
> "Range: bytes-live=340000-*" to pre-buffer/pre-prime the frame/sample 
> buffer),
> bytes Range request to retrieve an implementation-specific quantum of 
> time or data (e.g. "Range: bytes=3456789-3556789") and render.
>
> */[TL] In (2), How does the client determine the byte offset? fMP4 
> requires precise byteoffset, In case of TS, the client can sync to the 
> stream by first searching for 0x47 sync bytes. In (3), how does the 
> client determine “outside of the index”? Seems that some sort of 
> manifest is implicitly needed, which allows the client to understand 
> the latest byte pos. /*
>
> [cp] (2) is media-format-specific. For MP4/ISO BMFF, it would use the 
> built-in metadata, for MP2, it would either use an index file or a 
> heuristic.
>
> "outside the index". That is, the time the client is trying to access 
> isn't randomly accessible, and the client should just jump to the live 
> point.
>
> *//*
>
>
>
> */[TL] Would be good to clarify, what information the client needs to 
> get in order to do the operations. How the client gets the info can be 
> left out-of-scope./*
>
> small or overlapping ranges." This part of 4.4 applies to *all* Range 
> */[TL] ok. /*
> representation". This could probably be elaborated on a bit. But this 
> is supposed to be the "hook" into the 4.4 language.
>
>
> */[TL] Gets a bit clearer, but I still don’t understand the “mimic HLS 
> or DASH”. DASH / HLS focuses on CDN optimization by creating a 
> sequence of individual files. The client can work out the live-point 
> URL from the manifest. Each segment is a “good” access point (in DASH 
> always box boundaries and in HLS always TS boundaries even with PAT / 
> PMT). So, the key issue here is to clarify, how the client gets the 
> byte offsets of the fragment boundaries for range requests./*
>
> [cp] If it's still a bit unclear how this is performed, I can go into 
> more detail. But like I say, I should really reword that section of 
> the draft since I think I've created some confusion. The point I was 
> trying to make was that *polling* a non-segmented representation would 
> - other than being inefficient - have the kind of multi-second latency 
> that segmented live streaming would have.
>
> [cp] But the difficulty of expressing this (secondary) benefit in the 
> bytes-live is probably not worth the trouble. I'll see if I can reword 
> the draft to make it less confusing. I don't think this point is 
> necessary to "sell" the concept of bytes-live (or a bytes-live-like 
> feature).
>
> *From:*Craig Pratt [mailto:[email-masked]]
> *Sent:* Monday, April 18, 2016 11:04 AM
> *To:* [email-masked] <mailto:[email-masked]>; [email-masked] 
> <mailto:[email-masked]>
> *Cc:* Göran Eriksson AP; [email-masked] <mailto:[email-masked]>; 
> [email-masked] <mailto:[email-masked]>; 
> [email-masked] <mailto:[email-masked]>; [email-masked] 
> <mailto:[email-masked]>; [email-masked] 
> <mailto:[email-masked]>; [email-masked] 
> <mailto:[email-masked]>
> *Subject:* Re: Issue with "bytes" Range Unit and live streaming
>
> On 4/18/16 12:34 AM, [email-masked] <mailto:[email-masked]> wrote:
>
>     On Friday,15 April 2016 22:43,[email-masked] <mailto:[email-masked]>  wrote:
>
>     [1]https://lists.w3.org/Archives/Public/ietf-http-wg/2015AprJun/0122.html
>
>     [2]https://tools.ietf.org/html/draft-combs-http-indeterminate-range-01
>
>     [3]https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/1327.html
>
> there's no good way for a client to indicate it can support reception 
> of a Content-Range without a last-byte-pos. So I would fully expect 
> many clients to fail in "unpredictable ways" (disconnecting, crashing, 
> etc).
>
> your first citation introduces a "Accept-Indefinite-Ranges" header to 
> prevent this issue. But I think this brings with it some other 
> introduced in the future? How can a Client issue a request that starts 
> at the "live point"? It feels like it has one hand tied behind its back.
>
> [cp] If I could, I would prefer to go back in time and advocate for an 
> alternate ABNF for the bytes Range Unit. Seeing as that's not an 
> option, I think using this well- and long-defined Range Unit extension 
> mechanism seems like a good path forward as it should not create 
> well-defined, and isn't redundant, it seems reasonable that it should 
> be accepted as it shouldn't impact existing HTTP/1.1 semantics. In 
> fact, the gzip case (referenced in your third citation) seems like a 
> perfect application of the Range Unit (better than bytes-live). If 
> [email-masked] <mailto:[email-masked]>
>
> 503.746.8008
>
> 	
> 	
>
> -- 
>
>
> craig pratt
>
> Caspia Consulting
>
> [email-masked] <mailto:[email-masked]>
>
> 503.746.8008
>
> 	
> 	
>
> -- 
>
> craig pratt
>
> Caspia Consulting
>
> [email-masked] <mailto:[email-masked]>
>
> 503.746.8008
>
> 	
> 	
>

-- 

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 20 Apr 2016 17:50:05 +1000

> On 19 Apr 2016, at 7:59 PM, Craig Pratt <[email-masked]> wrote:
> 
> How about "b-live"?

Better. If we choose to mint a new range-unit, it might be an opportunity to make other changes too, in which case it might be more generic; that would imply a different, more general name. But we're not there yet.

Cheers,

---

From: Martin Thomson <[email-masked]>
Date: Wed, 20 Apr 2016 23:11:30 +1000

> Better. If we choose to mint a new range-unit, it might be an opportunity to make other changes too, in which case it might be more generic; that would imply a different, more general name. But we'…

Maybe I'm not 100% across the constraints that we're operating with
here, but is there some way to achieve the basic goal here without
committing to changes to the protocol?

I ask because we have exactly one use of ranges in HTTP and that makes
me highly skeptical that this particular extension point is actually
extensible.  As noted up-thread, many implementations make
assumptions.  That makes the chances that we can successfully make
changes to the protocol quite slim, or at the very least they could be
expensive.

If we were to, say, accept that Range is how it is, and concentrated
on using the protocol as it is to solve the problem, could that work,
or is there some aspect to the problem that I'm missing?

For example...

If I were to request Range: 123456-* and that resulted in me receiving
an entirely different response (maybe via redirect) that included the
requested content with a "This-Is-Part-Of: <that other URL>;
from=123456", would that work?  Or, and I hesitate to suggest this,
but it does avoid the extra request... Vary: Range.

---

From: Thorsten Lohmar <[email-masked]>
Date: Wed, 20 Apr 2016 15:14:36 +0000

Hi Craig,

See inline.

It might be worthwhile to bring-up the issue in MPEG. E.g. as ISO-BMFF extension or DASH extension.

BR,
/Thorsten

> -----Original Message-----
> Sent: Tuesday, April 19, 2016 11:54 PM
> Cc: Göran Eriksson AP; [email-masked]; [email-masked]; ietf-
> [email-masked]; [email-masked]; [email-masked];
> > creates a bytes-live range request of form "Range:
> > bytes-live=654321-*". The server looksup a good fragment boundary
> > (lets say 654420)  and responds with "Content-Range: bytes-live:
> > 654420-*". Do I understand the proposal correctly?
> >
> amended content (via fragments). But it doesn't have facilities for front-end
> truncation. It's possible, just not easy. So this will go down the rabbit hole
> quickly.

[TL] Yes, it is possible to append to the end. But you cannot simply delete from the front when you realize a TSB. The question is, whether this issue should be solved on HTTP layer, since likely you also want to give an indication of the TSB to the Users (GUI representation of the timeshiftbuffer depth).

> 
> One would need to define the operation of the server for providing a time-
> shifted MP4 representation. Basically the idea is that the server would
> maintain a list of valid fragments and would have to maintain a valid movie
> box at the front of the representation as fragments were added and
> removed (if a client's expected to consume a spec-compliant ISO BMFF
> container).

[TL] Yes.

> 
> So I'd say you're mostly right, but let me paraphrase: When an MP4 client
> wants to come in at a precise time on a MP4/ISO BMFF (e.g. 5 minutes), it
> typically would:
> 
> 1) get the movie box from the front of the representation (a couple byte-
> Range requests);
> 2) access the movie fragment header boxes until the requested time offset is
> determined (multiple byte-Range requests);

[TL] Yes, but the Movie Box does not contain such information for a Live offering. So, something is missing here.

> 3) If the time offset is found in the currently-available fragments, perform a
> Range request to get the fragment containing the target time and start
> rendering it (one or two "bytes" Range request), and start fetching the next
> Fragment if/when desired. e.g. If the nearest random access point for 5
> minutes is at the start of a fragment with offset

[TL] Well, almost. You must always start at the fragment boundary. When the fragment contains e.g. 4sec of media data (e.g. with 1Sec Gops), the client must start fetching from the beginning of the fragment and then skip the media data before the desire playtime start. The client can also do an open range request.

> fragments that are added on (one "bytes-live" range request). e.g. if the last
> fragment's offset is 5555555 with length 40001:
> 

[TL] Ok, understood.
> that sample time 0 is not media time 0. But I'd rather not get too far into that.
> 
> ** As with live content, TSBs are non-cachable. (but in-progress recordings
> *are* cache-able)

[TL] Since we talk about a single HTTP transaction with progressive media data, the session is anyhow not cachable.

> behavior (a Client should be driven off the Content-Range response header).
> But I think this is somewhat disingenuous and might be inconsistent with
> RFC7233 (I'd have to look closely). And I think this is another example of
> where a different Range Unit would make sense.
> 
> carries this contract (of providing the most-immediately-preceding "decoder-
> friendly position") in the returned media. Now, this was defined to work with
> HTTP/1.0, so defining a new Range Unit wasn't an option. But if so,
> TimeSeekRange.dlna.org could be easily replaced with a "npt" Range Unit
> (normal play time). And what you're talking about would be similar but with
> byte offsets. Both carry a similar assumption:

[TL] Yes, started also thinking about time range requests, but I couldn't remember, whether that was define in DLNA, OIPF of DVB. Are the time range headers widely used today? Can such a solution make sense for H1.1 or H2?
> (media-sensitive offset) Range Unit which could take a time offset or a byte
> offset (and incorporate live semantics).
> 
> The bytes-live draft (or whatever we end up calling it) is intended to assume
> all knowledge is client-side - just as the case with the bytes Range Unit. I
> think you're assuming some of this server-side structure knowledge that I
> was not intending. But "mso" would make another great Range Unit.

[TL] I still think, that the client should have precise information of the TSB (i.e. precise range and depth) in order to properly render the TSB representation on the GUI (Media Players often render the TSB in a progress bar together with the progress of the Live Session).

> 
> There are many other potential applications of something like bytes-live I
> believe - which is why we're bringing this to the IETF. But really it's all about
> getting to the "live point" in a couple different ways.

[TL] Maybe, it is better to discuss this issue in MPEG, either as ISO-BMFF extensions or DASH extensions.  

> >         data "from the live point" either with or without HTTP chunked
> >     ID says "All bytes currently in the representation and those
> >     processed". I get the impression, that the server is deleting all
> >     *I can understand, that the server must signal "growing resource"
> >     in the response.*
> >
> > In any case, how does the client know "good" byte range offsets (i.e.
> > */But the email talks about "none-segmented" media. The draft talks
> > about "mimicking segmented media". fMP4 is actually the way to create
> > '*' to indicate the unknown content-length, correct? /*
> >
> > client determine "outside of the index"? Seems that some sort of
> > */[TL] Gets a bit clearer, but I still don't understand the "mimic HLS
> > or DASH". DASH / HLS focuses on CDN optimization by creating a
> > URL from the manifest. Each segment is a "good" access point (in DASH
> [email-masked]
> >     A very similar idea was proposed before [1] as an I-D [2] by Rodger
> Coombs. We've also brought this up informally with other members of the
> WG.
> >
> >
> >
> >     Alas, in our experience range requests don't seem to be a high priority :(
> For example, the problem of combining gzip with range requests is still
> unsolved [3].
> >
> >
> >
> >
> > This email message is intended only for the use of the named recipient.
> Information contained in this email message and its attachments may be
> privileged, confidential and protected from disclosure. If you are not the
> intended recipient, please do not read, copy, use or disclose this
> communication to others. Also please notify the sender by replying to this
> message and then delete it from your system.
> >
> > --
> >
> >
> >
> > craig pratt
> >
> > Caspia Consulting
> >

---

From: Craig Pratt <[email-masked]>
Date: Wed, 20 Apr 2016 11:21:16 -0700

Hey Thorsten,

I'm not clear about what you think is missing today from MP4/ISO-BMFF.

HTTP/1.1 MP4-aware client has everything it needs today to resolve time 
offsets to byte offsets and find the nearest random access point(s) - 
whether unfragmented, static-fragmented, or live-fragmented - and I have 
code to demonstrate this. I'm sure this must be a misunderstanding, but 
I don't see what here requires a ISO-BMFF extension.

What *is* missing is the ability to get a continuous byte range on live 
content that starts at an arbitrary offset and the ability to directly 
jump to the live point.

More thoughts in-line.

cp

> [TL] Yes, it is possible to append to the end. But you cannot simply delete from the front when you realize a TSB. The question is, whether this issue should be solved on HTTP layer, since likely y…
[cp] Yeah - there always needs to be a mov box at the front. And while I 
believe Content-Range is sufficient to communicate front-end truncation, 
anything time-related is another matter. In MP4, this can be 
communicated in the timeline via an elst box (edit list). Anything else 
requires some kind of time-based Range Unit. And that's orthogonal to 
our draft - and live content in general.

> [TL] Yes, but the Movie Box does not contain such information for a Live offering. So, something is missing here.
[cp] Whether the MP4 is being actively appended to or not, walking the 
fragments (moofs) will provide a client the time offsets. I have code 
demonstrating this or you can do a protocol analysis of VLC to see it 
performing the bytes-Range requests. Are we getting tripped up on the 
definition of "live"? "Live" != "HLS Live Streaming", right?
> [TL] Well, almost. You must always start at the fragment boundary. When the fragment contains e.g. 4sec of media data (e.g. with 1Sec Gops), the client must start fetching from the beginning of the…
[cp] Agreed - which is why I said "the nearest random access point for 5 
minutes is at the start of a fragment with offset 4444444". The actual 
sample/frames corresponding with the 5-minute mark would be at some 
point after 4444444.
>> wait for async notification) and render the new fragment(s) as they come in.
[cp] It's important to remember that this is the feature we care about 
in context of this draft. If you're good with this, then we're on the 
same page.
> [TL] Since we talk about a single HTTP transaction with progressive media data, the session is anyhow not cachable.
[cp] Well, in-progress recordings should still be byte-wise cachable, 
correct? Byte 0 will always be byte 0, byte X will always be byte X. So 
a cache can be populated with the currently-accessible bytes and can 
satisfy Range requests. There might just be a cache miss if a client 
accesses bytes not-yet-cached. But a proxy should know that this is a 
possibility since the Content-Range responses have a "*" in place of the 
content length.
> [TL] Yes, started also thinking about time range requests, but I couldn't remember, whether that was define in DLNA, OIPF of DVB. Are the time range headers widely used today? Can such a solution m…
[cp] TimeSeekRange is widely used in DLNA. But they're more useful for 
less structured content such as MP2 Program/Transport streams. (for MP4 
representations, one never sees DLNA clients using TimeSeekRange - but 
servers are required to support TSR for all media formats in CVP-2).

> [TL] I still think, that the client should have precise information of the TSB (i.e. precise range and depth) in order to properly render the TSB representation on the GUI (Media Players often rend…
[cp] I do think it could be useful to define something like 
TimeSeekRange.dlna.org as a Range Unit. This would be very useful for 
random access on MPEG2-contained content. If you want to start a draft, 
I'm happy to co-author. ;^J
> [TL] Maybe, it is better to discuss this issue in MPEG, either as ISO-BMFF extensions or DASH extensions.
[cp] Perhaps. But the mechanism of how the transfer occurs - which is 
what this draft is related to - is definitely in the transport space. 
For some complex formats, it may be in the "necessary but not 
sufficient" category. But as I mentioned at the top, for 
(linear/progressive) MP4/ISO-BMFF, everything should be there.

-- 

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Craig Pratt <[email-masked]>
Date: Wed, 20 Apr 2016 11:23:37 -0700

On 4/20/16 12:50 AM, Mark Nottingham
wrote:

How about "b-live"?

Cheers,

[cp] I'll see what I can come up with and revise the draft (and try
to not think about that "yet" you have on there... ;^J)...

--

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Thorsten Lohmar <[email-masked]>
Date: Wed, 20 Apr 2016 19:31:14 +0000

Hi Craig,

> What *is* missing is the ability to get a continuous byte range on live content
> that starts at an arbitrary offset and the ability to directly jump to the live
> point.

Yes, and there are two solutions to the issue:
A. Enable it on HTTP Layer through the definition of a new range request method
B. Enable working with existing HTTP procedures, i.e. client can workout the precise byte offsets.

BR,
/Thorsten

> Sent: Wednesday, April 20, 2016 8:21 PM
> code to demonstrate this. I'm sure this must be a misunderstanding, but I
> don't see what here requires a ISO-BMFF extension.
> 
> What *is* missing is the ability to get a continuous byte range on live content
> that starts at an arbitrary offset and the ability to directly jump to the live
> point.
> 
> More thoughts in-line.
> 
> cp
> 
> >>> Maybe we should focus the discussion on the fMP4 format for some
> >>> time, since tune-in into fMP4 requires random access to fragment
> boundaries.
> >>> media stream and calculates a rough byte offset (i.e. 5min x
> >>> estimated bitrate, lets say byte pos 654321) [all out of scope of
> >>> the ID] and creates a bytes-live range request of form "Range:
> >> TSBs and MP4s are a nasty combination. ISO BMFF containers
> >> accommodate amended content (via fragments). But it doesn't have
> >> facilities for front-end truncation. It's possible, just not easy. So
> >> this will go down the rabbit hole quickly.
> > [TL] Yes, it is possible to append to the end. But you cannot simply delete
> from the front when you realize a TSB. The question is, whether this issue
> should be solved on HTTP layer, since likely you also want to give an
> indication of the TSB to the Users (GUI representation of the timeshiftbuffer
> depth).
> anything time-related is another matter. In MP4, this can be communicated
> in the timeline via an elst box (edit list). Anything else requires some kind of
> time-based Range Unit. And that's orthogonal to our draft - and live content
> in general.
> 
> >> time- shifted MP4 representation. Basically the idea is that the
> >> server would maintain a list of valid fragments and would have to
> >> maintain a valid movie box at the front of the representation as
> >> fragments were added and removed (if a client's expected to consume a
> >> spec-compliant ISO BMFF container).
> > [TL] Yes.
> >
> >> byte- Range requests);
> >> perform a Range request to get the fragment containing the target
> >> time and start rendering it (one or two "bytes" Range request), and
> >> start fetching the next Fragment if/when desired. e.g. If the nearest
> >> random access point for 5 minutes is at the start of a fragment with
> >> offset
> > [TL] Well, almost. You must always start at the fragment boundary. When
> the fragment contains e.g. 4sec of media data (e.g. with 1Sec Gops), the
> client must start fetching from the beginning of the fragment and then skip
> the media data before the desire playtime start. The client can also do an
> open range request.
> >> 4) If the time offset is not found to be after the
> >> currently-available fragments, jump to the live point by grabbing the
> >> last fragment and any fragments that are added on (one "bytes-live"
> >> range request). e.g. if the last fragment's offset is 5555555 with length
> 40001:
> >>
> >> buffer), put the framebuffer on-screen, and then block on the socket
> >> (or wait for async notification) and render the new fragment(s) as they
> come in.
> > [TL] Ok, understood.
> [cp] It's important to remember that this is the feature we care about in
> context of this draft. If you're good with this, then we're on the same page.
> >> * Note that one might use MP4 timelines in here to communicate the
> >> fact that sample time 0 is not media time 0. But I'd rather not get too far
> into that.
> >>
> > [TL] Since we talk about a single HTTP transaction with progressive media
> data, the session is anyhow not cachable.
> [cp] Well, in-progress recordings should still be byte-wise cachable, correct?
> Byte 0 will always be byte 0, byte X will always be byte X. So a cache can be
> populated with the currently-accessible bytes and can satisfy Range
> requests. There might just be a cache miss if a client accesses bytes not-yet-
> cached. But a proxy should know that this is a possibility since the Content-
> Range responses have a "*" in place of the content length.
> >
> >>> If yes, should the solution be limited to live cases?  If I
> >>> understand it correctly, then you are looking for a solution where
> >>> the client indicates a rough byte range in the request and the
> >>> server responds with another range, which is fulfilling some
> >>> condition. In case of a live session with fMP4, the server looks for
> >>> a random access point into the stream. The random access point must
> >>> be a fragment boundary in case of fMP4 and can be PAT, PMT, PES in
> case of TS.
> >>>
> header).
> >> RFC7233 (I'd have to look closely). And I think this is another
> >> example of where a different Range Unit would make sense.
> >>
> >> "decoder- friendly position") in the returned media. Now, this was
> >> option. But if so, TimeSeekRange.dlna.org could be easily replaced
> >> with a "npt" Range Unit (normal play time). And what you're talking
> >> about would be similar but with byte offsets. Both carry a similar
> assumption:
> > [TL] Yes, started also thinking about time range requests, but I couldn't
> remember, whether that was define in DLNA, OIPF of DVB. Are the time
> range headers widely used today? Can such a solution make sense for H1.1
> or H2?
> [cp] TimeSeekRange is widely used in DLNA. But they're more useful for less
> structured content such as MP2 Program/Transport streams. (for MP4
> >> (media-sensitive offset) Range Unit which could take a time offset or
> >> a byte offset (and incorporate live semantics).
> >>
> >> The bytes-live draft (or whatever we end up calling it) is intended
> >> to assume all knowledge is client-side - just as the case with the
> >> bytes Range Unit. I think you're assuming some of this server-side
> >> structure knowledge that I was not intending. But "mso" would make
> another great Range Unit.
> > [TL] I still think, that the client should have precise information of the TSB
> (i.e. precise range and depth) in order to properly render the TSB
> representation on the GUI (Media Players often render the TSB in a progress
> bar together with the progress of the Live Session).
> random access on MPEG2-contained content. If you want to start a draft, I'm
> happy to co-author. ;^J
> >> There are many other potential applications of something like
> >> bytes-live I believe - which is why we're bringing this to the IETF.
> >> But really it's all about getting to the "live point" in a couple different
> ways.
> [cp] Perhaps. But the mechanism of how the transfer occurs - which is what
> this draft is related to - is definitely in the transport space.
> For some complex formats, it may be in the "necessary but not sufficient"
> category. But as I mentioned at the top, for
> >>> movie and fragment header(s). For something like MP2, it's trickier of
> course.
> >>>
> >>> */[TL] Well, in case of fMP4, the client needs to get the Movie
> >>> Header for initialization. Then, proper access point are fragment
> boundaries.
> >>> about "mimicking segmented media". fMP4 is actually the way to
> >>> create ISO-BMFF segments. So, it is for segmented media, but without
> >>> a separate manifest?/*
> >>>
> >>> representation being appended to periodically (usually one fragment at
> a time).
> >>>
> >>> */[TL] So, how does the client know, that the proper fragment
> >>> boundary is at byte position 120000? Do you assume that the client
> >>> first fetches a time-to-byte offset file, which tells the client
> >>> that a access point (e.g. a fragment boundary) is at byte pos
> >>> 120000? If yes, why does the client need the HEAD request, when it
> >>> already has the byte position?/*
> >>>
> >>> unindexed
> >>> fixed amount of content in the hopes of capturing a keyframe (really
> >>> a heuristic).
> >>>
> >>> How should the client know, which byte ranges are already available
> >>> on the server? When the client is playing back from the recorded
> >>> part and would like to skip 5min forward, how does the client know,
> >>> whether a normal range request is needed or whether the client
> >>> should as for the live point? What type of HTTP Status code should
> >>> be provided, when the range request is not yet available of the server?
> >>>
> >>> what data range is available by utilizing a HEAD request with a "Range:
> >>>      (3) if the time is outside the index, jump to the live point
> >>> and update the time to the last-index time or other means (e.g.
> >>> using
> >>> requires precise byteoffset, In case of TS, the client can sync to
> >>> the stream by first searching for 0x47 sync bytes. In (3), how does
> >>> the client determine "outside of the index"? Seems that some sort of
> >>> */[TL] Would be good to clarify, what information the client needs
> >>> to get in order to do the operations. How the client gets the info
> >>> can be left out-of-scope./*
> >>>
> >>> */[TL] Gets a bit clearer, but I still don't understand the "mimic
> >>> HLS or DASH". DASH / HLS focuses on CDN optimization by creating a
> >>> URL from the manifest. Each segment is a "good" access point (in
> >>> DASH always box boundaries and in HLS always TS boundaries even with
> >>> PAT / PMT). So, the key issue here is to clarify, how the client
> >>> gets the byte offsets of the fragment boundaries for range
> >>> requests./*
> >>>
> >>> representation would
> >>> - other than being inefficient - have the kind of multi-second
> >>> latency that segmented live streaming would have.
> >>>
> >> Coombs. We've also brought this up informally with other members of
> >> the WG.
> >>>
> >>>
> >> For example, the problem of combining gzip with range requests is
> >> still unsolved [3].
> >>>
> >>>
> >>>
> >>> [1]https://lists.w3.org/Archives/Public/ietf-http-wg/2015AprJun/0122
> >>> .h
> >>> tml
> >>>
> >>>
> >>> [3]https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/1327
> >>> .h
> >>> tml
> >>>
> >>> starts at the "live point"? It feels like it has one hand tied behind its back.
> >>>
> >>> than bytes-live). If there's interest, I'll write up an RFC to demonstrate...
> >>>
> >>>
> >>>
> >>>
> >>>
> >> Information contained in this email message and its attachments may
> >> be privileged, confidential and protected from disclosure. If you are
> >> not the intended recipient, please do not read, copy, use or disclose
> >> this communication to others. Also please notify the sender by
> >> replying to this message and then delete it from your system.
> >>> --
> >>>
> >>>
> >>>
> >>> craig pratt
> >>>
> >>> Caspia Consulting
> >>>

---

From: Craig Pratt <[email-masked]>
Date: Wed, 20 Apr 2016 12:57:28 -0700

Hey Thornsten,

> B. Enable working with existing HTTP procedures, i.e. client can workout the precise byte offsets.
[cp] I guess I don't see these as mutually exclusive.

[cp] In DLNA we have both time-based and byte-based range methods. And 
what we find is that some clients want the server to "help out" for some 
formats (e.g. MPEG2 content) and these clients utilize time-based range 
units. And for other clients just want to use byte-wise access (esp 
clients accessing ISO-BMFF/MP4).

[cp] Both methods need to accommodate aggregated content. 
TimeSeekRange.dlna.org accommodates range/seek requests that include 
aggregated content. And Range doesn't (which is why we're submitting 
this draft).

[cp] It would be great to bring something like TSR in as a Range Unit, 
but it wouldn't replace bytes-live (at least if it's in the same form). 
And it really make sense (IMHO) for time-based and byte-based seek to be 
completely discreet as time-seek requires a media-content-aware HTTP 
server. Bytes (and bytes-live) can be implemented by a much "dumber" 
("content-dumb"?) HTTP-server.

[cp] I hope I'm making sense. (and we haven't lost too many people...)

-- 

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 21 Apr 2016 06:14:12 +1000

> > What *is* missing is the ability to get a continuous byte range on live
content
> B. Enable working with existing HTTP procedures, i.e. client can workout
the precise byte offsets.

C. Use a different URL (e.g. a ?start= query parameter). It's less optimal
for caching, but it works today without any new specs.

Cheers

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 21 Apr 2016 06:17:08 +1000

wrote:
> C. Use a different URL (e.g. a ?start= query parameter). It's less
optimal for caching, but it works today without any new specs..

Sorry, that's B(ii) isn't it.

---

From: Thorsten Lohmar <[email-masked]>
Date: Wed, 20 Apr 2016 20:40:56 +0000

Hi Craig, Matthew, all,

Yes, there are more than two solutions to the use-case. I think, Matthew solution is more a solution C than a solution B(ii).
Of course, another solution (D) is to use the existing DASH-Live or HLS solution, which utilizes a sequence of segment requests with a manifest. 

And, these solutions are not exclusive. 

BR,
/Thorsten

From: [email-masked] [mailto:[email-masked]] On Behalf Of Matthew Kerwin
Sent: Wednesday, April 20, 2016 10:17 PM
To: Thorsten Lohmar
Cc: [email-masked]
Subject: RE: Issue with "bytes" Range Unit and live streaming

On 21/04/2016 6:14 AM, "Matthew Kerwin" <[email-masked]> wrote:
>
>
> On 21/04/2016 5:37 AM, "Thorsten Lohmar" <[email-masked]> wrote:
> >
> > Hi Craig,
> >
> C. Use a different URL (e.g. a ?start= query parameter). It's less optimal for caching, but it works today without any new specs..

---

From: Göran Eriksson AP <[email-masked]>
Date: Wed, 20 Apr 2016 21:03:22 +0000

>[cp] I hope I'm making sense. (and we haven't lost too many people...)

I have a feeling that is not the case...

---

From: Craig Pratt <[email-masked]>
Date: Wed, 20 Apr 2016 14:08:00 -0700

Reply in-line.

cp

On 4/20/16 1:14 PM, Matthew Kerwin wrote:
>
>
> On 21/04/2016 5:37 AM, "Thorsten Lohmar" <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >
> > Hi Craig,
> >
> optimal for caching, but it works today without any new specs.
>
[cp] We can't use a different URI in our application (the UPnP CDS and 
DLNA RUIH guidelines only one URI per resource).

[cp] We can define a custom header. But (a) it appears that the feature 
we're describing has been requested many times over, (b) at least having 
an RFC enables caches to be implemented. Using a custom header or URI 
will require caching to be disabled by the origin server.

[cp] One point I haven't mentioned about caching with bytes-live: Since 
the bytes-live Range Unit is designed to work in concert with the bytes 
Range Unit (not superceded it), caching can still be performed by 
proxies that don't understand bytes-live. The read-through and hit ratio 
will be reduced. But there's still opportunities to cache.

-- 

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Craig Pratt <[email-masked]>
Date: Wed, 20 Apr 2016 14:18:37 -0700

> I have a feeling that is not the case...
Yeah, I had a feeling that was a rhetorical question. ;^J

I think it is important to consider the most complicated (real) 
scenarios and make sure we're at least considering them (or are at last 
make sure the feature is not incompatible or missing an opportunity at 
progress toward it).

But I think Thorsten and I could keep our media streaming ruminations 
off-list, if there's no one finding value in detailing the intricacies 
of media streaming MP4s and whatnot. Please speak up if otherwise...

cp

-- 

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Craig Pratt <[email-masked]>
Date: Wed, 20 Apr 2016 14:36:11 -0700

> And, these solutions are not exclusive.
Switching to HLS or DASH is not an option for every application, of 
course. And that certainly implies an additional layer of complexity 
when one just wants to provide access to a progressively-updated 
representation.

Media streaming aside, there's always been awareness of 
progressively-updated representations in HTTP (IMHO). All we're really 
trying to do with this draft is enable a Range request that accommodates 
progressively-updated representations. I see this as a simple oversight 
with the bytes Range Unit that we're trying to remedy in a 
least-impactful way - one that many people have recognized but has not 
yet been addressed by an RFC.

Happy Wednesday,

cp

-- 

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: Thorsten Lohmar <[email-masked]>
Date: Wed, 20 Apr 2016 22:22:44 +0000

HI there,

> Switching to HLS or DASH is not an option for every application, of course.
> And that certainly implies an additional layer of complexity when one just
> wants to provide access to a progressively-updated representation.

[TL] Yes, Understood, that this is not for all cases a solution.

> 
> Media streaming aside, there's always been awareness of progressively-
> updated representations in HTTP (IMHO). All we're really trying to do with
> this draft is enable a Range request that accommodates progressively-
> updated representations. I see this as a simple oversight with the bytes
> Range Unit that we're trying to remedy in a least-impactful way - one that
> many people have recognized but has not yet been addressed by an RFC.

[TL] Yes, I think that I understand your use-case and your proposed now. I don't have a strong opinion, whether to solve the use-case on HTTP layer or not. 

> >>>> What *is* missing is the ability to get a continuous byte range on
> >>>> live content that starts at an arbitrary offset and the ability to
> >>>> directly jump to the live point.
> >>> A. Enable it on HTTP Layer through the definition of a new range
> >>> request method B. Enable working with existing HTTP procedures, i.e.
> client can workout the precise byte offsets.
> for caching, but it works today without any new specs..

---

From: Craig Pratt <[email-masked]>
Date: Wed, 20 Apr 2016 15:46:13 -0700

Hey Thorsten,

Well, if we can demonstrate that a new Range Unit can be added safely 
with this draft, maybe adding some application-specific Range Units will 
be an easier sell.

The two RUs I'd see as being good candidates would be a time-based unit 
and a cleartext range unit - both of which have application in media 
streaming. And both can still accommodate caching (e.g. the 
TimeSeekRange response in DLNA includes the byte range corresponding 
with the response body).

Both of these could enable some pretty interesting possibilities in 
DASH/HLS as well. e.g. A DASH/HLS-aware server could field requests made 
against a manifest - allowing the manifest processing logic to live in 
the HTTP server instead of the client and reducing the amount of HTTP 
connections and chatter in the process.

Anyway, a conversation for a different day...

Happy Wednesday,

cp

-- 

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 20 Apr 2016 17:36:57 -0700

> On Apr 20, 2016, at 2:08 PM, Craig Pratt <[email-masked]> wrote:
> 
> Reply in-line.
> 
> cp
> 
>> On 21/04/2016 5:37 AM, "Thorsten Lohmar" <[email-masked] <mailto:[email-masked]>> wrote:
>> >
>> > Hi Craig,
>> >
>> C. Use a different URL (e.g. a ?start= query parameter). It's less optimal for caching, but it works today without any new specs.
>> 
> [cp] We can't use a different URI in our application (the UPnP CDS and DLNA RUIH guidelines only one URI per resource).
> 
> [cp] We can define a custom header. But (a) it appears that the feature we're describing has been requested many times over, (b) at least having an RFC enables caches to be implemented. Using a custom header or URI will require caching to be disabled by the origin server.
> 
> [cp] One point I haven't mentioned about caching with bytes-live: Since the bytes-live Range Unit is designed to work in concert with the bytes Range Unit (not superceded it), caching can still be performed by proxies that don't understand bytes-live. The read-through and hit ratio will be reduced. But there's still opportunities to cache.

This doesn't make any sense.  The live stream URI (whatever it is) always identifies
the current live stream head.  An HTTP cache cannot save that because each new request
expects the latest live stream.  What it can do is parallel cache the live stream's
representation under a different "past stream" URI (or an array of such) and include
instructions on how to use that within the live stream metadata.  Regardless, the live
stream HTTP response must be marked as non-cacheable to avoid filling up normal caches.

On balance, I think this is a poor use of HTTP.  The Apple live streaming protocol,
which is just a dynamically updated representation of a queue by reference, is
able to perform the same task without twisting backwards.  By indirectly referencing
the data streams, the primary queue can remain non-cacheable (until the stream dies
and is archived) without impacting (partial) requests on the referenced data streams.

....Roy

---

From: Craig Pratt <[email-masked]>
Date: Wed, 20 Apr 2016 20:18:02 -0700

Hey Roy,

> stream HTTP response must be marked as non-cacheable to avoid filling up normal caches.
[cp] I think you need to go back and read the rest of the thread - or 
even just my introduction. This is not the use case described. But I'll 
describe it again another way - in the hopes of avoiding misunderstandings.

The URI in this case refers to a random-accessible representation which 
is periodically appended to - not a live-only stream. The representation 
supports the "bytes" Range Unit, so a client can access bytes 0-1000, 
40000-90000, etc. of the growing resource at its discretion. And at some 
point a client wants to be able to get to the live point - just 
receiving the newly-appended (live) data just like you're describing. 
And to be consistent with HTTP semantics (and enable cache hits btw), a 
Range-less GET will get you the beginning of the representation, not the 
end. So there's no good way to get to the live point currently (polling 
being "non-good").

Re: Representation caching

Whether a representation is considered cacheable in this use case is at 
the discretion of the origin server and specific to the use 
case/application - as it should be (imho). There's no *necessity* in 
having a periodically-appended resource marked non-cachable, correct? If 
the resource mutates, it's not cacheable. If it's just being appended 
to, it is cacheable. And if an appended resource stops being appended 
to, it doesn't invalidate the cached representation.

And with the high compression rates of audio/video streams, and the 
larger capacity of proxy caches, I think video resources are more 
cache-worthy than they have ever been (esp live ones that can put high 
demands on a proxy). But they're still free to implement policy based on 
mime type, expiration, etc. - again, as it should be.
> and is archived) without impacting (partial) requests on the referenced data streams.
[cp] I'm going to assume this conclusion is in question since there's 
misunderstanding of the scenario. If not, I'll describe why both basic 
single-representation content and segmented content have their places. 
And please enlighten me if any of the proposed semantics are not 
consistent with the HTTP representation model.

Happy Wednesday,

cp

-- 

craig pratt

Caspia Consulting

[email-masked]

503.746.8008

---

From: <[email-masked]>
Date: Thu, 21 Apr 2016 10:25:07 +0000

On Thursday,21 April 2016 05:18 [email-masked] wrote:

I couldn't agree more. However, it seemed the prevailing sentiment when we tried to resolve the related issue of ranges before content codings, with a new bbcc unit (bytes-before-content-coding), was that the use cases for append-only growth represent an insignificant portion of HTTP traffic. "We live by app-specific protocols to handle these cases. What is so special ... that it must be addressed by http (in a very ugly way)?" [1]

[1] https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/1383.html