Subject: [GGIE] HTTP redirect for HTTP Adaptive Streaming (HAS) in CDN use cases

From: Ori Finkelman <[email-masked]>
Date: Sat, 3 Nov 2018 09:07:32 +0200

Hello ,

HTTP redirect, usually using 302 Found, is widely used in video CDN for
HTTP Adaptive Streaming (HAS) protocols, specifically for HLS and DASH.
Using HTTP redirect instead of the more commonly used delegation by CNAME
is a much more powerful and flexible solution.

At a high level, the required behavior is that after getting an HTTP
redirect to a playlist address that takes the client to a CDN cache, if the
playlist contains
relative references to the media segments, the client should be "sticky"
and request the subsequent media segments directly from the cache without
the need
for repeated redirects that would add undesired latency and harm QoE.
However, due to misinterpretations of the RFC in some cases, and
contradiction between this requirement and the existing RFC in other cases,
we face many
issues with clients that behave differently in the case of HTTP redirect,
when relative references are being used.
A discussion of this use case can also be found in section 2.2.1 of RFC6983
<https://tools.ietf.org/html/rfc6983#section-2.2.1>.

The Streaming Video Alliance (SVA) and DASH Industry Forum (DASH-IF) are
attempting to resolve these issues and create clear guidelines describing
how video players,
browsers and streaming applications should behave in such HTTP redirect
cases.

The attached I-D includes a detailed description of this issue, examples
with our understanding of the current behavior and proposals for
alternative approaches for solution.
We were hoping that the members of HTTPBIS working group could share their
views regarding the preferred  approach in order to resolve this issue.

If there is interest and some available time I can prepare a quick slide
deck for the httpbis meetings next week.

Best regards,
Ori

-- 

*Ori Finkelman*Qwilt | Work: +972-72-2221647 | Mobile: +972-52-3832189 |
[email-masked]
-- 

*Ori Finkelman*Qwilt | Work: +972-72-2221647 | Mobile: +972-52-3832189 |
[email-masked]

---

From: Aaron Falk <[email-masked]>
Date: Sat, 3 Nov 2018 14:44:39 +0700

Hi Ori-

Your draft on 302 is generating a 404.  Is 
draft-finkelman-httpbis-has-redirecting-00 a published doc?

--aaron

---

From: "Ali C. Begen" <[email-masked]>
Date: Sat, 3 Nov 2018 10:58:05 +0300

We have not submitted yet. Hopefully on monday. Can you check the draft Ori
attached in the mean time?

On Sat, Nov 3, 2018 at 10:47 AM Aaron Falk <[email-masked]> wrote:

> A discussion of this use case can also be found in section 2.2.1 of
> RFC6983 <https://tools.ietf.org/html/rfc6983#section-2.2.1>.
>
> _______________________________________________
> GGIE mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ggie
>

---

From: Thomas Peterson <[email-masked]>
Date: Sat, 03 Nov 2018 09:43:52 +0000

I think you should also clarify the scenario where multiple endpoints are provided (both HLS and DASH support, DASH permits multiple BaseURL entities). Is the redirection behaviour at a per-BaseURL?

Regards

From: GGIE <[email-masked]> on behalf of Ori Finkelman <[email-masked]>
Date: Saturday, 3 November 2018 at 07:07
To: <[email-masked]>, "Begen, Ali (Contractor)" <[email-masked]>, <[email-masked]>
Subject: [GGIE] HTTP redirect for HTTP Adaptive Streaming (HAS) in CDN use cases

Hello ,

HTTP redirect, usually using 302 Found, is widely used in video CDN for HTTP Adaptive Streaming (HAS) protocols, specifically for HLS and DASH.

Using HTTP redirect instead of the more commonly used delegation by CNAME is a much more powerful and flexible solution.

At a high level, the required behavior is that after getting an HTTP redirect to a playlist address that takes the client to a CDN cache, if the playlist contains 

relative references to the media segments, the client should be "sticky" and request the subsequent media segments directly from the cache without the need 

However, due to misinterpretations of the RFC in some cases, and  contradiction between this requirement and the existing RFC in other cases, we face many 

issues with clients that behave differently in the case of HTTP redirect, when relative references are being used.

A discussion of this use case can also be found in section 2.2.1 of RFC6983.

The Streaming Video Alliance (SVA) and DASH Industry Forum (DASH-IF) are attempting to resolve these issues and create clear guidelines describing how video players, 

browsers and streaming applications should behave in such HTTP redirect cases. 

The attached I-D includes a detailed description of this issue, examples with our understanding of the current behavior and proposals for alternative approaches for solution.

We were hoping that the members of HTTPBIS working group could share their views regarding the preferred  approach in order to resolve this issue.  

If there is interest and some available time I can prepare a quick slide deck for the httpbis meetings next week. 

Best regards,

Ori

-- 

Ori Finkelman
Qwilt | Work: +972-72-2221647 | Mobile: +972-52-3832189 | [email-masked]

-- 

Ori Finkelman
Qwilt | Work: +972-72-2221647 | Mobile: +972-52-3832189 | [email-masked]

_______________________________________________ GGIE mailing list [email-masked] https://www.ietf.org/mailman/listinfo/ggie

---

From: Ori Finkelman <[email-masked]>
Date: Sat, 3 Nov 2018 15:01:26 +0200

[Resending as I have sent to the wrong list]

Hello ,

Best regards,
Ori

--

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 3 Nov 2018 13:32:48 +0000

Ori,

Thanks for sharing this, and the authors for writing this up. What's
captured in the document broadly matches my experiences in this space.

The problems can be real. A broker server, dimensioned to handle a low rate
of playlist requests, may end up getting overwhelmed with client media
requests. This is hard to predict because client implementations are
diverse; new things come along, and existing ones may regress. I am aware
of non-IETF efforts that attempt to characterise client behaviour like
this, and it wouldn't surprise me if the effort is duplicated across
different content or service providers.

I support discussion that can address this issue but have no strong opinion
on the final solution at this time.

Lucas

---

From: Ori Finkelman <[email-masked]>
Date: Sat, 3 Nov 2018 23:59:59 +0200

Thanks Lucas and Thomas for your inputs.
Thomas, it's a good point, we will have to address the case of multiple
endpoints, we will address it in the next version once we get some inputs
from HTTPbis on the preferred direction.

All, my apologies for distributing by mail, we will submit on Monday
morning as soon as the submission tool is open.
Attached is a newer revision with minor typo fixes.

Thanks,
Ori

On Sat, Nov 3, 2018 at 3:33 PM Lucas Pardue <[email-masked]>
wrote:

> The problems can be real. A broker server, dimensioned to handle a low
> rate of playlist requests, may end up getting overwhelmed with client media
> I support discussion that can address this issue but have no strong
> opinion on the final solution at this time.
>
> Lucas
>
>> We were hoping that the members of HTTPBIS working group could share
>> their views regarding the preferred  approach in order to resolve this
>> issue.
>>

--

---

From: Patrick McManus <[email-masked]>
Date: Sun, 4 Nov 2018 10:35:28 +0700

Hi Ori,

As Chair:
Thanks for the draft and the note.. please submit any/all drafts to the
datatracker as soon as it reopens (which I believe is Monday morning).

As far as I understand your topic, your primary concern is that you are
looking for a clarification on what the base URI used to resolve a relative
URI is when there has been a redirect involved.. specifically in your case
the relative URI is in a manifest file and the manifest was obtained via
302 - is the base URI from the original manifest URI or the one that was
given in the 302 location?

Given that a major focus of our meeting will be on http-core, and this
seems like a clarification of core semantics let's just discuss it there.
Mark and I have huddled and we think
https://github.com/httpwg/http-core/issues/38 seems like the right place.
We'll have discussion time at the meeting and of course, the list is always
a fine place for that kind of discussion too.

As Individual:

* its against the terminal (i.e. redirected) uri - though I had trouble
finding language to quote on it.. so some guidance seems like a good idea
at first glance

* alternative services might also fit your use case - rfc 7838 .. in
particular they have better fallback behavior if you want to revert to the
original URI in case of reachability.

-Patrick

---

From: Ori Finkelman <[email-masked]>
Date: Sun, 4 Nov 2018 09:52:47 +0200

Hi Patrick,
Thanks for the feedback.
Alt-Svc  can serve as a good solution instead of redirect.
Nevertheless, for those applications that do use HTTP redirect, I tend to
think that a clear guidance on the relative reference resolution would be
in place, and the more advanced behavior like failover can be specified as
best practices instead of a standard.

Thanks,
Ori

--

---

From: Lucas Pardue <[email-masked]>
Date: Sun, 4 Nov 2018 08:58:53 +0000

Hey Patrick,

> original URI in case of reachability.

The sad reality is that, in my experience, client support for using Alt-Svc
like this is pretty bad. The last time I checked, even Chrome can't handle
host redirection for HTTP/1.1 or HTTP/2 connections (I can dig out the bug
if you like).

For non-browser clients, which Ori's document is likely targeting, I'd be
shocked if they had any support for Alt-Svc in a fully compliant way. Given
that there is trouble implementing redirects ;-).

Firefox is the only one that seems to do Alt-Svc to its full extent
(protocol support aside).

Lucas

---

From: Julian Reschke <[email-masked]>
Date: Sun, 4 Nov 2018 11:12:44 +0100

On 2018-11-03 14:01, Ori Finkelman wrote:
> ...

Interesting, but...:

> 6.2.  Example 2: Relative reference in an Encapsulating Entity
> 
>    Let us take the same example but instead of the player being passed
>    an absolute URI, let us assume the URI is relative and embedded
>    within another entity (e.g., within a top-level HTML page
>    https://video.cdn.example.com/index.html) In that case /vod/movie/
>    master.m3u8 (the encapsulating entity of 1800K/1800_complete.m3u8) is
>    itself encapsulated (in /index.html) and Section 5.1.2 does apply
>    this time, and the base URI of /vod/movie/master.m3u8 depends on the
>    base URI of /index.html

No.

The entity for </vod/movie/master.m3u8> is *not* embedded in the HTML. 
It's *URI* is. This is very different.

Unless I'm missing something, this is simply a misinterpretation of the 
terminology.

Best regards, Julian

---

From: Ori Finkelman <[email-masked]>
Date: Sun, 4 Nov 2018 13:19:49 +0200

Thanks Julian.

In that case, for Example 2:
Would you say that the base URI for /vod/movie/master.m3u8 is resolved by
section 5.1.3 from the retrieval URI?
In that case the base URI for /vod/movie/master.m3u8 is
https://edge1.dcdn.example.net/vod/movie/master.m3u8.
Since the relative reference 1800K/1800_complete.m3u8 is encapsulated
within master.m3u8, it should be resolved according to section 5.1.2 and
should therefore be
https://edge1.dcdn.example.net/vod/movie/1800K/1800_complete.m3u8

So, example 2 behaves well, according to this interpretation.

For example 3:
Following your input, master.m3u8 IS NOT encapsulated in the HTML (the URI
is), and therefore its base URI should be computed by section 5.1.3 from
the retrieval URI, as in Example 2.
That would lead to the same result that the
vod/movie/1800K/1800_complete.m3u8 base URI is at the edge cache.
https://edge1.dcdn.example.net/vod/movie/1800K/1800_complete.m3u8

Is this the correct interpretation ?
If so, then all these examples lead to the same conclusion that the client
should be 'sticky'.

Thanks,
Ori

--

---

From: Julian Reschke <[email-masked]>
Date: Sun, 4 Nov 2018 12:42:50 +0100

On 2018-11-04 12:19, Ori Finkelman wrote:
> Would you say that the base URI for /vod/movie/master.m3u8 is resolved 
> by section 5.1.3 from the retrieval URI?
> Following your input, master.m3u8 IS NOT encapsulated in the HTML (the 
> URI is), and therefore its base URI should be computed by section 5.1.3 
> from the retrieval URI, as in Example 2.
> If so, then all these examples lead to the same conclusion that the 
> client should be 'sticky'.
> 
> Thanks,
> Ori
> ...

Absolutely.

Best regards, Julian

---

From: Ori Finkelman <[email-masked]>
Date: Sun, 4 Nov 2018 13:45:57 +0200

If that is the case, then we should seek a way to communicate that to the
industry as we see many cases of misinterpretations.

--

---

From: Julian Reschke <[email-masked]>
Date: Sun, 4 Nov 2018 16:14:28 +0100

On 2018-11-04 12:45, Ori Finkelman wrote:
> If that is the case, then we should seek a way to communicate that to 
> the industry as we see many cases of misinterpretations.
> ...

Assuming the WG agrees with my interpretation, I would recommend to 
simply update the internet draft with these conclusions and to point 
anybody who disagrees to that document.

Best regards, Julian

---

From: Ori Finkelman <[email-masked]>
Date: Sun, 4 Nov 2018 19:24:31 +0200

Draft was submitted.
Thanks Ali.

---------- Forwarded message ---------
From: Begen, Ali (Contractor) <[email-masked]>
Date: Sun, Nov 4, 2018, 19:11
Subject: Re: [EXTERNAL] [E] [GGIE] HTTP redirect for HTTP Adaptive
Streaming (HAS) in CDN use cases
To: Ori Finkelman <[email-masked]>

Hi Ori

I submitted the draft. Apparently, the submission got open.

https://datatracker.ietf.org/doc/draft-finkelman-httpbis-has-redirecting/

Attached are the text and xml files.

--

---

From: Ori Finkelman <[email-masked]>
Date: Sun, 4 Nov 2018 19:26:58 +0200

Draft was submitted.

Thanks Ali.

Regards,
Ori

--

---

From: Patrick McManus <[email-masked]>
Date: Mon, 5 Nov 2018 17:01:34 +0700

On Sun, Nov 4, 2018 at 3:59 PM Lucas Pardue <[email-masked]>
wrote:

> The sad reality is that, in my experience, client support for using
> Alt-Svc like this is pretty bad. The last time I checked, even Chrome can't
> handle host redirection for HTTP/1.1 or HTTP/2 connections (I can dig out
> the bug if you like).
>

chrome should speak for themselves if they like, but my understanding from
previous comments is that they intend to fix this as part of the work for
supporting HQ at least for h2. (h1 has always been a little concerning
there as it uses the host and port implicitly)

> that there is trouble implementing redirects ;-).
I bring this up because its the only thing that I know of that satisfies
both relative references against the derived location _and_ fallback to the
original location in the case of reachability failure (which relative
references to the redirected url do not) - which is a nice pattern.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 5 Nov 2018 17:22:59 +0700

My .02 - 

I was thinking that at most, we'd look at adding a bit of text to 3xx to make it clear that a redirected request is a genuinely new request, and therefore has its own base URL, etc.  The fact that some clients follow them automatically is immaterial.

Cheers,

> On 4 Nov 2018, at 10:14 pm, Julian Reschke <[email-masked]> wrote:
> 
>> If that is the case, then we should seek a way to communicate that to the industry as we see many cases of misinterpretations.
>> ...
> 
> Assuming the WG agrees with my interpretation, I would recommend to simply update the internet draft with these conclusions and to point anybody who disagrees to that document.
> 
> Best regards, Julian
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 5 Nov 2018 11:31:09 +0100

> I was thinking that at most, we'd look at adding a bit of text to 3xx to make it clear that a redirected request is a genuinely new request, and therefore has its own base URL, etc.  The fact that…

...I agree with that fact, but I believe browser might have to do some 
homework to get the final URI exposed in XMLHttpRequest...

Best regards, Julian

---

From: Thomas Peterson <[email-masked]>
Date: Mon, 05 Nov 2018 10:48:39 +0000

Not all implementations are using XMLHttpRequest, dash.js has migrated over to using fetch whereas hls.js does not appear to. This only stands to further complicate considerations for implementations and perhaps adding text to the 3xx is not suitable.

Regards

On 05/11/2018, 10:31, "GGIE on behalf of Julian Reschke" <[email-masked] on behalf of [email-masked]> wrote:

    > I was thinking that at most, we'd look at adding a bit of text to 3xx to make it clear that a redirected request is a genuinely new request, and therefore has its own base URL, etc.  The fact t…
    
    ....I agree with that fact, but I believe browser might have to do some 
    homework to get the final URI exposed in XMLHttpRequest...
    
    Best regards, Julian

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 5 Nov 2018 14:23:06 -0800

Hey Patrick,

Some replies inline.

> there as it uses the host and port implicitly)

My understanding is that Chrome host-based Alt-Svc is supported in gQUIC
but not for H2 ( from a Google Groups thread "layering of socket pools
makes implementing this for HTTP/2 significantly more complex"). If that
complexity has been overcome it would be good news, it would be nice to
have a public statement to that effect.

> references to the redirected url do not) - which is a nice pattern.

Totally agree, I think there is untapped potential. Especially when
combined with ORIGIN and Secondary certs.

I could imagine some deployment that offers several alternatives, to avoid
falling back to the broker. Although I don't think implementations would
presently work that way.

DASH also offers a feature to specify multiple origins in the manifest
itself, with the client choosing them in an ordered way. There's
potentially some interesting interplay with Alt-Svc here.

Lucas

---

From: Ryan Hamilton <[email-masked]>
Date: Mon, 5 Nov 2018 17:23:34 -0800

On Mon, Nov 5, 2018 at 2:23 PM, Lucas Pardue <[email-masked]>
wrote:

>> chrome should speak for themselves if they like, but my understanding
>> from previous comments is that they intend to fix this as part of the work
>> for supporting HQ at least for h2. (h1 has always been a little concerning

Alas, this is still the case.

---

From: Ori Finkelman <[email-masked]>
Date: Tue, 6 Nov 2018 03:51:06 +0200

While I agree that Alt-Svc may be the to better solution looking forward,
we still need to resolve the issue for existing HTTP redirect
architectures, mainly for HLS as it is still the dominating video streaming
protocol, but for DASH also.
HTTP redirect is widely in use and clients that don't conform with the
relative resolution, as it seems to be the case, degrades qoe and cause a
significant performance hit on CDN load balancers.

If there is agreement that in case the playlist is requested separately,
and it is not embedded within another encapsulating entity, then the
playlist base URI is the final retrieval URI, after redirect, it would be
great if there could be some clarifications that establish this
understanding.

We could then work with player software developers to adjust their code
accordingly.
If there are gaps in browser XMLHttpRequest or fetch, we can also
articulate how this should be bridged.

Thanks,
Ori

On Tue, Nov 6, 2018 at 3:23 AM Ryan Hamilton <[email-masked]> wrote:

>> I could imagine some deployment that offers several alternatives, to
>> avoid falling back to the broker. Although I don't think implementations
>> would presently work that way.
>>

--

---

From: Julian Reschke <[email-masked]>
Date: Tue, 6 Nov 2018 03:07:38 +0100

On 2018-11-06 02:51, Ori Finkelman wrote:
> While I agree that Alt-Svc may be the to better solution looking 
> forward, we still need to resolve the issue for existing HTTP redirect 
> architectures, mainly for HLS as it is still the dominating video 
> streaming protocol, but for DASH also.
> relative resolution, as it seems to be the case, degrades qoe and cause 
> a significant performance hit on CDN load balancers.
> 
> playlist base URI is the final retrieval URI, after redirect, it would 
> be great if there could be some clarifications that establish this 
> understanding.
> ...

Where exactly do you think a clarification is needed?

Best regards, Julian

---

From: Ori Finkelman <[email-masked]>
Date: Thu, 8 Nov 2018 23:40:37 +0700

Hi Julian,
I think clarifications would help relating section 5.1.2 of rfc3986.

   ... For a document that is enclosed
   within another entity, such as a message or archive, the retrieval
   context is that entity.  Thus, the default base URI of a
   representation is the base URI of the entity in which the
   representation is encapsulated.

Now that I know that encapsulating the URI is not encapsulating the the
document itself, it is clear, however since this was under discussion of
quite a few people, before bringing it to HTTPbis, and we all missed it, it
might be good to help the reader by emphasizing this difference.
I think one confusing point was the use of HTML5 embedding tags such as
<embed> or <video> may lead you to the wrong conclusion that your document
is "embedded" within the encapsulation, while it is only logically embedded.

I am not sure what would be a good phrasing for that, but maybe something
along the line of:
"Note that a URI enclosed within another entity is not an encapsulation of
a document.
Therefore the base URI of such a document that is retrieved by such a URI,
must be established according to section 5.1.3"

If it doesn't make sense to add such a clarification, we can do it on an
external document we (the SVA) will be publishing for the video players
development community.

Best regards,
Ori

--