Subject: [Moq] Warp

From: Luke Curley <[email-masked]>
Date: Fri, 11 Feb 2022 11:22:14 -0800

Hey MOQ, I just published a draft for Warp
<https://datatracker.ietf.org/doc/draft-lcurley-warp/>. Here's a quick FAQ:

*What is Warp?*
Twitch has developed a new video distribution protocol to replace our
custom low-latency HLS stack. Warp uses QUIC streams to deliver media
segments, prioritizing streams based on content and age. This allows
viewers to skip old video content during congestion instead of buffering;
improving the user experience and reducing latency.

*What about contribution?*
Warp is very similar to Facebook's RUSH
<https://www.ietf.org/archive/id/draft-kpugin-rush-00.html> and can be used
as a contribution protocol. There's a few fundamental differences, like the
prioritization scheme and transferring media as segments. This first
version of the draft focuses on these core differences and omits anything
else that could be a distraction.

*Why not WebRTC?*
We initially used WebRTC (both media and data channels) for
last mile-delivery but the user experience was significantly worse than our
existing stack. There were so many minor issues, primarily caused by
WebRTC's focus on real-time latency and the inability to control the client
(browser) behavior. I personally had to scrap years of work on a custom
SFU. 😔

*Why not use datagrams?*
Warp uses QUIC streams because it dramatically simplifies the protocol. We
get the full benefit of QUIC's fragmentation, congestion control, flow
control, recovery, cancellation, multiplexing, etc. Using datagrams gives
you extra flexibility but it also means you have to reimplement everything
on every platform.

*Why not use HTTP?*
Good question! The key to warp is the prioritization mechanism, which could
work with HTTP/3 and possibly HTTP/2. Twitch has the benefit of running our
own network so it was just simpler to make a push-based protocol using QUIC
and WebTransport. I've got some ideas for a more complicated HTTP solution
that would enable CDN support..

*How is media delivered?*
Warp sends each segment (group of pictures) over a QUIC stream. Audio and
newer video segments are prioritized, causing older video segments to
starve during congestion. Either side can cancel the stream to effectively
drop the tail of a segment. Media is quite linear by nature and most frames
need to be processed in decode order.

*Why not drop individual frames?*
We decided that it wasn't worth dropping non-reference frames, given their
infrequency and relatively small size for high quality media. Our hardware
encodes (QuickSync) have only reference frames and we've seen software
encodes with only 3% non-reference frames by file size. And of course,
dropping reference frames will cause artifacting or freezing so that wasn't
an option.

*How could this be improved?*
We want to experiment with layered coding (ex. SVC) at some point in the
future. This would involve transferring non-reference frames/slices on a
different QUIC stream so they can be deprioritized. Simulcast would work
the same way: transfer each rendition on a different QUIC stream
prioritized based on the resolution.

*Why use fMP4?*
HLS and DASH support CMAF: a standard for fragmenting MP4 files. Warp uses
this file format so we can deliver the same segment data regardless of the
delivery protocol. The Warp MP4 atom uses JSON because I was too lazy to do
things "properly" for this first draft. The wire format doesn't matter!

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Fri, 11 Feb 2022 20:59:32 +0100

Hi luke,

QUICK question, what is the target glass to glass latency for WARP?

Best regards
Sergio

El vie, 11 feb 2022 20:22, Luke Curley <[email-masked]> escribió:

> <https://www.ietf.org/archive/id/draft-kpugin-rush-00.html> and can be
> used as a contribution protocol. There's a few fundamental differences,
> like the prioritization scheme and transferring media as segments. This
> first version of the draft focuses on these core differences and omits
> anything else that could be a distraction.
>
> *Why not WebRTC?*
> Good question! The key to warp is the prioritization mechanism, which
> could work with HTTP/3 and possibly HTTP/2. Twitch has the benefit of
> running our own network so it was just simpler to make a push-based
> protocol using QUIC and WebTransport. I've got some ideas for a more
> complicated HTTP solution that would enable CDN support..
>
> --
> Moq mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/moq
>

---

From: Luke Curley <[email-masked]>
Date: Fri, 11 Feb 2022 13:10:43 -0800

Hey Sergio,

Warp has flexible latency depending on the broadcaster and viewer(s).

The broadcaster chooses their encoding settings, for example using b-frames
(higher latency/quality) or using a larger look-ahead buffer (better
compression and rate control). The viewer dynamically chooses their buffer
size, dictating how long to wait before skipping the end of a segment.

With a perfect network, Warp would transfer each video frame from the
encoder to decoder as they are generated. However, congestion makes that
impossible, which is why it's necessary to have a dynamic player buffer for
smooth playback. For example, a viewer with a reliable connection may have
a 500ms buffer, while a viewer with a cellular connection may have a 2s
buffer, while a viewer in a developing country may have a 5s buffer, while
a service that archives the stream may have a 30s buffer for maximum
reliability.

The broadcaster and any intermediate proxies do not know or care about each
viewer's desired latency. They just create QUIC streams, transmit packets
based on stream priority, and eventually close any streams if they reach
some maximum upper bound. This makes it ideal for video distribution
especially when multiple caches and proxies are involved.

On Fri, Feb 11, 2022 at 11:59 AM Sergio Garcia Murillo <
[email-masked]> wrote:

>> Warp uses QUIC streams because it dramatically simplifies the protocol.
>> We get the full benefit of QUIC's fragmentation, congestion control, flow
>> We decided that it wasn't worth dropping non-reference frames, given
>> their infrequency and relatively small size for high quality media. Our
>> hardware encodes (QuickSync) have only reference frames and we've seen
>> software encodes with only 3% non-reference frames by file size. And of
>> course, dropping reference frames will cause artifacting or freezing so
>> that wasn't an option.
>>
>> HLS and DASH support CMAF: a standard for fragmenting MP4 files. Warp
>> uses this file format so we can deliver the same segment data regardless of
>> the delivery protocol. The Warp MP4 atom uses JSON because I was too lazy
>> to do things "properly" for this first draft. The wire format doesn't
>> matter!
>> --
>> Moq mailing list
>> [email-masked]

---

From: "Law, Will" <[email-masked]>
Date: Sat, 12 Feb 2022 00:27:44 +0000

@Luke – how does WARP handle throughput variation across the connection (the equivalent of ABR with HAS)? The draft indicates that older frames are dropped in the face of congestion. This implies that resolution and encoded bitrate remain constant and that it’s the rendered frame rate that drops on the client to compensate for any throughput degradation. If that is correct, then at what point can the client decide I’m tired of receiving the 4K feed at 8fps, I’d rather get 1080p at 30fps? Conceivably it could request the server to begin sending a lower resolution/bitrate stream of data, however the established streams are unidirectional and no control back-channel is defined. It could also tune-in to a new QUIC stream at the appropriate bitrate, if there was some standard metadata to define what was available and how to access it.   Do you consider discovery and service description to be out of scope of this core protocol definition? If so, has any thought be given to extending WARP so that it includes service discovery and description and perhaps a control back-channel?

Cheers
Will

From: Luke Curley <[email-masked]>
Date: Friday, February 11, 2022 at 1:11 PM
To: Sergio Garcia Murillo <[email-masked]>
Cc: MOQ Mailing List <[email-masked]>
Subject: Re: [Moq] Warp

Hey Sergio,

The broadcaster chooses their encoding settings, for example using b-frames (higher latency/quality) or using a larger look-ahead buffer (better compression and rate control). The viewer dynamically chooses their buffer size, dictating how long to wait before skipping the end of a segment.

With a perfect network, Warp would transfer each video frame from the encoder to decoder as they are generated. However, congestion makes that impossible, which is why it's necessary to have a dynamic player buffer for smooth playback. For example, a viewer with a reliable connection may have a 500ms buffer, while a viewer with a cellular connection may have a 2s buffer, while a viewer in a developing country may have a 5s buffer, while a service that archives the stream may have a 30s buffer for maximum reliability.

The broadcaster and any intermediate proxies do not know or care about each viewer's desired latency. They just create QUIC streams, transmit packets based on stream priority, and eventually close any streams if they reach some maximum upper bound. This makes it ideal for video distribution especially when multiple caches and proxies are involved.

On Fri, Feb 11, 2022 at 11:59 AM Sergio Garcia Murillo <[email-masked]<mailto:[email-masked]>> wrote:
Hi luke,

Best regards
Sergio

El vie, 11 feb 2022 20:22, Luke Curley <[email-masked]<mailto:[email-masked]>> escribió:
Hey MOQ, I just published a draft for Warp<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-lcurley-warp/__;!!GjvTz_vk!CN-FzuL40h3RSvNUobOIUtEEMChMR2oAcW4N7QAzHt3yJISvAijnqM0MaK7E$>. Here's a quick FAQ:

What is Warp?
Twitch has developed a new video distribution protocol to replace our custom low-latency HLS stack. Warp uses QUIC streams to deliver media segments, prioritizing streams based on content and age. This allows viewers to skip old video content during congestion instead of buffering; improving the user experience and reducing latency.

What about contribution?
Warp is very similar to Facebook's RUSH<https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-kpugin-rush-00.html__;!!GjvTz_vk!CN-FzuL40h3RSvNUobOIUtEEMChMR2oAcW4N7QAzHt3yJISvAijnqK2Y57G-$> and can be used as a contribution protocol. There's a few fundamental differences, like the prioritization scheme and transferring media as segments. This first version of the draft focuses on these core differences and omits anything else that could be a distraction.

Why not WebRTC?
We initially used WebRTC (both media and data channels) for last mile-delivery but the user experience was significantly worse than our existing stack. There were so many minor issues, primarily caused by WebRTC's focus on real-time latency and the inability to control the client (browser) behavior. I personally had to scrap years of work on a custom SFU. 😔

Why not use datagrams?
Warp uses QUIC streams because it dramatically simplifies the protocol. We get the full benefit of QUIC's fragmentation, congestion control, flow control, recovery, cancellation, multiplexing, etc. Using datagrams gives you extra flexibility but it also means you have to reimplement everything on every platform.

Why not use HTTP?
Good question! The key to warp is the prioritization mechanism, which could work with HTTP/3 and possibly HTTP/2. Twitch has the benefit of running our own network so it was just simpler to make a push-based protocol using QUIC and WebTransport. I've got some ideas for a more complicated HTTP solution that would enable CDN support..

How is media delivered?
Warp sends each segment (group of pictures) over a QUIC stream. Audio and newer video segments are prioritized, causing older video segments to starve during congestion. Either side can cancel the stream to effectively drop the tail of a segment. Media is quite linear by nature and most frames need to be processed in decode order.

Why not drop individual frames?
We decided that it wasn't worth dropping non-reference frames, given their infrequency and relatively small size for high quality media. Our hardware encodes (QuickSync) have only reference frames and we've seen software encodes with only 3% non-reference frames by file size. And of course, dropping reference frames will cause artifacting or freezing so that wasn't an option.

How could this be improved?
We want to experiment with layered coding (ex. SVC) at some point in the future. This would involve transferring non-reference frames/slices on a different QUIC stream so they can be deprioritized. Simulcast would work the same way: transfer each rendition on a different QUIC stream prioritized based on the resolution.

Why use fMP4?
HLS and DASH support CMAF: a standard for fragmenting MP4 files. Warp uses this file format so we can deliver the same segment data regardless of the delivery protocol. The Warp MP4 atom uses JSON because I was too lazy to do things "properly" for this first draft. The wire format doesn't matter!
--
Moq mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/moq<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/moq__;!!GjvTz_vk!CN-FzuL40h3RSvNUobOIUtEEMChMR2oAcW4N7QAzHt3yJISvAijnqCbRA_82$>

---

From: Luke Curley <[email-masked]>
Date: Fri, 11 Feb 2022 17:32:44 -0800

Hey Will,

Unlike HLS, the media sender is responsible for ABR. Our server pulls the
estimated bitrate directly from the QUIC congestion controller (BBR, Cubic,
etc) and switches renditions at segment boundaries. This is a dramatic
improvement over client-side ABR because it's the actual rate at which
media can be sent. It's also the primary challenge with using Warp over
HTTP/3 with CDN support.

Also I want to clarify that this draft is not complete. I wanted to focus
on what I felt were the core concepts that would shape a WG. That may have
been a mistake because it's come up a few times... and in fact. the client
can create streams. These are used to send messages like
load/play/pause/track but somehow I completely neglected to document it.

On Fri, Feb 11, 2022 at 4:27 PM Law, Will <[email-masked]> wrote:

> @Luke – how does WARP handle throughput variation across the connection
> (the equivalent of ABR with HAS)? The draft indicates that older frames are
> dropped in the face of congestion. This implies that resolution and encoded
> bitrate remain constant and that it’s the rendered frame rate that drops on
> the client to compensate for any throughput degradation. If that is
> correct, then at what point can the client decide I’m tired of receiving
> the 4K feed at 8fps, I’d rather get 1080p at 30fps? Conceivably it could
> request the server to begin sending a lower resolution/bitrate stream of
> data, however the established streams are unidirectional and no control
> back-channel is defined. It could also tune-in to a new QUIC stream at the
> appropriate bitrate, if there was some standard metadata to define what was
> available and how to access it.   Do you consider discovery and service
> description to be out of scope of this core protocol definition? If so, has
> any thought be given to extending WARP so that it includes service
> discovery and description and perhaps a control back-channel?
>
>
>
> Cheers
>
> Will
>
>
>
>
>
> *From: *Luke Curley <[email-masked]>
> *Date: *Friday, February 11, 2022 at 1:11 PM
> *To: *Sergio Garcia Murillo <[email-masked]>
> *Cc: *MOQ Mailing List <[email-masked]>
> *Subject: *Re: [Moq] Warp
>
>
>
> Hey Sergio,
>
>
>
> The broadcaster chooses their encoding settings, for example using
> b-frames (higher latency/quality) or using a larger look-ahead buffer
> (better compression and rate control). The viewer dynamically chooses their
> buffer size, dictating how long to wait before skipping the end of a
> segment.
>
>
>
> The broadcaster and any intermediate proxies do not know or care about
> each viewer's desired latency. They just create QUIC streams, transmit
> packets based on stream priority, and eventually close any streams if they
> reach some maximum upper bound. This makes it ideal for video distribution
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-lcurley-warp/__;!!GjvTz_vk!CN-FzuL40h3RSvNUobOIUtEEMChMR2oAcW4N7QAzHt3yJISvAijnqM0MaK7E$>.
> Here's a quick FAQ:
>
>
>
> *What is Warp?*
>
> <https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-kpugin-rush-00.html__;!!GjvTz_vk!CN-FzuL40h3RSvNUobOIUtEEMChMR2oAcW4N7QAzHt3yJISvAijnqK2Y57G-$> and
> can be used as a contribution protocol. There's a few fundamental
> differences, like the prioritization scheme and transferring media as
> segments. This first version of the draft focuses on these core differences
> and omits anything else that could be a distraction.
>
>
>
> *Why not WebRTC?*
>
> * How could this be improved?*
>
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/moq__;!!GjvTz_vk!CN-FzuL40h3RSvNUobOIUtEEMChMR2oAcW4N7QAzHt3yJISvAijnqCbRA_82$>
>
>

---

From: Luke Curley <[email-masked]>
Date: Fri, 11 Feb 2022 21:20:24 -0800

...and to clarify what I mean by "CDN support", I mean using HTTP/3
requests instead of QUIC streams. A client could request each HLS/DASH
segment in parallel providing the Warp priority as a header
<https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-priority>. You
effectively get the same segment data and prioritization but encapsulated
in a HTTP response.

However it does get quite a bit more complicated than that. The biggest
issue is that prioritization is not guaranteed, especially when multiple
connections are involved (ex. different hostnames). It's also very
difficult for the server to provide a timely bandwidth estimate for ABR. We
opted to take the simpler route and push via WebTransport instead of
pulling via HTTP/3.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 15 Feb 2022 04:30:59 +0000

Hi Luke,

Warp transfers independent media segments over QUIC. Instead of bare QUIC, if you’re willing to use HTTP/3, which it seems like you’re open to considering, then traditional HTTP streaming (any flavor, DASH, HLS, or their low latency variants) would seemingly support everything (including prioritization via httpbis-priority) with no custom protocol required.

If you had H3/QUIC stacks which both support prioritization, with independent media segments as separate requests/resources on separate QUIC streams in the usual way, would you need any other aspect of Warp?

Is the reverse direction of contribution as opposed to distribution a problem? Is the following simplification an over-simplification?

Contribution: producer/client pushes segmentX.mp4 to consumer/server via H3 PUT/POST with priority headers over QUIC streamX
Distribution: consumer/client pulls segmentX.mp4 from producer/server via H3 GET with priority headers over QUIC streamX

In both cases, the HTTP client specifies the desired priority, which seems like the right thing, and the server response can echo that to downstream intermediaries (if there are any).

Latency would depend on how a server bridges contribution to distribution, which could vary from a single packet to a full segment of latency, if left purely to implementation optimization with no standard.

This seems too simple and obvious. Why isn’t everyone already doing this? What dragons lurk?

Cheers,
Mo

From: Moq <[email-masked]> on behalf of Luke Curley <[email-masked]>
Date: Friday, February 11, 2022 at 2:23 PM
To: MOQ Mailing List <[email-masked]>
Subject: [Moq] Warp

Hey MOQ, I just published a draft for Warp<https://datatracker.ietf.org/doc/draft-lcurley-warp/>. Here's a quick FAQ:

What is Warp?

What about contribution?
Warp is very similar to Facebook's RUSH<https://www.ietf.org/archive/id/draft-kpugin-rush-00.html> and can be used as a contribution protocol. There's a few fundamental differences, like the prioritization scheme and transferring media as segments. This first version of the draft focuses on these core differences and omits anything else that could be a distraction.

Why not WebRTC?

Why not use datagrams?

Why not use HTTP?

How is media delivered?

Why use fMP4?

---

From: Luke Curley <[email-masked]>
Date: Mon, 14 Feb 2022 22:23:01 -0800

Hey Mo,

Absolutely!

Twitch is using QUIC (actually WebTransport) instead of HTTP/3 just to
avoid some extra headaches caused by HTTP semantics. I'm not sure which
approach is easier to standardize, but we should take it!

At one point I started a draft called DASH-WARP but I'll be honest, I
didn't want to learn the DASH terminology. I'm also not a fan of how
divergent contribution and distribution protocols have become. This is a
good opportunity to reevaluate why we currently need a separate protocol
for both use cases. I like your idea of flipping the HTTP verb while
keeping everything else the same. You'll notice that I used the terms
"sender" and "receiver" instead of "server" and "client" in the Warp draft
for the very same reason.

As for why this hasn't been done before; no clue, and maybe it has. I
really only stumbled upon the idea thanks to the recent efforts to actually
eliminate head-of-line blocking in HTTP.

On Mon, Feb 14, 2022, 8:31 PM Mo Zanaty (mzanaty) <[email-masked]> wrote:

> Hi Luke,
>
>
>
> Warp transfers independent media segments over QUIC. Instead of bare QUIC,
> if you’re willing to use HTTP/3, which it seems like you’re open to
> considering, then traditional HTTP streaming (any flavor, DASH, HLS, or
> their low latency variants) would seemingly support everything (including
> prioritization via httpbis-priority) with no custom protocol required.
>
>
>
> If you had H3/QUIC stacks which both support prioritization, with
> independent media segments as separate requests/resources on separate QUIC
> streams in the usual way, would you need any other aspect of Warp?
>
>
>
> Is the reverse direction of contribution as opposed to distribution a
> problem? Is the following simplification an over-simplification?
>
>
>
> Contribution: producer/client pushes segmentX.mp4 to consumer/server via
> H3 PUT/POST with priority headers over QUIC streamX
>
> Distribution: consumer/client pulls segmentX.mp4 from producer/server via
> H3 GET with priority headers over QUIC streamX
>
>
>
> In both cases, the HTTP client specifies the desired priority, which seems
> like the right thing, and the server response can echo that to downstream
> intermediaries (if there are any).
>
>
>
> Latency would depend on how a server bridges contribution to distribution,
> which could vary from a single packet to a full segment of latency, if left
> purely to implementation optimization with no standard.
>
>
>
> *From: *Moq <[email-masked]> on behalf of Luke Curley <
> [email-masked]>
> *Date: *Friday, February 11, 2022 at 2:23 PM
> *To: *MOQ Mailing List <[email-masked]>
> *Subject: *[Moq] Warp
>
>
>

---

From: Luke Curley <[email-masked]>
Date: Mon, 14 Feb 2022 22:47:37 -0800

Oh and there are definitely some dragons. But like I mentioned in another
email, Warp is currently enabled in production for 1% of Twitch Chrome
users. I can share results once they're statistically significant and some
lingering bugs have been fixed.

Our biggest issue right now is optimizing QUIC, as delivering UDP packets
is significantly more expensive than TCP given the current Linux APIs
<https://blog.cloudflare.com/accelerating-udp-packet-transmission-for-quic/>.
There's also some issues with BBR and latency, but that's no worse than the
current state of TCP protocols. And finally, platform support, which is why
we're making a big push to get this tech out there and adopted.

On Mon, Feb 14, 2022 at 10:23 PM Luke Curley <[email-masked]> wrote:

>> Warp transfers independent media segments over QUIC. Instead of bare
>> QUIC, if you’re willing to use HTTP/3, which it seems like you’re open to
>> In both cases, the HTTP client specifies the desired priority, which
>> seems like the right thing, and the server response can echo that to
>> downstream intermediaries (if there are any).
>>
>>
>>
>> Latency would depend on how a server bridges contribution to
>> distribution, which could vary from a single packet to a full segment of
>> latency, if left purely to implementation optimization with no standard.
>>
>>
>>

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 15 Feb 2022 07:41:03 +0000

Hey,

Thanks for publishing Luke, it's been a good conversation catalyst.

> eliminate head-of-line blocking in HTTP.

I could speculate that part of the reason such an approach wasn't done
before was the absence of conmon APIs and/or absence of standard
intermediary-to-origin and origin-to-intermediary signalling.

For APIs, I mean, an ability for clients to control the value of a priority
signal; an ability for server applications to control how servers actually
schedule.

For signalling, I mean a way to talk about the priority of the
client-to-intermediary connection properties outside of the actual
connection.

RFC 7540 priority signals (the first incarnation of HTTP/2 signals) support
the notion of signalling strict priority order where newer streams are more
important. As suggested in Warp. It just wasn't really exposed very well to
let people use it.  For example, trying to do so using Fetch. So even if
there was a full end-to-end H2 chain across intermediaries (e.g. low
latency HLS) it would be difficult to have explicit control over the
sending of response data to clients.

As Mo suggested, Extensible Priorities [1] addresses some of these
challenges, especially where there are multiple hops that might support
different HTTP versions or where APIs are version agnostic (Fetch).
However, out of the box Extensible Prioirites suggests serving data of the
same importance in the order it was requested. There's ways to do the
reverse; Luke and I have talked previously about different possible
methods. I'm confident that we could find an answer with enough discussion,
design and implementation experience.

All that said, I'm not overwhelmingly convinced that HTTP has to be used.
It introduces the potential for problems where an intermediary shards or
coalesces requests over different connections and ordering is affected. In
contrast, WebTransport stands a better chance of keeping streams bound to
the connection, aiding strict ordering. It also has a web-accessible API,
avoiding problems with going native QUIC.

Cheers
Lucas

[1] https://datatracker.ietf.org/doc/draft-ietf-httpbis-priority/