Subject: Re: [AVTCORE] Comments on draft-westerlund-avtcore-rtp-simulcast-00

From: Magnus Westerlund <[email-masked]>
Date: Thu, 3 Nov 2011 15:31:34 +0100

Ray,

Thanks for the review. Some replies inline.

On 2011-10-26 09:27, Brandenburg, R. (Ray) van wrote:
> Hi Magnus,
> 
>  
> 
> I reviewed your draft draft-westerlund-avtcore-rtp-simulcast-00 and have
> some questions/comments:
> 
>  
> 
> -          Section 3: This section lacks a summary/conclusion. A number
> of different scenarios are introduced, some of which are explicitly
> stated to be out-of-scope (e.g. 3.5), and some are stated to be in-scope
> (e.g. 3.3). Some scenarios however (e.g. 3.2) are not specified to be
> either in- or out-of-scope.
> 

We consider Multicast in scope but with less importance that the mixer
based scenarios.

>  
> 
> -          Section 3: This section talks quite a bit about the relation
> between Simulcast and Scalable Coding and in which situations one of the
> two is more suitable. I’m not sure why this is relevant. Simulcast and
> Scalable Coding might be two techniques that in some cases can be used
> to solve the same problem, the layer on which they do this is completely
> different (codec-level versus transport level). The fact that Scalable
> Coding might, in some situations, be a better solution than Simulcast
> does not mean that Simulcast should not be standardized to handle those
> situations. I understand that you’ve written this mainly from a video
> conferencing perspective, in which Scalable Coding might be used more
> often, but in a lot of situations, such as IPTV, Scalable Coding is just
> not an option due to its inherent complexity. In these situations
> Simulcast might be able to solve some real issues, despite the fact that
> Scalable Coding might be able to solve these issues more efficiently.
> 

I think layered encoding is both a codec level and a transport level
thing. Just as in simulcast you need transport level methods for
selecting the media streams/layers that you want to receive.

I fully agree that there are use cases where one might be considered
much more appropriate than the other. And that is part of our arguments
for why it should be defined for usage also, even though we have layered
coding specified.

>  
> 
> -          Section 3.1.1 See comment above. Why is this section relevant
> to the rest of the document?
> 

See above. But will attempt to clarify this.

>  
> 
> -          Section 3.2.1: This scenario is the only one of the scenarios
> in section 3 to make a clear pro/con of SSRC versus Session based
> multiplexing. Furthermore, it does so before the actual discussion of
> SSRC vs. Session-based multiplexing is introduced in section 4.
> 

Ok, for consistency we should likely move the pro and cons to the
analysis section.

>  
> 
> -          Section 3.3: For the tiled streaming use case described
> below, this scenario is especially relevant. Is there any technical
> reason why it has less emphasis than the RTP-Mixer scenario?

If my understanding of tiled is correct, I don't see how this use case
maps onto tiling. Please elaborate.

> 
>  
> 
> -          Section 5.5: Your conclusion here is that Session Based
> Multiplexing is the best choice, since SSRC multiplexing seems to
> require a large amount of extensions in order to work. How does this
> conclusion relate to the draft-lennox-rtcweb-rtp-media-type-mux-00 which
> seems to suggest that SSRC multiplexing is not that difficult? (I should
> note that I have no experience with sending multiple streams inside a
> single RTP session, so it could be that I don’t understand the issues
> correctly)

If you review RTP Multiplexing architecture you are likely going realize
that there are a number of cases where I consider SSRC multiplexed media
streams to be the best and most appropriate choice. However, it is
important that one consider the use case. So this document argues that
for Simulcast the best choice is to have the simulcast version in
different RTP sessions. However, if one have multiple media streams one
should in fact use multiple SSRC in each of these sessions to carry the
different media sources.

And in sessions where all media streams going to be sent are of the same
type and same configuration then using SSRC multiplexing becomes easy.
The added twist in Lennox's draft is the multiple media types and that
do bring a few issues, but not that many.
https://datatracker.ietf.org/doc/draft-westerlund-avtcore-multiplex-architecture/
and
https://datatracker.ietf.org/doc/draft-westerlund-avtcore-transport-multiplexing/
has discussion on that proposal.

The main point of the multiplexing architecture document is to make
clear that one size fits all does not apply to using multiple SSRCs vs
using several RTP sessions.

> 
>  
> 
> -          Section 6.2: How do you suggest the sequence numbers and RTP
> timestamps are handled between multiple alternative streams? Would it
> make things easier if these were aligned across multiple streams?

No, I don't believe in alignment. first of all the sequence numbers
needs to progress with the packets actually sent for a media stream.
Encoding alternatives are quite likely to have different packet rates,
especially for video as soon as one comes to a media bit-rate where some
alternative requires multiple packets per video frame.

When it comes to timestamp I don't see any need either. For the best
robustness and simplicity one should simply skip using alignment and
instead use the mechanism for synchronizing media clocks with each other
that exist, i.e. RTCP SR and Rapid Synchronization of RTP flows (RFC 6051).

>  
> 
> And one more remark:
> 
>  
> 
> -          In the introduction session you describe three different ways
> in which the encoding of a media content can differ: bit-rate, codec and
> sampling. Recent work in the area of immersive media has proposed a
> fourth method: tiled video. With tiled video (or spatial segmentation),
> the video resulting from a single camera is split into a number of
> areas, each focusing on a particular spatial area of the video (e.g. a
> single video source could be tiled into four separate video streams, one
> describing the topleft quarter of the video and three more describing
> the topright, bottomleft and bottomright quarters respectively).
> 

If I understand this correct, what you really are talking about are
breaking up the output of one camer or several aligned cameras that can
be considered to produce a single video image is divided into several
separate streams. Then on the receiver side these are commonly one
device per media stream. So a practical example would be to split the
output of 4k camera into four 1080p media streams and on the receiver
side use 4 1080 projectors that are aligned so the right edges matches.

If I am correct in my understanding that isn't simulcast. There are no
two or more  different alternatives of the media source. From my
perspective it looks like your splitting your single media source into
multiple synchronized sources. However, it is clear that if you are
doing Tiled video then you would like to consider the transport options
for it. Because it depends if the receiver is actually a single
end-point with 4 displays connected to them or if one actually have four
different end-points and like to have each stream only being delivered
to each end-point, not all of the others.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Brandenburg, R. (Ray) van" <[email-masked]>
Date: Thu, 3 Nov 2011 15:27:42 +0000

Hi Magnus, 

See comments inline. 

Ray

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]] 
Sent: donderdag 3 november 2011 15:32
To: Brandenburg, R. (Ray) van
Cc: [email-masked]
Subject: Re: [AVTCORE] Comments on draft-westerlund-avtcore-rtp-simulcast-00

Ray,

On 2011-10-26 09:27, Brandenburg, R. (Ray) van wrote:
> Hi Magnus,
> 
>  
> 
> I reviewed your draft draft-westerlund-avtcore-rtp-simulcast-00 and 
> have some questions/comments:
> 
>  
> 
> stated to be out-of-scope (e.g. 3.5), and some are stated to be 
> in-scope (e.g. 3.3). Some scenarios however (e.g. 3.2) are not 
> specified to be either in- or out-of-scope.
> 

We consider Multicast in scope but with less importance that the mixer based scenarios.

[RAY: Ok. Maybe you could add the reasoning to the document?]

> between Simulcast and Scalable Coding and in which situations one of 
> the two is more suitable. I'm not sure why this is relevant. Simulcast 
> and Scalable Coding might be two techniques that in some cases can be 
> used to solve the same problem, the layer on which they do this is 
> completely different (codec-level versus transport level). The fact 
> that Scalable Coding might, in some situations, be a better solution 
> than Simulcast does not mean that Simulcast should not be standardized 
> to handle those situations. I understand that you've written this 
> mainly from a video conferencing perspective, in which Scalable Coding 
> might be used more often, but in a lot of situations, such as IPTV, 
> Scalable Coding is just not an option due to its inherent complexity. 
> In these situations Simulcast might be able to solve some real issues, 
> despite the fact that Scalable Coding might be able to solve these issues more efficiently.
> 

I think layered encoding is both a codec level and a transport level thing. Just as in simulcast you need transport level methods for selecting the media streams/layers that you want to receive.

I fully agree that there are use cases where one might be considered much more appropriate than the other. And that is part of our arguments for why it should be defined for usage also, even though we have layered coding specified.

[RAY: I don't agree with you that layered coding is a transport-level thing (or maybe I don't understand your definition of layered coding. With layered coding I mean e.g. SVC). It might be that layered-coding requires some specific transport-level properties (such as alignment), however these are not unique to layered coding. Your draft (or specifically section 3) seems to imply that Simulcast can be considered an alternative to Scalable Coding and that you therefore have to argue why Simulcast is better in some situations. In my opinion, Scalable Coding is an alternative to creating multiple bitrates/resolutions (i.e. Adaptive Streaming) and not to Simulcast. My understanding is that Simulcast is a method to deliver those multiple bitrates, the same way it could be used to deliver Scalable Coding streams.]

[RAY: OK]

> SSRC vs. Session-based multiplexing is introduced in section 4.

Ok, for consistency we should likely move the pro and cons to the analysis section.

[RAY: OK]

> reason why it has less emphasis than the RTP-Mixer scenario?

If my understanding of tiled is correct, I don't see how this use case maps onto tiling. Please elaborate.

[RAY: see my comment below]

> conclusion relate to the draft-lennox-rtcweb-rtp-media-type-mux-00 
> which seems to suggest that SSRC multiplexing is not that difficult? 
> (I should note that I have no experience with sending multiple streams 
> inside a single RTP session, so it could be that I don't understand 
> the issues
> correctly)

If you review RTP Multiplexing architecture you are likely going realize that there are a number of cases where I consider SSRC multiplexed media streams to be the best and most appropriate choice. However, it is important that one consider the use case. So this document argues that for Simulcast the best choice is to have the simulcast version in different RTP sessions. However, if one have multiple media streams one should in fact use multiple SSRC in each of these sessions to carry the different media sources.

And in sessions where all media streams going to be sent are of the same type and same configuration then using SSRC multiplexing becomes easy.
The added twist in Lennox's draft is the multiple media types and that do bring a few issues, but not that many.
https://datatracker.ietf.org/doc/draft-westerlund-avtcore-multiplex-architecture/
and
https://datatracker.ietf.org/doc/draft-westerlund-avtcore-transport-multiplexing/
has discussion on that proposal.

The main point of the multiplexing architecture document is to make clear that one size fits all does not apply to using multiple SSRCs vs using several RTP sessions.

[RAY: Thanks for clearing that up]

> make things easier if these were aligned across multiple streams?

No, I don't believe in alignment. first of all the sequence numbers needs to progress with the packets actually sent for a media stream.
Encoding alternatives are quite likely to have different packet rates, especially for video as soon as one comes to a media bit-rate where some alternative requires multiple packets per video frame.

When it comes to timestamp I don't see any need either. For the best robustness and simplicity one should simply skip using alignment and instead use the mechanism for synchronizing media clocks with each other that exist, i.e. RTCP SR and Rapid Synchronization of RTP flows (RFC 6051).

[RAY: For the sequence numbers I agree, however about the timestamps I'm not so sure. In some cases it might be particularly useful to align timestamps. One example are areas where you need bit-level alignment (such as SVC and spatially segmented streams). For these situations NTP-based solutions such as RTCP SR are not accurate enough]

> in which the encoding of a media content can differ: bit-rate, codec 
> and sampling. Recent work in the area of immersive media has proposed 
> a fourth method: tiled video. With tiled video (or spatial 
> segmentation), the video resulting from a single camera is split into 
> a number of areas, each focusing on a particular spatial area of the 
> video (e.g. a single video source could be tiled into four separate 
> video streams, one describing the topleft quarter of the video and 
> three more describing the topright, bottomleft and bottomright quarters respectively).
> 

If I understand this correct, what you really are talking about are breaking up the output of one camer or several aligned cameras that can be considered to produce a single video image is divided into several separate streams. Then on the receiver side these are commonly one device per media stream. So a practical example would be to split the output of 4k camera into four 1080p media streams and on the receiver side use 4 1080 projectors that are aligned so the right edges matches.

If I am correct in my understanding that isn't simulcast. There are no two or more  different alternatives of the media source. From my perspective it looks like your splitting your single media source into multiple synchronized sources. However, it is clear that if you are doing Tiled video then you would like to consider the transport options for it. Because it depends if the receiver is actually a single end-point with 4 displays connected to them or if one actually have four different end-points and like to have each stream only being delivered to each end-point, not all of the others.

[RAY: Sorry for not being any clearer. We have a different understanding of tiled streaming (or spatial segmentation as is probably more descriptive). The use case is not having four network-connected projectors. The use case is having a mobile device with a relatively low resolution (e.g. 480x320 pixels) being able to navigate through high resolution (4k/8k) content. By splitting the output of a single 4k camera into 'tiles' of for example 240x160 pixels, and storing each of these as a separate stream, a client can request multiple streams simultaneously and reconstruct part of the original video (in this case it would request 4 'tiles'). When a user wants to navigate through the content (e.g. pan/zoom) it requests a different subset of tiles. 	
As you can see, what we have here is a large number of tiles (streams), coming from the same source, with different clients requesting different subsets of these tiles. In that aspect, it is not that dissimilar from some of the use cases you describe in your draft.] 

Cheers

Magnus Westerlund

This e-mail and its contents are subject to the DISCLAIMER at http://www.tno.nl/emaildisclaimer

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 4 Nov 2011 10:58:34 +0100

Ray,

I have removed all points where the is nothing to follow up. We will
take your comments into account when preparing an update.

On 2011-11-03 16:27, Brandenburg, R. (Ray) van wrote:
> -----Original Message----- From: Magnus Westerlund
> [mailto:[email-masked]] Sent: donderdag 3 november
> 2011 15:32 To: Brandenburg, R. (Ray) van Cc: [email-masked] Subject:
> Re: [AVTCORE] Comments on draft-westerlund-avtcore-rtp-simulcast-00
> 
>> 
>> -          Section 3: This section talks quite a bit about the
>> relation between Simulcast and Scalable Coding and in which
>> situations one of the two is more suitable. I'm not sure why this
>> is relevant. Simulcast and Scalable Coding might be two techniques
>> that in some cases can be used to solve the same problem, the layer
>> on which they do this is completely different (codec-level versus
>> transport level). The fact that Scalable Coding might, in some
>> situations, be a better solution than Simulcast does not mean that
>> Simulcast should not be standardized to handle those situations. I
>> understand that you've written this mainly from a video
>> conferencing perspective, in which Scalable Coding might be used
>> more often, but in a lot of situations, such as IPTV, Scalable
>> Coding is just not an option due to its inherent complexity. In
>> these situations Simulcast might be able to solve some real issues,
> much more appropriate than the other. And that is part of our
> arguments for why it should be defined for usage also, even though we
> have layered coding specified.
> 
> [RAY: I don't agree with you that layered coding is a transport-level
> thing (or maybe I don't understand your definition of layered coding.
> With layered coding I mean e.g. SVC). It might be that layered-coding
> requires some specific transport-level properties (such as
> alignment), however these are not unique to layered coding. Your
> draft (or specifically section 3) seems to imply that Simulcast can
> be considered an alternative to Scalable Coding and that you
> therefore have to argue why Simulcast is better in some situations.
> In my opinion, Scalable Coding is an alternative to creating multiple
> bitrates/resolutions (i.e. Adaptive Streaming) and not to Simulcast.
> My understanding is that Simulcast is a method to deliver those
> multiple bitrates, the same way it could be used to deliver Scalable
> Coding streams.]

Yes, SVC is one realization of layered encoding. And what I am getting
at is that SVC being a encoding technology do have specific transport
requirements to enable its functionality in some systems. For SVC this
is either a media aware network entity (MANE) that can remove layers
en-route. Or using multicast or other methods on the transport layers to
select the layers encoded.

I will not strongly disagree with your characterization in the second
half of the paragraph. I wouldn't use "adaptive streaming" as we are
interested in real-time conversational use cases, rather than content
delivery with somewhat relaxed source to receiver delay requirements.
But yes what we are discussing are methods for realization of adaptive
media delivery in point to multi-point scenarios.

>> 
>> 
>> 
>> -          Section 6.2: How do you suggest the sequence numbers and
>> RTP timestamps are handled between multiple alternative streams?
>> Would it make things easier if these were aligned across multiple
>> streams?
> 
> Encoding alternatives are quite likely to have different packet
> rates, especially for video as soon as one comes to a media bit-rate
> where some alternative requires multiple packets per video frame.
> 
> instead use the mechanism for synchronizing media clocks with each
> other that exist, i.e. RTCP SR and Rapid Synchronization of RTP flows
> (RFC 6051).
> 
> [RAY: For the sequence numbers I agree, however about the timestamps
> I'm not so sure. In some cases it might be particularly useful to
> align timestamps. One example are areas where you need bit-level
> alignment (such as SVC and spatially segmented streams). For these
> situations NTP-based solutions such as RTCP SR are not accurate
> enough]

I would not completely dismiss timestamp alignment, however it has on
multiple times become clear that mechanism that attempts to rely on more
of the basic and well defined behaviors of RTP header fields have
encountered issues or resulted in limitations with which RTP extensions
they can interop. So it is more to caution people here.

I don't quite see what you mean with bit-level alignment in the
discussion of timestamps. To my knowledge the SVC payload format is so
defined that this isn't an issue. In SST all layers are sent under a
single SSRC in a single timestamp space in one single media stream. Thus
common timeline is used for all NALUs in all layers. When using MST the
layers will be in differnt media streams in different RTP sessions. Here
using the same timestamp values or not is still dependent on them
mapping to the common sender clock represented by the NTP time. I think
RTP has no issues mapping multiple SSRC timestamp value with random
offset to the same time base clock to a granularity that is equivalent
to the the precision of the NTP values provided in the RTCP SR packets.
Taking care I think a precision better than 1/2^30 of a second i
possible. That is more than sufficient to align frames.

If you with bit-alignment means that you need to indicate that the
content of the packet is starting at the 10th line from the top and 240
pixels in, then I think using the timestamp for that is the wrong
choice. Instead the payload formats should use the timestamp to
correctly id the frame, and then use other explicit indicators for
off-set or relation.

> 
> [RAY: Sorry for not being any clearer. We have a different
> understanding of tiled streaming (or spatial segmentation as is
> probably more descriptive). The use case is not having four
> network-connected projectors. The use case is having a mobile device
> with a relatively low resolution (e.g. 480x320 pixels) being able to
> navigate through high resolution (4k/8k) content. By splitting the
> output of a single 4k camera into 'tiles' of for example 240x160
> pixels, and storing each of these as a separate stream, a client can
> request multiple streams simultaneously and reconstruct part of the
> original video (in this case it would request 4 'tiles'). When a user
> wants to navigate through the content (e.g. pan/zoom) it requests a
> different subset of tiles. As you can see, what we have here is a
> large number of tiles (streams), coming from the same source, with
> different clients requesting different subsets of these tiles. In
> that aspect, it is not that dissimilar from some of the use cases you
> describe in your draft.]

Thanks for the explanation, it was needed. I haven't heard of that usage
of tiled in the context of video before.

First of all, I think this is something new and we should discuss the
most appropriate way or possible ways to transport this. I don't think
this belongs in the RTP simulcast draft as the use cases are currently
defined. The reason is that in all the cases of the simulcast we are
discussing are that the alternative media streams tries to represent the
full source, although at different quality levels when it comes to SNR,
resolution etc. While tiled video is dealing with the selection and
handling of multiple partial representations of a source.

I think one way of dealing with tiled video is to make each tile
separate media stream, where each tile will use its own SSRC. Thus one
could potentially on SSRC level control if a stream is being delivered
or not. Another way of viewing the problem is to only use as many SSRCs
as you need simultaneously delivered tiles. Then you use something other
than SSRCs to indicate which tile is currently delivered in each media
stream.

I think this is an interesting problem, if you are interested in
standardizing a recommended way of handling tiled video you should
create a separate internet draft to discuss this problem.

Cheers

Magnus Westerlund