Subject: Re: [clue] Example of MCC - consumer driven layout and switching

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 11 Oct 2013 08:57:25 -0700

I'm branching this topic off the MCC discussion, keeping just a small section of previous email.

I have another suggestion to consider about making media switching decisions - see below.  Has anybody else considered an idea like this?

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Robert Hansen
> Sent: Monday, September 30, 2013 12:43 PM
> To: [email-masked]
> Subject: Re: [clue] Example of MCC
>
> > On 28/09/2013 1:04 AM, Duckworth, Mark wrote:
> >> One downside to this approach is that every change the MCU makes in
> >> how it assigns contributing VCs to an MCC (e.g. make a switch based
> >> on current talker) could result in the consumer renderer needing to
> >> change how it maps decoder output to display locations.  I need to
> >> think more about if this would really be a problem or not.
> 
> [RH] My issue with using the spatial coordinates of the MCC is that it requires
> the MCU to anticipate the rendering desires of the endpoint, as the MCU will
> need to advertise MCCs with spatial coordinates that allow the endpoint to
> do what it wants to do. My concern has always been that this may lead to
> huge numbers of MCCs and capture scene entries if the MCU is trying to
> anticipate potential layouts by the endpoint.

I agree it is good if the MCU does not have to anticipate the rendering (layout) desires of the endpoint consumer.  So I consider if it is better if the consumer can make even more of the decisions, including dynamically selecting which source capture should be included in an MCC.  draft-groves-clue-multi-content-00 says "The Consumer may or may not  want to receive all the indicated captures.  Therefore it can choose to receive a sub-set of captures indicated by the MCC."  This means the consumer can choose, in a configure message, to receive the one particular capture it wants at that moment, for each MCC.  And it can send new configure messages any time, to change this choice.  I think this would not require any SDP exchanges.

If the consumer makes these choices (which still must be consistent with the advertisement), then the provider doesn't have to make any switching choices any more.  The consumer can make its choices based on its own local policy for layout and rendering.

But the consumer would need some more information about current activity in the conference in order to make these decisions.  The consumer would need to know about audio activity, for example in the form of a "most recent talker list" or something like that, in order to make the decisions about which MC it wants to receive within each MCC.  I don't know of any existing standard way to convey that information.  It seems to me a provider could send a "most recent talker list" to a consumer via the CLUE channel.  Is anybody interested in pursuing this idea?

Mark

---

From: Christian Groves <[email-masked]>
Date: Tue, 22 Oct 2013 17:21:08 +1100

Hello Mark,

I guess I saw this slightly differently. Rather than chose the MC based 
on voice activity I saw that a consumer would chose the MC in the MCC 
based on the sources it wanted. It would then choose an applicable 
policy (i.e. most active, two most active, etc). The MCU would provide 
the appropriate MC based on this policy.

For the "most recent talker list" it sounds like a subscribe/notify 
mechanism where the Consumer would subscribe to events and then be 
notified of it. Is this the sort of thing that you were thinking of?

Regards, Christian

On 12/10/2013 2:57 AM, Duckworth, Mark wrote:
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: "Duckworth, Mark" <[email-masked]>
Date: Tue, 22 Oct 2013 03:13:57 -0700

Hi Christian,

I understand the scenario you are talking about, where the MCU chooses the MC for the MCC based on some policy. But I think it leads to some potential problems.  Like we said below, the MCU has to anticipate the rendering desires of the receiving endpoint, at least to some extent.  And also the receiving endpoint might have to remap a video decoder to a different rendering location every time the MCU makes a switch.  This seems like a problem for an endpoint that has particular decoders hardwired to different displays.  I think these problems can be resolved if the receiver makes more explicit choices about what it wants to receive in each MCC.

Yes, I was thinking of something like a subscribe/notify mechanism for a "recent talker list".  I'm not sure it's really needed though, maybe instead the receiver can get this information from the audio streams, if the MCU sends something like the N most active audio streams.

Mark

> Christian Groves
> Sent: Tuesday, October 22, 2013 2:21 AM
> To: [email-masked]
> Subject: Re: [clue] Example of MCC - consumer driven layout and switching
> 
> Hello Mark,
> 
> I guess I saw this slightly differently. Rather than chose the MC based on
> voice activity I saw that a consumer would chose the MC in the MCC based
> on the sources it wanted. It would then choose an applicable policy (i.e. most
> active, two most active, etc). The MCU would provide the appropriate MC
> based on this policy.
> 
> For the "most recent talker list" it sounds like a subscribe/notify mechanism
> where the Consumer would subscribe to events and then be notified of it. Is
> this the sort of thing that you were thinking of?
> 
> Regards, Christian
> 
> > I have another suggestion to consider about making media switching
> decisions - see below.  Has anybody else considered an idea like this?
> >
> >> From: [email-masked] [mailto:[email-masked]] On Behalf
> >> Of Robert Hansen
> >> [RH] My issue with using the spatial coordinates of the MCC is that
> >> it requires the MCU to anticipate the rendering desires of the
> >> endpoint, as the MCU will need to advertise MCCs with spatial
> >> coordinates that allow the endpoint to do what it wants to do. My
> >> concern has always been that this may lead to huge numbers of MCCs
> >> and capture scene entries if the MCU is trying to anticipate potential
> layouts by the endpoint.
> > I agree it is good if the MCU does not have to anticipate the rendering
> (layout) desires of the endpoint consumer.  So I consider if it is better if the
> consumer can make even more of the decisions, including dynamically
> selecting which source capture should be included in an MCC.  draft-groves-
> clue-multi-content-00 says "The Consumer may or may not  want to receive
> all the indicated captures.  Therefore it can choose to receive a sub-set of
> captures indicated by the MCC."  This means the consumer can choose, in a
> configure message, to receive the one particular capture it wants at that
> moment, for each MCC.  And it can send new configure messages any time,
> to change this choice.  I think this would not require any SDP exchanges.
> >
> > If the consumer makes these choices (which still must be consistent with
> the advertisement), then the provider doesn't have to make any switching
> choices any more.  The consumer can make its choices based on its own local
> policy for layout and rendering.
> >
> > But the consumer would need some more information about current
> activity in the conference in order to make these decisions.  The consumer
> would need to know about audio activity, for example in the form of a "most
> recent talker list" or something like that, in order to make the decisions about
> which MC it wants to receive within each MCC.  I don't know of any existing
> standard way to convey that information.  It seems to me a provider could
> send a "most recent talker list" to a consumer via the CLUE channel.  Is
> anybody interested in pursuing this idea?
> >
> > Mark

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 23 Oct 2013 13:30:04 -0400

Mark,

IIUC you are suggesting that the consumer might be making decisions and 
sending configure messages about the captures it wants to receive every 
time the active speaker changes.

And it requires that the active speaker mapping to source capture id be 
available to the consumer

Isn't it unrealistic to think that this could work well at that time scale?

	Thanks,
	Paul

---

From: "Duckworth, Mark" <[email-masked]>
Date: Mon, 28 Oct 2013 15:57:27 -0700

Hi Paul,

Yes, you understand correctly. Yes, it requires the consumer to know (or be able to figure out) which video streams are associated with the active speaker.  I think this is needed anyway, for other things (lip sync, spatial rendering, UI indications). I think it is realistic to do it this way.  I intend to discuss more with my colleagues, who have more experience in this aspect of media switching.

Are you concerned about the frequency of these decisions being a problem? Or about the latency for each individual decision? I think latency is more likely to be a problem.

Mark

> Paul Kyzivat
> Sent: Wednesday, October 23, 2013 1:30 PM
> To: [email-masked]
> > I understand the scenario you are talking about, where the MCU chooses
> the MC for the MCC based on some policy. But I think it leads to some
> potential problems.  Like we said below, the MCU has to anticipate the
> rendering desires of the receiving endpoint, at least to some extent.  And
> also the receiving endpoint might have to remap a video decoder to a
> different rendering location every time the MCU makes a switch.  This seems
> like a problem for an endpoint that has particular decoders hardwired to
> different displays.  I think these problems can be resolved if the receiver
> makes more explicit choices about what it wants to receive in each MCC.
> >
> > Yes, I was thinking of something like a subscribe/notify mechanism for a
> "recent talker list".  I'm not sure it's really needed though, maybe instead the
> receiver can get this information from the audio streams, if the MCU sends
> something like the N most active audio streams.
> >
> > Mark
> >
> >> Of Christian Groves
> >> I guess I saw this slightly differently. Rather than chose the MC
> >> based on voice activity I saw that a consumer would chose the MC in
> >> the MCC based on the sources it wanted. It would then choose an
> >> applicable policy (i.e. most active, two most active, etc). The MCU
> >> would provide the appropriate MC based on this policy.
> >>
> >>> I'm branching this topic off the MCC discussion, keeping just a
> >>> small section
> >> of previous email.
> >>>
> >>>> From: [email-masked] [mailto:[email-masked]] On
> >>>> Behalf Of Robert Hansen
> >>>>>> One downside to this approach is that every change the MCU makes
> >>>>>> in how it assigns contributing VCs to an MCC (e.g. make a switch
> >>>>>> based on current talker) could result in the consumer renderer
> >>>>>> needing to change how it maps decoder output to display
> >>>>>> locations.  I need to think more about if this would really be a
> problem or not.
> >> (layout) desires of the endpoint consumer.  So I consider if it is
> >> better if the consumer can make even more of the decisions, including
> >> dynamically selecting which source capture should be included in an
> >> MCC.  draft-groves-
> >> clue-multi-content-00 says "The Consumer may or may not  want to
> >> receive all the indicated captures.  Therefore it can choose to
> >> receive a sub-set of captures indicated by the MCC."  This means the
> >> consumer can choose, in a configure message, to receive the one
> >> particular capture it wants at that moment, for each MCC.  And it can
> >> send new configure messages any time, to change this choice.  I think this
> would not require any SDP exchanges.
> >>>
> >> the advertisement), then the provider doesn't have to make any
> >> switching choices any more.  The consumer can make its choices based
> >> on its own local policy for layout and rendering.
> >>>
> >> activity in the conference in order to make these decisions.  The
> >> consumer would need to know about audio activity, for example in the
> >> form of a "most recent talker list" or something like that, in order
> >> to make the decisions about which MC it wants to receive within each
> >> MCC.  I don't know of any existing standard way to convey that
> >> information.  It seems to me a provider could send a "most recent
> >> talker list" to a consumer via the CLUE channel.  Is anybody interested in
> pursuing this idea?
> >>>
> >>> Mark

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 30 Oct 2013 09:42:34 -0400

On 10/28/13 6:57 PM, Duckworth, Mark wrote:
> Hi Paul,
>
> Yes, you understand correctly. Yes, it requires the consumer to know (or be able to figure out) which video streams are associated with the active speaker.

OK.

> I think this is needed anyway, for other things (lip sync, spatial rendering, UI indications). I think it is realistic to do it this way.

Then I think we do need to say more about *how* this is done.
I suppose the mechanisms for lip sync could be used to correlate audio 
and video to some extent. But I don't think that is enough to figure out 
which display holds loudest speaker. Maybe spatial info could help. But 
we have had input that the spatial info for audio has a much different 
meaning than spatial info for video, so correlating them could be difficult.

> I intend to discuss more with my colleagues, who have more experience in this aspect of media switching.

And maybe others here can speak to that too.

> Are you concerned about the frequency of these decisions being a problem? Or about the latency for each individual decision? I think latency is more likely to be a problem.

Both. But it is only "concern". I don't have enough knowledge to decide 
if my concern is justified.

	Thanks,
	Paul