Subject: [clue] voice activated switching example for the Framework

From: "Duckworth, Mark" <[email-masked]>
Date: Tue, 6 May 2014 14:22:29 -0700

I had an action item from IETF89 to add an example to the Framework to show how an MCU can implement "voice activated switching" with endpoints that send multiple video streams, so the MCU can choose one to send to a simple endpoint that receives only one video stream.  To do this, we want to use a switched MCC from the multi-camera endpoints to the MCU to carry the active talker video from each endpoint.

This example also addresses, at least in some cases, the issue Steve brought up about cases where "talker identification does not work".  http://www.ietf.org/mail-archive/web/clue/current/msg03733.html

Here is my proposed text to add to the Framework, in the Examples (Informative) section:

================ begin proposed text ========================
12.3.4. Heterogeneous conference with voice activated switching

This example illustrates how multipoint "voice activated switching" behavior can be realized, with an endpoint making its own decision about which of its outgoing video streams is considered the "active talker" from that endpoint.  Then an MCU can decide which is the active talker among the whole conference.

Consider a conference between endpoints with the following characteristics:

   Endpoint A - 3 screens, 3 cameras
   Endpoint B - 3 screens, 3 cameras
   Endpoint C - 1 screen, 1 cameras

This example focuses on what the user at endpoint C sees.  The user would like to see the video capture of the current talker, without composing it with any other video capture.  In this example endpoint C is capable of receiving only a single video stream.  The following tables describe advertisements from A and B to the MCU, and from the MCU to C, that can be used to accomplish this.

     +-----------------------+---------------------------------+
     | Capture Scene #1      | Description=Endpoint x          |
     +-----------------------|---------------------------------+
     | VC1                   | CaptureArea=Left                |
     |                       | EncodingGroup=1                 |
     | VC2                   | CaptureArea=Center              |
     |                       | EncodingGroup=1                 |
     | VC3                   | CaptureArea=Right               |
     |                       | EncodingGroup=1                 |
     | MCC1(VC1,VC2,VC3)     | MaxCaptures=1                   |
     |                       | CaptureArea=whole scene         |
     |                       | Policy=SoundLevel:0             |
     |                       | EncodingGroup=1                 |
     | AC1                   | CaptureArea=whole scene         |
     |                       | EncodingGroup=2                 |
     | CSE1(VC1, VC2, VC3)   |                                 |
     | CSE2(MCC1)            |                                 |
     | CSE3(AC1)             |                                 |
     +---------------------------------------------------------+
Table 25: Advertisement received at the MCU from Endpoints A and B

Endpoints A and B are advertising each individual video capture, and also a switched capture MCC1 which switches between the other three based on who is the active talker.  These endpoints do not advertise distinct audio captures associated with each individual video capture, so it would be impossible for the MCU (as a media consumer) to make its own determination of which video capture is the active talker based just on information in the audio streams.

     +-----------------------+---------------------------------+
     | Capture Scene #1      | Description=conference          |
     +-----------------------|---------------------------------+
     | MCC1()                | CaptureArea=Left                |
     |                       | MaxCaptures=1                   |
     |                       | SynchronisationID=1             |
     |                       | Policy=SoundLevel:0             |
     |                       | EncodingGroup=1                 |
     |                       |                                 |
     | MCC2()                | CaptureArea=Center              |
     |                       | MaxCaptures=1                   |
     |                       | SynchronisationID=1             |
     |                       | Policy=SoundLevel:0             |
     |                       | EncodingGroup=1                 |
     |                       |                                 |
     | MCC3()                | CaptureArea=Right               |
     |                       | MaxCaptures=1                   |
     |                       | SynchronisationID=1             |
     |                       | Policy=SoundLevel:0             |
     |                       | EncodingGroup=1                 |
     |                       |                                 |
     | MCC4()                | CaptureArea=whole scene         |
     |                       | MaxCaptures=1                   |
     |                       | Policy=SoundLevel:0             |
     |                       | EncodingGroup=1                 |
     |                       |                                 |
     | MCC5() (for audio)    | CaptureArea=whole scene         |
     |                       | MaxCaptures=1                   |
     |                       | Policy=SoundLevel:0             |
     |                       | EncodingGroup=2                 |
     |                       |                                 |
     | MCC6() (for audio)    | CaptureArea=whole scene         |
     |                       | MaxCaptures=1                   |
     |                       | Policy=SoundLevel:1             |
     |                       | EncodingGroup=2                 |
     | CSE1(MCC1,MCC2,MCC3   |                                 |
     | CSE2(MCC4)            |                                 |
     | CSE3(MCC5,MCC6)       |                                 |
     +---------------------------------------------------------+
         Table 26: Advertisement sent from the MCU to C

The MCU advertises one scene, with four video MCCs.  Three of them in CSE1 give a left, center, right view of the conference, with "site switching". MCC4 provides a single video capture representing a view of the whole conference, which is simply switched between all the other original source captures.

Endpoint C, in its configure message to the MCU, requests to receive MCC4.  In order for the MCU to get the information it needs to construct MCC4, it has to send configure messages to A and B asking to receive MCC1 from each of them, along with their AC1 audio.  Now the MCU can use audio energy information from the two incoming audio streams from A and B to determine which of those alternatives is the current talker.  Based on that, the MCU uses either MCC1 from A or MCC1 from B as the source of MCC4 to send to C.
================ end proposed text =========================

---

From: Christian Groves <[email-masked]>
Date: Fri, 09 May 2014 13:05:51 +1000

Hello Mark,

Please see my comments below.

Regards, Christian

>     Endpoint C - 1 screen, 1 cameras
[CNG] Editorial nit: 1 camera<s> delete
> This example focuses on what the user at endpoint C sees.  The user would like to see the video capture of the current talker, without composing it with any other video capture.  In this example en…
[CNG] According to the above table from a MC perspective it wouldn't 
know which sources are included. Nothing about the original sources has 
been advertised.
There's no description of MCC5 and MCC6 and CSE3? Does the MCU need to 
send two Audio MCCs?

>
> Endpoint C, in its configure message to the MCU, requests to receive MCC4.  In order for the MCU to get the
[CNG] And an audio capture/s?
> information it needs to construct MCC4, it has to send configure messages to A and B asking to receive MCC1 from each of them, along with their AC1 audio.  Now the MCU can use audio energy information from the two incoming audio streams from A and B to determine which of those alternatives is the current talker.  Based on that, the MCU uses either MCC1 from A or MCC1 from B as the source of MCC4 to send to C.
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 9 May 2014 14:51:00 -0700

Hi Christian,
thanks for the feedback.  more below.
Mark

> -----Original Message-----
> From: clue [mailto:[email-masked]] On Behalf Of Christian Groves
> Sent: Thursday, May 08, 2014 11:06 PM
> To: [email-masked]
> Subject: Re: [clue] voice activated switching example for the Framework
> 
> Hello Mark,
> 
> > I had an action item from IETF89 to add an example to the Framework to
> show how an MCU can implement "voice activated switching" with endpoints
> that send multiple video streams, so the MCU can choose one to send to a
> simple endpoint that receives only one video stream.  To do this, we want to
> use a switched MCC from the multi-camera endpoints to the MCU to carry
> the active talker video from each endpoint.
> >
> > This example also addresses, at least in some cases, the issue Steve
> > brought up about cases where "talker identification does not work".
> > http://www.ietf.org/mail-archive/web/clue/current/msg03733.html
> >
> 12.3.4.
> > Heterogeneous conference with voice activated switching
> >
> > This example illustrates how multipoint "voice activated switching"
> behavior can be realized, with an endpoint making its own decision about
> which of its outgoing video streams is considered the "active talker" from
> that endpoint.  Then an MCU can decide which is the active talker among the
> whole conference.
> >
> > This example focuses on what the user at endpoint C sees.  The user would
> like to see the video capture of the current talker, without composing it with
> any other video capture.  In this example endpoint C is capable of receiving
> only a single video stream.  The following tables describe advertisements
> from A and B to the MCU, and from the MCU to C, that can be used to
> accomplish this.
> >
> > Endpoints A and B are advertising each individual video capture, and also a
> switched capture MCC1 which switches between the other three based on
> who is the active talker.  These endpoints do not advertise distinct audio
> captures associated with each individual video capture, so it would be
> impossible for the MCU (as a media consumer) to make its own
> determination of which video capture is the active talker based just on
> information in the audio streams.
> >
> > The MCU advertises one scene, with four video MCCs.  Three of them in
> CSE1 give a left, center, right view of the conference, with "site switching".
> MCC4 provides a single video capture representing a view of the whole
> conference, which is simply switched between all the other original source
> captures.
> [CNG] According to the above table from a MC perspective it wouldn't know
> which sources are included. Nothing about the original sources has been
> advertised.

[Duckworth, Mark] Correct.  That is allowed according to the definition of an MCC.
I want to keep this example as simple as possible to illustrate the point, and it isn't
necessary to include all those other captures without encoding groups in order to do that.

> There's no description of MCC5 and MCC6 and CSE3? Does the MCU need to

 [Duckworth, Mark] MCC5 and MCC6 have a bunch of attributes describing them, so I'm not sure what you mean?
Maybe the MCU should advertise:
CSE3(MCC5, MCC6)
CSE4(MCC5)
This would make it clear the advertisement suggests MCC5 if the consumer wants just one audio capture.

> [CNG] And an audio capture/s?

[Duckworth, Mark] Yes, I will add that for completeness in the text.

> > information it needs to construct MCC4, it has to send configure messages
> to A and B asking to receive MCC1 from each of them, along with their AC1
> audio.  Now the MCU can use audio energy information from the two
> incoming audio streams from A and B to determine which of those
> alternatives is the current talker.  Based on that, the MCU uses either MCC1
> from A or MCC1 from B as the source of MCC4 to send to C.

---

From: Christian Groves <[email-masked]>
Date: Mon, 12 May 2014 12:28:43 +1000

Hello Mark,

Please see below.

Regards, Christian

> necessary to include all those other captures without encoding groups in order to do that.
[CNG] That's fine, however I think its worth to add a note that the 
source captures would be sent but not included for brevity reasons. 
Otherwise the MC doesn't know that they are simply switched between all 
the other original source captures."
> This would make it clear the advertisement suggests MCC5 if the consumer wants just one audio capture.
[CNG] I mean there's no text procedural description saying what these 
are. The video MCCs also have attributes but you describe the video in 
the paragraph below the table.

---

From: "Duckworth, Mark" <[email-masked]>
Date: Mon, 12 May 2014 05:41:32 -0700

Hi Christian,
More below.
Mark

> From: Christian Groves [mailto:[email-masked]]
> Sent: Sunday, May 11, 2014 10:29 PM
> To: Duckworth, Mark; [email-masked]
> >> show how an MCU can implement "voice activated switching" with
> >> endpoints that send multiple video streams, so the MCU can choose one
> >> to send to a simple endpoint that receives only one video stream.  To
> >> do this, we want to use a switched MCC from the multi-camera
> >> endpoints to the MCU to carry the active talker video from each endpoint.
> >> behavior can be realized, with an endpoint making its own decision
> >> about which of its outgoing video streams is considered the "active
> >> talker" from that endpoint.  Then an MCU can decide which is the
> >> active talker among the whole conference.
> >> like to see the video capture of the current talker, without
> >> composing it with any other video capture.  In this example endpoint
> >> C is capable of receiving only a single video stream.  The following
> >> tables describe advertisements from A and B to the MCU, and from the
> >> MCU to C, that can be used to accomplish this.
> >> who is the active talker.  These endpoints do not advertise distinct
> >> audio captures associated with each individual video capture, so it
> >> would be impossible for the MCU (as a media consumer) to make its own
> >> determination of which video capture is the active talker based just
> >> on information in the audio streams.
> >> conference, which is simply switched between all the other original
> >> source captures.
> >> know which sources are included. Nothing about the original sources
> >> has been advertised.
> > I want to keep this example as simple as possible to illustrate the
> > point, and it isn't necessary to include all those other captures without
> encoding groups in order to do that.
> [CNG] That's fine, however I think its worth to add a note that the source
> captures would be sent but not included for brevity reasons.

[Duckworth, Mark] The MCU, as a provider, certainly could advertise information about all the other endpoint scenes, but it doesn't have to.  My intent with this particular example is the MCU does not advertise all those other scenes.  I think the example works fine without it.  The endpoints, as consumers, do not *need* to know all that.  Possibly if that information was included in the MCUs advertisement it could enable other behavior, but that's not what this example is about.

> >> There's no description of MCC5 and MCC6 and CSE3? Does the MCU need
> >> to send two Audio MCCs?
> [CNG] I mean there's no text procedural description saying what these are.
> The video MCCs also have attributes but you describe the video in the
> paragraph below the table.

[Duckworth, Mark] I understand.  I can add a sentence or two.

---

From: Christian Groves <[email-masked]>
Date: Tue, 13 May 2014 09:56:55 +1000

Hello Mark,

Please see below.

Regards, Christian
..snip..
> [Duckworth, Mark] The MCU, as a provider, certainly could advertise information about all the other endpoint scenes, but it doesn't have to.  My intent with this particular example is the MCU does…
[CNG] I agree the MCU as the provider can provide what ever it wants. 
The point is with the current example the text says "MCC4 provides a 
single video capture representing a view of the whole conference, which 
is simply switched between all the other original source captures." The 
MP may know this but all the MC knows from the Advertisement (without 
the sources) is that MCC4 is a switched capture. It doesn't know that 
its "switched between all the other original sources". There's the 
intent of the MP and what is actually advertised.

---

From: "Duckworth, Mark" <[email-masked]>
Date: Mon, 12 May 2014 17:25:07 -0700

Hi Christian,
Thanks for clarifying, please see below.
Mark

> Sent: Monday, May 12, 2014 7:57 PM
> >>>> MCC4 provides a single video capture representing a view of the
> >>>> whole conference, which is simply switched between all the other
> >>>> original source captures.
> >>>> [CNG] According to the above table from a MC perspective it
> >>>> wouldn't know which sources are included. Nothing about the
> >>>> original sources has been advertised.
> >>> [Duckworth, Mark] Correct.  That is allowed according to the
> >>> definition of
> >> an MCC.
> >> Otherwise the MC doesn't know that they are simply switched between
> >> all the other original source captures."
> > [Duckworth, Mark] The MCU, as a provider, certainly could advertise
> information about all the other endpoint scenes, but it doesn't have to.  My
> intent with this particular example is the MCU does not advertise all those
> other scenes.  I think the example works fine without it.  The endpoints, as
> consumers, do not *need* to know all that.  Possibly if that information was
> included in the MCUs advertisement it could enable other behavior, but
> that's not what this example is about.
> >
> The point is with the current example the text says "MCC4 provides a single
> video capture representing a view of the whole conference, which is simply
> switched between all the other original source captures." The MP may know
> this but all the MC knows from the Advertisement (without the sources) is
> that MCC4 is a switched capture. It doesn't know that its "switched between
> all the other original sources". There's the intent of the MP and what is
> actually advertised.
[Duckworth, Mark] You are right.  I will clarify that in the text, about what the MCU MP is doing, vs. how much information it gives in the advertisement.  Do you agree the example still works even though the endpoint consumer does not know MCC4 is "simply switched between all the other original source captures"?