Subject: [clue] Need to signal "active video capture" information?

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 19 Feb 2014 14:44:16 -0800

I had this in a slide from Feb 11 design team, but we didn't get to it.
This is an old issue we previously said was important, but haven't followed up on it.  I think it is still important.

Old slides from October 2011 interim - http://trac.tools.ietf.org/wg/clue/trac/raw-attachment/wiki/WikiStart/5-VAD-in-CLUE.ppt

Summary:
*       Signaling the "active video capture", not using just audio energy in the audio stream(s).  Discussed at October 2011 interim meeting, we never followed up on it. "VAD in CLUE"
*       Example: endpoint provider sends 3 video captures, and 1 mono audio
-      Consumer (MCU topo-mixer) wants to know which video has the loudest talker, so it can send that one to a simple endpoint that just receives one video stream
-      How does this MCU consumer know which one it is?
*       Provider can signal this somehow, if it knows locally which video is associated with the talker
-      in RTP?
-      in CLUE channel?
-      provider sends a switched MCC for this, plus the individual captures?
*       This gets back to issue of sending one packet stream that fulfills multiple encodings at the same time?
-      other way?

Does the group want to address this issue?

Mark

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 20 Feb 2014 14:28:25 -0500

Speaking as an individual, I would like to see a way to address this.

But I'm not (yet anyway) confident that there is a mechanism that would 
be sufficiently useful and also practical for us to specify. And if not, 
then I don't want to spend a lot of time flogging it.

I would be happy if *somebody* came back with a concrete proposal, with 
explanation of utility and feasibility. Or with several alternatives and 
the pros/cons of each. This in enough detail that we could quickly 
choose, or not.

	Thanks,
	Paul

On 2/19/14 5:44 PM, Duckworth, Mark wrote:
> I had this in a slide from Feb 11 design team, but we didn’t get to it.
>
> This is an old issue we previously said was important, but haven’t
> followed up on it.  I think it is still important.
>
> Old slides from October 2011 interim -
> http://trac.tools.ietf.org/wg/clue/trac/raw-attachment/wiki/WikiStart/5-VAD-in-CLUE.ppt
>
> Summary:
>
> •Signaling the “active video capture”, not using just audio energy in
> the audio stream(s).  Discussed at October 2011 interim meeting, we
> never followed up on it. “VAD in CLUE”
>
> •Example: endpoint provider sends 3 video captures, and 1 mono audio
>
> –Consumer (MCU topo-mixer) wants to know which video has the loudest
> talker, so it can send that one to a simple endpoint that just receives
> one video stream
>
> –How does this MCU consumer know which one it is?
>
> •Provider can signal this somehow, if it knows locally which video is
> associated with the talker
>
> –in RTP?
>
> –in CLUE channel?
>
> –provider sends a switched MCC for this, plus the individual captures?
>
> •This gets back to issue of sending one packet stream that fulfills
> multiple encodings at the same time?
>
> –other way?
>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: Christian Groves <[email-masked]>
Date: Fri, 21 Feb 2014 16:36:41 +1100

Hello,

+1 to Paul's comments.

Regards,
Christian

On 21/02/2014 6:28 AM, Paul Kyzivat wrote:
> But I'm not (yet anyway) confident that there is a mechanism that 
> would be sufficiently useful and also practical for us to specify. And 
> if not, then I don't want to spend a lot of time flogging it.
>
> I would be happy if *somebody* came back with a concrete proposal, 
> with explanation of utility and feasibility. Or with several 
> alternatives and the pros/cons of each. This in enough detail that we 
> could quickly choose, or not.
>
>     Thanks,
>     Paul
>

---

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 21 Feb 2014 09:11:46 -0800

Here is a high level suggestion for how this could be done, using (mostly) existing CLUE mechanisms.  I think if we eventually have a solution for the Media-9 requirement from draft-ietf-clue-rtp-mapping, then it would work well.  It could even work without Media-9, but with extra overhead of sending duplicate packet streams.

Scenario - multipoint conference, with middlebox MCU, including some endpoints with multiple cameras, some endpoints receiving multiple video streams, some endpoints receiving only single video stream.  When a multi-camera endpoint is the active talker, the multi stream receivers want to receive video from all the talker's cameras, but the single stream receiver wants to receive video from only the camera with the active talker.

- The multi camera endpoint provider can offer multiple video CSEs, one with all the camera captures, and one with a single switched MCC representing the active talker.
- The MCU consumer receives all these captures.
- The MCU consumer now has all the information it needs, in order to provide captures to other multi-stream receivers, and to the single stream receivers

Example advertisement from multi camera endpoint:
VC1 left
VC2 center
VC3 right
AC1 mono audio
MCC1(VC1,VC2,VC3) maxCaptures=1; policy=soundlevel:0
CSE1(VC1,VC2,VC3)
CSE2(AC1)
CSE3(MCC1)

>From draft-ietf-clue-rtp-mapping-01:
Media-9: If a given source is being sent on the same transport flow
for more than one reason (e.g. if it corresponds to more than one
switched capture at once, or to a static capture), it should be
possible for a sender to send only one copy of the source.

This seems like a clear case where we would want to use this Media-9 requirement so the provider can send just one copy of an encoding for each of VC1, VC2, and VC3 and somehow indicate which one of those is fulfilling MCC1 at any point in time.  This also seems to imply there should be a way for the provider to advertise only three possible encodings, yet still allow the consumer to request all four captures.  I don't think we have a way to do that yet.

Another possibility is the provider could offer enough encodings so the MCU could request a separate distinct encoding for MCC1, for example requesting VC1, VC2, and VC3 at 1080p while requesting MCC1 at 720p.  This case would not require a solution to Media-9.

Any comments?

Mark

> -----Original Message-----
> From: clue [mailto:[email-masked]] On Behalf Of Christian Groves
> Sent: Friday, February 21, 2014 12:37 AM
> To: [email-masked]
> Subject: Re: [clue] Need to signal "active video capture" information?
> 
> Hello,
> 
> +1 to Paul's comments.
> 
> Regards,
> Christian
> 
> >> This is an old issue we previously said was important, but haven't
> >> *Signaling the "active video capture", not using just audio energy in
> >> never followed up on it. "VAD in CLUE"
> >>
> >> *Example: endpoint provider sends 3 video captures, and 1 mono audio
> >>
> >> -Consumer (MCU topo-mixer) wants to know which video has the
> loudest
> >> talker, so it can send that one to a simple endpoint that just
> >> receives one video stream
> >>
> >> -How does this MCU consumer know which one it is?
> >>
> >> *Provider can signal this somehow, if it knows locally which video is
> >> -in RTP?
> >> -in CLUE channel?
> >> -provider sends a switched MCC for this, plus the individual captures?
> >>
> >> *This gets back to issue of sending one packet stream that fulfills
> >> -other way?