Subject: Re: [clue] Kick-off Item 2: Layout, composition, switching discussion

From: "Duckworth, Mark" <[email-masked]>
Date: Tue, 1 Nov 2011 10:35:07 -0700

Hi Roni,

The two topics you listed are items 3 and 4 from my message.

I think your item 1 and my item 3 are about source selection, which needs a discussion to include selecting sources for a mix, and also selecting sources a consumer wants to receive individually (not part of a mix).  Jonathan's presentation from the Andover meeting (13-CLUE-Interim-Sources.ppt) raised some good points about this.  Do you agree this all falls under the same topic?

I think your item 2 and my item 4 are the same topic.  Are you talking about switching streams, or composing streams, or both?  Do you see these switching modes as something a particular consumer can request from a provider, that will not impact other consumers? Or is it more like a conference-wide mode? (this gets into Paul's point as well).
My concern about this is I don't see how to bound it.  Are you suggesting we should define a few specific composing/switching modes that a provider can enumerate, and a way for a consumer to choose one?

Mark

From: Roni Even [mailto:[email-masked]]
Sent: Thursday, October 27, 2011 5:55 PM
To: Duckworth, Mark; [email-masked]
Subject: RE: [clue] Kick-off Item 2: Layout, composition, switching discussion

Hi Mark,
I understand that you have two topics

1.       Allowing the consumer to select specific media streams for a mix (for example, left camera from room 1, middle camera from room 2 and left camera from room 3) which I think should be possible. This will require information and protocol that will allow this functionality.

2.       Allowing the provider  to specify the mixture algorithm and for the consumer to ask for a specific one. I think that we need at least to enable the provider and consumer to specify or request site switch or segment switch mode.

Roni

From: [email-masked] [mailto:[email-masked]] On Behalf Of Duckworth, Mark
Sent: Thursday, October 27, 2011 5:30 PM
To: [email-masked]
Subject: [clue] Kick-off Item 2: Layout, composition, switching discussion

This is following up on the major issues list from the interim meeting in Andover.
Issues for discussion:

1.       Item 5 video composition (discussion kicked off by Stephan) is a part of this.  But more generally, the same sort of issue applies to other media types, not just video.

a.       For example, should a provider that mixes audio be able to indicate to the consumer which audio sources are in the mix?

b.      Proposal - the more general case should be consistent with the outcome of the Item 5 discussio

2.       Should a provider that mixes or composes other media captures/streams into a new stream be able to indicate some information about the algorithm it uses to choose which other captures/streams to mix?

a.       Proposal - no, CLUE does not need this.

3.       Should a provider that mixes or composes media be able to allow the consumer to make a choice about which other captures/streams are included in the mix?

a.       This should be covered by Major Issue Item 3 source selection - (discussion to be kicked off by Jonathan)

4.       Do we need support in CLUE for allowing a media consumer to request the provider to compose or mix particular media sources into a new media stream, using a particular algorithm?

a.       Proposal - no, CLUE does not need this.

b.      But if something like item 5 idea is accepted, then maybe a provider can offer a few alternatives and a consumer can choose among them

Mark Duckworth

---

From: Roni Even <[email-masked]>
Date: Wed, 02 Nov 2011 17:10:03 +0200

Hi Mark,

The reason I shorted the list is because if we want item 4 support than item
2 is needed to let the consumer know what algorithms can be used. 

As for item 1 I do not understand why we are even discussing it. In current
RTP you know the SSRCs of the mixed sources as they are part of CSRC, so for
CLUE we need to verify that this is not broken. The change in the mix should
be reflected in RTCP but we will need the association to the CLUE
information.

My view is that we need to support  both switching and mixing (so switching
here is a case of mixing). The issue of conference-wide mode or per user is
application or implementation specific. The provider can decide how to
respond to a request based on its capabilities and policies.

As for enumerating the cases.  There is more than one dimension. I think
that we can look at general policy (site switch, segment switch), per
receive stream (Voice activated, round robin). The usage can be automated,
so he consumer asks for the mode and the actual content is decided by the
producer based on the mode (Example segment switch, stream 1 voice
activated, stream 2 round robin- this will send in stream 1 the current
speaker and in stream 2 go through the rest of the streams (all rooms and
all cameras) in round robin).

If the consumer will want to build a specific mix he should be able to ask
for a specific streams for example in the previous example the consumer can
ask also for a third stream that will have the Manager TP system middle
camera.

I think that it will be good in the first stage to allow the producer to
provide the information about the modes (segment switch, site switch, voice
activate,..) and to allow the consumer to identify who is in the mix/switch.
The protocol from the consumer to provider can be a second stage

Roni

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Duckworth, Mark
Sent: Tuesday, November 01, 2011 7:35 PM
To: [email-masked]
Subject: Re: [clue] Kick-off Item 2: Layout, composition, switching
discussion

Hi Roni,

I think your item 1 and my item 3 are about source selection, which needs a
discussion to include selecting sources for a mix, and also selecting
sources a consumer wants to receive individually (not part of a mix).
Jonathan's presentation from the Andover meeting
(13-CLUE-Interim-Sources.ppt) raised some good points about this.  Do you
agree this all falls under the same topic?

I think your item 2 and my item 4 are the same topic.  Are you talking about
switching streams, or composing streams, or both?  Do you see these
switching modes as something a particular consumer can request from a
provider, that will not impact other consumers? Or is it more like a
conference-wide mode? (this gets into Paul's point as well).

My concern about this is I don't see how to bound it.  Are you suggesting we
should define a few specific composing/switching modes that a provider can
enumerate, and a way for a consumer to choose one?

Mark

From: Roni Even [mailto:[email-masked]] 
Sent: Thursday, October 27, 2011 5:55 PM
To: Duckworth, Mark; [email-masked]
Subject: RE: [clue] Kick-off Item 2: Layout, composition, switching
discussion

Hi Mark,

1.       Allowing the consumer to select specific media streams for a mix
(for example, left camera from room 1, middle camera from room 2 and left
camera from room 3) which I think should be possible. This will require
information and protocol that will allow this functionality.

2.       Allowing the provider  to specify the mixture algorithm and for the
consumer to ask for a specific one. I think that we need at least to enable
the provider and consumer to specify or request site switch or segment
switch mode.

Roni

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Duckworth, Mark
Sent: Thursday, October 27, 2011 5:30 PM
To: [email-masked]
Subject: [clue] Kick-off Item 2: Layout, composition, switching discussion

This is following up on the major issues list from the interim meeting in
Andover.

Issues for discussion:

1.       Item 5 video composition (discussion kicked off by Stephan) is a
part of this.  But more generally, the same sort of issue applies to other
media types, not just video.

a.       For example, should a provider that mixes audio be able to indicate
to the consumer which audio sources are in the mix?

b.      Proposal - the more general case should be consistent with the
outcome of the Item 5 discussio

2.       Should a provider that mixes or composes other media
captures/streams into a new stream be able to indicate some information
about the algorithm it uses to choose which other captures/streams to mix?

3.       Should a provider that mixes or composes media be able to allow the
consumer to make a choice about which other captures/streams are included in
the mix?

a.       This should be covered by Major Issue Item 3 source selection -
(discussion to be kicked off by Jonathan)

4.       Do we need support in CLUE for allowing a media consumer to request
the provider to compose or mix particular media sources into a new media
stream, using a particular algorithm?

b.      But if something like item 5 idea is accepted, then maybe a provider
can offer a few alternatives and a consumer can choose among them

Mark Duckworth

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 2 Nov 2011 14:41:55 -0700

Hi Roni,

Thanks for the clarifications.

For identifying what is in a mixed or composed media stream, I agree it makes sense to use the existing SSRC and CSRC mechanism rather than something new.

I think you are saying you want to define a few enumerated "mixing policies" and  "mixing modes" (switching is a subset of mixing) that will cover some small number of useful cases, is that right?  But not try to be all-inclusive with a never-ending list of possibilities?

What do you mean by "The protocol from the consumer to provider can be a second stage"?  Do you mean leave that for a possible future extension?

Mark

From: Roni Even [mailto:[email-masked]]
Sent: Wednesday, November 02, 2011 11:10 AM
To: Duckworth, Mark; [email-masked]
Subject: RE: [clue] Kick-off Item 2: Layout, composition, switching discussion

Hi Mark,
The reason I shorted the list is because if we want item 4 support than item 2 is needed to let the consumer know what algorithms can be used.
As for item 1 I do not understand why we are even discussing it. In current RTP you know the SSRCs of the mixed sources as they are part of CSRC, so for CLUE we need to verify that this is not broken. The change in the mix should be reflected in RTCP but we will need the association to the CLUE information.

My view is that we need to support  both switching and mixing (so switching here is a case of mixing). The issue of conference-wide mode or per user is application or implementation specific. The provider can decide how to respond to a request based on its capabilities and policies.

As for enumerating the cases.  There is more than one dimension. I think that we can look at general policy (site switch, segment switch), per receive stream (Voice activated, round robin). The usage can be automated, so he consumer asks for the mode and the actual content is decided by the producer based on the mode (Example segment switch, stream 1 voice activated, stream 2 round robin- this will send in stream 1 the current speaker and in stream 2 go through the rest of the streams (all rooms and all cameras) in round robin).
If the consumer will want to build a specific mix he should be able to ask for a specific streams for example in the previous example the consumer can ask also for a third stream that will have the Manager TP system middle camera.

I think that it will be good in the first stage to allow the producer to provide the information about the modes (segment switch, site switch, voice activate,..) and to allow the consumer to identify who is in the mix/switch. The protocol from the consumer to provider can be a second stage
Roni

From: [email-masked] [mailto:[email-masked]] On Behalf Of Duckworth, Mark
Sent: Tuesday, November 01, 2011 7:35 PM
To: [email-masked]
Subject: Re: [clue] Kick-off Item 2: Layout, composition, switching discussion

Hi Roni,

Mark

Hi Mark,

Roni

Mark Duckworth

---

From: "Andrew Pepperell (apeppere)" <[email-masked]>
Date: Thu, 3 Nov 2011 05:31:40 +0100

Roni, Mark,

My feeling on the whole use of CSRCs to identify the SSRCs of the mixed
sources is that perhaps this shouldn't be relied upon for CLUE - I'm not
so convinced this works for multi-stream systems, it's possible to make
composed layouts that would require too many CSRCs to be included, plus
it only goes so far in describing the mix (for instance, it can't
include information on who's being placed where etc.). The idea here I
favor would be an additional optional provider -> consumer message to
describe (in as full terms as we like) the make-up of a composed (or
even switched) stream. This isn't to say that including CSRC values into
a mixed stream is wrong, just that it isn't necessarily sophisticated
enough for systems that would want to use CLUE to its fullest.

To me, enumerating "a few possible modes" for a number of specific use
cases is in line with the thinking behind the "switched" / "composed"
media capture attributes already present in the model. A given middle
box might advertise a set of switched or loudest speaker captures and a
set of pre-composed streams as alternatives and the consumer then
chooses whether to receive loudest speaker streams and compose them into
a layout itself or to receive the pre-composed streams and map them 1 to
1 onto its screens. However, it's not completely clear if this could /
should be taken further to include alternatives for site switching /
segment switching, a 2 x 2 layout vs 3 x 3, fast / slow switching, round
robin panes etc. I wonder if each of these cases needs reasonably
careful consideration as to how it might be done within the existing
ideas of media capture attributes. I also wonder if this sort of thing
strengthens the case for the inclusion of the set of supported media
capture attributes in an initial consumer -> provider message.

I do think though that layout is a special case here - I don't see CLUE
as needing to provide a way for the consumer to specify to the provider
(e.g. in the pre-composed case) any form of description of the layout it
wants. To me, the pre-composed case should be the consumer choosing to
take what the provider decides to compose (modulo any site / segment
switching variations as per the above) an endpoints wishing to implement
specific layouts should instead be asking for the "n" loudest streams
and then perform the composition themselves. This avoids the need to
CLUE to somehow be able to describe layouts while allowing different
vendors' middle boxes to continue to enjoy their current freedom in
terms of implementing sensible / innovative layouts.

In terms of the ability of consumers to choose specific streams to
receive, again I think we need to bear in mind certain complexities
here. Specifically, assuming the large conference case where there are
100, 1000 or more different systems in a conference, each capable of
sending 1, 2, 3, or 4 camera versions of their media, it's clearly not
practical for every single CLUE consumer to be able to choose from the
thousands of individual streams on offer - I think we should aim to
avoid the case where a single system can ask for, say, systems A's left
screen on its own right, system B's center on its left and some other
system C's left screen to be displayed on its center. Apart from
anything else, this opens up the possibility of multiple simultaneous
conflicting requests which would be individually valid but impossible
together. I would see it as the job of the middle box in this case to
present a sensible set of choices to each consumer, in conjunction with
a roster list, enabling each consumer some freedom in what it receives
without the potential for complete chaos / unimplementability that the
"full mesh" scheme would allow.

I think there is a reasonably sensible middle ground to be reached here
to allow some level of selection while avoiding the worst case -
specifically, if we allowed a consumer to choose who it would prefer to
see in each selectable stream and combine this with the provider ->
consumer stream tagging  message we'd be able to accomplish almost of
the sensible cases I think without getting to the pathological worst
cases.

Andy

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Duckworth, Mark
Sent: Wednesday, November 02, 2011 9:42 PM
To: [email-masked]
Subject: Re: [clue] Kick-off Item 2: Layout, composition, switching
discussion

Hi Roni,

For identifying what is in a mixed or composed media stream, I agree it
makes sense to use the existing SSRC and CSRC mechanism rather than
something new.

I think you are saying you want to define a few enumerated "mixing
policies" and  "mixing modes" (switching is a subset of mixing) that
will cover some small number of useful cases, is that right?  But not
try to be all-inclusive with a never-ending list of possibilities?

What do you mean by "The protocol from the consumer to provider can be a
second stage"?  Do you mean leave that for a possible future extension?

Mark

Hi Mark,

The reason I shorted the list is because if we want item 4 support than
item 2 is needed to let the consumer know what algorithms can be used. 

As for item 1 I do not understand why we are even discussing it. In
current RTP you know the SSRCs of the mixed sources as they are part of
CSRC, so for CLUE we need to verify that this is not broken. The change
in the mix should be reflected in RTCP but we will need the association
to the CLUE information.

My view is that we need to support  both switching and mixing (so
switching here is a case of mixing). The issue of conference-wide mode
or per user is application or implementation specific. The provider can
decide how to respond to a request based on its capabilities and
policies.

As for enumerating the cases.  There is more than one dimension. I think
that we can look at general policy (site switch, segment switch), per
receive stream (Voice activated, round robin). The usage can be
automated, so he consumer asks for the mode and the actual content is
decided by the producer based on the mode (Example segment switch,
stream 1 voice activated, stream 2 round robin- this will send in stream
1 the current speaker and in stream 2 go through the rest of the streams
(all rooms and all cameras) in round robin).

If the consumer will want to build a specific mix he should be able to
ask for a specific streams for example in the previous example the
consumer can ask also for a third stream that will have the Manager TP
system middle camera.

I think that it will be good in the first stage to allow the producer to
provide the information about the modes (segment switch, site switch,
voice activate,..) and to allow the consumer to identify who is in the
mix/switch. The protocol from the consumer to provider can be a second
stage

Roni

Hi Roni,

I think your item 1 and my item 3 are about source selection, which
needs a discussion to include selecting sources for a mix, and also
selecting sources a consumer wants to receive individually (not part of
a mix).  Jonathan's presentation from the Andover meeting
(13-CLUE-Interim-Sources.ppt) raised some good points about this.  Do
you agree this all falls under the same topic?

I think your item 2 and my item 4 are the same topic.  Are you talking
about switching streams, or composing streams, or both?  Do you see
these switching modes as something a particular consumer can request
from a provider, that will not impact other consumers? Or is it more
like a conference-wide mode? (this gets into Paul's point as well).

My concern about this is I don't see how to bound it.  Are you
suggesting we should define a few specific composing/switching modes
that a provider can enumerate, and a way for a consumer to choose one?

Mark

Hi Mark,

1.       Allowing the consumer to select specific media streams for a
mix (for example, left camera from room 1, middle camera from room 2 and
left camera from room 3) which I think should be possible. This will
require information and protocol that will allow this functionality.

2.       Allowing the provider  to specify the mixture algorithm and for
the consumer to ask for a specific one. I think that we need at least to
enable the provider and consumer to specify or request site switch or
segment switch mode.

Roni

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Duckworth, Mark
Sent: Thursday, October 27, 2011 5:30 PM
To: [email-masked]
Subject: [clue] Kick-off Item 2: Layout, composition, switching
discussion

This is following up on the major issues list from the interim meeting
in Andover.

Issues for discussion:

1.       Item 5 video composition (discussion kicked off by Stephan) is
a part of this.  But more generally, the same sort of issue applies to
other media types, not just video.

a.       For example, should a provider that mixes audio be able to
indicate to the consumer which audio sources are in the mix?

2.       Should a provider that mixes or composes other media
captures/streams into a new stream be able to indicate some information
about the algorithm it uses to choose which other captures/streams to
mix?

3.       Should a provider that mixes or composes media be able to allow
the consumer to make a choice about which other captures/streams are
included in the mix?

4.       Do we need support in CLUE for allowing a media consumer to
request the provider to compose or mix particular media sources into a
new media stream, using a particular algorithm?

b.      But if something like item 5 idea is accepted, then maybe a
provider can offer a few alternatives and a consumer can choose among
them

Mark Duckworth

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 03 Nov 2011 13:27:33 -0400

> enough for systems that would want to use CLUE to its fullest.

I'm trying to understand what you are proposing.

Are you saying that the information conveyed from producer to supplier 
would be the complete list of candidate sources for a capture, that 
*might* be mixed in? (Meanwhile, the actual ones being mixed in at any 
point in time will be available via the CSRC.)

E.g. If you have a "loudest speaker" capture, then the description of 
the capture would include all the captures that are being selected among.

	Thanks,
	Paul

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Fri, 4 Nov 2011 11:03:44 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of Paul Kyzivat
> Sent: Thursday, November 03, 2011 10:28 AM
> To: [email-masked]
> Subject: Re: [clue] Kick-off Item 2: Layout, composition,switching
discussion
> the capture would include all the captures that are being selected
among.

Elaborating on this a bit more, I am envisioning that the CLUE mechanism
would tell you the complete set of candidate sources. A CLUE update
would be needed whenever a candidate source is added or removed. The RTP
CSRC/SSRC would tell you in real-time the actual mix, which would be
some subset of the candidate sources. As such, there will be some
mapping of CSRC/SSRC value to candidate source ID.

Cheers,
Charles

> 
> 	Thanks,
> 	Paul
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

---

From: "Andrew Pepperell (apeppere)" <[email-masked]>
Date: Sat, 5 Nov 2011 09:15:15 +0100

Hi Paul,

>point in time will be available via the CSRC.)

Apologies for not being clearer here - I'm not proposing that the list
of sources that *might* be mixed in is ever sent from provider to
consumer, just an (optional) more fully featured version of essentially
the CSRC information. For instance, a list of CSRCs alone wouldn't be
able to tell the consumer about where other participants' capture
streams had been placed within a composed layout, nor the equivalent
left / right audio panning information about constituent audio streams.

>the capture would include all the captures that are being selected
among.

I'm not convinced that this would scale well, though I'd anticipate that
information on all the other *participants* whose captures could at any
time be the loudest speaker would be available via roster list
protocols.

Regards,

Andy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Paul Kyzivat
Sent: Thursday, November 03, 2011 5:28 PM
To: [email-masked]
Subject: Re: [clue] Kick-off Item 2: Layout, composition,switching
discussion

E.g. If you have a "loudest speaker" capture, then the description of 
the capture would include all the captures that are being selected
among.

	Thanks,
	Paul

---

From: "Andrew Pepperell (apeppere)" <[email-masked]>
Date: Sat, 5 Nov 2011 09:27:36 +0100

>mapping of CSRC/SSRC value to candidate source ID.

My feeling is that this would give scalability issues; I'd very keen,
especially in the multipoint case, that it not end up effectively
becoming a requirement that a middle box need keep every consumer
informed of the complete set of media captures within the conference. To
my mind, the roster list would be the place to tell devices of
participants joining / leaving, with the finer-grained media capture
selection being [able to be] taken care of by the middle box itself (to
avoid cases of potentially multiple conflicting stream requests from
consumers).

Andy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Charles Eckel (eckelcu)
Sent: Friday, November 04, 2011 6:04 PM
To: Paul Kyzivat; [email-masked]
Subject: Re: [clue] Kick-off Item 2: Layout, composition,switching
discussion

Of Paul Kyzivat
discussion
among.

Cheers,
Charles

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Mon, 7 Nov 2011 11:11:26 -0800

I think that is fine, assuming you have a way to identify a participant
in the roster list by its RTP SDES CNAME value. You can then use that to
associate the SSRC/CSRC values in the media streams with each
participant/CNAME.

Cheers,
Charles

> From: Andrew Pepperell (apeppere)
> Sent: Saturday, November 05, 2011 1:28 AM
> To: Charles Eckel (eckelcu); Paul Kyzivat; [email-masked]
> Subject: RE: [clue] Kick-off Item 2: Layout, composition,switching
discussion
> My feeling is that this would give scalability issues; I'd very keen,
especially in the multipoint
> case, that it not end up effectively becoming a requirement that a
middle box need keep every consumer
> informed of the complete set of media captures within the conference.
To my mind, the roster list
> would be the place to tell devices of participants joining / leaving,
with the finer-grained media
> capture selection being [able to be] taken care of by the middle box
itself (to avoid cases of
> potentially multiple conflicting stream requests from consumers).
> 
> Andy
> 
> 
Of Charles Eckel (eckelcu)
discussion
> > > sources is that perhaps this shouldn't be relied upon for CLUE -
I'm
> not
> > > include information on who's being placed where etc.). The idea
here
> I
> > > even switched) stream. This isn't to say that including CSRC
values
> into