Subject: Re: [clue] #7: Is composed attribute a boolean or data structure

From: "Roni Even" <[email-masked]>
Date: Wed, 1 Feb 2012 12:30:17 +0200

Hi Espen,
I called site or segment switch a policy but I am not sure that it is a switch policy by itself since segment switch for example can happen based on active speakers or some round robin timing which is the switching policy.
Roni

> -----Original Message-----
> From: Espen Berger (espeberg) [mailto:[email-masked]]
> Sent: Tuesday, January 31, 2012 5:38 PM
> To: Roni Even; [email-masked]
> Subject: RE: [clue] #7: Is composed attribute a boolean or data
> structure
> 
> Hi Rony
> 
> To choose between site and segment switch is more a policy than a
> <video-layout>, so in that case I would argue that you could offer a
> Capture stream with a switch-policy = {site, segment}
> 
> My point about interoperability is that a single composed stream with
> alternative layouts are easier to understand, than multiple capture
> streams with alternative configurations. The last requires to pick the
> first one or requires some sort of default marking.
> 
> Cheers
> 
> -Espen
> 
> 
> -----Original Message-----
> From: Roni Even [mailto:[email-masked]]
> Sent: 31. januar 2012 00:06
> To: Espen Berger (espeberg); [email-masked]
> Subject: RE: [clue] #7: Is composed attribute a boolean or data
> structure
> 
> Hi Espen,
> If the provider wants to offer site switch and segment switch option to
> the consumer he cannot do it just with A. this is a basic use case.
> 
> In general, your option does not provide any interoperability since
> both sides may have different views what composed means. In the
> framework example of a PIP as composed image it is an assumption that
> if the provider offers this composed offer, the consumer can understand
> that it is a PIP mix but there is nothing in the offer that will
> indicate that it is.
> 
> Roni
> 
> > -----Original Message-----
> > From: Espen Berger (espeberg) [mailto:[email-masked]]
> > Sent: Monday, January 30, 2012 5:21 PM
> > To: Roni Even; [email-masked]
> > Subject: RE: [clue] #7: Is composed attribute a boolean or data
> > structure
> >
> > Hi Roni
> >
> > If you’re a particular use cases requires control over the composed
> > layouts you need a) and b). In my example I used a single
> > advertisement composed stream with optional meta-information about
> > possible layout choices. A receiver that wants the default composed
> > layout can request the composed stream and skip the optional video-
> layout information.
> > Optionally you could request the same composed stream with layouts
> > hints for the receiver.
> >
> > For me use case c) does not include the selection mechanisms, only
> the
> > content you see in the video stream.
> >
> > Cheers
> >
> > -Espen
> >
> > -----Original Message-----
> > From: Roni Even [mailto:[email-masked]]
> > Sent: 30. januar 2012 15:39
> > To: Espen Berger (espeberg); [email-masked]
> > Subject: RE: [clue] #7: Is composed attribute a boolean or data
> > structure
> >
> > Hi Espen,
> > If you will look at the notes from the last call there was a support
> > that a) is not enough. Having just compose does not address even the
> > use cases I provided which are also based on the use case draft.
> >
> >
> > I am not sure what you mean by C since what in the composed stream
> can
> > be either which VC you see or what is the selection criteria for
> being
> > in a composed stream. If you meant the second, my view is that C and
> > also b should be in the basic framework and not in an extension.
> >
> > Roni Even
> >
> > > -----Original Message-----
> > > From: Espen Berger (espeberg) [mailto:[email-masked]]
> > > Sent: Monday, January 30, 2012 4:17 PM
> > > To: Roni Even; [email-masked]
> > > Subject: RE: [clue] #7: Is composed attribute a boolean or data
> > > structure
> > >
> > >
> > > To help with the direction of the discussions I think it's useful
> to
> > > divide the discussions into three use cases for composed streams.
> > >  a) How to express if a capture stream is composed or not
> > >  b) How to express layout choices? (e.g. with a video-layout
> > > element)
> > >  c) How to express what's inside a composed stream
> > >
> > > For use case a) a composed attribute should be sufficient, you
> > > either ask for the composed stream or you do not.
> > > I also believe this should cover the interoperability requirement
> we
> > > have across multiple types of endpoints.
> > >
> > > For use case b) both mediactrl and xcon uses a <video-layout>
> > > element to describe possible layouts and also to request the
> > > preferred layout based on options. (as described in Roni's email)
> > >
> > > An example could be (inspired by mediactrl and xcon usage of
> <video-
> > > layout>):
> > >
> > > CLUE Advertisement
> > >   Capture id=2 Purpose=people Composed=false
> > >   Capture id=3 Purpose=people Composed=false
> > >   Capture id=4 Purpose=People Composed=true
> > >       Video-layout="'automatic', 'dual-view', ' single-view'"
> > >
> > > CLUE configure // Default composed stream
> > >   Capture id=4
> > >
> > > // Or composed stream with layout hints
> > >    Capture id=4 Video-layout='dual-view'
> > >
> > > The video-layout element is a list of strings and each string is a
> > > layout-hint that can be requested. Video-layout is optional. A
> > > layout hints about the requested rendering and the source is free
> to
> > > replace any stream as long as they fit the layout.
> > >
> > > Use case c) is more open in the sense that what you actually
> receive
> > > in a composed stream is depending on status of a room or who is in
> a
> > > MCU conference. E.g. from a room you could get a mix of different
> > > capture stream representing cameras and from a transcoding MCU it
> > > could be different room, different cameras or another mix of
> > > possible input video streams.
> > >
> > > Having support for both b) and c) could be a good test for the
> > > extensibility of CLUE. If the basis of CLUE only does use case a)
> > > there should be room to extend CLUE with support for b) and c) as a
> > > CLUE + layout description extension.
> > >
> > > Cheers
> > >
> > > -Espen
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: [email-masked] [mailto:[email-masked]] On
> Behalf
> > > Of Roni Even
> > > Sent: 30. januar 2012 11:18
> > > To: [email-masked]
> > > Subject: Re: [clue] #7: Is composed attribute a boolean or data
> > > structure
> > >
> > > Hi,
> > > During the last call I volunteered to provide some input.
> > >
> > > In current IETF work (XCON, Mediactrl WGs there is a video layout
> > > element)
> > >
> > > http://tools.ietf.org/html/draft-ietf-mediactrl-mixer-control-
> > package-
> > > 14#section-4.2.1.4.2.1
> > >
> > > and
> > > http://tools.ietf.org/html/draft-ietf-xcon-common-data-model-
> > > 32#section-4.2.7 (look at video-layout)
> > >
> > > I would like first to try to define the term "composed video" since
> > it
> > > looks to me like we have different views here, and provide my
> > > initial view on what should be described.
> > >
> > > Composed video can describes both the layout and the selection
> > > algorithm for composing the  content of the sub-windows in the
> > "mixed"
> > > video.
> > >
> > > The video layout which just describes the geometry of the composed
> > > image and I think that the above references provide good structure
> > > to define this part of the composed video attribute.
> > >
> > > The other part is the algorithm by which the provider select the
> > > content of each element in the layout. Since the content of each
> > > element may change dynamically by the provider this attribute only
> > > address the static information which is the algorithm and not the
> > > current content (who we see now in each element) which will need to
> > be
> > > conveyed also but probably not using this attribute. Note that the
> > > information is valid for point to point and multipoint so the
> > > current content should reflect the TP end point and the specific VC
> > > used from it.
> > >
> > > The algorithms may be global or per element(or sub-window). The
> > global
> > > algorithms I see are site switch or segment switch. The per element
> > > may be voice activated, round robin (switch every x seconds) and
> > fixed
> > > (the same VC is displayed there (may be changed by some control
> > > mechanism)
> > >
> > >
> > > Roni
> > >
> > > > -----Original Message-----
> > > > From: [email-masked] [mailto:[email-masked]] On
> > Behalf
> > > > Of clue issue tracker
> > > > Sent: Tuesday, January 24, 2012 12:43 AM
> > > > To: [email-masked];
> > > > [email-masked]
> > > > Cc: [email-masked]
> > > > Subject: Re: [clue] #7: Is composed attribute a boolean or data
> > > > structure
> > > >
> > > > #7: Is composed attribute a boolean or data structure
> > > >
> > > > Changes (by mary.ietf.barnes@…):
> > > >
> > > >  * type:  defect => task
> > > >
> > > >
> > > > --
> > > > --------------------------------+--------------------------------
> -
> > > > --------------------------------+-
> > -
> > > > --------------------------------+-
> > > -
> > > > --------------------------------+-
> > > > ----
> > > >  Reporter:  mary.ietf.barnes@…  |       Owner:  draft-ietf-clue-
> > > > framework@…
> > > >      Type:  task                |      Status:  new
> > > >  Priority:  major               |   Milestone:
> > > > Component:  framework           |     Version:
> > > >  Severity:  Active WG Document  |  Resolution:
> > > >  Keywords:                      |
> > > > --------------------------------+--------------------------------
> -
> > > > --------------------------------+-
> > -
> > > > --------------------------------+-
> > > -
> > > > --------------------------------+-
> > > > ----
> > > >
> > > > Ticket URL:
> > > > <http://trac.tools.ietf.org/wg/clue/trac/ticket/7#comment:2>
> > > > clue <http://tools.ietf.org/wg/clue/>
> > > >
> > > > _______________________________________________
> > > > clue mailing list
> > > > [email-masked]
> > > > https://www.ietf.org/mailman/listinfo/clue
> > >
> > > _______________________________________________
> > > clue mailing list
> > > [email-masked]
> > > https://www.ietf.org/mailman/listinfo/clue
>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Wed, 1 Feb 2012 14:00:01 +0100

I would still call it switch-policy, e.g. then possible values could be: 
* Site: Change all at the same time 
* Segment: Only change the active segment 
* Round robin: Change active segment and then change to other segments in 10 sec intervals. 

How you switch between streams and how you render has different requirements and policies, so I find it very useful to discuss them as two separate issues to avoid confusion. 

Cheers 

-Espen 

-----Original Message-----
From: Roni Even [mailto:[email-masked]] 
Sent: 1. februar 2012 11:30
To: Espen Berger (espeberg); [email-masked]
Subject: RE: [clue] #7: Is composed attribute a boolean or data structure

Hi Espen,

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 01 Feb 2012 12:29:19 -0500

Espen,

ISTM that for a switched stream you have:
- the set of candidate streams to be switched among
- an algorithm for picking among them

If the receiver has multiple screens, then there will be a stream for 
each one, and each could have different candidate input streams and a 
different policy.

What you are describing as "site switching" adds another wrinkle, since 
it is coupling the switching policies for multiple streams. Also, I'm 
not sure how it works if the sites have differing numbers of screens and 
cameras. How would you describe the individual streams from the MCU so 
that a room could select an appropriate set of streams to get site 
switching?

It would be helpful (to me anyway) if you could describe the interesting 
cases by enumerating the offered streams together with the inputs and 
the algorithm used for each. Right now, I'm not sure how to do that.

	Thanks,
	Paul

On 2/1/12 8:00 AM, Espen Berger (espeberg) wrote:
>>>> For use case b) both mediactrl and xcon uses a<video-layout>
>>>>> clue<http://tools.ietf.org/wg/clue/>
>>>>>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Thu, 2 Feb 2012 17:42:19 +0100

Hi Paul 

I will make a brief attempt to answer. 

The scenarios are how to map a 3x camera system to a 2x screen system. The triple offer 
 Strema = 1 - 3 
 Policies = Switched, static, composed 

As a receiver (with 2x screens) I would do this choices 

1) How many streams to receive 
   1 => The triple sends a single stream by some sender preferred policy 
   2 => The triple sends two streams by some policy 
   3 => The triple sends three cameras in spatial order, and 
      the receiver decide how to render across two screens 

2) Which switching policy 
  Static => The requester ask for explicit camera streams, e.g. cam1 or (cam2+cam3)  
  Switched/Composed => The triple chooses the streams to send, e.g. the two loudest segments or it could make a nice rendering of 3x cameras rendered on two streams. 

I assume that you won't use different switching policies for the different streams, since it gives the sender of media the chance to optimize what it send on the streams requested. 

The 'site' switching policy makes most sense from an MCU. I interprete the policy as try the best you can to switch in all capture streams from a room when the room is active. Example 
 Room A has two cameras 
 Room B has three cameras
 Room C has a single camera  
 Room D has three screens 

1) Room D receives media from room A and C 
2) Room B starts talking
3a) with site switching 
  Room D receives media only from D across all three screens 
3b) with segment switching 
  Room D receives media from A, and the new active segment from D 

Cheers  
 
-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Kyzivat
Sent: 1. februar 2012 18:29
To: [email-masked]
Subject: Re: [clue] #7: Is composed attribute a boolean or data structure

Espen,

	Thanks,
	Paul

---

From: "Roni Even" <[email-masked]>
Date: Fri, 3 Feb 2012 00:17:35 +0200

Hi Espen,
I see from your example that you make some assumptions while I have different ones.

I think that we probably see differently how the switch / compose is done. My view is that the sender should list all the modes he supports and let the receiver choose. The receiver must be able to select what it wants to view and in order to do it the receiver need to know what the sender is capable to produce.

You write "The triple sends a single stream by some sender preferred policy" - my view is that the sender can offer different options by listing more than one VC and let the receiver select. 
I also think that there can be different policies for different streams. For the triple sends two streams one may be the active speaker and the other may be static (for example middle camera or a mix of the other two cameras).

Roni Even

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Espen Berger (espeberg)
> Sent: Thursday, February 02, 2012 6:42 PM
> To: Paul Kyzivat; [email-masked]
> The scenarios are how to map a 3x camera system to a 2x screen system.
> The triple offer  Strema = 1 - 3  Policies = Switched, static, composed
> 
>   Switched/Composed => The triple chooses the streams to send, e.g. the
> two loudest segments or it could make a nice rendering of 3x cameras
> rendered on two streams.
> 
> I assume that you won't use different switching policies for the
> different streams, since it gives the sender of media the chance to
> optimize what it send on the streams requested.
> 
> The 'site' switching policy makes most sense from an MCU. I interprete
> the policy as try the best you can to switch in all capture streams
> from a room when the room is active. Example  Room A has two cameras
> Room B has three cameras  Room C has a single camera  Room D has three
> screens
> 
> not sure how it works if the sites have differing numbers of screens
> and cameras. How would you describe the individual streams from the MCU
> so that a room could select an appropriate set of streams to get site
> switching?
> 
> It would be helpful (to me anyway) if you could describe the
> interesting cases by enumerating the offered streams together with the
> inputs and the algorithm used for each. Right now, I'm not sure how to
> do that.
> 
> 	Thanks,
> 	Paul
> 
> > How you switch between streams and how you render has different
> requirements and policies, so I find it very useful to discuss them as
> two separate issues to avoid confusion.
> >
> > Cheers
> >
> > -Espen
> >
> >
> > I called site or segment switch a policy but I am not sure that it is
> a switch policy by itself since segment switch for example can happen
> based on active speakers or some round robin timing which is the
> switching policy.
> > Roni
> >
> >> streams with alternative configurations. The last requires to pick
> >> the first one or requires some sort of default marking.
> >>
> >> Cheers
> >>
> >> -Espen
> >>
> >>
> >> If the provider wants to offer site switch and segment switch option
> >> to the consumer he cannot do it just with A. this is a basic use
> case.
> >>
> >> if the provider offers this composed offer, the consumer can
> >> understand that it is a PIP mix but there is nothing in the offer
> >> that will indicate that it is.
> >>
> >> Roni
> >>

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 02 Feb 2012 19:07:43 -0500

> I will make a brief attempt to answer.

Its good to get into the details. As exhibited by Roni's reply, its not 
clear that all are on the same page.

>   Policies = Switched, static, composed

IIUC, in the above you have separated the policies from the streams - 
somehow the policies apply to all the streams.

It was my impression that each offered stream is independent, and may be 
static, switched, or composed.

So, I would assume that if the 3 camera system wants to accommodate 
dissimilar systems, it would offer more than three streams. E.g.:

- left, static (!composed)
- middle, static
- right, static
- composed (left, middle, and right)
- switched (active speaker)
- ...

>     1 =>  The triple sends a single stream by some sender preferred policy

This really confuses me.
I assumed that if the receiver only wants one stream (perhaps because it 
has only one screen), it would simply pick one of the streams offered 
(as you described, it would be one of the three.)

You seem to be suggesting that the act of requesting one stream will 
cause the sender to alter the policy used to produce that stream. That 
makes no sense to me.

>     2 =>  The triple sends two streams by some policy

More of same confusion above. ISTM that the if the recipient wants two 
streams to display, that it should be choosing two based on the 
descriptions. Not that the sender decides what to include in the two 
based on the fact that two have been requested.

>        the receiver decide how to render across two screens

This is clearly the most straightforward case.

> I assume that you won't use different switching policies for the different streams, since it gives the sender of media the chance to optimize what it send on the streams requested.

What I take from this is that you are assuming a single policy that 
impacts all the streams sent to this recipient.

But I thought the intent was that the sender describes *each* available 
stream with an independent policy, and the recipient just chooses those 
that it wishes.

>    Room D receives media from A, and the new active segment from D

I don't even want to *start* talking about this until the stuff above is 
resolved.

	Thanks,
	Paul

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Mon, 6 Feb 2012 20:01:11 +0100

I should have documented the use cases I used better. I will make a summary during the next days that hopefully is clearer. 

See inline. 

-Espen 

-----Original Message-----
From: Paul Kyzivat [mailto:[email-masked]] 
Sent: 3. februar 2012 01:08
To: Espen Berger (espeberg)
Cc: [email-masked]
Subject: Re: [clue] #7: Is composed attribute a boolean or data structure

EBE>> Details are good. 

EBE>> It's natural that you have at least one alternative in a capture set where all captures have the same policies. If we write them out as the same policy on individual capture streams that should mean the same as a single policy on all captures in a capture set alternative.  

EBE>> Agree, I made an example where also two screen alternatives are listed. In the example Left, middle, right represent the cameras, VCx are logical streams, I also assume that the only three streams alternative allowed is to request all 3x camera streams. 
Static 3-streams {left, middle, right}
Static 2-streams {left, middle}, {left, right} or {middle, right}
Static 1-stream {left}, {middle} or {right}
Composed 2-streams {VC0, VC1} 
Composed 1-stream {VC2} 
Switched 2-streams {Vc3, Vc4} 
Switched 1-stream {VC5} 
(What is the preferred alternative for a two screen system?)

This really confuses me.

EBE>> In the use cases I assumed that a media consumer want to select the alternative that the media provider think is most sensible for a the number of streams selected. E.g. if an endpoint request a single capture streams from a triple camera endpoint, the preferred alternative might be a composed view. If the media consumer wants a particular alternative, it can parse them all and select the one that matches what it wants. 

EBE>> Same as above. 

EBE>> It's a limitation to only allow the same policy on all streams requested, but in many use cases I think it's natural to use the same policy across multiple streams. 

EBE>> agree, we can wait with this one.

	Thanks,
	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 06 Feb 2012 20:32:55 -0500

> I should have documented the use cases I used better. I will make a summary during the next days that hopefully is clearer.

Comments inline too.

> (What is the preferred alternative for a two screen system?)

You show this as 7 alternatives, but its actually 12 streams, right?
Each row above is a recommended selection of multiple streams that you 
think makes sense.

And as you show it, the policy is associated with a set of streams, 
rather than a particular stream. I can't tell if that is the way you 
want to represent it, or just what you found convenient for this example.

But I presume there is nothing to prevent selecting some other subset of 
those 12 streams, right? (E.g. middle, VC5)

> EBE>>  In the use cases I assumed that a media consumer want to select the alternative that the media provider think is most sensible for a the number of streams selected. E.g. if an endpoint reque…

This presumes that the advertisement indicates a recommended grouping of 
streams for a particular number of screens. I hadn't previously seen a 
proposal to include that info.

> EBE>>  It's a limitation to only allow the same policy on all streams requested, but in many use cases I think it's natural to use the same policy across multiple streams.

This needs to be sorted out, because it is pretty fundamental to the way 
that all this info is represented.

	Thanks,
	Paul

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 8 Feb 2012 07:53:45 -0800

I'll add my thoughts to this, for how I see this scenario working (3x camera system sending to a 2x screen system), using Espen's ideas for what the alternatives are for this particular example, given this particular hypothetical 3 screen provider's capabilities.

Provider advertises these 9 video captures:
VC0 (left) not composed; not switched
VC1 (middle) not composed; not switched
VC2 (right) not composed; not switched
VC3 (left half of two screen composition) composed; not switched
VC4 (right half of two screen composition) composed; not switched
VC5 (left half of two screen switched) not composed; switched
VC6 (right half of two screen switched) not composed; switched
VC7 (single stream composition) composed, not switched
VC8 (single stream switched) not composed, switched

VC3 through VC8 could also have additional attributes describing the available video layout choices for composition or the available switching policies the provider is offering.  The consumer could choose among these.  I think that is the point Roni has been making.  Details of this need further discussion.  (Maybe the composed and switched choices should be collapsed into one set of stuff to choose from for a particular VC, rather than separate VCs, so that would reduce the number of VCs here from 9 to 6 ?).

Provider advertises a capture set with 5 entries:
Entry 1 (VC0, VC1, VC2)
Entry 2 (VC3, VC4)
Entry 3 (VC5, VC6)
Entry 4 (VC7)
Entry 5 (VC8)

By advertising the capture set in this way, the provider is suggesting which video captures are most useful to receive together, to get a view of the whole scene.  Each entry in the capture set is a suggested alternative.

A 2 screen consumer can choose VC3 and VC4 if it wants a composition.  It can choose VC5 and VC6 if it wants a dynamically switched view.  A 2 screen consumer is also free to ignore the suggestions in the capture set and instead choose VC0 and VC1 if that is what it wants.  It could also choose other combinations from different entries, say VC1 and VC8, assuming the provider's advertised simultaneous sets and encoding groups do not prohibit that combination.

Mark

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Espen Berger (espeberg)
Sent: Thursday, February 02, 2012 11:42 AM
To: Paul Kyzivat; [email-masked]
Subject: Re: [clue] #7: Is composed attribute a boolean or data structure

Hi Paul

The scenarios are how to map a 3x camera system to a 2x screen system. The triple offer  Strema = 1 - 3  Policies = Switched, static, composed

The 'site' switching policy makes most sense from an MCU. I interprete the policy as try the best you can to switch in all capture streams from a room when the room is active. Example  Room A has two cameras  Room B has three cameras  Room C has a single camera  Room D has three screens

Cheers

-Espen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Kyzivat
Sent: 1. februar 2012 18:29
To: [email-masked]
Subject: Re: [clue] #7: Is composed attribute a boolean or data structure

Espen,

If the receiver has multiple screens, then there will be a stream for each one, and each could have different candidate input streams and a different policy.

What you are describing as "site switching" adds another wrinkle, since it is coupling the switching policies for multiple streams. Also, I'm not sure how it works if the sites have differing numbers of screens and cameras. How would you describe the individual streams from the MCU so that a room could select an appropriate set of streams to get site switching?

It would be helpful (to me anyway) if you could describe the interesting cases by enumerating the offered streams together with the inputs and the algorithm used for each. Right now, I'm not sure how to do that.

        Thanks,
        Paul

On 2/1/12 8:00 AM, Espen Berger (espeberg) wrote:
>> to the consumer he cannot do it just with A. this is a basic use case.
>>

---

From: "Roni Even" <[email-masked]>
Date: Wed, 8 Feb 2012 18:58:02 +0200

Hi Mark,
I can add two more options.

1. Two out of three which will include the current and previous speaker
2. Two out of three which will rotate between the three every 10 seconds
These two cannot be differentiated with binary composed and switched.

If we go from three to one we have even more options that cannot be differentiated by binary values and I think that three to one will be more common.

Roni

> Duckworth, Mark
> Sent: Wednesday, February 08, 2012 5:54 PM
> To: [email-masked]
> I'll add my thoughts to this, for how I see this scenario working (3x
> camera system sending to a 2x screen system), using Espen's ideas for
> what the alternatives are for this particular example, given this
> particular hypothetical 3 screen provider's capabilities.
> 
> VC3 through VC8 could also have additional attributes describing the
> available video layout choices for composition or the available
> switching policies the provider is offering.  The consumer could choose
> among these.  I think that is the point Roni has been making.  Details
> of this need further discussion.  (Maybe the composed and switched
> choices should be collapsed into one set of stuff to choose from for a
> particular VC, rather than separate VCs, so that would reduce the
> number of VCs here from 9 to 6 ?).
> 
> By advertising the capture set in this way, the provider is suggesting
> which video captures are most useful to receive together, to get a view
> of the whole scene.  Each entry in the capture set is a suggested
> alternative.
> 
> A 2 screen consumer can choose VC3 and VC4 if it wants a composition.
> It can choose VC5 and VC6 if it wants a dynamically switched view.  A 2
> screen consumer is also free to ignore the suggestions in the capture
> set and instead choose VC0 and VC1 if that is what it wants.  It could
> also choose other combinations from different entries, say VC1 and VC8,
> assuming the provider's advertised simultaneous sets and encoding
> groups do not prohibit that combination.
> 
> Mark
> 
> Paul Kyzivat

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 8 Feb 2012 10:14:17 -0800

Hi Roni,
Yes, I agree.  That is what I meant by the statement "VC3 through VC8 could also have additional attributes describing the available video layout choices for composition or the available switching policies the provider is offering.  The consumer could choose among these."

Mark

-----Original Message-----
From: Roni Even [mailto:[email-masked]]
Sent: Wednesday, February 08, 2012 11:58 AM
To: Duckworth, Mark; [email-masked]
Subject: RE: [clue] #7: Is composed attribute a boolean or data structure

Hi Mark,

1. Two out of three which will include the current and previous speaker 2. Two out of three which will rotate between the three every 10 seconds These two cannot be differentiated with binary composed and switched.

Roni

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Duckworth, Mark
> switching policies the provider is offering.  The consumer could
> choose among these.  I think that is the point Roni has been making.
> Details of this need further discussion.  (Maybe the composed and
> switched choices should be collapsed into one set of stuff to choose
> from for a particular VC, rather than separate VCs, so that would
> reduce the number of VCs here from 9 to 6 ?).
>
> which video captures are most useful to receive together, to get a
> view of the whole scene.  Each entry in the capture set is a suggested
> alternative.
>
> It can choose VC5 and VC6 if it wants a dynamically switched view.  A
> 2 screen consumer is also free to ignore the suggestions in the
> capture set and instead choose VC0 and VC1 if that is what it wants.
> It could also choose other combinations from different entries, say
> VC1 and VC8, assuming the provider's advertised simultaneous sets and
> encoding groups do not prohibit that combination.
>
> Mark
>
>   Static => The requester ask for explicit camera streams, e.g. cam1
> or
> (cam2+cam3)
>   Switched/Composed => The triple chooses the streams to send, e.g.
> the two loudest segments or it could make a nice rendering of 3x
> cameras rendered on two streams.
>
> What you are describing as "site switching" adds another wrinkle,
> since it is coupling the switching policies for multiple streams.
> Also, I'm not sure how it works if the sites have differing numbers of
> screens and cameras. How would you describe the individual streams
> from the MCU so that a room could select an appropriate set of streams
> to get site switching?
>
> >> <video-layout>, so in that case I would argue that you could offer
> >> a Capture stream with a switch-policy = {site, segment}
> >>
> >> If the provider wants to offer site switch and segment switch
> >> option to the consumer he cannot do it just with A. this is a basic
> >> use
> case.
> >>
> >>> If you’re a particular use cases requires control over the
> >>> composed layouts you need a) and b). In my example I used a single
> >>> possible layout choices. A receiver that wants the default
> >>> composed layout can request the composed stream and skip the
> >>> optional video-
> >> layout information.
> >>>> The video-layout element is a list of strings and each string is
> >>>> a layout-hint that can be requested. Video-layout is optional. A
> >>>> The video layout which just describes the geometry of the
> >>>> composed image and I think that the above references provide good
> >>>> structure to define this part of the composed video attribute.
> >>>>
> >>>> element may change dynamically by the provider this attribute
> >>>> only address the static information which is the algorithm and
> >>>> not the current content (who we see now in each element) which
> >>>> will need
> to
> >>> be
> >>>> conveyed also but probably not using this attribute. Note that
> >>>> the information is valid for point to point and multipoint so the
> >>>>> --------------------------------+-------------------------------
> >>>>> --------------------------------+-------------------------------

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 8 Feb 2012 13:27:40 -0800

Hi Marshall,

I'm trying to put your proposal in perspective of some other related issues.  Basically you are proposing a way for a media provider to advertise composed media captures, including a list of sources that go into the composition, right?  And this could potentially be done by adding some additional attributes to media captures. (I'm using the term capture, to be consistent with the framework, where you use the term stream).

So this would work for cases where the source list does not change often, right?  Or are you thinking this would work for cases where the source list in a composed capture changes due to a change in current talker or a due to a participant joining or leaving the conference?  Would you suggest the provider must advertise a media capture with new attribute values every time a change like this occurs?  I was thinking that would not be practical.

This proposal sounds similar to Stephan's proposal from the discussion "Kick-Off Item 5: description of composed pictures" about three months ago.  Do you see this as a subset of Stephan's proposal, without the x,y bitstream coordinates?  Or do you see this as something else entirely?  It seems like a subset to me.

Separate from your proposal, which you say does not cover detailed formatting information, we could also potentially define a video-layout attribute like what Roni referred to from mediactrl and xcon.  I think the two ideas could be used together.  Also we could potentially add something about switching/composing policies or algorithms, of which your special character "A = active speaker" is just one example.

Mark

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Marshall Eubanks
Sent: Tuesday, January 24, 2012 11:21 AM
To: clue issue tracker
Cc: [email-masked]; [email-masked]
Subject: Re: [clue] #7: Is composed attribute a boolean or data structure (was: Is compose attribute a boolean or data structure)

Here is a specific suggestion based on the discussion today :

Composition is described by two (or more) booleans, and associated lists

First boolean : Is the stream composed of other streams ? Yes, or no.

Second boolean : Is this composed stream composed of other streams that the offerer is also offering.
Associated List : A list of those streams.

This second boolean has no meaning and MUST be ignored  if the first boolean is missing or negative.

So, suppose there is a 4 screen unit (with streams 1-4), which also offers

- Stream 5 :  a composition of screens 1 and 2
- Stream 6 :  a composition of screens 3 and 4
- Stream 7 :  a composition of screens 1, 2, 3 and 4

So, stream 5 might have a pseudocode description like

composed=true {composed_stream_set=true {stream_set="1,2"}}

while stream 7 would be

composed=true {composed_stream_set=true {stream_set="1,2,3,4"}}

I would strongly suggest that composed streams can be nested (as indeed they are frequently in practice).

I would also suggest two "special characters" - call them

D == "data"

and

A == "active  speaker"

So, a stream 8 with

composed=true {composed_stream_set=true {stream_set="7, A, D"}}

would mean a composed combination of (a composition of all of the true
streams) + the screen containing the active speaker + a data screen.

It would be useful to allow for virtual streams (i.e., maybe in my example stream "7" only exists inside the unit and is not available to others). That would make description more straightforward.  It is not clear to me whether or not virtual streams should have a virtual attribute, or whether or not that can conveyed within the existing framework. Of course, dependency loops MUST be avoided.

The  "A" attribute would be most useful for middle boxes. If  a composition is really coming from a 4 screen unit, then "A" may be at some remote unit, so  this would be not so useful. However, if a composition is actually coming from a middle box, then A can be the speaker, and having the attribute would be useful.

I believe that this proposal would capture essentially all of the composition use cases that do not require detailed formatting information.

Regards
Marshall

On Mon, Jan 23, 2012 at 5:15 PM, clue issue tracker <[email-masked]> wrote:
> --------------------------------+-------------------------------------
> --------------------------------+-----
>  Reporter:  mary.ietf.barnes@.  |       Owner:  
> draft-ietf-clue-framework@.
>     Type:  defect              |      Status:  new
> --------------------------------+-------------------------------------
> --------------------------------+-----
>
> Ticket URL: 
> <http://trac.tools.ietf.org/wg/clue/trac/ticket/7#comment:1>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Wed, 8 Feb 2012 23:42:25 +0100

I agree with Marks summary of the provider capabilities for a triple camera system.  

Roni's comments pointed out the fact that we could apply different policies to a capture set or individual streams. We have discussed policies for switched streams and video layouts, summarized here:

Switched policies 
 - Segment 
 - Site 
 - Round robin (e.g. iterate over available segments, change every 10 sec) 

Video-layout (Inspired by xcon [1])
 - dual-view
 - single-view 
 - Hollywood-squares
 - // and so on

A model where a media producer announce the availability of a policy could work. A capture set can then announce a video-layout attribute to let a media consumer request a captures with a preferred video-layout alternative, if alternatives are available. The video-layout attribute could also be announced on the individual capture.

An example based on Marks email: 
 Capture VC7 (single stream composition) composed, not switched
 Capture set Alternative 4 (VC7) video-layout {dual-view, single-view, Hollywood-squares } 

To request the single composed stream (with default video-layout)
  VC5

To request the single composed stream, with a specific video-layout
  VC5, video-layout='dual-view'

The example Roni mentioned with detailed control of who is included in a composed layout is not mentioned in the use case document, so maybe we could see it as out of scope for now? 

Regards 

-Espen 
 [1] http://tools.ietf.org/html/draft-ietf-xcon-common-data-model-32#section-4.2.7

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Duckworth, Mark
Sent: 8. februar 2012 19:14
To: Roni Even; [email-masked]
Subject: Re: [clue] #7: Is composed attribute a boolean or data structure

Hi Roni,

Mark

-----Original Message-----
From: Roni Even [mailto:[email-masked]]
Sent: Wednesday, February 08, 2012 11:58 AM
To: Duckworth, Mark; [email-masked]
Subject: RE: [clue] #7: Is composed attribute a boolean or data structure

Hi Mark,

Roni

> Of Espen Berger (espeberg)
> Of Paul Kyzivat

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 08 Feb 2012 21:25:34 -0500

> So this would work for cases where the source list does not change often, right?  Or are you thinking this would work for cases where the source list in a composed capture changes due to a change i…

ISTM that you would report the set of sources that are *candidates* for 
composition or switching. Especially in the case of switching, the list 
of candidates will normally be greater than the number in the mix at any 
one time. At least for switching, the actual source included can be 
determined by examining the media.

For composition it is harder. Perhaps we could say that in the case of 
composition, all the described sources are included. But that only 
partially solves the problem. There may be a case where a composition 
includes a source that is itself switched. Then it could be problematic 
to discover at any particular time which source is being shown in each 
portion of the composition.

	Thanks,
	Paul

> On Mon, Jan 23, 2012 at 5:15 PM, clue issue tracker<[email-masked]>  wrote: