Subject: [clue] #7: Is compose attribute a boolean or data structure

From: "clue issue tracker" <[email-masked]>
Date: Mon, 23 Jan 2012 22:15:41 -0000

#7: Is compose attribute a boolean or data structure

 Need to determine WG consensus as to whether the composed attribute is a
 boolean (currently in the framework) or whether it should be a data
 structure describing the composed image.
 (Note: this ticket is a result of closing Issue #2).

-- 
--------------------------------+-----------------------------------------
 Reporter:  mary.ietf.barnes@…  |      Owner:  draft-ietf-clue-framework@…
     Type:  defect              |     Status:  new
 Priority:  major               |  Milestone:
Component:  framework           |    Version:
 Severity:  Active WG Document  |   Keywords:
--------------------------------+-----------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/clue/trac/ticket/7>
clue <http://tools.ietf.org/wg/clue/>

---

From: "clue issue tracker" <[email-masked]>
Date: Mon, 23 Jan 2012 22:15:56 -0000

#7: Is composed attribute a boolean or data structure

-- 
--------------------------------+------------------------------------------
 Reporter:  mary.ietf.barnes@…  |       Owner:  draft-ietf-clue-framework@…
     Type:  defect              |      Status:  new
 Priority:  major               |   Milestone:
Component:  framework           |     Version:
 Severity:  Active WG Document  |  Resolution:
 Keywords:                      |
--------------------------------+------------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/clue/trac/ticket/7#comment:1>
clue <http://tools.ietf.org/wg/clue/>

---

From: "clue issue tracker" <[email-masked]>
Date: Mon, 23 Jan 2012 22:43:27 -0000

Changes (by mary.ietf.barnes@…):

 * type:  defect => task

     Type:  task                |      Status:  new

Ticket URL: <http://trac.tools.ietf.org/wg/clue/trac/ticket/7#comment:2>
clue <http://tools.ietf.org/wg/clue/>

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 24 Jan 2012 11:20:58 -0500

Here is a specific suggestion based on the discussion today :

Composition is described by two (or more) booleans, and associated lists

First boolean : Is the stream composed of other streams ? Yes, or no.

Second boolean : Is this composed stream composed of other streams
that the offerer is also offering.
Associated List : A list of those streams.

This second boolean has no meaning and MUST be ignored  if the first
boolean is missing or negative.

So, suppose there is a 4 screen unit (with streams 1-4), which also offers

- Stream 5 :  a composition of screens 1 and 2
- Stream 6 :  a composition of screens 3 and 4
- Stream 7 :  a composition of screens 1, 2, 3 and 4

So, stream 5 might have a pseudocode description like

composed=true {composed_stream_set=true {stream_set="1,2"}}

while stream 7 would be

composed=true {composed_stream_set=true {stream_set="1,2,3,4"}}

I would strongly suggest that composed streams can be nested (as
indeed they are frequently in practice).

I would also suggest two "special characters" - call them

D == "data"

and

A == "active  speaker"

So, a stream 8 with

composed=true {composed_stream_set=true {stream_set="7, A, D"}}

would mean a composed combination of (a composition of all of the true
streams) + the screen containing the active speaker + a data screen.

It would be useful to allow for virtual streams (i.e., maybe in my
example stream "7" only exists inside the unit and is not available to
others). That would make description more straightforward.  It is not
clear to me whether or not virtual streams should have a virtual
attribute, or whether or not that can conveyed within the existing
framework. Of course, dependency loops MUST be avoided.

The  "A" attribute would be most useful for middle boxes. If  a
composition is really coming from a 4 screen unit, then "A" may be at
some remote unit, so  this would be not so useful. However, if a
composition is actually coming from a middle box, then A can be the
speaker, and having the attribute would be useful.

I believe that this proposal would capture essentially all of the
composition use cases that do not require detailed formatting
information.

Regards
Marshall

On Mon, Jan 23, 2012 at 5:15 PM, clue issue tracker
<[email-masked]> wrote:
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 24 Jan 2012 11:46:23 -0500

[resend, tying into the #7 thread]

Based on what I have heard, I see the following possibilities:

1) there is no explicit indication of whether streams are composed or 
not. But still a composed stream will ideally indication the portion of 
the scene it covers, which would presumably be a union of those for the 
streams being composed. This could be nasty if that portion isn't a 
rectangle. (E.g. composition of left and right streams, but not the 
center screen.) It also is a problem if the streams being composed 
aren't part of the same scene. In that case it can't really describe the 
geometry of the composed scene at all. In the case of switching, such as 
active speaker, presumably the portion of the scene reported would be 
the union of all the potential speakers.

2) same as (1), but with a boolean indication for composed streams. IMO 
this doesn't add anything useful.

3) same as (1) but with addition of a free text description of what the 
stream contains. This isn't of any use for automated selection of 
streams to receive/display, but at least it can enable a user interface 
that can allow an end user in the receiving room to manually configure 
what streams should be displayed. Not ideal, but perhaps still useful 
for dealing with interop problems.

4) enumeration of the streams that are inputs to the composition of a 
composed stream. This is independent of the composition algorithm. 
stream that tiles the input streams would be described the same as one 
that switches among them. This would require that the streams that go 
into the composition also be (separately) described. This would be no 
issue if they are also available for independent selection. But if there 
are to be cases where the composition is available but the independent 
streams are not, then this would require more work.

5) description of how the composition is accomplished. It might be time 
or space multiplexing of the pixels. If time multiplexing, then the 
algorithm for doing so (active speaker, ...). If space multiplexing, 
then it could include which region of the composed stream contains each 
composed stream. There could also be a combination of time and space 
multiplexing. And this could also be hierarchical, if one or more of the 
input streams for composition are themselves composed.

	Thanks,
	Paul

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 24 Jan 2012 11:59:56 -0500

On Tue, Jan 24, 2012 at 11:46 AM, Paul Kyzivat <[email-masked]> wrote:
> 1) there is no explicit indication of whether streams are composed or not.
> But still a composed stream will ideally indication the portion of the scene
> it covers, which would presumably be a union of those for the streams being
> composed. This could be nasty if that portion isn't a rectangle. (E.g.
> composition of left and right streams, but not the center screen.) It also
> is a problem if the streams being composed aren't part of the same scene. In
> that case it can't really describe the geometry of the composed scene at
> all. In the case of switching, such as active speaker, presumably the
> portion of the scene reported would be the union of all the potential
> speakers.
>

I don't like presuming things in this context.

> 2) same as (1), but with a boolean indication for composed streams. IMO this
> doesn't add anything useful.
>

I agree.

> stream contains. This isn't of any use for automated selection of streams to
> receive/display, but at least it can enable a user interface that can allow
> an end user in the receiving room to manually configure what streams should
> be displayed. Not ideal, but perhaps still useful for dealing with interop
> problems.

I think that text descriptions are both useful and not reliable. Any
actual text is worthwhile, but
as you cannot rely on it as

- it may not be there  or
- it may not make sense or be relevant or
- it may be "canned," from some default or some previous session, and
thus not apply to current events.

So, sure, allow for this, but it doesn't really solve any problems.

> composed stream. This is independent of the composition algorithm. stream
> that tiles the input streams would be described the same as one that
> switches among them. This would require that the streams that go into the
> composition also be (separately) described. This would be no issue if they
> are also available for independent selection. But if there are to be cases
> where the composition is available but the independent streams are not, then
> this would require more work.

I am going to paste MY proposal here, as it is for this case

while stream 7 would be

D == "data"

and

A == "active  speaker"

So, a stream 8 with

>
> 5) description of how the composition is accomplished. It might be time or
> space multiplexing of the pixels. If time multiplexing, then the algorithm
> for doing so (active speaker, ...). If space multiplexing, then it could
> include which region of the composed stream contains each composed stream.
> There could also be a combination of time and space multiplexing. And this
> could also be hierarchical, if one or more of the input streams for
> composition are themselves composed.
>

I frankly think that this is too early, as this parameter space is
vast, and we would need to know just what pieces
of it to select.

Regards
Marshall

---

From: "Roni Even" <[email-masked]>
Date: Mon, 30 Jan 2012 12:18:14 +0200

Hi,
During the last call I volunteered to provide some input. 

In current IETF work (XCON, Mediactrl WGs there is a video layout element)

http://tools.ietf.org/html/draft-ietf-mediactrl-mixer-control-package-14#section-4.2.1.4.2.1

and 
http://tools.ietf.org/html/draft-ietf-xcon-common-data-model-32#section-4.2.7 (look at video-layout)

I would like first to try to define the term "composed video" since it looks to me like we have different views here, and provide my initial view on what should be described.

Composed video can describes both the layout and the selection algorithm for composing the  content of the sub-windows in the "mixed" video. 

The video layout which just describes the geometry of the composed image and I think that the above references provide good structure to define this part of the composed video attribute.

The other part is the algorithm by which the provider select the content of each element in the layout. Since the content of each element may change dynamically by the provider this attribute only address the static information which is the algorithm and not the current content (who we see now in each element) which will need to be conveyed also but probably not using this attribute. Note that the information is valid for point to point and multipoint so the current content should reflect the TP end point and the specific VC used from it.

The algorithms may be global or per element(or sub-window). The global algorithms I see are site switch or segment switch. The per element may be voice activated, round robin (switch every x seconds) and fixed (the same VC is displayed there (may be changed by some control mechanism)

Roni

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> clue issue tracker
> Sent: Tuesday, January 24, 2012 12:43 AM
> To: [email-masked];
> [email-masked]
> Cc: [email-masked]
> Subject: Re: [clue] #7: Is composed attribute a boolean or data
> structure
> 
> Ticket URL:
> <http://trac.tools.ietf.org/wg/clue/trac/ticket/7#comment:2>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Mon, 30 Jan 2012 15:17:06 +0100

To help with the direction of the discussions I think it's useful to divide the discussions into three use cases for composed streams. 
 a) How to express if a capture stream is composed or not 
 b) How to express layout choices? (e.g. with a video-layout element)  
 c) How to express what's inside a composed stream 

For use case a) a composed attribute should be sufficient, you either ask for the composed stream or you do not. 
I also believe this should cover the interoperability requirement we have across multiple types of endpoints. 

For use case b) both mediactrl and xcon uses a <video-layout> element to describe possible layouts and also to request the preferred layout based on options. (as described in Roni's email)

An example could be (inspired by mediactrl and xcon usage of <video-layout>): 

CLUE Advertisement
  Capture id=2 Purpose=people Composed=false 
  Capture id=3 Purpose=people Composed=false 
  Capture id=4 Purpose=People Composed=true  
      Video-layout="'automatic', 'dual-view', ' single-view'"  

CLUE configure // Default composed stream 
  Capture id=4 

// Or composed stream with layout hints
   Capture id=4 Video-layout='dual-view' 

The video-layout element is a list of strings and each string is a layout-hint that can be requested. Video-layout is optional. A layout hints about the requested rendering and the source is free to replace any stream as long as they fit the layout.

Use case c) is more open in the sense that what you actually receive in a composed stream is depending on status of a room or who is in a MCU conference. E.g. from a room you could get a mix of different capture stream representing cameras and from a transcoding MCU it could be different room, different cameras or another mix of possible input video streams.

Having support for both b) and c) could be a good test for the extensibility of CLUE. If the basis of CLUE only does use case a) there should be room to extend CLUE with support for b) and c) as a CLUE + layout description extension. 

Cheers 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Roni Even
Sent: 30. januar 2012 11:18
To: [email-masked]
Subject: Re: [clue] #7: Is composed attribute a boolean or data structure

Hi,

and 

Roni

---

From: "Roni Even" <[email-masked]>
Date: Mon, 30 Jan 2012 16:38:32 +0200

Hi Espen,
If you will look at the notes from the last call there was a support that a) is not enough. Having just compose does not address even the use cases I provided which are also based on the use case draft. 

I am not sure what you mean by C since what in the composed stream can be either which VC you see or what is the selection criteria for being in a composed stream. If you meant the second, my view is that C and also b should be in the basic framework and not in an extension.

Roni Even

> From: Espen Berger (espeberg) [mailto:[email-masked]]
> Sent: Monday, January 30, 2012 4:17 PM
> To: Roni Even; [email-masked]
> Subject: RE: [clue] #7: Is composed attribute a boolean or data
> structure
> 
> 
> To help with the direction of the discussions I think it's useful to
> divide the discussions into three use cases for composed streams.
> For use case a) a composed attribute should be sufficient, you either
> ask for the composed stream or you do not.
> For use case b) both mediactrl and xcon uses a <video-layout> element
> to describe possible layouts and also to request the preferred layout
> based on options. (as described in Roni's email)
> 
> The video-layout element is a list of strings and each string is a
> layout-hint that can be requested. Video-layout is optional. A layout
> hints about the requested rendering and the source is free to replace
> any stream as long as they fit the layout.
> 
> Use case c) is more open in the sense that what you actually receive in
> a composed stream is depending on status of a room or who is in a MCU
> conference. E.g. from a room you could get a mix of different capture
> stream representing cameras and from a transcoding MCU it could be
> different room, different cameras or another mix of possible input
> video streams.
> 
> Having support for both b) and c) could be a good test for the
> extensibility of CLUE. If the basis of CLUE only does use case a) there
> should be room to extend CLUE with support for b) and c) as a CLUE +
> layout description extension.
> 
> Cheers
> 
> -Espen
> 
> 
> 
> Roni Even
> I would like first to try to define the term "composed video" since it
> looks to me like we have different views here, and provide my initial
> view on what should be described.
> 
> Composed video can describes both the layout and the selection
> algorithm for composing the  content of the sub-windows in the "mixed"
> video.
> 
> The video layout which just describes the geometry of the composed
> image and I think that the above references provide good structure to
> define this part of the composed video attribute.
> 
> The other part is the algorithm by which the provider select the
> content of each element in the layout. Since the content of each
> element may change dynamically by the provider this attribute only
> address the static information which is the algorithm and not the
> current content (who we see now in each element) which will need to be
> conveyed also but probably not using this attribute. Note that the
> information is valid for point to point and multipoint so the current
> content should reflect the TP end point and the specific VC used from
> it.
> 
> The algorithms may be global or per element(or sub-window). The global
> algorithms I see are site switch or segment switch. The per element may
> be voice activated, round robin (switch every x seconds) and fixed (the
> same VC is displayed there (may be changed by some control mechanism)
> 
> 
> Roni
> 
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of clue issue tracker
> > --------------------------------+------------------------------------
> -
> > --------------------------------+-
> > ----
> > --------------------------------+------------------------------------
> -
> > --------------------------------+-
> > ----
> >
> > Ticket URL:

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Mon, 30 Jan 2012 16:20:58 +0100

Hi Roni 

If you’re a particular use cases requires control over the composed layouts you need a) and b). In my example I used a single advertisement composed stream with optional meta-information about possible layout choices. A receiver that wants the default composed layout can request the composed stream and skip the optional video-layout information. Optionally you could request the same composed stream with layouts hints for the receiver. 

For me use case c) does not include the selection mechanisms, only the content you see in the video stream. 

Cheers 

-Espen 

-----Original Message-----
From: Roni Even [mailto:[email-masked]] 
Sent: 30. januar 2012 15:39
To: Espen Berger (espeberg); [email-masked]
Subject: RE: [clue] #7: Is composed attribute a boolean or data structure

Hi Espen,

Roni Even

---

From: "Roni Even" <[email-masked]>
Date: Tue, 31 Jan 2012 01:06:04 +0200

Hi Espen,
If the provider wants to offer site switch and segment switch option to the consumer he cannot do it just with A. this is a basic use case.

In general, your option does not provide any interoperability since both sides may have different views what composed means. In the framework example of a PIP as composed image it is an assumption that if the provider offers this composed offer, the consumer can understand that it is a PIP mix but there is nothing in the offer that will indicate that it is.

Roni 

> Sent: Monday, January 30, 2012 5:21 PM
> If you’re a particular use cases requires control over the composed
> layouts you need a) and b). In my example I used a single advertisement
> composed stream with optional meta-information about possible layout
> choices. A receiver that wants the default composed layout can request
> the composed stream and skip the optional video-layout information.
> Optionally you could request the same composed stream with layouts
> hints for the receiver.
> 
> If you will look at the notes from the last call there was a support
> that a) is not enough. Having just compose does not address even the
> use cases I provided which are also based on the use case draft.
> 
> 
> I am not sure what you mean by C since what in the composed stream can
> be either which VC you see or what is the selection criteria for being
> in a composed stream. If you meant the second, my view is that C and
> also b should be in the basic framework and not in an extension.
> 
> Roni Even
> 
> > Use case c) is more open in the sense that what you actually receive
> > in a composed stream is depending on status of a room or who is in a
> > MCU conference. E.g. from a room you could get a mix of different
> > capture stream representing cameras and from a transcoding MCU it
> > could be different room, different cameras or another mix of possible
> > input video streams.
> >
> > extensibility of CLUE. If the basis of CLUE only does use case a)
> > there should be room to extend CLUE with support for b) and c) as a
> > CLUE + layout description extension.
> >
> > Cheers
> >
> > -Espen
> >
> >
> >
> > Of Roni Even
> > http://tools.ietf.org/html/draft-ietf-mediactrl-mixer-control-
> package-
> > 14#section-4.2.1.4.2.1
> >
> > and
> > algorithms I see are site switch or segment switch. The per element
> > may be voice activated, round robin (switch every x seconds) and
> fixed
> > (the same VC is displayed there (may be changed by some control
> > mechanism)
> >
> >
> > Roni
> >
> > > --------------------------------+----------------------------------
> -
> > > --------------------------------+----------------------------------
> -

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Tue, 31 Jan 2012 16:37:57 +0100

Hi Rony 

To choose between site and segment switch is more a policy than a <video-layout>, so in that case I would argue that you could offer a Capture stream with a switch-policy = {site, segment}

My point about interoperability is that a single composed stream with alternative layouts are easier to understand, than multiple capture streams with alternative configurations. The last requires to pick the first one or requires some sort of default marking.  

Cheers 

-Espen 

-----Original Message-----
From: Roni Even [mailto:[email-masked]] 
Sent: 31. januar 2012 00:06
To: Espen Berger (espeberg); [email-masked]
Subject: RE: [clue] #7: Is composed attribute a boolean or data structure

Hi Espen,

Roni