Subject: [clue] Layout negotiation

From: Christer Holmberg <[email-masked]>
Date: Tue, 21 Jun 2011 12:51:42 +0200

Hi,

A while ago there were some discussions about having requirements to negotiate the layout type, and at least according to Stephen B it was within the scope of the work.

        "The ability to negotiate an audio or video layout is clearly within scope, and has general value it the group wants to take it on.
        For instance, if you combine video switching of multiple sources with audio mixing/transcoding, then the receiver is responsible
        for the video layout but has no control over the audio layout.  In that case, there are advantages to allowing the receiver to tell the
        central audio mixer where it wants the sources placed on the sound stage.  I have no objection in principle to adding that kind of
        negotiation to the requirements, though I think the group should consider the impact on the schedule."

However, I haven't seen any input after that.

My idea would be that it is very similar to the rendering type: CLUE defines a mechanism to negotiate the layout type, but doesn't define specific types (instead an IANA registry would be created for that). So, I don't think it would have any significant impact on our schedule.

Whether it, in addition to negotiation a layout type (which specifies the source locations), should be possible for the user to more explicit place individual sources, as mentioned by Stephen, can be discussed. However, that would probably require a little more work, so at least I would be happy with a simple layout type negotiation at this point.

So, would people be ok with such requirement?

(Another alternative would be to "embed" the layout type into the rendering type, but I think that would be very clumsy. Because, in that case you would need to define mulitple types for the same rendering, but with different layouts.)

Regards,

Christer

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Tue, 21 Jun 2011 08:45:59 -0700

Hi Christer,

I think I was the one that brought up layouts originally. I was more
concerned with video layouts than audio layouts. Currently, the
requirements draft seems to focus more on audio layout/spatial audio.
Requirements 3 and 16 both deal with parts of this. I feel there needs
to be more clear requirements on the ability for the to represent the
potential layouts of the audio/video can that be offered. For example, I
have 3 camera and 3 mics. I can provide:
1) 3 individual video each with corresponding audio
2) 1 active speaker only switched video with mixed audio
3) 1 composed video including active speaker and 'n' non-active speakers
(where 'n' is may be all or some fixed limit) and mixed audio

If the receiving endpoint wants to control the layout itself, it goes
with option 1. If it has limited resources/bandwidth/screens/etc it may
prefer option 2 or 3.

Cheers,
Charles 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of Christer Holmberg
> Sent: Tuesday, June 21, 2011 3:52 AM
> To: [email-masked]
> Subject: [clue] Layout negotiation
> 
> Hi,
> 
> A while ago there were some discussions about having requirements to
negotiate the layout type, and at
> least according to Stephen B it was within the scope of the work.
> 
>         "The ability to negotiate an audio or video layout is clearly
within scope, and has general
> value it the group wants to take it on.
>         For instance, if you combine video switching of multiple
sources with audio
> mixing/transcoding, then the receiver is responsible
>         for the video layout but has no control over the audio layout.
In that case, there are
> advantages to allowing the receiver to tell the
>         central audio mixer where it wants the sources placed on the
sound stage.  I have no objection
> in principle to adding that kind of
>         negotiation to the requirements, though I think the group
should consider the impact on the
> schedule."
> My idea would be that it is very similar to the rendering type: CLUE
defines a mechanism to negotiate
> the layout type, but doesn't define specific types (instead an IANA
registry would be created for
> that). So, I don't think it would have any significant impact on our
schedule.
> 
> Whether it, in addition to negotiation a layout type (which specifies
the source locations), should be
> possible for the user to more explicit place individual sources, as
mentioned by Stephen, can be
> discussed. However, that would probably require a little more work, so
at least I would be happy with
> a simple layout type negotiation at this point.
> 
> (Another alternative would be to "embed" the layout type into the
rendering type, but I think that
> would be very clumsy. Because, in that case you would need to define
mulitple types for the same
> rendering, but with different layouts.)
> 
> Regards,
> 
> Christer
> 
>

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 12:09:53 +0200

Hi Charles, 

>I think I was the one that brought up layouts originally. I 
>was more concerned with video layouts than audio layouts. 
>Currently, the requirements draft seems to focus more on 
>audio layout/spatial audio.
>Requirements 3 and 16 both deal with parts of this. I feel 
>there needs to be more clear requirements on the ability for 
>the to represent the potential layouts of the audio/video can 
>that be offered. For example, I have 3 camera and 3 mics. I 
>can provide:
>3) 1 composed video including active speaker and 'n' 
>non-active speakers (where 'n' is may be all or some fixed 
>limit) and mixed audio

I actually think that your examples, as written, are different RENDERING TYPES, because they only talk about mixing and composition.

In my opinion, a LAYOUT TYPE is a description of how sources are placed in a virtual room.

Of course, in order to do the mixing, a layout description is often needed as an "input" for the rendering algorithm.

Regards,

Christer

>If the receiving endpoint wants to control the layout itself, 
>it goes with option 1. If it has limited 
>resources/bandwidth/screens/etc it may prefer option 2 or 3.
> 
> Cheers,
> Charles 
>

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 22 Jun 2011 10:44:22 -0400

On Wed, Jun 22, 2011 at 6:09 AM, Christer Holmberg <
[email-masked]> wrote:

> I actually think that your examples, as written, are different RENDERING
> TYPES, because they only talk about mixing and composition.
>
Confirmation that I am clueless about rendering types.

> In my opinion, a LAYOUT TYPE is a description of how sources are placed in
For me layout is about composition.  Layouts do not necessarily create a
virtual room experience, particularly in multipoint calls.

>
> Of course, in order to do the mixing, a layout description is often needed
> as an "input" for the rendering algorithm.
>
> Regards,
>
> Christer
>
>
>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 17:17:10 +0200

Hi, 
		
>Confirmation that I am clueless about rendering types. 

A value identifying the rendering algorithm used to produce a media stream sent towards the client.
	
>For me layout is about composition. Layouts do not necessarily create a virtual room experience, particularly in multipoint calls. 

Ok, so we need to make sure we have a common understanding of layout then :)

So, just to clarify, do you think that source selection is layout?
	
Regards,

Christer

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 22 Jun 2011 11:40:50 -0400

> A value identifying the rendering algorithm used to produce a media stream
In a client-to-client call this is backwards.  "Capture" describes the
stream production process better, rendering is that the client does with the
streams it receives.  Anyway, changing "Rendering type" to "rendering
algorithm" doesn't help me much.

> So, just to clarify, do you think that source selection is layout?
I generally separate them, though they are related.  For instance, in a
traditional Video MCU you can often choose a source-independent layout (2x2
array of equal size windows, a 1+5 layout, where the "1" window is bigger
than the others, and at the top left, etc).  Then you can control the
mapping of sources to windows separately.  Not sure this is the only way to
partition it, just the way I tend to use.

I see four distinct elements in your mobile binaural case - (a) the sources
chosen for placement, (b) the set of placements used in creating a sound
stage, (c) the placement used for each particular source at the moment, and
(d) whether the outbound audio is binaural or not.  I'd personally call (b)
the audio layout, though you can equally well call (b+c) the audio
layout.

>                > >         "The ability to negotiate an audio or video
> layout
>                > is clearly
>                > > Whether it, in addition to negotiation a layout type
> (which
>                > specifies

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 18:46:02 +0200

Hi, 

>In a client-to-client call this is backwards.  "Capture" describes the stream production process better, rendering is that the client does with the 
>streams it receives. Anyway, changing "Rendering type" to "rendering algorithm" doesn't help me much. 

It's probably easier to understand rendering type if one knows what we mean by render.

Our proposed definition is:

"Rendering:   Composing a media signal from one or more media signals using a specific composition algorithm, often according to a specific spatial model/layout."

So, the "rendering type" is a value associated with that algorithm.

So, I think the important thing in our "rendering" definition is that an output media signal is generated.

Now, if people think that our usage of "rendering" is confusing, we can of course use other wording instead.

>I generally separate them, though they are related.  For instance, in a traditional Video MCU you can often choose a source-independent layout (2x2 array 
>of equal size windows, a 1+5 layout, where the "1" window is bigger than the others, and at the top left, etc).  Then you can control the mapping of 
>sources to windows separately.  Not sure this is the only way to partition it, just the way I tend to use.  

Ok. So, first, I guess there is "source-dependent layout" and "source-independent layout".

Second, there is an algorithm that creates the output stream, using the layout as input.

Of course, in this case one might not need to be able to explicitly indicate the algorithm - if it's enough to indicate the layout.

>I see four distinct elements in your mobile binaural case - (a) the sources chosen for placement,

Correct. That's "source selection".

>(b) the set of placements used in creating a sound stage,
>
>(c) the placement used for each particular source at the moment, 
>and (d) whether the outbound audio is binaural or not.
>I'd personally call (b) the audio layout, though you can equally well call (b+c) the audio layout.    

Ok. So, that would be a "source-dependent layout", I guess?

Then, (d) would be the actual rendering algorithm, using the layout as input.

Regards,

Christer

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Wed, 22 Jun 2011 18:49:19 +0200

The CLUE definition document talks about both source selection,
rendering and layouts. 

Using the terms from that document I do not see source selection as a
part of layout, but they are used together to create a good user
experience. 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Christer Holmberg
Sent: 22. juni 2011 17:17
To: Stephen Botzko
Cc: [email-masked]
Subject: Re: [clue] Layout negotiation

Hi, 
		
>I actually think that your examples, as written, are different
RENDERING TYPES, because they only talk about mixing and composition.
>Confirmation that I am clueless about rendering types. 

A value identifying the rendering algorithm used to produce a media
stream sent towards the client.

Ok, so we need to make sure we have a common understanding of layout
then :)

Regards,

Christer

layout
		> >         for the video layout but has no control over
the
		> audio layout.
(which
		> > discussed. However, that would probably require a
little
		> more work, so

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Wed, 22 Jun 2011 10:03:00 -0700

> From: Christer Holmberg [mailto:[email-masked]]
> Sent: Wednesday, June 22, 2011 9:46 AM
> To: Stephen Botzko
> Cc: Charles Eckel (eckelcu); [email-masked]
> Subject: RE: [clue] Layout negotiation
> 
> 
> Hi,
> 
> >In a client-to-client call this is backwards.  "Capture" describes the stream production process
> better, rendering is that the client does with the
> "Rendering:   Composing a media signal from one or more media signals using a specific composition
> algorithm, often according to a specific spatial model/layout."
> 

This works for me. It may be useful to expand with typical audio and video examples, such as mixing multiple audio stream inputs into a single audio stream output, or composing multiple video stream inputs into a single video stream output.

Cheers,
Charles

> >I generally separate them, though they are related.  For instance, in a traditional Video MCU you can
> often choose a source-independent layout (2x2 array
> >of equal size windows, a 1+5 layout, where the "1" window is bigger than the others, and at the top
> left, etc).  Then you can control the mapping of

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 20:17:36 +0200

Hi, 

> > >>>I think I was the one that brought up layouts originally. I was 
> > >>>more concerned with video layouts than audio layouts.
> > >>>Currently, the requirements draft seems to focus more on audio 
> > >>>layout/spatial audio.
> > >>>Requirements 3 and 16 both deal with parts of this. I feel there 
> > >>>needs to be more clear requirements on the ability for the to 
> > >>>represent the potential layouts of the audio/video can that be 
> > >>>offered. For example, I have 3 camera and 3 mics. I can provide:
> > >>A value identifying the rendering algorithm used to 
> produce a media stream sent towards the client.
> > 
> > >streams it receives. Anyway, changing "Rendering type" to 
> "rendering algorithm" doesn't help me much.
> > 
> > "Rendering:   Composing a media signal from one or more 
> media signals 
> > using a specific composition algorithm, often according to 
> a specific spatial model/layout."
> > 
> > So, I think the important thing in our "rendering" 
> definition is that 
> > an output media signal is generated.
> > 
> This works for me. It may be useful to expand with typical 
> audio and video examples, such as mixing multiple audio 
> stream inputs into a single audio stream output, or composing 
> multiple video stream inputs into a single video stream output.

I did try to provide some examples in the "Definition and examples" e-mail.

Regards,

Christer

> > >>>For me layout is about composition. Layouts do not necessarily 
> > >>>create a virtual room experience,
> > particularly in multipoint calls.
> > >>
> > >>
> > >I generally separate them, though they are related.  For 
> instance, in 
> > >a traditional Video MCU you can
> > >of equal size windows, a 1+5 layout, where the "1" window 
> is bigger 
> > >than the others, and at the top
> > >sources to windows separately.  Not sure this is the only way to 
> > Of course, in this case one might not need to be able to explicitly 
> > indicate the algorithm - if it's enough to indicate the layout.
> > 
> > 
> > 
> > 
> > >I see four distinct elements in your mobile binaural case 
> - (a) the 
> > >sources chosen for placement,
> > 
> > >(c) the placement used for each particular source at the 
> moment, and 
> > >(d) whether the outbound audio is binaural or not.
> > >I'd personally call (b) the audio layout, though you can 
> equally well call (b+c) the audio layout.
> > 
> > 		               > >         "The ability to 
> negotiate an audio or video layout
> > 		               > is clearly
> > 		               > >         for the video layout 
> but has no control over the
> > 		               > audio layout.
> > 		               > In that case, there are
> > 		               > > Whether it, in addition to 
> negotiation a layout type (which
> > 		               > specifies
> > 		               > > possible for the user to 
> more explicit place individual sources, as
> > 		               > > discussed. However, that 
> would probably require a little
> > 		               > more work, so
> > 		               > > would be very clumsy. 
> Because, in that case you would need to define

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 20:36:31 +0200

Hi, 

>The CLUE definition document talks about both source 
>selection, rendering and layouts. 
> 
>Using the terms from that document I do not see source 
>selection as a part of layout, but they are used together to 
>create a good user experience. 

Yes.

And, the idea is that the rendering algorithm could use the source selection and layout as input when producing the output stream.

Of course, source selection might not be needed for all algorithms.

Regards,

Christer

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Christer Holmberg
> was more 
> >>concerned with video layouts than audio layouts.
> there needs 
> >>potential layouts of the audio/video can that be offered. 
> For example, 
> >>I have 3 camera and 3 mics. I can provide:
> 		> > Whether it, in addition to negotiation a 
> layout type (which
> 		> specifies
> 		> > discussed. However, that would probably 
> require a little
> 		> more work, so

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 22 Jun 2011 14:55:44 -0400

I'm not sure yet that we need any requirements on creating a general
mechanism just yet, but that we should continue to develop the use cases.

Stephen B.

On Wed, Jun 22, 2011 at 1:03 PM, Charles Eckel (eckelcu)
<[email-masked]>wrote:

> stream production process
> > So, I think the important thing in our "rendering" definition is that an
> output media signal is
> > generated.
> >
> > Now, if people think that our usage of "rendering" is confusing, we can
> of course use other wording
> > instead.
>
> This works for me. It may be useful to expand with typical audio and video
> examples, such as mixing multiple audio stream inputs into a single audio
> stream output, or composing multiple video stream inputs into a single video
> stream output.
>
> Cheers,
> Charles
>
>
> >
> >
> >
> > >>>For me layout is about composition. Layouts do not necessarily create
> a virtual room experience,
> traditional Video MCU you can
> >                              > >         "The ability to negotiate an
> audio or video layout
> >                              > is clearly
> >                              > >         for the video layout but has no
> control over the
> >                              > audio layout.
> >                              > In that case, there are

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Wed, 22 Jun 2011 12:31:20 -0700

Huh?
Perhaps one of you could summarize where we are in the conversation, and what is agreed upon so far?

One thing to remind you of is that we have agreed that CLUE does not specify how rendering should be done, that is up to the receiver. The sender (and receiver) can send info, but CLUE isn't specifying what the receiver (renderer) does with it, meaning how it does  the rendering, what choices it makes. I thought Charles' original email may have been suggesting specifying rendering, wasn't really sure.

Thanks,
Allyn

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Charles Eckel (eckelcu)
Sent: Wednesday, June 22, 2011 10:03 AM
To: Christer Holmberg; Stephen Botzko
Cc: [email-masked]
Subject: Re: [clue] Layout negotiation

Cheers,
Charles

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 21:43:57 +0200

Hi Allyn, 

>Huh?
>Perhaps one of you could summarize where we are in the 
>conversation, and what is agreed upon so far?

I think we're having more or less a definition discussion.

>One thing to remind you of is that we have agreed that CLUE 
>does not specify how rendering should be done, that is up to 
>the receiver. The sender (and receiver) can send info, but 
>CLUE isn't specifying what the receiver (renderer) does with 
>it, meaning how it does  the rendering, what choices it 
>makes. I thought Charles' original email may have been 
>suggesting specifying rendering, wasn't really sure.

I think what charles is talking is about being able to indicate:

1. Which streams are used to create an output stream
2. How does the output stream look like

It seems like people consider that being part of LAYOUT.

Regards,

Christer

> Behalf Of Charles Eckel (eckelcu)

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Wed, 22 Jun 2011 14:31:48 -0700

Here is the current definition of layout from http://www.ietf.org/id/draft-wenger-clue-definitions-00.txt:

        Layout: How rendered media streams are spatially arranged with
        respect to each other on a single screen/mono audio telepresence
        endpoint, and how rendered media streams are arranged with
        respect to each other on a multiple screen/speaker telepresence
        endpoint.  Note that audio as well as video is encompassed by
        the term layout--in other words, included is the placement of
        audio streams on speakers as well as video streams on video
        screens.  Edit. note: this is on the RENDERER side.

This definition is from the perspective of the physical speakers and displays on the endpoint. I am thinking more about how the available media streams are encoded and transmitted as some subset of the individual media streams, or as some composition of the individual media streams. It is still up to the receiving endpoint to decide the layout of these media streams with respect to its physically speakers and displays. 

Cheers,
Charles

> Sent: Wednesday, June 22, 2011 12:44 PM
> To: Allyn Romanow (allyn); Charles Eckel (eckelcu); Stephen Botzko
> Cc: [email-masked]

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Wed, 22 Jun 2011 21:12:21 -0700

Charles,

What does your note have to do with the requirements?
It seems to me you're out there designing a solution. Could you save it for the design discussion?
Or, perhaps you could state succinctly what statement you want added, subtracted, or modified in the requirements?

Thanks,
Allyn

> From: Charles Eckel (eckelcu)
> Sent: Wednesday, June 22, 2011 2:32 PM
> To: Christer Holmberg; Allyn Romanow (allyn); Stephen Botzko
> Cc: [email-masked]
> Here is the current definition of layout from
> http://www.ietf.org/id/draft-wenger-clue-definitions-00.txt:
> 
> This definition is from the perspective of the physical speakers and
> displays on the endpoint. I am thinking more about how the available
> media streams are encoded and transmitted as some subset of the
> individual media streams, or as some composition of the individual
> media streams. It is still up to the receiving endpoint to decide the
> layout of these media streams with respect to its physically speakers
> and displays.
> 
> Cheers,
> Charles
> 
> was
> there
> > > > >In a client-to-client call this is backwards.  "Capture"
> describes
> > > > >the stream production process
> > > > Now, if people think that our usage of "rendering" is confusing,
> we
> > > > can of course use other wording instead.
> > >

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 23 Jun 2011 11:39:21 +0200

Hi Charles, 

>         respect to each other on a single screen/mono audio 
>         telepresence endpoint, and how rendered media streams are arranged with
>         respect to each other on a multiple screen/speaker 
>         telepresence endpoint.  Note that audio as well as video is encompassed by
> This definition is from the perspective of the physical 
> speakers and displays on the endpoint. I am thinking more 
> about how the available media streams are encoded and 
> transmitted as some subset of the individual media streams, 
> or as some composition of the individual media streams.

Exactly, that is what I am thinking about also. The layout used by an MCU to genereate the output stream(s) towards an endpoint.

I have called that "rendering algorithm", but I don't really care what we call it - as long as we all have a common understanding of what we mean :)

>It is still up to the receiving endpoint to decide the layout of 
>these media streams with respect to its physically speakers 
>and displays. 

Yes.

Regards,

Christer

> > >One thing to remind you of is that we have agreed that 
> CLUE does not 
> > >specify how rendering should be done, that is up to the 
> receiver. The 
> > >sender (and receiver) can send info, but CLUE isn't 
> specifying what 
> > >the receiver (renderer) does with it, meaning how it does  the 
> > >rendering, what choices it makes. I thought Charles' 
> original email 
> > >may have been suggesting specifying rendering, wasn't really sure.
> > 
> > 1. Which streams are used to create an output stream 2. How 
> does the 
> > output stream look like
> > 
> > > Of Charles Eckel (eckelcu)
> > > > >>>that be offered. For example, I have 3 camera and 3 
> mics. I can provide:
> > > > >describes the stream production process
> > > > we can of course use other wording instead.
> > >
> > > This works for me. It may be useful to expand with 
> typical audio and 
> > > video examples, such as mixing multiple audio stream 
> inputs into a 
> > > single audio stream output, or composing multiple video stream 
> > > inputs into a single video stream output.
> > >
> > > Cheers,
> > > Charles
> > >
> > >
> > > >
> > > >
> > > >
> > > > Of course, in this case one might not need to be able to 
> > > > explicitly indicate the algorithm - if it's enough to 
> indicate the layout.
> > > >
> > > >
> > > >
> > > >

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 23 Jun 2011 11:41:17 +0200

Hi Allyn,

>Or, perhaps you could state succinctly what statement you
>want added, subtracted, or modified in the requirements?

In my opinion, Charles is not designing a solution.

One problem we have had in the requirements discussions is that we seem to have different understandings of terminology, and I think layout is one such example.

Regards,

Christer

> > telepresence
> telepresence
> speakers and
> > > >One thing to remind you of is that we have agreed that CLUE does
> > > >not specify how rendering should be done, that is up to the
> > > >receiver. The sender (and receiver) can send info, but
> CLUE isn't
> > > >specifying what the receiver (renderer) does with it,
> meaning how
> > > >it does  the rendering, what choices it makes. I thought
> Charles'
> > > >original email may have been suggesting specifying rendering,
> > > >wasn't really sure.
> > >
> > > 1. Which streams are used to create an output stream 2.
> How does the
> > > output stream look like
> > >
> > > > > >>>represent the potential layouts of the audio/video
> can that
> > > > > >>>be offered. For example, I have 3 camera and 3 mics. I can
> > provide:
> typical audio
> > > > and video examples, such as mixing multiple audio stream inputs
> > > > into a single audio stream output, or composing multiple video
> > > > stream inputs into a single video stream output.
> > > >
> > > > Cheers,
> > > > Charles
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > >>>For me layout is about composition. Layouts do not
> > > > > >>>necessarily create a virtual room experience,
> > > > > >sources to windows separately.  Not sure this is the
> only way
> > > > > >to partition it, just the way I tend to
> > > > > use.
> > > > >
> > explicitly
> > > > >                              > > Sent: Tuesday, June
> 21, 2011 3:52
> > AM
> > > > >                              > > in principle to
> adding that kind
> > of