Subject: Re: [clue] audio mixed attribute

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Mon, 2 Apr 2012 15:52:20 +0200

An audio mixing attribute is useful in the scenario where an MCU can
support both transcoding and  switching. 

Given this scenario: A MCU supports transcoding and switching off audio
streams. The MCU can offer 3x loudest audio streams or 3x pre-mixed
audio streams that can be played out on left, Center, Right speakers. 

Example advertisement 
 AC1, AC2, AC3 audio_type = natural // Audio is switched between the
natural sources 
 AC4, AC5, AC6 audio_type = mixed // Audio is mixed into three fixed
positions 
 
 CaptureSet // AC4 - AC6 belongs together
   { AC4, AC5, AC6 } 

A consumer chooses either to request AC1 - AC3, does local mixing and
play out placement. If the endpoint chooses stream AC4 - AC6 the audio
should be played out on speaker that matches a left to right position.  

In this scheme it might be better to have a multi value attribute
describing audio_type, to avoid the possible confusion multiple Boolean
might have.

Audio_type =  { natural | mixed } 

Regards 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Paul Kyzivat
Sent: 29. mars 2012 21:54
To: [email-masked]
Subject: Re: [clue] audio mixed attribute

I think you all have very clearly demonstrated that just being told that
a stream is mixed or not-mixed is insufficient to make any useful
decision about what to do with it.

	Thanks,
	Paul (as individual)

On 3/29/12 7:18 PM, John Leslie wrote:
> Espen Berger (espeberg)<[email-masked]>  wrote:
>>
>> John, you have listed a set of valid acoustical challenges which I 
>> recognize from our experience. Some of the acoustical challenges will

>> be present if a room use 1 or 2+ microphones to record, e.g. when a 
>> person moves around in the room.
>
>     This is true.
>
>> We should thrust endpoints that chooses to use multiple microphones 
>> for capture to do proper audio pre-processing, and keep the audio
quality.
>
>     I do not intend anything other than trusting endpoint operators.
> But there are inherent differences between single-pickup and multiple 
> pickups mixed. For many purposes, multiple pickups will be better...
>
>> I would suggest that we reserve the mixed attribute to scenarios 
>> where a middle box mix audio captures from different endpoints.
>
>     I don't believe it helpful to define "mixed" based on where the 
> mixing is done.
>
>     You raise a valid point about "mixing" audio from different rooms.
> This, indeed, is something to "hardly-ever" prefer. Thus, to me, I'd 
> rather it not be an ordinary offer.
>
>     When I describe "mixed" as a warning-label, I did not mean to 
> infer that "mixed" means "low-quality".
>
>     It sounds to me as if we should seriously consider including a 
> description of _what_ is being mixed...
>
> --
> John Leslie<[email-masked]>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

_______________________________________________
clue mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/clue

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 2 Apr 2012 10:40:34 -0400

in-line

>  AC1, AC2, AC3 audio_type = natural // Audio is switched between the
[sb] perhaps you mean "original" sources?
Also, are you thinking that AC1, AC2, AC3 vary in their spatial position
(that AC1 will suddenly switch from right to left???)

> [sb] - I am wondering if you are implying that AC1, AC2, AC3 can not be
> advertised in a capture set????
>

> Audio_type =  { natural | mixed }
[sb] "natural" seems to imply that it is somehow preferred (that "mixed" is
somehow un-natural).
One benefit of the mixed version is that you are more likely to hear onset
speech from participants who are not presently seen (assuming I understand
what switching algorithm you have in mind).

I am not yet convinced that we need signaling for this. Though if we are
thinking that a capture's spatial position can change w/o notice, that
might be worth further discussion. That will complicate Jonathan's
disaggregated media requirement.

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 2 Apr 2012 11:30:34 -0400

On Thu, Mar 29, 2012 at 11:26 AM, John Leslie <[email-masked]> wrote:

> Duckworth, Mark <[email-masked]> wrote:
> >
> > Hi John,
> > I'm trying to understand your view on labeling audio as mixed.
> > Here is one example scenario that might help me understand.
> > Suppose I am a provider, and I can capture and send audio in two
> > different ways:
> > Option 1: I can send one mono audio capture.  It comes from one
> >           microphone element that picks up audio from the whole room.
> > Option 2: I can send one mono audio capture.  It comes from several
> >           microphone elements mixed together, that collectively pick
> >           up audio from the whole room.
> >
> > I think you are saying it would be useful to label option 1 as not
> > mixed, and label option 2 as mixed.  Is that correct?
>
>    Yes.
>
> > What might a receiver do differently if it receives option 1 vs.
> > receiving option 2?
>
>    (To answer, I need to assume the receiver can to some degree choose
> which to receive.)
>
>   The receiver, for example, could reasonably guess that the "mixed"
> version will have a more uniform level when speakers are in different
> parts of the room. There's always a trade-off in applyinng level
> "correction" in order to improve comprehension -- this might be more
> necessary for the "unmixed" audio.
>

[sb] Whether level correction is useful depends on the actual level being
received at the moment [and whether the original human speaker intended
their whispered comment to be heard by everyone!].
I do not think that the pick-up pattern is the primary consideration here.
In most telepresence rooms, the microphones are intentionally placed to
give proper pickup of the entire room.  So I would not make the guess that
you are making.

>
>   My concern, actually, is more with how to "audibly" "place" the
> received audio in a virtual room. I believe the "mixed" version needs
> to be given a less-distinct "place" than the "unmixed". YMMV, of course.
>

[sb]I don't really get this.  How does the renderer give an audio stream a
"less distinct" placement?  And why would it do so?
In my view, it gives every audio stream the placement it's capture
information describes.  In some cases, the audio might have better spatial
separation than others.
But I don't see how the renderer can determine this from a "mixed"
attribute, and I certainly don't get how or why a renderer would
post-process the streams differently.

>
>   There's another issue, hopefully less abstruse: when working from
> _both_ mixed and unmixed audio captures: the receiver needs to be
> careful about mixing pre-mixed sources.
>
>   The nature of sound is a very slow propagation rate in air: roughly
> one foot per millisecond. The echo of blending sound from one speaker
> with 100 millisecond propagation difference is obvious -- but the ear
> can be confused by apparent acoustic effects with differences much
> smaller than that. The human ear adjusts pretty well to different
> acoustics if they're constant, but gets confused about apparent position
> if these acoustic effects change unpredictably.
>

[sb] You are making an assumption here - in particular you are assuming
that the audio mix done at the source does not take the propagation into
account. I am not sure why you are envisioning that the source processing
is naive, but the render processing is sophisticated.

I do agree that the way the human ear perceives sound location is quite
complex.  With multiple loudspeakers, generally the first arriving sound
dominates (which is a well known effect that auditorium sound-reinforcement
systems use to their advantage). If the second sound arrives too late, it
will not be fused, resulting in reverberance or even echo.

Of course in Mark's scenario, the captured sound (from however many
microphones) is converted to monophonic *at the source*. The apparent
position of the audio stream will always be stable, as long as it is
rendered consistently at the receiver.  Multiple human voices within the
capture will all sound like they are coming from the same place- which is
artificial, but is also something we are all used to and hear every day in
audio bridges.  This perception does not depend on the number of
microphones blended in the original capture.

>
>   This is probably not anything most of us want to worry about in our
> initial release: I am thinking ahead, hoping to avoid need to kludge
> something later on. Folks are already used to "surround-sound" home
> theatres: we should have some idea how to use such sound systems when
> we get a round-tuit. ;^)
>
> [sb]  I would agree here.  Though my surround-sound system has no idea if
the original sound comes from independent single microphones [or not].  The
calibration in my receiver takes care of local delay and equalization
issues due to the placement of my speakers in my room.  The producer is
responsible for providing sound tracks that sound appropriate when rendered
in the standard speaker positions.  I am imagining a similar model here.
As long as the producer provides sound that is consistent with its
advertisement for the streams, we should be fine.

> --
> John Leslie <[email-masked]>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Mon, 2 Apr 2012 18:35:13 +0200

With a natural source I mean a audio capture that is not changed from
the original endpoint, through the switched MCU. An endpoint receiving
multiple natural (or original) audio captures should be able to decode
and mix together all audio streams, if not it should request the mixed
audio capture where the MCU has already done that work. A receiving
endpoint could also mix locally and play out the audio at the position
matching the position of video from the same endpoint, e.g. by using
SDES CNAME and/or SRCNAME. 

The rest of the comments inline  

-Espen 

From: Stephen Botzko [mailto:[email-masked]] 
Sent: 2. april 2012 16:41
To: Espen Berger (espeberg)
Cc: Paul Kyzivat; [email-masked]
Subject: Re: [clue] audio mixed attribute

in-line

On Mon, Apr 2, 2012 at 9:52 AM, Espen Berger (espeberg)
<[email-masked]> wrote:

Example advertisement

[sb] perhaps you mean "original" sources?
[Espen] Original source might be a better name, should mean the same
thing

Also, are you thinking that AC1, AC2, AC3 vary in their spatial position
(that AC1 will suddenly switch from right to left???) 
[Espen] Yes. In a switched conference the audio captures does not need a
fixed position. If the MCU chooses to replace the content in AC1 with
endpoint A instead of B, the audio could be moved from right to left
based on where the video is displayed.

	 AC4, AC5, AC6 audio_type = mixed // Audio is mixed into three
fixed
	positions
	
	 CaptureSet // AC4 - AC6 belongs together
	  { AC4, AC5, AC6 }
	[sb] - I am wondering if you are implying that AC1, AC2, AC3 can
not be advertised in a capture set????
	[Espen] They can be in a capture set, but in this example they
do not share any information.  

	A consumer chooses either to request AC1 - AC3, does local
mixing and
	play out placement. If the endpoint chooses stream AC4 - AC6 the
audio
	should be played out on speaker that matches a left to right
position.
	
	In this scheme it might be better to have a multi value
attribute
	describing audio_type, to avoid the possible confusion multiple
Boolean
	might have.

[sb] "natural" seems to imply that it is somehow preferred (that "mixed"
is somehow un-natural).
One benefit of the mixed version is that you are more likely to hear
onset speech from participants who are not presently seen (assuming I
understand what switching algorithm you have in mind).

[Espen] The MCU could use a N-loudest selection mechanism, so a receiver
could receive the 5x loudest audio streams based on audio level. Audio
level can be transferred in the RTP extension headers to avoid audio
decoding on the server. 

[Espen] If not signalled, how do you differentiate between 'original'
and 'mixed' audio? 

	Regards

	-Espen

	-----Original Message-----
	From: [email-masked] [mailto:[email-masked]] On
Behalf Of

	Paul Kyzivat

	I think you all have very clearly demonstrated that just being
told that
	a stream is mixed or not-mixed is insufficient to make any
useful
	decision about what to do with it.
	
	       Thanks,
	       Paul (as individual)

	quality.

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 2 Apr 2012 12:49:31 -0400

[Espen] If not signalled, how do you differentiate between ‘original’ and
‘mixed’ audio?
I am not sure we need to differentiate them.  We don't have any use cases
that show the need.

However, if some switched captures can suddenly change position (whether
audio or video), then we do have a problem   I think we agreed to
Jonathan's RTP requirement that disaggregated receivers needed to receive
the capture attributes prior to receiving the first RTP packet.  Also, that
it needed to be possible to direct left, right, center streams to different
transport addresses, so they could be sent to specific decoder instances
running in dedicated hardware.  Changing a capture position on the fly
would therefore also imply changing its destination transport address w/o
signaling.  Using an "original" or "mixed" attribute doesn't accomplish
those requirements, so changing position on the fly needs more discussion.

Stephen Botzko

On Mon, Apr 2, 2012 at 12:35 PM, Espen Berger (espeberg) <[email-masked]
> wrote:

> ** **
>
> With a natural source I mean a audio capture that is not changed from the
> original endpoint, through the switched MCU. An endpoint receiving multiple
> natural (or original) audio captures should be able to decode and mix
> together all audio streams, if not it should request the mixed audio
> capture where the MCU has already done that work. A receiving endpoint
> could also mix locally and play out the audio at the position matching the
> position of video from the same endpoint, e.g. by using SDES CNAME and/or
> SRCNAME. ****
>
> ** **
>
> The rest of the comments inline  ****
>
> ** **
>
> -Espen ****
>
> ** **
>
> ** **
>
> *From:* Stephen Botzko [mailto:[email-masked]]
> *Sent:* 2. april 2012 16:41
> *To:* Espen Berger (espeberg)
> *Cc:* Paul Kyzivat; [email-masked]
>
> *Subject:* Re: [clue] audio mixed attribute****
>
> ** **
>
> in-line****
>
> On Mon, Apr 2, 2012 at 9:52 AM, Espen Berger (espeberg) <
> [email-masked]> wrote:****
>
> natural sources****
> [Espen] Original source might be a better name, should mean the same thing
> ****
>
> endpoint A instead of B, the audio could be moved from right to left based
> on where the video is displayed.****
>
> [Espen] They can be in a capture set, but in this example they do not
> share any information.  ****
>
>  ****
>
> Audio_type =  { natural | mixed }****
>
> what switching algorithm you have in mind).****
>
> [Espen] If not signalled, how do you differentiate between ‘original’ and
> ‘mixed’ audio? ****
>
> ** **
>
>
> Regards****
>
>
> -Espen
>
>
> From: [email-masked] [mailto:[email-masked]] On Behalf Of***
> *
>
> Paul Kyzivat
> Subject: Re: [clue] audio mixed attribute****
>
> https://www.ietf.org/mailman/listinfo/clue****
>
> ** **
>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Mon, 2 Apr 2012 21:28:43 +0200

An MCU can basically give me two types of audio streams, 

*         Mixed - The MCU mixes together audio and deliver a single
audio stream based on some MCU cleverness 

*         Original - The MCU forward a audio stream un-modified from the
source (based on VAD-level selection)

The purpose of the audio_type is only to make this distinction. 

How to model and implement various spatial audio scenarios need further
discussions, not the audio-type attribute. 

-Espen 

Sent: 2. april 2012 18:50

[Espen] If not signalled, how do you differentiate between 'original'
and 'mixed' audio? 
I am not sure we need to differentiate them.  We don't have any use
cases that show the need.

However, if some switched captures can suddenly change position (whether
audio or video), then we do have a problem   I think we agreed to
Jonathan's RTP requirement that disaggregated receivers needed to
receive the capture attributes prior to receiving the first RTP packet.
Also, that it needed to be possible to direct left, right, center
streams to different transport addresses, so they could be sent to
specific decoder instances running in dedicated hardware.  Changing a
capture position on the fly would therefore also imply changing its
destination transport address w/o signaling.  Using an "original" or
"mixed" attribute doesn't accomplish those requirements, so changing
position on the fly needs more discussion.

Stephen Botzko

On Mon, Apr 2, 2012 at 12:35 PM, Espen Berger (espeberg)
<[email-masked]> wrote:

-Espen 

in-line

Example advertisement

	

	

	
	Regards

	-Espen

	-----Original Message-----

	Paul Kyzivat

	       Thanks,
	       Paul (as individual)

	quality.

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 02 Apr 2012 15:37:03 -0400

On 4/2/12 3:28 PM, Espen Berger (espeberg) wrote:
> ·Mixed – The MCU mixes together audio and deliver a single audio stream
> based on some MCU cleverness
>
> ·Original – The MCU forward a audio stream un-modified from the source
> (based on VAD-level selection)
>

I'm pretty sure I'm more of a neophyte on the details of this than 
anybody else in the wg. But based on what I've observed, I still get the 
feeling that being aware of this distinction is insufficient to make any 
useful decisions about how to choose and use audio media.

Every example I've seen requires making additional, and unmotivated, 
assumptions about what has been mixed, or how, or both.

	Thanks,
	Paul

> *From:*Stephen Botzko [mailto:[email-masked]]
> *Sent:* 2. april 2012 18:50
> *Subject:* Re: [clue] audio mixed attribute
>
> [Espen] If not signalled, how do you differentiate between ‘original’
> and ‘mixed’ audio?
> <[email-masked] <mailto:[email-masked]>> wrote:
>
> *From:*Stephen Botzko [mailto:[email-masked]
> <mailto:[email-masked]>]
> *Cc:* Paul Kyzivat; [email-masked] <mailto:[email-masked]>
>
>
> *Subject:* Re: [clue] audio mixed attribute
>
> in-line
>
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     [sb] - I am wondering if you are implying that AC1, AC2, AC3 can not
>     be advertised in a capture set????
>
>
>     [Espen] They can be in a capture set, but in this example they do
>     not share any information.
>
> [Espen] If not signalled, how do you differentiate between ‘original’
> and ‘mixed’ audio?
>
>
>     Regards
>
>
>     -Espen
>
>
>     From: [email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>] On
>     Behalf Of
>
>     Paul Kyzivat
>     To: [email-masked] <mailto:[email-masked]>
>      > Espen Berger (espeberg)<[email-masked]
>     <mailto:[email-masked]>> wrote:
>      >>
>      > John Leslie<[email-masked] <mailto:[email-masked]>>
>      > [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Brian Rosen <[email-masked]>
Date: Mon, 2 Apr 2012 15:46:55 -0400

Agree.

To illustrate:

Top 4 speakers (switched) mixed to a stereo stream based on placement on the screen(s). 

Mixed or Original?

And regardless, is it sufficient to do ANYTHING?

Brian
On Apr 2, 2012, at 3:37 PM, Paul Kyzivat wrote:

> I'm pretty sure I'm more of a neophyte on the details of this than anybody else in the wg. But based on what I've observed, I still get the feeling that being aware of this distinction is insufficient to make any useful decisions about how to choose and use audio media.
> 
> Every example I've seen requires making additional, and unmotivated, assumptions about what has been mixed, or how, or both.
> 
> 	Thanks,
> 	Paul
>

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 2 Apr 2012 15:55:55 -0400

in-line, snipped...

On Mon, Apr 2, 2012 at 3:37 PM, Paul Kyzivat <[email-masked]> wrote:

> I'm pretty sure I'm more of a neophyte on the details of this than anybody
> else in the wg. But based on what I've observed, I still get the feeling
> that being aware of this distinction is insufficient to make any useful
> decisions about how to choose and use audio media.
>

[sb] This is one of the points I am trying to make as well.  I am certainly
open to signaling attributes that aid rendering, but am not seeing how this
one helps either selection or rendering.

I do understand the goal of the composition flag for video - the idea there
is to prevent nested hollywood squares. The definition may need some work,
but the goal is clear.

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Tue, 3 Apr 2012 10:48:07 +0200

I will highlight one potential use for the audio type, which Espen has
actually already pointed out.

If the audio stream is natural/original, it can be expected to
correspond to a single video stream. The renderer can go search for this
connection in other attributes, and then render the audio at the correct
position. An important feature in a complex layout on a wide display
setup.

If the audio is mixed by the MCU there is no point in searching for a
connection, play audio in the center or across all loudspeakers.

The same function could be accomplished without the audio attribute if
there were mechanisms for explicitly announcing spatial relationships
from the MCU to the renderer. I don't think the capture area concept is
well suited for this.

Regards

Johan

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Stephen Botzko
Sent: 2. april 2012 21:56
To: Paul Kyzivat
Cc: [email-masked]
Subject: Re: [clue] audio mixed attribute

in-line, snipped...

On Mon, Apr 2, 2012 at 3:37 PM, Paul Kyzivat <[email-masked]>
wrote:

*Mixed - The MCU mixes together audio and deliver a single audio stream
based on some MCU cleverness

*Original - The MCU forward a audio stream un-modified from the source
(based on VAD-level selection)

[sb] This is one of the points I am trying to make as well.  I am
certainly open to signaling attributes that aid rendering, but am not
seeing how this one helps either selection or rendering. 

I do understand the goal of the composition flag for video - the idea
there is to prevent nested hollywood squares. The definition may need
some work, but the goal is clear. 

	       Thanks,
	       Paul

	How to model and implement various spatial audio scenarios need
further
	discussions, not the audio-type attribute.
	
	-Espen

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Tue, 3 Apr 2012 12:44:35 +0200

A mono audio capture can originate from a several microphones mixed in
external gear that the telepresence system don't know about. This is
very common. So even a single microphone input can be mixed.

The idea of a middle box or renderer applying automatic gain corrections
to a non-mixed signal due to expected larger level differences from
talkers at different distances to the microphone is admirable. However,
normally AGC would be applied by the provider, and this is also the
preferred way as there can be extra data available in the mic processing
enabling better AGC designs. Network or receiver AGCs have a tendency to
perform less well, especially if they process on streams that are
already level corrected by the senders and partially corrupted by coding
or other processing artefacts.

So an audio stream marked "non-mixed" may actually be both mixed and
already gain-corrected. Trusting the sender to provide the best possible
pick-up and processing is in my view the best strategy, I agree with
Stephen.

I like John's suggestion about a "less distinct" placement. It can be
done by rendering an audio stream on several loudspeakers instead of
one. And it should be done for those streams where there is no spatial
information, or where the stream is known to represent a large pickup
area like a whole room which is rendered on a wide display area.
However, I think it should be connected to spatial information (or lack
of such), and not to a mixed attribute, which may be ambigious.

I also think John's comments about the multitude of audio captures,
mixed and unmixed, is important. Not necessarily for position effects,
but for quality. We should trust the provider to mix in a sensible way.
But the framework must prevent the MCU or the renderer choosing and
mixing audio captures that shouldn't be mixed later on. A mixed
attribute could help, but I guess other mechanisms in the capture set
announcements could solve it too, ref. the mutually-exclusive
discussion.

I think the goal that John keeps referring to of keeping positional
information deserves much more discussion and solutions. It should be
clarified in the framework. Preserving timing info between multiple
audio captures is one way, but requires taking care of microphone and
loudspeaker layouts. 

In my view a more abstract and general model would find more
applications. The model in the protocols should be abstract and totally
independent on the microphone pickup system and the loudspeaker
rendering system, very much like for surround sound that Stephen is
referring to, and other multichannel audio formats. Provide information
that is useful for the renderer, leave out the rest. 

Regards

Johan

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Stephen Botzko
Sent: 2. april 2012 17:31
To: John Leslie
Cc: [email-masked]
Subject: Re: [clue] audio mixed attribute

  Yes.

[sb] Whether level correction is useful depends on the actual level
being received at the moment [and whether the original human speaker
intended their whispered comment to be heard by everyone!].
I do not think that the pick-up pattern is the primary consideration
here.  In most telepresence rooms, the microphones are intentionally
placed to give proper pickup of the entire room.  So I would not make
the guess that you are making.

	  My concern, actually, is more with how to "audibly" "place"
the
	received audio in a virtual room. I believe the "mixed" version
needs
	to be given a less-distinct "place" than the "unmixed". YMMV, of
course.

[sb]I don't really get this.  How does the renderer give an audio stream
a "less distinct" placement?  And why would it do so?
In my view, it gives every audio stream the placement it's capture
information describes.  In some cases, the audio might have better
spatial separation than others. 
But I don't see how the renderer can determine this from a "mixed"
attribute, and I certainly don't get how or why a renderer would
post-process the streams differently.

	  There's another issue, hopefully less abstruse: when working
from
	_both_ mixed and unmixed audio captures: the receiver needs to
be
	careful about mixing pre-mixed sources.
	
	  The nature of sound is a very slow propagation rate in air:
roughly
	one foot per millisecond. The echo of blending sound from one
speaker
	with 100 millisecond propagation difference is obvious -- but
the ear
	can be confused by apparent acoustic effects with differences
much
	smaller than that. The human ear adjusts pretty well to
different
	acoustics if they're constant, but gets confused about apparent
position
	if these acoustic effects change unpredictably.

[sb] You are making an assumption here - in particular you are assuming
that the audio mix done at the source does not take the propagation into
account. I am not sure why you are envisioning that the source
processing is naive, but the render processing is sophisticated.

I do agree that the way the human ear perceives sound location is quite
complex.  With multiple loudspeakers, generally the first arriving sound
dominates (which is a well known effect that auditorium
sound-reinforcement systems use to their advantage). If the second sound
arrives too late, it will not be fused, resulting in reverberance or
even echo.

Of course in Mark's scenario, the captured sound (from however many
microphones) is converted to monophonic at the source. The apparent
position of the audio stream will always be stable, as long as it is
rendered consistently at the receiver.  Multiple human voices within the
capture will all sound like they are coming from the same place- which
is artificial, but is also something we are all used to and hear every
day in audio bridges.  This perception does not depend on the number of
microphones blended in the original capture. 

	  This is probably not anything most of us want to worry about
in our
	initial release: I am thinking ahead, hoping to avoid need to
kludge
	something later on. Folks are already used to "surround-sound"
home
	theatres: we should have some idea how to use such sound systems
when
	we get a round-tuit. ;^)

[sb]  I would agree here.  Though my surround-sound system has no idea
if the original sound comes from independent single microphones [or
not].  The calibration in my receiver takes care of local delay and
equalization issues due to the placement of my speakers in my room.  The
producer is responsible for providing sound tracks that sound
appropriate when rendered in the standard speaker positions.  I am
imagining a similar model here.  As long as the producer provides sound
that is consistent with its advertisement for the streams, we should be
fine.

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 3 Apr 2012 06:45:54 -0400

[inline]
Stephen

On Tue, Apr 3, 2012 at 4:48 AM, Johan Ludvig Nielsen (johaniel) <
[email-masked]> wrote:

> actually already pointed out.****
>
> ** **
>
> If the audio stream is natural/original, it can be expected to correspond
> to a single video stream. The renderer can go search for this connection in
> other attributes, and then render the audio at the correct position. An
> important feature in a complex layout on a wide display setup.
>
[sb] Actually CLUE does not require a 1-1 correspondence between audio and
video captures.  You can for instance use 3 audio captures to span 2 video
captures. As displays continue to get larger, I am thinking this
configuration will occur in real systems.

> connection, play audio in the center or across all loudspeakers.****
>
[sb] If the MCU is composing video on multiple sites on the displays, then
it could also re-process the audio so that the audio and video positioning
is preserved. It is just as capable of doing this as the receiving
endpoint.  Of course, if the audio in a capture has no correspondence to
the video, the MCU can choose to signal it as being centered if it wishes,
and it can adjust the capture area to the full range.  It has other options
though - for instance It could also position it off-screen.  I see no
reason for the receiver to second-guess the position, since (like sourcing
endpoints) the MCU is advertising the position of the audio and video in a
common coordinate space.

**
> there were mechanisms for explicitly announcing spatial relationships from
> the MCU to the renderer. I don’t think the capture area concept is well
> suited for this.
>
[sb]The capture scene concept is the mechanism for advertising spatial
relationships both for original captures and mixed/composed captures, for
both MCUs and endpoints.  If you think it is not well suited, then it would
be extremely helpful to hear more.

> ****
>
> ** **
>
> Regards****
>
> Johan****
>
> ** **
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Stephen Botzko
> *Sent:* 2. april 2012 21:56
> *To:* Paul Kyzivat
> *Cc:* [email-masked]
>
> in-line, snipped...****
> wrote:****
> On 4/2/12 3:28 PM, Espen Berger (espeberg) wrote:****
>
> The purpose of the audio_type is only to make this distinction.****
>
> ** **
>
> assumptions about what has been mixed, or how, or both.****
>
>
> there is to prevent nested hollywood squares. The definition may need some
> work, but the goal is clear. ****
>
>
>        Thanks,
>        Paul****
>
> -Espen****

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Tue, 3 Apr 2012 13:19:51 +0200

Stephen,

I think we have a common goal, I agree with what you write including the
expansion beyond 1:1 relationships, the features you describe are great.

If the capture scene / area concepts are meant to solve these features
for the MCU case, I would very much like to see an example of how, as I
am not able to figure it out from the framework description. I have
asked for that on this list before without response. The example in the
framework doesn't encompass spatial audio.

For the MCU composing layouts on multiple displays and multiple sites,
how would the MCU construct and announce capture scene and areas to the
renderers to signal the spatial connections? And how will CLUE
signalling be done to be sufficiently fast to keep up with changes in
the connections and spatial positions when active speakers constantly
change?

The same questions are even more interesting with a fully switched MCU.

Regards

Johan

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Stephen Botzko
Sent: 3. april 2012 12:46
To: Johan Ludvig Nielsen (johaniel)
Cc: [email-masked]
Subject: Re: [clue] audio mixed attribute

[inline]
Stephen

On Tue, Apr 3, 2012 at 4:48 AM, Johan Ludvig Nielsen (johaniel)
<[email-masked]> wrote:

[sb] Actually CLUE does not require a 1-1 correspondence between audio
and video captures.  You can for instance use 3 audio captures to span 2
video captures. As displays continue to get larger, I am thinking this
configuration will occur in real systems.

	If the audio is mixed by the MCU there is no point in searching
for a connection, play audio in the center or across all loudspeakers.

[sb] If the MCU is composing video on multiple sites on the displays,
then it could also re-process the audio so that the audio and video
positioning is preserved. It is just as capable of doing this as the
receiving endpoint.  Of course, if the audio in a capture has no
correspondence to the video, the MCU can choose to signal it as being
centered if it wishes, and it can adjust the capture area to the full
range.  It has other options though - for instance It could also
position it off-screen.  I see no reason for the receiver to
second-guess the position, since (like sourcing endpoints) the MCU is
advertising the position of the audio and video in a common coordinate
space.

	The same function could be accomplished without the audio
attribute if there were mechanisms for explicitly announcing spatial
relationships from the MCU to the renderer. I don't think the capture
area concept is well suited for this.

[sb]The capture scene concept is the mechanism for advertising spatial
relationships both for original captures and mixed/composed captures,
for both MCUs and endpoints.  If you think it is not well suited, then
it would be extremely helpful to hear more.

	Regards

	Johan

Behalf Of Stephen Botzko

	in-line, snipped...

	On Mon, Apr 2, 2012 at 3:37 PM, Paul Kyzivat
<[email-masked]> wrote:

	
	*Mixed - The MCU mixes together audio and deliver a single audio
stream
	based on some MCU cleverness
	
	*Original - The MCU forward a audio stream un-modified from the
source
	(based on VAD-level selection)

	I'm pretty sure I'm more of a neophyte on the details of this
than anybody else in the wg. But based on what I've observed, I still
get the feeling that being aware of this distinction is insufficient to
make any useful decisions about how to choose and use audio media.
	
	Every example I've seen requires making additional, and
unmotivated, assumptions about what has been mixed, or how, or both.

	[sb] This is one of the points I am trying to make as well.  I
am certainly open to signaling attributes that aid rendering, but am not
seeing how this one helps either selection or rendering. 
	
	I do understand the goal of the composition flag for video - the
idea there is to prevent nested hollywood squares. The definition may
need some work, but the goal is clear. 

		       Thanks,
		       Paul

		How to model and implement various spatial audio
scenarios need further
		discussions, not the audio-type attribute.
		
		-Espen