Subject: Re: [clue] Improving treatment of audio

From: Stephen Botzko <[email-masked]>
Date: Thu, 1 May 2014 06:58:55 -0400

In-line

On Thu, May 1, 2014 at 12:09 AM, Christian Groves <
[email-masked]> wrote:

> Hello Mark,
>
> Please see my replies below.
>
> Regards, Christian
>
>
> On 30/04/2014 11:28 PM, Duckworth, Mark wrote:
>
>> Hi Christian,
>> Thanks for elaborating on the idea.  I have some more questions inline.
>> I see basically what you are trying to do, but I still think it doesn't
>> work in practice any better than what we already have, as I explain below.
>> Mark
>>
>>  -----Original Message-----
>>> From: Christian Groves [mailto:[email-masked]]
>>> Sent: Monday, April 28, 2014 10:15 PM
>>> To: Duckworth, Mark; [email-masked]
>>> Subject: Re: [clue] Improving treatment of audio
>>>
>>> Hello Mark,
>>>
>>> Please see below.
>>>
>>> Regards, Christian
>>>
>>> On 29/04/2014 11:36 AM, Duckworth, Mark wrote:
>>>
>>>> Hi Christian,
>>>> I'm coming back to this message, because I'm trying to understand your
>>>>
>>> proposal.  You proposed "in each audio capture we say what VC it relates
>>> to."
>>> What does this mean?  What does it mean for an audio capture to relate
>>> to a
>>> video capture?  What exactly is the producer telling the consumer?  Can
>>> an
>>> audio capture relate to more than one video capture?
>>> [CNG] The meaning has largely the same semantic as if the ADV used the
>>> same capture area on a VC and AC. Basically its saying that the audio
>>> comes
>>> from the same "region" as what the video capture does. I think the
>>> difference is that its not trying to put a mathematical certainty about
>>> what
>>> that region is. Its a way for the producer to say to the consumer if you
>>> choose
>>> video capture A you probably want to choose audio capture B for a good
>>> experience.
>>>
>> [Duckworth, Mark] If that is the meaning, then would it be better for the
>> provider to advertise a list of audio captures that are "related to" each
>> video capture?  Rather than the other way around?  Maybe I'm not
>> understanding the problem you are trying to solve.
>>
> [CNG] I don't think it matters the way it is defined. video related to
> audio or audio related to video is essentially the same thing. The problem
> I'm trying to solve is how does the media consumer know which audio
> captures to choose when presented with a list of captures.
>
> As I mentioned in an earlier email I think people where seeing the "area
> of capture" in two separate ways. The first way was using the area of
> capture to determine which video captures the audio related to (i.e.
> capture selection). The second was concentrating on how to use the area of
> capture information for audio rendering (i.e. capture rendering). I'm
> focusing on the first, capture selection.
>
> [SB] I am wondering if you are interested in static selection - based on
spatial information alone - or if you are more focused on dynamic capture
selection - finding the video of the current talker for instance.
Also, it might matter if you are trying to select video captures that go
along with audio captures, or the other way around.

My view is that selecting a subset of the audio captures in the room for
rendering or distribution will likely result in bad audio much of the time
- unless the sending system has expressly designed its audio pickup and
seating arrangements to enable that. And a lot of systems won't be designed
to allow it. I don't think it will be all that easy to detect that from
details of the pickup pattern of each microphone or capture information.
 The general acoustics of the room also play into it, and I think the
seating arrangement does also.

It is possible that senders will create alternative sets of audio captures
- for instance,a set of spatial audio captures that together provide full
room pickup, and a single downmixed monophonic channel that also covers the
full room.  Receivers obviously need to sort that out, just like they do in
the corresponding video case.

My view is also that trying to select an appropriate video capture for
display by examining the voice activity in all the audio captures in the
room will result in showing the wrong video quite often.  This is based on
practical interoperability experience with TIP.

It might be helpful if we first try to reach agreement on the limitations
of this kind of selection before we dive into the protocol mechanics.[/SB]

>
>
>
>
>>  In terms of whether an audio capture can relate to more than
>>> one video capture yes I think so. You could have one audio capture for a
>>> room and three video captures.
>>>
>>>> More comments inline below.
>>>>
>>>> Regards,
>>>> Mark
>>>>
>>>>  -----Original Message-----
>>>>> From: Christian Groves [mailto:[email-masked]]
>>>>> Sent: Sunday, April 13, 2014 11:18 PM
>>>>> To: Duckworth, Mark; [email-masked]
>>>>> Subject: Re: [clue] Improving treatment of audio
>>>>>
>>>>> Hello Mark,
>>>>>
>>>>> Sorry I didn't consider the entire 12.1.1. if I do that, according to
>>>>> that
>>>>> example:
>>>>>
>>>>> Video areas of capture:
>>>>>
>>>>>           bottom left    bottom right  top left         top right
>>>>>       VC0 (-2011,2850,0) (-673,3000,0) (-2011,2850,757) (-673,3000,757)
>>>>>       VC1 ( -673,3000,0) ( 673,3000,0) ( -673,3000,757) ( 673,3000,757)
>>>>>       VC2 (  673,3000,0) (2011,2850,0) (  673,3000,757) (2011,3000,757)
>>>>>       VC3 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
>>>>>       VC4 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
>>>>>       VC5 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
>>>>>       VC6 none
>>>>>
>>>>> Areas of capture for audio (from 12.1.1):
>>>>>
>>>>>           bottom left    bottom right  top left         top right
>>>>>
>>>>>       AC0 (-2011,2850,0) (-673,3000,0) (-2011,2850,757) (-673,3000,757)
>>>>>       AC1 (  673,3000,0) (2011,2850,0) (  673,3000,757) (2011,3000,757)
>>>>>       AC2 ( -673,3000,0) ( 673,3000,0) ( -673,3000,757) ( 673,3000,757)
>>>>>       AC3 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
>>>>>       AC4 none
>>>>>
>>>>> Using a reference rather than area of capture:
>>>>>        AC0 (VC0)
>>>>>        AC1 (VC2)
>>>>>        AC2 (VC1)
>>>>>        AC3 (VC3,VC4,VC5)
>>>>> I would think they are basically conveying the same information???
>>>>>
>>>> [Duckworth, Mark] Looking from the consumer point of view, in trying to
>>>>
>>> understand the advertisement, what does the consumer know from the
>>> advertisement?  Suppose the consumer wants to get all three of VC0, VC1,
>>> VC2, but it only wants a single audio capture.  It should probably ask
>>> for AC3,
>>> but according to the advertisement AC3 doesn't "relate to" VC0, VC1, or
>>> VC2.
>>> [CNG] I agree. If the provider wanted to indicate that AC3 could also be
>>> used
>>> for VC0,VC1,VC2 it could advertise:
>>>        AC0 (VC0)
>>>        AC1 (VC2)
>>>        AC2 (VC1)
>>>        AC3 (VC0,VC1,VC2,VC3,VC4,VC5)
>>>
>> [Duckworth, Mark] I agree that  AC3 (the whole room audio) "relates to"
>> all those video captures.  But earlier you said "the audio comes from the
>> same 'region' as what the video capture does."  So now do we have to modify
>> that to say something like "the audio comes from the same region as the
>> video, or a subset of that region, or a superset, or an overlapping
>> region"?  In this example, AC3 covers a region that is a superset of the
>> VC0 region, a superset of the VC1 region, and a superset of the VC2 region.
>>
> [CNG] I was trying to illustrate the concept when I used the term
> "region". I didn't mean for that to relate to a specific area. I prefer
> "relates to" because that is essentially I'm trying specify. For the
> purposes of capture selection you could specify the relation in terms of a
> mathematical area or you can simply say the captures are related.
>
>
>
>>  [Duckworth, Mark] For a case with a different consumer, suppose the
>>>>
>>> consumer wants the single video capture VC5, but it would like to receive
>>> and render spatial audio.  There is no audio capture with multiple
>>> channels,
>>> so it could ask for AC0, AC1, and AC2 if it knew how they were spatially
>>> "related to" VC5.  But according to the advertisement, AC0, AC1 and AC2
>>> aren't "related to" VC5 at all.
>>> [CNG] So in this case there is no AC3? In that case the advertisement
>>> could
>>> look like:
>>>        AC0 (VC0,VC5)
>>>        AC1 (VC2,VC5)
>>>        AC2 (VC1,VC5)
>>>
>> [Duckworth, Mark] I didn't mean to remove AC3.  That would still be there
>> in this example.  So now since the provider forms it's advertisement
>> without knowing what the consumer wants to do, the provider has to give all
>> this information:
>>    AC0 relates to (VC0, VC3, VC4, VC5)
>>    AC1 relates to (VC2, VC3, VC4, VC5)
>>    AC2 relates to (VC1, VC3, VC4, VC5)
>>    AC3 relates to (VC0, VC1, VC2, VC3, VC4, VC5)
>> [Duckworth, Mark] Does this information really help the consumer?  I
>> don't think it is any better than what we already have with area of capture.
>>
> [CNG] Well it does say to the MC a number of things, for exmaple:
> - if you're going to choose VC0 then either AC0 and AC3 would be the best
> choice.
> - AC3 is the best choice if you want one audio capture as it relates to
> all video captures.
>
> My proposal was only duplicate what is supported by area of capture in
> terms of relating audio and video captures for capture selection. The way
> the discussion is going it seems that people are dubious about specifying
> capture area for audio. The proposal is trying to get around that.
> Another option would be to indicate that the capture area when used for
> audio actually relates to a video capture region. However that seems to be
> confusing to people.
>
>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: Christian Groves <[email-masked]>
Date: Thu, 01 May 2014 21:34:41 +1000

Hello Stephen,

Please see below.

Regards, Christian

On 1/05/2014 8:58 PM, Stephen Botzko wrote:
> In-line
>
>
> On Thu, May 1, 2014 at 12:09 AM, Christian Groves 
> <[email-masked] <mailto:[email-masked]>> 
> wrote:
>
>     Hello Mark,
>
>         I see basically what you are trying to do, but I still think
>         it doesn't work in practice any better than what we already
>         have, as I explain below.
>         Mark
>
>             From: Christian Groves
>             [mailto:[email-masked]
>             <mailto:[email-masked]>]
>             To: Duckworth, Mark; [email-masked] <mailto:[email-masked]>
>             proposal.  You proposed "in each audio capture we say what
>             VC it relates to."
>             What does this mean?  What does it mean for an audio
>             capture to relate to a
>             video capture?  What exactly is the producer telling the
>             consumer?  Can an
>             same capture area on a VC and AC. Basically its saying
>             that the audio comes
>             difference is that its not trying to put a mathematical
>             certainty about what
>             that region is. Its a way for the producer to say to the
>             consumer if you choose
>         [Duckworth, Mark] If that is the meaning, then would it be
>         better for the provider to advertise a list of audio captures
>         that are "related to" each video capture?  Rather than the
>         other way around?  Maybe I'm not understanding the problem you
>         are trying to solve.
>
>     [CNG] I don't think it matters the way it is defined. video
>     related to audio or audio related to video is essentially the same
>     thing. The problem I'm trying to solve is how does the media
>     consumer know which audio captures to choose when presented with a
>     list of captures.
>
>     As I mentioned in an earlier email I think people where seeing the
>     "area of capture" in two separate ways. The first way was using
>     the area of capture to determine which video captures the audio
>     related to (i.e. capture selection). The second was concentrating
>     on how to use the area of capture information for audio rendering
>     (i.e. capture rendering). I'm focusing on the first, capture
>     selection.
>
> [SB] I am wondering if you are interested in static selection - based 
> on spatial information alone - or if you are more focused on dynamic 
> capture selection - finding the video of the current talker for instance.
[CNG] I was thinking about a static selection.

> Also, it might matter if you are trying to select video captures that 
> go along with audio captures, or the other way around.
>
> My view is that selecting a subset of the audio captures in the room 
> for rendering or distribution will likely result in bad audio much of 
> the time - unless the sending system has expressly designed its audio 
> pickup and seating arrangements to enable that. And a lot of systems 
> won't be designed to allow it. I don't think it will be all that easy 
> to detect that from details of the pickup pattern of each microphone 
> or capture information.  The general acoustics of the room also play 
> into it, and I think the seating arrangement does also.
>
> It is possible that senders will create alternative sets of audio 
> captures - for instance,a set of spatial audio captures that together 
> provide full room pickup, and a single downmixed monophonic channel 
> that also covers the full room.  Receivers obviously need to sort that 
> out, just like they do in the corresponding video case.
[CNG] Its the aspect of alternate sets of audio captures that I was 
concentrating on. If a MP sends several audio captures and video 
captures AND we don't have audio capture area how does the MP figure out 
which audio to select?

> display by examining the voice activity in all the audio captures in 
> the room will result in showing the wrong video quite often.  This is 
> based on practical interoperability experience with TIP.
>
> It might be helpful if we first try to reach agreement on the 
> limitations of this kind of selection before we dive into the protocol 
> mechanics.[/SB]
>
>
>

---

From: Stephen Botzko <[email-masked]>
Date: Thu, 1 May 2014 07:58:15 -0400

inline

On Thu, May 1, 2014 at 7:34 AM, Christian Groves <
[email-masked]> wrote:

>> [email-masked] <mailto:[email-masked]>>
>> wrote:
>>
>>     Hello Mark,
>>

>> provide full room pickup, and a single downmixed monophonic channel that
>> also covers the full room.  Receivers obviously need to sort that out, just
>> like they do in the corresponding video case.
>>
> concentrating on. If a MP sends several audio captures and video captures
> AND we don't have audio capture area how does the MP figure out which audio
> to select?
>
> ]SB} I agree that we need to ensure that alternate sets of audio captures
of the room can be advertised and distinguished.  I think it is quite
likely that systems will include an alternate monophonic capture of the
entire room at least.  Taking individual audio captures from different
alternate sets and attempting to render them together will certainly result
in bad audio..

The group may disagree on the value of taking a subset of the full room
audio, but I think the need to distinguish alternate sets of audio captures
is something we can agree on.

---

From: Paul Coverdale <[email-masked]>
Date: Thu, 1 May 2014 11:50:13 -0400

My view is that selecting a subset of the audio captures in the room for rendering or distribution will likely result in bad audio much of the time - unless the sending system has expressly designed its audio pickup and seating arrangements to enable that. And a lot of systems won't be designed to allow it. I don't think it will be all that easy to detect that from details of the pickup pattern of each microphone or capture information.  The general acoustics of the room also play into it, and I think the seating arrangement does also.

It is possible that senders will create alternative sets of audio captures - for instance,a set of spatial audio captures that together provide full room pickup, and a single downmixed monophonic channel that also covers the full room.  Receivers obviously need to sort that out, just like they do in the corresponding video case.

My view is also that trying to select an appropriate video capture for display by examining the voice activity in all the audio captures in the room will result in showing the wrong video quite often.  This is based on practical interoperability experience with TIP.

[Paul Coverdale] I agree with this. Using voice activity to trigger the appropriate video in a multi-site conference is one thing, but to reliably differentiate between talkers in the same room is much more complicated.

It might be helpful if we first try to reach agreement on the limitations of this kind of selection before we dive into the protocol mechanics.[/SB]

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 01 May 2014 12:33:19 -0400

On 5/1/14 6:58 AM, Stephen Botzko wrote:

> room will result in showing the wrong video quite often.  This is based
> on practical interoperability experience with TIP.
>

I think it would be helpful to describe the kinds of choices we intend 
to facilitate, and those that we think are non-goals and likely not 
supported by the mechanism. I think it is important that we agree on 
what these are.

Then, for the kinds of choices we intend to facilitate it would be good 
to (somewhere) have some existence proofs (examples) of how this can be 
achieved. These need not be normative.

Part of our problem right now may be that we *don't* agree on this.

For instance, for video, I think it is in scope for a consumer to select 
a subset of the captures - even a subset of a CSE that might be useful 
*to some consumers*. (For instance somebody who only wants to monitor 
the behavior of his boss.)

For audio, I see hints of a consensus that a subset of the captures in a 
CSE will almost never provide a useful experience.

But perhaps the general assertion ought to be that one complete video 
CSE and one complete audio CSE per scene is the minimum that can be 
expected to provide a good experience. Subsets of that are allowed, but 
without any promises about the acceptability of the result.

	Thanks,
	Paul

---

From: Stephen Botzko <[email-masked]>
Date: Thu, 1 May 2014 12:54:45 -0400

On Thu, May 1, 2014 at 12:33 PM, Paul Kyzivat <[email-masked]> wrote:

> I think it would be helpful to describe the kinds of choices we intend to
> facilitate, and those that we think are non-goals and likely not supported
> by the mechanism. I think it is important that we agree on what these are.
>
> Then, for the kinds of choices we intend to facilitate it would be good to
> (somewhere) have some existence proofs (examples) of how this can be
> For instance, for video, I think it is in scope for a consumer to select a
> subset of the captures - even a subset of a CSE that might be useful *to
> some consumers*. (For instance somebody who only wants to monitor the
> behavior of his boss.)
>
> But perhaps the general assertion ought to be that one complete video CSE
> and one complete audio CSE per scene is the minimum that can be expected to
> provide a good experience. Subsets of that are allowed, but without any
> promises about the acceptability of the result.
>
[sb]I think this is in the right direction, though perhaps it needs some
more refinement.  We probably need to acknowledge some differences between
audio and video in any general assertions.

For video, arguably you can get a better experience in some cases if you
use fewer captures displayed at a larger size - getting closeups of a few
people instead of a crowd of indistinguishable people.

You can of course also render video captures in more than one place on the
screen, which makes no sense for audio.

With video you need to manage screen real estate - especially in
multipoint, where it quickly becomes a scarce resource. There is no
corresponding thing to manage for audio.[\sb]

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 01 May 2014 14:48:08 -0400

On 5/1/14 12:54 PM, Stephen Botzko wrote:
>
> On Thu, May 1, 2014 at 12:33 PM, Paul Kyzivat <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     I think it would be helpful to describe the kinds of choices we
>     intend to facilitate, and those that we think are non-goals and
>     likely not supported by the mechanism. I think it is important that
>     we agree on what these are.
>
>     Then, for the kinds of choices we intend to facilitate it would be
>     good to (somewhere) have some existence proofs (examples) of how
>     this can be achieved. These need not be normative.
>
>     For instance, for video, I think it is in scope for a consumer to
>     select a subset of the captures - even a subset of a CSE that might
>     be useful *to some consumers*. (For instance somebody who only wants
>     to monitor the behavior of his boss.)
>
>     For audio, I see hints of a consensus that a subset of the captures
>     in a CSE will almost never provide a useful experience.
>
>     But perhaps the general assertion ought to be that one complete
>     video CSE and one complete audio CSE per scene is the minimum that
>     can be expected to provide a good experience. Subsets of that are
>     allowed, but without any promises about the acceptability of the result.
>
> more refinement.

I'm *sure* it needs more refinement!

> We probably need to acknowledge some differences
> between audio and video in any general assertions.
>
> use fewer captures displayed at a larger size - getting closeups of a
> few people instead of a crowd of indistinguishable people.
>
> You can of course also render video captures in more than one place on
> the screen, which makes no sense for audio.
>

I'm hoping those of you who know more about this than I do can do the 
refinement.

I'm approaching this from an information theoretic perspective - what 
inferences are possible given the information in the framework. And at 
the moment it seems to me that lots of unexplained assumptions are 
required to use the information in the advertisement to correlate audio 
with video, or to extract enough about the audio to decide how best to 
render it.

	Thanks,
	Paul