Subject: [clue] VAD and speaker coordinates

From: Paul Kyzivat <[email-masked]>
Date: Mon, 06 Feb 2012 16:31:53 -0500

(As individual)

In the design team meeting last week there was discussion of 
representing the speaker location as part of the VAD data - concern with 
optimizing it.

I'm somewhat naive here, but ISTM that there are principles to work out 
before getting to this. In particular, as I understand it a lot of 
inference is required to get from energy levels to VAD to speaker 
detection to speaker location.

E.g. consider a case where there is a composed video stream that tiles a 
number of captures, and a corresponding composed audio stream that mixes 
corresponding audio captures.

The audio mixer could do its own speaker detection by processing the 
audio, and tag it with the location in the video where the corresponding 
video is placed.

But its possible that the sources for the individual audio and video 
streams have already tagged the audio with VAD info. If so, how would 
that information be handled by the mixer? Does it circumvent the 
explicit audio processing and replace it with a distinct mixing process 
for VAD info?

Also, is it the case that energy level is filtered in some way so that 
loud non-voice noises are distinguished from actual speech?

	Thanks,
	Paul

---

From: John Leslie <[email-masked]>
Date: Mon, 6 Feb 2012 18:41:07 -0500

> detection to speaker location.

   I believe the technical term is "PFM". ;^)

> corresponding audio captures.

   Yes, that could happen too...

> audio, and tag it with the location in the video where the corresponding 

   I'll let someone else volunteer to do that...

> explicit audio processing and replace it with a distinct mixing process 

   This is a good question: if we had good speaker-location info, what
could we do with it?

> loud non-voice noises are distinguished from actual speech?

   That is technically feasible...

   But I'd like to suggest we instead think in terms of identifying the
physical location of the _microphone_ with greatest speech-like sound.
That physical location is pretty easy to know; and if folks really
care about identifying _speaker_ location, they can use more mikes.

   IMHO, the current state of the art can only _guess_ at speaker
location when speakers share a microphone; and I'd rather not base
our spec on guesses...

--
John Leslie <[email-masked]>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 06 Feb 2012 20:16:06 -0500

On 2/6/12 6:41 PM, John Leslie wrote:
> Paul Kyzivat<[email-masked]>  wrote:
>>

PFM = P??? F??? M???

>     I'll let someone else volunteer to do that...

This actually seems like one of the easy things, if it is mixing matched 
pairs of audio/video. It only has to decide which audio is the loudest 
speaker and then tag it with the location in the mixed video where the 
corresponding video stream is being placed.

> location when speakers share a microphone; and I'd rather not base

I'm easy - if that is the goal, then go ahead and specify that. At least 
there is a reasonable chance of defining it.

	Thanks,
	Paul

---

From: "Duckworth, Mark" <[email-masked]>
Date: Tue, 7 Feb 2012 18:23:10 -0800

John Leslie wrote:
   "But I'd like to suggest we instead think in terms of identifying the physical location of the _microphone_ with greatest speech-like sound.  That physical location is pretty easy to know; and if folks really care about identifying _speaker_ location, they can use more mikes."

I disagree.  If a media provider wants to estimate the speaker (talker) location based on which microphone has the greatest speech-like sound, that is fine.  But some microphone arrangements (along with the audio processing algorithms) just don't work that way.   For example, microphone arrays where there is no direct constant relationship between a microphone element and an encoded audio channel.

So if those systems identified the location of the microphone with greatest speech-like sound it wouldn't necessarily have any relation to the location of the talker.  I think it is better for the media provider to estimate (usually better than a guess!) where the talker is, using whatever method makes sense for that provider.  We don't want the media consumer to be making incorrect assumptions about how the provider's microphone system works.

Mark

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of John Leslie
Sent: Monday, February 06, 2012 6:41 PM
To: Paul Kyzivat
Cc: CLUE
Subject: Re: [clue] VAD and speaker coordinates

Paul Kyzivat <[email-masked]> wrote:
> representing the speaker location as part of the VAD data - concern 
> with optimizing it.
> 
> I'm somewhat naive here, but ISTM that there are principles to work 
> out before getting to this. In particular, as I understand it a lot of 

> E.g. consider a case where there is a composed video stream that tiles 
> a number of captures, and a corresponding composed audio stream that 
> mixes corresponding audio captures.

> audio, and tag it with the location in the video where the 
> corresponding video is placed.

> explicit audio processing and replace it with a distinct mixing 
> process for VAD info?

   This is a good question: if we had good speaker-location info, what could we do with it?

   But I'd like to suggest we instead think in terms of identifying the physical location of the _microphone_ with greatest speech-like sound.
That physical location is pretty easy to know; and if folks really care about identifying _speaker_ location, they can use more mikes.

   IMHO, the current state of the art can only _guess_ at speaker location when speakers share a microphone; and I'd rather not base our spec on guesses...

--
John Leslie <[email-masked]>
_______________________________________________
clue mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/clue

---

From: John Leslie <[email-masked]>
Date: Wed, 8 Feb 2012 16:59:05 -0500

Duckworth, Mark <[email-masked]> wrote:
> John Leslie wrote:
> 
>> "But I'd like to suggest we instead think in terms of identifying
>> the physical location of the _microphone_ with greatest speech-like
>> sound. That physical location is pretty easy to know; and if folks
>> really care about identifying _speaker_ location, they can use more
>> mikes."
> 
> I disagree.  If a media provider wants to estimate the speaker (talker)
> location based on which microphone has the greatest speech-like sound,
> that is fine.  But some microphone arrangements (along with the audio
> processing algorithms) just don't work that way.

   Umm...

> For example, microphone arrays where there is no direct constant
> relationship between a microphone element and an encoded audio channel.

   If you're talking about the likes of Polycom HDX, I consider that
a microphone "pointed" down with a time-variable pattern.

   Granted, "greatest speech-like sound" is hand-waving, but I think
we're stuck with that problem anyway: there needs to be an algorithm
to say "none of the individual mikes seem to be close to this speaker;
thus we select the room-general mike".

   And as a sound guy, I want to know that the room-general mike is
the one selected _before_ I worry about the coordinates of the actual
speaker. I would likely treat the computed coordinates as a hint
rather than an established fact.

> So if those systems identified the location of the microphone with
> greatest speech-like sound it wouldn't necessarily have any relation
> to the location of the talker.

   Correct!

> I think it is better for the media provider to estimate (usually better
> than a guess!) where the talker is, using whatever method makes sense
> for that provider.

   It is reasonable to provide that estimate in addition to the location
of the room-general microphone.

   But IMHO, different providers will use different algorithms, and I
don't see how to know the accuracy of the estimate.

> We don't want the media consumer to be making incorrect assumptions
> about how the provider's microphone system works.

   Exactly!

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Thu, 9 Feb 2012 00:52:59 +0100

Are we discussing a use case where we want information about the speaker
position, e.g

"User Bob is in a call with Alice, Bob see that Alice is standing left
in the video stream. Bob wants to hear the audio from Alice being from
the left side"  

Some comments 
* A single audio stream is enough 
* Without any speaker position information the default is to play out
audio in the center of the video stream 
* If dynamic speaker position is sent, you could play out the audio in
the correct position 
* Speaker position is not always possible to detect or some rooms might
skip the whole thing, so it's optional. 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
John Leslie
Sent: 8. februar 2012 22:59
To: Duckworth, Mark
Cc: CLUE
Subject: Re: [clue] VAD and speaker coordinates

   Umm...

   Correct!

   Exactly!

---

From: John Leslie <[email-masked]>
Date: Mon, 13 Feb 2012 11:23:07 -0500

This really deserves a reply, but it is proving hard to write: :^(

> in the video stream. Bob wants to hear the audio from Alice being from

   It's _really_ not that simple.

   Having mono audio bounce between speakers will sort-of work if we're
all agreed there are only two speakers. If there are more than two, it
won't work at all.

> * A single audio stream is enough 

   Enough for what?

   A single monophonic speaker does have the advantage of never confusing
the listener by having the current speaker's position move arbitrarily...

>   audio in the center of the video stream

   Yes, that is a reasonable default.
 
> * If dynamic speaker position is sent, you could play out the audio in

   There-Ain't-No-Such-Thing-As-A "correct position". :^(

   We're trying to create a virtual room; and except in the trivial case
where one room has half the speakers and a second room has the other
half, we're reduced to asking the participants to suspend their disbelief
that they're in this virtual room.

   For that trivial case, we can treat each set of screens as a virtual
glass partition, let participants listen to the folks on their side
without any amplification, and reproduce the audio which "would" pass
through the partition if it weren't there.

   (There _will_ be some extended echo problems, but those can be managed.)

   Once we get past this virtual-room-in-two-pieces, we're in trouble!
IMHO, we shouldn't even try to define how the audio will be managed --
but we should provide a means of passing the information which may be
needed for _different_ ways of presenting the audio.

   One example would be to keep the "divided-room" paradigm, and add in
individuals (perhaps shown in insets on one of the screens), always in
the same virtual (three-dimensional) audio position, when they are
speaking. (A better way, of course, would be to give them their own
screen and speaker.)

   IMHO, the critical information is the _virtual_ position of the speaker,
the actual position of the microphone, and _perhaps_ the relative actual
position of the speaker to the microphone. Room acoustics is a critical
part of how humans "place" a sound source; and screwing up the room
acoustics to "help" them won't help. (IMHO, of course...)

> skip the whole thing, so it's optional. 

   Agreed!

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Tue, 14 Feb 2012 21:36:33 +0100

Hi John 

Discussions of Audio recording and playback is something that will
benefits from a whiteboard! 

See inline. 

-Espen 

-----Original Message-----
From: John Leslie [mailto:[email-masked]] 
Sent: 13. februar 2012 17:23
To: Espen Berger (espeberg)
Cc: CLUE
Subject: Re: [clue] VAD and speaker coordinates

   It's _really_ not that simple.
[Espen Berger (espeberg)] My use case is this simple, other use cases
might be more complex. With a single speaker in a room, dynamic speaker
position work nicely.
[Espen]

   Having mono audio bounce between speakers will sort-of work if we're
all agreed there are only two speakers. If there are more than two, it
won't work at all.
[Espen Berger (espeberg)] I do not see that modeling more than two
active talkers is hard. To do proper playback is hard, but last resort
is always to mix all audio together and play back on a single speaker. 

> * A single audio stream is enough 

   Enough for what?
[Espen Berger (espeberg)] Enough to play out on active talker in any
position, as long as you receive the dynamic position information. 

   A single monophonic speaker does have the advantage of never
confusing
the listener by having the current speaker's position move
arbitrarily...

 

   We're trying to create a virtual room; and except in the trivial case
where one room has half the speakers and a second room has the other
half, we're reduced to asking the participants to suspend their
disbelief
that they're in this virtual room.
[Espen Berger (espeberg)] The term Virtual room is not well defined. 

   (There _will_ be some extended echo problems, but those can be
managed.)
[Espen Berger (espeberg)] AFAIK the echo cancelation is a implementation
challenge more than a modeling challenge. 

[Espen Berger (espeberg)] Agree, we should focus on the meta-information
sent and received.

   IMHO, the critical information is the _virtual_ position of the
speaker,
the actual position of the microphone, and _perhaps_ the relative actual
position of the speaker to the microphone. Room acoustics is a critical
part of how humans "place" a sound source; and screwing up the room
acoustics to "help" them won't help. (IMHO, of course...)
[Espen Berger (espeberg)] I still believe that the capture room is
better at figuring out the speaker position than the reproducing room.
At capture time you can do pre-processing if needed to calculate
position and also mix multiple microphone inputs together into a single
audio stream, e.g. when two microphones picks up the audio from the same
speaker.

   Agreed!

---

From: John Leslie <[email-masked]>
Date: Wed, 15 Feb 2012 06:53:00 -0500

> benefits from a whiteboard! 

   Indeed... especially a three-dimensional one!

   ;^)

> might be more complex. With a single speaker in a room, dynamic speaker

   We really ought to come up with a better way of distinguishing a
human speaking from an audio transducer...

   There are use-cases simple enough to be processed into a position
in a stereo or surround system -- I don't find them interesting...

> is always to mix all audio together and play back on a single speaker. 

   Indeed, that works.

> position, as long as you receive the dynamic position information. 

   Well, no.

   If we're dealing with a lapel mike in a reasonably dead room, we
can process that into a "position" in stereo / surround. In practice,
it's seldom that straightforward. Folks share microphones, the room
is too noisy, echos become too confusing...

>>> * If dynamic speaker position is sent, you could play out the audio
>>>   in the correct position 
>>
>> disbelief that they're in this virtual room.
>

   My bad.

   I see that my post could be read to ask CLUE to define a virtual room.

   That was not my intent. It's merely how I think about the audio issue:
others are free to think about it differently. I think about it that way
because that is my model of how _participants_ experience telepresence:
they're in a room which doesn't actually exist, but "feels" like it does.

> challenge more than a modeling challenge. 

   Actually, there is a modeling challenge there -- to accurately map
the delay times so the jitter doesn't kill you...

   But I inherently don't trust "echo cancellation" algorithms. They
work pretty well in simple cases (two speakers, no jitter), but fall
apart when you switch more speakers in and out in the presence of
jitter.

>> IMHO, the critical information is the _virtual_ position of the

   My bad, again. :^(

   For _my_ rendering purposes, I need this virtual position. That does
_not_ mean it needs to be the same as any other renderer's virtual
position for that (human) speaker.

> better at figuring out the speaker position than the reproducing room.

   Agreed, the capture room has better information with which to guess
the (human) speaker postion.

   But the rendering room must take responsibility for the audio
experience of its occupants. Trusting the capture room too much about
the physical positions of its occupants is hazardous.

> audio stream, e.g. when two microphones picks up the audio from the same

   Indeed you can -- though for the most part I wish you wouldn't.
Calculating position should be harmless (assuming you don't change the
audio being captured); but mixing two audio streams screws with the
room acoustics.

   My intended point, really, is that I prefer knowing microphone
positions to guessing (human) speaker positions. Were I to accurately
map each microphone audio capture to a virtual audio transducer in
my corresponding virtual room (and not change things too suddenly),
the human listeners would pretty much work it out by themselves.

   What does this mean to a CLUE protocol? Not a whole lot, perhaps.
If someone is mixing audio to create the streams sent, I want to be
warned: other than that, I really want microphone positions.

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Sun, 19 Feb 2012 23:12:28 +0100

I think we need a better understanding of what should be done at record
time and what should be done at playback is needed. It would be valuable
to have more detailed use cases, to make us better equipped to discuss
the same scenarios.

The best would be to find a model where a room can do pre-processing and
send a single audio stream with position information, which I also is
important for interoperability with systems only capable of process a
limited set of audio information. At the same time it would be great to
allow for more meta-information to be sent and received when needed. 

Cheers

-Espen  

-----Original Message-----
From: John Leslie [mailto:[email-masked]] 
Sent: 15. februar 2012 12:53
To: Espen Berger (espeberg)
Cc: CLUE
Subject: Re: [clue] VAD and speaker coordinates

   ;^)

>>> "User Bob is in a call with Alice, Bob see that Alice is standing 
>>> left in the video stream. Bob wants to hear the audio from Alice 
>>> being from the left side"
>>
> might be more complex. With a single speaker in a room, dynamic 
> speaker position work nicely.

   There are use-cases simple enough to be processed into a position in
a stereo or surround system -- I don't find them interesting...

>> all agreed there are only two speakers. If there are more than two, 
>> it won't work at all.
> 

   Indeed, that works.

   Well, no.

   If we're dealing with a lapel mike in a reasonably dead room, we can
process that into a "position" in stereo / surround. In practice, it's
seldom that straightforward. Folks share microphones, the room is too
noisy, echos become too confusing...

   My bad.

   I see that my post could be read to ask CLUE to define a virtual
room.

   That was not my intent. It's merely how I think about the audio
issue:
others are free to think about it differently. I think about it that way
because that is my model of how _participants_ experience telepresence:
they're in a room which doesn't actually exist, but "feels" like it
does.

> [Espen Berger (espeberg)] AFAIK the echo cancelation is a 
> implementation challenge more than a modeling challenge.

   My bad, again. :^(

>> the actual position of the microphone, and _perhaps_ the relative 
>> actual position of the speaker to the microphone. Room acoustics is a

>> critical part of how humans "place" a sound source; and screwing up 
>> the room acoustics to "help" them won't help. (IMHO, of course...)
> 

> position and also mix multiple microphone inputs together into a 
> single audio stream, e.g. when two microphones picks up the audio from

> the same speaker.

   Indeed you can -- though for the most part I wish you wouldn't.
Calculating position should be harmless (assuming you don't change the
audio being captured); but mixing two audio streams screws with the room
acoustics.

   My intended point, really, is that I prefer knowing microphone
positions to guessing (human) speaker positions. Were I to accurately
map each microphone audio capture to a virtual audio transducer in my
corresponding virtual room (and not change things too suddenly), the
human listeners would pretty much work it out by themselves.