Subject: Re: [clue] Steered microphone arrays

From: Stephen Botzko <[email-masked]>
Date: Mon, 2 Apr 2012 12:02:57 -0400

in-line

On Tue, Mar 27, 2012 at 6:32 PM, John Leslie <[email-masked]> wrote:

> Stephen Botzko <[email-masked]> wrote:
> > On Tue, Mar 27, 2012 at 6:12 AM, John Leslie <[email-masked]> wrote:
> >> Stephen Botzko <[email-masked]> wrote:
> >>
> >>> [sb]  Are you suggesting that every system using a steered microphone
> >>> array would be tagged as "mixed"?
> >>
> >> Actually, yes: in the case of a steered _array_ of microphones, the
> >> arrival-time information will be lost, just as in an audio mixer with
> >> multiple microphones.
> >>
> >> This is distinct from the case of a steered "shotgun" microphone where
> >> the mike position is fixed (though its directional pattern changes),
> thus
> >> arrival-time information is preserved.
> >>
> >> [sb] I believe steered microphone arrays are used commonly in sonar
> >> acquisition,
>
>   This is true...
>
> >> so I don't believe arrival time information has to be lost.
> >> Usually the microphones in such an array are in fixed positions.
>
>   As usual, the devil is in the details...
>
>   In sonar, there's a very clearly-positioned sound source, and the
> return times of the microphone array are calibrated to that source.
> (Actually, the source likely produces a "chirp" of varying wavelength
> and the phase differences are processed during the chirp response.)
>
>   In our microphone array, the position of the sound source is
> variable and there's no control of its wavelengths. It is technically
> possible to compare phase information of several microphones and
> process the sound output to that which "would be received" by a mike
> at a given position (within the range of the microphone array, of
> course); but if any existing system does that, I haven't heard of it.
>
[sb] we have built systems that do that, and that is exactly what I mean by
a steerable array.

>
>   Again, we're going to have to trust operators to set the "mixed"
> bit: we can only offer guidance...
>
> --
> John Leslie <[email-masked]>
>

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Tue, 3 Apr 2012 09:37:14 +0200

Hi

A microphone array can behave similar to a steered directional microphone, with a fixed position (center of array) and focussing its beam in different directions. Like a passive sonar.

The concept of preserving arrival-time information seems to me to be useful only when you have several microphones somehow translating into several output streams, in which case an array with a single processed output and a shotgun would behave equally as one of those microphones.

An array can also have several output streams, looking in several directions and picking up spatially separated sound sources simultaneously. However, the effective mic position would still be fixed and the same for the different sources, no inherent arrival-time differences.

The array can, however, know which direction it is looking, and use this to embed time (and/or level) differences in output streams. Or (perhaps better) add meta data about direction or even position to the streams.

I would not tag the single stream output of a (steered or not steered) microphone array as "mixed". Logically it is a single audio capture with potentially high (and time-varying) directivity, but a fixed position.

However, John's goal of preserving sound source direction information is important and should definitely be worked on further.

The "virtual microphone" is a slightly different thing, where an array in one position is processed to a single output with the illusion of being a microphone placed somewhere else. I agree with Stephen that it can be done, but it has some challenges and I doubt that it will be an important usecase in telepresence anytime soon.

Regards
Johan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Stephen Botzko
Sent: 2. april 2012 18:03
To: John Leslie
Cc: [email-masked]
Subject: Re: [clue] Steered microphone arrays

in-line
On Tue, Mar 27, 2012 at 6:32 PM, John Leslie <[email-masked]> wrote:
Stephen Botzko <[email-masked]> wrote:
>> the mike position is fixed (though its directional pattern changes), thus

  This is true...

[sb] we have built systems that do that, and that is exactly what I mean by a steerable array.

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 3 Apr 2012 07:22:15 -0400

[in line]

On Tue, Apr 3, 2012 at 3:37 AM, Johan Ludvig Nielsen (johaniel) <
[email-masked]> wrote:

> Hi
>
> A microphone array can behave similar to a steered directional microphone,
> with a fixed position (center of array) and focussing its beam in different
> directions. Like a passive sonar.
>
> The concept of preserving arrival-time information seems to me to be
> useful only when you have several microphones somehow translating into
> several output streams, in which case an array with a single processed
> output and a shotgun would behave equally as one of those microphones.
>
> An array can also have several output streams, looking in several
> directions and picking up spatially separated sound sources simultaneously.
> However, the effective mic position would still be fixed and the same for
> the different sources, no inherent arrival-time differences.
>
> The array can, however, know which direction it is looking, and use this
> to embed time (and/or level) differences in output streams. Or (perhaps
> better) add meta data about direction or even position to the streams.
>
> [sb]generally agree  I would rather see the source send the proper level
rather than force receivers to adjust.  One reason is that VAD detection
generally includes energy, so VAD will be mismatched if senders use
mismatched levels.  This biases switching decisions.  Though I think this
is out of scope for CLUE.

I would not tag the single stream output of a (steered or not steered)
> microphone array as "mixed". Logically it is a single audio capture with
> potentially high (and time-varying) directivity, but a fixed position.
>

[sb] I agree it is not mixed.  Time-varying directivity is a problem -
signaling in the media plane would help, but there is still the
disaggregated media challenge.  Another option is to generate multiple
captures from the steered array, to allow the capture attributes to be
fixed.  For instance, generate a stereo capture, using panning to get the
current position correct.  The directivity then results in good noise
rejection.  Some future codecs (SAOC) would allow the position of each
human speaker in the capture to be signaled in the audio stream - which
should not be precluded.

>
> However, John's goal of preserving sound source direction information is
> important and should definitely be worked on further
>
[sb] We have talked much more about video than audio so far.  It is good to
see more audio discussion.

The "virtual microphone" is a slightly different thing, where an array in
> one position is processed to a single output with the illusion of being a
> microphone placed somewhere else. I agree with Stephen that it can be done,
> but it has some challenges and I doubt that it will be an important usecase
> in telepresence anytime soon.

[sb]The protocol will hopefully be relevant for many years.  We have used
microphone arrays for a long time to do automatic camera steering,  It is
forseeable that such arrays will be used for telepresence.  One advantage
they have is they allow multiple capture advertisements to be generated
from a single set of sensors.

---

From: Andy Pepperell <[email-masked]>
Date: Fri, 6 Apr 2012 09:16:34 +0100

It's clear from the discussions that there's some contention over the
proposed use of the "composed" attribute. While I maintain that there
is use in a consumer knowing that a video stream is not suitable for
use in a new composed, multiple pane arrangement, it's also clear that
a capture could be composed / mixed without this being true (the
simple case of 2 adjacent seats covered by a single camera [capture]).

Perhaps there's some merit in separating these concepts? Specifically,
we could have a "composed" attribute which the provider would use to
mean "this capture can be broken down into constituent elements" and a
separate "artificial" attribute to signify "this is something that
shouldn't be recombined with other captures if there's an
alternative".

This would leave the composed / mixed attribute open for all provider
devices to use (i.e. not just those performing MCU-style layout
compositions), and open to being extended to cover the dynamic update
case (a provider indicating changes in who's being shown where as that
changes, which could be said to be equally applicable to, say, the 4
panes of a 2 x 2 MCU layout as to 2 people sat next to each other at a
desk covered by a single camera [capture]).

Regards,

Andy

On Fri, Mar 30, 2012 at 2:52 PM, Charles Eckel (eckelcu)
<[email-masked]> wrote:
> Reiterating a comment I made a the mic, I would like to see the addition
> of semantics that describe the composed/mixed media capture in terms of
> other media captures, when applicable. I think this would be very useful
> information for a consumer.
>
> Cheers,
> Charles
>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of
>> Paul Kyzivat
>> Sent: Wednesday, March 28, 2012 11:40 PM
>> To: Duckworth, Mark
>> Cc: [email-masked]
>> Subject: Re: [clue] Use of the media capture "composed" attribute
>>
>> I think this further verifies that we need a better explanation of the
>> intended purpose(s) of these (mixed/composed) attributes so that we
> can
>> assess whether we have an actionable definition sufficient to the
> need.
>>
>>       Thanks,
>>       Paul
>>
>> On 3/28/12 10:52 AM, Duckworth, Mark wrote:
>> > Paul asked:
>> >> If mixed and composed aren't intended to drive decisions about
>> further
>> >> mixing and composing, then what *are* they for?
>> >
>> > I was thinking one other case where the composed attribute would be
>> used is when a consumer (an endpoint in this case) is choosing which
>> media captures to receive from a provider.  Suppose the provider
>> advertises capture scene entries that are the same except one of them
>> has video captures that use composed=true, and the other uses
>> composed=false.  I think the consumer can reasonably assume the one
>> with composed=true includes more information somehow, possibly with
>> "Hollywood squares" or picture-in-picture video layout.  The consumer
>> can use this limited information to make a choice.  This choice has
>> nothing to do with whether or not the receiving endpoint is going to
>> re-compose the images.
>> >
>> > Mark
>> >
>> >> -----Original Message-----
>> >> From: [email-masked] [mailto:[email-masked]] On
> Behalf
>> Of
>> >> Paul Kyzivat
>> >> Sent: Tuesday, March 27, 2012 11:35 PM
>> >> To: [email-masked]
>> >> Subject: Re: [clue] Use of the media capture "composed" attribute
>> >>
>> >> [I don't know if this is as chair or individual. Maybe both.]
>> >>
>> >> The discussion of the composed and mixed attributes today exposed a
>> lot of
>> >> vagueness about what they mean. I think this needs to be firmed up
>> >> considerably.
>> >>
>> >> Here are some thoughts of my own just to keep the discussion going:
>> >>
>> >> Regarding mixed (audio):
>> >>
>> >> The fact that an audio capture has mixed content doesn't preclude
>> mixing it
>> >> again. But there are properties of it that might make such mixing
>> >> unsatisfactory. For instance, if AC3 is a mix of (AC1, AC2), then
> it
>> is ok to mix
>> >> AC3 and independent capture AC4. But it would be unwise to mix it
>> with AC2
>> >> again. And it would also be unwise to mix it with AC5 that is
> itself
>> a mix of
>> >> (AC1, AC4).
>> >>
>> >> It *might* also be the case that a mix is unwise if it includes
>> >> (transitively) too many original inputs.
>> >>
>> >> Fundamentally it seems like the decision of whether a mix makes
>> sense
>> >> requires some complex decision making based on way more data than
>> >> whether the immediate inputs are themselves mixed or not.
>> >>
>> >> Regarding composed (video):
>> >>
>> >> This seems potentially more complex. So far the only use I have
>> heard for
>> >> this attribute is to drive a decision about whether it makes sense
>> to compose
>> >> this capture with others. But I expect that isn't always true
>> either. You could
>> >> have one uncomposed capture that shows two people
>> >> (heads) side by side. You could have another *composed* capture
> that
>> also
>> >> shows two people in independently side by side each inside a frame,
>> that
>> >> were assembled from other captures. (Would anyone argue that this
>> isn't
>> >> composed?) But I can't see why it would be any more or less
>> desirable to
>> >> compose one of these captures than another.
>> >>
>> >> Again it seems like you need more information about what has been
>> >> composed, and at what scale (and maybe position) before you can
> make
>> a
>> >> decision about whether it makes sense to compose it. And then that
>> may
>> >> also depend on *how* you want to compose it.
>> >>
>> >> If mixed and composed aren't intended to drive decisions about
>> further
>> >> mixing and composing, then what *are* they for?
>> >>
>> >>    Thanks,
>> >>    Paul
>> >> _______________________________________________
>> >> clue mailing list
>> >> [email-masked]
>> >> https://www.ietf.org/mailman/listinfo/clue
>> >
>>
>> _______________________________________________
>> clue mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/clue
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

---

From: Andy Pepperell <[email-masked]>
Date: Fri, 6 Apr 2012 09:24:03 +0100

Hi Charles,

re: this specific point, it could be said that to an extent the
structure / semantics of the capture scene entries do give you this
information, in that each such entry in a capture scene captures
"equivalent" data, so, if, say, you have a capture scene entry
consisting of just VC1 and there's another entry in the same capture
scene consisting of VC2, VC3 and VC4 then, whether or not VC1 is
tagged as composed/mixed, it to an extent "covers" VC2, VC3 and VC4
(and the area of capture co-ordinates can be used to determine this in
more detail).

One issue with separate semantics for describing captures in terms of
other captures, to my mind, is that it wouldn't give as fine-grained
information as one might want. In the case of a single VCn being
supplied from a single-camera endpoint, presumably a fairly common
case, we might want to have a provider -> consumer mechanism for
describing who's on the left and right of the image without the need
to invent new sub-captures for this purpose.

Andy

---

From: Marshall Eubanks <[email-masked]>
Date: Fri, 6 Apr 2012 08:03:01 -0400

On Tue, Mar 27, 2012 at 10:14 AM, Stephen Botzko
<[email-masked]> wrote:
>> > On Wed, Mar 21, 2012 at 9:23 PM, John Leslie <[email-masked]> wrote:
>> >> Roni Even <[email-masked]> wrote:
>> >>> [Andy Pepperell wrote:]
>> >>>>
>> >>>> With more thought on this, we also came to the conclusion that the
>> >>>> use
>> >>>> for this attribute for video captures doesn't necessarily translate
>> >>>> exactly equivalently to audio captures. In a network of cascaded MCUs
>> >>>> it *would* often be perfectly acceptable to use a pre-mixed audio
>> >>>> stream from a peer MCU whereas it would be highly desirable to not
>> >>>> use pre-composed, multi-pane, video from those media providers.
>>
>>   (I agree that pre-mixed audio may sometimes be preferred to doing
>> your own mixing, for the same room where pre-composed video would _not_
>> be preferred.)
>>
>> >>>> Largely because of the potential different use of the attribute by
>> >>>> media stream consumers, our thinking is that we *shouldn't* define
>> >>>> the "composed" attribute to apply to both video and audio captures,
>> >>>> and instead use, if required, a separate "mixed" attribute for the
>> >>>> audio case.
>> >>
>> >> I'm not sure it needs to be "separate", but I would hope we have a
>> >> tag to show that two or more microphones in different positions were
>> >> "mixed" into a single audio stream.
>> >>
>> >> I view this as a warning label -- and a single bit suffices unless
>> >> we include details of the microphones being mixed (which I don't
>> >> believe
>> >> is the current plan).
>> >>
> acquisition, so I don't believe arrival time information has to be lost.

In a steered array there has to be an array phase reference location (where
all of the signals are delayed to be in phase at). It doesn't have to
be the same
location for each steered beam, but I think it generally is in
practice. (It also doesn't have to be the
location of any physical microphone.) Note that if it is in the middle
of the array, then
in general some inputs would need to be retarded, some advanced, to be
in sync, and so
(for a real time system) the output will actually be what a virtual
microphone at the array
phase reference would have heard some time before (that time being at
least the delay to
the last receiver to receive that phase).

So, if the phased array picks some convenient spot in the middle to be the
phase reference, it will of necessity output the sound delayed by some
amount (at least the phase delay between
that spot and the last receiver). Of course, this phase shift delay
has to be added to any processing delays and digitization delays, but
these
are probably similar to those for conventional microphones.

Suppose the array is 1 meter in radius; then the array phase delay
will be < or ~ 3 msec.  That would be a tiny
delay for two telepresence systems not located in the same building,
so could probably be ignored. Even if
the array spanned the entire telepresence space (say, 10 meters), the
delay is only 30 msec. Only if the array was spanning,
say, an entire IETF plenary would the delay become problematic. (Note,
however, that arrays surrounding sources can give
rise to more complexities.)

Given all of this, I don't see any intrinsic reason why an array
system can't be regarded as giving proper arrival time information.

Regards
Marshall

>>
>> > I don't see that as helpful, so I think we would need to define the
>> > meaning of "mixed" here carefully.
>>
>>   We probably do need a careful definition...
>>
>>   To me, the essential point is that a mixer greatly obscures any
>> ability to process for actual physical location of the microphone --
>> thus I'm probably going to fail if I try to aid the listeners in
>> recovering the actual physical location of the speaker(s).
>>
>>   Of course, regardless of our definition, such a "mixed" bit will end
>> up being set wrong sometimes -- but I'd still like the warning label.
>>
>>   In an actual room, listeners with binaural hearing determine the
>> speaker's location quite unconsciously -- but they do use physical
>> position as one input into speaker recognition (and where to focus
>> their attention). When a remote speaker is presented through a
>> monophonic speaker, they describe it as a "dissociated voice" and
>> are subconsciously confused.
>>
>>   Given "mixed" audio, I can still process it to a single apparent
>> position, but I can't relate that position to any other sound source;
>> nor can I usefully vary it as different speakers are included in that
>> audio stream.
>>
>> --

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 06 Apr 2012 08:53:21 -0400

> shouldn't be recombined with other captures if there's an

I'm still confused by these. You are stating these definitions from the 
perspective of the recipient, but without much of an operational 
definition for the provider. And even from the point of view of the 
recipient this is rather unmotivated.

(What does "this capture can be broken down into constituent elements" 
mean to the recipient? Without some added data about where the 
constituent elements are in the composed image, and what they represent, 
this is not helpful. And how does one decide whether "this is something 
that shouldn't be recombined"?)

ISTM that it would be more helpful to define attributes in operational 
terms that the provider can use to decide the value of the attribute. If 
that is well defined (rather than being subjective) then the recipient 
can then make its own decisions about what is appropriate to do with the 
capture.

	Thanks,
	Paul (as individual)

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 06 Apr 2012 09:01:15 -0400

> (and the area of capture co-ordinates can be used to determine this in

ISTM that the area of capture is more meaningful here.
Its a leap of faith to assume that just because VC1 is an alternative to 
VC2+VC3+VC4 that it must contain a rendition of everything in 
VC2+VC3+VC4. It might just show the active speaker, or it might just 
show the most important person, or just the presentation.

> information as one might want.

This is all hypothetical right now. Presumably it can give as detailed a 
description as clue makes provision for describing.

> In the case of a single VCn being

AFAIK we don't currently have any mechanism to describe who is where, 
regardless of how many captures there are. (Though perhaps we should.)

	Thanks,
	Paul (as individual)