Subject: [clue] audio mixed attribute

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 28 Mar 2012 02:06:31 -0700

Hi John,
I'm trying to understand your view on labeling audio as mixed.
Here is one example scenario that might help me understand.
Suppose I am a provider, and I can capture and send audio in two different ways:
Option 1: I can send one mono audio capture.  It comes from one microphone element that picks up audio from the whole room.
Option 2: I can send one mono audio capture.  It comes from several microphone elements mixed together, that collectively pick up audio from the whole room.

I think you are saying it would be useful to label option 1 as not mixed, and label option 2 as mixed.  Is that correct?

What might a receiver do differently if it receives option 1 vs. receiving option 2?

Regards,
Mark

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> John Leslie
> Sent: Tuesday, March 27, 2012 6:13 AM
> To: Stephen Botzko
> Cc: [email-masked]
> Subject: Re: [clue] Use of the media capture "composed" attribute
> 
> Stephen Botzko <[email-masked]> wrote:
> > On Wed, Mar 21, 2012 at 9:23 PM, John Leslie <[email-masked]> wrote:
> >> Roni Even <[email-masked]> wrote:
> >>> [Andy Pepperell wrote:]
> >>>>
> >>>> With more thought on this, we also came to the conclusion that the
> >>>> use for this attribute for video captures doesn't necessarily
> >>>> translate exactly equivalently to audio captures. In a network of
> >>>> cascaded MCUs it *would* often be perfectly acceptable to use a
> >>>> pre-mixed audio stream from a peer MCU whereas it would be highly
> >>>> desirable to not use pre-composed, multi-pane, video from those
> media providers.
> 
>    (I agree that pre-mixed audio may sometimes be preferred to doing your
> own mixing, for the same room where pre-composed video would _not_ be
> preferred.)
> 
> >>>> Largely because of the potential different use of the attribute by
> >>>> media stream consumers, our thinking is that we *shouldn't* define
> >>>> the "composed" attribute to apply to both video and audio captures,
> >>>> and instead use, if required, a separate "mixed" attribute for the
> >>>> audio case.
> >>
> >> I'm not sure it needs to be "separate", but I would hope we have a
> >> tag to show that two or more microphones in different positions were
> >> "mixed" into a single audio stream.
> >>
> >> I view this as a warning label -- and a single bit suffices unless we
> >> include details of the microphones being mixed (which I don't believe
> >> is the current plan).
> >>
> > [sb]  Are you suggesting that every system using a steered microphone
> > array would be tagged as "mixed"?
> 
>    Actually, yes: in the case of a steered _array_ of microphones, the arrival-
> time information will be lost, just as in an audio mixer with multiple
> microphones.
> 
>    This is distinct from the case of a steered "shotgun" microphone where the
> mike position is fixed (though its directional pattern changes), thus arrival-
> time information is preserved.
> 
> > I don't see that as helpful, so I think we would need to define the
> > meaning of "mixed" here carefully.
> 
>    We probably do need a careful definition...
> 
>    To me, the essential point is that a mixer greatly obscures any ability to
> process for actual physical location of the microphone -- thus I'm probably
> going to fail if I try to aid the listeners in recovering the actual physical
> location of the speaker(s).
> 
>    Of course, regardless of our definition, such a "mixed" bit will end up being
> set wrong sometimes -- but I'd still like the warning label.
> 
>    In an actual room, listeners with binaural hearing determine the speaker's
> location quite unconsciously -- but they do use physical position as one input
> into speaker recognition (and where to focus their attention). When a
> remote speaker is presented through a monophonic speaker, they describe
> it as a "dissociated voice" and are subconsciously confused.
> 
>    Given "mixed" audio, I can still process it to a single apparent position, but I
> can't relate that position to any other sound source; nor can I usefully vary it
> as different speakers are included in that audio stream.
> 
> --
> John Leslie <[email-masked]>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

---

From: John Leslie <[email-masked]>
Date: Thu, 29 Mar 2012 11:26:00 -0400

Duckworth, Mark <[email-masked]> wrote:
> Option 1: I can send one mono audio capture.  It comes from one
>           microphone element that picks up audio from the whole room.
> Option 2: I can send one mono audio capture.  It comes from several
>           microphone elements mixed together, that collectively pick
>           up audio from the whole room.
> 
> I think you are saying it would be useful to label option 1 as not
> mixed, and label option 2 as mixed.  Is that correct?

   Yes.

> What might a receiver do differently if it receives option 1 vs.

   (To answer, I need to assume the receiver can to some degree choose
which to receive.)

   The receiver, for example, could reasonably guess that the "mixed"
version will have a more uniform level when speakers are in different
parts of the room. There's always a trade-off in applyinng level
"correction" in order to improve comprehension -- this might be more
necessary for the "unmixed" audio.

   My concern, actually, is more with how to "audibly" "place" the
received audio in a virtual room. I believe the "mixed" version needs
to be given a less-distinct "place" than the "unmixed". YMMV, of course.

   There's another issue, hopefully less abstruse: when working from
_both_ mixed and unmixed audio captures: the receiver needs to be
careful about mixing pre-mixed sources.

   The nature of sound is a very slow propagation rate in air: roughly
one foot per millisecond. The echo of blending sound from one speaker
with 100 millisecond propagation difference is obvious -- but the ear
can be confused by apparent acoustic effects with differences much
smaller than that. The human ear adjusts pretty well to different
acoustics if they're constant, but gets confused about apparent position
if these acoustic effects change unpredictably.

   This is probably not anything most of us want to worry about in our
initial release: I am thinking ahead, hoping to avoid need to kludge
something later on. Folks are already used to "surround-sound" home
theatres: we should have some idea how to use such sound systems when
we get a round-tuit. ;^)

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Thu, 29 Mar 2012 18:50:37 +0200

John, you have listed a set of valid acoustical challenges which I
recognize from our experience. Some of the acoustical challenges will be
present if a room use 1 or 2+ microphones to record, e.g. when a person
moves around in the room. We should thrust endpoints that chooses to use
multiple microphones for capture to do proper audio pre-processing, and
keep the audio quality. 

I would suggest that we reserve the mixed attribute to scenarios where a
middle box mix audio captures from different endpoints. 

Cheers 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
John Leslie
Sent: 29. mars 2012 17:26
To: Duckworth, Mark
Cc: [email-masked]
Subject: Re: [clue] audio mixed attribute

   Yes.

   My concern, actually, is more with how to "audibly" "place" the
received audio in a virtual room. I believe the "mixed" version needs to
be given a less-distinct "place" than the "unmixed". YMMV, of course.

   This is probably not anything most of us want to worry about in our
initial release: I am thinking ahead, hoping to avoid need to kludge
something later on. Folks are already used to "surround-sound" home
theatres: we should have some idea how to use such sound systems when we
get a round-tuit. ;^)

---

From: John Leslie <[email-masked]>
Date: Thu, 29 Mar 2012 13:18:43 -0400

Espen Berger (espeberg) <[email-masked]> wrote:
> moves around in the room.

   This is true.

> We should thrust endpoints that chooses to use multiple microphones for
> capture to do proper audio pre-processing, and keep the audio quality. 

   I do not intend anything other than trusting endpoint operators.
But there are inherent differences between single-pickup and multiple
pickups mixed. For many purposes, multiple pickups will be better...

> middle box mix audio captures from different endpoints. 

   I don't believe it helpful to define "mixed" based on where the mixing
is done.

   You raise a valid point about "mixing" audio from different rooms.
This, indeed, is something to "hardly-ever" prefer. Thus, to me, I'd
rather it not be an ordinary offer.

   When I describe "mixed" as a warning-label, I did not mean to infer
that "mixed" means "low-quality".

   It sounds to me as if we should seriously consider including a
description of _what_ is being mixed...

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 29 Mar 2012 21:54:05 +0200

I think you all have very clearly demonstrated that just being told that 
a stream is mixed or not-mixed is insufficient to make any useful 
decision about what to do with it.

	Thanks,
	Paul (as individual)

On 3/29/12 7:18 PM, John Leslie wrote:
> Espen Berger (espeberg)<[email-masked]>  wrote:
>>
> John Leslie<[email-masked]>