Subject: [clue] Improving treatment of audio

From: Paul Kyzivat <[email-masked]>
Date: Tue, 01 Apr 2014 13:11:45 -0400

In today's design team meeting we had a discussion of what is lacking 
about our treatment of audio, and how to fix it.

I want to open a ticket on this topic, but I need some help to properly 
describe the task. IMO it has to do with what sort of spatial 
information should be provided for audio captures, how it can be used to 
correlate audio captures with video captures, and how it can be used to 
choose which audio captures to configure.

Can somebody (John?) make a *concise* statement of what is needed?

	Thanks,
	Paul

---

From: John Leslie <[email-masked]>
Date: Thu, 10 Apr 2014 18:27:05 -0400

Paul Kyzivat <[email-masked]> wrote:
> Date: Tue, 01 Apr 2014 13:11:45 -0400
> From: Paul Kyzivat <[email-masked]>
> To: CLUE <[email-masked]>
> Subject: [clue] Improving treatment of audio
> 
> In today's design team meeting (01 April) we had a discussion of what is
> lacking about our treatment of audio, and how to fix it.
> 

   No.

   I don't believe we understand the problem space well enough to agree
on a _concise_ statement.

   I've been playinng around with an "Audio 101" presentation about
the problem space. I'm not particularly happy with it, but it's high
time I posted it and moved on.

   (Hopefully, it's light-hearted enough that reading it won't be too
painful...)

====

Audio 101

Q: What is sound?
A: Sound is periodic localized air-pressure differences in the
   audio-frequency range.

Q: What do you mean by "periodic"?
A: Periodic refers to some sort of repetition enabling reinforcement
   by combining with a time-delayed signal from the past.

Q: What is the audio-frequency range?
A: Human hearing is generally accepted to cover the range from
   20 to 20,000 Hertz (repetitions per second).

Q: How does sound propagate?
A: Nature abhors a vacuum. Air molecules move towards lower pressure.
   Sound can also travel in solid objects, such as a conference tables
   when the CEO is pounding his fist on it.

Q: What is a microphone?
A: It's a device with a membrane which responds to air-pressure
   differences between one side and the other, converting those
   differences to electrical signals.

Q: What is the output of a microphone?
A: Roughly speaking, it's a amplitude modulated voltage which tracks
   the air-pressure difference, within limits.

Q: What is an audio mixer?
A: It's a device that combines multiple input signals at adjustable
   levels into one or more output signals. Incoming signals may be
   muted, reduced, or amplified depending on the desired result.

Q: How well does that work, really?
A: Better than you'd expect! The human ear expects any original sound
   to be combined with "reverberation"

Q: What is "reverberation"?
A: Every room and even outdoor spaces have some level of reflection
   of sounds. A microphone or human ear can only hear the combination
   of original sound and reflections. The combination of reflections
   is called reverberation.

Q: Isn't that confusing?
A: Actually rather un-nerving to find yourself in a space designed
   to prevent any reverberation: with no reverberation your voice
   "disappears". Your ears have long since learned how to "extract"
   the original sound from the reverberation.

Q: With so many reflections, doesn't reverberation overwhelm the
   original sound?
A: Yes, but no real-world reflection is 100%, and in most cases the
   sound continues to dissipate with distance. The ear can make sense
   of it in "ordinary" rooms even when the original sound becomes
   inaudible.

Q: What's "dissipation with distance"?
A: In an object-free space, a localized air-pressure difference
   (sound) will spread in _every_ direction; thus there's less
   pressure difference as you get farther from the source. Thus,
   at twice the distance there's only one-quarter the pressure
   difference.

Q: Is that why we try to put microphones near the person speaking?
A: Yes: If the microphone is close enough, the person speaking
   dominates all other sound sources.

Q: What if there are more speakers than microphones?
A: We tend to put microphones within arm's reach and encourage
   speakers to grab the nearest one. (There is some equipment
   that can cope with multiple speakers at arms length: it depends
   on hardware and configuration.)

Q: Does that work well?
A: No.

Q: Why not?
A: First, because often they don't (grab). People assume if they can
   hear themselves, so can the other people.

Q: So, should we have a microphone in front of every speaker?
A: It would help, but it may not be close enough. A lapel mike isi
   perhaps 6 inches from the mouth. A tabletop may be 3 feet.

Q: How would you manage sound mixing for lapel mikes for every
   participant?
A: Usually, you can simply send each signal into a mixer, more or
   less equalize the levels, and leave it alone.

Q: How would you manage sound mixing for a cardioid mike in front
   of each person:
A: Poorly!

Q: Why can't it be the same as for lapel mikes?
A: If you feed an in-room speaker with the mix, you get feedback; if
   you feed it only to remote rooms, you get echo.

Q: So what do sound engineers do?
A: They "ride gain" by keeping most levels low and bringing up the
   person speaking.

Q: Can that be automated?
A: Easily.

Q: Why can't the same trick be used for one mike per three persons?
A: It can; but it will sound as if they're speaking from three different
   rooms. And, you'll probably get feedback.

Q: Aren't there devices to "eliminate" feedback?
A: Indeed there are: when the problem is mild they work pretty well;
   but if the problem is severe, instead of deafening feedback, it
   sounds like the mike is malfunctioning.

Q: What do good sound engineers do faced with that difficult situation?
A: Order another beer. Alcohol improves the sound.

Q: What is "feedback"?
A: When a microphone "hears" the signal it sent to the mixer returning
   from a loudspeaker in the room, the delay between the microphone
   picking up a sound and the sound returning from the loudspeaker
   will be just right to reinforce certain frequencies: if that
   reinforcing signal is not sufficiently damped, that "feedback"
   frequency will overwhelm normal speech.

Q: Are there ways to automatically damp such "feedback"?
A: Many, but that's outside the scope of this tutorial.

Q: How does "feedback" differ from "echo"?
A: Mainly it's the amount of delay. With loudspeakers being unable to
   reproduce sounds of less than 20 Hertz, the sound no longer gets
   loud enough to overwhelm, but the ear is confused by an acoustic
   very unlike any room it knows.

Q: So "echo cancellation" is needed?
A: Yes, although "echo avoidance" is better. Echo can be avoided if
   everyone wears earbuds, for example.

Q: Is that why telephones work so well?
A: Basically yes. However, telephones fake a slight echo by feeding
   back (electrically) a signal from the mouthpiece to the earpiece
   so that the person can "believe" the other end of the call is
   hearing him/her.

Q: Can we do that trick in CLUE?
A: To some degree, yes. It works quite well for headsets, but only if
   the actual echo from the room(s) at the other end of the call can
   be kept quiet enough.

Q: What about room to room, with no headsets?
A: Almost always it's better to depend on the room acoustics; but
   the trick can still be used to simulate echo from the other room(s)
   at an "appropriate" delay (coming from the loudspeakers).

Q: How is echo cancellation done?
A: It's a black art! Fundamentally the equipment guesses at the actual
   echo delay and "subtracts" a predicted echo from the actual
   returned echo. The details go way beyond this tutorial.

Q: Is that a worse problem for CLUE than for telephony?
A: Much worse! The actual delay changes unpredictably, especially if
   anything about the audio is "switched".

Q: What do you recommend that CLUE does about it?
A: To fix it, we'd need to synchronize time at both rooms and find
   the loudspeaker-to-microphone delays. That's too hard right now.
   To some degree, the problem can be hidden with artificial
   reverberation.

Q: What is artificial reverberation?
A: That's generating an artificial signal to simulate a larger room
   than what you're in, usually by applying an "impulse response"
   to the original signal.

Q: What's an "impulse response"?
A: Conceptually, it's what you would hear in a real room where the
   only sound input is an "infinitely short" step function. In practice
   it's often done by recording a (blank) pistol shot.

Q: Will that magic actually fool the human ear?
A: Close enough. It will sound "almost natural" and the brain will
   suspend its disbelief. :^)

====

   If I succeeded in outlining the scope, perhaps somebody else will take
a stab at the wording Paul wants...

--
John Leslie <[email-masked]>

---

From: Paul Coverdale <[email-masked]>
Date: Thu, 10 Apr 2014 19:17:38 -0400

I don't disagree with John's "Audio 101" (well, most of it anyway). It takes
me back to my old days in the Audio & Acoustics group at BNR/Nortel when we
were trying to figure out what to do about acoustic echo and
de-reverberation. But those problems have pretty well been solved for audio
conferencing systems. So it's not clear to me what extra we need to do from
a CLUE perspective. What is the problem?

...Paul

>-----Original Message-----
>From: clue [mailto:[email-masked]] On Behalf Of John Leslie
>Sent: Thursday, April 10, 2014 6:27 PM
>To: Paul Kyzivat
>Cc: CLUE
>Subject: Re: [clue] Improving treatment of audio
>
>> In today's design team meeting (01 April) we had a discussion of what
>> is lacking about our treatment of audio, and how to fix it.
>>
>> I want to open a ticket on this topic, but I need some help to
>> properly describe the task. IMO it has to do with what sort of spatial
>> information should be provided for audio captures, how it can be used
>> to correlate audio captures with video captures, and how it can be
>> used to choose which audio captures to configure.
>>
>   I've been playinng around with an "Audio 101" presentation about the
>problem space. I'm not particularly happy with it, but it's high time I
>posted it and moved on.
>
>   If I succeeded in outlining the scope, perhaps somebody else will
>take a stab at the wording Paul wants...
>
>--
>_______________________________________________
>clue mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/clue

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 10 Apr 2014 22:26:22 -0400

John,

While interesting, I can definitely say that what you gave is *not* what 
I was looking for. :-)

The concise statement I am looking for is not a description of the 
solution we are seeking. I am only (right now) looking for a description 
of the *issue* that needs to be solved.

E.g. "Define what sort of spatial information should be provided with 
the description of audio captures in order to support consumer selection 
of captures and proper playback."

*Then* the next step will be to actually do it, in the framework and the 
data model. So the issue should be phrased in a way that is achievable.

	Thanks,
	Paul

---

From: Christian Groves <[email-masked]>
Date: Fri, 11 Apr 2014 12:37:38 +1000

"So it's not clear to me what extra we need to do from a CLUE 
perspective. What is the problem?"
I guess this is the pertinent point.

 From John's Audio 101 it seems that anything to do with spatial 
information that would relate to reverberation is in the too hard 
basket. So it seems "area of capture" for audio could be marked "not 
applicable" in the framework. There doesn't seem to be any driver for 
having the "point of capture" apply to an audio capture either.

 From the Audio101 there does seem to be a dependency on where the 
microphone is located with respect to the person speaking (i.e. lapel 
mic, desk mic, room mic) to how it is handled at mixing/playout. Perhaps 
this is useful to signal via CLUE? If it is possible to signal this them 
perhaps tying a particular audio capture to a video capture makes sense? 
i.e. a talker giving a presentation using a lapel mic captured by a 
particular video.

Regards, Christian

---

From: Paul Coverdale <[email-masked]>
Date: Fri, 11 Apr 2014 09:52:34 -0400

>> It takes me back to my old days in the Audio & Acoustics group at
>> BNR/Nortel when we were trying to figure out what to do about acoustic
>> echo and de-reverberation. But those problems have pretty well been
>> solved for audio conferencing systems.
>
>   They have indeed been "solved" for POTS. I am less convinced they
>have been solved for audio-conferencing systems -- though I'd be happy
>to be convinced...

Well, I think that there are some quite sophisticated acoustic echo
cancellers and even de-reverberation algorithms on the market nowadays, so I
believe that the potential is certainly there to give a high-quality audio
experience within Telepresence. Having said that, the implementers of these
systems need to have some skill in setting them up to work correctly.
However, I think this goes beyond the scope of CLUE.

>   But IMHO, prior solutions can't simply be tacked on -- we need to
>recognize where these problems come from; then we need to evaluate what
>new difficulties arise in our immersion-teleconferencing setting.
>
>   I don't mean to _solve_ these problems here; but I do want to make
>sufficient tools available to the folks that will have to solve them:
>preferably early in CLUE's rollout, but at worst "eventually".
>
>   Recall that we will have to deal with wildly varying delays and
>jitter, and that the human ear's expectations become more discriminating
>when trying to match the imagined room. Et cetera...
>
Don't disagree, but it's still not clear to me what specifically needs to be
done in a CLUE context. 

>> So it's not clear to me what extra we need to do from a CLUE
>> perspective.
>
>   That's the question, isn't it. I expect we will need to better inform
>echo-cancellation about the varying delays, for one thing.
>
Well, it's not going to be easy to figure out the end to end delay, since it
will depend on the terminal equipment and the network transport. But even if
we had such a figure, I'm not sure what the acoustic echo-canceller would do
with it. Basically, the canceller plus non-linear processor tries to reduce
the level of any echo to below the level of audibility, regardless of the
end to end delay. The only delay it needs to worry about is in the
tail-path, which is at the local end anyway.

---

From: Paul Coverdale <[email-masked]>
Date: Fri, 11 Apr 2014 10:24:51 -0400

>From: clue [mailto:[email-masked]] On Behalf Of Christian Groves
>Sent: Thursday, April 10, 2014 10:38 PM
>To: [email-masked]

Yes, I think we will be opening a can of worms if we try to get too fancy
about signalling spatial audio information. However, I think that as a
minimum we need to signal the position of each microphone, its pattern (and
axis of directivity if directional), and its relation to the audio capture
and video capture.

...Paul

---

From: John Leslie <[email-masked]>
Date: Fri, 11 Apr 2014 11:00:55 -0400

Christian Groves <[email-masked]> wrote:
> basket.

   What precisely does "too hard basket" mean?

   I haven't even reached the point of suggesting what metrics to
define the _ability_ to send. To me, "too hard" merely means that
an individual site could choose not to send them or to ignore them
on receipt.

   But it sounds as if you're suggesting reverberation is "to hard
to understand" and thus we should have no metrics about it.

   I _hope_ that's not what you mean.

> So it seems "area of capture" for audio could be marked "not 
> applicable" in the framework.

   I hope so.

> There doesn't seem to be any driver for having the "point of capture"
> apply to an audio capture either.

   I don't understand this. Point of capture for a microphone may be
"too hard" to track (today) for a microphone which moves; nonetheless
it seems to me the most fundamental metric there could be.

> mic, desk mic, room mic) to how it is handled at mixing/playout.

   I don't think I really got that far...

   IMHO, it's more flexible to have mixing be the responsibily of
the receiver, but I haven't tried to specify that and I'm not at all
sure I want to specify that.

   I expect the actual sound systems in different rooms to vary wildly,
from one monaural speaker to stereo to full surround-sound. Mixing
for these without knowing which is the actual target seems hard; but
I'm sure there will be sites which prefer to do so. A question which
will arise, IMHO, is how to specify the _intent_ of a mix generated
in one room to be fed to other rooms. (I'd prefer not to go there yet.)

> Perhaps this is useful to signal via CLUE? If it is possible to signal
> this then perhaps tying a particular audio capture to a video capture
> makes sense?

   I'm not thinking along those lines. (That doesn't mean we shouldn't
think along those lines...) I'm thinking in terms of providing several
audio streams per room, associated with position information about
the position of the source of those sounds, and allowing the receiver
to choose how to mix them and how to present the mix in his/her room.

> i.e. a talker giving a presentation using a lapel mic captured by a 

   In fact, there's only limited tendency for humans to strictly
attach the sound they hear to the video they see. Clearly, during a
presentation we want to _hear_ the presenter talking, but having the
sound move back and forth as the speaker walks can be confusing.

   At the same time, we will want to hear the questions to which the
presenter may respond. It will be far easier on the listener if these
do _not_ seem to be coming from the same physical position, especially
when the questioner _is_ in the same room as the presenter.

   Hope this helps...

---

From: John Leslie <[email-masked]>
Date: Fri, 11 Apr 2014 11:14:05 -0400

Paul Coverdale <[email-masked]> wrote:
> 
> Yes, I think we will be opening a can of worms if we try to get too
> fancy about signalling spatial audio information.

   There are worm-cans strewn about, I agree...

   But, even if we tend to be minimalist for the initial spec, I believe
we will need to be able to expand the limits of what we _can_ specify.

> However, I think that as a minimum we need to signal the position of
> each microphone,

   Yes, but there will be microphones which move (especially lapel
microphones of presenters). There is a real question how to specify
such motion (which I'm not in a hurry to get to yet).

> its pattern (and axis of directivity if directional),

   Pattern, to tell truth, always is related to axis of capture for
physically-achievable microphones. (I suspect, however, that sometimes
the patter will simply be "unknown". This isn't necessarily fatal.)

> and its relation to the audio capture and video capture.

   I wouldn't know how to specify "relation to the audio capture"
(or even what a "the audio capture" might be). "Relation to the video
capture" seems not worth the trouble, at first blush...

   When a sender _chooses_ to send a mix rather than individual audio
sources, I haven't given a lot of thought to how to describe the mix.
My initial guess would be to reference the sources (which hopefully
have a physical position) and the relative gain settings.

   To me, a mix doesn't really _have_ a physical position.

   Hope this helps...

---

From: Paul Coverdale <[email-masked]>
Date: Fri, 11 Apr 2014 13:56:25 -0400

>From: John Leslie [mailto:[email-masked]]
>Sent: Friday, April 11, 2014 11:14 AM
>To: Paul Coverdale
>Cc: [email-masked]
>microphones of presenters). There is a real question how to specify such
>motion (which I'm not in a hurry to get to yet).

Yes, it would be difficult to track the motion, but I don't think we need
to. The lapel microphone is likely to be the best option for picking up
audio, giving (hopefully) a good signal to noise ratio regardless of where
the user is. All we need to do is associate it with the appropriate video
capture.

>the patter will simply be "unknown". This isn't necessarily fatal.)

Maybe I didn't get the terminology right, but I was thinking of describing
the microphone position in terms of Point of Capture (7.1.1.1 in Framework).
By pattern, I meant the inherent microphone pattern (in free-space), eg
omni, cardioid. In the case of a directional microphone, the direction of
the major lobe could be described using Point of Capture and Point on Line
of Capture (7.1.1.2). I agree that Area of Capture has little meaning for
audio. As you say, in some cases the pattern may be unknown. In this case,
the default would be omni.

>capture" seems not worth the trouble, at first blush...

Again, maybe I didn't use the right terminology. What I meant was that a
given microphone needs to be associated with a particular audio capture, and
this audio capture is normally associated with a specific video capture eg,
you hear what the guy in the picture is saying. But I suppose there could be
situations where the audio capture has no associated video eg, the guy
doesn't want anyone to see the wart on his nose. And there may be situations
where the same audio capture is associated with more than one video capture,
ie several subjects are sharing the same microphone.

>My initial guess would be to reference the sources (which hopefully have
>a physical position) and the relative gain settings.
>

Agreed, in the case of a mix from more than microphone, just reference the
individual microphone locations and the relative gain settings.

>   Hope this helps...

Yes, it does.

---

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 11 Apr 2014 12:23:55 -0700

Hi all,
The original intent of using "area of capture" for all media types, including audio and video, was to provide a way to associate audio with video.  So the consumer can choose captures that go together and render them together.  An example of this is in the framework document, section 12.1.1.  I still think this works fine for this purpose.

In that example, the consumer can choose VC0, VC1, VC2, AC0, AC1, and AC2.  AC0 and VC0 have the same area of capture, indicating they are capturing the same area of the scene.  If the consumer wants to render AC0 from a loudspeaker close to the display for VC0 it can do so.  Similarly, AC3 has an area of capture covering the whole scene (the full extent of the areas of VC0, VC1, VC2) so the consumer knows AC3 includes audio associated with all of VC0, VC1, and VC2.

So I'm puzzled why you are proposing we remove the ability to use area of capture for audio for this purpose.

Regards,
Mark

> Sent: Friday, April 11, 2014 11:01 AM
> To: Christian Groves
> Cc: [email-masked]
>    I haven't even reached the point of suggesting what metrics to define the
> _ability_ to send. To me, "too hard" merely means that an individual site
> could choose not to send them or to ignore them on receipt.
> 
>    But it sounds as if you're suggesting reverberation is "to hard to
> understand" and thus we should have no metrics about it.
> 
>    I don't understand this. Point of capture for a microphone may be "too
> hard" to track (today) for a microphone which moves; nonetheless it seems
> to me the most fundamental metric there could be.
> 
>    IMHO, it's more flexible to have mixing be the responsibily of the receiver,
> but I haven't tried to specify that and I'm not at all sure I want to specify that.
> 
>    I expect the actual sound systems in different rooms to vary wildly, from
> one monaural speaker to stereo to full surround-sound. Mixing for these
> without knowing which is the actual target seems hard; but I'm sure there
> will be sites which prefer to do so. A question which will arise, IMHO, is how
> to specify the _intent_ of a mix generated in one room to be fed to other
> rooms. (I'd prefer not to go there yet.)
> 
>    I'm not thinking along those lines. (That doesn't mean we shouldn't think
> along those lines...) I'm thinking in terms of providing several audio streams
> per room, associated with position information about the position of the
> source of those sounds, and allowing the receiver to choose how to mix
> them and how to present the mix in his/her room.
> 
>    In fact, there's only limited tendency for humans to strictly attach the sound
> they hear to the video they see. Clearly, during a presentation we want to
> _hear_ the presenter talking, but having the sound move back and forth as
> the speaker walks can be confusing.
> 
> presenter may respond. It will be far easier on the listener if these do _not_
> seem to be coming from the same physical position, especially when the
> questioner _is_ in the same room as the presenter.
> 
>    Hope this helps...
> 
> --

---

From: Paul Coverdale <[email-masked]>
Date: Fri, 11 Apr 2014 16:48:27 -0400

Hi Mark,

I can see what you're trying to do in the example given in Framework section
12.1.1. The problem I have is that, because of the huge difference in
wavelength between light waves and audio waves, you can't really define an
area of capture for audio in the same way that you can for video. A camera
can focus on a scene defined by 4 co-planar X,Y,Z coordinates. It will
capture the video inside that quadrilateral, and nothing outside. A
microphone can capture audio coming from inside of the same quadrilateral,
but it will also capture a lot of audio from outside. What this means is
that we can never define a unique association of an audio capture with a
video capture based on pure geometrical considerations, but we can still
simply define that ACx is associated with VCx, but maybe with VCy and VCz
too.

...Paul

>From: clue [mailto:[email-masked]] On Behalf Of Duckworth, Mark
>Sent: Friday, April 11, 2014 3:24 PM
>To: John Leslie; Christian Groves
>Cc: [email-masked]
>The original intent of using "area of capture" for all media types,
>including audio and video, was to provide a way to associate audio with
>video.  So the consumer can choose captures that go together and render
>them together.  An example of this is in the framework document, section
>12.1.1.  I still think this works fine for this purpose.
>
>In that example, the consumer can choose VC0, VC1, VC2, AC0, AC1, and
>AC2.  AC0 and VC0 have the same area of capture, indicating they are
>capturing the same area of the scene.  If the consumer wants to render
>AC0 from a loudspeaker close to the display for VC0 it can do so.
>Similarly, AC3 has an area of capture covering the whole scene (the full
>extent of the areas of VC0, VC1, VC2) so the consumer knows AC3 includes
>audio associated with all of VC0, VC1, and VC2.
>
>> define the _ability_ to send. To me, "too hard" merely means that an
>> individual site could choose not to send them or to ignore them on
>receipt.
>>
>> > microphone is located with respect to the person speaking (i.e.
>> > lapel mic, desk mic, room mic) to how it is handled at
>mixing/playout.
>>
>>    IMHO, it's more flexible to have mixing be the responsibily of the
>> receiver, but I haven't tried to specify that and I'm not at all sure
>I want to specify that.
>>
>>    I expect the actual sound systems in different rooms to vary
>> wildly, from one monaural speaker to stereo to full surround-sound.
>> Mixing for these without knowing which is the actual target seems
>> hard; but I'm sure there will be sites which prefer to do so. A
>> question which will arise, IMHO, is how to specify the _intent_ of a
>> mix generated in one room to be fed to other rooms. (I'd prefer not to
>> go there yet.)
>>
>> > Perhaps this is useful to signal via CLUE? If it is possible to
>> > signal this then perhaps tying a particular audio capture to a video
>> > capture makes sense?
>>
>> audio streams per room, associated with position information about the
>> position of the source of those sounds, and allowing the receiver to
>> choose how to mix them and how to present the mix in his/her room.
>>

---

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 11 Apr 2014 14:42:04 -0700

Hi Paul,
Yes, I understand the area of capture for audio can't be nearly as precise as it can be for video.  But for this usage I think it doesn't matter.
Mark

> From: Paul Coverdale [mailto:[email-masked]]
> Sent: Friday, April 11, 2014 4:48 PM
> To: Duckworth, Mark; 'John Leslie'; 'Christian Groves'
> Cc: [email-masked]
> Subject: RE: [clue] Improving treatment of audio
> 
> Hi Mark,
> 
> can focus on a scene defined by 4 co-planar X,Y,Z coordinates. It will capture
> the video inside that quadrilateral, and nothing outside. A microphone can
> capture audio coming from inside of the same quadrilateral, but it will also
> capture a lot of audio from outside. What this means is that we can never
> define a unique association of an audio capture with a video capture based
> on pure geometrical considerations, but we can still simply define that ACx is
> associated with VCx, but maybe with VCy and VCz too.
> 
> ...Paul
> 
> >them together.  An example of this is in the framework document,
> >section 12.1.1.  I still think this works fine for this purpose.
> >
> >Similarly, AC3 has an area of capture covering the whole scene (the
> >full extent of the areas of VC0, VC1, VC2) so the consumer knows AC3
> >includes audio associated with all of VC0, VC1, and VC2.
> >
> >> mix generated in one room to be fed to other rooms. (I'd prefer not
> >> to go there yet.)
> >>
> >> > signal this then perhaps tying a particular audio capture to a
> >> > video capture makes sense?
> >>
> >>    I'm not thinking along those lines. (That doesn't mean we
> >> shouldn't think along those lines...) I'm thinking in terms of
> >> providing several audio streams per room, associated with position
> >> information about the position of the source of those sounds, and
> >> allowing the receiver to choose how to mix them and how to present the
> mix in his/her room.
> >>
> >> do _not_ seem to be coming from the same physical position,
> >> especially when the questioner _is_ in the same room as the presenter.
> >>
> >>    Hope this helps...
> >>
> >> --

---

From: Paul Coverdale <[email-masked]>
Date: Fri, 11 Apr 2014 20:12:54 -0400

Hi Mark,

Maybe I mis-understood, but I had the impression that the capture area would
provide the association between audio and video captures, without any other
explicit assignment. In other words, if capture areas for two different
media define exactly the same space, they must be associated with each
other. This is the part that I have some concerns about, for the reasons
already given for audio capture area. But I have no objection to defining an
association on the basis of the video capture area and an audio capture that
most closely intersects that area.

...Paul

>From: Duckworth, Mark [mailto:[email-masked]]
>Sent: Friday, April 11, 2014 5:42 PM
>To: Paul Coverdale
>Cc: [email-masked]
>Yes, I understand the area of capture for audio can't be nearly as
>precise as it can be for video.  But for this usage I think it doesn't
>matter.
>Mark
>
>> I can see what you're trying to do in the example given in Framework
>> section 12.1.1. The problem I have is that, because of the huge
>> difference in wavelength between light waves and audio waves, you
>> can't really define an area of capture for audio in the same way that
>> you can for video. A camera can focus on a scene defined by 4
>> co-planar X,Y,Z coordinates. It will capture the video inside that
>> quadrilateral, and nothing outside. A microphone can capture audio
>> coming from inside of the same quadrilateral, but it will also capture
>> a lot of audio from outside. What this means is that we can never
>> define a unique association of an audio capture with a video capture
>> based on pure geometrical considerations, but we can still simply
>define that ACx is associated with VCx, but maybe with VCy and VCz too.
>>
>> ...Paul
>>
>> >including audio and video, was to provide a way to associate audio
>> >with video.  So the consumer can choose captures that go together and
>> >render them together.  An example of this is in the framework
>> >document, section 12.1.1.  I still think this works fine for this
>purpose.
>> >
>> >So I'm puzzled why you are proposing we remove the ability to use
>> >area of capture for audio for this purpose.
>> >
>> >Regards,
>> >Mark
>> >
>> >> on
>> >>    I don't understand this. Point of capture for a microphone may
>> >> be "too hard" to track (today) for a microphone which moves;
>> >> nonetheless it seems to me the most fundamental metric there could
>be.
>> >>
>> >> the receiver, but I haven't tried to specify that and I'm not at
>> >> all sure
>> >I want to specify that.
>> >>
>> >> question which will arise, IMHO, is how to specify the _intent_ of
>> >> a mix generated in one room to be fed to other rooms. (I'd prefer
>> >> not to go there yet.)
>> >>
>> >> > i.e. a talker giving a presentation using a lapel mic captured by
>> >> > a particular video.
>> >>
>> >> presentation we want to _hear_ the presenter talking, but having
>> >> the sound move back and forth as the speaker walks can be
>confusing.
>> >>
>> >>    At the same time, we will want to hear the questions to which
>> >> the presenter may respond. It will be far easier on the listener if
>> >> these do _not_ seem to be coming from the same physical position,

---

From: Paul Coverdale <[email-masked]>
Date: Sun, 13 Apr 2014 15:59:10 -0400

So, going back to Paul K's request for a concise statement of what needs to
be done with respect to the treatment of audio in CLUE, I'm not sure what
else is really required apart from satisfying REQMT-2 and REQMT-3, with the
possible exception of including the directional characteristics of the
microphone with the audio capture.

...Paul

>From: clue [mailto:[email-masked]] On Behalf Of Paul Kyzivat
>Sent: Tuesday, April 01, 2014 1:12 PM
>To: CLUE

---

From: John Leslie <[email-masked]>
Date: Sun, 13 Apr 2014 19:31:54 -0400

Paul Coverdale <[email-masked]> wrote:
> else is really required apart from satisfying REQMT-2 and REQMT-3,

   I have to agree with Paul-C here: clue-telepresence-requirements is
pretty much set in stone, now that it has entered the RFCed queue. But
I don't think it fits Paul-K's criteria of "concise:
] 
] REQMT-2: The solution MUST support a description of the spatial
]          arrangement of captured source audio sent in audio streams
]          which enables a satisfactory reproduction at the receiver
]          in a spatially correct manner.  This applies to each site
]          in a point to point or a multipoint meeting and refers to
]          the spatial ordering within a site, not the ordering of
]          channels between sites.
]
]          Use case point to point symmetric, and all use cases,
]          especially heterogeneous.
]
]  REQMT-2a: The solution MUST support a means of preserving
]            the spatial order of audio in the captured
]            scene.  For example, if John sounds as if he is
]            at Susan's right in the captured audio, John
]            voice is also placed at Susan's right in the
]            rendered image.
]
]  REQMT-2b: The solution MUST support a means to identify
]            the number and spatial arrangement of audio
]            channels including monaural, stereophonic
]            (2.0), and 3.0 (left, center, right) audio
]            channels.
]
]  REQMT-2c: The solution MUST support a means to identify
]            the point of capture of individual audio
]            captures in three dimensions.
]
]  REQMT-2d: The solution MUST support a means to identify
]            the area of coverage of individual audio
]            captures in three dimensions.
]
] REQMT-3: The solution MUST enable individual audio streams to be
]          associated with one or more video image captures, and
]          individual video image captures to be associated with one
]          or more audio captures, for the purpose of rendering
]          proper position.
]
]          Use case is point to point symmetric, and all use cases.

   Worse, I'm not confident that all of this is even attainable. :^(

   Mea culpa, of course -- I should have said that in WGLC. :^( :^(

> with the possible exception of including the directional
> characteristics of the microphone with the audio capture.

   While I'd be happy to see that signaled, I'm really not pushing it
for our first spec -- merely hoping we'll get there someday.

   There is, alas, an elephant in each room which can render our
audio treatment useless -- it's called "echo". :^(

====
   Despite being too late, I feel I ought to comment on these REQMTs.
Feel free to ignore what follows:

] REQMT-2: The solution MUST support a description of the spatial
]          arrangement of captured source audio sent in audio streams
]          which enables a satisfactory reproduction at the receiver
]          in a spatially correct manner.

   Other than wondering what "spatially correct means, I view this as
attainable, since the responsibily must belong to the receiver, and
that only if the sender chooses to specify enough audio sources.

]  REQMT-2a: The solution MUST support a means of preserving
]            the spatial order of audio in the captured
]            scene.  For example, if John sounds as if he is
]            at Susan's right in the captured audio, John
]            voice is also placed at Susan's right in the
]            rendered image.

   Likewise, this responsibility must belong to the receiver.

]  REQMT-2b: The solution MUST support a means to identify
]            the number and spatial arrangement of audio
]            channels including monaural, stereophonic
]            (2.0), and 3.0 (left, center, right) audio
]            channels.

   This, IMHO, isn't worth doing (although perhaps I don't understand
what REQMT-2b means).

   The actual layout of loudspeakers should belong entirely to the
receiving room. Further, "stereo" doesn't actually define the relationship
of the two channels; and "3.0" suffers a similar problem.

   If a sender _actually_ sends "stereo" or "3.0" the receiver will have
to punt. I can only hope that our actual standard doesn't encourage this.

]  REQMT-2c: The solution MUST support a means to identify
]            the point of capture of individual audio
]            captures in three dimensions.

   Well stated. Adding point on line of capture helps slightly, but only
if we have a useful approximation of sensitivity pattern (which I don't
believe we can expect to get).

]  REQMT-2d: The solution MUST support a means to identify
]            the area of coverage of individual audio
]            captures in three dimensions.

   Poorly stated. :^( _Every_ microphone in a room can cover _every_
source of sound in a room. Thus, the "area of coverage" must necessarily
be the entire room...

   (It gets worse... much worse!)

   Inevitably, some yahoo is going to put one or more loudspeakers in
the room. And those will expand the "area of coverage" to include the
other rooms those loudspeakers attempt to render.

   With delay!!!

] REQMT-3: The solution MUST enable individual audio streams to be
]          associated with one or more video image captures, and
]          individual video image captures to be associated with one
]          or more audio captures, for the purpose of rendering
]          proper position.
]
]          Use case is point to point symmetric, and all use cases.

   I accept there are many folks who will insist on doing this. I don't
expect to stop them.

   But audio streams are intrinsically linked to rooms, not video captures.
And the most useful audio streams are linked to microphones "close" to
individual people, while video captures will _very_ often try to cover
more than one person -- thus video capures will very typically be associated
with several audio captures.

====

   It will help if more of us pay attention to the relationship of audio
in television in movies. I find that they _don't_ track each other too
closely.

   The audio provides the continuity; the video tantalizes the senses.
When the camera switches between two people talking to each other, the
audio doesn't switch -- it tries to make you believe you're in the room.

   Thus I most sincerely hope that folks won't choose to switch their
audio based on who's speaking -- but I won't try to stop them from doing
this. (And I must admit that sometimes I _shouldn't_ blame them, because
they find themselves sabotaged by audio sources contaminated by echo.)

====

   A word on "echo suppression" might help here: this is a well-solved
in plain-old-telephony where end-to-end delay can be held constand. We
won't be able to do that over the Internet.

   An individual room is probably able to cancel the echo in what it
_sends_ but a receiving room won't be able to cancel echo in what it
receives. :^(

====

   That's as much as I'm willing to sqeeze into this email.

   Hope _something_ in this helps...

---

From: Christian Groves <[email-masked]>
Date: Mon, 14 Apr 2014 11:49:28 +1000

Hello John,

Thanks for the reply. Please see my responses below.

Regards, Christian

>     What precisely does "too hard basket" mean?
[CNG] Well what did you mean by "That's too hard right now" in your 
audio 101? :-) The tone of the 101 (and previous emails) seemed to imply 
that using the current spatial information is not appropriate and using 
other sorts of physical information is going to be hard.
> an individual site could choose not to send them or to ignore them
[CNG] I didn't understand your usage of "too hard" to imply optionality 
in sending or receipt.
> to understand" and thus we should have no metrics about it.
[CNG] I'm not saying its too hard to understand, more questioning about 
the amount of effort that would be required at this point in time of the 
development of CLUE.
> it seems to me the most fundamental metric there could be.
[CNG] I didn't understand the 101 wanting the current "point of capture" 
metric as a physical point. I agree it is a fundamental metric. However 
rather than a physical point in space is it enough to say the microphone 
is associated with a certain video capture or a location (e.g. lapel 
mic.)? I guess this comes down to what people actually want to do with it.
>     I don't think I really got that far...
[CNG] You did mention different possibilities of handling sound from 
different microphone types and where they are placed. That's what I was 
picking up on.
> to choose how to mix them and how to present the mix in his/her room.
[CNG] I think we're thinking in a similar vain, the issue is how to 
specify the "position". Whether its a physical point or more 
representional, i.e. lapel mic.

---

From: Christian Groves <[email-masked]>
Date: Mon, 14 Apr 2014 12:08:03 +1000

Hello all,

Perhaps the confusion is that some see Audio Capture area as a means to 
associate an Audio capture with a video capture as you've shown Mark's 
example. e.g. The audio spatial information isn't really used for any 
audio transformation other than associating it with a particular video 
stream.

Whereas others were more thinking the audio spatial information as an 
input to mixing and more complicated audio processing.

If this is the case perhaps rather than linking ACs and VCs through the 
physical or virtual co-ordinates of the area of capture information, we 
simplify things and in each audio capture we say what VC it relates to? 
e.g.using Mark's example AC0(VC0), AC3(V0,V1,V2)

Regards, Christian

---

From: "Duckworth, Mark" <[email-masked]>
Date: Sun, 13 Apr 2014 19:26:18 -0700

Hello Christian,
please see below.
Mark

> Sent: Sunday, April 13, 2014 10:08 PM
> To: [email-masked]
> example. e.g. The audio spatial information isn't really used for any audio
> transformation other than associating it with a particular video stream.
> 
> Whereas others were more thinking the audio spatial information as an input
> to mixing and more complicated audio processing.

[Duckworth, Mark] You could be right about this being a source of confusion.

> e.g.using Mark's example AC0(VC0), AC3(V0,V1,V2)

[Duckworth, Mark] I'm not sure how this would really work.  Because even in this simple example, we also have AC0 relates to VC3 (sometimes), VC4, and VC5 (at least part of it), and so on.  AC3 relates to all of VC1 through VC5.  I think using area of capture works better than trying to do something like this.

Regards,
Mark

> > Yes, I understand the area of capture for audio can't be nearly as precise as
> it can be for video.  But for this usage I think it doesn't matter.
> > Mark
> >
> >> coming from inside of the same quadrilateral, but it will also
> >> capture a lot of audio from outside. What this means is that we can
> >> never define a unique association of an audio capture with a video
> >> capture based on pure geometrical considerations, but we can still simply
> >>> with video.  So the consumer can choose captures that go together
> >>> and render them together.  An example of this is in the framework
> >>> document, section 12.1.1.  I still think this works fine for this purpose.
> >>>
> >>> In that example, the consumer can choose VC0, VC1, VC2, AC0, AC1,
> >>> and AC2.  AC0 and VC0 have the same area of capture, indicating they
> >>> are capturing the same area of the scene.  If the consumer wants to
> >>> render
> >>>> the sound move back and forth as the speaker walks can be confusing.
> >>>>

---

From: Christian Groves <[email-masked]>
Date: Mon, 14 Apr 2014 13:18:03 +1000

Hello Mark,

Sorry I didn't consider the entire 12.1.1. if I do that, according to 
that example:

Video areas of capture:

        bottom left    bottom right  top left         top right
    VC0 (-2011,2850,0) (-673,3000,0) (-2011,2850,757) (-673,3000,757)
    VC1 ( -673,3000,0) ( 673,3000,0) ( -673,3000,757) ( 673,3000,757)
    VC2 (  673,3000,0) (2011,2850,0) (  673,3000,757) (2011,3000,757)
    VC3 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
    VC4 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
    VC5 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
    VC6 none

Areas of capture for audio (from 12.1.1):

        bottom left    bottom right  top left         top right

    AC0 (-2011,2850,0) (-673,3000,0) (-2011,2850,757) (-673,3000,757)
    AC1 (  673,3000,0) (2011,2850,0) (  673,3000,757) (2011,3000,757)
    AC2 ( -673,3000,0) ( 673,3000,0) ( -673,3000,757) ( 673,3000,757)
    AC3 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
    AC4 none

Using a reference rather than area of capture:
     AC0 (VC0)
     AC1 (VC2)
     AC2 (VC1)
     AC3 (VC3,VC4,VC5)
I would think they are basically conveying the same information???

Regards, Christian

---

From: John Leslie <[email-masked]>
Date: Mon, 14 Apr 2014 11:05:49 -0400

Duckworth, Mark <[email-masked]> wrote:
> I'm not sure how this would really work.  Because even in this simple
> example, we also have AC0 relates to VC3 (sometimes), VC4, and VC5
> (at least part of it), and so on. AC3 relates to all of VC1 through VC5.

   Agreed.

> I think using area of capture works better than trying to do something
> like this.

   I need to admit I'm pretty clueless what Mark means here. :^(

   Area-of-capture _is_ four points in 3-space, hopefully co-planar.
I find the concept of relating area-of-capture for a video source to
area-of-capture for an audio source (where the two area-of-captures
will not be coplanar) _extremely_ confusing -- and I really don't
understand how future operators of telepresence systems are going to
find it less confusing.

   (Sorry, Mark...)

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 14 Apr 2014 11:46:04 -0400

>     Worse, I'm not confident that all of this is even attainable. :^(

I don't know if they are unattainable. But IIUC they cannot be attained 
with the information we currently have in the fw and data model.

If we want to give up on attaining some of them, then let's do so, and 
explicitly note that we have. And then ensure that we have sufficient 
information to attain the rest.

> audio treatment useless -- it's called "echo". :^(

IMO, if we don't have a solution that can prevent this then we don't 
have a solution worth publishing.

> Feel free to ignore what follows:

As I noted above, I think it is important to confront these issues.

> that only if the sender chooses to specify enough audio sources.

What do you mean by "attainable"? Can this be achieved using the data 
model we have now? (I don't think so.)

ISTM that the responsibility is *shared*:
- the advertiser must provide sufficient sources, *and* sufficient
   description of those sources to permit the receiver to do the right
   thing.

- the receiver must utilize the available information in order to
   reproduce the audio in a spatially correct manner, within the limits
   of its available equipment.

- *we* are responsible for providing sufficient expressiveness in
   the advertisement and RTP so that the advertiser and receiver can
   do the above. *That* is the part we need to work on now.

> to punt. I can only hope that our actual standard doesn't encourage this.

Sounds like something we need to talk about. Where?

> other rooms those loudspeakers attempt to render.

OK. But surely the point of capture isn't sufficient by itself. 
*Something* more is required. Can you suggest something that is stated 
better, that would be attainable and useful for our purposes?

> with several audio captures.

Are you saying that within a scene the choice of audio captures is 
unrelated to the choice of video captures? (Other than if you configure 
*some* video then you probably also want to configure *some* audio.)

If so, then does it ever make sense to choose a subset of the advertised 
audio? And if that is true, then what criteria make sense?

> they find themselves sabotaged by audio sources contaminated by echo.)

I assume you mean when the speakers are in the same room, right?

Isn't the situation different when we are switching among speakers in 
different rooms?

>     Hope _something_ in this helps...

Some! This is going to take more work.

	Thanks,
	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 14 Apr 2014 11:51:57 -0400

> input to mixing and more complicated audio processing.

Yes.

> e.g.using Mark's example AC0(VC0), AC3(V0,V1,V2)

Based on John's comments, I wonder if this is possible at all, for a 
single scene.

Perhaps within a scene it makes no sense to choose audio based on which 
video is chosen. Instead maybe we should simply be trying to satisfy the 
need for mixing and audio processing.

	Thanks,
	Paul

>>> section
>>> never
>>> based

---

From: Paul Coverdale <[email-masked]>
Date: Mon, 14 Apr 2014 14:11:48 -0400

>have a solution worth publishing.

Of course echo is important, but I wouldn't get too pessimistic about it.
While it's always possible to screw it up, I suspect that any serious
Telepresence vendor understands acoustic echo control quite well and will
have taken steps to control it. I don't know that there's much else we can
usefully do from a CLUE perspective. 

...Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 14 Apr 2014 14:16:40 -0400

> usefully do from a CLUE perspective.

I will happily accept the judgement of those who know about this, as 
long as they actively think it is ok. But its bad if there is risk and 
people haven't thought about it.

	Thanks,
	Paul

---

From: Paul Coverdale <[email-masked]>
Date: Mon, 14 Apr 2014 15:20:15 -0400

>> Telepresence vendor understands acoustic echo control quite well and
>> will have taken steps to control it. I don't know that there's much
>> else we can usefully do from a CLUE perspective.
>

[PVC]: Well, if we want to be really bullet-proof about echo we would need
to write detailed specifications and implementation guidelines for acoustic
echo cancellers. But it seems to me that this goes way beyond the current
CLUE charter.

...Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 14 Apr 2014 15:52:48 -0400

> echo cancellers. But it seems to me that this goes way beyond the current

I'll reiterate: we only need to ensure that there is enough information 
so that the endpoints can do this. We don't have to tell them how.

Based on John's comments, it sounds like we might need to specify that 
the advertiser is responsible for removing echo from the captures it 
supplies, rather than thinking that the receiver can do it. (But I don't 
really know if what I said makes any sense.)

	Thanks,
	Paul

---

From: "Duckworth, Mark" <[email-masked]>
Date: Mon, 14 Apr 2014 13:41:12 -0700

I agree echo cancellation is outside the scope of CLUE.  Echo cancellation for audio/video conferencing is nothing new related to CLUE.  CLUE doesn't make echo cancellation any easier or harder than conferencing without CLUE.

Mark

> From: clue [mailto:[email-masked]] On Behalf Of Paul Coverdale
> Sent: Monday, April 14, 2014 2:12 PM
> To: 'Paul Kyzivat'; [email-masked]

---

From: Paul Coverdale <[email-masked]>
Date: Mon, 14 Apr 2014 17:36:39 -0400

>From: Paul Kyzivat [mailto:[email-masked]]
>Sent: Monday, April 14, 2014 3:53 PM
>To: Paul Coverdale; [email-masked]
>>>>> IMO, if we don't have a solution that can prevent this then we
>>>>> don't have a solution worth publishing.
>>>>
>>>> While it's always possible to screw it up, I suspect that any
>>>> serious Telepresence vendor understands acoustic echo control quite
>>>> well and will have taken steps to control it. I don't know that
>>>> there's much else we can usefully do from a CLUE perspective.
>>>
>>> long as they actively think it is ok. But its bad if there is risk
>>> and people haven't thought about it.
>>
>> [PVC]: Well, if we want to be really bullet-proof about echo we would
>> need to write detailed specifications and implementation guidelines
>> for acoustic echo cancellers. But it seems to me that this goes way
>> beyond the current CLUE charter.
>

[PVC]: The endpoints should already know what to do. It's not clear to me
what CLUE information needs to be signaled between endpoints.
>really know if what I said makes any sense.)

[PVC]: I'm not sure what this means. It is the far-end that is always
responsible for removing echo. The near-end can do nothing. Unless by echo
you mean reverberance (barrel effect). The best way to get rid of
reverberance is acoustically (microphone closer to talker, or more
directional microphone), but de-reverberation algorithms exist which can
work on the electrical signal. These algorithms could be applied at the
near-end or the far-end.

...Paul

---

From: John Leslie <[email-masked]>
Date: Tue, 15 Apr 2014 09:58:40 -0400

Paul Kyzivat <[email-masked]> wrote:
>>] REQMT-2:...

   ISTM we're not ready to give up on any of them, but I'm willing to
continue the conversation...

> have a solution worth publishing.

   I think this deserves a separate thread...

> model we have now? (I don't think so.)

   Perhaps REQMT-2 means different things to different folk... :^(

   (Unquestionably, we have different ideas of _how_ to "enable a
satisfactory reproduction" -- but this doesn't bother me: I expect
end-users to adapt to the reproduction they pay for.)

   (BTW, I don't think we need to establish a single meaning for
"spatially correct" -- I only meant to stipulate that I was trying to
avoid that issue.)

   IMHO, a "satisfactory reproduction" can be achieved given separate
audio feeds with a clear specification of the location of each (so long
as you don't have to deal with echo of audio you sent yourself).

   Others, no doubt, prefer a different paradigm where each room sends
a surround-sound representation of their own room. It _is_ possible
to process that into a _different_ surround-sound representation which
will be satisfactory. (Of course, if the received surround-sound has
echo of what _you_ sent, my mind starts to boggle about cancelling echo.)

   Yes.

>   do the above. *That* is the part we need to work on now.

   Speaking for myself, I'll settle for multiple channels of audio
with point-of-capture for each mike.

   (I know I won't always get that -- but I don't blame the CLUE spec
for that.)

> Sounds like something we need to talk about. Where?

   I have no suggestions...

> better, that would be attainable and useful for our purposes?

   Distance from the speaker's mouth would help. If it's "close enough"
the receiver needn't worry -- when an individual want's to mute, that's
the sender's problem.

>> ... the most useful audio streams are linked to microphones "close" to
>> more than one person -- thus video capures will very typically be 
>> associated with several audio captures.
> 
> unrelated to the choice of video captures?

   Mostly, yes.

> audio? And if that is true, then what criteria make sense?

   Clearly, there will be cases where an individual in one room is
particularly interested in some limited area within another room. IMHO
this is _in_addition_to_ the general room audio, so I see no reason
to worry about it in our specification.

> I assume you mean when the speakers are in the same room, right?

   What I meant was audio feeds they receive coming in with so much echo
that they find themselves abliged to exclude some of their own mikes
from what they send.

   This, to tell truth, is a somewhat weird corner case. They should be
able to exclude the _received_ echo from their outgoing feed by normal
echo-cancellation algorithms. If I had thought this through while
writing the above text, I would have erased it before sending.

> Isn't the situation different when we are switching among speakers in 

   I don't understand the question...

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 15 Apr 2014 12:54:50 -0400

> continue the conversation...

We may have a conflict here between continuing the conversation and 
getting something done. We've already been working on this stuff for 
years. If we haven't figured it out by now, then maybe we need to simplify.

So please continue the conversation, maybe for as much as a few days. 
But not for a few more months!

>     I think this deserves a separate thread...

Fine. I don't understand. I get the impression that there is something 
that must be said about echo. I leave it to you.

> end-users to adapt to the reproduction they pay for.)

IMO it is fine that some things are left to "quality of implementation", 
as long as those things are entirely within the scope of a single endpoint.

> echo of what _you_ sent, my mind starts to boggle about cancelling echo.)

Your last two paragraphs above leave me thinking that there is something 
more that must be said in the framework. But I have no idea what it is.

>     (I know I won't always get that -- but I don't blame the CLUE spec

Well, if there is some minimal set of information that will always be 
required to get a workable result, then we should make that mandatory.

E.g., suppose the advertisement has a scene with multiple audio captures 
and *no* spatial information about them. Can a receiver to anything 
useful with that? If not, then we should require more.

>> Sounds like something we need to talk about. Where?

Then maybe we want to declare it "out of scope".

> the receiver needn't worry -- when an individual want's to mute, that's

I want those of you who know something about this to decide what we 
ought to make provision for in the advertisement.

>> unrelated to the choice of video captures?

That would be a really good thing to say in the framework!

> to worry about it in our specification.

What about a case where the receiver is limited in the number of audio 
captures it can receive and/or process? Can the advertisement help the 
receiver decide which one(s) it should get?

(We do have the priority attribute if nothing else.)

>>> they find themselves sabotaged by audio sources contaminated by echo.)

Maybe I misunderstood you above. By "folks" did you mean providers or 
consumers? I was thinking you meant providers.

> writing the above text, I would have erased it before sending.

Again, I don't fully understand, and probably don't need to. But it 
sounds like something more is needed in the fw to discuss expectations.

>     I don't understand the question...

I was thinking of an MCU advertising, among other things, switched audio 
and video captures, where the alternatives come from different rooms. 
Then the whole point is to switch the audio and video together based on 
who is speaking.

	Thanks,
	Paul

---

From: Stephen Botzko <[email-masked]>
Date: Thu, 17 Apr 2014 14:25:56 -0400

Just a general comment on echo...

There are telepresence systems in the field already from multiple vendors.
 They use multiple microphones and multiple speakers, rendering
multichannel audio.  Echo cancellation in those systems works, even if they
are disparate systems using TIP.

Even my home non-telepresence system negotiates full duplex fullband stereo
audio, and it properly cancels the echo even when the received sound is
being rendered through 5 speakers via an external surround sound processor.

And if you build a system which can't handle multichannel echo
cancellation, you can still be CLUE-full.  All you need to do is downmix
the receive audio to mono before you render it and use your single
microphone for capture.  That keeps your system from injecting echo into
the conference.

So echo is not a problem for CLUE.  I agree with Mark that it is
out-of-scope.  We don't see the need to talk about echo when we write up
SIP signalling for multichannel payload formats, and I think the same
principle applies here.

BR,
Steve

On Tue, Apr 15, 2014 at 12:54 PM, Paul Kyzivat <[email-masked]>wrote:

> getting something done. We've already been working on this stuff for years.
> If we haven't figured it out by now, then maybe we need to simplify.
>
> So please continue the conversation, maybe for as much as a few days. But
> not for a few more months!
>
>
> and *no* spatial information about them. Can a receiver to anything useful
> with that? If not, then we should require more.
>
>
> I want those of you who know something about this to decide what we ought
> to make provision for in the advertisement.
>
>
> Again, I don't fully understand, and probably don't need to. But it sounds
> like something more is needed in the fw to discuss expectations.
>
>
> and video captures, where the alternatives come from different rooms. Then
> the whole point is to switch the audio and video together based on who is
> speaking.
>
>         Thanks,
>         Paul
>
>

---

From: John Leslie <[email-masked]>
Date: Tue, 22 Apr 2014 10:48:58 -0400

> Audio 101

   continuing...

Audio 102

Q: What is Echo?
A: Echo is a reflection of sound arriving with sufficient delay to be perceived separately.

Q: How is that different from reverberation?
A: It's a question of perception: when the delay is much less than 100 milliseconds, the ear doesn't perceive it as separate.

Q: But isn't reverberation sometimes measured in seconds (thousands of milliseconds)?
A: Indeed it is. Reverberation often exceeds five seconds in a large cathedral.

Q: So why isn't that "echo"?
A: No individual reflection stands out enough to be perceived as separate. "Reverberation" comes from multiple reflections off different surfaces, which can stretch out for seconds.

Q: Why is echo considered a problem when reverberation is not?
A: But reverberation _is_ a problem when it exceeds a few hundred milliseconds.

Q: Huh?
A: Excessive reverberation "muddies" the speech, and is perceived by the speaker, who speaks more slowly.

Q: Why doesn't that work for echo?
A: The ear perceives separate sound "sources" and the brain is confused by them.

Q: What did people do about this before electronics were invented?
A: They met somewhere else. ;^)

Q: When did we decide to "fix" echo?
A: We felt a need to fix it when we couldn't "meet" somewhere else: specifically when we could only "meet" by telephone.

Q: How's that?
A: As telephones became practical over long distances, they became the only way two people could "meet".

Q: But don't electric signals travel at the speed of light?
A: Actually they never go that fast; but even if they did, 900 miles would produce audible echo. By 1960, satellites were used, giving delays over 100 milliseconds.

Q: What did telephone companies do about it?
A: They made devices to detect echo and "suppress" it by reducing the signal containing echo.

Q: Did that actually work?
A: Well enough for government work ;^) -- folks learned not to talk while the other person talked.

Q: But we've improved on that, haven't we?
A: Indeed: they figured how to do "echo cancellation" instead of just "echo suppression".

Q: How does that work?
A: They predict the echo signal and "cancel" it by inserting the exact opposite.

Q: Isn't that risky -- don't you end up amplifying if you mis-guess the phase?
A: Yes.

Q: So why did it work at all?
A: Telephony was limited to about three kilohertz -- you'd have to be off by 100 microseconds to get in trouble. Besides, their problem was much simpler: no room reverberation, just electrical reflection

Q: Have we improved on that since 1960?
A: We've improved the estimates, yes; but the acoustic echo is far more complex, and we're fighting to track the changes in reflection.

Q: How do we "track" these changes?
A: We run continuous algorithms to update the estimates.

Q: Isn't that expensive?
A: We dedicate on-chip Digital Signal Processors; so the cost is manageable.

Q: What are the practical limits of constantly-updated estimates?
A: Raw processing speed, deviations from the model, and out-of-model events all can lead to bad results. Modern devices generally add non-linear processing, beyond the scope of this tutorial.

Q: Isn't processing speed always increasing due to Moore's law?
A: Yes, but 1) it's not increasing as fast as memory capacity, and 2) equipment upgrades are pretty slow.

Q: What are some typical deviations from the model?
A: Many models assume a clear distinction between originated signal and echos, and guess badly when speech overlaps in direction.

Q: What are some out-of-model events?
A: Many models assume all "interesting" echo will come within N milliseconds. N tends to be much less than 500. :^(

Q: Is there anything we can do about these problems?
A: We task each sender to analyze and cancel the acoustic echo of any sound it receives and sends to loudspeakers in the room.
This keeps the distinction clear between sound originated in the room vs. echo of what it receives (and the number of milliseconds you need to process is kept reasonably small).

Q: Does it make a difference which particular algorithm is used?
A: Yes, but mostly in overall speed and how large N can be. There are also differences in how fast they react to changes.

Q: Is there any reason for us to recommend particular algorithm(s)?
A: No. They tend to be proprietary, and new ones are introduced every year.

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 22 Apr 2014 11:15:27 -0400

John,

IIUC, out of this there is one thing that you are suggesting go into our 
documents:

> We task each sender to analyze and cancel the acoustic echo of any
> sound it receives and sends to loudspeakers in the room.

And then we put this issue to bed. Is that right?

	Thanks,
	Paul

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 22 Apr 2014 13:34:51 -0400

As I tried to say before, acoustic echo cancellation has been provided in
speakerphones and videoconferencing systems for decades. I am still not
seeing anything here that CLUE needs to deal with, and nothing unique to
telepresence.

But I absolutely agree with the key phrase "we put this issue to bed".

>>>We task each sender to analyze and cancel the acoustic echo of any sound
it receives and sends to loudspeakers in the room.

Paul, do you have a specific document in mind where you think this would
fit?  And John - would this really end it, or will there be other warning
labels that you will want to add somewhere?

BR
Stephen

On Tue, Apr 22, 2014 at 11:15 AM, Paul Kyzivat <[email-masked]>wrote:

>> A: It's a question of perception: when the delay is much less than 100
>> milliseconds, the ear doesn't perceive it as separate.
>>
>> A: No individual reflection stands out enough to be perceived as
>> separate. "Reverberation" comes from multiple reflections off different
>> surfaces, which can stretch out for seconds.
>>
>> A: We felt a need to fix it when we couldn't "meet" somewhere else:
>> specifically when we could only "meet" by telephone.
>>
>> Q: How's that?
>> A: Actually they never go that fast; but even if they did, 900 miles
>> would produce audible echo. By 1960, satellites were used, giving delays
>> over 100 milliseconds.
>>
>> A: Telephony was limited to about three kilohertz -- you'd have to be off
>> by 100 microseconds to get in trouble. Besides, their problem was much
>> simpler: no room reverberation, just electrical reflection
>>
>> A: We've improved the estimates, yes; but the acoustic echo is far more
>> complex, and we're fighting to track the changes in reflection.
>>
>> A: Raw processing speed, deviations from the model, and out-of-model
>> events all can lead to bad results. Modern devices generally add non-linear
>> processing, beyond the scope of this tutorial.
>>
>> A: Many models assume a clear distinction between originated signal and
>> echos, and guess badly when speech overlaps in direction.
>>
>> A: Many models assume all "interesting" echo will come within N
>> milliseconds. N tends to be much less than 500. :^(
>>
>> A: We task each sender to analyze and cancel the acoustic echo of any
>> This keeps the distinction clear between sound originated in the room vs.
>> echo of what it receives (and the number of milliseconds you need to
>> process is kept reasonably small).
>>
>> A: Yes, but mostly in overall speed and how large N can be. There are
>> also differences in how fast they react to changes.
>>

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Fri, 25 Apr 2014 08:25:05 +0000

On the echo issue:

I fully agree with those that argue that acoustic echo cancellation is out of scope for CLUE. It is a local responsibility and should be much higher than CLUE on the feature list of any loudspeaking endpoint. While a warning label can be put in some CLUE document, it is really not necessary.

By echo I mean the audio signal received from the network, played out on your local loudspeaker(s), reverberated through the room back to the microphone(s) where it is superimposed on the the near-end talker signal. This echo should be cancelled or suppressed before the microphone signal is encoded and sent.

Regards
Johan

From: Stephen Botzko <[email-masked]<mailto:[email-masked]>>
Date: Tuesday 22 April 2014 19:34
To: Paul Kyzivat <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [clue] Improving treatment of audio

As I tried to say before, acoustic echo cancellation has been provided in speakerphones and videoconferencing systems for decades. I am still not seeing anything here that CLUE needs to deal with, and nothing unique to telepresence.

>>>We task each sender to analyze and cancel the acoustic echo of any sound it receives and sends to loudspeakers in the room.

Paul, do you have a specific document in mind where you think this would fit?  And John - would this really end it, or will there be other warning labels that you will want to add somewhere?

BR
Stephen

On Tue, Apr 22, 2014 at 11:15 AM, Paul Kyzivat <[email-masked]<mailto:[email-masked]>> wrote:
John,

IIUC, out of this there is one thing that you are suggesting go into our documents:

        Thanks,
        Paul

On 4/22/14 10:48 AM, John Leslie wrote:
John Leslie <[email-masked]<mailto:[email-masked]>> wrote:

Audio 101

    continuing...

Audio 102

Q: What is Echo?

Q: Huh?

Q: How's that?

Q: How does that work?

Q: Isn't that expensive?

--
John Leslie <[email-masked]<mailto:[email-masked]>>

_______________________________________________
clue mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/clue

_______________________________________________
clue mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/clue

---

From: John Leslie <[email-masked]>
Date: Fri, 25 Apr 2014 06:27:27 -0400

Johan Ludvig Nielsen (johaniel) <[email-masked]> wrote:
> 
> I fully agree with those that argue that acoustic echo cancellation is
> out of scope for CLUE.

   I don't agree it's helpful to call it "out of scope," but...

> It is a local responsibility and should be much higher than CLUE on
> the feature list of any loudspeaking endpoint.

   Yes, it's a local responsibility.

   Note that not all endpoints of interest will have loudspeakers.

> While a warning label can be put in some CLUE document, it is really
> not necessary.

   I much prefer a parameter saying the responsibility is satisfied
to a warning label in some document nobody reads.

> By echo I mean the audio signal received from the network, played out
> on your local loudspeaker(s), reverberated through the room back to
> the microphone(s) where it is superimposed on the the near-end talker
> signal. This echo should be cancelled or suppressed before the
> microphone signal is encoded and sent.

   This is a rather good statement of the problem (better than mine,
certainly).

   How about a parameter saying "cancelled" vs. "suppressed"?

   BTW, audio is on the agenda for the May 13 design team call.

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Fri, 25 Apr 2014 13:48:28 +0000

I have not been following CLUE lately, please forgive wrong terminology.

We had a very similar discussion on this list more than two years ago
about spatial association of audio to video. At that time I suggested
relating ACs to VCs or a group of VCs, both to assist consumer selection
of streams and to assist mixing/routing for playback on the rendering
side. This was very similar to what Christian and Paul C. has mentioned.

My understanding of the current framework is that receivers calculate the
spatial association between an audio capture and the video scene from the
areas of capture. This will probably work. But I still think having an
explicit association of captures would be a much simpler, more robust and
more scalable solution.

(Many (even advanced multi-capture) systems won´t have a clue about the
capture area and axes of their cameras and microphones, but an association
of audio to video captures can still make perfect sense. For instance in
the education scenario.)

>From the point of view of the renderer, what is needed is info that helps
decide where to play out the audio streams it receives. Whatever the
shortcomings or unknowns of the microphone system in the sending room, and
the problems of linking physical audio and video capture areas, the
playout direction is linked to the video scene to be rendered. Normally
the aim would be to render audio spatially coherent with the video scene,
but it can also be (as John has commented well on) to be able to separate
sound sources for increased intelligibility and scene perception, or to
keep the audio scene static by choice.

An audio capture, be it a single mic signal or a mix, represents a sound
source or a collection of sound sources, which are more often than not
visible in a given video capture or scene and should therefore be
connected to that capture or scene of captures. Naturally, several audio
captures can be connected to the same video capture or scene.

Coordinates can be of interest to convey a finer granularity of the
position of a sound source in the video capture, if available. Information
about the sending room and its microphones (like point or area of capture,
sensitivity, directivity, etc.) are fundamental characteristics of the
capture system, but they are of no use in rendering direction. And
therefore do not need to be advertised or signalled.

More fancy spatial audio schemes exist and may see applications in
telepresence in the (somewhat distant) future, but will probably come in
the form of single stream multichannel formats where the spatial
information is embedded in the format like in surround sound, or
explicitly in the encoded stream like in spatial audio object coding.

Best regards
Johan

On 14/04/14 05:18, "Christian Groves" <[email-masked]>
wrote:

>>>audio
>>>input
>> [Duckworth, Mark] I'm not sure how this would really work.  Because
>>even in this simple example, we also have AC0 relates to VC3
>>(sometimes), VC4, and VC5 (at least part of it), and so on.  AC3 relates
>>to all of VC1 through VC5.  I think using area of capture works better
>>than trying to do something like this.
>>
>> Regards,
>> Mark
>>
>>> Regards, Christian
>>>
>>>>precise as

---

From: John Leslie <[email-masked]>
Date: Fri, 25 Apr 2014 11:05:18 -0400

Johan Ludvig Nielsen (johaniel) <[email-masked]> wrote:
> areas of capture. This will probably work.

   Poorly, at best, IMHO. I'd like to discourage it.

> But I still think having an explicit association of captures would be
> a much simpler, more robust and more scalable solution.

   I entirely agree. I'd like to propose an optional attribute of
video captures to link audio captures in the area it covers.

> From the point of view of the renderer, what is needed is info that
> helps decide where to play out the audio streams it receives.

   While I'm not enthusiastic about choosing that way, this seems an
entirely reasonable piece of information to supply.

> explicitly in the encoded stream like in spatial audio object coding.

   Agreed -- that's not something to settle in our initial spec.

---

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 25 Apr 2014 14:28:28 -0700

Hello Johan,
Thanks very much for the input.
Comments inline.
Mark

> From: Johan Ludvig Nielsen (johaniel) [mailto:[email-masked]]
> Sent: Friday, April 25, 2014 9:48 AM
> To: Christian Groves; Duckworth, Mark; [email-masked]
> We had a very similar discussion on this list more than two years ago about
> spatial association of audio to video. At that time I suggested relating ACs to
> VCs or a group of VCs, both to assist consumer selection of streams and to
> assist mixing/routing for playback on the rendering side. This was very similar
> to what Christian and Paul C. has mentioned.
>

[Duckworth, Mark] Yes, that is the idea

> But I still think having an explicit
> association of captures would be a much simpler, more robust and more
> scalable solution.

[Duckworth, Mark] You might be right.  Christian's proposal is something like this, but to me it still seems incomplete and would need to continue using the spatial coordinates to make sense, which I think was not Christian's intent.  Does anybody want to make further proposals along these lines?

> of audio to video captures can still make perfect sense. For instance in the
> education scenario.)
>
> shortcomings or unknowns of the microphone system in the sending room,
> and the problems of linking physical audio and video capture areas, the
> playout direction is linked to the video scene to be rendered. Normally the
> aim would be to render audio spatially coherent with the video scene, but it
> can also be (as John has commented well on) to be able to separate sound
> sources for increased intelligibility and scene perception, or to keep the
> audio scene static by choice.
[Duckworth, Mark] I agree

> An audio capture, be it a single mic signal or a mix, represents a sound source
> or a collection of sound sources, which are more often than not visible in a
> given video capture or scene and should therefore be connected to that
> capture or scene of captures. Naturally, several audio captures can be
> connected to the same video capture or scene.

[Duckworth, Mark] I agree.  Today in CLUE we can explicitly associate a set of audio captures with a set of video captures by putting them in the same scene.  The difficulty seems to be associating audio with video when there are more than one of each in the same scene.  The intent was to do that with the area of capture attribute, but maybe we can do better.

> Coordinates can be of interest to convey a finer granularity of the position of
> a sound source in the video capture, if available. Information about the
> sending room and its microphones (like point or area of capture, sensitivity,
> directivity, etc.) are fundamental characteristics of the capture system, but
> they are of no use in rendering direction.

[Duckworth, Mark] I don't understand, because those last two sentences seem to contradict each other.  Wouldn't the rendering system be able to make use of the more fine grain positioning information if it wanted to?  Can you explain more?

> And therefore do not need to be
> advertised or signalled.
>
> telepresence in the (somewhat distant) future, but will probably come in the
> form of single stream multichannel formats where the spatial information is
> embedded in the format like in surround sound, or explicitly in the encoded
> stream like in spatial audio object coding.

[Duckworth, Mark] I am focusing on the single stream (single capture) multichannel formats.  Secondarily, I am considering the multiple audio capture scenarios.  Both are mentioned in the use cases: "The audio may be transmitted as multi-channel (stereo/surround sound) or as distinct and separate monophonic streams"

> >>> Perhaps the confusion is that some see Audio Capture area as a means
> >>>to  associate an Audio capture with a video capture as you've shown
> >>>Mark's  example. e.g. The audio spatial information isn't really used
> >>>for any audio  transformation other than associating it with a
> >>>particular video stream.
> >>>
> >>> Whereas others were more thinking the audio spatial information as
> >>>an input  to mixing and more complicated audio processing.
> >>> If this is the case perhaps rather than linking ACs and VCs through
> >>> the physical or virtual co-ordinates of the area of capture
> >>> information, we simplify things and in each audio capture we say what
> VC it relates to?
> >>(sometimes), VC4, and VC5 (at least part of it), and so on.  AC3
> >>relates to all of VC1 through VC5.  I think using area of capture
> >>works better than trying to do something like this.
> >>
> >> Regards,
> >> Mark
> >>
> >>> Regards, Christian
> >>>
> >>>>> I can see what you're trying to do in the example given in
> >>>>>Framework  section 12.1.1. The problem I have is that, because of
> >>>>>the huge  difference in wavelength between light waves and audio
> >>>>>waves, you  can't really define an area of capture for audio in the
> >>>>>same way that  you can for video. A camera can focus on a scene
> >>>>>defined by 4  co-planar X,Y,Z coordinates. It will capture the
> >>>>>video inside that  quadrilateral, and nothing outside. A microphone
> >>>>>can capture audio  coming from inside of the same quadrilateral,
> >>>>>but it will also  capture a lot of audio from outside. What this
> >>>>>means is that we can  never define a unique association of an audio
> >>>>>capture with a video  capture based on pure geometrical
> >>>>>considerations, but we can still simply
> >>>>>> The original intent of using "area of capture" for all media
> >>>>>>types,  including audio and video, was to provide a way to
> >>>>>>associate audio  with video.  So the consumer can choose captures
> >>>>>>that go together  and render them together.  An example of this is
> >>>>>>in the framework  document, section 12.1.1.  I still think this
> >>>>>>works fine for this purpose.
> >>>>>>
> >>>>>> and AC2.  AC0 and VC0 have the same area of capture, indicating
> >>>>>> they are capturing the same area of the scene.  If the consumer
> >>>>>> wants to render
> >>>>>> Similarly, AC3 has an area of capture covering the whole scene
> >>>>>> (the full extent of the areas of VC0, VC1, VC2) so the consumer
> >>>>>> knows AC3 includes audio associated with all of VC0, VC1, and VC2.
> >>>>>>
> >>>>>>>>   From John's Audio 101 it seems that anything to do with
> >>>>>>>> spatial information that would relate to reverberation is in
> >>>>>>>> the too hard basket.
> >>>>>>>      I haven't even reached the point of suggesting what metrics
> >>>>>>> to define the _ability_ to send. To me, "too hard" merely means
> >>>>>>> that an individual site could choose not to send them or to
> >>>>>>> ignore them on
> >>>>>> receipt.
> >>>>>>>      But it sounds as if you're suggesting reverberation is "to
> >>>>>>>hard  to understand" and thus we should have no metrics about it.
> >>>>>>>
> >>>>>>>      I don't understand this. Point of capture for a microphone
> >>>>>>> may be "too hard" to track (today) for a microphone which moves;
> >>>>>>>>   From the Audio101 there does seem to be a dependency on
> where
> >>> the
> >>>>>>>      IMHO, it's more flexible to have mixing be the responsibily
> >>>>>>> of the receiver, but I haven't tried to specify that and I'm not
> >>>>>>> at all sure
> >>>>>> I want to specify that.
> >>>>>>>      I expect the actual sound systems in different rooms to
> >>>>>>> vary wildly, from one monaural speaker to stereo to full surround-
> sound.
> >>>>>>> Mixing for these without knowing which is the actual target
> >>>>>>> seems hard; but I'm sure there will be sites which prefer to do
> >>>>>>> so. A question which will arise, IMHO, is how to specify the
> >>>>>>> _intent_ of a mix generated in one room to be fed to other
> >>>>>>> providing several audio streams per room, associated with
> >>>>>>> position information about the position of the source of those
> >>>>>>> sounds, and allowing the receiver to choose how to mix them and
> >>>>>>> how to present the
> >>>>> mix in his/her room.
> >>>>>>>> i.e. a talker giving a presentation using a lapel mic captured
> >>>>>>>> by a particular video.
> >>>>>>>      In fact, there's only limited tendency for humans to
> >>>>>>>strictly  attach the sound they hear to the video they see.
> >>>>>>>Clearly, during a  presentation we want to _hear_ the presenter
> >>>>>>>talking, but having  the sound move back and forth as the speaker
> >>>>>>>walks can be confusing.
> >>>>>>>
> >>>>>>>      At the same time, we will want to hear the questions to
> >>>>>>> which the presenter may respond. It will be far easier on the
> >>>>>>> listener if these do _not_ seem to be coming from the same
> >>>>>>> physical position, especially when the questioner _is_ in the
> >>>>>>> same room as the
> >>> presenter.
> >>>>>>>      Hope this helps...
> >>>>>>>
> >>>>>>> --

---

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 25 Apr 2014 14:42:16 -0700

Hi John,
Can you elaborate on your proposal about linking audio and video captures?  I'd like to see more detail.

Mark

> Sent: Friday, April 25, 2014 11:05 AM
> To: Johan Ludvig Nielsen (johaniel)
> Cc: Christian Groves; Duckworth, Mark; [email-masked]
> > My understanding of the current framework is that receivers calculate
> > the spatial association between an audio capture and the video scene
> > from the areas of capture. This will probably work.
> 
>    I entirely agree. I'd like to propose an optional attribute of video captures to
> link audio captures in the area it covers.
> 
>    While I'm not enthusiastic about choosing that way, this seems an entirely
> reasonable piece of information to supply.
> 
> > telepresence in the (somewhat distant) future, but will probably come
> > in the form of single stream multichannel formats where the spatial

---

From: Stephen Botzko <[email-masked]>
Date: Fri, 25 Apr 2014 19:52:51 -0400

> to a warning label in some document nobody reads.

How could such a parameter be used?   If my system isn't using loudspeakers
then there is need for my system to do echo cancellation, so it would
presumably set the parameter to "no".  If my system does have loudspeakers,
then it does need to do it, and it would presumably set the parameter to
"yes".  In either case there is no action needed anywhere else.  So I don't
see how any other system could use such a parameter.

>    How about a parameter saying "cancelled" vs. "suppressed"?
What would the use case be? Either way my rendering doesn't change.  Also,
echo cancellers all do some amount of suppression, so one would need to be
very careful on the definition.

---

From: Christian Groves <[email-masked]>
Date: Mon, 28 Apr 2014 11:19:10 +1000

Hello Mark,

Please see my comments below.

Regards, Christian
On 26/04/2014 7:28 AM, Duckworth, Mark wrote:
..snip..
> [Duckworth, Mark] You might be right.  Christian's proposal is something like this, but to me it still seems incomplete and would need to continue using the spatial coordinates to make sense, which…
[CNG] Yes my intent was only to use one set of spatial co-ordinates i.e. 
the ones on the video captures. The reference in the audio captures 
indicates which video captures it relates to. Could you elaborate what 
seems to be incomplete?

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 28 Apr 2014 16:24:00 -0400

>     Poorly, at best, IMHO. I'd like to discourage it.

I still don't understand how it would work.

The area of capture is a planar quadrilateral. But of course neither a 
camera nor a microphone capture their media solely from a planar 
quadrilateral in three-space. (Though this might be a reasonable 
representation of a powerpoint or screen capture.)

The camera captures roughly a pyramid shaped region of the scene, with 
the point of capture being the peak of the pyramid, and the area of 
capture bounding the intersection of the pyramid and a plane. Not all of 
that will be in focus, and some of it may be occluded. I guess typically 
the content of interest will fall in the portion of the pyramid behind 
the area of capture.

How to relate the area of capture of an audio capture to what is going 
on in the scene seems much less obvious. Does it mean it only captures 
audio emitted within the pyramid? Or that it captures audio that may be 
heard from within the pyramid?

*How* do I relate the area of capture of an audio and video capture to 
decide what audio captures I need to make sense of the video capture? Do 
I intersect the planar areas of capture? (Which yields nothing if they 
aren't coplanar.) Or do I intersect the pyramids?

Or, as John has suggested, will I always need all the audio from a scene 
to understand any video from the scene?

If this isn't defined well we may get very poor interoperation.

	Thanks,
	Paul

---

From: Paul Coverdale <[email-masked]>
Date: Mon, 28 Apr 2014 19:09:33 -0400

Comments in-line...

>Sent: Monday, April 28, 2014 4:24 PM
>To: [email-masked]

[PVC] Me neither.

>heard from within the pyramid?

[PVC] I think the sooner we give up trying to define an audio area of
capture in the same manner as a video area of capture, the better. It just
doesn't work.

---

From: "Duckworth, Mark" <[email-masked]>
Date: Mon, 28 Apr 2014 18:36:06 -0700

Hi Christian,
I'm coming back to this message, because I'm trying to understand your proposal.  You proposed "in each audio capture we say what VC it relates to."  What does this mean?  What does it mean for an audio capture to relate to a video capture?  What exactly is the producer telling the consumer?  Can an audio capture relate to more than one video capture?

More comments inline below.

Regards,
Mark

> From: Christian Groves [mailto:[email-masked]]
> Sent: Sunday, April 13, 2014 11:18 PM
> To: Duckworth, Mark; [email-masked]
> Sorry I didn't consider the entire 12.1.1. if I do that, according to that
> example:
> 
> Video areas of capture:
> 

[Duckworth, Mark] Looking from the consumer point of view, in trying to understand the advertisement, what does the consumer know from the advertisement?  Suppose the consumer wants to get all three of VC0, VC1, VC2, but it only wants a single audio capture.  It should probably ask for AC3, but according to the advertisement AC3 doesn't "relate to" VC0, VC1, or VC2.

[Duckworth, Mark] For a case with a different consumer, suppose the consumer wants the single video capture VC5, but it would like to receive and render spatial audio.  There is no audio capture with multiple channels, so it could ask for AC0, AC1, and AC2 if it knew how they were spatially "related to" VC5.  But according to the advertisement, AC0, AC1 and AC2 aren't "related to" VC5 at all.

> >> for any audio transformation other than associating it with a particular
> video stream.
> >>
> >> information, we simplify things and in each audio capture we say what VC
> it relates to?
> > [Duckworth, Mark] I'm not sure how this would really work.  Because even
> in this simple example, we also have AC0 relates to VC3 (sometimes), VC4,
> and VC5 (at least part of it), and so on.  AC3 relates to all of VC1 through VC5.
> I think using area of capture works better than trying to do something like
> this.
...snip...

---

From: Christian Groves <[email-masked]>
Date: Tue, 29 Apr 2014 12:15:22 +1000

Hello Mark,

Please see below.

Regards, Christian

> I'm coming back to this message, because I'm trying to understand your proposal.  You proposed "in each audio capture we say what VC it relates to."  What does this mean?  What does it mean for an…
[CNG] The meaning has largely the same semantic as if the ADV used the 
same capture area on a VC and AC. Basically its saying that the audio 
comes from the same "region" as what the video capture does. I think the 
difference is that its not trying to put a mathematical certainty about 
what that region is. Its a way for the producer to say to the consumer 
if you choose video capture A you probably want to choose audio capture 
B for a good experience. In terms of whether an audio capture can relate 
to more than one video capture yes I think so. You could have one audio 
capture for a room and three video captures.
> [Duckworth, Mark] Looking from the consumer point of view, in trying to understand the advertisement, what does the consumer know from the advertisement?  Suppose the consumer wants to get all thre…
[CNG] I agree. If the provider wanted to indicate that AC3 could also be 
used for VC0,VC1,VC2 it could advertise:
      AC0 (VC0)
      AC1 (VC2)
      AC2 (VC1)
      AC3 (VC0,VC1,VC2,VC3,VC4,VC5)
> [Duckworth, Mark] For a case with a different consumer, suppose the consumer wants the single video capture VC5, but it would like to receive and render spatial audio.  There is no audio capture wi…
[CNG] So in this case there is no AC3? In that case the advertisement 
could look like:
      AC0 (VC0,VC5)
      AC1 (VC2,VC5)
      AC2 (VC1,VC5)

Each audio capture could relate to a particular video or be part of the 
zoomed out video capture if there was no room audio.

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 29 Apr 2014 00:10:20 -0400

On 4/28/14 10:15 PM, Christian Groves wrote:
>> I'm coming back to this message, because I'm trying to understand your
>> proposal.  You proposed "in each audio capture we say what VC it
>> relates to."  What does this mean?  What does it mean for an audio
>> capture to relate to a video capture?  What exactly is the producer
>> telling the consumer?  Can an audio capture relate to more than one
>> video capture?

Do you think this can be defined well enough that two people, given the 
same configuration of equipment, would reach the same conclusion about 
which audios to associate with which videos?

	Thanks,
	Paul

>>> that
>> [Duckworth, Mark] Looking from the consumer point of view, in trying
>> to understand the advertisement, what does the consumer know from the
>> advertisement?  Suppose the consumer wants to get all three of VC0,
>> VC1, VC2, but it only wants a single audio capture.  It should
>> probably ask for AC3, but according to the advertisement AC3 doesn't
>> "relate to" VC0, VC1, or VC2.
>> [Duckworth, Mark] For a case with a different consumer, suppose the
>> consumer wants the single video capture VC5, but it would like to
>> receive and render spatial audio.  There is no audio capture with
>> multiple channels, so it could ask for AC0, AC1, and AC2 if it knew
>> how they were spatially "related to" VC5.  But according to the
>> advertisement, AC0, AC1 and AC2 aren't "related to" VC5 at all.
>>>>> particular
>>>> even

---

From: Christian Groves <[email-masked]>
Date: Tue, 29 Apr 2014 19:41:10 +1000

On 29/04/2014 2:10 PM, Paul Kyzivat wrote:
> Do you think this can be defined well enough that two people, given 
> the same configuration of equipment, would reach the same conclusion 
> about which audios to associate with which videos?

I can't see why not? CLUE already suggests relationships between 
captures i.e.through CSE and GCSEs. You can't assume that the spatial 
co-ordinates will give the relationships in these cases as the spatial 
attributes aren't mandatory.

Christian

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 29 Apr 2014 10:35:18 -0400

This may be a digression, but I think it would be useful to identify
specific scenarios that work (and don't work) before we go in and try and
improve things.  I'm providing my assessment below for comments/feedback.

BR
Stephen

-Room to Room rendering works.

In the normal telepresence point-to-point case, there is a large video wall
in the front, with some number of speakers also in the front.  The size of
the wall, the number of cameras/screens, and the number of loudspeakers is
mismatched.

On the video side, there is enough spatial information to allow the
receivers to render the far end room "properly", and several strategies
that receivers can use.  Once the receiver knows the mapping between its
local coordinates and the far end, it can then make corresponding
adjustments to the received audio.  For instance, for each received audio
channel, it can identify the nearest two loudspeakers, and can use normal
panning to create a sound source at a point in between.  The spatial audio
is "soft" - similar to the results you get when you derive a center channel
from two stereo speakers.  A center speaker in the right place certainly
sounds more realistic, but using two stereo speakers to simulate it works
acceptably.

For users, even the "soft" spatial audio improves the user experience.  For
instance, when someone starts talking, even the "soft" spatial audio is
enough to allow the participants to automatically look in the right
direction when someone new starts talking.

If you wanted to use rear speakers to place some sounds behind the local
audience, that would not work.  But I don't think that's a very important
case.  It is probably more common to use any rear speakers for normal sound
reinforcement (exploiting the Haas effect).

So I believe this scenario is already enabled.  Industry experience with
interoperability today (using TIP) pretty much supports that view.

Could we do better?  In theory some approaches (MPEG SAOC) might be able to
create a better spatial sound. Although SAOC proponents sometimes refer to
interactive conferencing as an application, I don't think it is
well-enabled in practice.  Locating and isolating each sound source in the
room in real time is not an easy task, and SAOC gives no hints on how to do
it.  SAOC's real application is gaming, where pre-recorded and synthetic
sounds are being blended into a virtual reality.  Also, SAOC uses in-band
transmission of audio object information, so it could be used with CLUE in
a future system.  There would likely need to be some form of tagging the
objects, so that the video rendering could also take advantage of them. But
that could be added in the future.  SAOC is likely encumbered (just a guess
I haven't checked).

So at some point we could perhaps do better, but it is still a bit of a
research project.  I think the current functionality is enough.

-Whole Room Composition works

Similarly, in a multipoint case you frequently end up giving up on full
size rendering and eye-contact because you simply don't have the screen
real estate.  In those cases, you are scaling the room video down (usually
to something a lot smaller) and placing it somewhere in the composition.

On the audio side, the panning techniques in the full room-to-room case
still apply, and give a satisfactory (though soft") spatial audio
experience.  Sounds that are off-camera in the far-end room might be placed
in an adjacent room in the composition.  However the existing spatial
system allows receivers to detect that the far-end sound stage is wider
than the video scene.  And there is a remedy - receivers can also choose to
down-mix the far-end room audio to a single mono capture, and align that
with the video.  In many cases, that works just as well (since the room
view on the screen is relatively small), and it ensures that all the audio
from the room is aligned with the video composition.

So I also think the current functionality is enough.

-Rendering in a deliberately different spatial alignment does not work.

One could chose to totally ignore the spatial information for video.  For
instance, you might wish to present independent video tiles for the various
talkers, putting the current talker in the center, and other recent talkers
around the side - totally ignoring the spatial information and what room
the talker is in.

You can do this with video.  However, re-processing the sound from the far
end rooms in this case doesn't work well.  The spatial sound would not be
coherent or natural, because each audio capture includes some sound from
the other parts of the room, and in general there is no way to remove that
sound.  So in this scenario, receivers probably need to fall back to
monophonic audio (rendering all the audio from all senders monophonically).

It would be nice to do better here.  But I think the problem is really in
the audio capture itself.  Changing the signaling won't fix it.  So the
current functionality is not enough for this case, but I think for now
commercial systems can't do better than monophonic rendering.

-Talker identification does not work.

It is common in videoconferencing to use "voice-activated switching".
That works by detecting speech in the far-end audio channel, and then
automatically switching to the corresponding video channel.

This is a problem area for CLUE (and it is also a problem area I've seen
with systems using TIP).

Part of the problem grounded in the audio capture itself.  A talker in the
room is to some extent picked up by all the microphones.  You might think
it is easy to identify the right capture from the volume level, but in many
microphone arrangements that doesn't work reliably.  Participants sitting
between microphones are hard to locate.  Participants also turn towards
other people in the local room when they talk, which often changes the
microphone that picks them up best.  And often there are multiple
simultaneous talkers in the room - we are not always polite.  So even
detecting the approximate talker location(s) from the far end is
problematic.  My company's products can locate the talker(s) within the
room accurately, but they are using information that is not available to
far-end systems, and which would be hard to send in a product-independent
way.

Personally I favor explicit signaling from the sender on which capture(s)
carry the current talker(s).  Middle boxes and far end systems can
determine which rooms have active talkers by down-mixing to mono.  To get
finer granularity within the room, you could then use the explicit
signaling.

The second aspect of the problem is associating the talker with a video
capture.  Again, this seems somewhat difficult, especially if the audio
captures are not precisely aligned to the video captures. We can probably
improve this - but if we need explicit talker signaling anyway, it could
potentially identify the best video capture. In fact, since I personally
think that whole-room audio rendering should always be done, I am much more
interested in the video capture than the audio capture.

Other approaches might work, the main point I am making here is that remote
identification of talker locations in the room is probably broken.

-Partial Room Rendering will not always work well.

If you are selecting some audio captures from a room, and discarding
others, you might not get natural sounding audio.  How good it sounds will
depend on the details of the microphone pickup in the far end room, how
controlled participant seating is, and which captures you happen to discard.

Since the goal for CLUE is interoperability across disparate room designs,
we can't make a lot of assumptions about microphone pickup.  I think we can
assume that the full room audio will sound appropriate - if it doesn't,
then that means that the sender doesn't have a well-designed system.  But
once you start filtering out captures in the middle, the results will vary.
 I'm not seeing much CLUE can do about that, other than point it out.  As
noted above, I think that rendering the entire room's transmitted sound
field is going to sound the best.  If there is a need for more conditioning
(say noise reduction) it is probably better to do it locally at the sender.

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 29 Apr 2014 12:49:46 -0400

Stephen,

(First my usual disclaimer that I know nothing about audio, so these may 
be naive comments.)

I more-or-less follow what you are saying below. My main concern is the 
same as I mentioned yesterday. You aren't giving me enough detail about 
how you are assuming certain things are determined from the advertisement.

E.g., you say "for each received audio channel, it can identify the 
nearest two loudspeakers". But nearest to *what*? Nearest to the center 
of the Area of Capture of the audio after coordinate mapping? Or nearest 
to the Point of Capture of the audio?

	Thanks,
	Paul

On 4/29/14 10:35 AM, Stephen Botzko wrote:
> specific scenarios that work (and don't work) before we go in and try
> and improve things.  I'm providing my assessment below for
> comments/feedback.
>
> BR
> Stephen
>
>
>     In the normal telepresence point-to-point case, there is a large
>     video wall in the front, with some number of speakers also in the
>     front.  The size of the wall, the number of cameras/screens, and the
>     number of loudspeakers is mismatched.
>
>     receivers to render the far end room "properly", and several
>     strategies that receivers can use.  Once the receiver knows the
>     mapping between its local coordinates and the far end, it can then
>     make corresponding adjustments to the received audio.  For instance,
>     for each received audio channel, it can identify the nearest two
>     loudspeakers, and can use normal panning to create a sound source at
>     a point in between.  The spatial audio is "soft" - similar to the
>     results you get when you derive a center channel from two stereo
>     speakers.  A center speaker in the right place certainly sounds more
>     realistic, but using two stereo speakers to simulate it works
>     acceptably.
>
>     For users, even the "soft" spatial audio improves the user
>     experience.  For instance, when someone starts talking, even the
>     "soft" spatial audio is enough to allow the participants to
>     automatically look in the right direction when someone new starts
>     talking.
>
>     If you wanted to use rear speakers to place some sounds behind the
>     local audience, that would not work.  But I don't think that's a
>     very important case.  It is probably more common to use any rear
>     speakers for normal sound reinforcement (exploiting the Haas effect).
>
>     So I believe this scenario is already enabled.  Industry experience
>     with interoperability today (using TIP) pretty much supports that view.
>
>     Could we do better?  In theory some approaches (MPEG SAOC) might be
>     able to create a better spatial sound. Although SAOC proponents
>     sometimes refer to interactive conferencing as an application, I
>     don't think it is well-enabled in practice.  Locating and isolating
>     each sound source in the room in real time is not an easy task, and
>     SAOC gives no hints on how to do it.  SAOC's real application is
>     gaming, where pre-recorded and synthetic sounds are being blended
>     into a virtual reality.  Also, SAOC uses in-band transmission of
>     audio object information, so it could be used with CLUE in a future
>     system.  There would likely need to be some form of tagging the
>     objects, so that the video rendering could also take advantage of
>     them. But that could be added in the future.  SAOC is likely
>     encumbered (just a guess I haven't checked).
>
>     So at some point we could perhaps do better, but it is still a bit
>     of a research project.  I think the current functionality is enough.
>
>
>
>     Similarly, in a multipoint case you frequently end up giving up on
>     full size rendering and eye-contact because you simply don't have
>     the screen real estate.  In those cases, you are scaling the room
>     video down (usually to something a lot smaller) and placing it
>     somewhere in the composition.
>
>     On the audio side, the panning techniques in the full room-to-room
>     case still apply, and give a satisfactory (though soft") spatial
>     audio experience.  Sounds that are off-camera in the far-end room
>     might be placed in an adjacent room in the composition.  However the
>     existing spatial system allows receivers to detect that the far-end
>     sound stage is wider than the video scene.  And there is a remedy -
>     receivers can also choose to down-mix the far-end room audio to a
>     single mono capture, and align that with the video.  In many cases,
>     that works just as well (since the room view on the screen is
>     relatively small), and it ensures that all the audio from the room
>     is aligned with the video composition.
>
>     One could chose to totally ignore the spatial information for video.
>       For instance, you might wish to present independent video tiles
>     for the various talkers, putting the current talker in the center,
>     and other recent talkers around the side - totally ignoring the
>     spatial information and what room the talker is in.
>
>     You can do this with video.  However, re-processing the sound from
>     the far end rooms in this case doesn't work well.  The spatial sound
>     would not be coherent or natural, because each audio capture
>     includes some sound from the other parts of the room, and in general
>     there is no way to remove that sound.  So in this scenario,
>     receivers probably need to fall back to monophonic audio (rendering
>     all the audio from all senders monophonically).
>
>     It would be nice to do better here.  But I think the problem is
>     really in the audio capture itself.  Changing the signaling won't
>     fix it.  So the current functionality is not enough for this case,
>     but I think for now commercial systems can't do better than
>     monophonic rendering.
>
>     It is common in videoconferencing to use "voice-activated
>     switching".   That works by detecting speech in the far-end audio
>     channel, and then automatically switching to the corresponding video
>     channel.
>
>     This is a problem area for CLUE (and it is also a problem area I've
>     seen with systems using TIP).
>
>     Part of the problem grounded in the audio capture itself.  A talker
>     in the room is to some extent picked up by all the microphones.  You
>     might think it is easy to identify the right capture from the volume
>     level, but in many microphone arrangements that doesn't work
>     reliably.  Participants sitting between microphones are hard to
>     locate.  Participants also turn towards other people in the local
>     room when they talk, which often changes the microphone that picks
>     them up best.  And often there are multiple simultaneous talkers in
>     the room - we are not always polite.  So even detecting the
>     approximate talker location(s) from the far end is problematic.  My
>     company's products can locate the talker(s) within the room
>     accurately, but they are using information that is not available to
>     far-end systems, and which would be hard to send in a
>     product-independent way.
>
>     Personally I favor explicit signaling from the sender on which
>     capture(s) carry the current talker(s).  Middle boxes and far end
>     systems can determine which rooms have active talkers by down-mixing
>     to mono.  To get finer granularity within the room, you could then
>     use the explicit signaling.
>
>     The second aspect of the problem is associating the talker with a
>     video capture.  Again, this seems somewhat difficult, especially if
>     the audio captures are not precisely aligned to the video captures.
>     We can probably improve this - but if we need explicit talker
>     signaling anyway, it could potentially identify the best video
>     capture. In fact, since I personally think that whole-room audio
>     rendering should always be done, I am much more interested in the
>     video capture than the audio capture.
>
>     Other approaches might work, the main point I am making here is that
>     remote identification of talker locations in the room is probably
>     broken.
>
>     others, you might not get natural sounding audio.  How good it
>     sounds will depend on the details of the microphone pickup in the
>     far end room, how controlled participant seating is, and which
>     captures you happen to discard.
>
>     Since the goal for CLUE is interoperability across disparate room
>     designs, we can't make a lot of assumptions about microphone pickup.
>       I think we can assume that the full room audio will sound
>     appropriate - if it doesn't, then that means that the sender doesn't
>     have a well-designed system.  But once you start filtering out
>     captures in the middle, the results will vary.  I'm not seeing much
>     CLUE can do about that, other than point it out.  As noted above, I
>     think that rendering the entire room's transmitted sound field is
>     going to sound the best.  If there is a need for more conditioning

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 29 Apr 2014 14:48:50 -0400

In-line

On Tue, Apr 29, 2014 at 12:49 PM, Paul Kyzivat <[email-masked]>wrote:

> same as I mentioned yesterday. You aren't giving me enough detail about how
> you are assuming certain things are determined from the advertisement.
>
> nearest two loudspeakers". But nearest to *what*? Nearest to the center of
> the Area of Capture of the audio after coordinate mapping? Or nearest to
> the Point of Capture of the audio?
>
[sb]Center of the area was what I was thinking, but Point of Capture could
probably also be used in practice.. Either way, that point is mapped onto a
corresponding pixel on the video display (which is showing that same point
in the far end room's video).  If the point isn't displayed because it is
off camera, then you can still compute where it would have been displayed
if you had a big enough screen.

The rendered audio capture is panned so it sounds like it is coming from
that pixel.  To do that you can use the two nearest speakers to that pixel,
and you send the right proportion of sound to each (proportionally more
sound to the closer speaker - just linear interpolation).  In practice
people won't be able to localize the sound very precisely with this
approach, which is one of things I was trying to communicate with the
"soft" spatial audio description.  Most multi-channel loudspeaker systems
don't allow humans to localize the sound all that accurately - which I
think was one of the points John Leslie also made.  But it is good enough
to be useful.

What I mean by localize: imagine blindfolding people and ask them to point
to the apparent sound source.  They won't be spot-on with the CLUE
approach, and they might feel tentative on the direction they choose (this
is often the case with spatial audio rendered through multiple
loudspeakers).  But they should be pointing in approximately the right
direction most of the time.

I can provide more details if you ask more questions.  Though I am also
wanting folks who are conversant with audio to provide their own thoughts
on what works and what doesn't.[/sb]

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 29 Apr 2014 15:44:10 -0400

I realize this is probably not a topic that ought to be normative. But 
ISTM that *something* should be said about it. I think the advertiser 
and the consumer need to be in agreement on what this information means 
or how it is intended to be used in order to get interoperability. There 
seems to be no one obvious way to pick an Area of Capture for a 
microphone, but different decisions about what to choose may well effect 
the way the consumer's algorithm works.

 From what you describe, perhaps there is no value in area of capture 
for audio, and axis of capture would be more useful. And I assume that 
in the absence of axis of capture, one may approximate it with a line 
through the center of the area of capture and perpendicular to it.

	Thanks,
	Paul

On 4/29/14 2:48 PM, Stephen Botzko wrote:
> In-line
>
>
> On Tue, Apr 29, 2014 at 12:49 PM, Paul Kyzivat <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Stephen,
>
>     (First my usual disclaimer that I know nothing about audio, so these
>     may be naive comments.)
>
>     I more-or-less follow what you are saying below. My main concern is
>     the same as I mentioned yesterday. You aren't giving me enough
>     detail about how you are assuming certain things are determined from
>     the advertisement.
>
>     nearest two loudspeakers". But nearest to *what*? Nearest to the
>     center of the Area of Capture of the audio after coordinate mapping?
>     Or nearest to the Point of Capture of the audio?
>
> [sb]Center of the area was what I was thinking, but Point of Capture
> could probably also be used in practice.. Either way, that point is
> mapped onto a corresponding pixel on the video display (which is showing
> that same point in the far end room's video).  If the point isn't
> displayed because it is off camera, then you can still compute where it
> would have been displayed if you had a big enough screen.
>
> that pixel.  To do that you can use the two nearest speakers to that
> pixel, and you send the right proportion of sound to each
> (proportionally more sound to the closer speaker - just linear
> interpolation).  In practice people won't be able to localize the sound
> very precisely with this approach, which is one of things I was trying
> to communicate with the "soft" spatial audio description.  Most
> multi-channel loudspeaker systems don't allow humans to localize the
> sound all that accurately - which I think was one of the points John
> Leslie also made.  But it is good enough to be useful.
>
> What I mean by localize: imagine blindfolding people and ask them to
> point to the apparent sound source.  They won't be spot-on with the CLUE
> approach, and they might feel tentative on the direction they choose
> (this is often the case with spatial audio rendered through multiple
> wanting folks who are conversant with audio to provide their own
> thoughts on what works and what doesn't.[/sb]
>
>
>              Thanks,
>              Paul
>
>
>              (say noise reduction) it is probably better to do it
>         locally at the
>              sender.
>
>
>
>
>
>
>
>
>         _________________________________________________
>         clue mailing list
>         [email-masked] <mailto:[email-masked]>
>         https://www.ietf.org/mailman/__listinfo/clue
>         <https://www.ietf.org/mailman/listinfo/clue>
>
>
>     _________________________________________________
>     clue mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/__listinfo/clue
>     <https://www.ietf.org/mailman/listinfo/clue>
>
>

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 29 Apr 2014 16:06:56 -0400

On Tue, Apr 29, 2014 at 3:44 PM, Paul Kyzivat <[email-masked]> wrote:

> ISTM that *something* should be said about it. I think the advertiser and
> the consumer need to be in agreement on what this information means or how
> it is intended to be used in order to get interoperability. There seems to
> be no one obvious way to pick an Area of Capture for a microphone, but
> different decisions about what to choose may well effect the way the
> consumer's algorithm works.

I agree that some information is needed. Perhaps continuing to drill down
on some of these cases will help clarify what needs to be said.

>
>
>From what you describe, perhaps there is no value in area of capture for
> audio, and axis of capture would be more useful. And I assume that in the
> absence of axis of capture, one may approximate it with a line through the
> center of the area of capture and perpendicular to it.
>
Axis of capture could be more useful, though I am not aware of any
techniques that would let you align your output audio along a different
axis from the loudspeaker's natural orientation.

---

From: Paul Coverdale <[email-masked]>
Date: Tue, 29 Apr 2014 19:50:27 -0400

I realize this is probably not a topic that ought to be normative. But ISTM that *something* should be said about it. I think the advertiser and the consumer need to be in agreement on what this information means or how it is intended to be used in order to get interoperability. There seems to be no one obvious way to pick an Area of Capture for a microphone, but different decisions about what to choose may well effect the way the consumer's algorithm works.

I agree that some information is needed. Perhaps continuing to drill down on some of these cases will help clarify what needs to be said.

>From what you describe, perhaps there is no value in area of capture for audio, and axis of capture would be more useful. And I assume that in the absence of axis of capture, one may approximate it with a line through the center of the area of capture and perpendicular to it.

Axis of capture could be more useful, though I am not aware of any techniques that would let you align your output audio along a different axis from the loudspeaker's natural orientation. 

[Paul Coverdale]  I’m not aware of such a technique either, but are you saying that the idea is that the axis of capture might be used to align the output audio at the receiving side to where the microphone at the sending side is located? Is this useful? What happens if this microphone is not in a very good location anyway? It seems to me that the “best” audio rendering direction at the receiving side would be perpendicular to the display, for a given video capture. In other words, microphone types and their physical location should be a matter for the sending side only, the only criterion being that they should be able to be associated with any possible video capture in that room, and provide a high signal to noise ratio for sending to the receiving side. This would cover the use of a single omni, multiple directional, headsets, beam-steering arrays etc.

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 29 Apr 2014 21:51:44 -0400

On Tue, Apr 29, 2014 at 7:50 PM, Paul Coverdale <[email-masked]>wrote:

> *[Paul Coverdale] * I’m not aware of such a technique either, but are you
> saying that the idea is that the axis of capture might be used to align the
> output audio at the receiving side to where the microphone at the sending
> side is located? Is this useful? What happens if this microphone is not in
> a very good location anyway? It seems to me that the “best” audio rendering
> direction at the receiving side would be perpendicular to the display, for
> a given video capture. In other words, microphone types and their physical
> location should be a matter for the sending side only, the only criterion
> being that they should be able to be associated with any possible video
> capture in that room, and provide a high signal to noise ratio for sending
> to the receiving side. This would cover the use of a single omni, multiple
> directional, headsets, beam-steering arrays etc.
>
>
>
>
>
>   I was thinking that if you had a way to align the audio capture axis in
sender and receiver, that you would have the audio analog of a
line-of-sight.  I don't see any way to use the audio capture axis
information without such a method.

I agree that for most systems the rendering axis of the speakers will be
perpendicular to their displays, and that it would be sensible for senders
to take that into account when they design their audio captures.

I also agree that microphone types and physical location need be a matter
for the sending side only (and that loudspeaker types and loudspeaker
locations also need to be a matter for the receiving side only).  There is
a price for that design constraint - if you have full control over the both
sender and receiver equipment you can design a more convincing sense of
presence (both for audio and for video).  But that's a price worth paying
to get broad interoperability.

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 30 Apr 2014 06:28:33 -0700

Hi Christian,
Thanks for elaborating on the idea.  I have some more questions inline.
I see basically what you are trying to do, but I still think it doesn't work in practice any better than what we already have, as I explain below.
Mark

> Sent: Monday, April 28, 2014 10:15 PM
> proposal.  You proposed "in each audio capture we say what VC it relates to."
> What does this mean?  What does it mean for an audio capture to relate to a
> video capture?  What exactly is the producer telling the consumer?  Can an
> audio capture relate to more than one video capture?
> same capture area on a VC and AC. Basically its saying that the audio comes
> from the same "region" as what the video capture does. I think the
> difference is that its not trying to put a mathematical certainty about what
> that region is. Its a way for the producer to say to the consumer if you choose
> video capture A you probably want to choose audio capture B for a good
> experience.

[Duckworth, Mark] If that is the meaning, then would it be better for the provider to advertise a list of audio captures that are "related to" each video capture?  Rather than the other way around?  Maybe I'm not understanding the problem you are trying to solve.

> In terms of whether an audio capture can relate to more than
> one video capture yes I think so. You could have one audio capture for a
> room and three video captures.
> >
> > [Duckworth, Mark] Looking from the consumer point of view, in trying to
> understand the advertisement, what does the consumer know from the
> advertisement?  Suppose the consumer wants to get all three of VC0, VC1,
> VC2, but it only wants a single audio capture.  It should probably ask for AC3,
> but according to the advertisement AC3 doesn't "relate to" VC0, VC1, or VC2.
> [CNG] I agree. If the provider wanted to indicate that AC3 could also be used
> for VC0,VC1,VC2 it could advertise:
>       AC0 (VC0)
>       AC1 (VC2)
>       AC2 (VC1)

[Duckworth, Mark] I agree that  AC3 (the whole room audio) "relates to" all those video captures.  But earlier you said "the audio comes from the same 'region' as what the video capture does."  So now do we have to modify that to say something like "the audio comes from the same region as the video, or a subset of that region, or a superset, or an overlapping region"?  In this example, AC3 covers a region that is a superset of the VC0 region, a superset of the VC1 region, and a superset of the VC2 region.

> consumer wants the single video capture VC5, but it would like to receive
> and render spatial audio.  There is no audio capture with multiple channels,
> so it could ask for AC0, AC1, and AC2 if it knew how they were spatially
> "related to" VC5.  But according to the advertisement, AC0, AC1 and AC2
> aren't "related to" VC5 at all.
> [CNG] So in this case there is no AC3? In that case the advertisement could
> look like:
>       AC0 (VC0,VC5)
>       AC1 (VC2,VC5)
>       AC2 (VC1,VC5)

[Duckworth, Mark] I didn't mean to remove AC3.  That would still be there in this example.  So now since the provider forms it's advertisement without knowing what the consumer wants to do, the provider has to give all this information:
  AC0 relates to (VC0, VC3, VC4, VC5)
  AC1 relates to (VC2, VC3, VC4, VC5)
  AC2 relates to (VC1, VC3, VC4, VC5)
  AC3 relates to (VC0, VC1, VC2, VC3, VC4, VC5)
[Duckworth, Mark] Does this information really help the consumer?  I don't think it is any better than what we already have with area of capture.  With my two different consumer examples here, the first consumer would simply pick AC3 because it appears by itself in a CSE, and the consumer would render it as "whole scene" audio.  It doesn't need any more information than that.  The second consumer has a tougher job.  It would choose AC0, AC1, AC2 because they appear in a CSE.  The tricky part is figuring out how to render them spatially.  With your proposal, I think the consumer has to look at the list of "relates to" video, make assumptions about how each AC relates to those VCs in the list, assuming for example that AC0 is more focused on the region of VC0 rather than the regions of the others, and then associate the VC0 area of capture as a subset of the VC5 area (because it is really receiving VC5) and then associate that with the AC0 rendering area.  It seems to me it is more straightforward for the provider to associate AC0 directly with the same area of capture as VC0 in the first place, as we already have today.

[Duckworth, Mark] For another provider example, suppose the provider also offered mono audio captures AC5 and AC6, capturing the left and right halves of the room.  How would your proposal handle that?  Would it be like this?
  AC5 relates to (VC0, VC1, VC3, VC4, VC5)
  AC6 relates to (VC1, VC2, VC3, VC4, VC5)

> Each audio capture could relate to a particular video or be part of the zoomed
> out video capture if there was no room audio.
> >
> >> Regards, Christian
> >>
> >>>> Perhaps the confusion is that some see Audio Capture area as a
> >>>> means to associate an Audio capture with a video capture as you've
> >>>> shown Mark's example. e.g. The audio spatial information isn't
> >>>> really used for any audio transformation other than associating it
> >>>> with a particular
> >> video stream.
> >> in this simple example, we also have AC0 relates to VC3 (sometimes),
> >> VC4, and VC5 (at least part of it), and so on.  AC3 relates to all of VC1
> through VC5.
> >> I think using area of capture works better than trying to do
> >> something like this.
> > ...snip...
> >

---

From: John Leslie <[email-masked]>
Date: Wed, 30 Apr 2014 11:40:15 -0400

(I realize I'm falling behind in replies -- I'm afraid that will
continue for the rest of this week. I do mean to get things out well
before May 13th.)

Stephen Botzko <[email-masked]> wrote:
>>... In other words, microphone types and their physical location should
>> be a matter for the sending side only, the only criterion being that
>> they should be able to be associated with any possible video capture
>> in that room, and provide a high signal to noise ratio for sending to
>> the receiving side. This would cover the use of a single omni, multiple

   This sounds like hand-waving to me.

> I was thinking that if you had a way to align the audio capture axis
> in sender and receiver, that you would have the audio analog of a
> line-of-sight.

   I can't imagine what such an "analog" would be.

> I don't see any way to use the audio capture axis information without
> such a method.

   Audio capture axis isn't intended to solve the problem I think you're
trying to solve. It's intended to convey the _meaning_ of the microphone
capture.

   If I understand correctly, you're trying to find a way to direct the
listener's attention to the screen of the person talking. This is trivial
if every person speaking has an individual microphone, and nearly
impossible otherwise. The best you can do is compare the relative audio
levels of various microphones to place the person on a line perpendicular
to the line between two microphones; guess from the video where on that
line the person lies; and mix the sound between speakers to match the
position on the screen.

   This may be worth doing, for all I know; but myself, I wouldn't bother.

> to take that into account when they design their audio captures.

   Telepresence system designers can't reasonably "take that into account"
for the wide spectrum of future telepresence systems. Nor do I believe
there's anything we could write in our spec to enable that.

> for the sending side only

   In practice, we will often see a single audio stream from a room, which
is the output of a mixer. To a first approximation, what inputs go into
that mixer are indeed a matter for the sending side only.

   But the actual location, line-of-capture, and directional-pattern of
a microphone whose audio is sent as a separate stream is _critical_ to
intelligent use of that stream. (IMHO, there's plenty of bandwidth
available to typical users of telepresence equipment to send one audio
stream per person expected to talk in addition to the mixer output.)

   Of course, when there is only a single mixer output sent, the tricks
being discussed here won't help anyway...

> (and that loudspeaker types and loudspeaker locations also need to be
> a matter for the receiving side only).

   I don't believe it could ever be helpful to the sender to know the
placement of loudspeakers. The closest we might come is for the sender
to announce to the receiver what placement it _intends_ when it sends
stereo or multi-channel streams.

> There is a price for that design constraint - if you have full control
> over both sender and receiver equipment you can design a more convincing
> sense of presence (both for audio and for video).

   That, IMHO, can only happen if both ends use the same vendor. We're
writing standards for interoperation of different vendors' equipment.

> But that's a price worth paying to get broad interoperability.

   Selecting the same vendor will indeed make economic sense for many
customers -- quite regardless of anything we specify here.

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 30 Apr 2014 12:34:56 -0400

Replying generally to this thread:

When I asked whether axis of capture might be more useful than area of 
capture I was responding to Steven's comment about computing which video 
capture was "closest" to the audio capture. Steven seemed to be 
suggesting that that distance would be computed between the centers of 
the audio and video areas of capture. I was thinking that computing 
which video area of capture an audio axis of capture came closest to. 
Also, ISTM that point of capture and axis of capture are more natural 
attributes to provide for audio than is an area of capture.

This is moot if we instead explicitly indicate associations between 
audio and video captures. (If we do that, I don't think it makes a lot 
of difference whether audio points to video or visa versa.)

More below.

On 4/30/14 11:40 AM, John Leslie wrote:
[snip]
> Stephen Botzko <[email-masked]> wrote:
[snip]
> customers -- quite regardless of anything we specify here.

ISTM that there are some truths here that may be more apparent to some 
than others - that a certain division of responsibility is being made 
between the advertiser and the consumer, and a limited ability to share 
information via the advertisement. This limits how faithful the 
rendition can be.

I think we all recognize that if the equipment in two rooms is very 
inconsistent then it may be impossible to get a really good rendering.

But it is also the case that we could have two rooms that are quite 
different, but both very capable. If each had total knowledge of the 
other, and they could do arbitrary negotiation, then the advertiser 
might be able to construct (using the equipment it has) a set of 
captures that the consumer could process (using equipment it has) those 
captures to provide an excellent rendering.

However, the advertise/configure mechanism won't be sufficient to allow 
that to happen in an optimal way in all cases. We are making tradeoffs 
to simplify the process and make it practical to implement and deploy. 
It is unclear *to me* how are we are ending up from the ideal.

I *hope* that what we provide is sufficient so that implementations 
don't need to employ different algorithms when interoperating with known 
and unknown equipment.

	Thanks,
	Paul

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 30 Apr 2014 22:40:59 -0400

On Wed, Apr 30, 2014 at 12:34 PM, Paul Kyzivat <[email-masked]>wrote:

> capture was "closest" to the audio capture. Steven seemed to be suggesting
> that that distance would be computed between the centers of the audio and
> video areas of capture. I was thinking that computing which video area of
> capture an audio axis of capture came closest to. Also, ISTM that point of
> capture and axis of capture are more natural attributes to provide for
> audio than is an area of capture.
>
I believe I was actually describing how to pan an audio capture so that
when rendering it is placed at the correct point in the rendered video
scene.  I wasn't attempting to describe how to identify which video capture
is closest to which audio capture.

>
> This is moot if we instead explicitly indicate associations between audio
> and video captures. (If we do that, I don't think it makes a lot of
> difference whether audio points to video or visa versa.)
>
> More below.
>
> information via the advertisement. This limits how faithful the rendition
> can be.
>
> different, but both very capable. If each had total knowledge of the other,
> and they could do arbitrary negotiation, then the advertiser might be able
> to construct (using the equipment it has) a set of captures that the
> consumer could process (using equipment it has) those captures to provide
> an excellent rendering.
>
> that to happen in an optimal way in all cases. We are making tradeoffs to
> simplify the process and make it practical to implement and deploy. It is
> unclear *to me* how are we are ending up from the ideal.
>
> I believe your overall assessment is correct - there are trade-offs here.
 In my view, a simple but adequate framework is more practical than one
that turns rendering into a research problem in acoustics.  Also, we need
to ensure that the information that is sent is something that can be easily
understood and verified by the folks that are doing the signaling
implementations - otherwise the information will almost certainly be wrong.

I do understand that the impact of the compromises might be unclear.
 Hopefully the design team meetings will help clarify.  Though we should
also keep in mind that some amount of telepresence interoperability between
disparate systems already happens.  Advertisement/configures are not being
used, but the basic audio techniques are,

> I *hope* that what we provide is sufficient so that implementations don't
> need to employ different algorithms when interoperating with known and
> unknown equipment.
>
>         Thanks,
>         Paul
>
>

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 30 Apr 2014 23:50:12 -0400

>    I can't imagine what such an "analog" would be.

It's simple enough.  A camera/display pair is aligned to provide a line of
sight between a pair of participants, maintaining eye-contact.  That's
described in the CLUE tutorial we did some time ago.  A microphone and
speaker pair can potentially be configured to do something similar, if you
align the microphone capture axis with the loudspeaker axis in the remote
room.

> trying to solve. It's intended to convey the _meaning_ of the microphone
Why does the rendering system need to know that meaning?  How might it use
that information?  If it doesn't need to know it, why should I take the
trouble to measure it and signal it?

>    This may be worth doing, for all I know; but myself, I wouldn't bother.
(a) it can be done without individual miking
(b) Panning audio captures to align them with the video is worth doing.

>
> Personally I believe it is reasonable for senders to make that particular
assumption about rendering.  If you want to make different assumptions when
you set up your microphone arrangements and capture advertisements, you are
free to do so.  I wasn't proposing that my assumptions be written into a
spec or that they needed to be.

> intelligent use of that stream.

Here we disagree.  I am not understanding why you think any of those things
are critical, or how you are proposing that a renderer or middle box would
use them.
Why, for instance, does a receiver need to know that my system is using
ceiling microphones instead of boundary microphones on the table?

> (IMHO, there's plenty of bandwidth
> stereo or multi-channel streams

I think we are agreeing that loudspeaker placement shouldn't be signaled.

> writing standards for interoperation of different vendors' equipment.
Which was my point. And that means there will be inevitably be some
compromises in the user experience

> Certainly.  But that doesn't mean that those customers don't value or need
interoperability.
In some cases they will be talking to people in a different organization
which chose a different vendor. Also, even a single vendor can market more
than one type of system.  We do.

---

From: Christian Groves <[email-masked]>
Date: Thu, 01 May 2014 14:09:14 +1000

Hello Mark,

Please see my replies below.

Regards, Christian

> [Duckworth, Mark] If that is the meaning, then would it be better for the provider to advertise a list of audio captures that are "related to" each video capture?  Rather than the other way around?…
[CNG] I don't think it matters the way it is defined. video related to 
audio or audio related to video is essentially the same thing. The 
problem I'm trying to solve is how does the media consumer know which 
audio captures to choose when presented with a list of captures.

As I mentioned in an earlier email I think people where seeing the "area 
of capture" in two separate ways. The first way was using the area of 
capture to determine which video captures the audio related to (i.e. 
capture selection). The second was concentrating on how to use the area 
of capture information for audio rendering (i.e. capture rendering). I'm 
focusing on the first, capture selection.

> [Duckworth, Mark] I agree that  AC3 (the whole room audio) "relates to" all those video captures.  But earlier you said "the audio comes from the same 'region' as what the video capture does."  So…
[CNG] I was trying to illustrate the concept when I used the term 
"region". I didn't mean for that to relate to a specific area. I prefer 
"relates to" because that is essentially I'm trying specify. For the 
purposes of capture selection you could specify the relation in terms of 
a mathematical area or you can simply say the captures are related.

> [Duckworth, Mark] Does this information really help the consumer?  I don't think it is any better than what we already have with area of capture.
[CNG] Well it does say to the MC a number of things, for exmaple:
- if you're going to choose VC0 then either AC0 and AC3 would be the 
best choice.
- AC3 is the best choice if you want one audio capture as it relates to 
all video captures.

My proposal was only duplicate what is supported by area of capture in 
terms of relating audio and video captures for capture selection. The 
way the discussion is going it seems that people are dubious about 
specifying capture area for audio. The proposal is trying to get around 
that.
Another option would be to indicate that the capture area when used for 
audio actually relates to a video capture region. However that seems to 
be confusing to people.