Subject: [clue] Requirements - RE: Improving treatment of audio

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 16 Apr 2014 11:08:38 -0700

I think we might not have a common understanding about what the requirements mean, and what are the use cases regarding audio.  Maybe we should discuss this before getting too deep into the solution.

I included some notes below about my interpretation of the requirements.

Regards,
Mark

> -----Original Message-----
> From: clue [mailto:[email-masked]] On Behalf Of Paul Kyzivat
> Sent: Monday, April 14, 2014 11:46 AM
> To: [email-masked]
> Subject: Re: [clue] Improving treatment of audio
> 
> On 4/13/14 7:31 PM, John Leslie wrote:
> > Paul Coverdale <[email-masked]> wrote:
> >>
> >> So, going back to Paul K's request for a concise statement of what
> >> needs to be done with respect to the treatment of audio in CLUE, I'm
> >> not sure what else is really required apart from satisfying REQMT-2
> >> and REQMT-3,
> >
> >     I have to agree with Paul-C here: clue-telepresence-requirements
> > is pretty much set in stone, now that it has entered the RFCed queue.
> > But I don't think it fits Paul-K's criteria of "concise:
> > ]
> > ] REQMT-2: The solution MUST support a description of the spatial
> > ]          arrangement of captured source audio sent in audio streams
> > ]          which enables a satisfactory reproduction at the receiver
> > ]          in a spatially correct manner.  This applies to each site
> > ]          in a point to point or a multipoint meeting and refers to
> > ]          the spatial ordering within a site, not the ordering of
> > ]          channels between sites.
> > ]
> > ]          Use case point to point symmetric, and all use cases,
> > ]          especially heterogeneous.
> > ]
> > ]  REQMT-2a: The solution MUST support a means of preserving
> > ]            the spatial order of audio in the captured
> > ]            scene.  For example, if John sounds as if he is
> > ]            at Susan's right in the captured audio, John
> > ]            voice is also placed at Susan's right in the
> > ]            rendered image.
> > ]
> > ]  REQMT-2b: The solution MUST support a means to identify
> > ]            the number and spatial arrangement of audio
> > ]            channels including monaural, stereophonic
> > ]            (2.0), and 3.0 (left, center, right) audio
> > ]            channels.
> > ]
> > ]  REQMT-2c: The solution MUST support a means to identify
> > ]            the point of capture of individual audio
> > ]            captures in three dimensions.
> > ]
> > ]  REQMT-2d: The solution MUST support a means to identify
> > ]            the area of coverage of individual audio
> > ]            captures in three dimensions.
> > ]
> > ] REQMT-3: The solution MUST enable individual audio streams to be
> > ]          associated with one or more video image captures, and
> > ]          individual video image captures to be associated with one
> > ]          or more audio captures, for the purpose of rendering
> > ]          proper position.
> > ]
> > ]          Use case is point to point symmetric, and all use cases.
> >
> >     Worse, I'm not confident that all of this is even attainable. :^(
> 
> I don't know if they are unattainable. But IIUC they cannot be attained with
> the information we currently have in the fw and data model.
> 
> If we want to give up on attaining some of them, then let's do so, and
> explicitly note that we have. And then ensure that we have sufficient
> information to attain the rest.
> 
> >     Mea culpa, of course -- I should have said that in WGLC. :^( :^(
> >
> >> with the possible exception of including the directional
> >> characteristics of the microphone with the audio capture.
> >
> >     While I'd be happy to see that signaled, I'm really not pushing it
> > for our first spec -- merely hoping we'll get there someday.
> >
> >     There is, alas, an elephant in each room which can render our
> > audio treatment useless -- it's called "echo". :^(
> 
> IMO, if we don't have a solution that can prevent this then we don't have a
> solution worth publishing.
> 
> > ====
> >     Despite being too late, I feel I ought to comment on these REQMTs.
> > Feel free to ignore what follows:
> 
> As I noted above, I think it is important to confront these issues.
> 
> > ] REQMT-2: The solution MUST support a description of the spatial
> > ]          arrangement of captured source audio sent in audio streams
> > ]          which enables a satisfactory reproduction at the receiver
> > ]          in a spatially correct manner.
> >
> >     Other than wondering what "spatially correct means, I view this as
> > attainable, since the responsibily must belong to the receiver, and
> > that only if the sender chooses to specify enough audio sources.
> 
> What do you mean by "attainable"? Can this be achieved using the data
> model we have now? (I don't think so.)
> 
> ISTM that the responsibility is *shared*:
> - the advertiser must provide sufficient sources, *and* sufficient
>    description of those sources to permit the receiver to do the right
>    thing.
> 
> - the receiver must utilize the available information in order to
>    reproduce the audio in a spatially correct manner, within the limits
>    of its available equipment.
> 
> - *we* are responsible for providing sufficient expressiveness in
>    the advertisement and RTP so that the advertiser and receiver can
>    do the above. *That* is the part we need to work on now.

[Duckworth, Mark] I think REQMT-2 means only that audio can be rendered spatially consistent with video, within a scene.  To me this is very simple, like what a DVD player (with home theater receiver) does.  It renders audio and video, spatially consistent with each other (person on left of screen can be heard coming from left side of room).  It can render different audio formats (e.g. stereo, surround) depending on the physical loudspeaker configuration and the audio rendering capability.  Likewise in CLUE, the sender can send one audio capture (could be multiple channels) for a scene, and the receiver can just render it in same manner as the DVD player would.  The receiver could choose different encoding of the audio capture depending on the receiver's capability (different encoding for mono, stereo, 3-channel, for example), and assuming the sender advertises those encodings. (or maybe they are different alternative captures, rather than different encodings of the same capture.  Need to think about this and how it's related to SDP m-line for audio).

[Duckworth, Mark] It gets a bit more complicated when the sender wants to send multiple captures for the same scene, for example different captures that represent different parts of the room.  This is how TIP works, with three separate captures for left, center, and right.  Now we have to identify spatially how those multiple audio captures relate to the video.  This is where the capture area was intended to be used.  But people are confused about that, and have reasons why it won't work well.  Maybe this topic needs more discussion.  TIP is very simple and limited, in that it forces exactly 3 video and 3 audio with one to one correspondence with no flexibility.

> > ]  REQMT-2a: The solution MUST support a means of preserving
> > ]            the spatial order of audio in the captured
> > ]            scene.  For example, if John sounds as if he is
> > ]            at Susan's right in the captured audio, John
> > ]            voice is also placed at Susan's right in the
> > ]            rendered image.
> >
> >     Likewise, this responsibility must belong to the receiver.

[Duckworth, Mark] I think this is very simple, as explained above.  The basic scenario is the media consumer receives one audio capture for room audio (for example stereo) and renders it.  The receiver doesn't have to do anything special, other than know how to decode and render stereo audio.  This works today in typical videoconferencing, and it can work easily the same when using CLUE to spatially arrange multiple video displays for a single scene.

> > ]  REQMT-2b: The solution MUST support a means to identify
> > ]            the number and spatial arrangement of audio
> > ]            channels including monaural, stereophonic
> > ]            (2.0), and 3.0 (left, center, right) audio
> > ]            channels.
> >
> >     This, IMHO, isn't worth doing (although perhaps I don't understand
> > what REQMT-2b means).
> >
> >     The actual layout of loudspeakers should belong entirely to the
> > receiving room. Further, "stereo" doesn't actually define the
> > relationship of the two channels; and "3.0" suffers a similar problem.
> >
> >     If a sender _actually_ sends "stereo" or "3.0" the receiver will
> > have to punt. I can only hope that our actual standard doesn't encourage
> this.
> 
> Sounds like something we need to talk about. Where?

[Duckworth, Mark] Conferencing systems have been using stereo and 3-channel audio for years, without CLUE.  I don't understand the concern.  Like I said above, I'm not sure any more about identifying different formats using the "audio channel format" CLUE capture attribute, or by using information that would already be in the SDP audio m-lines anyway.

> > ]  REQMT-2c: The solution MUST support a means to identify
> > ]            the point of capture of individual audio
> > ]            captures in three dimensions.
> >
> >     Well stated. Adding point on line of capture helps slightly, but
> > only if we have a useful approximation of sensitivity pattern (which I
> > don't believe we can expect to get).

[Duckworth, Mark] We have this now in CLUE as a media capture attribute, but I don't understand how it can be used by the audio receiver as an enhancement to the rendering process.  I don't see a use case for it.

> > ]  REQMT-2d: The solution MUST support a means to identify
> > ]            the area of coverage of individual audio
> > ]            captures in three dimensions.
> >
> >     Poorly stated. :^( _Every_ microphone in a room can cover _every_
> > source of sound in a room. Thus, the "area of coverage" must
> > necessarily be the entire room...

[Duckworth, Mark] I think this requirement was for the use case where you need multiple audio captures for covering an entire scene, like the TIP case.  The "area of capture" attribute was supposed to provide this information, as a generalization of the limited TIP example of just left, center, right.

> >     (It gets worse... much worse!)
> >
> >     Inevitably, some yahoo is going to put one or more loudspeakers in
> > the room. And those will expand the "area of coverage" to include the
> > other rooms those loudspeakers attempt to render.
> >
> >     With delay!!!

[Duckworth, Mark] I think echo cancellation is already well understood in the industry and outside the scope of CLUE.

> OK. But surely the point of capture isn't sufficient by itself.
> *Something* more is required. Can you suggest something that is stated
> better, that would be attainable and useful for our purposes?
> 
> > ] REQMT-3: The solution MUST enable individual audio streams to be
> > ]          associated with one or more video image captures, and
> > ]          individual video image captures to be associated with one
> > ]          or more audio captures, for the purpose of rendering
> > ]          proper position.
> > ]
> > ]          Use case is point to point symmetric, and all use cases.
> >
> >     I accept there are many folks who will insist on doing this. I
> > don't expect to stop them.
> >
> >     But audio streams are intrinsically linked to rooms, not video captures.
> > And the most useful audio streams are linked to microphones "close" to
> > individual people, while video captures will _very_ often try to cover
> > more than one person -- thus video capures will very typically be
> > associated with several audio captures.

[Duckworth, Mark] Since REQMT-3 is about "rendering proper position", I think it doesn't really add any additional requirement that isn't already in REQMT-2.  Maybe REQMT-3 is starting to assume something about the solution, and putting lower level requirement on it?
 
> Are you saying that within a scene the choice of audio captures is unrelated
> to the choice of video captures? (Other than if you configure
> *some* video then you probably also want to configure *some* audio.)
> 
> If so, then does it ever make sense to choose a subset of the advertised
> audio? And if that is true, then what criteria make sense?
> 
> > ====
> >
> >     It will help if more of us pay attention to the relationship of
> > audio in television in movies. I find that they _don't_ track each
> > other too closely.
> >
> >     The audio provides the continuity; the video tantalizes the senses.
> > When the camera switches between two people talking to each other, the
> > audio doesn't switch -- it tries to make you believe you're in the room.
> >
> >     Thus I most sincerely hope that folks won't choose to switch their
> > audio based on who's speaking -- but I won't try to stop them from
> > doing this. (And I must admit that sometimes I _shouldn't_ blame them,
> > because they find themselves sabotaged by audio sources contaminated
> > by echo.)
> 
> I assume you mean when the speakers are in the same room, right?
> 
> Isn't the situation different when we are switching among speakers in
> different rooms?
> 
> > ====
> >
> >     A word on "echo suppression" might help here: this is a
> > well-solved in plain-old-telephony where end-to-end delay can be held
> > constand. We won't be able to do that over the Internet.
> >
> >     An individual room is probably able to cancel the echo in what it
> > _sends_ but a receiving room won't be able to cancel echo in what it
> > receives. :^(
> >
> > ====
> >
> >     That's as much as I'm willing to sqeeze into this email.
> >
> >     Hope _something_ in this helps...
> 
> Some! This is going to take more work.
> 
> 	Thanks,
> 	Paul
> 
> > --
> > John Leslie <[email-masked]>

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 16 Apr 2014 18:59:58 -0400

I generally agree with Mark's points. See my comments inline.

..Paul

>From: clue [mailto:[email-masked]] On Behalf Of Duckworth, Mark
>Sent: Wednesday, April 16, 2014 2:09 PM
>To: Paul Kyzivat; [email-masked]
>Subject: [clue] Requirements - RE: Improving treatment of audio
>
>I think we might not have a common understanding about what the
>requirements mean, and what are the use cases regarding audio.  Maybe we
>should discuss this before getting too deep into the solution.
>
>> >> needs to be done with respect to the treatment of audio in CLUE,
>> >> I'm not sure what else is really required apart from satisfying
>> >> REQMT-2 and REQMT-3,
>> >
>> I don't know if they are unattainable. But IIUC they cannot be
>> attained with the information we currently have in the fw and data
>model.
>>
>> >     While I'd be happy to see that signaled, I'm really not pushing
>> > it for our first spec -- merely hoping we'll get there someday.
>> >
>> IMO, if we don't have a solution that can prevent this then we don't
>> have a solution worth publishing.
>>
>> > ====
>> >     Other than wondering what "spatially correct means, I view this
>> > as attainable, since the responsibily must belong to the receiver,
>> > and that only if the sender chooses to specify enough audio sources.
>>
>[Duckworth, Mark] I think REQMT-2 means only that audio can be rendered
>spatially consistent with video, within a scene.  To me this is very
>simple, like what a DVD player (with home theater receiver) does.  It
>renders audio and video, spatially consistent with each other (person on
>left of screen can be heard coming from left side of room).  It can
>render different audio formats (e.g. stereo, surround) depending on the
>physical loudspeaker configuration and the audio rendering capability.
>Likewise in CLUE, the sender can send one audio capture (could be
>multiple channels) for a scene, and the receiver can just render it in
>same manner as the DVD player would.  The receiver could choose
>different encoding of the audio capture depending on the receiver's
>capability (different encoding for mono, stereo, 3-channel, for
>example), and assuming the sender advertises those encodings. (or maybe
>they are different alternative captures, rather than different encodings
>of the same capture.  Ne  ed to think about this and how it's related to
>SDP m-line for audio).

[PVC] This is my interpretation of REQMT-2, also.

>[Duckworth, Mark] It gets a bit more complicated when the sender wants
>to send multiple captures for the same scene, for example different
>captures that represent different parts of the room.  This is how TIP
>works, with three separate captures for left, center, and right.  Now we
>have to identify spatially how those multiple audio captures relate to
>the video.  This is where the capture area was intended to be used.  But
>people are confused about that, and have reasons why it won't work well.
>Maybe this topic needs more discussion.  TIP is very simple and limited,
>in that it forces exactly 3 video and 3 audio with one to one
>correspondence with no flexibility.

[PVC] I agree that the TIP example is very simple, but for the more general
case of multiple audio and video captures, why can't the Advertisement from
the Provider provide information on the association between them?

>[Duckworth, Mark] I think this is very simple, as explained above.  The
>basic scenario is the media consumer receives one audio capture for room
>audio (for example stereo) and renders it.  The receiver doesn't have to
>do anything special, other than know how to decode and render stereo
>audio.  This works today in typical videoconferencing, and it can work
>easily the same when using CLUE to spatially arrange multiple video
>displays for a single scene.

>> >     This, IMHO, isn't worth doing (although perhaps I don't
>> > understand what REQMT-2b means).
>> >
>[Duckworth, Mark] Conferencing systems have been using stereo and 3-
>channel audio for years, without CLUE.  I don't understand the concern.
>Like I said above, I'm not sure any more about identifying different
>formats using the "audio channel format" CLUE capture attribute, or by
>using information that would already be in the SDP audio m-lines anyway.

>> > only if we have a useful approximation of sensitivity pattern (which
>> > I don't believe we can expect to get).
>
>[Duckworth, Mark] We have this now in CLUE as a media capture attribute,
>but I don't understand how it can be used by the audio receiver as an
>enhancement to the rendering process.  I don't see a use case for it.

[PVC] This may not necessary for satisfying current use cases, but knowing
the spatial relationships of all microphones at the sending end (and their
sensitivities and directional patterns), could allow the receiving end to do
some fancy spatial audio processing. 

>> >     Poorly stated. :^( _Every_ microphone in a room can cover
>> > _every_ source of sound in a room. Thus, the "area of coverage" must
>[Duckworth, Mark] I think this requirement was for the use case where
>you need multiple audio captures for covering an entire scene, like the
>TIP case.  The "area of capture" attribute was supposed to provide this
>information, as a generalization of the limited TIP example of just
>left, center, right.

[PVC] As a practical matter, this requirement which doesn't much sense.

>> >     Inevitably, some yahoo is going to put one or more loudspeakers
>> > in the room. And those will expand the "area of coverage" to include
>> > the other rooms those loudspeakers attempt to render.
>> >
>> >     With delay!!!
>
>[Duckworth, Mark] I think echo cancellation is already well understood
>in the industry and outside the scope of CLUE.

[PVC] I agree!

>> > And the most useful audio streams are linked to microphones "close"
>> > to individual people, while video captures will _very_ often try to
>> > cover more than one person -- thus video capures will very typically
>> > be associated with several audio captures.
>
>[Duckworth, Mark] Since REQMT-3 is about "rendering proper position", I
>think it doesn't really add any additional requirement that isn't
>already in REQMT-2.  Maybe REQMT-3 is starting to assume something about
>the solution, and putting lower level requirement on it?

[PVC] Yes, REQMT-3 just seems to be re-stating REQT-2.

>> Are you saying that within a scene the choice of audio captures is
>> unrelated to the choice of video captures? (Other than if you
>> configure
>> If so, then does it ever make sense to choose a subset of the
>> advertised audio? And if that is true, then what criteria make sense?
>>
>> > ====
>> >
>> > When the camera switches between two people talking to each other,
>> > the audio doesn't switch -- it tries to make you believe you're in
>the room.
>> >
>> >     Thus I most sincerely hope that folks won't choose to switch
>> > their audio based on who's speaking -- but I won't try to stop them
>> > from doing this. (And I must admit that sometimes I _shouldn't_
>> > blame them, because they find themselves sabotaged by audio sources
>> > contaminated by echo.)
>>
>> > well-solved in plain-old-telephony where end-to-end delay can be
>> > held constand. We won't be able to do that over the Internet.
>> >
>> >     An individual room is probably able to cancel the echo in what
>> > it _sends_ but a receiving room won't be able to cancel echo in what
>> > it receives. :^(
>> >
>> > ====
>> >
>_______________________________________________
>clue mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/clue

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 16 Apr 2014 17:23:41 -0700

Hi Paul,
A couple of responses are inline below.
Mark

> From: Paul Coverdale [mailto:[email-masked]]
> Sent: Wednesday, April 16, 2014 7:00 PM
> To: Duckworth, Mark; [email-masked]
> Subject: RE: [clue] Requirements - RE: Improving treatment of audio
> 
 
 ...snip...

> >renders audio and video, spatially consistent with each other (person
> >on left of screen can be heard coming from left side of room).  It can
> >they are different alternative captures, rather than different
> >encodings of the same capture.  Ne  ed to think about this and how it's
> >related to SDP m-line for audio).
> 
> >works, with three separate captures for left, center, and right.  Now
> >we have to identify spatially how those multiple audio captures relate
> >to the video.  This is where the capture area was intended to be used.
> >But people are confused about that, and have reasons why it won't work
> well.
> >Maybe this topic needs more discussion.  TIP is very simple and
> >limited, in that it forces exactly 3 video and 3 audio with one to one

[Duckworth, Mark] The area of capture attribute was intended to provide this association between audio and video captures.  But now using that mechanism is in question.  If somebody has another proposal, we can discuss it.

...snip...

> >> >     Well stated. Adding point on line of capture helps slightly,
> >> > but only if we have a useful approximation of sensitivity pattern
> >> > (which I don't believe we can expect to get).
> >
> >[Duckworth, Mark] We have this now in CLUE as a media capture
> >attribute, but I don't understand how it can be used by the audio
> >receiver as an enhancement to the rendering process.  I don't see a use
> case for it.
> 

[Duckworth, Mark] I wasn't suggesting we remove these attributes.  I wouldn't oppose adding new attributes for audio capture sensitivities and directional patterns either, as long as it isn't mandatory for an advertisement to include them.
 
...snip...