Subject: [clue] Requirement on centralized media mixing

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Mon, 20 Jun 2011 14:47:01 -0700

Hi Christer,

I think that you are proposing that CLUE needs a mechanism to pass
information, such as a tag, on "rendering type", for example, binaural,
threshold, etc.
Is that correct?

And that you are not asking for any more from CLUE than simply
identifying the rendering type.
Is that correct?

IF both of these statements are correct, than personally I think it is
fine TO include the information of rendering type, such as "binaural",
by some mechanism, such as a tag, AS LONG AS there is an accepted Use
Case that shows the necessity of communicating the rendering type, such
as "thresholding" or "panning".

How do other people feel about this?  

Thanks,
Allyn

-----Original Message-----
From: Christer Holmberg [mailto:[email-masked]] 
Sent: Monday, June 20, 2011 12:23 PM
To: Allyn Romanow (allyn); Mary Barnes
Cc: [email-masked]; Botzko, Stephen; Bill Mauchly (bmauchly)
Subject: RE: [clue] Requirement on centralized media mixing

Hi Allyn,

>Still trying to get clarification-
>
>I read your description of various types of rendering.
>Still trying to understand what you want from CLUE-
>If we have a tag that says "binaural" or stereo, sub binaural, are we
>done?

Of course we could have a binaural specific SDP attribute, e.g.
a=binaural.

But, again, what we are proposing is a general mechanism, that could
look something like:

a=rendering-type = <value>

<value> could then e.g. be "binaural", but it could also be something
else (we showed a few example in the e-mail).

>Or do you want CLUE to describe how to do some kind of rendering? To
say
>something more about specific rendering functions?

No, CLUE doesn't need to say anything about that. 

There would of course need to be a rendering description associated with
each rendering type value, but CLUE wouldn't define those.

The idea is that an IANA registry is created, where people can register
their rendering type values.

Regards,

Christer

> -----Original Message-----
> From: Christer Holmberg [mailto:[email-masked]]
> Sent: Sunday, June 19, 2011 10:29 PM
> To: Allyn Romanow (allyn); Mary Barnes
> Cc: [email-masked]; Botzko, Stephen; Bill Mauchly (bmauchly)
> Subject: RE: [clue] Requirement on centralized media mixing
>
>
> Hi Allyn,
>
> I appologise that it took some time to reply.
>
> We've been working on some definition/example text (I sent it to the
> list just a few minutes ago), which I hope will clarify some of the
> questions that have been asked.
>
> >Also, what kind of support are you asking for binaural? I believe
it's
> often considered as a subset of stereo. What are you asking for? A
> simple tag that says "binaural", or  the specification
> >of a rendering system for binaural? Precisely what are asking for?
>
> From CLUE, we are only asking for a general mechanism to request and
> indicate different rendering types (see definition in other e-mail).
>
> An example of such rendering type is binaural, BUT we are *NOT* asking
> CLUE to define any binaural specific SDP attributes etc. If such are
> needed, we agree that it needs to be done as a separate task.
>
> Before I try to answer your other questions, please take a look at the
> definition/example e-mail, which I hope will clarify what we mean by
> rendering and rendering type.
>
> Regards,
>
> Christer
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>       Hi Christer -
>
>       In spirit I am enthusiastic about what you are proposing
(meaning
> good quality mobile participation). I'm not clear on how the spirit
> becomes instantiated in our protocol, so I have some questions below J
>
>       Thanks,
>
>       Allyn
>
>
>
>       From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Christer Holmberg
>       Sent: Tuesday, June 14, 2011 7:23 AM
>       To: [email-masked]
>       Subject: [clue] Requirement on centralized media mixing
>
>
>
>
>
>       Hi,
>
>
>
>       During the last interim meeting (May 12th) we discussed a
> requirement proposal
>
>       stating that an endpoint must be able to request central audio
> rendering
>
>       of different predefined formats, including 3D binaural
rendering.
>
>       (Link to proposal: http://www.ietf.org/mail-
> archive/web/clue/current/msg00179.html)
>
>
>
>       The proposal was well received, with the reservation that it
> should not be mandatory
>
>       for the solution to implement centralized 3D binaural rendering,
> i.e.an endpoint should
>
>       be able to request 3D binaural rendering, but had no guaranty
> that such rendering was
>
>       supported by the system.
>
>
>
>       It was decided to look how such requirement could look like.
>
>
>
>       When I look the -03 version of the req draft, Requirement 2c
> states:
>
>
>
>           "The solution MUST NOT preclude the use of binaural audio."
>
>
>
>       I think that requirement is unclear, and it doesn't really give
> any guidance to our work.
>
>
>
>       Due to that, and also due to the fact that rather than talking
> about specific rendering types,
>
>       I would like to propose more general requirement text on the
> negotiation of the media mixing type,
>
>       and the usage of centralized media mixing in the first place.
>
>
>
>       The mechanism would be extendable, so new types can be added in
> future.
>
>
>
>       So, the proposed requirements are:
>
>
>
>       REQ-x:    It MUST be possible to negotiate the usage of
> centralized media mixing. System support of centralized media mixing
is
> optional.
>
>
>
>       REQ-y:    When centralized media mixing is used, it MUST be
> possible to negotiate the type of media rendering provided for each
> media stream received by a client.
>
>        ---
>
>       [ar] I have a few questions/comments:
>
>       1.         The term "negotiate" troubles me as I think it means
a
> particular form of communication that dictates the solution. What is
> being asked for is the receiver to know that it is receiving
> "centralized media mixing". This may not be the end product of a
> negotiation. The same is true of the use of the term negotiate in the
> second requirement. What is needed is a mechanism by which the
receiver
> may know what type of "media rendering" is being received.
>
>            So, I'd rather offer, The solution must support a means for
> identifying a stream which is "centralized media mixing".
>
>       And, When "centralized media mixing" is used, the solution must
> support a means for identifying...
>
>
>
>       2.       Also,  I'm not too sure what "media rendering",  the
> phrase in the second reqmt, represents- SDP and RTP recognize audio
> channels and recognize codecs. Is this what is referred to as "media
> rendering"? Or is it something different? What very specifically?
>
>
>
>       3.       After reading more of the discussion, I realize that I
> don't fully understand what is meant in the first reqmt by
"centralized
> media mixing" or "rendering", which term is used in another email. I
> had thought you meant the usual audio mixing such as discussed in RFC
> 5117.( I figured it was an understood part of the topology so didn't
> need to  be called out in particular. But I don't mind calling it
out.)
> In any case, now I'm not sure what you mean, maybe it is something
> other than typical MCU mixing behavior as described in RFC 5117?
>
>
>
>       4.       As for the discussion of binaural.. here is what it
> seems like to me.. AVP, RFC 3551 specifies  the number of  audio
> channels, what they refer to,  and types of codecs. Binaural is
> neither. As you say, Christer, it's a type of stereo channel.  Steve
> feels that "binaural" must be recognized in SDP, which of course, it
is
> not currently.
>
>
>
>       I'm not entirely sure what needs to be specified about binaural,
> given that it is neither channel -type nor codec-type. What needs to
be
> standardized? This question is probably more for Steve, or both of
you.
> If the receiving device knows it's getting binaural stereo, then it
can
> run some algorithms to improve quality and if it isn't binaural, it
> won't run those algs., or something analogous? If that is the case, I
> don't see what needs to be standardized that isn't already
> standardized... But I'm eager to learn.
>
>
>
>       Thanks-
>
>       Allyn
>
>
>
>       Regards,
>
>
>
>       Christer
>
>
>
>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 20 Jun 2011 18:12:04 -0400

> information, such as a tag, on "rendering type", for example, binaural,

How does this differ from simply making an offer with the audio codec 
and parameters that you want to use?

	Thanks,
	Paul

> a=rendering-type =<value>
>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 21 Jun 2011 10:22:19 +0200

Hi Paul, 

>>information, such as a tag, on "rendering type", for example, 
>>binaural, threshold, etc.
>>Is that correct?
> 
>How does this differ from simply making an offer with the 
>audio codec and parameters that you want to use?

Assuming we would use an SDP element (e.g. an SDP attribute) to indicate the rendering-type, there should be no difference.

Regards,

Christer

> > Case that shows the necessity of communicating the rendering type, 
> > such as "thresholding" or "panning".
> >
> >> Still trying to understand what you want from CLUE- If we 
> have a tag 
> >> that says "binaural" or stereo, sub binaural, are we done?
> >
> > There would of course need to be a rendering description associated 
> > with each rendering type value, but CLUE wouldn't define those.
> >
> > The idea is that an IANA registry is created, where people can 
> > register their rendering type values.
> >
> > Regards,
> >
> > Christer
> >
> >
> >
> >> An example of such rendering type is binaural, BUT we are *NOT* 
> >> asking CLUE to define any binaural specific SDP attributes etc. If 
> >> such are needed, we agree that it needs to be done as a 
> separate task.
> >>
> >> Before I try to answer your other questions, please take a look at 
> >> the definition/example e-mail, which I hope will clarify 
> what we mean 
> >> by rendering and rendering type.
> >>
> >> Regards,
> >>
> >> Christer
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>        Hi Christer -
> >>
> >> becomes instantiated in our protocol, so I have some 
> questions below 
> >> J
> >>
> >>        Thanks,
> >>
> >>        Allyn
> >>
> >>
> >>
> >>        for the solution to implement centralized 3D binaural 
> >> rendering, i.e.an endpoint should
> >>
> >>        I think that requirement is unclear, and it doesn't really 
> >> give any guidance to our work.
> >>
> >>
> >>
> >>        The mechanism would be extendable, so new types can 
> be added 
> >> in future.
> >>
> >>
> >>
> >>             So, I'd rather offer, The solution must 
> support a means 
> >> for identifying a stream which is "centralized media mixing".
> >>
> >>        And, When "centralized media mixing" is used, the solution 
> >> must support a means for identifying...
> >>
> >>
> >>
> >>        I'm not entirely sure what needs to be specified about 
> >> binaural, given that it is neither channel -type nor 
> codec-type. What 
> >> needs to
> > be

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 21 Jun 2011 10:23:09 +0200

Hi Allyn, 

>I think that you are proposing that CLUE needs a mechanism to 
>pass information, such as a tag, on "rendering type", for 
>example, binaural, threshold, etc.
>Is that correct?

Yes.

>And that you are not asking for any more from CLUE than 
>simply identifying the rendering type.
>Is that correct?

Yes. Identifying, per stream, the rendering type(s) that the receiver is willing to receive, and the rendering type(s) eventually associated with the stream.

Regards,

Christer

>IF both of these statements are correct, than personally I 
>think it is fine TO include the information of rendering 
>type, such as "binaural", by some mechanism, such as a tag, 
>AS LONG AS there is an accepted Use Case that shows the 
>necessity of communicating the rendering type, such as 
>"thresholding" or "panning".
>
> 
> >Still trying to understand what you want from CLUE- If we have a tag 
> But, again, what we are proposing is a general mechanism, 
> that could look something like:
> 
> <value> could then e.g. be "binaural", but it could also be 
> something else (we showed a few example in the e-mail).
> 
> There would of course need to be a rendering description 
> associated with each rendering type value, but CLUE wouldn't 
> define those.
> 
> The idea is that an IANA registry is created, where people 
> can register their rendering type values.
> 
> Regards,
> 
> Christer
> 
> 
> 
> questions below J
> be added in

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 21 Jun 2011 08:40:12 -0400

> Assuming we would use an SDP element (e.g. an SDP attribute) to indicate the rendering-type, there should be no difference.

I'm sure I don't fully understand this.
I *gather* that what you are after is a single audio media stream that 
carries a left and right channel (stereo). And that in addition there is 
some specialized audio processing of original sources to determine what 
is presented in the left and right channel.

So, one question is whether that specialized processing is separable 
from the encoding and delivery of the media stream via a codec. (E.g. 
could that specialized processing be used with a variety of codecs, or 
is it intrinsically linked to a particular codec?) The answer to that 
will impact how it is represented in SDP, and probably the sort of work 
that would be required to get it standardized.

But regardless, do you agree that this is fundamentally distinct from 
telepresence, and something that belongs in SDP, one way or another?

	Thanks,
	Paul

>> think it is

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 21 Jun 2011 14:57:09 +0200

Hi Paul, 

>>>> I think that you are proposing that CLUE needs a 
>>>> mechanism to pass information, such as a tag, on "rendering type", for example, 
>> Assuming we would use an SDP element (e.g. an SDP 
>> attribute) to indicate the rendering-type, there should be no 
>> difference.
> 
>I *gather* that what you are after is a single audio media 
>stream that carries a left and right channel (stereo).
>
>And that in addition there is some specialized audio processing 
>of original sources to determine what is presented in the 
>left and right channel.

In the audio binaural case, yes, but it of course depends on the media type etc.

But, again, CLUE would not define what the processing is. 

(CLUE would only define what information is needed when one registers a rendering type, associated with an algorithm, with IANA.)

>So, one question is whether that specialized processing is 
>separable from the encoding and delivery of the media stream 
>via a codec. (E.g. could that specialized processing be used with a variety of 
>codecs, or is it intrinsically linked to a particular codec?) 

It is not linked to a particular codec.

>The answer to that will impact how it is represented in SDP, 
>and probably the sort of work that would be required to get 
>it standardized.
> 
>But regardless, do you agree that this is fundamentally 
>distinct from telepresence, and something that belongs in 
>SDP, one way or another?

What we are discussing now are requirements (eventhough I know we have used SDP terminology in order to better describe the mechanism, and at the moment it seems quite clear that SDP would be the best solution).

The actual definition of the an attribute might then take place elsewhere, e.g. in MMUSIC. But, the CLUE requirements don't need to care about that.

Regards,

Christer

> associated 
> >>>>         When I look the -03 version of the req draft, 
> Requirement 
> >>>> 2c
> >>>> states:
> >>>>
> >>>>
> >>>>

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 21 Jun 2011 09:44:14 -0400

Christer,

What I'm trying to discern is whether there is *anything* about the 
binaural capability that is specific to CLUE.

ISTM that it would be equally likely that a UA with binaural capability 
would want to use that on a point to point audio call without CLUE, or a 
regular audio conference.

I can best make sense of that in the context of codecs and codec 
parameters. But I sense you are viewing it as something else - some 
higher level of audio processing, independent of codec, that also has to 
be negotiated. AFAIK there is currently no such notion. If so, and that 
is what you need, then you should be taking it up with MMUSIC.

	Thanks,
	Paul

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 21 Jun 2011 12:20:44 -0400

I would like to see use cases, particularly for non-binaural proposals.  It
seems to me that a receiver doesn't really care if the stereo it is
receiving is "panned", "plain" or "thresholded", or if it is linear or
non-linear. So use cases motivating those choices would be helpful.

Also, the proposal is for a specific mechanism ("rendering type").  It may
be premature to identify the mechanism in the requirements - since often
there are multiple ways to achieve the desired ends.

Regards
Steve B.

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 21 Jun 2011 20:08:21 +0200

Hi Paul,

>would want to use that on a point to point audio call without CLUE, 

No.

>or a regular audio conference.

I guess it depends on what you mean by "regular". You do need some kind of "virtual room" concept, where you place the different participants, and I don't think you normally have that in "regular" audio conferences.

>I can best make sense of that in the context of codecs and codec parameters. But I sense you are viewing it as something else - some
>higher level of audio processing, independent of codec, that also has to be negotiated. AFAIK there is currently no such notion.

With SDP, you don't only negotiate codecs. You negotiate media streams, identified by the SDP m- line, and parameters (including codecs) associated with those. For example, every SDP attribute has (or, at least should have) defined offer/answer procedures.

And, IF we would use SDP for this, it would most likely be a media level attribute.

Regards,

Christer

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 21 Jun 2011 20:11:44 +0200

Hi,

>I would like to see use cases, particularly for non-binaural proposals.  It seems to me that a receiver doesn't really >care if the stereo it is receiving is "panned", "plain" or "thresholded", or if it is linear or non-linear. So use cases >motivating those choices would be helpful.
>
>Also, the proposal is for a specific mechanism ("rendering type").  It may be premature to identify the mechanism >in the requirements - since often there are multiple ways to achieve the desired ends.

Again, I am not suggesting to specify a specific mechanism - only a mechanism to negotiate the rendering type mechanism.

Yes, I am using binaural as a rendering type EXAMPLE, but you are the one who keep saying that I am trying to specify binaural :)

Regards,

Christer

On Mon, Jun 20, 2011 at 5:47 PM, Allyn Romanow (allyn) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Christer,

Thanks,
Allyn

-----Original Message-----
From: Christer Holmberg [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Monday, June 20, 2011 12:23 PM
To: Allyn Romanow (allyn); Mary Barnes
Cc: [email-masked]<mailto:[email-masked]>; Botzko, Stephen; Bill Mauchly (bmauchly)
Subject: RE: [clue] Requirement on centralized media mixing

Hi Allyn,

say

Regards,

Christer

> From: Christer Holmberg [mailto:[email-masked]<mailto:[email-masked]>]
> Cc: [email-masked]<mailto:[email-masked]>; Botzko, Stephen; Bill Mauchly (bmauchly)
it's
(meaning
>       From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On
>       To: [email-masked]<mailto:[email-masked]>
rendering.
> i.e.an<http://i.e.an> endpoint should
>
is
a
receiver
"centralized
out.)
is
be
you.
can
_______________________________________________
clue mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/clue

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 21 Jun 2011 14:52:44 -0400

Hi Christer

I understand you are advocating a general mechanism, which is why I
especially want to see use cases showing the need for the panned, plain,
thresholded, linear, non-linear rendering types you proposed.  If someone
has other rendering types, then they would be helpful too.

For me the negotiation of "rendering type" is a mechanism.  I am thinking
all we need is number of channels, but I am quite willing to change my view
if the use cases show more is needed.

Regards,
Stephen Botzko

On Tue, Jun 21, 2011 at 2:11 PM, Christer Holmberg <
[email-masked]> wrote:

> Hi,
>
> >I would like to see use cases, particularly for non-binaural proposals.
>  It seems to me that a receiver doesn't really >care if the stereo it is
> non-linear. So use cases >motivating those choices would be helpful.
> >
> be premature to identify the mechanism >in the requirements - since often
> Again, I am not suggesting to specify a specific mechanism - only a
> mechanism to negotiate the rendering type mechanism.
>
>
>
> Yes, I am using binaural as a rendering type EXAMPLE, but you are the one
> who keep saying that I am trying to specify binaural :)
>
>
>
> Regards,
>
>
>
> Christer
>
>
>
> >       From: [email-masked]<mailto:[email-masked]> [mailto:
> [email-masked]<mailto:[email-masked]>] On

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 07:24:34 +0200

Hi Stephen,

>I understand you are advocating a general mechanism, which is why I especially want to see use cases showing the need for the panned, plain, thresholded, linear, non-linear rendering types you 
>proposed.  If someone has other rendering types, then they would be helpful too.
>	
>For me the negotiation of "rendering type" is a mechanism.  I am thinking all we need is number of channels, but I am quite willing to change my view if the use cases show more is needed.

What do you mean by "number of channels"?

In the SDP a=rtpmap attribute you are able to indicate the number of audio channels, but I don't think that helps.

Regards,

Christer

	On Tue, Jun 21, 2011 at 2:11 PM, Christer Holmberg <[email-masked]> wrote:

		Hi,

		

		

		Regards,

		Christer

		Hi Christer,

		

		Thanks,
		Allyn
		
		-----Original Message-----

		
		Hi Allyn,

		

		
		say

		
		
		Regards,
		
		Christer

		
		
		it's
		(meaning

		
		
		rendering.

		is
		a
		receiver
		"centralized
		out.)
		is
		be
		you.
		can

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 22 Jun 2011 07:28:53 -0400

On Wed, Jun 22, 2011 at 1:24 AM, Christer Holmberg <
[email-masked]> wrote:

> thresholded, linear, non-linear rendering types you
> Setting aside binaural for now, I am envisioning that I am receiving three
audio channels, and that CLUE mechanisms are telling me "left, center,
right" for each channel.  I think that's all I need to know in order to
render the audio properly, but I am thinking you don't agree.  That I would
also need to know "panning" vs "plain"???  I don't understand what you are
thinking I would do differently if I knew that, which is why I am wanting
more use cases.

Regards,
>                >I would like to see use cases, particularly for
> non-binaural proposals.  It seems to me that a receiver doesn't really >care
> if the stereo it is receiving is "panned", "plain" or "thresholded", or if
> it is linear or non-linear. So use cases >motivating those choices would be
> helpful.
>                >
>                >Also, the proposal is for a specific mechanism ("rendering
> type").  It may be premature to identify the mechanism >in the requirements
> - since often there are multiple ways to achieve the desired ends.
>
>
>
>
>                Again, I am not suggesting to specify a specific mechanism -
> only a mechanism to negotiate the rendering type mechanism.
>
>
>
>                Yes, I am using binaural as a rendering type EXAMPLE, but
> you are the one who keep saying that I am trying to specify binaural :)
>
>
>
>                Regards,
>
>
>
>                Christer
>
>
>
>
>                On Mon, Jun 20, 2011 at 5:47 PM, Allyn Romanow (allyn) <
> [email-masked]<mailto:[email-masked]>> wrote:
>
>
>                Hi Christer,
>
>                From: Christer Holmberg [mailto:
> [email-masked]<mailto:[email-masked]>]
> something
> associated with
>                > From: Christer Holmberg [mailto:
> [email-masked]<mailto:[email-masked]>]
>                >       From: [email-masked]<mailto:
> [email-masked]> [mailto:[email-masked]<mailto:
> [email-masked]>] On
>
> Requirement 2c

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 14:14:22 +0200

Hi,

>>>I understand you are advocating a general mechanism, which is why I especially want to see use cases showing
>>>the need for the panned, plain, thresholded, linear, non-linear rendering types you
>Setting aside binaural for now, I am envisioning that I am receiving three audio channels, and that
>CLUE mechanisms are telling me "left, center, right" for each channel.  I think that's all I need to
>know in order to render the audio properly, but I am thinking you don't agree.

It depends on WHO is rendering.

An entity receiving all "raw material" can of course do whatever it wants with it. But, in the centralized rendering case the idea is that the rendering is done on behalf of a client, since the client due to different reasons can't do it itself.

>That I would also need to know "panning" vs "plain"??? I don't understand what you are thinking I would do differently if I knew that, which is why I am wanting more use cases.

Maybe there is no need to separate "panning" vs "plain". I will need to look into that (there for sure is a need to separate binaural from other stereo, though).

Other use-cases is which video and/or audio streams are sent towards a client in the first place. E.g. Sending the video of the most active speaker, sending the audio of the 4 most active speakers, etc. I consider that being different rendering types also, because they affect how the output stream is generated.

Regards,

Christer

                               Hi,

                               Regards,

                               Christer

                               Hi Christer,

                               Thanks,
                               Allyn

                               -----Original Message-----

                               Hi Allyn,

                               say

                               Regards,

                               Christer

                               it's
                               (meaning

                               rendering.

                               is
                               a
                               receiver
                               "centralized
                               out.)
                               is
                               be
                               you.
                               can

---

From: Roni Even <[email-masked]>
Date: Wed, 22 Jun 2011 15:30:18 +0300

Hi,

In XCON data model
http://tools.ietf.org/html/draft-ietf-xcon-common-data-model-31#section-4.2.
13 there is a specification of video layout control that describes the
continuous presence layout. There was no requirement for something similar
for audio. I assume that Christer wants to see something similar to that
definition for audio but I am not sure if there is a use case for that,
currently SDP RTPMAP has the encoding parameters that allows the
specification of the number of audio channels which as far as I understand
is sufficient for telepresence systems.

If there is a general need to negotiate not only the number of channels but
also a non codec specific way to negotiate audio mixing parameter, I am not
sure this is a CLUE topic. In which case its support will be part of
interoperability requirement with SIP systems that do not support
telepresence extensions.

Regards

Roni Even

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Stephen Botzko
Sent: Wednesday, June 22, 2011 2:29 PM
To: Christer Holmberg
Cc: [email-masked]
Subject: Re: [clue] Requirement on centralized media mixing

On Wed, Jun 22, 2011 at 1:24 AM, Christer Holmberg
<[email-masked]> wrote:

Hi Stephen,

In the SDP a=rtpmap attribute you are able to indicate the number of audio
channels, but I don't think that helps.

Regards,

Christer

       On Tue, Jun 21, 2011 at 2:11 PM, Christer Holmberg
<[email-masked]> wrote:

               Hi,

               >I would like to see use cases, particularly for non-binaural
proposals.  It seems to me that a receiver doesn't really >care if the
stereo it is receiving is "panned", "plain" or "thresholded", or if it is
linear or non-linear. So use cases >motivating those choices would be
helpful.

               Yes, I am using binaural as a rendering type EXAMPLE, but you
are the one who keep saying that I am trying to specify binaural :)

               Regards,

               Christer

               On Mon, Jun 20, 2011 at 5:47 PM, Allyn Romanow (allyn)
<[email-masked]<mailto:[email-masked]>> wrote:

               Hi Christer,

pass
binaural,
               threshold, etc.
               Is that correct?

simply

               IF both of these statements are correct, than personally I
think it is
               fine TO include the information of rendering type, such as
"binaural",
               by some mechanism, such as a tag, AS LONG AS there is an
accepted Use
               Case that shows the necessity of communicating the rendering
type, such
               as "thresholding" or "panning".

               Thanks,
               Allyn

               -----Original Message-----

               From: Christer Holmberg
[mailto:[email-masked]<mailto:[email-masked]
>]

               Cc: [email-masked]<mailto:[email-masked]>; Botzko, Stephen;
Bill Mauchly (bmauchly)
               Subject: RE: [clue] Requirement on centralized media mixing

               Hi Allyn,

               >If we have a tag that says "binaural" or stereo, sub
binaural, are we

               Of course we could have a binaural specific SDP attribute,
e.g.
               a=binaural.

that could
               look something like:

               say

can register

               Regards,

               Christer

               > From: Christer Holmberg
[mailto:[email-masked]<mailto:[email-masked]
>]

               it's
               (meaning
questions below J

               >       From:
[email-masked]<mailto:[email-masked]>
[mailto:[email-masked]<mailto:[email-masked]>] On

               rendering.
               >       for the solution to implement centralized 3D binaural
rendering,

Requirement 2c
be added in
               is
               a
               receiver
               "centralized
               out.)
               is
               be
               you.
               can

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 22 Jun 2011 10:30:35 -0400

On Wed, Jun 22, 2011 at 8:14 AM, Christer Holmberg <
[email-masked]> wrote:

> especially want to see use cases showing
> >>>For me the negotiation of "rendering type" is a mechanism.  I am
> thinking all we need is number of channels, but I am quite willing to change
> my view if the use cases show more is needed.
> >>
> >>In the SDP a=rtpmap attribute you are able to indicate the number of
> audio channels,
> >>but I don't think that helps.
> >
> audio channels, and that
> An entity receiving all "raw material" can of course do whatever it wants
> with it. But, in the centralized rendering case the idea is that the
> rendering is done on behalf of a client, since the client due to different
> reasons can't do it itself.
>
Even there, if I can receive 3 channels I am not sure why I need to tell the
rendering server anything else.  (I do get why I might want to choose the
audio layout, but am thinking that is a distinct facility).

>
> >That I would also need to know "panning" vs "plain"??? I don't understand
> what you are thinking I would do differently if I knew that, which is why I
> am wanting more use cases.
>
> Maybe there is no need to separate "panning" vs "plain". I will need to
> look into that (there for sure is a need to separate binaural from other
> stereo, though).
>
Although we disagree on whether binaural is a telepresence feature, we agree
that it has value and that its use should [at least] not be precluded.  I am
trying to set it aside, because I think I understand the requirements for
that adequately [for now anyway]. I agree that binaural needs to be
distinguished from normal stereo.  Devices using speakers shouldn't (ideally
must not) get binaural audio; since it will often degrade their audio
quality; devices using stereo earphones would prefer to get it; since it
enhances their audio quality.  And you must not apply the hrtf filter more
than once on the signal path.

Where I remain somewhat confused is on the generalization proposal.  I am
not sure I can define "rendering type" very well, and the examples I've seen
posted are actually confusing me more, rather than less.  I am hoping that
re-focusing on desired behaviors/use cases will be more productive.  If we
capture the behavioral requirements correctly [specifically for the central
server case], then I am sure that we will come up with an appropriate
protocol design.

> Other use-cases is which video and/or audio streams are sent towards a
> client in the first place. E.g. Sending the video of the most active
> speaker, sending the audio of the 4 most active speakers, etc. I consider
> that being different rendering types also, because they affect how the
> output stream is generated.
>
> Regards,
>
> Christer
>
>
>
>
>
>
>
>                               Hi,
>
>
>                               >I would like to see use cases, particularly
> for non-binaural proposals.  It seems to me that a receiver doesn't really
> >care if the stereo it is receiving is "panned", "plain" or "thresholded",
> or if it is linear or non-linear. So use cases >motivating those choices
> would be helpful.
>                               >
>                               >Also, the proposal is for a specific
> mechanism ("rendering type").  It may be premature to identify the mechanism
> >in the requirements - since often there are multiple ways to achieve the
> desired ends.
>
>
>
>
>                               Again, I am not suggesting to specify a
> specific mechanism - only a mechanism to negotiate the rendering type
> mechanism.
>
>
>
>                               Yes, I am using binaural as a rendering type
> EXAMPLE, but you are the one who keep saying that I am trying to specify
> binaural :)
>
>
>
>                               Regards,
>
>
>
>                               Christer
>
>
>
>
>                               On Mon, Jun 20, 2011 at 5:47 PM, Allyn
> Romanow (allyn) <[email-masked]<mailto:[email-masked]>> wrote:
>
>
>                               Hi Christer,
>
>                               > We've been working on some
> definition/example text (I sent it to the

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 22 Jun 2011 10:51:48 -0400

> An entity receiving all "raw material" can of course do whatever it wants with it. But, in the centralized rendering case the idea is that the rendering is done on behalf of a client, since the cli…

There is a distinction here between describing "what you have" or "what 
you want".

AFAIK the intent right now is that what is described is "what I have", 
and "what I can provide". E.g.

- I have some microphones, speakers, cameras, displays in a particular 
arrangement

- I can provide some selection of media streams having content, either 
renditions of particular cameras, microphones, etc. or specific 
combinations of those.

It seems to me that binaural can fit into this model. The room 
configuration suitable to receive binaural involves "speakers" 
(headphones) arranged in a particular way. I guess that then there would 
need be a way to specify that a media stream that a room can provide is 
encoded as a binaural "mix" of a particular set of microphones on the 
source side.

It isn't in principle necessary to "ask" for binaural. Rather what is 
being done is "offer" binaural. But the actual mechanism for doing that 
seems independent of CLUE.

	Thanks,
	Paul

>                                 >I would like to see use cases, particularly for non-binaural proposals.  It seems to me that a receiver doesn't really>care if the stereo it is receiving is "panned", "plain" or "thresholded", or if it is linear or non-linear. So use cases>motivating those choices would be helpful.
>                                 >
>                                 >Also, the proposal is for a specific mechanism ("rendering type").  It may be premature to identify the mechanism>in the requirements - since often there are multiple ways to achieve the desired ends.
>
>
>
>
>                                 On Mon, Jun 20, 2011 at 5:47 PM, Allyn Romanow (allyn)<[email-masked]<mailto:[email-masked]>>  wrote:
>
>
>                                 Hi Christer,
>

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 17:07:54 +0200

Hi Roni,

>In XCON data model http://tools.ietf.org/html/draft-ietf-xcon-common-data-model-31#section-4.2.13 there is a specification of video layout control that
>describes the continuous presence layout. There was no requirement for something similar for audio. I assume that Christer wants to see something similar
>to that definition for audio but I am not sure if there is a use case for that, currently SDP RTPMAP has the encoding parameters that allows the
>specification of the number of audio channels which as far as I understand is sufficient for telepresence systems.< mal>
>
>If there is a general need to negotiate not only the number of channels but also a non codec specific way to negotiate audio mixing parameter, I am not
>sure this is a CLUE topic. In which case its support will be part of interoperability requirement with SIP systems that do not support telepresence
>extensions.

We do have a requirement to interoperate with "single stream devices" (that might still be CLUE aware), and then there is a need to specify source selection, both for audio and video.

In the most simple case, the generated stream towards the device is done simply by switching, but in more advanced cases there will be need for mixing, where source selection migth be needed for the mixing algorithm.

Regards,

Christer

        From: [email-masked] [mailto:[email-masked]] On Behalf Of Stephen Botzko
        Sent: Wednesday, June 22, 2011 2:29 PM
        To: Christer Holmberg
        Cc: [email-masked]
        Subject: Re: [clue] Requirement on centralized media mixing

        On Wed, Jun 22, 2011 at 1:24 AM, Christer Holmberg <[email-masked]> wrote:

        Hi Stephen lass=MsoNormal style='margin-bottom:12.0pt'>

        Setting aside binaural for now, I am envisioning that I am receiving three audio channels, and that CLUE mechanisms are telling me &q uot; for each channel.  I think that's all I need to know in order to render the audio properly, but I am thinking you don't agree.  That I would also need to know "panning" vs "plain"???  I don't understand what you are thinking I would do differently if I knew that, which is why I am wanting more use cases.

                Regards,

                Christer

                               Hi,

                               >I would like to ly for non-binaural proposals.  It seems to me that a receiver doesn't really >care if the stereo it is receiving is "panned", "plain" or "thresholded", or if it is linear or non-linear. So use cases >motivating those choices would be helpful.
                               >

                               Yes, I am using binaural as a rendering type EXAMPLE, but you are the one who keep saying that I am try )

                               Regards,

                               Christer

                               Hi Christer,

                               And that you ar from CLUE than simply
                               identifying the rendering type.
                               Is that correct?

                               Thanks,

                               -----Original Message-----

                               Hi A ;            >Still trying to get clarification-

                               say

                & ;        The idea is that an IANA registry is created, where people can register
                               their rendering type values.

                               Regards,

                               Christer

                               > Cc: [email-masked]>; Botzko, Stephen; Bill Mauchly (bmauchly)
                               > asked.
                               it's
                               > indicate different rendering types (see definition in other e-mail).
                &nbs nbsp;      >
                               > rendering and rendering type.
                    &n    >
                               > Christer
                      & ;  >       Hi Christer -
                               (meaning
                               > becomes instantiated in our protocol, so I have some questions below J
                            bsp;              >

                               >       Subject: [clue] Requirement on central p;              >
                               >
                               >
                               >
                               >
                               >       Hi,
                               >
                               >
                               >
                               >       stating that an endpoint must be able to >               > rendering
                               >
                               rendering.
                               >       The proposal was well received, it
                               > should not be mandatory
                               >

                               >       be able to request 3D binaural rendering, but had no guaranty
                      p;  >
                               >           "The solution MU f binaural audio."
                               >
                               >
                               >
                              &n bsp; I would like to propose more general requirement text on the
                               is
                               > optional.
                               >
                               >
                               >
                               >       REQ-y:    When centralized media mixing i sp;              > possible to negotiate the type of media rendering provided for each
                               a
                               > particular form of communication that dictates the solution. What is< nbsp;        > being asked for is the receiver to know that it is receiving
                               receiver
                               > identifying a ntralized media mixing".
                               >
                  & ;      > rendering"? Or is it something different? What very specifically?
                               "centralized
                               > 5117.( stood part of the topology so didn't
                               out.)
                               > seems like to me.. AVP, RFC 3551 specifies  the number of  audio
                    & ;    > channels, what they refer to,  and types of codecs. Binaural is
                               is
                               > given that it is neither channel -type nor codec-type. What needs to
                    &nbs nbsp;  be
                               you.
                               can
                               > standardized... But I'm eager to learn.
                            p;              >
                               >       Christer
                &nbs nbsp;      >

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Wed, 22 Jun 2011 10:18:50 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of Paul Kyzivat (pkyzivat)
> Sent: Wednesday, June 22, 2011 7:52 AM
> To: [email-masked]
> >>>> I understand you are advocating a general mechanism, which is why
I especially want to see use
> cases showing
thinking all we need is number
> of channels, but I am quite willing to change my view if the use cases
show more is needed.
> >>> In the SDP a=rtpmap attribute you are able to indicate the number
of audio channels,
> >> CLUE mechanisms are telling me "left, center, right" for each
channel.  I think that's all I need
> to
> > An entity receiving all "raw material" can of course do whatever it
wants with it. But, in the
> centralized rendering case the idea is that the rendering is done on
behalf of a client, since the
> client due to different reasons can't do it itself.
> 

Let me refer to this as the raw inputs.

> renditions of particular cameras, microphones, etc. or specific

Exactly, and I can provide multiple different combinations of the raw
inputs. This may include, for example:
(a) all the raw inputs
(b) some subset of the raw inputs
(c) some composition of the raw inputs

The end device may choose to receive (a) or (b) or (c) or perhaps even
(a)+(c), etc.
As for the requirements for CLUE, it is not clear to me that this is
covered currently. I would like to see that it is added.
As for how to do it, SDPCapNeg and the SDP grouping frameworks provide
tools that may help. We may choose to leverage these or not, but I agree
that is not in scope for the requirements.
 
Cheers,
Charles

> >> That I would also need to know "panning" vs "plain"??? I don't
understand what you are thinking I
> would do differently if I knew that, which is why I am wanting more
use cases.
> >
> > Maybe there is no need to separate "panning" vs "plain". I will need
to look into that (there for
> sure is a need to separate binaural from other stereo, though).
> >
> > Other use-cases is which video and/or audio streams are sent towards
a client in the first place.
> E.g. Sending the video of the most active speaker, sending the audio
of the 4 most active speakers,
> etc. I consider that being different rendering types also, because
they affect how the output stream
> is generated.
> proposals.  It seems to me that a receiver doesn't really>care if the
stereo it is receiving is
> "panned", "plain" or "thresholded", or if it is linear or non-linear.
So use cases>motivating those
> choices would be helpful.
> >                                 >
> >                                 >Also, the proposal is for a
specific mechanism ("rendering type").
> It may be premature to identify the mechanism>in the requirements -
since often there are multiple
> ways to achieve the desired ends.
> >
> >
> >
> >
> >                                 Yes, I am using binaural as a
rendering type EXAMPLE, but you are
> the one who keep saying that I am trying to specify binaural :)
> >
> >
> >
> >                                 Regards,
> >
> >
> >
> >                                 Christer
> >
> >
> >
> >
> >                                 On Mon, Jun 20, 2011 at 5:47 PM,
Allyn Romanow
> (allyn)<[email-masked]<mailto:[email-masked]>>  wrote:
> >
> >
> >                                 Hi Christer,
> >
> >                                 IF both of these statements are
correct, than personally I think it
> >                                 by some mechanism, such as a tag, AS
LONG AS there is an accepted
> Use
[mailto:[email-masked]<mailto:christer.holmberg@ericsson
.com>]
> >                                 Cc:
[email-masked]<mailto:[email-masked]>; Botzko, Stephen; Bill
> Mauchly (bmauchly)
> >                                 >If we have a tag that says
"binaural" or stereo, sub binaural, are
> we
> >                                 >Or do you want CLUE to describe how
to do some kind of rendering?
> To
> with
[mailto:[email-masked]<mailto:christer.holmberg@ericsson
.com>]
> >                                 >  Cc:
[email-masked]<mailto:[email-masked]>; Botzko, Stephen; Bill
> Mauchly (bmauchly)
definition/example text (I sent it to
> the
> >                                 >  list just a few minutes ago),
which I hope will clarify some of
> the
> >                                 >  >Also, what kind of support are
you asking for binaural? I
> believe
> >                                 it's
> >                                 >  often considered as a subset of
stereo. What are you asking for?
> A
> >                                 >  >of a rendering system for
binaural? Precisely what are asking
> for?
> >                                 >
> >                                 >   From CLUE, we are only asking
for a general mechanism to request
> and
> >                                 >  indicate different rendering
types (see definition in other e-
> mail).
> asking
> >                                 >  CLUE to define any binaural
specific SDP attributes etc. If such
> are
> >                                 >  Before I try to answer your other
questions, please take a look
> at the
> >                                 >  definition/example e-mail, which
I hope will clarify what we mean
> >                                 >  good quality mobile
participation). I'm not clear on how the
> spirit
> >                                 >  becomes instantiated in our
protocol, so I have some questions
> below J
> >                                 >        stating that an endpoint
must be able to request central
> audio
> >                                 >        The proposal was well
received, with the reservation that
> it
> >                                 >        be able to request 3D
binaural rendering, but had no
> guaranty
> >                                 >  that such rendering was
> >                                 >
> >                                 >        supported by the system.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        It was decided to look how
such requirement could look
> like.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        When I look the -03 version
of the req draft, Requirement
> >                                 >            "The solution MUST NOT
preclude the use of binaural
> audio."
> >                                 >        Due to that, and also due
to the fact that rather than
> talking
> >                                 >        I would like to propose
more general requirement text on
> the
> >                                 >        and the usage of
centralized media mixing in the first
> place.
> >                                 >        The mechanism would be
extendable, so new types can be
> added in
> >                                 >  future.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        So, the proposed
requirements are:
> >                                 >
> >                                 >
> >                                 >
> >                                 >        REQ-x:    It MUST be
possible to negotiate the usage of
> >                                 >  centralized media mixing. System
support of centralized media
> mixing
> >                                 is
> >                                 >  optional.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        REQ-y:    When centralized
media mixing is used, it MUST be
> >                                 >  possible to negotiate the type of
media rendering provided for
> each
> >                                 >        1.         The term
"negotiate" troubles me as I think it
> means
> >                                 a
> >                                 >  particular form of communication
that dictates the solution. What
> >                                 >  negotiation. The same is true of
the use of the term negotiate in
> the
> >                                 >             So, I'd rather offer,
The solution must support a
> means for
> must
> >                                 >        2.       Also,  I'm not too
sure what "media rendering",
> the
> >                                 >  phrase in the second reqmt,
represents- SDP and RTP recognize
> audio
> >                                 >  channels and recognize codecs. Is
this what is referred to as
> "media
> >                                 >        3.       After reading more
of the discussion, I realize
> that I
> >                                 >  media mixing" or "rendering",
which term is used in another
> email. I
> >                                 >  had thought you meant the usual
audio mixing such as discussed in
> RFC
> >                                 >  5117.( I figured it was an
understood part of the topology so
> didn't
> >                                 >  need to  be called out in
particular. But I don't mind calling it
> >                                 out.)
> >                                 >  In any case, now I'm not sure
what you mean, maybe it is
> >                                 >        4.       As for the
discussion of binaural.. here is what
> it
> >                                 >  neither. As you say, Christer,
it's a type of stereo channel.
> Steve
> >                                 >  feels that "binaural" must be
recognized in SDP, which of course,
> it
> >                                 >  given that it is neither channel
-type nor codec-type. What needs
> to
> >                                 be
> >                                 >  standardized? This question is
probably more for Steve, or both
> of
> >                                 you.
> >                                 >  If the receiving device knows
it's getting binaural stereo, then
> it
> >                                 can
> >                                 >  run some algorithms to improve
quality and if it isn't binaural,
> it
> >                                 >  won't run those algs., or
something analogous? If that is the
> case, I

---

From: Roni Even <[email-masked]>
Date: Wed, 22 Jun 2011 20:44:16 +0300

Hi Christer,
What you describe is not CLUE specific. It applies to multipoint even
without telepresence extension and I am not sure 

> Sent: Wednesday, June 22, 2011 6:08 PM
> To: Roni Even; 'Stephen Botzko'
> Cc: [email-masked]
> >In XCON data model http://tools.ietf.org/html/draft-ietf-xcon-common-
> data-model-31#section-4.2.13 there is a specification of video layout
> control that
> >describes the continuous presence layout. There was no requirement for
> something similar for audio. I assume that Christer wants to see
> something similar
> >to that definition for audio but I am not sure if there is a use case
> for that, currently SDP RTPMAP has the encoding parameters that allows
> the
> >specification of the number of audio channels which as far as I
> understand is sufficient for telepresence systems.< mal>
> >
> >If there is a general need to negotiate not only the number of
> channels but also a non codec specific way to negotiate audio mixing
> parameter, I am not
> telepresence
> We do have a requirement to interoperate with "single stream devices"
> (that might still be CLUE aware), and then there is a need to specify
> source selection, both for audio and video.
> 
> In the most simple case, the generated stream towards the device is
> done simply by switching, but in more advanced cases there will be need
> for mixing, where source selection migth be needed for the mixing
> algorithm.

What you describe is not CLUE specific. It applies to multipoint even
without telepresence extension and I am not sure what is the new requirement
here. In the XCON + SIP case the XCON data model describe the video layout
for the conference but the selection of sources to mix is done using SDP.
What I was saying that there was no requirement in the past to describe to
the MCU the mixing algorithm and if there was one it would be done by
special conference definition like XCON for the whole conference and not in
the protocol.
In your example you want specific mixing mode per participant done by the
MCU. I think that it may be needed for video (personal layout) and if there
is a use case for audio. I am not sure it is CLUE work since it applies also
to current SIP systems.

Roni

> Behalf Of Stephen Botzko
>         >I understand you are advocating a general mechanism, which is
> why I especially want to see use cases showing the need for the panned,
> plain, thresholded, linear, non-linear rendering types you
>         >For me the negotiation of "rendering type" is a mechanism.  I
> am thinking all we need is number of channels, but I am quite willing
> to change my view if the use cases show more is needed.
> 
>         In the SDP a=rtpmap attribute you are able to indicate the
> number of audio channels, but I don't think that helps.
> 
>         Setting aside binaural for now, I am envisioning that I am
> receiving three audio channels, and that CLUE mechanisms are telling me
> &q uot; for each channel.  I think that's all I need to know in order
> to render the audio properly, but I am thinking you don't agree.  That
> I would also need to know "panning" vs "plain"???  I don't understand
> what you are thinking I would do differently if I knew that, which is
> why I am wanting more use cases.
> 
>                 Regards,
> 
>                 Christer
> 
> 
> 
> 
> 
> 
> 
>                        On Tue, Jun 21, 2011 at 2:11 PM, Christer
> Holmberg <[email-masked]> wrote:
> 
> 
>                                Hi,
> 
> 
>                                >I would like to ly for non-binaural
> stereo it is receiving is "panned", "plain" or "thresholded", or if it
> is linear or non-linear. So use cases >motivating those choices would
> be helpful.
>                                >
> mechanism ("rendering type").  It may be premature to identify the
> mechanism >in the requirements - since often there are multiple ways to
> achieve the desired ends.
> 
> 
> 
> 
>                                Yes, I am using binaural as a rendering
> type EXAMPLE, but you are the one who keep saying that I am try )
> 
> 
> 
>                                Regards,
> 
> 
> 
>                                Christer
> 
> 
> 
> 
> LONG AS there is an accepted Use
> [mailto:[email-masked]<mailto:christer.holmberg@ericsso
> n.com>]
> [mailto:[email-masked]<mailto:christer.holmberg@ericsso
> n.com>]
> stereo. What are you asking for? A
> questions, please take a look at the
>                                >       From: clue-
> [email-masked]<mailto:[email-masked]> [mailto:clue-
> [email-masked]<mailto:[email-masked]>] On
> 
> extendable, so new types can be added in
>                                > future.
>                                >
>                                >
>                                >
> support of centralized media mixing
>                                is
>                                > optional.
>                                >
>                                >
>                                >
>                                >       REQ-y:    When centralized media
> mixing i sp;              > possible to negotiate the type of media
> rendering provided for each
>                                > particular form of communication that
> dictates the solution. What is< nbsp;        > being asked for is the
> receiver to know that it is receiving
> represents- SDP and RTP recognize audio
> recognized in SDP, which of course, it
>                                is
>                                > not currently.
>                                >
>                                >
>                                >
> probably more for Steve, or both of
>                                you.

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 20:15:07 +0200

Hi Roni,

> > >In XCON data model
> http://tools.ietf.org/html/draft-ietf-xcon-common-
> > done simply by switching, but in more advanced cases there will be
> > need for mixing, where source selection migth be needed for
> the mixing
> > algorithm.
>
> What you describe is not CLUE specific. It applies to
> multipoint even without telepresence extension and I am not
> sure what is the new requirement here. In the XCON + SIP case
> the XCON data model describe the video layout for the
> conference but the selection of sources to mix is done using SDP.
> What I was saying that there was no requirement in the past
> to describe to the MCU the mixing algorithm and if there was
> one it would be done by special conference definition like
> XCON for the whole conference and not in the protocol.
> In your example you want specific mixing mode per participant
> done by the MCU. I think that it may be needed for video
> (personal layout) and if there is a use case for audio. I am
> not sure it is CLUE work since it applies also to current SIP systems.

Our task is to identify requirements for CLUE.

Based on those requirements, we will choose existing protocol mechanisms, or define new ones (either ourselves, or ask some other WG to do it).

Whether those protocol mechanisms can also be used for non-CLUE systems is irrelevant.

Regards,

Christer

> > why I especially want to see use cases showing the need for the
> > panned, plain, thresholded, linear, non-linear rendering types you
> > receiving three audio channels, and that CLUE mechanisms
> are telling
> > me &q uot; for each channel.  I think that's all I need to know in
> > order to render the audio properly, but I am thinking you
> don't agree.
> > understand what you are thinking I would do differently if I knew
> > that, which is why I am wanting more use cases.
> >
> >                 Regards,
> >
> >                 Christer
> >
> >
> >
> >
> >
> >
> >
> > proposals.  It seems to me that a receiver doesn't really
> >care if the
> specific
> > mechanism >in the requirements - since often there are
> multiple ways
> > to achieve the desired ends.
> >
> >
> >
> >
> > correct, than personally I think it is
> > [email-masked]<mailto:[email-masked]>;
> > Botzko, Stephen; Bill Mauchly (bmauchly)
> >                                Hi A ;            >Still
> trying to get
> > clarification-
> "binaural"
> > or stereo, sub binaural, are we
> >                                >done?
> >
> >                                But, again, what we are
> proposing is a
> > general mechanism, that could
> >                                look something like:
> >
> >
> >                                <value> could then e.g. be
> "binaural",
> > but it could also be something
> >                                else (we showed a few
> example in the e-
> > mail).
> >
> >                                >Or do you want CLUE to
> describe how to
> > do some kind of rendering? To
> >                                say
> >                                No, CLUE doesn't need to say
> anything
> > about that.
> >
> >                                > Cc: [email-masked]>;
> Botzko, Stephen;
> >                                > I appologise that it took
> some time
> > to reply.
> >                                >
> >                                > list just a few minutes
> ago), which I
> > hope will clarify some of the
> >                                > asked.
> >                                >
> >                                > >Also, what kind of
> support are you
> > asking for binaural? I believe
> >                                it's
> >                                > simple tag that says
> "binaural", or
> > the specification
> binaural?
> > Precisely what are asking for?
> >                                >
> >                                > From CLUE, we are only
> asking for a
> > general mechanism to request and
> >                                > An example of such
> rendering type is
> > binaural, BUT we are *NOT* asking
> >                                > CLUE to define any
> binaural specific
> > SDP attributes etc. If such are
> >                                > needed, we agree that it
> needs to be
> > done as a separate task.
> >                                >
> >                                > becomes instantiated in
> our protocol,
> > so I have some questions below J
> >                                >
> >                                >       Thanks,
> >                                >
> >                                >       Allyn
> >                                >
> >                             bsp;              >
> >
> >                                >       From: clue-
> >                                >       Subject: [clue]
> Requirement on
> > central p;              >
> >                                >       During the last
> interim meeting
> > (May 12th) we discussed a
> >                                > requirement proposal
> >                                >
> >                                >       stating that an
> endpoint must be
> > able to >               > rendering
> >                                >
> >                                >       of different
> predefined formats,
> > including 3D binaural
> >                                rendering.
> >                                >
> >                                >       (Link to proposal:
> >                                >       The proposal was
> well received,
> >                                >       be able to request
> 3D binaural
> > rendering, but had no guaranty
> >                                > that such rendering was
> >                                >
> >                                >       supported by the system.
> >                       p;  >
> >                                >
> >                                >
> >                                >       It was decided to
> look how such
> > requirement could look like.
> >                                >
> >                                >
> >                                >
> >                                >       When I look the -03
> version of
> > the req draft, Requirement 2c
> >                                > states:
> >                                >
> >                                >
> >                                >
> >                                >           "The solution MU
> f binaural
> > audio."
> >                                >
> >                                >
> >                                >
> >                                >       Due to that, and
> also due to the
> > fact that rather than talking
> >                                >       REQ-x:    It MUST be
> "negotiate"
> > troubles me as I think it means
> >                                a
> >                                > particular form of
> communication that
> >                                > "centralized media
> mixing". This may
> > not be the end product of a
> >                                > negotiation. The same is
> true of the
> > use of the term negotiate in the
> >                                > second requirement. What
> is needed is
> > a mechanism by which the
> >                                receiver
> >                                >            So, I'd rather
> offer, The
> > solution must support a means for
> >                                > identifying a ntralized
> media mixing".
> >                                >
> >                                >       2.       Also,  I'm
> not too sure
> > what "media rendering",  the
> > this what is referred to as "media
> >                   & ;      > rendering"? Or is it something
> different?
> > What very specifically?
> >                                >
> >                                >
> >                                >
> >                                >       3.       After
> reading more of
> > the discussion, I realize that I
> >                                > don't fully understand
> what is meant
> > in the first reqmt by
> >                                "centralized
> >                                > media mixing" or
> "rendering", which
> > term is used in another email. I
> >                                > had thought you meant the
> usual audio
> > mixing such as discussed in RFC
> >                                > 5117.( stood part of the
> topology so
> > didn't
> >                                > need to  be called out in
> particular.
> discussion
> > of binaural.. here is what it
> >                     & ;    > channels, what they refer to,
> and types
> > of codecs. Binaural is
> >                                > neither. As you say,
> Christer, it's a
> > type of stereo channel.  Steve
> >                                >       I'm not entirely
> sure what needs
> > to be specified about binaural,
> > -type nor codec-type. What needs to
> >                     &nbs nbsp;  be
> > quality and if it isn't binaural, it
> > analogous? If that is the case, I

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 20:25:12 +0200

Hi Charles, 

> receiving three audio channels, and that
> Exactly, and I can provide multiple different combinations of 
> the raw inputs. This may include, for example:
> (a) all the raw inputs
> The end device may choose to receive (a) or (b) or (c) or 
> perhaps even (a)+(c), etc.

A single-stream device would choose to receive (c).

>As for the requirements for CLUE, it is not clear to me that 
>this is covered currently. I would like to see that it is added.

Me too. And, based on the discussions, my understanding is that (a),(b) and (c) are layout options (either source-dependent or source-independent). 

>As for how to do it, SDPCapNeg and the SDP grouping 
>frameworks provide tools that may help. We may choose to 
>leverage these or not, but I agree that is not in scope for 
>the requirements.

I don't think that would help, because it doesn't describe the "raw input combinations" of the media streams.

Regards,

Christer

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 22 Jun 2011 20:32:49 +0200

Hi Paul,

>>>>> I understand you are advocating a general mechanism,
>>>>> which is why I especially want to see use cases showing the need for
>>>>> the panned, plain, thresholded, linear, non-linear rendering types
>>>>> you proposed. If someone has other rendering types, then they
>>>>> would be helpful too.
>>>>>
>>>>> For me the negotiation of "rendering type" is a
>>>>> mechanism. I am thinking all we need is number of channels,
>>>>> but I am quite willing to change my view if the use cases
>>>>> show more is needed.
>>>>
>>>> audio channels, but I don't think that helps.
>>>
>>> receiving
>>> three audio channels, and that CLUE mechanisms are telling
>>> me "left, center, right" for each channel. I think that's all I
>>> need to know in order to render the audio properly, but I am
>>> thinking you don't agree.
>>
>> An entity receiving all "raw material" can of course do
>> whatever it wants with it. But, in the centralized rendering
>> case the idea is that the rendering is done on behalf of a
>> client, since the client due to different reasons can't do it itself.
>
> There is a distinction here between describing "what you
> have" or "what you want".
>
> AFAIK the intent right now is that what is described is "what
> I have", and "what I can provide". E.g.
>
> - I have some microphones, speakers, cameras, displays in a
> particular arrangement
>
> - I can provide some selection of media streams having
> content, either renditions of particular cameras,
> microphones, etc. or specific combinations of those.
>
> It seems to me that binaural can fit into this model. The
> room configuration suitable to receive binaural involves "speakers"
> (headphones) arranged in a particular way. I guess that then
> there would need be a way to specify that a media stream that
> a room can provide is encoded as a binaural "mix" of a
> particular set of microphones on the source side.

The room doesn't need to provide binaural media.

>It isn't in principle necessary to "ask" for binaural. Rather
>what is being done is "offer" binaural. But the actual
>mechanism for doing that seems independent of CLUE.

I am not sure I am following.

A client needs to indicate that it is willing to receive a binaural audio stream, and the MCU, when sending the stream, needs to indicate that it is binaural.

Regards,

Christer

>
>       Thanks,
>       Paul
>
> >> That I would also need to know "panning" vs "plain"??? I
> don't understand what you are thinking I would do differently
> if I knew that, which is why I am wanting more use cases.
> >
> > Maybe there is no need to separate "panning" vs "plain". I
> will need to look into that (there for sure is a need to
> separate binaural from other stereo, though).
> >
> > Other use-cases is which video and/or audio streams are
> sent towards a client in the first place. E.g. Sending the
> video of the most active speaker, sending the audio of the 4
> most active speakers, etc. I consider that being different
> rendering types also, because they affect how the output
> stream is generated.
> >
> > Regards,
> >
> > Christer
> >
> >
> >
> >
> >
> >
> >
> >                                 Hi,
> >
> >
> >                                 >I would like to see use
> cases, particularly for non-binaural proposals.  It seems to
> me that a receiver doesn't really>care if the stereo it is
> receiving is "panned", "plain" or "thresholded", or if it is
> linear or non-linear. So use cases>motivating those choices
> would be helpful.
> >                                 >
> >                                 >Also, the proposal is for
> a specific mechanism ("rendering type").  It may be premature
> to identify the mechanism>in the requirements - since often
> >                                 Again, I am not suggesting
> to specify a specific mechanism - only a mechanism to
> negotiate the rendering type mechanism.
> >
> >
> >
> > rendering type EXAMPLE, but you are the one who keep saying
> that I am
> > trying to specify binaural :)
> >
> >
> >
> >                                 Regards,
> >
> >
> >
> >                                 Christer
> >
> >
> >
> >
> >                                 On Mon, Jun 20, 2011 at
> 5:47 PM, Allyn Romanow
> >                                 I think that you are
> proposing that CLUE needs a mechanism to pass
> >                                 information, such as a tag,
> on "rendering type", for example, binaural,
> >                                 threshold, etc.
> >                                 Is that correct?
> >
> >                                 IF both of these statements
> are correct, than personally I think it is
> >                                 fine TO include the
> information of rendering type, such as "binaural",
> >                                 by some mechanism, such as
> a tag, AS LONG AS there is an accepted Use
> >                                 Case that shows the
> necessity of communicating the rendering type, such
> [mailto:[email-masked]<mailto:christer.holmber
> [email-masked]>]
> "binaural" or stereo, sub binaural, are we
> >                                 >done?
> >
> proposing is a general mechanism, that could
> >                                 look something like:
> >
> "binaural", but it could also be something
> >                                 else (we showed a few
> example in the e-mail).
> >
> >                                 >Or do you want CLUE to
> describe how to do some kind of rendering? To
> >                                 say
> >                                 >something more about
> >                                 There would of course need
> to be a rendering description associated with
> >                                 The idea is that an IANA
> registry is created, where people can register
> [mailto:[email-masked]<mailto:christer.holmber
> [email-masked]>]
> >                                 >  Sent: Sunday, June 19,
> 2011 10:29 PM
> >                                 >  We've been working on
> some definition/example text (I sent it to the
> >                                 >  list just a few minutes
> ago), which I hope will clarify some of the
> support are you asking for binaural? I believe
> >                                 it's
> >                                 >  often considered as a
> subset of stereo. What are you asking for? A
> >                                 >  simple tag that says
> "binaural", or  the specification
> >                                 >  >of a rendering system
> for binaural? Precisely what are asking for?
> >                                 >
> >                                 >   From CLUE, we are only
> asking for a general mechanism to request and
> >                                 >  indicate different
> rendering types (see definition in other e-mail).
> >                                 >
> >                                 >  An example of such
> rendering type is binaural, BUT we are *NOT* asking
> >                                 >  CLUE to define any
> binaural specific SDP attributes etc. If such are
> >                                 >  needed, we agree that it
> needs to be done as a separate task.
> >                                 >
> >                                 >  Before I try to answer
> your other questions, please take a look at the
> >                                 >  definition/example
> e-mail, which I hope will clarify what we mean by
> >                                 >        In spirit I am
> enthusiastic about what you are proposing
> >                                 (meaning
> >                                 >  good quality mobile
> participation). I'm not clear on how the spirit
> >                                 >  becomes instantiated in
> our protocol, so I have some questions below J
> >                                 >
> >                                 >        Thanks,
> >                                 >
> >                                 >        Allyn
> >                                 >
> >                                 >
> >                                 >
> >
> >                                 >        From:
> Requirement on centralized media mixing
> >                                 >
> >                                 >
> >                                 >
> >                                 >
> >                                 >
> >                                 >        Hi,
> >                                 >
> >                                 >
> >                                 >
> >                                 >        During the last
> interim meeting (May 12th) we discussed a
> >                                 >  requirement proposal
> >                                 >
> >                                 >        stating that an
> endpoint must be able to request central audio
> >                                 >  rendering
> >                                 >
> >                                 >        of different
> predefined formats, including 3D binaural
> >                                 rendering.
> >                                 >
> >                                 >        (Link to proposal:
> well received, with the reservation that it
> >                                 >  should not be mandatory
> >                                 >
> >                                 >        for the solution
> to implement centralized 3D binaural rendering,
> >
> 3D binaural rendering, but had no guaranty
> >                                 >  that such rendering was
> >                                 >
> >                                 >        supported by the system.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        It was decided to
> look how such requirement could look like.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        When I look the
> -03 version of the req draft, Requirement 2c
> >                                 >  states:
> >                                 >
> >                                 >
> >                                 >
> >                                 >            "The solution
> MUST NOT preclude the use of binaural audio."
> >                                 >
> >                                 >
> >                                 >
> >                                 >        I think that
> requirement is unclear, and it doesn't really give
> also due to the fact that rather than talking
> >                                 >        I would like to
> propose more general requirement text on the
> >                                 >  negotiation of the media
> mixing type,
> >                                 >
> >                                 >        and the usage of
> centralized media mixing in the first place.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        The mechanism
> would be extendable, so new types can be added in
> >                                 >  future.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        So, the proposed
> requirements are:
> >                                 >
> >                                 >
> >                                 >
> >                                 >        REQ-x:    It MUST
> >                                 >  centralized media
> mixing. System support of centralized media mixing
> >                                 is
> >                                 >  optional.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        REQ-y:    When
> centralized media mixing is used, it MUST be
> >                                 >  possible to negotiate
> the type of media rendering provided for each
> >                                 >  media stream received by
> a client.
> >                                 >
> >                                 >         ---
> >                                 >
> >                                 >        [ar] I have a few
> questions/comments:
> >                                 >
> >                                 >        1.         The
> term "negotiate" troubles me as I think it means
> >                                 a
> >                                 >  particular form of
> communication that dictates the solution. What is
> >                                 >  being asked for is the
> mixing". This may not be the end product of a
> >                                 >  negotiation. The same is
> true of the use of the term negotiate in the
> >                                 >  second requirement. What
> is needed is a mechanism by which the
> >                                 receiver
> >                                 >  may know what type of
> >                                 >             So, I'd
> rather offer, The solution must support a means for
> >                                 >  identifying a stream
> >                                 >        And, When
> "centralized media mixing" is used, the solution must
> >                                 >  support a means for
> identifying...
> >                                 >
> >                                 >
> >                                 >
> >                                 >        2.       Also,
> I'm not too sure what "media rendering",  the
> >                                 >  phrase in the second
> reqmt, represents- SDP and RTP recognize audio
> >                                 >  channels and recognize
> codecs. Is this what is referred to as "media
> >                                 >  rendering"? Or is it
> something different? What very specifically?
> >                                 >
> >                                 >
> >                                 >
> >                                 >        3.       After
> reading more of the discussion, I realize that I
> >                                 >  don't fully understand
> what is meant in the first reqmt by
> >                                 "centralized
> >                                 >  media mixing" or
> "rendering", which term is used in another email. I
> >                                 >  had thought you meant
> the usual audio mixing such as discussed in RFC
> >                                 >  5117.( I figured it was
> an understood part of the topology so didn't
> >                                 >  need to  be called out
> in particular. But I don't mind calling it
> >                                 out.)
> >                                 >  In any case, now I'm not
> sure what you mean, maybe it is something
> >                                 >  other than typical MCU
> mixing behavior as described in RFC 5117?
> >                                 >
> >                                 >
> >                                 >
> >                                 >        4.       As for
> the discussion of binaural.. here is what it
> >                                 >  seems like to me.. AVP,
> >                                 >  channels, what they
> refer to,  and types of codecs. Binaural is
> >                                 >  neither. As you say,
> Christer, it's a type of stereo channel.  Steve
> >                                 >  feels that "binaural"
> must be recognized in SDP, which of course, it
> >                                 is
> >                                 >  not currently.
> >                                 >
> >                                 >
> >                                 >
> >                                 >        I'm not entirely
> sure what needs to be specified about binaural,
> >                                 >  given that it is neither
> channel -type nor codec-type. What needs to
> >                                 be
> >                                 >  standardized? This
> question is probably more for Steve, or both of
> >                                 you.
> >                                 >  If the receiving device
> knows it's getting binaural stereo, then it
> >                                 can
> >                                 >  run some algorithms to
> improve quality and if it isn't binaural, it
> >                                 >  won't run those algs.,
> or something analogous? If that is the case, I
> >                                 >  don't see what needs to