Subject: [clue] Requirement on centralized media mixing

From: Christer Holmberg <[email-masked]>
Date: Tue, 14 Jun 2011 16:22:57 +0200

Hi,

During the last interim meeting (May 12th) we discussed a requirement proposal
stating that an endpoint must be able to request central audio rendering
of different predefined formats, including 3D binaural rendering.
(Link to proposal: http://www.ietf.org/mail-archive/web/clue/current/msg00179.html)

The proposal was well received, with the reservation that it should not be mandatory
for the solution to implement centralized 3D binaural rendering, i.e.an endpoint should
be able to request 3D binaural rendering, but had no guaranty that such rendering was
supported by the system.

It was decided to look how such requirement could look like.

When I look the -03 version of the req draft, Requirement 2c states:

    "The solution MUST NOT preclude the use of binaural audio."

I think that requirement is unclear, and it doesn't really give any guidance to our work.

Due to that, and also due to the fact that rather than talking about specific rendering types,
I would like to propose more general requirement text on the negotiation of the media mixing type,
and the usage of centralized media mixing in the first place.

The mechanism would be extendable, so new types can be added in future.

So, the proposed requirements are:

REQ-x:    It MUST be possible to negotiate the usage of centralized media mixing. System support of centralized media mixing is optional.

REQ-y:    When centralized media mixing is used, it MUST be possible to negotiate the type of media rendering provided for each media stream received by a client.

Regards,

Christer

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 14 Jun 2011 10:52:44 -0400

If some other WG (for instance MMUSIC) defined a mechanism for general use
of binaural audio for SIP/RTP, then I would fully support requirements that
such a facility should be enabled in CLUE.

But I do not think that CLUE is the right place to invent negotiation of
binaural audio.  Telepresence system users generally do not use headsets.

Regards,
Stephen Botzko

On Tue, Jun 14, 2011 at 10:22 AM, Christer Holmberg <
[email-masked]> wrote:

> (Link to proposal: *
> http://www.ietf.org/mail-archive/web/clue/current/msg00179.html*<http://www.ietf.org/mail-archive/web/clue/current/msg00179.html>
> )
>
> for the solution to implement centralized 3D binaural rendering, i.e.anendpoint should
> REQ-x:    It MUST be possible to negotiate the usage of centralized media
> mixing. System support of centralized media mixing is optional.
>
> REQ-y:    When centralized media mixing is used, it MUST be possible to
> negotiate the type of media rendering provided for each media stream
> received by a client.
>
> Regards,
>
> Christer
>
>
>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>
>

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 14 Jun 2011 18:10:46 +0200

Hi,

>If some other WG (for instance MMUSIC) defined a mechanism for general use of binaural audio for SIP/RTP, then
>I would fully support requirements that such a facility should be enabled in CLUE.
>
>But I do not think that CLUE is the right place to invent negotiation of binaural audio.  Telepresence system users
>generally do not use headsets.

No, but users communicating with a telepresence system using other types of devices may.

But, again, the proposed requirements are not specific to binaural audio.

It is about having a generic mechanism for negotiation of centralized media mixing usage, if the system supports it.

Regards,

Christer

On Tue, Jun 14, 2011 at 10:22 AM, Christer Holmberg <[email-masked]<mailto:[email-masked]>> wrote:

Hi,

The proposal was well received, with the reservation that it should not be mandatory
for the solution to implement centralized 3D binaural rendering, i.e.an<http://i.e.an> endpoint should
be able to request 3D binaural rendering, but had no guaranty that such rendering was
supported by the system.

Regards,

Christer

_______________________________________________
clue mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/clue

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 14 Jun 2011 13:04:23 -0400

On Tue, Jun 14, 2011 at 12:10 PM, Christer Holmberg <
[email-masked]> wrote:

> of binaural audio for SIP/RTP, then
> Yes they might.  But CLUE is not the place to create completely new
facilities that are not intended for telepresence systems.

> I don't understand what  "*negotiate the type of media rendering*" means.
Maybe you can describe more precisely what is meant?

> [email-masked]<mailto:[email-masked]>>
> wrote:
>
> Hi,
>
> (Link to proposal:
> http://www.ietf.org/mail-archive/web/clue/current/msg00179.html)
>

---

From: [email-masked]
Date: Wed, 15 Jun 2011 09:28:44 +0800

Christer,

I think this function is meaningful for operating network, such as IMS, 
NGN.

Thanks,

Gao

> REQ-x:    It MUST be possible to negotiate the usage of centralized 
> media mixing. System support of centralized media mixing is optional.
> 
> REQ-y:    When centralized media mixing is used, it MUST be possible
> to negotiate the type of media rendering provided for each media 
> stream received by a client.
> 
> Regards,
> 
> Christer
> 

--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail is solely property of the sender's organization. This mail communication is confidential. Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the originator of the message. Any views expressed in this message are those of the individual sender.
This message has been scanned for viruses and Spam by ZTE Anti-Spam system.

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 15 Jun 2011 12:08:11 +0200

Hi Stephen,
	
>>>If some other WG (for instance MMUSIC) defined a mechanism for 
>>>general use of binaural audio for SIP/RTP, then I would fully support requirements that such a facility should be enabled in CLUE.
>>>
>>>But I do not think that CLUE is the right place to invent negotiation 
>>>of binaural audio.  Telepresence system users generally do not use headsets.
>>	
>Yes they might. But CLUE is not the place to create completely new facilities that are not intended for telepresence systems. 

Being able to connect to telepresence system with non-TP devices is within the scope of the work, and we think it would create great added value if we have a mechanism which allows offering an optimized "near-telepresence experience" also to such devices.

>I don't understand what  "negotiate the type of media rendering" means.  Maybe you can describe more precisely what is meant? 

It's about two things:

First, to be able to find out whether the system supports centralized rendering in the first place. Note that such functionality is optional.

Second, to be able to request centralized rendering, and the type of rendering.

(Solution wise, we might be talking about some SDP attributes in order to do this)

Note that this proposal does not have any impact on the definitions, or on the telepresence "room model".

Regards,

Christer

		
		Hi,

		

		

		

		

		Regards,
		
		Christer

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 15 Jun 2011 06:47:45 -0400

On Wed, Jun 15, 2011 at 6:08 AM, Christer Holmberg <
[email-masked]> wrote:

> >>>general use of binaural audio for SIP/RTP, then I would fully support
> requirements that such a facility should be enabled in CLUE.
> >>>
> Being able to connect to telepresence system with non-TP devices is within
> the scope of the work, and we think it would create great added value if we
> have a mechanism which allows offering an optimized "near-telepresence
> experience" also to such devices.
>
Yes, connecting telepresence systems to legacy devices is within scope.  And
identifying relationships between streams is within scope.

Binaural audio is not in either category.  It is a specific mode of audio
which for some reason has not been standardized.  Telepresence systems do
not use it, and there is no SDP defined for legacy devices to use either
And It is not a stream relationship like left, right.

I am not saying that binaural audio has no value; I think it does.   But I
don't think it is in scope. There is trouble ahead if every shiny new media
idea becomes CLUE work because some hypothetical future non-telepresence
device might want to use it to talk to a telepresence system through a
middle box.

>
> >>It is about having a generic mechanism for negotiation of centralized
> media mixing usage, if the system supports it.
> >
> >I don't understand what  "negotiate the type of media rendering" means.
> First, to be able to find out whether the system supports centralized
> rendering in the first place. Note that such functionality is optional.
>
What is the* type* of rendering?  If this is a general facility not limited
to binaural audio, then I'd like some other "types" to be identified. And I
think we will need  a definition.  As it stands, I have no idea how to tell
if a proposed solution meets this requirement or not.

>                for the solution to implement centralized 3D binaural
> rendering, i.e.an<http://i.e.an> endpoint should
>
>                REQ-x:    It MUST be possible to negotiate the usage of
> centralized media mixing. System support of centralized media mixing is
> optional.
>
>                REQ-y:    When centralized media mixing is used, it MUST be
> possible to negotiate the type of media rendering provided for each media

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 15 Jun 2011 15:28:52 +0200

Hi Stephen,

>>>>>But I do not think that CLUE is the right place to invent 
>>>>>negotiation of binaural audio.  Telepresence system users generally do not use headsets.
>>>>
>>Being able to connect to telepresence system with non-TP devices is 
>>within the scope of the work, and we think it would create great added value if we have a mechanism which allows offering an optimized "near-telepresence experience" also to such devices.
>
>Yes, connecting telepresence systems to legacy devices is within scope.  And identifying relationships between streams is within scope. 
>
>Binaural audio is not in either category. It is a specific mode of 
>audio which for some reason has not been standardized. Telepresence systems do not use it, and there is no SDP defined for legacy devices to use either  And It is not a stream relationship like left, right.
>
>I am not saying that binaural audio has no value; I think it does. But 
>I don't think it is in scope. There is trouble ahead if every shiny new 
>media idea becomes CLUE work because some hypothetical future non-telepresence device might want to use it to talk to a telepresence system through a middle box.

We are not talking about any "hypothetical future non-telepresence device". We are talking about the most widely spread communication device in the world, and we are talking about providing means for optimizing the user experience for such a device. As we see it this is perfectly inline with the CLUE charter.

>What is the type of rendering? If this is a general facility not limited to binaural audio, then I'd like some other "types" to be identified. 

For audio, you could have "mono", and different types of "stereo" (e.g plain, paning, binaural).

Note that we are not suggesting that, within the CLUE work, start to define every possible type, or even specific types. We propose a generic mechanism.

For video, once you have negotiated the usage of centralized rendering, you could negotiate whether the rendered stream is "most active speaker", and different types of layout options. I do realize, however, that we might not be talking about "rendering type" in the same meaning as for audio in that case. It might be more related to "rendering control", and if we want that it should probably be covered in a separate requirement.

>And I think we will need a definition. As it stands, I have no idea how to tell if a proposed solution meets this requirement or not.

As discussed the interim meeting, the task was to come up with suitable wording. I've tried to do that, but of course I am willing to do modifications in order to clarify the text.

Regards,

Christer

________________________________

	From: Stephen Botzko [mailto:[email-masked]] 
	Sent: 15. kesäkuuta 2011 13:48
	To: Christer Holmberg
	Cc: [email-masked]
	Subject: Re: [clue] Requirement on centralized media mixing

	On Wed, Jun 15, 2011 at 6:08 AM, Christer Holmberg <[email-masked]> wrote:

		Hi Stephen,

	Yes, connecting telepresence systems to legacy devices is within scope.  And identifying relationships between streams is within scope. 
	
	Binaural audio is not in either category.  It is a specific mode of audio which for some reason has not been standardized.  Telepresence systems do not use it, and there is no SDP defined for legacy devices to use either  And It is not a stream relationship like left, right.  
	
	I am not saying that binaural audio has no value; I think it does.   But I don't think it is in scope. There is trouble ahead if every shiny new media idea becomes CLUE work because some hypothetical future non-telepresence device might want to use it to talk to a telepresence system through a middle box.

		It's about two things:

	What is the type of rendering?  If this is a general facility not limited to binaural audio, then I'd like some other "types" to be identified. And I think we will need  a definition.  As it stands, I have no idea how to tell if a proposed solution meets this requirement or not.

		Regards,
		
		Christer

		
		               Hi,

		

		

		

		

		               Regards,
		
		               Christer

---

From: <[email-masked]>
Date: Wed, 15 Jun 2011 16:03:09 +0000

Hi,

I agree that these requirements are meaningful for CLUE, and correspond to valid use cases of telepresence experience.

> For video, once you have negotiated the usage of centralized rendering, you could negotiate whether the rendered
> stream is "most active speaker", and different types of layout options. I do realize, however, that we might not be
> talking about "rendering type" in the same meaning as for audio in that case. It might be more related to "rendering
> control", and if we want that it should probably be covered in a separate requirement.

What do you think of the wording "rendering policy"?

Stephane. 

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]] De la part de Christer Holmberg
Envoyé : mercredi 15 juin 2011 15:29
À : Stephen Botzko
Cc : [email-masked]
Objet : Re: [clue] Requirement on centralized media mixing

Hi Stephen,

Regards,

Christer

		Hi Stephen,

	

		It's about two things:

		

		Regards,
		
		Christer

		
		               Hi,

		

		

		

		

		               Regards,
		
		               Christer

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 15 Jun 2011 12:07:40 -0400

On Wed, Jun 15, 2011 at 9:28 AM, Christer Holmberg <
[email-masked]> wrote:

> >>within the scope of the work, and we think it would create great added
> value if we have a mechanism which allows offering an optimized
> "near-telepresence experience" also to such devices.
> >
> >Yes, connecting telepresence systems to legacy devices is within scope.
>  And identifying relationships between streams is within scope.
> >
> >audio which for some reason has not been standardized. Telepresence
> systems do not use it, and there is no SDP defined for legacy devices to use
> either  And It is not a stream relationship like left, right.
> >
> >media idea becomes CLUE work because some hypothetical future
> non-telepresence device might want to use it to talk to a telepresence
> system through a middle box.
>
> We are not talking about any "hypothetical future non-telepresence device".
> We are talking about the most widely spread communication device in the
> world, and we are talking about providing means for optimizing the user
> experience for such a device. As we see it this is perfectly inline with the
> CLUE charter.
>

Then we disagree on our understanding of the charter...

Anyway, SIP doesn't negotiate binaural audio at all, so I don't see how
binaural audio is wide-spread communication.  Also, I suspect you'd want
binaural audio enabled generally in audio conferences, not just for CLUE
devices.  This is not really (IMO) related to multiple streams.  It is more
broadly applicable than that.

> >What is the type of rendering? If this is a general facility not limited
> to binaural audio, then I'd like some other "types" to be identified.
>

> Note that we are not suggesting that, within the CLUE work, start to define
> every possible type, or even specific types. We propose a generic mechanism.
>
> For video, once you have negotiated the usage of centralized rendering, you
> could negotiate whether the rendered stream is "most active speaker", and
> different types of layout options. I do realize, however, that we might not
> be talking about "rendering type" in the same meaning as for audio in that
> case. It might be more related to "rendering control", and if we want that
> it should probably be covered in a separate requirement.
>
> >And I think we will need a definition. As it stands, I have no idea how to
> tell if a proposed solution meets this requirement or not.
>
> As discussed the interim meeting, the task was to come up with suitable
> wording. I've tried to do that, but of course I am willing to do
> modifications in order to clarify the text.
>
> Regards,
>
> Christer
>
>
>
>
>                >>>general use of binaural audio for SIP/RTP, then I would
> fully support requirements that such a facility should be enabled in CLUE.
>                >>>
> negotiation
>                >Yes they might. But CLUE is not the place to create
> completely new facilities that are not intended for telepresence systems.
>
>
>                Being able to connect to telepresence system with non-TP
> devices is within the scope of the work, and we think it would create great
> added value if we have a mechanism which allows offering an optimized
> "near-telepresence experience" also to such devices.
>
>
>        Yes, connecting telepresence systems to legacy devices is within
> scope.  And identifying relationships between streams is within scope.
>
> audio which for some reason has not been standardized.  Telepresence systems
> do not use it, and there is no SDP defined for legacy devices to use either
>        I am not saying that binaural audio has no value; I think it does.
> But I don't think it is in scope. There is trouble ahead if every shiny new
> media idea becomes CLUE work because some hypothetical future
> non-telepresence device might want to use it to talk to a telepresence
> system through a middle box.
>
>
>
>                >>It is about having a generic mechanism for negotiation of
> centralized media mixing usage, if the system supports it.
>                >
>                >I don't understand what  "negotiate the type of media
> rendering" means.  Maybe you can describe more precisely what is meant?
>
>
>                It's about two things:
>
>                First, to be able to find out whether the system supports
> centralized rendering in the first place. Note that such functionality is
> optional.
>
>        What is the type of rendering?  If this is a general facility not
> limited to binaural audio, then I'd like some other "types" to be
> identified. And I think we will need  a definition.  As it stands, I have no
> idea how to tell if a proposed solution meets this requirement or not.
>
>
>
>
>                Note that this proposal does not have any impact on the
> definitions, or on the telepresence "room model".
>
>
>                Regards,
>
>                Christer
>
>
>
>
>
>                               On Tue, Jun 14, 2011 at 10:22 AM, Christer
> Holmberg <[email-masked]<mailto:
> [email-masked]>> wrote:
>
>                               Hi,
>
>                               The proposal was well received, with the
> reservation that it should not be mandatory
>
>                               for the solution to implement centralized 3D
> binaural rendering, i.e.an<http://i.e.an> endpoint should
>
>                               I think that requirement is unclear, and it
> doesn't really give any guidance to our work.
>
>                               Due to that, and also due to the fact that
> rather than talking about specific rendering types,
>                               I would like to propose more general
> requirement text on the negotiation of the media mixing type,
>                               REQ-x:    It MUST be possible to negotiate
> the usage of centralized media mixing. System support of centralized media
> mixing is optional.
>
>                               REQ-y:    When centralized media mixing is
> used, it MUST be possible to negotiate the type of media rendering provided
> for each media stream received by a client.
>
>                               Regards,
>
>                               Christer
>
>
>

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 15 Jun 2011 19:59:42 +0200

Hi Stephen,

>Then we disagree on our understanding of the charter...

The charter states:

      "The WG will create specifications for SIP-based conferencing systems
      to enable communication of information about media streams so that a
      sending system, receiving system, or intermediate system can make
      reasonable decisions about transmitting, selecting, and rendering
      media streams. This enables systems to make choices that optimize user
      experience."

...and what we are proposing is exactly that: namely to provide support for a

rendering type that is aimed for optimizing the user experience for mobile endpoints.

>Anyway, SIP doesn't negotiate binaural audio at all, so I don't see how binaural audio is wide-spread communication.  Also, I suspect you'd want
>binaural audio enabled generally in audio conferences, not just for CLUE devices.  This is not really (IMO) related to multiple streams.  It is more

It is related to interworking with multistream telepresence clients, and to be able to get the best possible experience when a mobile device supports

only a single stream.

Another example: if a mobile user is in a area with very bad radio covereage/low bandwidth, it could request the media to be rendered in mono.

Regards,

Christer

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 15 Jun 2011 15:24:42 -0400

On Wed, Jun 15, 2011 at 1:59 PM, Christer Holmberg <
[email-masked]> wrote:

> >>>>>>>general use of binaural audio for SIP/RTP, then I would fully
> support requirements that such a facility should be enabled in CLUE.
> >>>>>>>
> >>We are not talking about any "hypothetical future non-telepresence
> device". We are talking about the most widely spread communication device in
> the world, and we are talking about providing means for optimizing the user
> experience for such a device. As we see it this is perfectly
> >>inline with the CLUE charter.
> >
Yes, that is the introduction. The charter then goes on to list specifics,
and nothing further on leads me to believe that binaural audio
standardization is in scope.  Defining spatial relationships is in scope,
but binaural audio standardization is about signaling / negotiating the use
of HRTF (Head Related Transfer Functions) filters in the proposed central
mixer, which in itself is not about defining relationships between streams,
etc.  All I am saying is that binaural audio should be taken to MMUSIC.  Not
that it is a bad idea.

> It is related to interworking with multistream telepresence clients, and to
> be able to get the best possible experience when a mobile device supports
>
> only a single stream.
>
>
>
> Another example: if a mobile user is in a area with very bad radio
> covereage/low bandwidth, it could request the media to be rendered in mono.
>
>
>
> Regards,
>
>
>
> Christer
>
>
>
>
>

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Thu, 16 Jun 2011 02:06:47 +0200

My understanding is that it can define new requirements that can be used by telepresence systems, even if those new capabilities may have a more general use.

What may occur is that then the definition of a solution for a general capability may well be assigned to a more appropriate working group.

It is of course perfectly valid to argue that the requirement is inappropriate or not useful enough for telepresence sustems.

Regards

Keith

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Christer Holmberg
> Sent: 15 June 2011 11:08
> To: Stephen Botzko
> Cc: [email-masked]
> the scope of the work, and we think it would create great added value if
> we have a mechanism which allows offering an optimized "near-telepresence
> 		On Tue, Jun 14, 2011 at 10:22 AM, Christer Holmberg
> <[email-masked]<mailto:[email-masked]>>
> wrote:
> 
> 		Hi,
>

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Wed, 15 Jun 2011 21:45:50 -0700

Hi Christer -

In spirit I am enthusiastic about what you are proposing (meaning good
quality mobile participation). I'm not clear on how the spirit becomes
instantiated in our protocol, so I have some questions below J

Thanks,

Allyn

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Christer Holmberg
Sent: Tuesday, June 14, 2011 7:23 AM
To: [email-masked]
Subject: [clue] Requirement on centralized media mixing

Hi,

During the last interim meeting (May 12th) we discussed a requirement
proposal

(Link to proposal:

The proposal was well received, with the reservation that it should not
be mandatory 

for the solution to implement centralized 3D binaural rendering, i.e.an
endpoint should 

be able to request 3D binaural rendering, but had no guaranty that such
rendering was 

supported by the system.

I think that requirement is unclear, and it doesn't really give any
guidance to our work.

Due to that, and also due to the fact that rather than talking about
specific rendering types, 

I would like to propose more general requirement text on the negotiation
of the media mixing type, 

 ---

[ar] I have a few questions/comments:

1.         The term "negotiate" troubles me as I think it means a
particular form of communication that dictates the solution. What is
being asked for is the receiver to know that it is receiving
"centralized media mixing". This may not be the end product of a
negotiation. The same is true of the use of the term negotiate in the
second requirement. What is needed is a mechanism by which the receiver
may know what type of "media rendering" is being received.

     So, I'd rather offer, The solution must support a means for
identifying a stream which is "centralized media mixing".

And, When "centralized media mixing" is used, the solution must support
a means for identifying... 

2.       Also,  I'm not too sure what "media rendering",  the phrase in
the second reqmt, represents- SDP and RTP recognize audio channels and
recognize codecs. Is this what is referred to as "media rendering"? Or
is it something different? What very specifically?

3.       After reading more of the discussion, I realize that I don't
fully understand what is meant in the first reqmt by "centralized media
mixing" or "rendering", which term is used in another email. I had
thought you meant the usual audio mixing such as discussed in RFC 5117.(
I figured it was an understood part of the topology so didn't need to
be called out in particular. But I don't mind calling it out.)  In any
case, now I'm not sure what you mean, maybe it is something other than
typical MCU mixing behavior as described in RFC 5117?

4.       As for the discussion of binaural.. here is what it seems like
to me.. AVP, RFC 3551 specifies  the number of  audio channels, what
they refer to,  and types of codecs. Binaural is neither. As you say,
Christer, it's a type of stereo channel.  Steve feels that "binaural"
must be recognized in SDP, which of course, it is not currently. 

I'm not entirely sure what needs to be specified about binaural, given
that it is neither channel -type nor codec-type. What needs to be
standardized? This question is probably more for Steve, or both of you.
If the receiving device knows it's getting binaural stereo, then it can
run some algorithms to improve quality and if it isn't binaural, it
won't run those algs., or something analogous? If that is the case, I
don't see what needs to be standardized that isn't already
standardized... But I'm eager to learn.

Thanks-

Allyn 

Regards,

Christer

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Wed, 15 Jun 2011 21:48:17 -0700

Hi Gao, 

Would you care to describe more about what you have in mind? What are
you thinking about the relationship between CLUE - a way to specify
multiple streams in a telepresence conference, and network operation?

Best regards,

Allyn

From: [email-masked] [mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Tuesday, June 14, 2011 6:29 PM
To: Christer Holmberg
Cc: [email-masked]
Subject: Re: [clue] Requirement on centralized media mixing

Christer, 

Thanks, 

Gao 

--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail
is solely property of the sender's organization. This mail communication
is confidential. Recipients named above are obligated to maintain
secrecy and are not permitted to disclose the contents of this
communication to others.
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the
originator of the message. Any views expressed in this message are those
of the individual sender.
This message has been scanned for viruses and Spam by ZTE Anti-Spam
system.

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 16 Jun 2011 07:06:08 +0200

Hi Keith, 

>My understanding is that it can define new requirements that 
>can be used by telepresence systems, even if those new 
>capabilities may have a more general use.
> 
>What may occur is that then the definition of a solution for 
>a general capability may well be assigned to a more 
>appropriate working group.

I totally agree. I am not talking about who is going to specify what protocol elements - I am only talking about requirements.

Regards,

Christer

> It is of course perfectly valid to argue that the requirement 
> is inappropriate or not useful enough for telepresence sustems.
> 
> Regards
> 
> Keith
> 
> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> > Of Christer Holmberg
> > Sent: 15 June 2011 11:08
> > To: Stephen Botzko
> > Cc: [email-masked]
> > >>>support
> > >>>negotiation of binaural audio.  Telepresence system 
> users generally 
> > >>>do not use
> > headsets.
> > >>
> > >>No, but users communicating with a telepresence system 
> using other 
> > >>types
> > of devices may.
> > >
> > >>centralized
> rendering" means.
> centralized 
> > Second, to be able to request centralized rendering, and 
> the type of 
> > rendering.
> > 
> > (Solution wise, we might be talking about some SDP 
> attributes in order 
> > to do this)
> > 
> <[email-masked]<mailto:[email-masked]>
> > >
> > wrote:
> > 
> > 		Hi,
> > 
> > 		During the last interim meeting (May 12th) we 
> discussed a 
> > requirement proposal
> > 		stating that an endpoint must be able to 
> request central audio 
> > rendering
> > 		be able to request 3D binaural rendering, but 
> had no guaranty that 
> > such rendering was
> > 		supported by the system.
> > 
> > 		When I look the -03 version of the req draft, 
> Requirement 2c
> > states:
> > 
> > 		The mechanism would be extendable, so new types 
> can be added in 
> > future.
> > 
> > centralized media mixing. System support of centralized 
> media mixing 
> > is optional.
> > 
> used, it MUST be
> > possible to negotiate the type of media rendering provided for each 
> > media stream received by a client.
> > 
> > 		Regards,
> > 
> > 		Christer

---

From: [email-masked]
Date: Thu, 16 Jun 2011 13:11:27 +0800

Hi Allyn,
What I care is about centralized media mixing talked in Christer's mail.
Telcom carrier/operator usually deploy large scale conference systems(such 
as conference AS/MRF in IMS) in their operating network, then sell the 
conference service to their subscribers. The subscribers could be normal 
end users, or enterprise users(by system such as IP Centrex in the 
operating network).
Considering such use cases, I think it would be meaningful to consider 
centralized media mixing function for telepresence conference system, for 
operating network talked above.
Thanks,
Gao 

> Hi Gao, 
> 
> Would you care to describe more about what you have in mind? What 
> are you thinking about the relationship between CLUE – a way to 
> specify multiple streams in a telepresence conference, and network 
operation?
NGN.

---

From: Stephen Botzko <[email-masked]>
Date: Thu, 16 Jun 2011 07:33:27 -0400

>>> {allyn]
I’m not entirely sure what needs to be specified about binaural, given that
it is neither channel –type nor codec-type. What needs to be standardized?
This question is probably more for Steve, or both of you.  If the receiving
device knows it’s getting binaural stereo, then it can run some algorithms
to improve quality and if it isn’t binaural, it won’t run those algs., or
something analogous? If that is the case, I don’t see what needs to be
standardized that isn’t already standardized… But I’m eager to learn.

When you listen to live audio in a real space, the sound on your left side
reaches both of your ears.  But what you hear in your right ear is quite
different from what you hear in the left - this is because your head (and
the shape of your ears) modifies the sound.  Among other things, your head
creates an acoustic "shadow".

If you listen to ordinary stereo audio through earphones, sound in the left
earphone does not reach your the right earphone in any form.  The result is
that the spatial positioning of the audio sound stage is marred.  For
instance,if a singer is center-stage, through earphones she might sound like
she is singing inside your head.

Filtering the stereo sound through a special filter (called the HRTF)
largely eliminates this effect.  The "head-related transfer function"
adjusts the audio being presented to the earphones to something much closer
to what you'd hear in the ordinary room (by modeling the effect of a typical
human head).  Done well, you get a 3D soundstage, similar to what you would
hear naturally. To some degree, how well it works depends on how closely the
HRTF function being used matches your own head and ears.

Of course the HRTF-processed sound is intended to be heard through
earphones.  If you listen to it through normal speakers, the audio might not
sound good.  (There are several variations of HRTF, and some work much
better than others with normal speakers). And some receivers might have
their own approach to creating 3D sound through speakers, and
pre-application of HRTF would interfere with that.  So a receiver that is
rendering through speakers must not receive HRTF-processed audio.

Ordinarily, the HRTF processing would be simply done in the final receiver -
and in that case there would be no negotiation needed.  However, the HTRF
filters are quite complicated, and therefore Ericsson desires to move the
processing to a central server in order to reduce the complexity/power
requirements in the mobile device.

A second reason for doing the HRTF in the server is that the server
potentially has the original audio captures available, which allows the
server to "place" each source prior to mixing.  To take advantage of this in
the mobile device, you need to forward the original streams, which adds
bandwidth.

So this is not about being able to run "special algorithms" if you are
receiving binaural audio.  It is more about making sure HRTF is not
inappropriately applied (for a normal telepresence rooms using
loudspeakers).  Also, some special algorithms (including HRTF) can only be
run if you are NOT receiving binaural audio.from a central server.  One
obvious example - you wouldn't want to run HRTF twice.

The ability to negotiate an audio or video layout is clearly within scope,
and has general value it the group wants to take it on.  For instance, if
you combine video switching of multiple sources with audio
mixing/transcoding, then the receiver is responsible for the video layout
but has no control over the audio layout.  In that case, there are
advantages to allowing the receiver to tell the central audio mixer where it
wants the sources placed on the sound stage.  I have no objection in
principle to adding that kind of negotiation to the requirements, though I
think the group should consider the impact on the schedule.

All I've been trying to say is the negotiation of sender-side HRTF/binaural
audio itself does not appear to be in scope, and that part would be better
done in MMUSIC.  As Keith observes, the charter indicates that we can
redirect stuff like that to the appropriate group, and I'd suggest that we
do that in this case.  I don't see HRTF negotiation as a telepresence
requirement, it is really a mobile application, so I'd prefer to see
Ericsson do this directly.  I also don't think CLUE has a dependency on such
a negotiation, since the development of a protocol to specify audio layout
doesn't really depend on the availability of binaural negotiation. When I
drafted the "not preclude" requirement on binaural audio I was presuming
this kind of outcome.

As far as the appropriate mechanism for binaural negotiation, I'd imagine
this is a new a= parameter in SDP.  It can be applied to any stereo audio,
and is not codec dependent.

Hopefully this helps to clarify, as far as I know it is a correct and fair
summary of what Ericsson has in mind - the ability to negotiate an audio
layout independently from the video layout, the ability to modify such a
layout mid-call, and optionally negotiate the pre-application the HRTF
function (or equivalent binaural processing means) in a central server for
mobile devices.

Stephen Botzko

On Thu, Jun 16, 2011 at 12:45 AM, Allyn Romanow (allyn) <[email-masked]>wrote:

>  Hi Christer –
> quality mobile participation). I’m not clear on how the spirit becomes
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of
> *Christer Holmberg
> *Sent:* Tuesday, June 14, 2011 7:23 AM
> *To:* [email-masked]
> *Subject:* [clue] Requirement on centralized media mixing
>
>
>
>
>
> Hi,
>
>
>
> 1.         The term “negotiate” troubles me as I think it means a
> particular form of communication that dictates the solution. What is being
> asked for is the receiver to know that it is receiving “centralized media
> mixing”. This may not be the end product of a negotiation. The same is true
> of the use of the term negotiate in the second requirement. What is needed
> is a mechanism by which the receiver may know what type of “media rendering”
> is being received.
>
>      So, I’d rather offer, The solution must support a means for
> identifying a stream which is “centralized media mixing”.
>
> And, When “centralized media mixing” is used, the solution must support a
> means for identifying…
>
>
>
> 2.       Also,  I’m not too sure what “media rendering”,  the phrase in
> recognize codecs. Is this what is referred to as “media rendering”? Or is it
> something different? What very specifically?
>
>
>
> 3.       After reading more of the discussion, I realize that I don’t
> fully understand what is meant in the first reqmt by “centralized media
> mixing” or “rendering”, which term is used in another email. I had thought
> you meant the usual audio mixing such as discussed in RFC 5117.( I figured
> it was an understood part of the topology so didn’t need to  be called out
> in particular. But I don’t mind calling it out.)  In any case, now I’m not
> sure what you mean, maybe it is something other than typical MCU mixing
> behavior as described in RFC 5117?
>
>
>
> 4.       As for the discussion of binaural.. here is what it seems like to
> me.. AVP, RFC 3551 specifies  the number of  audio channels, what they refer
> to,  and types of codecs. Binaural is neither. As you say, Christer, it’s a
> type of stereo channel.  Steve feels that “binaural” must be recognized in
> SDP, which of course, it is not currently.
>
>
>
> I’m not entirely sure what needs to be specified about binaural, given that
> it is neither channel –type nor codec-type. What needs to be standardized?
> This question is probably more for Steve, or both of you.  If the receiving
> device knows it’s getting binaural stereo, then it can run some algorithms
> to improve quality and if it isn’t binaural, it won’t run those algs., or
> something analogous? If that is the case, I don’t see what needs to be
> standardized that isn’t already standardized… But I’m eager to learn.
>
>
>
> Thanks—
>
> Allyn
>
>
>
> Regards,
>
>
>
> Christer
>
>
>
>
>

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Sun, 19 Jun 2011 16:48:06 -0700

Hi Christer,

I'm resending my original message to you as I think the questions it
asks are important. I would like to try to get your requirements clear.
I don't think they are clear yet.

Also, as Mary said in her previous email, email discussions on the list
must be 3 days before the virtual meeting in order to be included in the
virtual meeting. Last minute emails won't be included in the virtual
discussion.

I cannot tell if by "centralized media mixing" you are referring to the
normal functions of an MCU, such as described in RTP Topologies, or if
you mean something else? The reason centralized media mixing wasn't
included in the requirements is because it's assumed behavior of an MCU.
You can say you want it in the requirements anyway.. the point is what
you mean by it?

Also, what kind of support are you asking for binaural? I believe it's
often considered as a subset of stereo. What are you asking for? A
simple tag that says "binaural", or  the specification of a rendering
system for binaural? Precisely what are asking for?

I suggest we do not say negotiate, as I think it is unnecessary and we
can use the notion of sending information between senders and receivers,
as is currently specified. Are you all right with the specific
suggestion I made to replace the term "negotiate" as below?

What did you mean by "media rendering", as below?

Thanks,

Allyn

 

Hi Christer -

Thanks,

Allyn

 

Hi,

(Link to proposal: 

supported by the system.

 ---

Thanks-

Allyn 

Regards,

Christer

---

From: Christer Holmberg <[email-masked]>
Date: Mon, 20 Jun 2011 07:28:46 +0200

Hi Allyn,

I appologise that it took some time to reply.

We've been working on some definition/example text (I sent it to the list just a few minutes ago), which I hope will clarify some of the questions that have been asked. 

>Also, what kind of support are you asking for binaural? I believe it's often considered as a subset of stereo. What are you asking for? A simple tag that says "binaural", or  the specification 
>of a rendering system for binaural? Precisely what are asking for?

>From CLUE, we are only asking for a general mechanism to request and indicate different rendering types (see definition in other e-mail).

An example of such rendering type is binaural, BUT we are *NOT* asking CLUE to define any binaural specific SDP attributes etc. If such are needed, we agree that it needs to be done as a separate task.

Before I try to answer your other questions, please take a look at the definition/example e-mail, which I hope will clarify what we mean by rendering and rendering type.

Regards,

Christer

	 

	Hi Christer -

	In spirit I am enthusiastic about what you are proposing (meaning good quality mobile participation). I'm not clear on how the spirit becomes instantiated in our protocol, so I have some questions below J

	Thanks,

	Allyn

	From: [email-masked] [mailto:[email-masked]] On Behalf Of Christer Holmberg
	Sent: Tuesday, June 14, 2011 7:23 AM
	To: [email-masked]
	Subject: [clue] Requirement on centralized media mixing

	Hi,

	supported by the system.

	 ---

	1.         The term "negotiate" troubles me as I think it means a particular form of communication that dictates the solution. What is being asked for is the receiver to know that it is receiving "centralized media mixing". This may not be the end product of a negotiation. The same is true of the use of the term negotiate in the second requirement. What is needed is a mechanism by which the receiver may know what type of "media rendering" is being received.

	     So, I'd rather offer, The solution must support a means for identifying a stream which is "centralized media mixing".

	And, When "centralized media mixing" is used, the solution must support a means for identifying... 

	2.       Also,  I'm not too sure what "media rendering",  the phrase in the second reqmt, represents- SDP and RTP recognize audio channels and recognize codecs. Is this what is referred to as "media rendering"? Or is it something different? What very specifically?

	3.       After reading more of the discussion, I realize that I don't fully understand what is meant in the first reqmt by "centralized media mixing" or "rendering", which term is used in another email. I had thought you meant the usual audio mixing such as discussed in RFC 5117.( I figured it was an understood part of the topology so didn't need to  be called out in particular. But I don't mind calling it out.)  In any case, now I'm not sure what you mean, maybe it is something other than typical MCU mixing behavior as described in RFC 5117?

	4.       As for the discussion of binaural.. here is what it seems like to me.. AVP, RFC 3551 specifies  the number of  audio channels, what they refer to,  and types of codecs. Binaural is neither. As you say, Christer, it's a type of stereo channel.  Steve feels that "binaural" must be recognized in SDP, which of course, it is not currently. 

	I'm not entirely sure what needs to be specified about binaural, given that it is neither channel -type nor codec-type. What needs to be standardized? This question is probably more for Steve, or both of you.  If the receiving device knows it's getting binaural stereo, then it can run some algorithms to improve quality and if it isn't binaural, it won't run those algs., or something analogous? If that is the case, I don't see what needs to be standardized that isn't already standardized... But I'm eager to learn.

	Thanks-

	Allyn 

	Regards,

	Christer

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Mon, 20 Jun 2011 12:06:55 -0700

Hi Christer,

Still trying to get clarification-

I read your description of various types of rendering.
Still trying to understand what you want from CLUE-
If we have a tag that says "binaural" or stereo, sub binaural, are we
done?

Or do you want CLUE to describe how to do some kind of rendering? To say
something more about specific rendering functions?

Or something additional altogether?

Thanks much-

> From: Christer Holmberg [mailto:[email-masked]]
> Sent: Sunday, June 19, 2011 10:29 PM
> To: Allyn Romanow (allyn); Mary Barnes
> Cc: [email-masked]; Botzko, Stephen; Bill Mauchly (bmauchly)
> Subject: RE: [clue] Requirement on centralized media mixing
> 
> 
> Hi Allyn,
> 
> We've been working on some definition/example text (I sent it to the
> list just a few minutes ago), which I hope will clarify some of the
> questions that have been asked.
> 
> simple tag that says "binaural", or  the specification
> From CLUE, we are only asking for a general mechanism to request and
> indicate different rendering types (see definition in other e-mail).
> 
> An example of such rendering type is binaural, BUT we are *NOT* asking
> CLUE to define any binaural specific SDP attributes etc. If such are
> needed, we agree that it needs to be done as a separate task.
> 
> Before I try to answer your other questions, please take a look at the
> definition/example e-mail, which I hope will clarify what we mean by
> rendering and rendering type.
> 
> Regards,
> 
> Christer
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 	Hi Christer -
> 
> 	In spirit I am enthusiastic about what you are proposing
(meaning
> good quality mobile participation). I'm not clear on how the spirit
> becomes instantiated in our protocol, so I have some questions below J
> 
> 	Thanks,
> 
> 	Allyn
> 
> 
> 
> 	2.       Also,  I'm not too sure what "media rendering",  the
> phrase in the second reqmt, represents- SDP and RTP recognize audio
> channels and recognize codecs. Is this what is referred to as "media
> rendering"? Or is it something different? What very specifically?
> 
> 
> 
> 	3.       After reading more of the discussion, I realize that I
> don't fully understand what is meant in the first reqmt by
"centralized
> media mixing" or "rendering", which term is used in another email. I
> had thought you meant the usual audio mixing such as discussed in RFC
> 5117.( I figured it was an understood part of the topology so didn't
> need to  be called out in particular. But I don't mind calling it
out.)
> In any case, now I'm not sure what you mean, maybe it is something
> other than typical MCU mixing behavior as described in RFC 5117?
> 
> 
> 
> 	4.       As for the discussion of binaural.. here is what it
> seems like to me.. AVP, RFC 3551 specifies  the number of  audio
> channels, what they refer to,  and types of codecs. Binaural is
> neither. As you say, Christer, it's a type of stereo channel.  Steve
> feels that "binaural" must be recognized in SDP, which of course, it
is
> not currently.
> 
> 
> 
> 	I'm not entirely sure what needs to be specified about binaural,
> given that it is neither channel -type nor codec-type. What needs to
be

---

From: Christer Holmberg <[email-masked]>
Date: Mon, 20 Jun 2011 21:22:43 +0200

Hi Allyn,

>If we have a tag that says "binaural" or stereo, sub binaural, are we

Of course we could have a binaural specific SDP attribute, e.g. a=binaural.

But, again, what we are proposing is a general mechanism, that could look something like:

a=rendering-type = <value>

<value> could then e.g. be "binaural", but it could also be something else (we showed a few example in the e-mail).

>something more about specific rendering functions?

No, CLUE doesn't need to say anything about that. 

There would of course need to be a rendering description associated with each rendering type value, but CLUE wouldn't define those.

The idea is that an IANA registry is created, where people can register their rendering type values.

Regards,

Christer

(meaning
"centralized
out.)
is
be

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 20 Jun 2011 16:10:58 -0400

What Christer is looking for may already be implicitly present in the 
existing requirements.

REQMT-2a/b/c cover some of this.

REQMT-4 also has some relevance. (The endpoint that wants binaural has 
two speakers for rendering audio. The source may have some different 
number.)

REQMT-10 has some relevance, since the cell phone that wants this has 
limited bandwidth for receiving the audio.

REQMT-13a almost covers part of what Christer is asking for. Its a 
little obscure, since it uses "transcoding" as a synonym for MCU. (And 
there is no definition of transcoding in this document or the 
definitions document - its in the use cases.) And as written it only 
applies to multipoint conferences. But maybe this isn't needed to get at 
what Christer is requesting.

*If* "binaural" support was representable as an audio codec and options 
in SDP, and the MCU and/or peer endpoint is willing/able to negotiate 
support for it, then it seems like it all should "just work" based on 
these requirements.

So I think for the most part this is a matter of accepting the use case, 
and maybe identifying how the use case is satisfied by the requirements. 
And maybe a "little" tweaking of the requirements.

	Thanks,
	Paul

On 6/20/2011 3:22 PM, Christer Holmberg wrote:
> a=rendering-type =<value>
>