Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt

From: Bo Burman <[email-masked]>
Date: Fri, 1 Jun 2012 17:36:49 +0200

Hi Paul, all,

See inline below.

/Bo B

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Kyzivat
Sent: Thursday, May 31, 2012 8:17 PM
To: [email-masked]
Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt

On 5/31/12 1:51 PM, Robert Hansen wrote:
> Hi Paul,
>
> As you say, CNAME is used for lipsyncby establishing which RTP streams 
> share a common clock.

Is the CNAME used for this always derived from the SSRC, or can it be derived from a CSRC?

[BoB]: CNAME is always related to an SSRC, but that SSRC may very well only appear as CSRC to the entity receiving the RTCP SR.

> However, in multistream there may be a more granular positional 
> association than just this synchronisation. For instance, in a case 
> where an endpoint produces three video streams and one audio stream, 
> all four streams would likely share a common clock and hence can be 
> sychronised. However, at any given time the audio may come from one of 
> three microphones associated with the three positions, and hence at 
> the far end we would want it to be played out in that position.

In such a case, couldn't the provider use three different CSRCs to indicate which mic was being sent? Or use three different SSRCs for that purpose, if necessary to identify the correlation.

[BoB]: It would definitely be possible to make this work. However, it would also de-couple the mics from a sync perspective and it is possible to describe cases when things would break. For example, assume a number of mics that cover parts of the same sound field (e.g. in the same room) that are not totally separated audio-wise (part of audio going into one mic will also go into one or more other mics) and assume they use different CNAMEs. Each CNAME is shared with a corresponding camera. Camera and mic does not necessarily have to map 1:1, there may for example be multiple mics for one camera. Further assume that one or more of the audio streams are subject to network jitter, making it necessary to compensate the playout delay somewhat in some audio jitter buffer. It will still be possible to sync the delayed audio with other audio and video sharing the same CNAME. However, the other CNAMEs in the same room will per definition have no sync relation, resulting in that audio that went into the "delayed mic" will be perceived as an echo since it can also be heard un-delayed from those other mics. That would not happen if all mics sharing the sound field also share CNAME.

> RTCP SDES packets also has the property that they are not received as 
> frequently or in sync with the RTP media packets; this isn't a 
> particular problem for lipsync, as the adjustments required to sync 
> streams are generally small and in the case of audio can be concealed 
> in moments of silence. However, starting to play out audio from one 
> speaker and then moving it a few seconds later when the SDES packet 
> arrives is much more noticeable to users.

I had assumed that the SSRC:CNAME correlation was cached. If so, I would think this wouldn't normally be a problem.

[BoB]: Yes, I would also say that the correlation is cached. It can however be a startup problem if RTP media reaches the receiver before any RTCP SR carrying CNAME is received (say e.g. that SR was lost or just delayed). RFC 6051 Rapid Synchronization of RTP Flows describes a solution to this problem, based on use of RTP Header Extension.

The point of my question is just to see if there is a way to use existing mechanisms rather than invent new ones.

	Thanks,
	Paul

> Rob
>
> On 31/05/2012 18:27, Paul Kyzivat wrote:
>> I'm really happy to see drafts such as this one and the others sent 
>> out at the same time. They really communicate a lot of thinking and 
>> really help to facilitate conversation.
>>
>> I do have some basic questions about the functionality proposed in 
>> this draft. (Note that I'm not well versed in media processing, so I 
>> won't feel bad to be told that my questions make no sense.)
>>
>> IIUC, RTP (together with some basic signaling in SDP) has a mechanism 
>> to specify lip-sync, based on matching CNAME values between an audio 
>> and video stream. Why is the lip-sync mechanism not sufficient for 
>> the consumer to match a particular packet from an audio capture to 
>> the
>> screen(s) that displayed recent video packets?
>>
>> I do see a different but related problem that I don't think has been 
>> addressed yet: how do I ensure that I have selected the proper set of 
>> audio captures that will carry the audio corresponding to the set of 
>> video captures that I have selected?
>>
>> Thanks,
>> Paul
>>
>> On 5/31/12 1:03 AM, Allyn Romanow (allyn) wrote:
>>>
>>>
>>> -----Original Message-----
>>> From: [email-masked] [mailto:[email-masked]]
>>> Sent: Wednesday, May 30, 2012 10:02 PM
>>> To: Allyn Romanow (allyn)
>>> Cc: [email-masked]; Robert Hansen (rohanse2); Brian 
>>> Baldino (bbaldino)
>>> Subject: New Version Notification
>>> fordraft-romanow-clue-audio-rendering-tag-00.txt
>>>
>>> A new version of I-D, draft-romanow-clue-audio-rendering-tag-00.txt
>>> has been successfully submitted by Allyn Romanow and posted to the 
>>> IETF repository.
>>>
>>> Filename: draft-romanow-clue-audio-rendering-tag
>>> Revision: 00
>>> Title: The need for audio rendering tag mechanism in the CLUE 
>>> Framework Creation date: 2012-05-31 WG ID: Individual Submission 
>>> Number of pages: 8
>>>
>>> Abstract:
>>> The purpose of this draft is for discussion in the CLUE working 
>>> group.
>>>
>>> It proposes adding an audio rendering tag to the CLUE framework 
>>> [I-D.ietf-clue-framework], which makes it possible for the consumer 
>>> to correctly render audio with respect to video in a multistream 
>>> video conference. The solution proposed is in partial response to 
>>> CLUE Task #10, Does Framework provide sufficient info for receiver?
>>>
>>>
>>>
>>>
>>> The IETF Secretariat
>>> _______________________________________________
>>> clue mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/clue
>>>
>>
>> _______________________________________________
>> clue mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/clue
>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

_______________________________________________
clue mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/clue

---

From: "Roni Even" <[email-masked]>
Date: Sat, 2 Jun 2012 01:00:29 +0300

Hi,
To further clarify, my assumption is that in the example in section 1 and
for the general case covered in the draft, each of the audio stream from the
different microphones will have a different SSRC coming from a different
sources. This is similar to the three video streams coming from the
different cameras each with its own SSRC.
Roni

> From: Roni Even [mailto:[email-masked]]
> Sent: Thursday, May 31, 2012 11:13 PM
> To: 'Allyn Romanow (allyn)'; 'CLUE'
> Subject: RE: [clue] FW: New Version Notification fordraft-romanow-clue-
> audio-rendering-tag-00.txt
> 
> Hi,
> I think that Magnus has a more general proposal in
> http://tools.ietf.org/id/draft-westerlund-avtext-rtcp-sdes-srcname-
> 00.txt
> Roni Even
> 
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Allyn Romanow (allyn)
> > Sent: Thursday, May 31, 2012 8:04 AM
> > To: CLUE
> > Subject: [clue] FW: New Version Notification fordraft-romanow-clue-
> > audio-rendering-tag-00.txt
> >
> >
> >
> > Subject: New Version Notification fordraft-romanow-clue-audio-
> > rendering-tag-00.txt
> >
> > Title:		 The need for audio rendering tag mechanism in the
> > CLUE Framework
> > Creation date:	 2012-05-31
> > WG ID:		 Individual Submission
> > Number of pages: 8
> >
> > Abstract:

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Sun, 3 Jun 2012 23:54:34 +0200

Related to

> the far end we would want it to be played out in that position.

What if a media provider assigned a unique SDES SRCNAME (as described in
[SRCNAME]) to each video stream, and dynamically assign SRCNAME values
to the audio stream to move it between the video positions. Since all
stream share the same CNAME they will be played out synchronized. 

The initial configuration of audio and video stream will be configured
in CLUE. 

Cheers

-Espen 
[SRCNAME]
http://tools.ietf.org/html/draft-westerlund-avtext-rtcp-sdes-srcname-00

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Bo Burman
Sent: 1. juni 2012 17:37
To: [email-masked]
Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt

Hi Paul, all,

See inline below.

/Bo B

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Paul Kyzivat
Sent: Thursday, May 31, 2012 8:17 PM
To: [email-masked]
Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt

> share a common clock.

Is the CNAME used for this always derived from the SSRC, or can it be
derived from a CSRC?

[BoB]: CNAME is always related to an SSRC, but that SSRC may very well
only appear as CSRC to the entity receiving the RTCP SR.

> the far end we would want it to be played out in that position.

In such a case, couldn't the provider use three different CSRCs to
indicate which mic was being sent? Or use three different SSRCs for that
purpose, if necessary to identify the correlation.

[BoB]: It would definitely be possible to make this work. However, it
would also de-couple the mics from a sync perspective and it is possible
to describe cases when things would break. For example, assume a number
of mics that cover parts of the same sound field (e.g. in the same room)
that are not totally separated audio-wise (part of audio going into one
mic will also go into one or more other mics) and assume they use
different CNAMEs. Each CNAME is shared with a corresponding camera.
Camera and mic does not necessarily have to map 1:1, there may for
example be multiple mics for one camera. Further assume that one or more
of the audio streams are subject to network jitter, making it necessary
to compensate the playout delay somewhat in some audio jitter buffer. It
will still be possible to sync the delayed audio with other audio and
video sharing the same CNAME. However, the other CNAMEs in the same room
will per definition have no sync relation, resulting in that audio t
hat went into the "delayed mic" will be perceived as an echo since it
can also be heard un-delayed from those other mics. That would not
happen if all mics sharing the sound field also share CNAME.

> arrives is much more noticeable to users.

I had assumed that the SSRC:CNAME correlation was cached. If so, I would
think this wouldn't normally be a problem.

[BoB]: Yes, I would also say that the correlation is cached. It can
however be a startup problem if RTP media reaches the receiver before
any RTCP SR carrying CNAME is received (say e.g. that SR was lost or
just delayed). RFC 6051 Rapid Synchronization of RTP Flows describes a
solution to this problem, based on use of RTP Header Extension.

The point of my question is just to see if there is a way to use
existing mechanisms rather than invent new ones.

	Thanks,
	Paul

---

From: Bo Burman <[email-masked]>
Date: Mon, 4 Jun 2012 15:58:14 +0200

I very much support that. /Bo B 

-----Original Message-----
From: Espen Berger (espeberg) [mailto:[email-masked]] 
Sent: Sunday, June 03, 2012 11:55 PM
To: Bo Burman; [email-masked]
Subject: RE: [clue] draft-romanow-clue-audio-rendering-tag-00.txt

Related to

What if a media provider assigned a unique SDES SRCNAME (as described in
[SRCNAME]) to each video stream, and dynamically assign SRCNAME values to the audio stream to move it between the video positions. Since all stream share the same CNAME they will be played out synchronized. 

The initial configuration of audio and video stream will be configured in CLUE. 

Cheers

-Espen
[SRCNAME]

-----Original Message-----

Hi Paul, all,

See inline below.

/Bo B

-----Original Message-----

	Thanks,
	Paul

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Tue, 5 Jun 2012 12:11:59 +0200

I like the logic behind the SRCNAME suggestion too.

However, we should bear in mind that abrupt switches occur. RTCP may
incur delays as Rob pointed out. Therefore having the audio-to-video
association in RTP header extension makes sense.

Also, it should be possible to associate a single audio stream to
several video streams.

And the mechanism chosen must work with both precomposed/mixed and
switched streams from an MCU to a consumer.

Johan

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Bo Burman
Sent: 4. juni 2012 15:58
To: Espen Berger (espeberg); [email-masked]
Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt

-----Original Message-----
From: Espen Berger (espeberg) [mailto:[email-masked]]
Sent: Sunday, June 03, 2012 11:55 PM
To: Bo Burman; [email-masked]
Subject: RE: [clue] draft-romanow-clue-audio-rendering-tag-00.txt

Related to

Cheers

-Espen
[SRCNAME]

-----Original Message-----

Hi Paul, all,

See inline below.

/Bo B

-----Original Message-----

[BoB]: It would definitely be possible to make this work. However, it
would also de-couple the mics from a sync perspective and it is possible
to describe cases when things would break. For example, assume a number
of mics that cover parts of the same sound field (e.g. in the same room)
that are not totally separated audio-wise (part of audio going into one
mic will also go into one or more other mics) and assume they use
different CNAMEs. Each CNAME is shared with a corresponding camera.
Camera and mic does not necessarily have to map 1:1, there may for
example be multiple mics for one camera. Further assume that one or more
of the audio streams are subject to network jitter, making it necessary
to compensate the playout delay somewhat in some audio jitter buffer. It
will still be possible to sync the delayed audio with other audio and
video sharing the same CNAME. However, the other CNAMEs in the same room
will per definition have no sync relation, resulting in that audio t hat
went into the "delayed mic" will be perceived as an echo since it can
also be heard un-delayed from those other mics. That would not happen if
all mics sharing the sound field also share CNAME.

	Thanks,
	Paul

---

From: bbaldino <[email-masked]>
Date: Wed, 06 Jun 2012 16:17:47 -0700

Espen,
When you say "assign a unique SDES SRCNAME per video stream" you mean per
individual video stream, not video capture, correct?  What would be the
behavior when audio was being sent whose corresponding video wasn't being
transmitted at that time?