Subject: [clue] Association of audio and video from switched captures (RE: draft-romanow-clue-audio-rendering-tag-00.txt)

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Thu, 14 Jun 2012 12:45:51 +0200

Hi all

The background for the suggestion to use SRCNAME was how we want to
associate audio with video for lip-sync and logical pairing of audio and
video from the same scene. I see the ability of dynamically associate
audio and video useful when a consumer receives switched captures from a
switching MCU, and the association is not statically defined up front. 

I found three levels of matching 
* Matching provider - to ensure that a consumer is able to associate all
media from the same provider 
* Matching capture scene (or source) - to ensure that a receiver is able
to match audio and video from the same scene, where a provider send
multiple scenes (e.g. 'main speaker' and 'presentation')  
* Matching segment within a capture scene - to ensure that a receiver
can associate audio and video from individual segments within a capture
scene.  

For the first one we can reuse the existing RTCP SDES CNAME mechanisms.
By recommending to use RFC6222 we can assume that CNAMEs are unique
across a conference. 

For the second we have SDES SRCNAME which can be used to group streams
from the same capture scene, e.g. audio and video from the same scene
uses the same SRCNAME. This is not use for lip-sync, but to associate
audio and video into different scenes and enable the ability to play out
audio with the corresponding video. 

For the third there are no current defined standard. One potential
solution could be to use hierarchical SRCNAME, to express segments like
SRCNAME='srcid/segmentid', e.g. SRCNAME='main/left'. The srcid should
refer to the capture scene id in the CLUE advertisement and the
segmentid should refer to the active capture scene entry configuration.
An audio stream belonging to all video captures in a scene can refer to
the srcid without additional segment information. 

Brian, If a consumer receives audio without corresponding video you can
either do default position (e.g. play out in the centre) or refer to the
CLUE advertisement/configuration information and use the static
configuration from there. 

Johan, for rapid association between audio and video we can look at a
similar scheme to what is suggested in RFC6051, Rapid Synchronisation of
RTP flows. In the case of captures being generated on the MCU the CNAME
and SRCNAME refers to names originating from the MCU, so the mechanisms
should be the same.

This suggestion should not change any of the CLUE work, mainly extending
it when we approach the solution phase. 

Cheers 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Brian Baldino (bbaldino)
Sent: 7. juni 2012 01:18
To: Johan Ludvig Nielsen (johaniel); [email-masked]
Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt

Espen,
When you say "assign a unique SDES SRCNAME per video stream" you mean
per individual video stream, not video capture, correct?  What would be
the behavior when audio was being sent whose corresponding video wasn't
being transmitted at that time?

On 6/5/12 3:11 AM, "Johan Ludvig Nielsen (johaniel)"
<[email-masked]>
wrote:

> I like the logic behind the SRCNAME suggestion too.
> 
> However, we should bear in mind that abrupt switches occur. RTCP may 
> incur delays as Rob pointed out. Therefore having the audio-to-video 
> association in RTP header extension makes sense.
> 
> Also, it should be possible to associate a single audio stream to 
> several video streams.
> 
> And the mechanism chosen must work with both precomposed/mixed and 
> switched streams from an MCU to a consumer.
> 
> Johan
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of Bo Burman
> Sent: 4. juni 2012 15:58
> To: Espen Berger (espeberg); [email-masked]
> Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt
> 
> I very much support that. /Bo B
> 
> -----Original Message-----
> From: Espen Berger (espeberg) [mailto:[email-masked]]
> Sent: Sunday, June 03, 2012 11:55 PM
> To: Bo Burman; [email-masked]
> Subject: RE: [clue] draft-romanow-clue-audio-rendering-tag-00.txt
> 
> Related to
> 
>> However, in multistream there may be a more granular positional 
>> association than just this synchronisation. For instance, in a case 
>> where an endpoint produces three video streams and one audio stream, 
>> all four streams would likely share a common clock and hence can be 
>> sychronised. However, at any given time the audio may come from one 
>> of
> 
>> three microphones associated with the three positions, and hence at 
>> the far end we would want it to be played out in that position.
> 
> What if a media provider assigned a unique SDES SRCNAME (as described 
> in
> [SRCNAME]) to each video stream, and dynamically assign SRCNAME values

> to the audio stream to move it between the video positions. Since all 
> stream share the same CNAME they will be played out synchronized.
> 
> The initial configuration of audio and video stream will be configured

> in CLUE.
> 
> Cheers
> 
> -Espen
> [SRCNAME]
> http://tools.ietf.org/html/draft-westerlund-avtext-rtcp-sdes-srcname-0
> 0
> 
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of Bo Burman
> Sent: 1. juni 2012 17:37
> To: [email-masked]
> Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt
> 
> Hi Paul, all,
> 
> See inline below.
> 
> /Bo B
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of Paul Kyzivat
> Sent: Thursday, May 31, 2012 8:17 PM
> To: [email-masked]
> Subject: Re: [clue] draft-romanow-clue-audio-rendering-tag-00.txt
> 
> On 5/31/12 1:51 PM, Robert Hansen wrote:
>> Hi Paul,
>> 
>> As you say, CNAME is used for lipsyncby establishing which RTP 
>> streams
> 
>> share a common clock.
> 
> Is the CNAME used for this always derived from the SSRC, or can it be 
> derived from a CSRC?
> 
> [BoB]: CNAME is always related to an SSRC, but that SSRC may very well

> only appear as CSRC to the entity receiving the RTCP SR.
> 
>> However, in multistream there may be a more granular positional 
>> association than just this synchronisation. For instance, in a case 
>> where an endpoint produces three video streams and one audio stream, 
>> all four streams would likely share a common clock and hence can be 
>> sychronised. However, at any given time the audio may come from one 
>> of
> 
>> three microphones associated with the three positions, and hence at 
>> the far end we would want it to be played out in that position.
> 
> In such a case, couldn't the provider use three different CSRCs to 
> indicate which mic was being sent? Or use three different SSRCs for 
> that purpose, if necessary to identify the correlation.
> 
> [BoB]: It would definitely be possible to make this work. However, it 
> would also de-couple the mics from a sync perspective and it is 
> possible to describe cases when things would break. For example, 
> assume a number of mics that cover parts of the same sound field (e.g.

> in the same room) that are not totally separated audio-wise (part of 
> audio going into one mic will also go into one or more other mics) and

> assume they use different CNAMEs. Each CNAME is shared with a
corresponding camera.
> Camera and mic does not necessarily have to map 1:1, there may for 
> example be multiple mics for one camera. Further assume that one or 
> more of the audio streams are subject to network jitter, making it 
> necessary to compensate the playout delay somewhat in some audio 
> jitter buffer. It will still be possible to sync the delayed audio 
> with other audio and video sharing the same CNAME. However, the other 
> CNAMEs in the same room will per definition have no sync relation, 
> resulting in that audio t hat went into the "delayed mic" will be 
> perceived as an echo since it can also be heard un-delayed from those 
> other mics. That would not happen if all mics sharing the sound field
also share CNAME.
> 
>> RTCP SDES packets also has the property that they are not received as

>> frequently or in sync with the RTP media packets; this isn't a 
>> particular problem for lipsync, as the adjustments required to sync 
>> streams are generally small and in the case of audio can be concealed

>> in moments of silence. However, starting to play out audio from one 
>> speaker and then moving it a few seconds later when the SDES packet 
>> arrives is much more noticeable to users.
> 
> I had assumed that the SSRC:CNAME correlation was cached. If so, I 
> would think this wouldn't normally be a problem.
> 
> [BoB]: Yes, I would also say that the correlation is cached. It can 
> however be a startup problem if RTP media reaches the receiver before 
> any RTCP SR carrying CNAME is received (say e.g. that SR was lost or 
> just delayed). RFC 6051 Rapid Synchronization of RTP Flows describes a

> solution to this problem, based on use of RTP Header Extension.
> 
> The point of my question is just to see if there is a way to use 
> existing mechanisms rather than invent new ones.
> 
> Thanks,
> Paul
> 
>> Rob
>> 
>> On 31/05/2012 18:27, Paul Kyzivat wrote:
>>> I'm really happy to see drafts such as this one and the others sent 
>>> out at the same time. They really communicate a lot of thinking and 
>>> really help to facilitate conversation.
>>> 
>>> I do have some basic questions about the functionality proposed in 
>>> this draft. (Note that I'm not well versed in media processing, so I

>>> won't feel bad to be told that my questions make no sense.)
>>> 
>>> IIUC, RTP (together with some basic signaling in SDP) has a 
>>> mechanism
> 
>>> to specify lip-sync, based on matching CNAME values between an audio

>>> and video stream. Why is the lip-sync mechanism not sufficient for 
>>> the consumer to match a particular packet from an audio capture to 
>>> the
>>> screen(s) that displayed recent video packets?
>>> 
>>> I do see a different but related problem that I don't think has been

>>> addressed yet: how do I ensure that I have selected the proper set 
>>> of
> 
>>> audio captures that will carry the audio corresponding to the set of

>>> video captures that I have selected?
>>> 
>>> Thanks,
>>> Paul
>>> 
>>> On 5/31/12 1:03 AM, Allyn Romanow (allyn) wrote:
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: [email-masked] [mailto:[email-masked]]
>>>> Sent: Wednesday, May 30, 2012 10:02 PM
>>>> To: Allyn Romanow (allyn)
>>>> Cc: [email-masked]; Robert Hansen (rohanse2); Brian

>>>> Baldino (bbaldino)
>>>> Subject: New Version Notification
>>>> fordraft-romanow-clue-audio-rendering-tag-00.txt
>>>> 
>>>> A new version of I-D, draft-romanow-clue-audio-rendering-tag-00.txt
>>>> has been successfully submitted by Allyn Romanow and posted to the 
>>>> IETF repository.
>>>> 
>>>> Filename: draft-romanow-clue-audio-rendering-tag
>>>> Revision: 00
>>>> Title: The need for audio rendering tag mechanism in the CLUE 
>>>> Framework Creation date: 2012-05-31 WG ID: Individual Submission 
>>>> Number of pages: 8
>>>> 
>>>> Abstract:
>>>> The purpose of this draft is for discussion in the CLUE working 
>>>> group.
>>>> 
>>>> It proposes adding an audio rendering tag to the CLUE framework 
>>>> [I-D.ietf-clue-framework], which makes it possible for the consumer

>>>> to correctly render audio with respect to video in a multistream 
>>>> video conference. The solution proposed is in partial response to 
>>>> CLUE Task #10, Does Framework provide sufficient info for receiver?
>>>> 
>>>> 
>>>> 
>>>> 
>>>> The IETF Secretariat
>>>> _______________________________________________
>>>> clue mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/clue
>>>> 
>>> 
>>> _______________________________________________
>>> clue mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/clue
>> 
>> _______________________________________________
>> clue mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/clue
>> 
> 
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

_______________________________________________
clue mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/clue