Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as WG item

From: <[email-masked]>
Date: Mon, 1 Sep 2008 15:56:54 +0300

Thanks Colin, I will submit another individual version for this.

A quick question. During the AVT session in Dublin, it was first agreed
that it is obvious to require aligned RTP timestamps for all the RTP
sessions in multi-session transmission, but later I heard that there was
an issue found out by Colin, Jonathan Lennox, Thomas Schierl et al.
Could you please clarify what is issue such that I know whether it is
fine to specify that cross-session decoding order recoevery in this
draft is based on RTP timestamps or NTP timestamps. 

BR, YK 

________________________________

	From: [email-masked] [mailto:[email-masked]] On
Behalf Of ext Colin Perkins
	Sent: 01 September, 2008 01:36
	To: Roni Even
	Cc: 'Cullen Jennings'; 'AVT WG'; 'Tom Taylor'
	Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as
WG item

	Hi, 

	On 23 Aug 2008, at 22:32, Roni Even wrote:

		During the Dublin meeting it was agreed to ask if
http://tools.ietf.org/id/draft-lakaniemi-avt-rtp-evbr-02.txt can become
a WG items.

		This is the RTP payload specification for G.718 that is
in final approval at the ITU. Payload specifications for standard codec
are within the charter of the WG.

		If there are no objections by the September 1st we will
accept it as a WG item and add this draft to the charter of AVT

	I have no objection in principle to taking this draft as a work
item, but I think there are a couple of problems with the current draft:

	- Section 3.6 introduces a new cross-stream/cross-layer
synchronisation mechanism, as an alternative to RTCP SR information.
Such a thing might be appropriate - we talked about the need for this
for SVC in Dublin - but it should be a general mechanism, rather than
something specific to a payload format. I think Jonathan Lennox was
going to write a draft on this subject?

	- As noted in Section 4, an RTCP APP packet isn't appropriate
for sending configuration data. This should likely be a new RTCP AVPF
feedback packet.

	I would suggest that these be addressed before the draft is
accepted as a work item, since there seemed to be consensus on both
directions in Dublin. 

	Also, a more minor issue is that Section 3.3 should reference
RFC 5117, and explain that such devices are RTP translators, and so must
follow RFC 3550, section 7.

	Cheers,
	-- 
	Colin Perkins
	http://csperkins.org/
	
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Colin Perkins <[email-masked]>
Date: Mon, 1 Sep 2008 23:11:31 +0100

Ye-Kui,

> draft is based on RTP timestamps or NTP timestamps.

I think Jonathan Lennox and Thomas Schierl were going to write a  
draft discussing these issues, and proposing a common way forward. It  
would be good if you could work with them to do that, rather than  
propose another payload-specific variant. We really don't want to  
have different ways of doing cross-layer synchronisation for  
different payload formats; that would complicate implementations for  
no good reason.

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 11:39:46 +0300

Colin,

I am not preparing to propose another payload-specific variant. What I
am considering is whether to remove our current method from the EVBR
draft but only use the simplest method based on aligned RTP timestamps. 

My understanding is as follows. Everybody (for both this EVBR draft and
the SVC draft) proposed to use aligned RTP timestamps in the beginning
and you said that that broke the random initial setting rule of RTP
timestamps accroding to RFC 3550 and one should use NTP timestamps based
on RTCP SRs. However, David Oran commented in the Dublin AVT session
that it was obvious to use aligned RTP timestamps and it was
temporarilly concluded that aligned RTP timestamps should be used at
that time. Later I heard that you and Jonathan and Thomas again found a
problem with aligned RTP timestamps. 

So my question is simple, what is the problem you guys have identified
for using aligned RTP timestamps. Could you or Jonathan or Thomas
clarify? 

A further question: Is the common way of doing cross-layer
synchronisation you are looking for targets replacing all the
cross-layer synchronisation methods in the SVC draft and in this EVBR
draft? Or you are just trying to replace the timestamps based methods?
Or what you are looking for is not really a generic method for
cross-layer synchronisation but just a method on how to solve the delay
issue associated with the RTCP SR based method?

A clarification of the above would be very helpful on how to process
both this and the SVC draft. Thank you!

BR, YK 

>From: ext Colin Perkins [mailto:[email-masked]] 
>Sent: 02 September, 2008 01:12
>To: Wang Ye-Kui (Nokia-NRC/Tampere)
>Cc: [email-masked]; [email-masked]; [email-masked]; 
>[email-masked]
>Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as WG item
>
>Ye-Kui,
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 2 Sep 2008 10:29:46 +0100

Ye-Kui,

> draft but only use the simplest method based on aligned RTP  

The standard RTP method for synchronisation is to use the RTCP SR  
packets. Both the current draft, and using aligned timestamps, use  
payload specific variants.

> for using aligned RTP timestamps. Could you or Jonathan or Thomas

I don't recall that we discussed this issue, sorry. The obvious  
problems with aligned timestamps are that they don't work unless all  
sessions use the same clock rate, and they make RFC 1889 encryption  
even less secure.

> draft? Or you are just trying to replace the timestamps based methods?

There is a standard way of doing accurate timing synchronisation  
across RTP sessions using RTCP SR packets. I see no reason why SVC,  
MVC, EVBR, and other formats can't use it. Timing synchronisation  
between sessions is separate to the way decoding order is determined  
(although some ways of determining decoding order might rely on  
accurate time synchronisation).

> issue associated with the RTCP SR based method?

We already have a generic cross-layer timing synchronisation method.  
There might be a need to reduce the delay associated with waiting for  
an RTCP SR packet, to speed that generic method up, but that's  
independent of the payload format.

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 13:39:59 +0300

Colin,

Thanks for your quick clarifications, which helped a lot! 

>The obvious problems with aligned timestamps are that they don't work
unless all  
>sessions use the same clock rate, and they make RFC 1889 encryption  

To me it seems use of the same clock rate for all the RTP sessions in a
multi-session transmission is obvious. You see any disadvantage of this?

For RFC 1889, it has been obsoleted by RFC 3550. Do we still need to
take an obsolete RFC into consideration or that issue still applies to
RFC 3550?

BR, YK

>-----Original Message-----
>Sent: 02 September, 2008 12:30
>Cc: [email-masked]; [email-masked]; [email-masked]

---

From: Colin Perkins <[email-masked]>
Date: Tue, 2 Sep 2008 12:08:04 +0100

On 2 Sep 2008, at 11:39, <[email-masked]> wrote:
>> The obvious problems with aligned timestamps are that they don't  
>> work unless all sessions use the same clock rate, and they make  
>> RFC 1889 encryption even less secure.
>

I can certainly envisage potential layered audio codecs where each  
layer has a different sampling rate, and hence uses a different RTP  
timestamp rate.

> take an obsolete RFC into consideration or that issue still applies to

That feature is present but deprecated in RFC 3550 (section 9). There  
used to be widely deployed implementations, although I don't know how  
prevalent they are now.

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 14:22:11 +0300

>layer has a different sampling rate, and hence uses a different RTP  

This is surely true. For this EVBR codec, layers L1 and L2 may have
sampling rate of 8 or 16 kHz, while other layers must have sampling rate
of 16 kHz. However, I don't see a problem of using the same clock rate
for RTP timestamps. For example, in the current draft, the timestamp
clock frequency of 32 kHz is specified.

BR, YK

---

From: Colin Perkins <[email-masked]>
Date: Tue, 2 Sep 2008 12:42:24 +0100

> clock frequency of 32 kHz is specified.

Sure, for this one case. The standard RTCP SR-based mechanism works  
for all cases though.

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 15:01:08 +0300

>>> sampling rate of 8 or 16 kHz, while other layers must have 
>sampling  
>>> rate

There are existing standard systems (e.g. 3GPP) using RTP without RTCP - I know that this has been discussed in AVT and those systems might be considered as "broken" by some. However, the systems do work and are deployed. The timestamp aligned method works even for such "broken" systems, while the RTCP-based method does not. And so far I don't where a real problem is for the timestamp aligned method. The RFC 1889 issue to me is not a real issue, as I cannot imagine a future standard will reference this RFC instead of RFC 3550. 

>>   
>Is it guaranteed that RTCP SRs allow sample-accurate alignment of the 
>streams? I heard rumors that many applications can only guarantee RTCP 
>SR accuracy in the range of ms.
>

BR, YK

---

From: Colin Perkins <[email-masked]>
Date: Tue, 2 Sep 2008 13:02:59 +0100

On 2 Sep 2008, at 12:53, Stefan Döhla wrote:
> Is it guaranteed that RTCP SRs allow sample-accurate alignment of  
> the streams?

Yes, if implemented correctly.

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Tue, 2 Sep 2008 14:00:40 +0100

Ye-Kui,

On 2 Sep 2008, at 13:01, <[email-masked]> wrote:
...
> There are existing standard systems (e.g. 3GPP) using RTP without  
> RTCP - I know that this has been discussed in AVT and those systems  
> might be considered as "broken" by some. However, the systems do  
> work and are deployed. The timestamp aligned method works even for  
> such "broken" systems, while the RTCP-based method does not. And so  
> far I don't where a real problem is for the timestamp aligned  
> method. The RFC 1889 issue to me is not a real issue, as I cannot  
> imagine a future standard will reference this RFC instead of RFC 3550.

You're completely missing the point.

We have a synchronisation mechanism, in the form of RTCP SR packets,  
that works in all cases, has been a standard for over 12 years, and  
is widely implemented. Rather than do something different for this  
payload format - slightly simplifying implementations that only  
support this format, but making general purpose implementations and  
future standards more complex - use the existing standard.

-- 
Colin Perkins
http://csperkins.org/

---

From: Stefan Döhla <[email-masked]>
Date: Tue, 02 Sep 2008 13:53:00 +0200

- Stefan

-- 
Dipl.-Ing. Stefan Döhla     | Email: [email-masked]
Multimedia Transport        | Phone: +49 (0)9131 776 6042 (!NEW!)
Audio Department            | Fax:   +49 (0)9131 776 6099 (!NEW!)
Fraunhofer IIS              |
Am Wolfmantel 33            |
91058 Erlangen, Germany     | Web:   http://www.iis.fraunhofer.de/amm

---

From: [email-masked]
Date: Tue, 02 Sep 2008 16:01:05 +0200

Hi all

This is an old debate that has never bring a right response until today.

>From an IPTV application point of view. If we consider that timestamp
may change between multiple sessions transporting a single content
(audio or video), I
see two main constraints, notably for SVC streams:

- A SVC stream may use a multicast transport (with an IP multicast
address for each video session) without RTCP messages, so synchronizing
timestamps with RTCP is not possible.

- IPTV services need very fast change between channels that cannot
accommodate long delays from RTCP messages to synchronize multiple
sessions

However we believe you cannot get away from a strict compatibility with
existing AVC and rfc 3984, and indeed a synchronization between RTP
sessions must exist.

According all these constraints, may we suggest you to add in SDP the
difference between a
wall clock and the
timestamp in an RTP session, that I name below "delta_time(layer)"?
Wall clock may be the clock of the encoder for SVC. Computing the
difference between timestamps from two sessions allows to synchronize
them, even without  information about this common wallclock. This is
possible because because these differences remain constants when the
time progress.

Assuming that:

delta_time(layer_0) = wall_clock - timestamp(layer_0)

delta_time(layer_1) =
wall_clock - timestamp(layer_1)

....

delta_time(layer_i =
wall_clock - timestamp(layer_i)

Then:

delta_time(layer_1) -
delta_time(layer_0) = timestamp(layer_1) - timestamp(layer_0)

...

so, synchronization become easy and faster than with RTCP messages, at
less for SVC with an answer at the application level.

If reference clocks are different, SDP must contain also the timestamp
value at which delta_time is computed.

Can you consider this
idea is a way to resolve some timestamp synchronization that could
adapt also other encoding schemes?

We intend to write a draft to develop this idea.

Regards

Gérard BABONNEAU, Xavier MARJOU

[email-masked] a écrit :

To me it seems use of the same clock rate for all the RTP sessions
in a
multi-session transmission is obvious. You see any disadvantage of
this?

sampling

rate
of 16 kHz. However, I don't see a problem of using the same

clock rate

BR, YK

--

BABONNEAU Gerard RD-MAPS-REN

REN/MAPS/DPC

Tel : +33 299124008

Email:
[email-masked]

R&D Rennes 4, rue du Clos-Courtel - BP 91226 - 35512 Cesson-Sevigne
Cedex - France

Mailer ThunderBird sous Mandriva

Installez simplement votre poste de travail Linux avec
http://web2000.rd.francetelecom.fr

---

From: Randell Jesup <[email-masked]>
Date: Tue, 02 Sep 2008 12:06:36 -0400

>sessions use the same clock rate, and they make RFC 1889 encryption  

Correct me if I'm wrong, but the only real reasons for random initial
timestamps (and sequence numbers) is for RFC 1889 encryption and (perhaps)
for reducing the ease of "hijacking" or disrupting unencrypted (or
un-authenticated) sessions by an attacker who isn't able to monitor the
media path.

>independent of the payload format.

That's the biggest problem so far with the generic method - startup
syncronization time, especially for AVP/SAVP (as opposed to AVPF/SAVPF).
Even for AVPF, just because the sender *can* send extra or repeated RTCP
SR/RR packets near the start of a stream doesn't mean that they will, and
there's no way in the Codec Control AVPF extensions to "request" an RTCP.

Possible solutions:
1) Synchronized initial timestamps
   Simple to implement for many people.  May not be for others.  Needs to
       be signaled and/or part of payload spec.
   Problem: if they don't use the same timebase the sync will drift; you
       still need SR/RR to fix it.  May also cause problems with RFC 1889,
       etc.  Can be conf
2) Multiple early RTCPs at start of stream
   Pretty simple to implement - just send RTCPs.
   Problem: requires AVPF/SAVPF.  Blows RTCP bandwidth that might be needed
       for other things.  Need to decide how much extra to send, and when
       to stop (new CCM message?).  May not work for non point-to-point
       topologies(?).  Probably should be signaled.
3) Add "I need an NTP sync point" CCM message.
   Not hard to implement.  Saves bandwidth.  Don't need to guess when to
       stop sending SR/RR.  Normally (baring non-compound RTCP) also
       includes your own sync info.
   Problems: Still some latency (though can be low), especially if there
       are lost packets (especially if the "I need an NTP sync point" gets
       lost).  Decision about how to handle loss is an issue (how long to
       wait).  Needs rtcp-fb additions (probably).
4) Specify the initial cross-layer sync in the negotiation.
   Allows sync on the very first packets in the streams.  Occurs on
       a reliable channel.
   Problems: sync may drift; RR/SR still needed.  May be hard for some
       applications to get what "will be" a sync point before stream is
       established, especially in decomposed UAs.  May affect RFC1889
       security (not sure, not sure if we care).  May yield larger SDPs,
       though you don't *need* to give the NTP time for each stream, just a
       set of timestamps that are of roughly equal NTP time (nearest packet
       to a common NTP time).

Others?
  
-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: Randell Jesup <[email-masked]>
Date: Tue, 02 Sep 2008 12:10:43 -0400

>clock frequency of 32 kHz is specified.

For a generic standard for sync, this is a problem - a decomposed UA may
have different timestamps-to-realtime rates for different streams.  Even a
UA that uses different HW sources (live video and audio) may have different
timestamp rates.  SVC is a "special" case in that it's multiple encodings
of the same sampled data, so inherently there's no skew/drift between the
streams.

---

From: Randell Jesup <[email-masked]>
Date: Tue, 02 Sep 2008 12:20:20 -0400

<[email-masked]> writes:
>There are existing standard systems (e.g. 3GPP) using RTP without RTCP - I
>know that this has been discussed in AVT and those systems might be
>considered as "broken" by some. However, the systems do work and are
>deployed. The timestamp aligned method works even for such "broken"
>systems, while the RTCP-based method does not. And so far I don't where a
>real problem is for the timestamp aligned method.

Those mechanisms (RTP without RTCP) will NOT work if there is any
timestamp-to-NTP skew between the streams.  There are two additional
assumptions being made:

   1) that the initial timestamps have a known value, OR the timestamp
      rates for each stream are identical, or you're sure you won't lose
      media off the "front" end of the stream.
   2) there is no skew between the timestamp-to-NTP rates for the two
      channels.  For example, if audio is (actually) sampled at 8001 HZ
      (on a channel with a nominal 8000HZ period), and video is sampled
      at 89990 HZ (on a channel with a nominal 90000 HZ rate), the two
      streams will drift out of sync.

Alignment can help you at the start of a stream (with some issues, like
front-end loss), but you still normally need RTCP to maintain sync.

RTP without RTCP is broken, especially for the case of syncing two streams
unless (per above) you make a number of simplifying assumptions/requirements.

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 21:30:34 +0300

Colin,
 
If the point is that the RTCP SR based method is the standard way for
cross-layer sync of layered encodings while the aligned timestamp based
method is not hence cannot be used, regardless of their pros and cons,
then yes I was and I am still missing this point. Maybe others (who are
still discussing the pros and cons of different methods) too :-) 
 
In RFC 3550, sections where layered encodings are mentioned are:
- 2.4, an introduction of layer encodings
- 6.2, mentioned the use of bandwidth modifier
- 6.3.9, mentioned the carriage of SDES and CNAME
- 8.11, about SSRC use
- 11, mentioned use of port numbers
 
The semantics of timestamp in Section 5.1 talks a lot on
synchronization, but that is about presentation synchronization of
different media types, as indicated by the last sentence of the
semantics (copied below for convenience):
A receiver can then synchronize presentation of the audio and video
packets by relating their RTP timestamps using the timestamp pairs in
RTCP SR packets.
The same is indicated by the last sentence in Section 2.4 (again copied
below for convenience), as Section 5.1 is not mentioned. Note that this
sentence itself seems not accurate as Section 6.2 is not mentioned
either - but maybe that was intentional as the word "should" is used in
Section 6.2 while in Sections 6.3.9, 8.11 and 11, the words "is",
"SHOULD" and "RECOMMENDED" are used. 
 
Details of the use of RTP with layered encodings are given in Sections
6.3.9, 8.3 and 11.
Therefore, I got a conclusion that RFC 3550 has not specified how to do
cross-layer sync of layered encodings. Please let me know if this
conclusion is incorrect. It may also be possible that even RFC 3550 has
not specified how to do cross-layer sync of layered encodings, the RTCP
SR based method has been widely deployed for cross-layer sync of layered
encodings. In that case, it would clarify things if someone can point
out these deployments.  
BR, YK

	From: ext Colin Perkins [mailto:[email-masked]] 
	Sent: 02 September, 2008 16:01
	To: Wang Ye-Kui (Nokia-NRC/Tampere)
	Cc: [email-masked]; [email-masked];
[email-masked]; [email-masked]
	Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as
WG item

	Ye-Kui, 

		There are existing standard systems (e.g. 3GPP) using
RTP without RTCP - I know that this has been discussed in AVT and those
systems might be considered as "broken" by some. However, the systems do
work and are deployed. The timestamp aligned method works even for such
"broken" systems, while the RTCP-based method does not. And so far I
don't where a real problem is for the timestamp aligned method. The RFC
1889 issue to me is not a real issue, as I cannot imagine a future
standard will reference this RFC instead of RFC 3550. 

	We have a synchronisation mechanism, in the form of RTCP SR
packets, that works in all cases, has been a standard for over 12 years,
and is widely implemented. Rather than do something different for this
payload format - slightly simplifying implementations that only support
this format, but making general purpose implementations and future
standards more complex - use the existing standard.

	-- 
	Colin Perkins
	http://csperkins.org/

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 22:45:03 +0300

Gérard, Xavier,
 
To me this method works too, but I don't know whether there are any problems with the carriage of the infromation using SDP, e.g. for systems not using RTP but not using SDP, and how serious are the problems mentioned by Randell in another email. 
 
The method is essentially similar to the method described in Section 2 of this EVBR draft, with the difference that the method in the current EVBR draft transmits the timestamp of the base layer in-band. 
 
BTW, wouldn't it be simpler by signaling the difference of the initial timestamp offset between sessions instead of the difference between the initial timestamp and the wall-clock time?
 
BR, YK

	From: ext [email-masked] [mailto:[email-masked]] 
	Sent: 02 September, 2008 17:01
	To: Wang Ye-Kui (Nokia-NRC/Tampere)
	Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; BABONNEAU Gerard RD-MAPS-REN; MARJOU Xavier RD-CORE-LAN
	Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as WG item

	Hi all
	
	This is an old debate that has never bring a right response until today.
	From an IPTV application point of view. If we consider that timestamp may change between multiple sessions transporting a single content (audio or video), I see two main constraints, notably for SVC streams:
	- A SVC stream may use a multicast transport (with an IP multicast address for each video session) without RTCP messages, so synchronizing timestamps with RTCP is not possible.
	- IPTV services need very fast change between channels that cannot accommodate long delays from RTCP messages to synchronize multiple sessions
	
	However we believe you cannot get away from a strict compatibility with existing AVC and rfc 3984, and indeed a synchronization between RTP  sessions must exist.
	
	According all these constraints, may we suggest you to add in SDP the difference between a wall clock and the timestamp in an RTP session, that I name below "delta_time(layer)"?  Wall clock may be the clock of the encoder for SVC. Computing the difference between timestamps from two sessions allows to synchronize them, even without  information about this common wallclock. This is possible because because these differences remain constants when the time progress.
	
	Assuming that: 
	delta_time(layer_0) = wall_clock - timestamp(layer_0)
	delta_time(layer_1) = wall_clock - timestamp(layer_1)
	....
	delta_time(layer_i = wall_clock - timestamp(layer_i)
	
	Then:
	delta_time(layer_1) - delta_time(layer_0) = timestamp(layer_1) - timestamp(layer_0)
	...
	so, synchronization become easy and faster than with RTCP messages, at less for SVC with an answer at the application level.
	
	If reference clocks are different, SDP must contain also the timestamp value at which delta_time is computed. 
	
	Can you consider this idea is a way to resolve some timestamp synchronization that could adapt also other encoding schemes?
	We intend to write a draft to develop this idea.
	
	Regards

		  

			sampling  

					rate

			clock rate

		

		
		BR, YK

	-- 
	 
	BABONNEAU Gerard RD-MAPS-REN
	REN/MAPS/DPC
	Tel : +33 299124008 
	Email: [email-masked] <mailto:[email-masked]> 
	R&D Rennes 4, rue du Clos-Courtel - BP 91226 - 35512 Cesson-Sevigne Cedex - France 

	Mailer ThunderBird sous Mandriva
	Installez simplement votre poste de travail Linux avec http://web2000.rd.francetelecom.fr

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 22:47:23 +0300

>   Problem: if they don't use the same timebase the sync will 
>drift; you still need SR/RR to fix it.  

But as there is no problem of requiring the use of the same clock rate
for timestamps for different sessions, this should not be a problem.

>      May also cause problems 
>with RFC 1889, etc.  

Will any existing implementations of RFC 1889 using the encryption
method deprecated by RFC 3550 be used for transport of future scalable
media (e.g. SVC and EVBR)?

> Can be conf

This is a incomplete sentence. 

BR, YK 

>From: ext Randell Jesup [mailto:[email-masked]] 
>Sent: 02 September, 2008 19:07
>To: Colin Perkins
>Cc: Wang Ye-Kui (Nokia-NRC/Tampere); [email-masked];

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 22:47:49 +0300

>sampling rate
>timestamp rates.  

Note that we are talking about cross-layer sync of scalable media, not
lip-sync between different media types. 

> SVC is a "special" case in that it's 
>multiple encodings

EVBR, or any other scalable media, is the same. 

BR, YK

---

From: <[email-masked]>
Date: Tue, 2 Sep 2008 22:49:13 +0300

>timestamp-to-NTP skew between the streams. 

Note that there are systems playing with only one media type, e.g.
speech or audio only. 

> There are two additional

If it is OR (i.e. only one of the two conditions holds), then yes loss
of some beginning frames will be problematic. However, there isl no such
problem with AND (both are valid). Is there any problem when both
conditions above are satisfied?

>      streams will drift out of sync.

Again you are talking about lip-sync, which is not the right topic. 

>front-end loss), but you still normally need RTCP to maintain sync.

I guess "sync" here again is lip-sync.

>assumptions/requirements.

Again, note that there are systems playing with only one media type. 

BR, YK

---

From: Colin Perkins <[email-masked]>
Date: Tue, 2 Sep 2008 23:33:59 +0100

Ye-Kui,

The sentence you quote from Section 5.1 of RFC 3550 is part of a  
paragraph that is clearly labelled as an example. The mechanism is  
general purpose, to be used to synchronise any RTP streams that need  
time synchronisation. Synchronisation is not discussed for layered  
coding, because there is no need for a special mechanism - the  
approach of Section 5.1 is used, same as for any other RTP sessions  
that must be synchronised.

If there are problems with this mechanism, they should be addressed  
in a general manner, suitable for any RTP streams, and not in a  
payload format specific way. The only problem mentioned seems to be  
slow synchronisation when joining a session, since one has to wait  
for an RTCP SR packet. That would seem solvable for unicast by  
sending an immediate RTCP SR packet (this is explicitly allowed in  
Section 6.2 of RFC 3550); for multicast, the RTP/AVPF could be used.

Note, once again, that these time synchronisation mechanisms say  
nothing about the decoding order for a layered codec, or any other  
codec.

Colin

On 2 Sep 2008, at 19:30, <[email-masked]> wrote:
> If the point is that the RTCP SR based method is the standard way  
> for cross-layer sync of layered encodings while the aligned  
> timestamp based method is not hence cannot be used, regardless of  
> their pros and cons, then yes I was and I am still missing this  
> point. Maybe others (who are still discussing the pros and cons of  
> different methods) too :-)
>
> packets by relating their RTP timestamps using the timestamp pairs  
> in RTCP SR packets.
> The same is indicated by the last sentence in Section 2.4 (again  
> copied below for convenience), as Section 5.1 is not mentioned.  
> Note that this sentence itself seems not accurate as Section 6.2 is  
> not mentioned either - but maybe that was intentional as the word  
> "should" is used in Section 6.2 while in Sections 6.3.9, 8.11 and  
> 11, the words "is", "SHOULD" and "RECOMMENDED" are used.
>
> Details of the use of RTP with layered encodings are given in  
> Sections 6.3.9, 8.3 and 11.
> Therefore, I got a conclusion that RFC 3550 has not specified how  
> to do cross-layer sync of layered encodings. Please let me know if  
> this conclusion is incorrect. It may also be possible that even RFC  
> 3550 has not specified how to do cross-layer sync of layered  
> encodings, the RTCP SR based method has been widely deployed for  
> cross-layer sync of layered encodings. In that case, it would  
> clarify things if someone can point out these deployments.
> BR, YK
>
>> RTCP - I know that this has been discussed in AVT and those  
>> systems might be considered as "broken" by some. However, the  
>> systems do work and are deployed. The timestamp aligned method  
>> works even for such "broken" systems, while the RTCP-based method  
>> does not. And so far I don't where a real problem is for the  
>> timestamp aligned method. The RFC 1889 issue to me is not a real  
>> issue, as I cannot imagine a future standard will reference this  
>> RFC instead of RFC 3550.
>
> packets, that works in all cases, has been a standard for over 12  
> years, and is widely implemented. Rather than do something  
> different for this payload format - slightly simplifying  
> implementations that only support this format, but making general  
> purpose implementations and future standards more complex - use the  
> existing standard.
>
> -- 
> Colin Perkins
> http://csperkins.org/
>
>
>

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Tue, 2 Sep 2008 23:41:06 +0100

On 2 Sep 2008, at 20:49, <[email-masked]> wrote:
>>> timestamp aligned method.
>>

The problem can still exist in those sessions, depending on how the  
media is captured.

> conditions above are satisfied?

No, but there is also no way for a receiver to know that both  
conditions are satisfied, so it must assume the worst case when  
resynchronising.

> Again you are talking about lip-sync, which is not the right topic.

The same problem can occur with layered or multi-description codecs  
too, depending on how the media is captured.

>> Alignment can help you at the start of a stream (with some issues,  
>> like front-end loss), but you still normally need RTCP to maintain  
>> sync.
>

No - it's potentially an issue for any streams.

>> RTP without RTCP is broken, especially for the case of syncing   
>> two streams unless (per above) you make a number of simplifying  
>> assumptions/requirements.
>

And, again, note that the problems Randell mentions can occur in  
those systems too.

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Tue, 2 Sep 2008 23:50:35 +0100

On 2 Sep 2008, at 17:06, Randell Jesup wrote:
> Correct me if I'm wrong, but the only real reasons for random  
> initial timestamps (and sequence numbers) is for RFC 1889  
> encryption and (perhaps) for reducing the ease of "hijacking" or  
> disrupting unencrypted (or un-authenticated) sessions by an  
> attacker who isn't able to monitor the media path.

Correct. They add some small (trivial?) amount of additional  
security, at essentially no cost (as you've noted, you still need the  
RTCP SR packets, even if timestamps are initially aligned, due to  
potential clock skew).

> syncronization time, especially for AVP/SAVP (as opposed to AVPF/ 
> SAVPF). Even for AVPF, just because the sender *can* send extra or  
> repeated RTCP
> SR/RR packets near the start of a stream doesn't mean that they  
> will, and there's no way in the Codec Control AVPF extensions to  
> "request" an RTCP.
>
> Possible solutions:
> drift; you

Right - this doesn't work in the general case.

>        topologies(?).  Probably should be signaled.

For unicast sessions, RFC 3550 explicitly allows for an immediate  
initial RTCP report without signalling. AVPF can be used in other cases.

>        wait).  Needs rtcp-fb additions (probably).

When coupled with an immediate initial RTCP packet, this would seem  
to solve the startup issue. It's also backwards compatible; non- 
extended clients will synchronise slowly, when they receive the  
eventual SR, but will synchronise.

>        set of timestamps that are of roughly equal NTP time  

I care less about the problems with RFC 1889 security, than with the  
problems due to it not working in the presence of clock skew between  
media.

Cheers,
-- 
Colin Perkins
http://csperkins.org/

---

From: Tom Taylor <[email-masked]>
Date: Wed, 03 Sep 2008 01:53:56 +0200

Let me summarize this discussion, and particularly Colin's point. That point is 
that if your RTP application implements a method of cross-stream synchronization 
that works for all cases, there is no need to also force the designer to 
implement a method that only works for special cases. We can focus on mitigating 
the drawbacks of that general method, and the outcome will still be a general 
method.

Ye-Kui counters that his special case (layered encoding) allows a method that 
escapes the drawbacks of the general method. He questions whether the general 
method has actually been implemented widely, and by implication, if it has not, 
the presumed benefits of generality aren't really there. Meantime he has a 
problem to solve.

I believe textual analysis of RFC 3550 is beside the point, and this captures 
the essence of the discussion. Do people agree?

Tom

---

From: "Jonathan Lennox" <[email-masked]>
Date: Tue, 2 Sep 2008 21:12:06 -0400

Ye-Kui Wang writes:
> Later I heard that you and Jonathan and Thomas again found a

(Again, to be clear, I'm specifically talking about the layered encoding
case, where multiple RTP streams are encoded based on a single input
media source.  Synchronization of separate-but-related media sources,
e.g. for lip sync, is a separate problem.) 

The only difficulty we found with aligned RTP timestamps, per se, was
that you must use the same clock rate for every session's RTP streams.
For audio this isn't necessarily always the most natural choice, e.g.
for wide-band enhancement of narrow-band audio, but it didn't sound like
this restriction would in practice be a problem for anyone.  (There are
already plenty of examples of audio codecs that use something other than
their natural clock rate, e.g. G.722, for various historical reasons.)

The difficulty we did find was with the related question of how to
associate related (base and enhancement) media streams in multi-sender
topologies.  RFC 3550's solution, aligned RTP SSRCs with collision
detection performed on the base RTP session, requires that there indeed
be a single base RTP session that every session member must subscribe
to.  For some (perhaps bizarre) use cases of SVC this isn't necessarily
the case, and for MVC or other future codecs such use cases may be more
natural.

Thomas and I are working on a draft with a taxonomy of existing and
proposed solutions to both source alignment and source correlation
issues that arise in multi-session transmission, with their advantages
and limitations.

-- 
Jonathan Lennox
Vidyo, Inc
[email-masked]

---

From: "Jonathan Lennox" <[email-masked]>
Date: Tue, 2 Sep 2008 22:59:27 -0400

> Yes, if implemented correctly.

I'm not sure this is the case.

If you're sending live (as opposed to pre-recorded) media, clock drift between the media clock (generating RTP timestamps) and the system time-of-day clock (generating NTP timestamps) can make sample-accurate stream alignment based on RTCP SRs extremely problematic, especially as RTCP report intervals lengthen.

Specifically, assuming each RTCP SR's NTP and RTP times accurately represent the time-of-day and sample-clock measurements (respectively) at SR transmission time, and SR transmission time is randomized independently for each RTCP session, max_sample_misalignment = max_rtcp_interval * abs(actual_clock_rate - nominal_clock_rate)/nominal_clock_rate.

If in addition there's imprecision in the time-of-day clock used as the source of NTP time in SRs, the sample mis-alignment increases correspondingly.  (E.g., the Win32 system clock is only updated on every system timer tick, typically 10 ms -- see <http://blogs.msdn.com/oldnewthing/archive/2005/09/02/459952.aspx>.)  This latter point can be compensated for by supplementing the time-of-day clock with a high-precision clock, but this can add considerable implementation complexity.

-- 
Jonathan Lennox
Vidyo, Inc
[email-masked]

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 10:06:11 +0300

Thanks Tom for the good summary.

Just an addition. My key point is that lip-sync between different media
types is not the same thing as cross-layer (for layered encodings) or
cross-channel (for multi-channel encodings) or cross-description (for
multiple description codings - MDC) sync,  as also mentioned by others,
e.g. Jonathan Lennox. The key difference is that lip-sync operates on
media streams independently captured, independently encoded and
independently decoded and only a non-exact sync is needed to present the
decoded samples together, while the other operates on different portions
of the same media captured at exactly the same time, encoded in a
connected way, and decoded in a connected way and in many cases the
decoding order must be exactly synchronized between the different
portions. A method itself that applies to all layered, multi-channel and
MDC codecs is already a generic method. This method should be allowed
unless the conventional method for lip-sync is at least equally good .
How, so far from the discussions, in my understanding, using the RTCP SR
based method is not as good as using the aligned initial timestamp for
cross-layer/channel/description sync. Therefore I don't get it why we
have to bind ourselves to the RTCP SR based method. 

BR, YK 

>From: ext Tom Taylor [mailto:[email-masked]] 
>Sent: 03 September, 2008 02:54
>To: Colin Perkins
>Cc: Randell Jesup; [email-masked]; 
>[email-masked]; Wang Ye-Kui 
>(Nokia-NRC/Tampere); [email-masked]

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 08:25:03 +0100

On 3 Sep 2008, at 03:59, Jonathan Lennox wrote:
> If you're sending live (as opposed to pre-recorded) media, clock  
> drift between the media clock (generating RTP timestamps) and the  
> system time-of-day clock (generating NTP timestamps) can make  
> sample-accurate stream alignment based on RTCP SRs extremely  
> problematic, especially as RTCP report intervals lengthen.
>
> Specifically, assuming each RTCP SR's NTP and RTP times accurately  
> represent the time-of-day and sample-clock measurements  
> (respectively) at SR transmission time, and SR transmission time is  
> randomized independently for each RTCP session,  
> max_sample_misalignment = max_rtcp_interval * abs(actual_clock_rate  
> - nominal_clock_rate)/nominal_clock_rate.

I'm not sure this expression is valid, since I'd expect the receiver  
to estimate the clock skew based on observing several SR's, and  
compensate for it.

> If in addition there's imprecision in the time-of-day clock used as  
> the source of NTP time in SRs, the sample mis-alignment increases  
> correspondingly.  (E.g., the Win32 system clock is only updated on  
> every system timer tick, typically 10 ms -- see <http:// 
> blogs.msdn.com/oldnewthing/archive/2005/09/02/459952.aspx>.)  This  
> latter point can be compensated for by supplementing the time-of- 
> day clock with a high-precision clock, but this can add  
> considerable implementation complexity.

If you have inaccurate clocks, clock synchronisation will be  
inaccurate, sure. That's not a protocol issue.

---

From: "Roni Even" <[email-masked]>
Date: Wed, 3 Sep 2008 10:29:51 +0300

YK,
I think that you are going to specific cases.
What was said is that using RTCP SR is the general solution it works for
systems that use only RTP with no SDP or other signaling and it is not
dependent on the relations between the streams (same source, pre-recorded,
etc)
What Colin was saying is 
" Section 3.6 introduces a new cross-stream/cross-layer synchronisation
mechanism, as an alternative to RTCP SR information. Such a thing might be
appropriate - we talked about the need for this for SVC in Dublin - but it
should be a general mechanism, rather than something specific to a payload
format. I think Jonathan Lennox was going to write a draft on this subject?"

I do not see voices against such a solution as long as it is a general one
that may be used for layered codecs but that will list the assumptions under
which it works.

Jonathan mentioned that he is working on such draft and it will make sense
that evbr should reference this draft. I assume that there are time issues
here so this is a question to Jonathan about when?

Roni Even

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Wednesday, September 03, 2008 10:06 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]
Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as WG item

BR, YK

---

From: Thomas Schierl <[email-masked]>
Date: Wed, 03 Sep 2008 09:41:20 +0200

Since I am back from vacation, we (me an Jonathan) planned to start
with the work this week. So relax a little bit. When we have a document
on the table, the discussion will be much easier.

My view of the discussion and I am with Collin here is: RTCP SR sync
should work also for live. There is only the delay issue at session
start up and for sure the SSRC collision detection. I know Jonathan has
a different opinion here, but we will reflect those opinions in the
draft.

--
Thomas Schierl
--------------
Fraunhofer HHI

Roni Even wrote:

YK,

Roni Even

-----Original Message-----

BR, YK

-----Original Message-----
From: ext Tom Taylor [mailto:[email-masked]]
Sent: 03 September, 2008 02:54
To: Colin Perkins
Cc: Randell Jesup; [email-masked];

Let me summarize this discussion, and particularly Colin's
point. That point is
that if your RTP application implements a method of
cross-stream synchronization
that works for all cases, there is no need to also force the
designer to
implement a method that only works for special cases. We can
focus on mitigating
the drawbacks of that general method, and the outcome will
still be a general
method.

Ye-Kui counters that his special case (layered encoding)
allows a method that
escapes the drawbacks of the general method. He questions
whether the general
method has actually been implemented widely, and by
implication, if it has not,
the presumed benefits of generality aren't really there.
Meantime he has a
problem to solve.

I believe textual analysis of RFC 3550 is beside the point,
and this captures
the essence of the discussion. Do people agree?

Tom

Colin Perkins wrote:

...

----

Visit us at

IBC 2008 / Amsterdam (NL), 12 - 16 September 2008 / Hall 8 Booth C81

more details

insite 2008 / Johannesburg (South Africa), Sandton Convention Centre / 15 - 17 September 2008

more details

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 10:52:32 +0300

Roni, 

Just some clarifications. 

>I think that you are going to specific cases.

OK, if you call that as "specific". 

>dependent on the relations between the streams (same source, 

But problems have been identified for the RTCP SR based method when used
for cross-layer/channel/description/view sync, which says that it may
not work for some cases or may not work as well as the "specific"
method. 

If you say that we anyway have to bind to the RTCP SR based method
regardless of its disadvantages, that's fine, and then I will shut up
and go ahead with only this method in this EVBR draft. But also note
that this "generic" method does not work well enough for SVC (and any
other codecs for which the transmission order may be different from
decoding order) and that's why we have the cross-session decoding order
number based methods specified in the SVC draft. 

>format. I think Jonathan Lennox was going to write a draft on 

I understand this but the topic is now a different one. The discussion
is not any more about whether to keep the alternative method currently
in the EVBR draft, but whether we should allow the use of the RTP
timestamp aligned method.

>that may be used for layered codecs but that will list the 

Right - but people are still discussing the potential problems and there
is not yet clear conclusion that it works (well) for all cases - and it
is clear that it does not work (well) for some SVC cases. 

>here so this is a question to Jonathan about when?

I think Jonathan and Thomas are working on a related but yet another
topic - on SSRC use in multi-sender topologies, as can be seen from his
clarification (copied below). 

"The difficulty we did find was with the related question of how to

and limitations."

BR, YK

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 10:55:12 +0300

See inline below, please.
 
BR, YK

	From: ext Colin Perkins [mailto:[email-masked]] 
	Sent: 03 September, 2008 10:37
	To: Wang Ye-Kui (Nokia-NRC/Tampere)
	Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]; [email-masked]
	Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as
WG item

	On 3 Sep 2008, at 08:06, <[email-masked]> wrote:

		Just an addition. My key point is that lip-sync between
different media types is not the same thing as cross-layer (for layered
encodings) or cross-channel (for multi-channel encodings) or
cross-description (for multiple description codings - MDC) sync,  as
also mentioned by others, e.g. Jonathan Lennox. The key difference is
that lip-sync operates on media streams independently captured,
independently encoded and independently decoded and only a non-exact
sync is needed to present the decoded samples together, while the other
operates on different portions of the same media captured at exactly the
same time, encoded in a connected way, and decoded in a connected way
and in many cases the decoding order must be exactly synchronized
between the different portions. A method itself that applies to all
layered, multi-channel and MDC codecs is already a generic method. This
method should be allowed unless the conventional method for lip-sync is
at least equally good . How, so far from the discussions, in my
understanding, using the RTCP SR based method is not as good as using
the aligned initial timestamp for cross-layer/channel/description sync.

	On the contrary, the RTCP SR based method is the only one that
works in all cases. Using aligned initial timestamps for synchronisation
doesn't work if there is clock skew, and you cannot guarantee that there
won't be clock skew between the components of all layered,
multi-channel, and MDC codecs.
	[Wang Ye-Kui] This is not true. The aligned initial timestamps
works fine even if there is clock skew, as the skew amount is the same
for all portions of one frame/sample.  

		Therefore I don't get it why we have to bind ourselves
to the RTCP SR based method. 

	Because it works, is already standardised and implemented, and
solves this problem. 

	-- 
	Colin Perkins
	http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 08:36:32 +0100

On 3 Sep 2008, at 08:06, <[email-masked]> wrote:
> Just an addition. My key point is that lip-sync between different  
> media types is not the same thing as cross-layer (for layered  
> encodings) or cross-channel (for multi-channel encodings) or cross- 
> description (for multiple description codings - MDC) sync,  as also  
> mentioned by others, e.g. Jonathan Lennox. The key difference is  
> independently encoded and independently decoded and only a non- 
> exact sync is needed to present the decoded samples together, while  
> the other operates on different portions of the same media captured  
> at exactly the same time, encoded in a connected way, and decoded  
> in a connected way and in many cases the decoding order must be  
> exactly synchronized between the different portions. A method  
> itself that applies to all layered, multi-channel and MDC codecs is  
> already a generic method. This method should be allowed unless the  
> conventional method for lip-sync is at least equally good . How, so  
> far from the discussions, in my understanding, using the RTCP SR  
> based method is not as good as using the aligned initial timestamp  
> for cross-layer/channel/description sync.

On the contrary, the RTCP SR based method is the only one that works  
in all cases. Using aligned initial timestamps for synchronisation  
doesn't work if there is clock skew, and you cannot guarantee that  
there won't be clock skew between the components of all layered,  
multi-channel, and MDC codecs.

> Therefore I don't get it why we have to bind ourselves to the RTCP  
> SR based method.

Because it works, is already standardised and implemented, and solves  
this problem.

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 09:10:11 +0100

On 3 Sep 2008, at 08:52, <[email-masked]> wrote:
>> What was said is that using RTCP SR is the general solution it  
>> works for systems that use only RTP with no SDP or other signaling  
>> and it is not dependent on the relations between the streams (same  
>> source, pre-recorded, etc)
>

What are those problems?

> number based methods specified in the SVC draft.

You're confusing two issues: time synchronisation and decoding order.  
Some payload formats might be able to base their decoding order  
decision on the time synchronised media (EVBR seems to be an example  
of this), while other payload formats might need separate approaches  
for decoding order and time synchronisation (SVC seems to be an  
example here).

>> What Colin was saying is " Section 3.6 introduces a new cross- 
>> stream/cross-layer synchronisation mechanism, as an alternative to  
>> RTCP SR information. Such a thing might be appropriate - we talked  
>> about the need for this for SVC in Dublin - but it should be a  
>> general mechanism, rather than something specific to a payload  
>> I do not see voices against such a solution as long as it is a  
>> general one that may be used for layered codecs but that will list  
>> the assumptions under which it works.
>

Again, you're confusing time synchronisation and decoding order  
recovery.

-- 
Colin Perkins
http://csperkins.org/

---

From: Thomas Schierl <[email-masked]>
Date: Wed, 03 Sep 2008 10:13:54 +0200

> for decoding order and time synchronisation (SVC seems to be an 
But, SVC seems to be also an example where you can send the data time 
aligned, see NI-T mode.

-- 
Thomas Schierl
--------------
Fraunhofer HHI

----
Visit us at

IBC 2008 / Amsterdam (NL), 12 - 16 September 2008 / Hall 8 Booth C81
http://www.hhi.fraunhofer.de/index.php?id=2399&L=1

http://www.insitex.co.za

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 09:14:00 +0100

On 3 Sep 2008, at 08:55, <[email-masked]> wrote:
>> description (for multiple description codings - MDC) sync,  as  
>> also mentioned by others, e.g. Jonathan Lennox. The key difference  
>> is that lip-sync operates on media streams independently captured,  
>> exact sync is needed to present the decoded samples together,  
>> while the other operates on different portions of the same media  
>> captured at exactly the same time, encoded in a connected way, and  
>> decoded in a     connected way and in many cases the decoding  
>> order must be exactly synchronized between the different portions.  
>> A method itself that applies to all layered, multi-channel and MDC  
>> codecs is already a generic method. This method should be allowed  
>> unless the conventional method for lip-sync is at least equally  
>> good . How, so far from the discussions, in my understanding,  
>> using the RTCP SR based method is not as good as using the aligned  
>> initial timestamp for cross-layer/channel/description sync.
>
> works in all cases. Using aligned initial timestamps for  
> synchronisation doesn't work if there is clock skew, and you cannot  
> guarantee that there won't be clock skew between the components of  
> all layered, multi-channel, and MDC codecs.
> works fine even if there is clock skew, as the skew amount is the  
> same for all portions of one frame/sample.

This only works if you're generating multiple layers from a single  
frame - that's a very restricted case. For example, it doesn't work  
for layered or MDC codecs that work across multiple frames captured  
by separate devices, since there might be skew between frames. The  
RTCP SR based mechanism does work in those cases.

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 09:16:31 +0100

> to estimate the clock skew based on observing several SR's, and

And, of course, even if your expression is valid, this just means you  
might need to allocate a higher RTCP bandwidth fraction for sessions  
with large reporting intervals and high clock skew, to reduce the  
reporting interval.

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 11:25:54 +0300

>> not work for some cases or may not work as well as the "specific"

I will summarize the problems identified and no invalidated so far in a
next email. 

>for decoding order and time synchronisation (SVC seems to be an  

No - if someone is confusing these - that's not me. All what we are
talking about here is for decoder order recovery, with time
synchonization being one possible approach to achieve that. I have never
tried to talk about this for any other purpose than decoder order
recovery. Point out please if there was one. 

BR, YK

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 11:32:53 +0300

[From Colin] This only works if you're generating multiple layers from a
single frame - that's a very restricted case. For example, it doesn't
work for layered or MDC codecs that work across multiple frames captured
by separate devices, since there might be skew between frames. The RTCP
SR based mechanism does work in those cases.
 
[From YK] For cases that work across multiple frames (taking purely
temporal scalability of SVC for example), the time-sync approaches
(including both the RTCP SR based method and the RTP timestamp aligned
method) do not work at all, unless you add some data to make aligned
frames - see the NI-T mode decoder order recovery method in the SVC
draft - which then become the same as cases what were talking about. 
 
BR, YK

	From: ext Colin Perkins [mailto:[email-masked]] 
	Sent: 03 September, 2008 11:14
	To: Wang Ye-Kui (Nokia-NRC/Tampere)
	Cc: [email-masked]; [email-masked];
[email-masked]; [email-masked]; [email-masked]
	Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as
WG item

		 
		BR, YK

			From: ext Colin Perkins
[mailto:[email-masked]] 
			Sent: 03 September, 2008 10:37
			To: Wang Ye-Kui (Nokia-NRC/Tampere)
			Cc: [email-masked]; [email-masked];
[email-masked]; [email-masked]; [email-masked]
			Subject: Re: [AVT] accepting
draft-lakaniemi-avt-rtp-evbr-02 as WG item

			On 3 Sep 2008, at 08:06, <[email-masked]>
wrote:

				Just an addition. My key point is that
lip-sync between different media types is not the same thing as
cross-layer (for layered encodings) or cross-channel (for multi-channel
encodings) or cross-description (for multiple description codings - MDC)
sync,  as also mentioned by others, e.g. Jonathan Lennox. The key
difference is that lip-sync operates on media streams independently
captured, independently encoded and independently decoded and only a
non-exact sync is needed to present the decoded samples together, while
the other operates on different portions of the same media captured at
exactly the same time, encoded in a connected way, and decoded in a
connected way and in many cases the decoding order must be exactly
synchronized between the different portions. A method itself that
applies to all layered, multi-channel and MDC codecs is already a
generic method. This method should be allowed unless the conventional
method for lip-sync is at least equally good . How, so far from the
discussions, in my understanding, using the RTCP SR based method is not
as good as using the aligned initial timestamp for
cross-layer/channel/description sync.

			On the contrary, the RTCP SR based method is the
only one that works in all cases. Using aligned initial timestamps for
synchronisation doesn't work if there is clock skew, and you cannot
guarantee that there won't be clock skew between the components of all
layered, multi-channel, and MDC codecs.
			[Wang Ye-Kui] This is not true. The aligned
initial timestamps works fine even if there is clock skew, as the skew
amount is the same for all portions of one frame/sample.  

	This only works if you're generating multiple layers from a
single frame - that's a very restricted case. For example, it doesn't
work for layered or MDC codecs that work across multiple frames captured
by separate devices, since there might be skew between frames. The RTCP
SR based mechanism does work in those cases.

	-- 
	Colin Perkins
	http://csperkins.org/

---

From: Thomas Schierl <[email-masked]>
Date: Wed, 03 Sep 2008 10:42:59 +0200

[email-masked] wrote:

[From YK] For cases that work across multiple
frames (taking purely temporal scalability of SVC for example),
the time-sync approaches (including both the RTCP SR based method and
the RTP timestamp aligned method) do not work at all, unless you add
some data to make aligned frames - see the NI-T mode decoder order
recovery method in the SVC draft - which then become the same as cases
what were talking about.

The non-interleaved timestamp based decoding order recovery (NI-T) mode
of the SVC draft works with NTP timestamp alignment! The NI-T mode just
guarantees that there is aligned data in the sessions.

--
Thomas Schierl
--------------
Fraunhofer HHI

----

Visit us at

more details

more details

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 11:52:40 +0300

Thomas,
 
What you said is no difference from what I said. After ensuring aligned
data in the sessions, the NI-T mode is the same as "generating multiple
layers from a single frame - that's a very restricted case". So thanks
for supporting my point. 
 
BR, YK

	From: ext Thomas Schierl [mailto:[email-masked]] 
	Sent: 03 September, 2008 11:43
	To: Wang Ye-Kui (Nokia-NRC/Tampere)
	Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]; [email-masked]
	Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as
WG item

		[From YK] For cases that work across multiple frames
(taking purely temporal scalability of SVC for example), the time-sync
approaches (including both the RTCP SR based method and the RTP
timestamp aligned method) do not work at all, unless you add some data
to make aligned frames - see the NI-T mode decoder order recovery method
in the SVC draft - which then become the same as cases what were talking
about. 

	The non-interleaved timestamp based decoding order recovery
(NI-T) mode of the SVC draft works with NTP timestamp alignment! The
NI-T mode just guarantees that there is aligned data in the sessions.

	-- 
	Thomas Schierl
	--------------
	Fraunhofer HHI

	----
	Visit us at
	
	IBC 2008 / Amsterdam (NL), 12 - 16 September 2008 / Hall 8 Booth
C81
	more details
<http://www.hhi.fraunhofer.de/index.php?id=2399&L=1> 
	
	insite 2008 / Johannesburg (South Africa), Sandton Convention
Centre / 15 - 17 September 2008
	more details <http://www.insitex.co.za>

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 10:05:29 +0100

> recovery. Point out please if there was one.

This thread started when I suggested that the EVBR draft use the  
standard RTCP SR based time synchronisation mechanism, rather than  
proposing a new, payload specific, variant. I said nothing about  
decoding order recovery, and it's not clear to me why you keep  
raising it in a thread about timing synchronisation.

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 12:30:45 +0300

>raising it in a thread about timing synchronisation.

The mentioning of the RTCP SR based method in the EVBR draft is for
decoding order recovery, not anything else. 

BR, YK

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 12:34:36 +0300

Hi Colin,

In the context of decoder order recovery, can the RTP timestamp aligned
method be used?

BR, YK 

>From: Wang Ye-Kui (Nokia-NRC/Tampere) 
>Sent: 03 September, 2008 12:31
>To: 'ext Colin Perkins'
>Cc: [email-masked]; [email-masked]; 
>[email-masked]; [email-masked]; [email-masked]
>Subject: RE: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as WG item
>
> 
>The mentioning of the RTCP SR based method in the EVBR draft 
>is for decoding order recovery, not anything else. 
>
>BR, YK
>

---

From: Stefan Döhla <[email-masked]>
Date: Wed, 03 Sep 2008 11:44:58 +0200

Colin Perkins wrote:
> /doesn't work/ if there is clock skew, and you cannot guarantee that 
For layered audio codecs you *must* make sure that there's no skew 
between the components - they are from a single source with a single 
clock! Otherwise this requires e.g. resampling for the bandwidth 
extension just to align the components.

In my understanding the components share a single media clock with at 
least identical skew. Whether or not the NTP timestamps drift doesn't 
matter but they also *must* reflect that there is no skew between the 
media clocks.
>
>> Therefore I don't get it why we have to bind ourselves to the RTCP SR 
>> based method. 
>
I think if you use RTCP only (without any additional signaling) you must 
be prepared at the receiver that the media components may drift. I don't 
think people will understand that they have to include media component 
synchronization (and adaption, i.e. resampling) on a frame by frame basis.

- Stefan

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 10:57:00 +0100

Ye-Kui,

No - if you want to base decoding order recovery on time  
synchronisation, you should base it on the existing standard time  
synchronisation mechanism, not invent another mechanism.

Colin

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Wed, 3 Sep 2008 13:13:52 +0300

Colin,

OK - this to me is saying that we MUST NOT specify any decoder order
recovery based on time synchronization not using the RTCP SR based
method, regardless of pros and cons. If this is fine with people, then
let's forget about the RTP timestamp aligned method (and those others
belonging to this category).

Also, my understanding is that for any improvement to the RTCP SR based
method, it MUST be generic to be qualified to be considered by AVT. 

BR, YK 

>Sent: 03 September, 2008 12:57

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 12:20:32 +0100

Ye-Kui,

> method, it MUST be generic to be qualified to be considered by AVT.

I might phrase it a little differently: time synchronisation of  
multiple RTP streams, for whatever purpose, must be done using the  
standard RTP mechanism based on RTCP SR packets. If that time  
synchronisation mechanism is found to have problems, those problems  
should be addressed in a general manner, not specific to any  
particular payload format, class of codecs, etc.

-- 
Colin Perkins
http://csperkins.org/

---

From: "Jonathan Lennox" <[email-masked]>
Date: Wed, 3 Sep 2008 17:08:20 -0400

Even if you have media captured from several different devices (as in
MDC video or multi-channel surround sound audio), I think that, by
definition, a layered or MDC codec has to be using a single encoding
process, which thus has to coerce or synchronize all its inputs to a
common media clock.  I don't see any way that you could have streams
with independently-running clocks and yet have decoding dependencies
between the streams.

-- 

Jonathan Lennox

Vidyo, Inc

[email-masked]

From: Colin Perkins [mailto:[email-masked]] 
Sent: Wednesday, September 03, 2008 4:14 AM
To: [email-masked]
Cc: [email-masked]; Jonathan Lennox; [email-masked];
[email-masked]; [email-masked]
Subject: Re: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as WG item

	BR, YK

			Just an addition. My key point is that lip-sync
between different media types is not the same thing as cross-layer (for
layered encodings) or cross-channel (for multi-channel encodings) or

		On the contrary, the RTCP SR based method is the only
one that works in all cases. Using aligned initial timestamps for
synchronisation doesn't work if there is clock skew, and you cannot
guarantee that there won't be clock skew between the components of all
layered, multi-channel, and MDC codecs.
		[Wang Ye-Kui] This is not true. The aligned initial
timestamps works fine even if there is clock skew, as the skew amount is
the same for all portions of one frame/sample.  

This only works if you're generating multiple layers from a single frame
- that's a very restricted case. For example, it doesn't work for
layered or MDC codecs that work across multiple frames captured by
separate devices, since there might be skew between frames. The RTCP SR
based mechanism does work in those cases.

-- 

Colin Perkins

http://csperkins.org/

---

From: "Jonathan Lennox" <[email-masked]>
Date: Wed, 3 Sep 2008 17:30:46 -0400

> to estimate the clock skew based on observing several SR's, and

I think this may be the disconnect we're seeing here.  It's fine for synchronization for lip sync to be estimated approximately sometime shortly after session startup, and slowly converge to greater precision over time as you receive more SRs.  Lip sync also degrades gracefully if the encoder's clocks are inaccurate or imprecise.

Layered codecs, by contrast, are *not decodable* until you've established the timing relationship between the layers, unless you use codec-specific in-band methods such as SVC's NI-C or I-C which avoid the use of timing information entirely.  Otherwise, if your receiver mis-calculates a timing relationship between streams you end up with garbage output from your decoder.

I think that for layered codecs, the requirements (100% accurate clock synchronization prior to any decoding) and the domain constraints (a single common media clock, as I argued in my previous e-mail) are sufficiently different from those of lip sync that it's not unreasonable to have a specific solution to the problem, rather than trying to re-use a solution to a different problem which has different requirements and constraints.
	
-- 
Jonathan Lennox
Vidyo, Inc
[email-masked]

---

From: <[email-masked]>
Date: Thu, 4 Sep 2008 01:23:24 +0300

Totally agreed.
 
BR, YK

	From: ext Jonathan Lennox [mailto:[email-masked]] 
	Sent: 04 September, 2008 00:08
	To: Colin Perkins; Wang Ye-Kui (Nokia-NRC/Tampere)
	Cc: [email-masked]; [email-masked];
[email-masked]; [email-masked]
	Subject: RE: [AVT] accepting draft-lakaniemi-avt-rtp-evbr-02 as
WG item

	Even if you have media captured from several different devices
(as in MDC video or multi-channel surround sound audio), I think that,
by definition, a layered or MDC codec has to be using a single encoding
process, which thus has to coerce or synchronize all its inputs to a
common media clock.  I don't see any way that you could have streams
with independently-running clocks and yet have decoding dependencies
between the streams.

	-- 

	Jonathan Lennox

	Vidyo, Inc

	[email-masked]

	Cc: [email-masked]; Jonathan Lennox;

		BR, YK

			From: ext Colin Perkins

	-- 

	Colin Perkins

	http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Wed, 3 Sep 2008 23:25:21 +0100

On 3 Sep 2008, at 22:30, Jonathan Lennox wrote:
> I think this may be the disconnect we're seeing here.  It's fine  
> for synchronization for lip sync to be estimated approximately  
> sometime shortly after session startup, and slowly converge to  
> greater precision over time as you receive more SRs.  Lip sync also  
> degrades gracefully if the encoder's clocks are inaccurate or  
> imprecise.
>
> Layered codecs, by contrast, are *not decodable* until you've  
> established the timing relationship between the layers, unless you  
> use codec-specific in-band methods such as SVC's NI-C or I-C which  
> avoid the use of timing information entirely.  Otherwise, if your  
> receiver mis-calculates a timing relationship between streams you  
> end up with garbage output from your decoder.

If there's no clock skew, then RTCP SR packets give exact  
synchronisation once the first SR has been received for each RTP  
session.

> I think that for layered codecs, the requirements (100% accurate  
> clock synchronization prior to any decoding) and the domain  
> constraints (a single common media clock, as I argued in my  
> previous e-mail) are sufficiently different from those of lip sync  
> that it's not unreasonable to have a specific solution to the  
> problem, rather than trying to re-use a solution to a different  
> problem which has different requirements and constraints.

The only limitation of the RTCP SR mechanism seems to be start-up  
latency. This can be reduced in the common case for unicast by  
sending immediate SR packets (which is allowed by RFC 3550 today,  
although perhaps not widely implemented). Clearly documenting that,  
and specifying an AVPF SR request packet, would seem to solve the  
start-up latency issue for layered coding, without needing an entire  
new synchronisation mechanism.

---

From: <[email-masked]>
Date: Thu, 4 Sep 2008 01:26:24 +0300

If we may still discuss this issue - I would say I fully agree with Jonathan. 

BR, YK 

>Sent: 04 September, 2008 00:31
>To: Colin Perkins
>Cc: Stefan Döhla; [email-masked]; Wang 
>Ye-Kui (Nokia-NRC/Tampere); [email-masked]
>I think this may be the disconnect we're seeing here.  It's 
>fine for synchronization for lip sync to be estimated 
>approximately sometime shortly after session startup, and 
>slowly converge to greater precision over time as you receive 
>more SRs.  Lip sync also degrades gracefully if the encoder's 
>clocks are inaccurate or imprecise.
>
>established the timing relationship between the layers, unless 
>you use codec-specific in-band methods such as SVC's NI-C or 
>I-C which avoid the use of timing information entirely.  
>Otherwise, if your receiver mis-calculates a timing 
>relationship between streams you end up with garbage output 
>from your decoder.
>
>I think that for layered codecs, the requirements (100% 
>accurate clock synchronization prior to any decoding) and the 
>domain constraints (a single common media clock, as I argued 
>in my previous e-mail) are sufficiently different from those 
>of lip sync that it's not unreasonable to have a specific 
>solution to the problem, rather than trying to re-use a 
>solution to a different problem which has different 
>requirements and constraints.
>	
>-- 
>Jonathan Lennox
>Vidyo, Inc
>[email-masked]
>
>

---

From: [email-masked]
Date: Thu, 04 Sep 2008 09:21:22 +0200

Colin and all,

Fast zapping is very important for Live program in multicast.

So, synchronisation latency must be shorter for multicast  video than
for unicast, and I do not see now how it is possible to avoid a fast
synchronization mechanism.

Colin Perkins a écrit :

---

From: Colin Perkins <[email-masked]>
Date: Thu, 4 Sep 2008 09:50:37 +0100

Gerard,

Sure - I understand. That's why I suggested an AVPF SR request  
packet, since that can be used as a (backwards compatible) fast  
synchronisation mechanism.

Colin

On 4 Sep 2008, at 08:21, [email-masked] wrote:
> So, synchronisation latency must be shorter for multicast  video  
> than for unicast, and I do not see now how it is possible to avoid  
> a fast synchronization mechanism.
>
>
> Colin Perkins a écrit :
>>
>>

---

From: Thomas Schierl <[email-masked]>
Date: Thu, 04 Sep 2008 13:40:11 +0200

There may be also channels without feedback capability as, e.g. DVB-H.

As one solution for such broadcast channels, a video sender may send 
RTCP SRs with each random access point in the video. If that should 
overflow the RTCP bandwidth limits, AVPF bandwidth modifiers may be used.

-- 
Thomas Schierl
--------------
Fraunhofer HHI

Colin Perkins wrote:
> Gerard,
>
> Sure - I understand. That's why I suggested an AVPF SR request packet, 
> since that can be used as a (backwards compatible) fast 

----
Visit us at

---

From: Thomas Schierl <[email-masked]>
Date: Thu, 04 Sep 2008 17:34:47 +0200

Jonathan,

I also agree with you for the live case. For the pre-encoded case, there 
is typically no deviation in the clock contained in the file and the 
media frames do have equidistant timings. Having such an accurate clock 
in the file does help to "tweak" the SR's NTP wallclock value in a way 
that it can be "forced" to be correct. As an example, you may take the 
timing from the file as a scheduler time and put this time + a delta as 
wallclock into your SR, while the SR is actually sent at the 
corresponding gettimeofday() value, which may have deviation from the 
scheduled (wallclock) time.

For the live case, i.e. you have to rely on the system's wallclock only, 
it's different. Your example was the deviations of the win32 get-time 
functions. In such a case with clock deviations around 10ms, there is 
indeed a problem! Even using other clocks as the one of the sound card 
or the one of the video capturer for the wallclock may not really help here.

Possible backward-compatible solutions (assuming the same clock is used) 
may be RTP timestamp alignment or sending RTCP SRs in the multiple 
sessions at the same time including the same NTP wallclock. Everything 
else may lead to inaccuracy in NTP media timestamp generation at the 
receiver. But this seems to be rather an implementation issue than a 
standardization issue. Although this issue may affect the standardization.
I think we should discuss this problem in detail in the draft and show 
what can be done, if no exact clock is available. Even, if not all 
systems are affected, there must be a way to work around such a problem.
Since e.g., the two above proposed solutions would extend the current 
RTP specification for the case of multi session transmission, we do not 
have an informational draft on "multi session transmission issues". It 
would be a normative one, right?

-- 
Thomas Schierl
--------------
Fraunhofer HHI

----
Visit us at

---

From: Randell Jesup <[email-masked]>
Date: Thu, 04 Sep 2008 12:16:28 -0400

Colin Perkins <[email-masked]> writes:

>>>> There are existing standard systems (e.g. 3GPP) using RTP without  RTCP
>>>> - I know that this has been discussed in AVT and those  systems might
>>>> be considered as "broken" by some. However, the  systems do work and
>>>> are deployed. The timestamp aligned method  works even for such
>>>> don't where a real problem is for the  timestamp aligned method.
>>>
>The problem can still exist in those sessions, depending on how the  media
>is captured.

In case it's not clear, Colin is referring to cases such as where multiple
audio streams are captured by separate hardware samplers.  That means the
sample rates of the two streams can skew.

>>> There are two additional assumptions being made:
>>>
>No, but there is also no way for a receiver to know that both  conditions
>are satisfied, so it must assume the worst case when  resynchronising.

I was pointing out that Ye-Kui's description of an implementation requires
at least one of those two things to be true - and that the receiver knows
that is the case (knows to assume it - and it has to know which).  The only
case where you don't have to assume those is if you *know* you won't lose
any packets (off the front of the streams).