Subject: [AVT] alignment of layers issue

From: "Ingemar Johansson S" <[email-masked]>
Date: Tue, 18 Nov 2008 00:07:29 +0100

Hi

Given the discussion around alignment of layers and the document 
draft-perkins-avt-rapid-rtp-sync-00
I can't get rid of the feeling that there is a probablity that layers may not be aligned of RTCP-SR based suchronization is used. 
I am not that worried about clock-skew. The problem I see is rather the update rate of clocks that are used for the NTP. If the update rate is 15msec you already have a relatively high probability that layers may slip (the slip may only be occasional and initial and can probably be fixed in the long run using some magic algo that irons out the 15msec update). 

But that put aside. How does the multi-anything decoder know that one or more layers (or views) are misaligned ?. 
I can't tell for SVC, MVC and the others but I fail to see how this verification is possible in G.718. The user will experience crap audio but how does the decoder know that one or more layers are misaligned ?. 

Regards
Ingemar

******************************************* 
Ingemar Johansson 
Senior Research Engineer, IETF "nethead" 
EAB/TVP - Multimedia Technologies 
Ericsson Research Ericsson AB 
Box 920 S-971 28 Luleå, Sweden 
Tel: +46 (0)8 4043042 
ECN: 850-43042 
Mobile: +46 (0)730 783289 
******************************************* 
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 17 Nov 2008 18:34:59 -0600

> I can't tell for SVC, MVC and the others but I fail to see how this verification is possible in G.718. The user will experience crap audio but how does the decoder know that one or more layers are…

I don't think this really is an issue if you correctly implement the
synchronization. You can completely avoid to have misalignment between
layers in the time-domain with the exception for any clock skew updates.
 Where such could exist in the time between the first RTCP SR packet
with an update and the last layer having received the corresponding update.

Thus there are some consideration that clock skew updates should not be
done to often and maybe some implementation tips would be to hold off
adjusting a multi-layered until SR packets with updates has been
received for all layers. This do work as clock skew fixing is only
adjusting its relation to other media streams, not the internal process.

Regarding how to generate correct RTCP SR packets this is how you do it:

At start up of codecs you take one clock value for the  media clock TS_O
and the corresponding system clock NTP_O. This reference is going to be
used for all layers, i.e. in all RTP Sessions for this media source.

When sending an RTCP SR at time p, the sender gets the current system
clock value NTP_P. It doesn't sample the media clock to determine the
RTP TS value, instead this is calculated based on the reference value.

TS_P= (NTP_P-NTP_O)/TS_RATE + TS_O

To determine clock skew you need to regularly sample system clock and
media clock and use that to determine if you have significant clock skew
that needs adjustments. This should be low pass filtered and probably
put on threshold to avoid unnecessary updates. Only when the clock skew
affects inter media synchronization to such a degree that issues arise
or the buffer handling becomes problematic should this be updated, i.e.
at least 20-50 ms error before reacting. Certain applications can take
much bigger errors before reacting.

If you have clock skew that becomes an adjustment parameter in the above
calculation, or an update of the NTP_O TS_O binding.

When using the above method you will not have an issue that the
synchronization information in the SR will depend on inaccuracy of the
system clock.

Cheers

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Thomas Schierl <[email-masked]>
Date: Tue, 18 Nov 2008 11:27:07 -0600

> adjusting its relation to other media streams, not the internal process.

But dealing with the updates seems to become really complicated. What 
would this mean? You would have to inform the receiver, that an update 
was going on in the binding of RTP and NTP. This would mean the receiver 
would have to get all "updated" RTCP reports in the different sessions 
first, before it can make use of it. So, another extra signaling?

Anyway, let us assume for the time being, we can make this work with a 
relatively complex implementation in the sender as well as in the receiver.

I still ask myself and I guess also other people do so, why not doing 
this little change towards "RTP timestamp alignment" for data of the 
same layered sample in the multiple sessions? It's so easy and the only 
real argument against it was the security issue which I guess was 
discussed earlier. With this, I further assume we can keep the SSRC 
alignment of the sessions in the session multiplex and we can also make 
sure, that using the same timescale in the different sessions is ok.

The thing what I would wish to have is a simple and robust mechanism, 
where I do not have to tweak timestamps and where I do not have to have 
extra signaling and where I do not have to care about lost or late 
packets and where I just can take two numerical values from the packets 
and know whether this is of the same sample or not. No rounding problems 
at all. The mechanism is there and it comes for free!

Remember for multi session transmission, the alignment of data is not 
required after decoding as for lip-sync, it is already required before 
you can start decoding.  And, there are not so many implementations 
which gets the synchronization right. For lip-sync it is just an 
quality-of-implementation issue. For data alignment of layered codecs, 
it becomes an correctness issue for the implementation.

Thomas

-- 
Thomas Schierl
--------------
Fraunhofer HHI

----
Visit us at

MEDICA 2008 / Duesseldorf, Germany / 19 - 22 November 2008 / Hall 16, Booth D55
http://www.medica.de

SOCCEREX 2008 / Gauteng, South Africa / 23 - 26 November 2008 / Booth V1-B1 a
http://www.soccerex.com

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 18 Nov 2008 14:53:22 -0600

Hi,

See inline

> first, before it can make use of it. So, another extra signaling?

No, a receiver can detect when the reference point by comparing data
from previous SR packet and the present if there has been a change. So
in the case of multi-layer the receiver continues to use the old
synchronization information until it has received changes on all media
streams.

> relatively complex implementation in the sender as well as in the receiver.

This is not complex, it is standard RTCP that you anyway need to
implement, with the exception of handling of changes that is new for
multi-layer.

> sure, that using the same timescale in the different sessions is ok.

The reason is that you anyway need to do it. You can't handle this in a
non-layered codec that is synched with something else without these
mechanism, so what is the difference between synching audio and video
with synching within a single media between the layers.

> at all. The mechanism is there and it comes for free!

I agree that the mechanism is in place and exist, it is called RTCP SR!
> it becomes an correctness issue for the implementation.

I do understand why if one has a single media that it appears that
aligning the timestamps solves all the problems. Well that isn't the
truth, as you anyway need to ensure that you bind the correct SSRC in
each session through CNAME to the right original source. So if you want
this to be possible to use you will require also SSRC alignment across
all sessions.

Yes, this has been done before, however it doesn't come for free and has
a latent problem if you have sources that aren't using the layered codec
and don't think about these restrictions. So by accepting this as a
general rule will result in restriction to the RTP model.

So I do want to question if we really should do this restriction which
only provides a benefit in the single media case. As soon you have more
than one media or streams that needs synchronization you anyway need to
use RTCP.

Cheers

Magnus Westerlund

---

From: "Jonathan Lennox" <[email-masked]>
Date: Tue, 18 Nov 2008 17:43:29 -0500

> synchronization information until it has received changes on all media

This would mean that as a receiver of RTCP, when I receive an SR with an
RTP/NTP mapping significantly offset from the previous one, I would need
to figure out whether a) this is an SR from a sender which implements
this "change the mapping only when clock skew gets too big" algorithm,
in which case I need to tell the decoding layer to ignore this SR for
the time being; or whether b) it's an SR generated by an implementation
using the standard naive gettimeofday()-based implementation, in which
case I need to proceed with it and pass it to the presentation layer as
normal.

This is certainly very different from what any existing RTCP receiver
would do -- today, when you receive an SR, you update your clock
association, without any magic knowledge about your peers or
cross-session correlation.

I'm also concerned that if a sender with a large clock skew is sending
to a session with a large RTCP interval (due to low session bandwidth or
a large sender population), it could be that the timestamp mapping would
need to get changed more often than SRs are sent, so this algorithm
wouldn't work.

I'm curious, also, how many existing implementations actually get this
right.  Every implementation I've found of SRs either a) uses
gettimeofday() or equivalent, and is thus subject to the whims of system
clock precision, or b) establishes an NTP/RTP mapping once at startup
and never changes it, making SRs useless for lip sync if there's any
clock skew at all.

> with synching within a single media between the layers.

The difference is that for lip sync, time accuracy is a
quality-of-implementation issue.  A 30 ms inaccuracy in lip sync, due to
imprecision in the sender's clocks, might produce a degraded user
experience, but in general, the stream is still usable.

However, for data alignment, timestamp accuracy is a correctness issue.
For the layered codecs I know about, if you attempt to decode having
mis-aligned the base and enhancement layers, the decoder will output
unusable gibberish.

> this to be possible to use you will require also SSRC alignment across

Yes, as specified in RFC 3550 (section 8.3).

Also, CNAME-based stream association doesn't work if a single
participant (with a single CNAME) is sending several different
(independent) layered streams in the same session group, e.g. from
multiple cameras.  Thomas and I discuss this in our draft.

> general rule will result in restriction to the RTP model.

I don't understand this paragraph at all.  This is only being proposed
for layered codecs.

> than one media or streams that needs synchronization you anyway need
to

No one is suggesting not using RTCP for all the things it's currently
used for -- lip sync, statistics reporting, etc.  We're just saying that
it has enough limitations, at least as implemented in almost all
existing implementations, that it shouldn't be extended to this new
domain.

-- 
Jonathan Lennox
Vidyo, Inc
[email-masked]

---

From: Ross Finlayson <[email-masked]>
Date: Tue, 18 Nov 2008 17:55:40 -0600

At 2:53 PM -0600 11/18/08, Magnus Westerlund wrote:
>implement

It seems to me that there's "an elephant in the room" here.  I can't 
help but feel that many (though not all) of the people who are 
opposing adopting the standard RTCP SR-based solution are really 
doing so because they don't want to implement RTCP.  I.e., they want 
to get away with implementing only a restricted version of RTP that 
works just for one particular layered video codec (and perhaps also 
one particular audio codec for which they would also align 
timestamps), without RTCP at all.

If this is really true, then it would be useful if people would come 
clean about this, so that we can address, or put to rest, whatever 
fears/concerns they may have about implementing RTCP.

	Ross.

---

From: Colin Perkins <[email-masked]>
Date: Wed, 19 Nov 2008 00:21:00 +0000

> Magnus Westerlund writes:
...
> existing implementations, that it shouldn't be extended to this new

The point, for which I agree with Magnus, is that you need to  
implement the RTCP SR-based mechanism anyway for lip-sync. If you  
implement it correctly, it also solves the synchronisation problem  
for layered codecs. Rather than spending all this energy inventing  
yet another different and incompatible synchronisation mechanism,  
spend the energy documenting how to implement the existing standard  
solution.

-- 
Colin Perkins
http://csperkins.org/

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 18 Nov 2008 18:34:28 -0600

> cross-session correlation.

Yes, but then you aren't really using a multi-layer codec either today
either. The added requirement of maintaining the synch between the
different layers is the thing that makes this a bit more requiring than
earlier.

> need to get changed more often than SRs are sent, so this algorithm

I agree that there are some corner cases in this. I noticed the issue
when writing the previous email.

> and never changes it, making SRs useless for lip sync if there's any

Very good question. I am not really running any real systems out there.
For most application B is actually better than A, it might affect the
buffering badly for low delay applications. I guess most streaming
servers never care to adjust anything.

> mis-aligned the base and enhancement layers, the decoder will output

Yes, fully agree. The synch data between the different layers need to
really provide sample exact alignment. That is possible with SR as long
as you use a common synch point.

> multiple cameras.  Thomas and I discuss this in our draft.

Okay, I think this is a something that needs clarification. However, it
is not obvious that the different cameras shouldn't how different
CNAMEs. I think that depends if you actually can provide them in a
synchronous manner or not. You anyway need to have a mechanism to
distinguish between the layers what is belonging together.

> I don't understand this paragraph at all.  This is only being proposed

If you have payload types of this type you forces all sources in the RTP
session to take this into consideration independently if they are
multi-layer or not. The RFC 4588 (Retransmission format) that has SSRC
alignment is very unlikely to get in its RTP session multiplexing mode
to get anything else than retransmission packet.

For a media codec that is not as true.

> existing implementations, that it shouldn't be extended to this new

Okay, but it seems to a large degree be a implementation issues. There
seem to be some shortcomings to clock skew adjustments for the SR. That
I think needs more analysis. But I think if we are producing documents
in this area we really should clear up the implementation issues so that
people make it work correctly. Because if you don't fix them correctly,
even with timestamp and SSRC alignment over the RTP session you will get
 the error in generating the SR packet results in a erroneous clock skew
correction in a receiver. Which will also affect quality, especially for
audio.

Cheers

Magnus Westerlund

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Wed, 19 Nov 2008 17:00:05 +0100

Hi
 
After some excel sheet hacking I have finally come to the point where I believe that the  method proposed by Magnus (see below) would actually solve the issue with slow update rate or inaccurate NTP clocks. 
Reading RFC3550 (6.4.1) again one could well find the below method embedded in the text and then again one can find another way to implement this especially as the text says 
"RTP timestamp: 32 bits
Corresponds .......
......Note that
in most cases this timestamp will not be equal to the RTP
timestamp in any adjacent data packet. Rather, it MUST be
calculated from the corresponding NTP timestamp using the
relationship between the RTP timestamp counter and real time a
maintained by periodically checking the wallclock time at a
sampling instant."

The "in most cases" statement leaves the door open for many alternative implementations. I would here suggest that any new document that describes layer alignment and synchronization makes it very clear what and RTP stack implementation should do. 

Regards
Ingemar
 
Excert from Magnus's email below 
=========================================

---

From: Thomas Schierl <[email-masked]>
Date: Wed, 19 Nov 2008 10:26:30 -0600

> Reading RFC3550 (6.4.1) again one could well find the below method embedded in the text and then again one can find another way to implement this especially as the text says 
But some implementations may like to make an assumption which is not 
directly given by the statement below.
> The "in most cases" statement leaves the door open for many alternative implementations. I would here suggest that any new document that describes 
I am not sure if this statement is applicable to this.
> layer alignment and synchronization makes it very clear what and RTP stack implementation should do. 
>   
I think we cannot change the RTP stack implemenation here, that has to 
be the same for all implementations including RTCP for deriving NTP 
timestamps. But we may give a guideline on the RTP timestamp generation 
for layered codecs used with session multiplexing, which may be used to 
"improve" the data alignment process or may make it a little bit easier 
for some implementations. RTCP with NTP stays and is mandatory to be 
used, however an layered receiver may also rely on the fact the RTP 
timestamps are aligned in the different sessions in this particular case.

Could that be a way to go forward?

Thomas

-- 
Thomas Schierl
--------------
Fraunhofer HHI

----
Visit us at

---

From: Colin Perkins <[email-masked]>
Date: Wed, 19 Nov 2008 16:58:11 +0000

On 19 Nov 2008, at 16:26, Thomas Schierl wrote:
...
> I think we cannot change the RTP stack implemenation here, that has  
> to be the same for all implementations including RTCP for deriving  
> NTP timestamps. But we may give a guideline on the RTP timestamp  
> generation for layered codecs used with session multiplexing, which  
> may be used to "improve" the data alignment process or may make it  
> a little bit easier for some implementations. RTCP with NTP stays  
> and is mandatory to be used, however an layered receiver may also  
> rely on the fact the RTP timestamps are aligned in the different  
> sessions in this particular case.
>

So, your "improvement" is that layered decoders should ignore the  
RTCP SR information, and assume the timestamps are aligned, but you'd  
send the SR data anyway? Sorry, no, that's just another way of saying  
you don't want to implement RTCP.

Guidelines on how to generate SR packets with correct timestamps, sure.

-- 
Colin Perkins
http://csperkins.org/

---

From: Randell Jesup <[email-masked]>
Date: Wed, 19 Nov 2008 14:24:28 -0500

>servers never care to adjust anything.

B is often implemented - but is WRONG, unless the application *knows* the
two streams share a clock base, and can never shift in relative sync no
matter what.  The might be the case for some non-live audio/video sources.

gettimeofday() coarseness - this is an issue (idiots who built the old PC
hardware (and DOS/Windows) and the bigger idiots who let that remain the
default for 25+ years...).

However, 16ms coarseness on the order of 2.5-7.5s updates would be in the 
range of 0.64%-0.213%, or at most just over 1/2 of a percent.  Relevant but
not major, and it should only come into play for sync.

Certain applications and UAs, though (*cough*Asterisk*cough*) can send you
RTCP packets that imply huge, wild shifts in NTP<->timestamp rates, like an
RTCP that says 40ms, or even 0ms(!), or -120ms (!!!!!)  have occured since
the last RTCP, but 40000 (or 450000) samples have occurred.

There are a few real-world cases to watch out for with system-time calls,
like what happens if a small NTPd adjustment is being made periodically, or
a large (seconds, minutes) NTPd-derived time change occurs during the life
of the stream, or if the user changes the system time (on a
non-NTPd-enabled system), or at daylight savings time (which *should* have
no effect if you're using TZ's properly, but could be bad if someone is
handling DST by hand.  These can even bite case B (with a much smaller
window), since two streams will sample gettimeofday() at different points,
potentially at different sides of a clock change.

Ugh.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Thu, 20 Nov 2008 17:11:23 +0100

Hi
 
My comment (based on Randell's real life observation below) is that I really don't see the conflict between RTCP SR and RTP timestamp alignment as proposed by Thomas and Jonathan. 
 
I believe nobody wants to get rid of RTCP SR (or RTCP whatever). RTP timestamp alignment is simply to be seen as an enhancement to make sure that sample (or decoding block) alignment is ensured with 100% probability for multi-layer/view/channel codecs. RTCP (SR) is needed for lipsync with other media and SSRC collision detection. My gut feeling is that, given all the possible implementation aspects with RTCP SR that there are simply too many loose ends to make it trustworthy for the exact alignment requirement with e.g layered codecs.
 
Also.. (IMHO very important). As it may be really tricky (or impossible) for the decoder to know if layers are aligned the "alignment device" must guarantee a near 100% successrate. 
The RTP timestamp alignment proposal can in my opinion guarantee 100% success rate. 
RTCP SR: I can't help getting the feeling that the failure rate may in some cases be quite high (depending on a whole lot of implementation issues). Additionally, what if there is an RTCP SR alignment tune in period of several seconds on session start of when the channel is switched ?. Is it acceptable to just decode the base layer for that long period, one cannot take the chance to add the enhancement layers until it is deemed near 100% safe to do so. And a last one...when is RTCP SR alignment tune in done and safe ?, as the receiver side doesn't really have a reference it must rely on some metrics that with a given probability tells that the alignment is correct and this by its nature introduces an element of uncertanity.
 
My 99 cents
/Ingemar

________________________________

Från: Randell Jesup [mailto:[email-masked]]
Skickat: on 2008-11-19 20:24
Till: Magnus Westerlund
Kopia: Jonathan Lennox; Ingemar Johansson S; [email-masked]
Ämne: Re: [AVT] alignment of layers issue

Ugh.

---

From: "Roni Even" <[email-masked]>
Date: Thu, 20 Nov 2008 23:45:10 +0200

Ross,
I want to believe that this is not the case. At least for video codec you
need RTCP for other reasons as defined in RTCP feedback and the extensions
in the ccm draft.
Roni Even

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross
Finlayson
Sent: Wednesday, November 19, 2008 1:56 AM
To: [email-masked]
Subject: Re: [AVT] alignment of layers issue

	Ross.

---

From: "Ver Steeg, Bill" <[email-masked]>
Date: Fri, 21 Nov 2008 09:26:00 -0500

Ross/Roni

I also believe that there are good reasons to implement the full RTP
suite, particularly for video flows. However, we are "preaching to the
choir" here. The readers of this list are a self selecting crowd. The
folks who do not use RTP for video flows do not read this list, and
probably do not attend the IETF.

It would be interesting to ask other SDOs (like CableLabs or W3C) to
comment on "the elephant in the room". It may even be more instructive
to ask implementers who operate outside of standards groups their
opinion on this matter. I am not sure how we solicit that data, though. 

I suspect that most applications programmers do not know that RTP
exists, much less understand its merits. In many cases, it is these
applications programmers that decide what encapsulation to use for their
data flows.

Bill VerSteeg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Roni Even
Sent: Thursday, November 20, 2008 4:45 PM
To: 'Ross Finlayson'; [email-masked]
Subject: Re: [AVT] alignment of layers issue

Ross,
I want to believe that this is not the case. At least for video codec
you need RTCP for other reasons as defined in RTCP feedback and the
extensions in the ccm draft.
Roni Even

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ross Finlayson
Sent: Wednesday, November 19, 2008 1:56 AM
To: [email-masked]
Subject: Re: [AVT] alignment of layers issue

It seems to me that there's "an elephant in the room" here.  I can't
help but feel that many (though not all) of the people who are opposing
adopting the standard RTCP SR-based solution are really doing so because
they don't want to implement RTCP.  I.e., they want to get away with
implementing only a restricted version of RTP that works just for one
particular layered video codec (and perhaps also one particular audio
codec for which they would also align timestamps), without RTCP at all.

	Ross.

     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

---

From: Thomas Schierl <[email-masked]>
Date: Fri, 21 Nov 2008 10:00:03 -0600

I have the feeling, that you as well as Ross and unfortunately also
others completely misunderstood the discussion we had up to now. No one
ever said "do not use RTCP". RTCP must be used what it is built for, as
for inter-session lip-sync. What we did was to raise the question,
whether this also works under all possible circumstances (available
clocks on today's PC systems) for sample-exact alignment of data
between sessions. Making this to a discussion on "using or not using
RTCP", does not make any sense and was absolutely not our intention.

Thomas

Ver Steeg, Bill wrote:

Ross/Roni

Bill VerSteeg

-----Original Message-----

Ross,

-----Original Message-----

> Anyway, let us assume for the time being, we can make this work
with a

relatively complex implementation in the sender as well as in the

receiver.

Ross.

----

Visit us at

http://www.medica.de

http://www.soccerex.com

---

From: <[email-masked]>
Date: Fri, 21 Nov 2008 17:38:02 +0100

Hi

I want first to remenber I wrote in the past that layer alignment with RTCP/SR may need a long time that providers get irritated with for hight quality IPTV programs. This is also, in a very different way, the main objective of draft-versteeg-avt-rapid-synchronization-for-rtp-

Even if RTCP/SR are possible in multicast sessions, the problem of a rapid layer alignment persits because RTCP and RTP are indeed companion session, but independant in sending messages.

In my understanding, layer alignment (and synchronisation) is more strictly a RTP problem rather than a control of the session (in my RTCP understanding). So, an efficient synchronisation process could be a layer alignment precisely when starting the send of a IDR. RTP according to the payload content has enough information to do it. Thus, a receiver can immediately decode multiple layers from the first IDR it receives.

So timestamp alignment is from a long way the best. Even if timestamp is random, I can convince that hazardously an encoder or a svc video source can set the same timestamp in at least two sessions relating to a unique video encoding. I see no reasons that could stop RTCP in a such case.

However, if timestamp alignment is fine for SVC, my very little problem is that I do not clearly understand how this can run to synchronize sessions from several sources that cannot share the same timestamp. The perfect should be another generic solution to invent, that could apply to every multi-session encoders.

Best regards,

Gérard BABONNEAU 

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]] De la part de Ingemar Johansson S
Envoyé : jeudi 20 novembre 2008 17:11
À : Randell Jesup; Magnus Westerlund
Cc : Ingemar Johansson S; Jonathan Lennox; [email-masked]; [email-masked]; [email-masked]
Objet : Re: [AVT] alignment of layers issue

Hi

 
 
My 99 cents
/Ingemar

Magnus Westerlund <[email-masked]> writes:

>> I'm curious, also, how many existing implementations actually get 
>> this right.  Every implementation I've found of SRs either a) uses
>> gettimeofday() or equivalent, and is thus subject to the whims of 
>> system clock precision, or b) establishes an NTP/RTP mapping once at 
>> startup and never changes it, making SRs useless for lip sync if 
>> there's any clock skew at all.
>

B is often implemented - but is WRONG, unless the application *knows* the two streams share a clock base, and can never shift in relative sync no matter what.  The might be the case for some non-live audio/video sources.

gettimeofday() coarseness - this is an issue (idiots who built the old PC hardware (and DOS/Windows) and the bigger idiots who let that remain the default for 25+ years...).

However, 16ms coarseness on the order of 2.5-7.5s updates would be in the range of 0.64%-0.213%, or at most just over 1/2 of a percent.  Relevant but not major, and it should only come into play for sync.

Certain applications and UAs, though (*cough*Asterisk*cough*) can send you RTCP packets that imply huge, wild shifts in NTP<->timestamp rates, like an RTCP that says 40ms, or even 0ms(!), or -120ms (!!!!!)  have occured since the last RTCP, but 40000 (or 450000) samples have occurred.

There are a few real-world cases to watch out for with system-time calls, like what happens if a small NTPd adjustment is being made periodically, or a large (seconds, minutes) NTPd-derived time change occurs during the life of the stream, or if the user changes the system time (on a non-NTPd-enabled system), or at daylight savings time (which *should* have no effect if you're using TZ's properly, but could be bad if someone is handling DST by hand.  These can even bite case B (with a much smaller window), since two streams will sample gettimeofday() at different points, potentially at different sides of a clock change.

Ugh.

--
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team [email-masked] "The fetters imposed on liberty at home have ever been forged out of the weapons provided for defence against real, pretended, or imaginary dangers from abroad."
                - James Madison, 4th US president (1751-1836)

---

From: Franceschini Guido <[email-masked]>
Date: Fri, 21 Nov 2008 17:48:51 +0100

Hi all,

I can present myself as a co-implementer of an RTP/RTCP suite.
I do use RTCP, base my lip-sync on RTCP and have no troubles with it.
However, when I generate RTCP reports my wallclock-RTPTS mapping is not exact, but suffers from the imprecision of the getTimeXXX functions.
Similarly, when I examine the RTCP reports I receive from other sources, I notice similar issues.
Such imprecision is a don't care for lip-sync.
But it becomes a must care for the layering stuff.
This is a key difference that, by the way, invalidates my current RTCP implementation (and I guess many others).

In my understanding, layered codecs need some exact correlation strategy. Timestamp can be used for that task only if they can be computed and matched exactly. Stretching the RTCP SR to cover this requirement sounds ineffective to me.
The alternative proposed so far, i.e. forcing all layers to share the same TSoffset, is preferable, although I don't like it that much. Indeed, I think that in theory (maybe just in theory) some or all layers could go through "recasting" devices, and such devices might be media agnostic and apply new TSoffsets, thus breaking the alignment constraint. Probably this could be ameliorated by explicit signaling.
For similar reasons (intermediate devices) I believe that any effort on the RTCP SR is prone to corner cases that would invalidate its effectiveness, in addition to clock precision problems on the peer themselves.

I haven't seen in this thread a discussion considering the carriage of an additional, explicit field to correlate the layers. This field could even contain a copy of the original TS itself, or any other "exact" information. Maybe this possibility was already explored and rejected. I can imagine that for the SVC case the requirement of backward compatibility with AVC prevents adding extra fields in the Payload Header of the base layer. Also, the desire of a common solution for the various layered codecs votes against Payload Header specificities. However, did you explored the less efficient but still available usage of RTP Header Extensions? Would this be a viable alternative? I have no experience/knowledge about RTP Header Extension and their management in intermediate devices, though.

BR
Guido

------------------------------------------------------------------
Telecom Italia
Guido FRANCESCHINI
Broadband Solutions and Multimedia Innovation
Via Guglielmo Reiss Romoli, 274 10148 Torino
office   +39 011 2286137
mobile +39 335 7669748

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ver
> Steeg, Bill
> Sent: venerdì 21 novembre 2008 15.26
> To: Roni Even; Ross Finlayson; [email-masked]
>      - - - - -                              Cisco
> - - - - -

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

Rispetta l'ambiente. Non stampare questa mail se non e' necessario.

www.avoicomunicare.it      Ogni giorno, il tuo luogo di dialogo.
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Colin Perkins <[email-masked]>
Date: Mon, 24 Nov 2008 23:35:21 +0000

On 21 Nov 2008, at 16:48, Franceschini Guido wrote:
> However, when I generate RTCP reports my wallclock-RTPTS mapping is  
> not exact, but suffers from the imprecision of the getTimeXXX  
> functions.
> This is a key difference that, by the way, invalidates my current  
> RTCP implementation (and I guess many others).

I understand that clock skew is an issue for some naive  
implementations that don't measure clock skew and compensate, but  
it's not clear what you mean by the "imprecision" of the getTimeXXX  
functions?

-- 
Colin Perkins
http://csperkins.org/