Subject: [rmcat] Send timestamps

From: Stefan Holmer <[email-masked]>
Date: Tue, 4 Dec 2012 10:23:10 +0100

Hi,

In draft-alvestrand-rtcweb-congestion-01<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-01>
we
proposed using a send timestamp RTP header extension for measuring the
packet inter-arrival time offset. In
02<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-02>
this
was revised to instead reference RFC 5450<http://tools.ietf.org/html/rfc5450>,
which specifies a header extension for specifying the send timestamp as an
offset relative to the RTP timestamp.

Thinking further about this, there seems to be a couple of benefits with
defining a new, absolute, send timestamp header extension. If a relay
approach to conference servers are used, the relaying server can simply
rewrite the send timestamp of a packet to correspond to the time it was
relayed. Doing so the receiver can jointly process all streams being
received from the relay server, even though they were initially captured
and sent by different senders (with different clocks).

With a send timestamp offset, accomplishing the same thing would require
rewriting both the timestamp offset and the NTP time in the corresponding
RTCP SR reports. And to keep video synchronized with audio the RTCP packet
for the audio streams must also be rewritten in a similar way. This may not
even be possible if the audio stream is relayed through a different server.

Do you see any problems with using an RTP header extension with absolute
send timestamps? Are there any benefits with the send timestamp offset
header extension I might be missing?

/Stefan

---

From: John Leslie <[email-masked]>
Date: Tue, 4 Dec 2012 12:27:28 -0500

> header extension I might be missing?

   I must admit to failing to understand what problem you think this would
solve. :^(

   I am far more concerned with deriving mouth-to-ear delay than anything
relating to some intermediary relay. (But, to tell truth, I'm still less
than clear how to derive even transducer-to-transducer delay -- perhaps
somebody could write up a "RTP-latency for dummies" piece?)

--
John Leslie <[email-masked]>

---

From: Kevin Gross <[email-masked]>
Date: Tue, 4 Dec 2012 12:30:29 -0700

For a discussion of latency in RTP, you should have a look at -
http://tools.ietf.org/html/draft-ietf-avtcore-idms-07.
http://tools.ietf.org/html/draft-ietf-avtcore-clksrc-01 may also be helpful.

Kevin Gross
+1-303-447-0517
Media Network Consultant
AVA Networks - www.AVAnw.com <http://www.avanw.com/>, www.X192.org

---

From: Stefan Holmer <[email-masked]>
Date: Wed, 5 Dec 2012 09:47:18 +0100

>    I must admit to failing to understand what problem you think this would

I might have gone into the details too fast above, sorry about that.

I'm specifically thinking of the problem of measuring the changes in
one-way delay by calculating the relative inter-arrival time. In
http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-03 we
calculate this as

d(i) = t(i) - t(i-1) - (T(i) - T(i-1)),

where t(i) is the receive-time and T(i) is the send-time of a packet. The
send-time can be stamped on the packet in different ways, either with an
offset relative to the RTP timestamp or as an absolute send time.

With a relaying conference server a receiving client may receive several
streams, originating from different sending clients. As all of those
streams have been relayed by the same conference server we may want to use
all of those streams to estimate bandwidth between the server and the
receiving client. When doing that, the problems described in my initial
e-mail arise.

Hope this somewhat clarifies what I'm talking about.

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Wed, 05 Dec 2012 10:27:53 +0100

El 05/12/2012 9:47, Stefan Holmer escribió:
>
>
>
> On Tue, Dec 4, 2012 at 6:27 PM, John Leslie <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Stefan Holmer <[email-masked] <mailto:[email-masked]>> wrote:
>     >
> where t(i) is the receive-time and T(i) is the send-time of a packet. 
> The send-time can be stamped on the packet in different ways, either 
> with an offset relative to the RTP timestamp or as an absolute send time.
>
> With a relaying conference server a receiving client may receive 
> several streams, originating from different sending clients. As all of 
> those streams have been relayed by the same conference server we may 
> want to use all of those streams to estimate bandwidth between the 
> server and the receiving client. When doing that, the problems 
> described in my initial e-mail arise.
>
>
Hi Steffan,

Not considering packet losses, to calculate the t(i) reception time of a 
frame, wouldn't we just need the timestamp in the last RTP packet of a 
frame? I am still struggling to understand the draft to implement it in 
my media server, so maybe my suggestion is quite dumb, but, if we had 
absolute timestamps for each packet, could we improve the algorithm to 
work on a per rtp packet base instead of a per frame?  Something like:

    At the receiving side we are observing incoming packets, where each
    packet has an absolute sent timestamp P(i).

    Each packet is assigned a receive time p(i), which corresponds to the
    time at which the packet has been received.  A packet is delayed
    relative to its predecessor if p(i)-p(i-1)>P(i)-P(i-1), i.e., if the
    arrival time difference is larger than the timestamp difference.

    We define the (relative) inter-arrival time, d(i) as

      d(i) = p(i)-p(i-1)-(P(i)-P(i-1))

Best regards
Sergio

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Sun, 16 Dec 2012 15:08:24 +0000

Please clarify "absolute". Do you mean another RTP timestamp using the same units but a random offset from the main RTP timestamp, and no implied sync across multiple RTP sessions? Or NTP timestamps with NTP units and implied sync across all RTP sessions (within the same sync domain, currently defined to be CNAME)?

Please also clarify "relay conference server", as there are many RTP topologies for conferencing. I think you are probably referring to a source projection mixer. Rewriting RTCP is standard practice for those, as well as most mixers. In fact, I can't envision when it would be possible to blindly relay RTCP in any mixer/topology. So I'm not sure what complications you see arising from the use of RFC 5450.

Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Stefan Holmer
Sent: Tuesday, December 04, 2012 4:23 AM
To: rmcat WG
Subject: [rmcat] Send timestamps

Hi,

In draft-alvestrand-rtcweb-congestion-01<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-01> we proposed using a send timestamp RTP header extension for measuring the packet inter-arrival time offset. In 02<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-02> this was revised to instead reference RFC 5450<http://tools.ietf.org/html/rfc5450>, which specifies a header extension for specifying the send timestamp as an offset relative to the RTP timestamp.

Thinking further about this, there seems to be a couple of benefits with defining a new, absolute, send timestamp header extension. If a relay approach to conference servers are used, the relaying server can simply rewrite the send timestamp of a packet to correspond to the time it was relayed. Doing so the receiver can jointly process all streams being received from the relay server, even though they were initially captured and sent by different senders (with different clocks).

With a send timestamp offset, accomplishing the same thing would require rewriting both the timestamp offset and the NTP time in the corresponding RTCP SR reports. And to keep video synchronized with audio the RTCP packet for the audio streams must also be rewritten in a similar way. This may not even be possible if the audio stream is relayed through a different server.

Do you see any problems with using an RTP header extension with absolute send timestamps? Are there any benefits with the send timestamp offset header extension I might be missing?

/Stefan

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 17 Dec 2012 12:00:51 +0100

I'll try to clarify with an example:

Let's say we have two clients (A and B) in a call. We want to estimate the
bandwidth from A to B by looking at the variations in one-way delay, which
we can measure by comparing deltas in RTP timestamp to deltas in
receive-time. The problem with using RTP timestamps is that those represent
capture time and not the send time of a frame, so we get a lot of noise
related to variations in, e.g., video encoding time. To get around that we
introduce RFC 5450 <http://tools.ietf.org/html/rfc5450> and now we can
measure something much closer to actual network time. If we want to jointly
process all RTP streams from A to B (at the receiver, B) we can convert the
RTP timestamps of all streams to NTP using the (RTP timestamp, NTP
timestamp) tuple in the RTCP SR report.

Now let's add a server (let's call it S) in between client A and B. This
server simply relays RTP packets (and might do some rewriting as well). Now
we want to estimate the bandwidth between A and S, and between S and B. S
will have to change the timestamp offset (RFC 5450) of the RTP packets to
correspond to its send-time, but besides that all is fine.

Now let's add a third client (C). S will relay packets from both A and C to
B, so ideally we should jointly process all of those packets when
estimating the bandwidth between S to B. However, packets sent from A will
be based on the tuple (RTP timestamp A, NTP timestamp A) and packets sent
from C will be based on (RTP timestamp C, NTP timestamp C), so we can't
just convert to NTP at B and compare them. This can be solved in the server
by rewriting RTP and NTP timestamps to (RTP timestamp S, NTP timestamp S)
before relaying to B, which requires rewriting both RTP packets and RTCP
packets. This is fine as long as you rewrite _all_ streams from the same
sending client in the same way, otherwise you won't be able to synchronize
them. Therefore all streams must go through the same server.

I'm suggesting that we could introduce a new "send timestamp" which doesn't
relate to the RTP timestamp (which is used for synchronized playback).
Whenever a packet is sent (no matter if it's sent from A, C or S) it will
be stamped with the send-time taken from the clock of that sender, for
instance at 1 kHz or 90 kHz. Since it's not related to the RTP timestamp we
won't have to rewrite the (RTP, NTP) tuple, and actually, we don't even
have to convert to NTP at the receiver since all send timestamps are taken
directly from the same clock. Removing the relation to the RTP timestamp is
what I mean with an "absolute" timestamp (i.e., not relative to RTP
timestamp).

/Stefan

On Sun, Dec 16, 2012 at 4:08 PM, Mo Zanaty (mzanaty) <[email-masked]>wrote:

>  Please clarify “absolute”. Do you mean another RTP timestamp using the
> same units but a random offset from the main RTP timestamp, and no implied
> sync across multiple RTP sessions? Or NTP timestamps with NTP units and
> implied sync across all RTP sessions (within the same sync domain,
> currently defined to be CNAME)?****
>
> ** **
>
> Please also clarify “relay conference server”, as there are many RTP
> topologies for conferencing. I think you are probably referring to a source
> projection mixer. Rewriting RTCP is standard practice for those, as well as
> most mixers. In fact, I can’t envision when it would be possible to blindly
> relay RTCP in any mixer/topology. So I’m not sure what complications you
> see arising from the use of RFC 5450.****
>
> ** **
>
> Mo****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Stefan Holmer
> *Sent:* Tuesday, December 04, 2012 4:23 AM
> *To:* rmcat WG
> *Subject:* [rmcat] Send timestamps****
>
> ** **
>
> Hi,****
>
> ** **
>
> In draft-alvestrand-rtcweb-congestion-01<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-01> we
> packet inter-arrival time offset. In 02<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-02> this
> offset relative to the RTP timestamp.****
>
> ** **
>
> and sent by different senders (with different clocks). ****
>
> ** **
>
> ****
> ** **
> header extension I might be missing?****
>
> ** **
>
> /Stefan****
>

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 17 Dec 2012 12:06:37 +0100

Separating the timestamp used as a congestion signal from timestamps 
that may be used for other purposes would seem to me to be an advantage 
in terms of architectural cleanliness.

RTP timestamps need to reflect media time. In a situation with relay 
nodes and changing paths, this may have uncomfortable amounts of 
variation from the time the packets enter the last hop to the recipient.

Given that we won't have all systems deploying the new extension from 
Day One, we need to be able to fall back to the RTP timestamp (it's 
better than nothing), but it seems we shouldn't marry it more than we 
have to.

               Harald

On 12/17/2012 12:00 PM, Stefan Holmer wrote:
> Let's say we have two clients (A and B) in a call. We want to estimate 
> the bandwidth from A to B by looking at the variations in one-way 
> delay, which we can measure by comparing deltas in RTP timestamp to 
> deltas in receive-time. The problem with using RTP timestamps is that 
> those represent capture time and not the send time of a frame, so we 
> get a lot of noise related to variations in, e.g., video encoding 
> time. To get around that we introduce RFC 5450 
> <http://tools.ietf.org/html/rfc5450> and now we can measure something 
> much closer to actual network time. If we want to jointly process all 
> RTP streams from A to B (at the receiver, B) we can convert the RTP 
> timestamps of all streams to NTP using the (RTP timestamp, NTP 
> Now let's add a server (let's call it S) in between client A and B. 
> This server simply relays RTP packets (and might do some rewriting as 
> well). Now we want to estimate the bandwidth between A and S, and 
> between S and B. S will have to change the timestamp offset (RFC 5450) 
> of the RTP packets to correspond to its send-time, but besides that 
> all is fine.
>
> Now let's add a third client (C). S will relay packets from both A and 
> C to B, so ideally we should jointly process all of those packets when 
> estimating the bandwidth between S to B. However, packets sent from A 
> will be based on the tuple (RTP timestamp A, NTP timestamp A) and 
> packets sent from C will be based on (RTP timestamp C, NTP timestamp 
> C), so we can't just convert to NTP at B and compare them. This can be 
> solved in the server by rewriting RTP and NTP timestamps to (RTP 
> timestamp S, NTP timestamp S) before relaying to B, which requires 
> rewriting both RTP packets and RTCP packets. This is fine as long as 
> you rewrite _all_ streams from the same sending client in the same 
> way, otherwise you won't be able to synchronize them. Therefore all 
> streams must go through the same server.
>
> I'm suggesting that we could introduce a new "send timestamp" which 
> doesn't relate to the RTP timestamp (which is used for synchronized 
> playback). Whenever a packet is sent (no matter if it's sent from A, C 
> or S) it will be stamped with the send-time taken from the clock of 
> that sender, for instance at 1 kHz or 90 kHz. Since it's not related 
> to the RTP timestamp we won't have to rewrite the (RTP, NTP) tuple, 
> and actually, we don't even have to convert to NTP at the receiver 
> since all send timestamps are taken directly from the same clock. 
> Removing the relation to the RTP timestamp is what I mean with an 
> "absolute" timestamp (i.e., not relative to RTP timestamp).
>
> /Stefan
>
>
> On Sun, Dec 16, 2012 at 4:08 PM, Mo Zanaty (mzanaty) 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Please clarify “absolute”. Do you mean another RTP timestamp using
>     the same units but a random offset from the main RTP timestamp,
>     and no implied sync across multiple RTP sessions? Or NTP
>     timestamps with NTP units and implied sync across all RTP sessions
>     (within the same sync domain, currently defined to be CNAME)?
>
>     Please also clarify “relay conference server”, as there are many
>     RTP topologies for conferencing. I think you are probably
>     referring to a source projection mixer. Rewriting RTCP is standard
>     practice for those, as well as most mixers. In fact, I can’t
>     envision when it would be possible to blindly relay RTCP in any
>     mixer/topology. So I’m not sure what complications you see arising
>     from the use of RFC 5450.
>
>     Mo
>
>     *From:[email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>]
>     *On Behalf Of *Stefan Holmer
>     *Subject:* [rmcat] Send timestamps
>
>     Hi,
>
>     In draft-alvestrand-rtcweb-congestion-01
>     <http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-01> we
>     proposed using a send timestamp RTP header extension for measuring
>     the packet inter-arrival time offset. In 02
>     <http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-02> this
>     was revised to instead reference RFC 5450
>     <http://tools.ietf.org/html/rfc5450>, which specifies a header
>     extension for specifying the send timestamp as an offset relative
>     to the RTP timestamp.
>
>     Thinking further about this, there seems to be a couple of
>     benefits with defining a new, absolute, send timestamp header
>     extension. If a relay approach to conference servers are used, the
>     relaying server can simply rewrite the send timestamp of a packet
>     to correspond to the time it was relayed. Doing so the receiver
>     can jointly process all streams being received from the relay
>     server, even though they were initially captured and sent by
>     different senders (with different clocks).
>
>     With a send timestamp offset, accomplishing the same thing would
>     require rewriting both the timestamp offset and the NTP time in
>     the corresponding RTCP SR reports. And to keep video synchronized
>     with audio the RTCP packet for the audio streams must also be
>     rewritten in a similar way. This may not even be possible if the
>     audio stream is relayed through a different server.
>
>     Do you see any problems with using an RTP header extension with
>     absolute send timestamps? Are there any benefits with the send
>     timestamp offset header extension I might be missing?
>
>     /Stefan
>
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 17 Dec 2012 14:45:32 +0000

Hi

I believe that it can make sense to use a separate timestamp for congestion control but I may need to read and understand some more. A few reflections though.

Unless I miss something, this is quite close to the TCP timestamps which are used for congestion avoidance in TCP Vegas and LEDBAT.
The problem in Vegas is that it is sensitive to congestion in the feedback path, not shure though how much concern that should be.
LEDBAT modifies this and implement the concept of base delay, but suffers from late comers advantage.
Both the above assume that one ACKs are returned to the sender almost immediately (delay ACK adds some uncertanity here) also one don't need to synchronize clocks.
Here I get unsure about what to do with the received timestamps, I guess you can compute a delta between sent and received timestamp but unless you have exactly synchronized clocks in sender(s) and receiver(s), you will have an unknown offset that can possibly also drift due to clock skew.

/Ingemar

From: Harald Alvestrand [mailto:[email-masked]]
Sent: den 17 december 2012 12:07
To: [email-masked]
Subject: Re: [rmcat] Send timestamps

Separating the timestamp used as a congestion signal from timestamps that may be used for other purposes would seem to me to be an advantage in terms of architectural cleanliness.

RTP timestamps need to reflect media time. In a situation with relay nodes and changing paths, this may have uncomfortable amounts of variation from the time the packets enter the last hop to the recipient.

Given that we won't have all systems deploying the new extension from Day One, we need to be able to fall back to the RTP timestamp (it's better than nothing), but it seems we shouldn't marry it more than we have to.

              Harald

Let's say we have two clients (A and B) in a call. We want to estimate the bandwidth from A to B by looking at the variations in one-way delay, which we can measure by comparing deltas in RTP timestamp to deltas in receive-time. The problem with using RTP timestamps is that those represent capture time and not the send time of a frame, so we get a lot of noise related to variations in, e.g., video encoding time. To get around that we introduce RFC 5450<http://tools.ietf.org/html/rfc5450> and now we can measure something much closer to actual network time. If we want to jointly process all RTP streams from A to B (at the receiver, B) we can convert the RTP timestamps of all streams to NTP using the (RTP timestamp, NTP timestamp) tuple in the RTCP SR report.

Now let's add a server (let's call it S) in between client A and B. This server simply relays RTP packets (and might do some rewriting as well). Now we want to estimate the bandwidth between A and S, and between S and B. S will have to change the timestamp offset (RFC 5450) of the RTP packets to correspond to its send-time, but besides that all is fine.

Now let's add a third client (C). S will relay packets from both A and C to B, so ideally we should jointly process all of those packets when estimating the bandwidth between S to B. However, packets sent from A will be based on the tuple (RTP timestamp A, NTP timestamp A) and packets sent from C will be based on (RTP timestamp C, NTP timestamp C), so we can't just convert to NTP at B and compare them. This can be solved in the server by rewriting RTP and NTP timestamps to (RTP timestamp S, NTP timestamp S) before relaying to B, which requires rewriting both RTP packets and RTCP packets. This is fine as long as you rewrite _all_ streams from the same sending client in the same way, otherwise you won't be able to synchronize them. Therefore all streams must go through the same server.

I'm suggesting that we could introduce a new "send timestamp" which doesn't relate to the RTP timestamp (which is used for synchronized playback). Whenever a packet is sent (no matter if it's sent from A, C or S) it will be stamped with the send-time taken from the clock of that sender, for instance at 1 kHz or 90 kHz. Since it's not related to the RTP timestamp we won't have to rewrite the (RTP, NTP) tuple, and actually, we don't even have to convert to NTP at the receiver since all send timestamps are taken directly from the same clock. Removing the relation to the RTP timestamp is what I mean with an "absolute" timestamp (i.e., not relative to RTP timestamp).

/Stefan

On Sun, Dec 16, 2012 at 4:08 PM, Mo Zanaty (mzanaty) <[email-masked]<mailto:[email-masked]>> wrote:
Please clarify "absolute". Do you mean another RTP timestamp using the same units but a random offset from the main RTP timestamp, and no implied sync across multiple RTP sessions? Or NTP timestamps with NTP units and implied sync across all RTP sessions (within the same sync domain, currently defined to be CNAME)?

Mo

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Stefan Holmer
Sent: Tuesday, December 04, 2012 4:23 AM
To: rmcat WG
Subject: [rmcat] Send timestamps

Hi,

/Stefan

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 17 Dec 2012 17:15:31 +0100

On Mon, Dec 17, 2012 at 3:45 PM, Ingemar Johansson S <
[email-masked]> wrote:

>  Hi****
>
> ** **
>
> I believe that it can make sense to use a separate timestamp for
> congestion control but I may need to read and understand some more. A few
> reflections though.****
>
> ** **
>
> Unless I miss something, this is quite close to the TCP timestamps which
> are used for congestion avoidance in TCP Vegas and LEDBAT. ****
>
> The problem in Vegas is that it is sensitive to congestion in the feedback
> path, not shure though how much concern that should be. ****
>
> LEDBAT modifies this and implement the concept of base delay, but suffers
> from late comers advantage.****
>
> Both the above assume that one ACKs are returned to the sender almost
> immediately (delay ACK adds some uncertanity here) also one don’t need to
> synchronize clocks.****
>
> Here I get unsure about what to do with the received timestamps, I guess
> you can compute a delta between sent and received timestamp but unless you
> have exactly synchronized clocks in sender(s) and receiver(s), you will
> have an unknown offset that can possibly also drift due to clock skew.
>

In draft-alvestrand-rtcweb-congestion<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-03>
we
are tracking the trend of that delta. If the receive-deltas are increasing
compared to send-deltas we assume it's because of congestion. Doing so we
don't have to deal with the unknown offset.

> ****
>
> ** **
>
> /Ingemar****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Harald Alvestrand [mailto:[email-masked]]
> *Sent:* den 17 december 2012 12:07
>
> *To:* [email-masked]
> *Subject:* Re: [rmcat] Send timestamps****
>
>  ** **
>
> Separating the timestamp used as a congestion signal from timestamps that
> may be used for other purposes would seem to me to be an advantage in terms
> of architectural cleanliness.
>
>
> RTP timestamps need to reflect media time. In a situation with relay nodes
> and changing paths, this may have uncomfortable amounts of variation from
> the time the packets enter the last hop to the recipient.
>
> Given that we won't have all systems deploying the new extension from Day
> One, we need to be able to fall back to the RTP timestamp (it's better than
> nothing), but it seems we shouldn't marry it more than we have to.
>
>               Harald
>
> On 12/17/2012 12:00 PM, Stefan Holmer wrote:****
>
>   I'll try to clarify with an example: ****
>
> ** **
>
> timestamp) tuple in the RTCP SR report.****
>
> ** **
>
> correspond to its send-time, but besides that all is fine.****
>
> ** **
>
> Now let's add a third client (C). S will relay packets from both A and C
> to B, so ideally we should jointly process all of those packets when
> them. Therefore all streams must go through the same server.****
>
> ** **
>
> playback). Whenever a packet is sent (no matter if it's sent from A, C or
> S) it will be stamped with the send-time taken from the clock of that
> sender, for instance at 1 kHz or 90 kHz. Since it's not related to the RTP
> timestamp we won't have to rewrite the (RTP, NTP) tuple, and actually, we
> don't even have to convert to NTP at the receiver since all send timestamps
> are taken directly from the same clock. Removing the relation to the RTP
> timestamp is what I mean with an "absolute" timestamp (i.e., not relative
> to RTP timestamp).****
>
> ** **
>
> /Stefan****
>
> ** **
>
> On Sun, Dec 16, 2012 at 4:08 PM, Mo Zanaty (mzanaty) <[email-masked]>
> wrote:****
>

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 17 Dec 2012 20:36:37 +0100

On 12/17/2012 03:45 PM, Ingemar Johansson S wrote:
> congestion control but I may need to read and understand some more. A 
> few reflections though.
>
> Unless I miss something, this is quite close to the TCP timestamps 
> which are used for congestion avoidance in TCP Vegas and LEDBAT.
>
> The problem in Vegas is that it is sensitive to congestion in the 
> feedback path, not shure though how much concern that should be.
>
If we keep a model of computation-at-receiver, it's not a problem at all.
> immediately (delay ACK adds some uncertanity here) also one don't need 
> to synchronize clocks.
>
> Here I get unsure about what to do with the received timestamps, I 
> guess you can compute a delta between sent and received timestamp but 
> unless you have exactly synchronized clocks in sender(s) and 
> receiver(s), you will have an unknown offset that can possibly also 
> drift due to clock skew.
>

As Stefan says - since we measure ~a moving average of difference 
between the (receive timestamp - send timestamp) of successive packets, 
not an absolute transit time, value, this is not likely to be an issue.

I have a hard time imagining clock drift so severe that it will 
significantly influence the deltas of packets that are sent less than a 
second apart. Clock jumps (like leap seconds) may be a measurable, but 
extremely transient, influence.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 17 Dec 2012 20:15:24 +0000

So we're back to two pendulums two months after deciding multi-hop is initially out of scope. I'm glad, because it will likely be a major (or perhaps even dominant) deployment scenario, which we should not defer based on complexity alone.

If we introduce a new send timestamp extension (which I'm unsure of the benefits over RFC 5450), I think it should be in NTP units not RTP media clock units. There is a precedent (in RTCP) for NTP timestamps that are synced across multiple RTP sessions, but there is no such precedent for synced RTP timestamps. Different media types with different media clocks would also present problems for synced RTP timestamps.

But even with this new send timestamp extension (using absolute synced NTP timestamps), I don't think it avoids Stefan's problem of rewriting every RTP and RTCP packet at the server. Every RTP packet must still be rewritten with the new extension using the server's NTP time, and every RTCP SR packet must still be rewritten for several reasons beyond congestion control.

So I really don't see any benefit that a new extension would provide beyond RFC 5450. Perhaps I'm missing it.

I'm also missing why you need to convert to NTP time in order to "jointly process all RTP streams", because only the deltas really matter not the actual NTP time, and RFC 5450 gives you all you need for deltas.

> If we want to jointly process all RTP streams from A to B (at the receiver, B) we can convert the RTP timestamps of all streams to NTP using the (RTP timestamp, NTP timestamp) tuple in the RTCP SR report.

Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Harald Alvestrand
Sent: Monday, December 17, 2012 6:07 AM
To: [email-masked]
Subject: Re: [rmcat] Send timestamps

              Harald

/Stefan

Mo

Hi,

/Stefan

---

From: Michael Welzl <[email-masked]>
Date: Mon, 17 Dec 2012 21:43:28 +0100

+1 to what harald and stefan said, from a transport perspective.
(i'll leave it to the rtp experts to say if that makes sense or not)

as for the two pendulums thing: i think it's okay, and probably important, for us to consider if signalling functions would also work in a multi-hop case; this is a necessity to build a two- (or n-)pendulum system.
surely we don't want to exclude such deployment scenarios later on!

when we talked about two pendulums, i think the point was to defer looking into two-pendulum-dynamics, which i'd translate into: the buffer management at an intermediate rtp hop.

cheers
michael

On Dec 17, 2012, at 9:15 PM, Mo Zanaty (mzanaty) <[email-masked]> wrote:

> So we’re back to two pendulums two months after deciding multi-hop is initially out of scope. I’m glad, because it will likely be a major (or perhaps even dominant) deployment scenario, which we should not defer based on complexity alone.
>  
> If we introduce a new send timestamp extension (which I’m unsure of the benefits over RFC 5450), I think it should be in NTP units not RTP media clock units. There is a precedent (in RTCP) for NTP timestamps that are synced across multiple RTP sessions, but there is no such precedent for synced RTP timestamps. Different media types with different media clocks would also present problems for synced RTP timestamps.
>  
> But even with this new send timestamp extension (using absolute synced NTP timestamps), I don’t think it avoids Stefan’s problem of rewriting every RTP and RTCP packet at the server. Every RTP packet must still be rewritten with the new extension using the server’s NTP time, and every RTCP SR packet must still be rewritten for several reasons beyond congestion control.
>  
> So I really don’t see any benefit that a new extension would provide beyond RFC 5450. Perhaps I’m missing it.
>  
> I’m also missing why you need to convert to NTP time in order to “jointly process all RTP streams”, because only the deltas really matter not the actual NTP time, and RFC 5450 gives you all you need for deltas.
>  
> Let's say we have two clients (A and B) in a call. We want to estimate the bandwidth from A to B by looking at the variations in one-way delay, which we can measure by comparing deltas in RTP timestamp to deltas in receive-time. The problem with using RTP timestamps is that those represent capture time and not the send time of a frame, so we get a lot of noise related to variations in, e.g., video encoding time. To get around that we introduce RFC 5450 and now we can measure something much closer to actual network time. If we want to jointly process all RTP streams from A to B (at the receiver, B) we can convert the RTP timestamps of all streams to NTP using the (RTP timestamp, NTP timestamp) tuple in the RTCP SR report.
>  
> On Sun, Dec 16, 2012 at 4:08 PM, Mo Zanaty (mzanaty) <[email-masked]> wrote:
> Please clarify “absolute”. Do you mean another RTP timestamp using the same units but a random offset from the main RTP timestamp, and no implied sync across multiple RTP sessions? Or NTP timestamps with NTP units and implied sync across all RTP sessions (within the same sync domain, currently defined to be CNAME)?
>  
> Please also clarify “relay conference server”, as there are many RTP topologies for conferencing. I think you are probably referring to a source projection mixer. Rewriting RTCP is standard practice for those, as well as most mixers. In fact, I can’t envision when it would be possible to blindly relay RTCP in any mixer/topology. So I’m not sure what complications you see arising from the use of RFC 5450.
>  
> Mo
>  
> In draft-alvestrand-rtcweb-congestion-01 we proposed using a send timestamp RTP header extension for measuring the packet inter-arrival time offset. In 02 this was revised to instead reference RFC 5450, which specifies a header extension for specifying the send timestamp as an offset relative to the RTP timestamp.
>

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 17 Dec 2012 22:20:34 +0100

On Mon, Dec 17, 2012 at 9:15 PM, Mo Zanaty (mzanaty) <[email-masked]>wrote:

>  So we’re back to two pendulums two months after deciding multi-hop is
> initially out of scope. I’m glad, because it will likely be a major (or
> perhaps even dominant) deployment scenario, which we should not defer based
> on complexity alone.****
>
> ** **
>
> If we introduce a new send timestamp extension (which I’m unsure of the
> benefits over RFC 5450), I think it should be in NTP units not RTP media
> clock units. There is a precedent (in RTCP) for NTP timestamps that are
> synced across multiple RTP sessions, but there is no such precedent for
> synced RTP timestamps. Different media types with different media clocks
> would also present problems for synced RTP timestamps.****
>
> ** **
>
> But even with this new send timestamp extension (using absolute synced NTP
> timestamps), I don’t think it avoids Stefan’s problem of rewriting every
> RTP and RTCP packet at the server. Every RTP packet must still be rewritten
> with the new extension using the server’s NTP time, and every RTCP SR
> packet must still be rewritten for several reasons beyond congestion
> control.
>

Perhaps. I don't see why rewriting the full (RTP timestamp, NTP timestamp)
tuple is necessary for other reasons though.

The problem as I see it is when we, at the receiving client B, are to
compute the delta between packets generated by A and packets generated by
C. Even if we convert them both to NTP, their NTP timestamps aren't likely
to be in sync. They can be if S has rewritten the RTCP SRs so that the NTP
timestamp of streams from A and C now are based on the clock of S (NTP
timestamp S). But when that has been done we also have to rewrite the RTCP
SR of all other streams from A and C which we at some point would like to
have synchronized playback with, which will require all those streams to
also go through S. So basically we will not be able to have to servers, S1
and S2, which for example handle video and audio streams respectively.

Maybe not a big issue, but to me it feels like a limitation without
benefits. I don't see the point of requiring that timestamps used for
congestion control should be set in relation to the RTP timestamps which is
meant for doing synchronized playback.

> ****
>
> ** **
>
> So I really don’t see any benefit that a new extension would provide
> beyond RFC 5450. Perhaps I’m missing it.****
>
> ** **
>
> I’m also missing why you need to convert to NTP time in order to “jointly
> process all RTP streams”, because only the deltas really matter not the
> actual NTP time, and RFC 5450 gives you all you need for deltas.
>

The deltas only have a meaning in relation to the corresponding RTP
timestamp. RTP timestamps of different streams (SSRCs) have different
random offsets, and therefore can't be compared before converted to NTP. At
least if you want to calculate the delta between packets belonging from two
different streams.

> ****
>
> ** **
>
> > If we want to jointly process all RTP streams from A to B (at the
> receiver, B) we can convert the RTP timestamps of all streams to NTP using
> the (RTP timestamp, NTP timestamp) tuple in the RTCP SR report.****
>
> ** **
>
> Mo****
>
> ** **
>
> ** **
>
> Of *Harald Alvestrand
> *Sent:* Monday, December 17, 2012 6:07 AM
>
> *To:* [email-masked]

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Tue, 18 Dec 2012 15:18:55 +0000

All,

Re: "As Stefan says - since we measure ~a moving average of difference between the (receive timestamp - send timestamp) of successive packets, not an absolute transit time, value, this is not likely to be an issue."

I think this discussion is heading in the right direction ... that is, an absolute, transport-not-media-capture-related ability to generate one-way delay DIFFERENCES ... without a need to keep track of absolute time (e.g., NTP) or detailed knowledge about the capture time. We are interested in congestion measures here, not play out information in this discussion.

As one may recall, the discussion started from a premise of inter-arrival time differences ... which I didn't think was wise or appropriate (even though a particular vendor happened to design a control based on inter-arrival time). I preferred a mechanism to measure the relative one-way time delay differences from which an inter-arrival time measure could be calculated (if you wanted one).

The issue I have with the above is that I also think it is silly to compute a moving average of the differences based on my original email on this thread (knowing the delay histogram is one sided and that we can EXPECT outliers). If there was non-linear filtering PRIOR TO a moving average metric ... then I would be happier ... but that also doesn't need to be specified in what we are trying to do with the timestamps on the wire. I can do that non-linear filtering with the timestamps as we are proposing in this thread.

Regards,

Michael Ramalho, Ph.D.

PS - Harold, until we specify what we need, a close approximation is to take the RTP timestamp of the FIRST packet of a multi-packet video frame. You will still have the OS and processing delay in there ... but it is better than nothing.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Harald Alvestrand
Sent: Monday, December 17, 2012 2:37 PM
To: Ingemar Johansson S
Cc: [email-masked]
Subject: Re: [rmcat] Send timestamps

As Stefan says - since we measure ~a moving average of difference between the (receive timestamp - send timestamp) of successive packets, not an absolute transit time, value, this is not likely to be an issue.

I have a hard time imagining clock drift so severe that it will significantly influence the deltas of packets that are sent less than a second apart. Clock jumps (like leap seconds) may be a measurable, but extremely transient, influence.

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 20 Dec 2012 09:53:25 +0100

On 2012-12-17 12:00, Stefan Holmer wrote:
> the bandwidth from A to B by looking at the variations in one-way delay,
> which we can measure by comparing deltas in RTP timestamp to deltas in
> receive-time. The problem with using RTP timestamps is that those
> represent capture time and not the send time of a frame, so we get a lot
> of noise related to variations in, e.g., video encoding time.

How big is this problem? can you show us some data showing where this 
completely breaks or cause huge problem in estimating bandwidth?

> around that we introduce RFC 5450
> server simply relays RTP packets (and might do some rewriting as well).
> Now we want to estimate the bandwidth between A and S, and between S and
> B.

I think you will have to explain the topology a bit more in details 
here. What kind of feedback loop you are assuming? why do you need to 
estimate bandwidth between A-S and S-B separately? as I understand it 
that a simple relay S will be transparent between A and B and all we 
need to be concerned about delay visible between A-B. But if you have a 
different view or requirements on S then this might not be a point to 
point session we are concern here in RMCAT.

S will have to change the timestamp offset (RFC 5450) of the RTP
> packets to correspond to its send-time, but besides that all is fine.
>
> packets sent from C will be based on (RTP timestamp C, NTP timestamp C),
> so we can't just convert to NTP at B and compare them. This can be
> rewriting both RTP packets and RTCP packets. This is fine as long as you
> rewrite _all_ streams from the same sending client in the same way,
> otherwise you won't be able to synchronize them. Therefore all streams
> must go through the same server.

Again, you will have to explain the relation between A, C, B and S in 
details? Depending on that relation, it can turned out to be a multicast 
case, a unicast case with all legs are handled separately or even a 
multi-party multi-hop case which is out of scope right now. In the later 
case we should not be so concerned now but if you want to change the 
charter then you are more than welcomed from my part.

BR

-- 
Zahed

============================================
ANM ZAHEDUZZAMAN SARKER

Ericsson AB
Multimedia Technologies (MMT)
Ericsson Research
P.O. Box 920, SE-971 28, Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]
www.ericsson.com
============================================

---

From: Stefan Holmer <[email-masked]>
Date: Thu, 20 Dec 2012 10:31:51 +0100

> completely breaks or cause huge problem in estimating bandwidth?

I think it's a fairly big problem. We want to make sure the RTP timestamp
is set to correspond to when the frame was captured, while the timestamp
used for bandwidth estimation should be as close to sending the
frame/packet as possible. For instance if your CPU is temporarily loaded
you will get jitter similar to network congestion. I don't have any numbers
to show you though.

>> <http://tools.ietf.org/html/**rfc5450<http://tools.ietf.org/html/rfc5450>>
>> and now we can measure something
>>
> I think you will have to explain the topology a bit more in details here.
> What kind of feedback loop you are assuming? why do you need to estimate
> bandwidth between A-S and S-B separately? as I understand it that a simple
> relay S will be transparent between A and B and all we need to be concerned
> about delay visible between A-B. But if you have a different view or
> requirements on S then this might not be a point to point session we are
> concern here in RMCAT.

I'm thinking of a topology where S is able to shape the outgoing streams to
fit the link between S and B you want to be able to estimate the bandwidth
on that link. How that shaping is done is of minor interest, but I can
imagine both stripping layers and transcoding.

> ==============================**==============
> ANM ZAHEDUZZAMAN SARKER
>
>
> Ericsson AB
> zaheduzzaman.sarker@ericsson.**com <[email-masked]>
> www.ericsson.com
> ==============================**==============
>

---

From: Stefan Holmer <[email-masked]>
Date: Thu, 20 Dec 2012 10:34:13 +0100

On Tue, Dec 18, 2012 at 4:18 PM, Michael Ramalho (mramalho) <
[email-masked]> wrote:

>  All,****
>
> ** **
>
> Re: “As Stefan says - since we measure ~a moving average of difference
> between the (receive timestamp - send timestamp) of successive packets, not
> an absolute transit time, value, this is not likely to be an issue.”****
>
> ** **
>
> I think this discussion is heading in the right direction … that is, an
> absolute, transport-not-media-capture-related ability to generate one-way
> delay DIFFERENCES … without a need to keep track of absolute time (e.g.,
> NTP) or detailed knowledge about the capture time. We are interested in
> congestion measures here, not play out information in this discussion.****
>
> ** **
>
> As one may recall, the discussion started from a premise of inter-arrival
> time differences … which I didn’t think was wise or appropriate (even
> though a particular vendor happened to design a control based on
> inter-arrival time). I preferred a mechanism to measure the relative
> one-way time delay differences from which an inter-arrival time measure
> could be calculated (if you wanted one).
>

What is your definition of "inter-arrival time differences" and "one-way
time delay differences"?

> ****
>
> ** **
>
> The issue I have with the above is that I also think it is silly to
> compute a moving average of the differences based on my original email on
> this thread (knowing the delay histogram is one sided and that we can
> EXPECT outliers). If there was non-linear filtering PRIOR TO a moving
> average metric … then I would be happier … but that also doesn’t need to be
> specified in what we are trying to do with the timestamps on the wire. I
> can do that non-linear filtering with the timestamps as we are proposing in
> this thread.****
>
> ** **
>
> Regards,****
>
> ** **
>
> Michael Ramalho, Ph.D.****
>
> ** **
>
> PS – Harold, until we specify what we need, a close approximation is to
> take the RTP timestamp of the FIRST packet of a multi-packet video frame.
> You will still have the OS and processing delay in there … but it is better
> than nothing.****
>
> ** **
>
> *Sent:* Monday, December 17, 2012 2:37 PM
> *To:* Ingemar Johansson S
> *Cc:* [email-masked]
>
> On 12/17/2012 03:45 PM, Ingemar Johansson S wrote:****
>
> Hi****
>
>  ****
>
> have an unknown offset that can possibly also drift due to clock skew.****
>
>
> As Stefan says - since we measure ~a moving average of difference between
> the (receive timestamp - send timestamp) of successive packets, not an
> absolute transit time, value, this is not likely to be an issue.
>
> extremely transient, influence.****
>

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Fri, 21 Dec 2012 14:12:17 +0100

On 2012-12-20 10:31, Stefan Holmer wrote:
>
>
>
> On Thu, Dec 20, 2012 at 9:53 AM, Zaheduzzaman Sarker
> <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     How big is this problem? can you show us some data showing where
>     this completely breaks or cause huge problem in estimating bandwidth?
>
>
> I think it's a fairly big problem. We want to make sure the RTP
> timestamp is set to correspond to when the frame was captured, while the
> timestamp used for bandwidth estimation should be as close to sending
> the frame/packet as possible.

This is where I think we need data to show us what is the percentage of 
performance gain if we use send timestamp instead of RTP timestamp for 
bandwidth estimation.

For instance if your CPU is temporarily
> loaded you will get jitter similar to network congestion. I don't have
> any numbers to show you though.

what is the worst case here? If I take Google's proposed algorithm then 
it will result in a temporary "over-use". The sender may have to adapt. 
But that might be good (specially in case of multi-stream), may be the 
app itself is causing the CPU load :-).

The CPU overload will be visible in other streams as well. Perhaps a 
smart co-relation can filter those event out, perhaps.

>         <http://tools.ietf.org/html/__rfc5450
>         <http://tools.ietf.org/html/rfc5450>> and now we can measure
>         something
>
>     here. What kind of feedback loop you are assuming? why do you need
>     to estimate bandwidth between A-S and S-B separately? as I
>     understand it that a simple relay S will be transparent between A
>     and B and all we need to be concerned about delay visible between
>     A-B. But if you have a different view or requirements on S then this
>     might not be a point to point session we are concern here in RMCAT.
>
>
> I'm thinking of a topology where S is able to shape the outgoing streams
> to fit the link between S and B you want to be able to estimate the
> bandwidth on that link. How that shaping is done is of minor interest,
> but I can imagine both stripping layers and transcoding.

It seems to me that this topology is out of scope of RMCAT. Here, S will 
have to manage A-S and S-B separately and S will have to handle RTCP 
report properly to reflect the shaping (I am using your terminology). A 
multi-hop case. The RMCAT only applies to A-S and S-B separately but not 
A-B in that picture. If S touches the media it will have to rewrite the 
RTP header inserting new RTP timestamps and etc. then A-S and S-B have 
different RTP timestamps to estimate corresponding bandwidth.

Even if you think this topology is with in scope of RMCAT then I would 
again like to know how much we gain in performance by adding a new RTP 
header extension.

If I have missed something, please correct me.

>     Again, you will have to explain the relation between A, C, B and S
>     in details? Depending on that relation, it can turned out to be a
>     multicast case, a unicast case with all legs are handled separately
>     or even a multi-party multi-hop case which is out of scope right
>     now. In the later case we should not be so concerned now but if you
>     want to change the charter then you are more than welcomed from my part.
>

-- 
Zahed

Ericsson AB

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Fri, 21 Dec 2012 14:15:59 +0000

FWIW - I agree with Stefan.

Comments below with "MAR:"

Michael Ramalho

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zaheduzzaman Sarker
Sent: Friday, December 21, 2012 8:12 AM
To: Stefan Holmer
Cc: rmcat WG; Mo Zanaty (mzanaty)
Subject: Re: [rmcat] Send timestamps

On 2012-12-20 10:31, Stefan Holmer wrote:
> timestamp is set to correspond to when the frame was captured, while 
> the timestamp used for bandwidth estimation should be as close to 
> sending the frame/packet as possible.

This is where I think we need data to show us what is the percentage of performance gain if we use send timestamp instead of RTP timestamp for bandwidth estimation.

MAR: The magnitude of the "problem" depends on how you deal with the problem.

MAR: Stefan is correct .. the usual RTP timestamp is representative of the capture time ... ANYTHING in the processing in between the capture time and the time a particular packet departs your NIC (potentially one of many from that capture instant, I might add) will appear as a random process. OSs burp a lot as seen by a particular application process ... oftentimes due to unrelated processes (such as encryption processes on my corporate machine, ughh ... but I digress).

MAR: Now, if your receive processing is "sophisticated enough", you MAY be able to discard that "outlier" caused by such processing - but I can assure you that you MUST EXPECT such outliers ... as there are other network sources of occasional delay (such as what I mentioned at the beginning of this thread ..sortware-only firewalls doing garbage collection and the like).

MAR: So .. how exactly does one show the magnitude of the problem (what you say as percentage of performance gain), when the "gain" is a function of how intelligent a design is? You can't.

MAR: The solution is to de-couple the problems and create a timestamp that IS MORE representative of the transmit time than what we have now. I say "more representative" because most of what we are proposing here is  at the application, not kernel, level processing.

> any numbers to show you though.

what is the worst case here? If I take Google's proposed algorithm then it will result in a temporary "over-use". The sender may have to adapt. 
But that might be good (specially in case of multi-stream), may be the app itself is causing the CPU load :-).

MAR: Unless I am mistaken (the Google folks can correct me here) ... ONE bad data point (one packet having additional delay) creates TWO bad "inter-arrival datapoint" samples. That is error multiplication. Now unless their designers have developed an algorithm that discounts that effect BEFORE performing normal Euclidean processing on those two bad data points ... they will take them as an "error" and then "square that error" in their estimator. In other words, the bad data points have now biased the estimator!

MAR: Now, hopefully, you will see the importance of getting as accurate of data as possible - as engineers we don't want to ignore what is going on "behind the math". Simple exponential moving averages work GREAT only for GAUSSIAN processing - we KNOW we are dealing with one-sided, non-Gaussian PDV processes. [Ditto for Kalman Filtering, but I don't have a better solution ;-).] Let's try to get our underlying data as accurate/clean as we can get it at reasonable (bandwidth/RTP header extension) cost.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Sat, 22 Dec 2012 21:31:45 +0000

Hi Stefan,

I thought some more about your problem, and I think it is caused by your desire to map to NTP. While it may seem intuitively logical to map all received packets to a single absolute NTP timeline, this may be unnecessary, and possibly even harmful. What you really care about computing is d(i)=t(i)-t(i-1)-(T(i)-T(i-1)). If you map T(i) to NTP before computing deltas, you unnecessarily introduce conversion and mapping errors, which may be harmful or negligible. If you simply compute deltas on T(i) directly, you avoid this. If you want to process multiple streams, just compute deltas per stream before feeding the computed d(i) into a single filter. Do you agree this would solve your problem?

Some more comments below, but these details are mostly orthogonal to the core issue above.

> Perhaps. I don't see why rewriting the full (RTP timestamp, NTP timestamp) tuple is necessary for other reasons though.

The server needs to rewrite NTP in RTCP SR if it wants to compute RTT. Instead of rewriting NTP on the wire, it could rewrite internally (i.e. maintain state internally for the NTP offset between source and server, so the NTP echoed in RTCP RR from receivers can be converted from source to server time). But this seems like equivalent or higher complexity (and possibly fragility) to maintain internal rewrites than rewrite on the wire. So I would not consider this an advantage.

More fundamental than any timestamps, servers must usually (always?) rewrite RTCP SR because they are rarely (never?) blindly forwarding all RTP packets for the life of the session. For example, a source projection mixer that switches video sources at keyframe boundaries based on active speaker audio levels. The source RTCP SRs will be periodic, not on keyframe boundaries, so what the mixer decides to forward can differ from what the source sent in that RTCP interval. The mixer effectively terminates and regenerates all RTCP with rules more complex than its simple RTP forwarding behavior, so it can't simply forward RTCP.

> The deltas only have a meaning in relation to the corresponding RTP timestamp.

Actually, the RFC 5450 offsets only relate to themselves, not directly to the RTP timestamp. See the example in section 3 where "the choice of x is essentially arbitrary". Also see the AVT archives circa 2005-2006 (search for draft-singer-rtp-hdrext) for discussions about "true offset" as defined by Ron vs. "deviation compared to a static delay" as defined by Magnus (which is what the final RFC wording reflects).

The problem RFC 5450 attempted to solve is exactly the same as RMCAT now wants to solve with send timestamps, to separate source and network jitter, for "transmission rate adaptation" (Magnus' words).

Happy Holidays!
Mo

From: Stefan Holmer [mailto:[email-masked]]
Sent: Monday, December 17, 2012 4:21 PM
To: Mo Zanaty (mzanaty)
Cc: Harald Alvestrand; [email-masked]
Subject: Re: [rmcat] Send timestamps

On Mon, Dec 17, 2012 at 9:15 PM, Mo Zanaty (mzanaty) <[email-masked]<mailto:[email-masked]>> wrote:
So we're back to two pendulums two months after deciding multi-hop is initially out of scope. I'm glad, because it will likely be a major (or perhaps even dominant) deployment scenario, which we should not defer based on complexity alone.

Perhaps. I don't see why rewriting the full (RTP timestamp, NTP timestamp) tuple is necessary for other reasons though.

The problem as I see it is when we, at the receiving client B, are to compute the delta between packets generated by A and packets generated by C. Even if we convert them both to NTP, their NTP timestamps aren't likely to be in sync. They can be if S has rewritten the RTCP SRs so that the NTP timestamp of streams from A and C now are based on the clock of S (NTP timestamp S). But when that has been done we also have to rewrite the RTCP SR of all other streams from A and C which we at some point would like to have synchronized playback with, which will require all those streams to also go through S. So basically we will not be able to have to servers, S1 and S2, which for example handle video and audio streams respectively.

Maybe not a big issue, but to me it feels like a limitation without benefits. I don't see the point of requiring that timestamps used for congestion control should be set in relation to the RTP timestamps which is meant for doing synchronized playback.

The deltas only have a meaning in relation to the corresponding RTP timestamp. RTP timestamps of different streams (SSRCs) have different random offsets, and therefore can't be compared before converted to NTP. At least if you want to calculate the delta between packets belonging from two different streams.

Mo

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Harald Alvestrand
Sent: Monday, December 17, 2012 6:07 AM

To: [email-masked]<mailto:[email-masked]>
Subject: Re: [rmcat] Send timestamps

              Harald

/Stefan

Mo

Hi,

/Stefan