Subject: Re: [rmcat] Send timestamps

From: Stefan Holmer <[email-masked]>
Date: Tue, 8 Jan 2013 09:05:25 +0100

Sorry for the late reply, holidays...

On Sat, Dec 22, 2012 at 10:31 PM, Mo Zanaty (mzanaty) <[email-masked]>wrote:

>  Hi Stefan,****
>
> ** **
>
> I thought some more about your problem, and I think it is caused by your
> desire to map to NTP. While it may seem intuitively logical to map all
> received packets to a single absolute NTP timeline, this may be
> unnecessary, and possibly even harmful. What you really care about
> computing is d(i)=t(i)-t(i-1)-(T(i)-T(i-1)). If you map T(i) to NTP before
> computing deltas, you unnecessarily introduce conversion and mapping
> errors, which may be harmful or negligible. If you simply compute deltas on
> T(i) directly, you avoid this. If you want to process multiple streams,
> just compute deltas per stream before feeding the computed d(i) into a
> single filter. Do you agree this would solve your problem?
>

Sure, it's a possible solution to the problem, but I'm not convinced it's a
good one. If you receive two streams, A and B, with the following packets:
[A1] [B1] [A2] [A3] [B2] ....
You will get d(i)'s which are overlapping. For instance, d_a(1) = t_a(2) -
t_a(1) - (T_a(2) - T_a(1)), and d_b(1) both partly cover the same time
slice. d_a(3) and d_b(2) will be even worse where the time slice covered by
d_a(3) will be accounted for twice since it's also accounted for by d_b(2).
I think it would be much simpler and more intuitive to be able to compute
the differences between all packets coming from the same sending client,
and we wouldn't get any weird bias issues due to the overlap problem.

> ****
>
> ** **
>
> Some more comments below, but these details are mostly orthogonal to the
> core issue above.****
>
> ** **
>
> > Perhaps. I don't see why rewriting the full (RTP timestamp, NTP
> timestamp) tuple is necessary for other reasons though.****
>
> ** **
>
> The server needs to rewrite NTP in RTCP SR if it wants to compute RTT.
> Instead of rewriting NTP on the wire, it could rewrite internally (i.e.
> maintain state internally for the NTP offset between source and server, so
> the NTP echoed in RTCP RR from receivers can be converted from source to
> server time). But this seems like equivalent or higher complexity (and
> possibly fragility) to maintain internal rewrites than rewrite on the wire.
> So I would not consider this an advantage.
>

Depends. It is an advantage if you want to be able to process audio and
video streams independently.

> ****
>
> ** **
>
> More fundamental than any timestamps, servers must usually (always?)
> rewrite RTCP SR because they are rarely (never?) blindly forwarding all RTP
> packets for the life of the session. For example, a source projection mixer
> that switches video sources at keyframe boundaries based on active speaker
> audio levels. The source RTCP SRs will be periodic, not on keyframe
> boundaries, so what the mixer decides to forward can differ from what the
> source sent in that RTCP interval. The mixer effectively terminates and
> regenerates all RTCP with rules more complex than its simple RTP forwarding
> behavior, so it can’t simply forward RTCP.
>

If the server simply chooses a set of streams to relay to the receiving
client this shouldn't be an issue. If the server chooses to stop relaying
stream A and start relaying stream B, the receiver will either already have
received RTCP SR from stream B before, in which case it can sync with any
related audio stream. If it has not received any RTCP SR from stream B, it
will have to wait for one before syncing. Or am I missing your point here?

> ****
>
> ** **
>
> > The deltas only have a meaning in relation to the corresponding RTP
> timestamp.****
>
> ** **
>
> Actually, the RFC 5450 offsets only relate to themselves, not directly to
> the RTP timestamp. See the example in section 3 where “the choice of x is
> essentially arbitrary”. Also see the AVT archives circa 2005-2006 (search
> for draft-singer-rtp-hdrext) for discussions about “true offset” as defined
> by Ron vs. “deviation compared to a static delay” as defined by Magnus
> (which is what the final RFC wording reflects).****
>
> ** **
>
> The problem RFC 5450 attempted to solve is exactly the same as RMCAT now
> wants to solve with send timestamps, to separate source and network jitter,
> for “transmission rate adaptation” (Magnus’ words).
>

Yes, I agree that RFC 5450 tries to solve a similar (or even the same)
problem. But I also believe that doing it the RFC 5450 way has limitations.
I think there are benefits of being able to compare send timestamps of
different RTP streams if we know they're sent from the same sender, which I
don't think is possible if you want to have separate processing of audio
and video streams in your media servers.

> ****
>
> ** **
>
> Happy Holidays!****
>
> Mo****
>
> ** **
>
> ** **
>
> *From:* Stefan Holmer [mailto:[email-masked]]
> *Sent:* Monday, December 17, 2012 4:21 PM
> *To:* Mo Zanaty (mzanaty)
> *Cc:* Harald Alvestrand; [email-masked]
>
> *Subject:* Re: [rmcat] Send timestamps****
>
> ** **
>
> ** **
>
> ** **
>
> On Mon, Dec 17, 2012 at 9:15 PM, Mo Zanaty (mzanaty) <[email-masked]>
> wrote:****
>
> So we’re back to two pendulums two months after deciding multi-hop is
> initially out of scope. I’m glad, because it will likely be a major (or
> perhaps even dominant) deployment scenario, which we should not defer based
> on complexity alone.****
>
>  ****
>
> If we introduce a new send timestamp extension (which I’m unsure of the
> benefits over RFC 5450), I think it should be in NTP units not RTP media
> clock units. There is a precedent (in RTCP) for NTP timestamps that are
> synced across multiple RTP sessions, but there is no such precedent for
> synced RTP timestamps. Different media types with different media clocks
> would also present problems for synced RTP timestamps.****
>
>  ****
>
> But even with this new send timestamp extension (using absolute synced NTP
> timestamps), I don’t think it avoids Stefan’s problem of rewriting every
> RTP and RTCP packet at the server. Every RTP packet must still be rewritten
> with the new extension using the server’s NTP time, and every RTCP SR
> packet must still be rewritten for several reasons beyond congestion
> control.****
>
> ** **
>
> Perhaps. I don't see why rewriting the full (RTP timestamp, NTP timestamp)
> tuple is necessary for other reasons though.****
>
> ** **
>
> The problem as I see it is when we, at the receiving client B, are to
> compute the delta between packets generated by A and packets generated by
> C. Even if we convert them both to NTP, their NTP timestamps aren't likely
> to be in sync. They can be if S has rewritten the RTCP SRs so that the NTP
> timestamp of streams from A and C now are based on the clock of S (NTP
> timestamp S). But when that has been done we also have to rewrite the RTCP
> SR of all other streams from A and C which we at some point would like to
> have synchronized playback with, which will require all those streams to
> also go through S. So basically we will not be able to have to servers, S1
> and S2, which for example handle video and audio streams respectively.****
>
> ** **
>
> Maybe not a big issue, but to me it feels like a limitation without
> benefits. I don't see the point of requiring that timestamps used for
> congestion control should be set in relation to the RTP timestamps which is
> meant for doing synchronized playback.****
>
>  ****
>
>   ****
>
> So I really don’t see any benefit that a new extension would provide
> beyond RFC 5450. Perhaps I’m missing it.****
>
>  ****
>
> I’m also missing why you need to convert to NTP time in order to “jointly
> process all RTP streams”, because only the deltas really matter not the
> actual NTP time, and RFC 5450 gives you all you need for deltas.****
>
>  ** **
>
> The deltas only have a meaning in relation to the corresponding RTP
> timestamp. RTP timestamps of different streams (SSRCs) have different
> random offsets, and therefore can't be compared before converted to NTP. At
> least if you want to calculate the delta between packets belonging from two
> different streams.****
>
>  ****
>
>    ****
>
> > If we want to jointly process all RTP streams from A to B (at the
> receiver, B) we can convert the RTP timestamps of all streams to NTP using
> the (RTP timestamp, NTP timestamp) tuple in the RTCP SR report.****
>
>  ****
>
> Mo****
>
>  ****
>
>  ****
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Harald Alvestrand
> *Sent:* Monday, December 17, 2012 6:07 AM****
>
>
> *To:* [email-masked]
> *Subject:* Re: [rmcat] Send timestamps****
>
>  ****
>
> Separating the timestamp used as a congestion signal from timestamps that
> may be used for other purposes would seem to me to be an advantage in terms
> of architectural cleanliness.****
>
>
>
> RTP timestamps need to reflect media time. In a situation with relay nodes
> and changing paths, this may have uncomfortable amounts of variation from
> the time the packets enter the last hop to the recipient.
>
> Given that we won't have all systems deploying the new extension from Day
> One, we need to be able to fall back to the RTP timestamp (it's better than
> nothing), but it seems we shouldn't marry it more than we have to.
>
>               Harald
>
> On 12/17/2012 12:00 PM, Stefan Holmer wrote:****
>
>  I'll try to clarify with an example: ****
>
>  ****
>
> Let's say we have two clients (A and B) in a call. We want to estimate the
> bandwidth from A to B by looking at the variations in one-way delay, which
> we can measure by comparing deltas in RTP timestamp to deltas in
> receive-time. The problem with using RTP timestamps is that those represent
> capture time and not the send time of a frame, so we get a lot of noise
> related to variations in, e.g., video encoding time. To get around that we
> introduce RFC 5450 <http://tools.ietf.org/html/rfc5450> and now we can
> measure something much closer to actual network time. If we want to jointly
> process all RTP streams from A to B (at the receiver, B) we can convert the
> RTP timestamps of all streams to NTP using the (RTP timestamp, NTP
> timestamp) tuple in the RTCP SR report.****
>
>  ****
>
> Now let's add a server (let's call it S) in between client A and B. This
> server simply relays RTP packets (and might do some rewriting as well). Now
> we want to estimate the bandwidth between A and S, and between S and B. S
> will have to change the timestamp offset (RFC 5450) of the RTP packets to
> correspond to its send-time, but besides that all is fine.****
>
>  ****
>
> Now let's add a third client (C). S will relay packets from both A and C
> to B, so ideally we should jointly process all of those packets when
> estimating the bandwidth between S to B. However, packets sent from A will
> be based on the tuple (RTP timestamp A, NTP timestamp A) and packets sent
> from C will be based on (RTP timestamp C, NTP timestamp C), so we can't
> just convert to NTP at B and compare them. This can be solved in the server
> by rewriting RTP and NTP timestamps to (RTP timestamp S, NTP timestamp S)
> before relaying to B, which requires rewriting both RTP packets and RTCP
> packets. This is fine as long as you rewrite _all_ streams from the same
> sending client in the same way, otherwise you won't be able to synchronize
> them. Therefore all streams must go through the same server.****
>
>  ****
>
> I'm suggesting that we could introduce a new "send timestamp" which
> doesn't relate to the RTP timestamp (which is used for synchronized
> playback). Whenever a packet is sent (no matter if it's sent from A, C or
> S) it will be stamped with the send-time taken from the clock of that
> sender, for instance at 1 kHz or 90 kHz. Since it's not related to the RTP
> timestamp we won't have to rewrite the (RTP, NTP) tuple, and actually, we
> don't even have to convert to NTP at the receiver since all send timestamps
> are taken directly from the same clock. Removing the relation to the RTP
> timestamp is what I mean with an "absolute" timestamp (i.e., not relative
> to RTP timestamp).****
>
>  ****
>
> /Stefan****
>
>  ****
>
> On Sun, Dec 16, 2012 at 4:08 PM, Mo Zanaty (mzanaty) <[email-masked]>
> wrote:****
>
> Please clarify “absolute”. Do you mean another RTP timestamp using the
> same units but a random offset from the main RTP timestamp, and no implied
> sync across multiple RTP sessions? Or NTP timestamps with NTP units and
> implied sync across all RTP sessions (within the same sync domain,
> currently defined to be CNAME)?****
>
>  ****
>
> Please also clarify “relay conference server”, as there are many RTP
> topologies for conferencing. I think you are probably referring to a source
> projection mixer. Rewriting RTCP is standard practice for those, as well as
> most mixers. In fact, I can’t envision when it would be possible to blindly
> relay RTCP in any mixer/topology. So I’m not sure what complications you
> see arising from the use of RFC 5450.****
>
>  ****
>
> Mo****
>
>  ****
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Stefan Holmer
> *Sent:* Tuesday, December 04, 2012 4:23 AM
> *To:* rmcat WG
> *Subject:* [rmcat] Send timestamps****
>
>  ****
>
> Hi,****
>
>  ****
>
> In draft-alvestrand-rtcweb-congestion-01<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-01> we
> proposed using a send timestamp RTP header extension for measuring the
> packet inter-arrival time offset. In 02<http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-02> this
> was revised to instead reference RFC 5450<http://tools.ietf.org/html/rfc5450>,
> which specifies a header extension for specifying the send timestamp as an
> offset relative to the RTP timestamp.****
>
>  ****
>
> Thinking further about this, there seems to be a couple of benefits with
> defining a new, absolute, send timestamp header extension. If a relay
> approach to conference servers are used, the relaying server can simply
> rewrite the send timestamp of a packet to correspond to the time it was
> relayed. Doing so the receiver can jointly process all streams being
> received from the relay server, even though they were initially captured
> and sent by different senders (with different clocks). ****
>
>  ****
>
> With a send timestamp offset, accomplishing the same thing would require
> rewriting both the timestamp offset and the NTP time in the corresponding
> RTCP SR reports. And to keep video synchronized with audio the RTCP packet
> for the audio streams must also be rewritten in a similar way. This may not
> even be possible if the audio stream is relayed through a different server.
> ****
>
>  ****
>
> Do you see any problems with using an RTP header extension with absolute
> send timestamps? Are there any benefits with the send timestamp offset
> header extension I might be missing?****
>
>  ****
>
> /Stefan****
>
>  ****
>
>  ****
>
>  ** **
>