Subject: [AVT] MPEG-2 elementary streams

From: Bodo Rüskamp <[email-masked]>
Date: Mon, 18 Nov 2002 16:47:09 +0100

Hi,

We have a problem in our company with RTP transmission of MPEG-2
data. We use a MPEG-2 hardware decoder chip that accepts the two
elementary streams (video & audio) through a FIFO. This chip is not
capable of doing RTP processing like reordering and dejitter.

So we wrote a RTP receiver, that receives a MPEG-2 program stream,
buffers the stream for some hundred milliseconds for reordering and
dejitter and then feeds the chips with the two elementary streams.
Since we are using the program stream, the relative order of video
and audio packets are preserved.

Most RTSP servers doesn't transmit program streams, they transmit
elementary streams using two RTP sessions.

In a program stream we have the SCR field in the pack header, which
is a monotonically increasing value. The decoding time (DTS) of
elementary packets in an elementary stream is also a monotonically
increasing value, but has no relation to the DTS of the other elementary
stream. We noticed that an audio packet is transmitted 100 to 500 msec
later than the video packet with the same DTS.

Question: how can the RTP receiver do the buffering for dejitter and
reordering without the timing information of the program stream (the
SCR in the pack header).

(1) transmission using program stream

         src ----- tx ----- rx ----- demux ===== decoder

(2) transmission using two elementary streams

	src ----- demux ===== tx ===== rx ===== decoder

----- program stream
===== elementary streams (audio & video)
src = file on disk (e.g. from VCD, S-VCD or DVD)
demux = splits program stream into two elementary streams
tx = RTP transmitter
rx = RTP receiver
decoder = MPEG decoder

; Bodo

-- 
Bodo Rüskamp <[email-masked]>
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Chris Sloan <[email-masked]>
Date: Mon, 18 Nov 2002 11:34:52 -0800

Hi Bodo,

If the rtp server is compliant with RFC 2250, you should be able to
synchronize the two elementary streams via the timestamp in the
RTP header.

(Excerpt from http://www.ietf.org/rfc/rfc2250.txt)
3.3 RTP Fixed Header for MPEG ES encapsulation

The RTP header fields are used as follows:

Payload Type: Distinct payload types should be assigned for video 
elementary streams and audio elementary streams. See [4] for payload type 
assignments.

M bit: For video, set to 1 on packet containing MPEG frame end code, 0 
otherwise. For audio, set to 1 on first packet of a "talk-spurt," 0 otherwise.

PT: MPEG video or audio stream ID.

timestamp: 32-bit 90K Hz timestamp representing presentation time of MPEG 
picture or audio frame. Same for all packets that make up a picture or 
audio frame. May not be monotonically increasing in video stream if B 
pictures present in stream. For packets that contain only a video sequence 
and/or GOP header, the timestamp is that of the subsequent picture.

Chris

---

From: Bodo Rüskamp <[email-masked]>
Date: Mon, 18 Nov 2002 21:21:57 +0100

Hi Chris,

> synchronize the two elementary streams via the timestamp in the

The timestamp in the RTP header is the presentation time (PTS), but
I need the SCR from the pack header. My RTP receiver doesn't do the
video decoding and has therefore no knowledge when a picture is to be 
displayed.

The following is an excerpt from a DVD movie. You can see, that the
PTS is not linear and the video packets are transmitted earlier 
(according
to the PTS) than the audio packets. The decoder chips fails if the order
of the video and audio packets are not in the right SCR order.

v 00001.44162 00001.72000 00001.68000
a 00001.46437 00001.52800 00001.52800
a 00001.49688 00001.56000 00001.56000
a 00001.52938 00001.59200 00001.59200
a 00001.56027 00001.62400 00001.62400
a 00001.59277 00001.65600 00001.65600
a 00001.62528 00001.68800 00001.68800
a 00001.65617 00001.72000 00001.72000
a 00001.68867 00001.78400 00001.78400
a 00001.75206 00001.81600 00001.81600
a 00001.78457 00001.84800 00001.84800
a 00001.81708 00001.88000 00001.88000
a 00001.84960 00001.91200 00001.91200
a 00001.88047 00001.94400 00001.94400
a 00001.91298 00001.97600 00001.97600

column 1 = video / audio
column 2 = SCR (from PAK header)
column 3 = PTS (from PES header)
column 4 = DTS (from PES header)

According to the SCRs, I have to deliver the audio packet with the 
presentation
time of 1.72 about .21455 seconds later than the video packet. This 
delay can
be computed from the SCR, but not from the PTS.

; Bodo

---

From: Chris Sloan <[email-masked]>
Date: Mon, 18 Nov 2002 15:53:09 -0800

At 09:21 PM 11/18/02 +0100, Bodo Rüskamp wrote:
The following is an excerpt from a DVD movie. You can see, that the
>PTS is not linear and the video packets are transmitted earlier (according

I'm not sure how the decoder chip can know what the "proper" SCR order is.
Do you input the SCR or PTS values for each frame?  How does the chip
synch. the audio and video rendering?

It seems to me, the decoder just needs each video/audio frame before the 
DTS for
that frame.  You can convert a video PTS to its DTS by buffering the video
frames and applying this rule to reference frames:

DTS(n+1) = PTS(n)

where n = reference frame number in decode order

and obviously for B frames DTS(n) = PTS(n)

If you control the state of the decoder, you just need to input the initial 
reference frame
before starting the decoder.

Since you are now generating continuous DTSs for each audio and video 
frame, you
should be able to input the appropriate audio/video frame to the decoder at 
the proper time.

Is there something else going on that I am not getting?

Chris

---

From: Bodo Rüskamp <[email-masked]>
Date: Tue, 19 Nov 2002 16:44:43 +0100

On Dienstag, November 19, 2002, at 12:53  Uhr, Chris Sloan wrote:
> Do you input the SCR or PTS values for each frame?

The chip gets the PTS. The SCR is needed only for the 
dejitter/reordering
buffer at the RTP receiver. After the demux, only the PTS is passed.

> How does the chip

I don't have internal documentation about the chip. Just a simple 
library
interface to delivers the data and the PTS.

> should be able to input the appropriate audio/video frame to the 
> decoder at the proper time.
>

Yes. My question is not about the timing in one elementary stream. It's
about the relative timing of the two elementary streams:

Back to the example from my last mail:

The transmitter sends the video packet with PTS 1.72 at T+1.44162 and
the audio packet with PTS 1.72 at T+1.65617. That's a temporal offset
of 0.21455 seconds.

If the network delay of the video packet is 0.5 seconds and the network
delay of the audio packet is 0.2 seconds, then the RTP receiver will 
get the
video packet at T+1.94162 and the audio packet at T+1.85617.

At the RTP transmitter, the audio packet was sent 0.21455 seconds late,
at the RTP receiver the audio packet is now 0.08545 seconds early.

My question: How does the RTP receiver restore the temporal order of the
two packets that was present when the packets were sent?

; Bodo

---

From: Stephen Casner <[email-masked]>
Date: Wed, 20 Nov 2002 07:07:43 -0800 (PST)

> two packets that was present when the packets were sent?

RTP does not provide any mechanism for the receiver to restore the
sending temporal order of the packets.

However, I believe this is not the right question.  There is no
assurance that the server will send the packets with the same temporal
order or spacing as they might have appeared in a Program Stream.
What you really need to do is restore the audio and video frames to
the decode order that the chip expects.  I believe you need to
construct and use the DTS as suggested in a previous message.

Furthermore, in the example you have given, the presentation
timestamps have the same offset.  This might not always be the case
because RTP calls for these timestamps to have a random offset.  RTP
provides a means to synchronize the two presentation timestamp streams
using the association between reference (NTP format) timestamps and
media (RTP format) timestamps in the RTCP Sender Report packet.

I am not an MPEG expert, so I don't know if a Program Stream will
contain an SCR for each frame.  However, I'm pretty sure that a
Transport Stream does not -- the SCR appears only periodically,
perhaps once every few seconds -- and Transport Streams are the more
common form used with the multiplexed stream mode of RFC 2250.  This
is similar to the way that RTCP SR packets are sent periodically.

                                                        -- Steve