Subject: RE: IETF AVT last call on draft-ietf-avt-rtp-mpeg4-es-03.txt - ti mestamp of short header VOP

From: Gary Sullivan <[email-masked]>
Date: Fri, 8 Sep 2000 14:44:15 -0700

Kris et al,

A bit of H.263 explanation with a bit of a history lesson is
spliced into Kris's quoted message below:

+> -----Original Message-----
+> From: Kris Huber [mailto:[email-masked]]
+> Sent: Friday, September 08, 2000 1:04 PM
+> To: Young-Kwon LIM; Franceschini Guido; Koji Imura; IETF 
+> AVT; Colin Perkins; 'Yoshihiro Kikuchi'; Varesio Andrea
+> Cc: 4on2andIP-sys; Gary Sullivan
+> Subject: RE: IETF AVT last call on 
+> draft-ietf-avt-rtp-mpeg4-es-03.txt - ti mestamp of short header VOP
+> 
+> 
+> Dear Young and all,
+> 
+> Unless "MAY" is changed to "SHOULD" in "When the short video 
+> header mode is
+> used, the RTP payload format used MAY be that specified for 
+> H.263 in the
+> relevant RFCs or in other relevant standards,"  I think additional
+> modifications are necessary to accommodate MPEG-4 short 
+> header mode.  This
+> is because vop_time_increment_resolution, modulo_time_base, 
+> and time_code
+> are not present in a short header elementary stream.  Here 
+> is my proposed
+> modification (new text is indicated by surrounding it by 
+> "<<" and ">>").
+> No deletions are required.
+> 
+>    Timestamp: The timestamp indicates the composition time, or the
+>    presentation time in a no-compositor decoder. A constant 
+> offset, which is
+>    random, is added for security reasons. The detailed 
+> definition of the
+>    timestamp is as follows:
+> <<  - For a video object plane with short header, the 
+> timestamps (after the
+> first random timestamp) are equal to the presentation time sequence
+> associated with the semantics of the temporal_reference field.
+> Specifically, each timestamp value SHALL be calculated by 
+> rounding the
+> value of a precise clock that advances delta_time with each 
+> successive
+> video object plane with short header.  The time increment SHOULD be
+> calculated as delta_time = (((temporal_reference + 256 -
+> (temporal_reference of previous VOP) modulo 256) * 
+> 1001/30000) for each
+> successive video object plane with short header.  The RTP 
+> timestamp should
+> be consistently rounded or truncated to the resolution of 
+> the RTP timestamp
+> field.>>
+>    - <<Otherwise,>> for a video object plane, it is defined as
+> vop_time_increment (in units
+>      of 1/vop_time_increment_resolution seconds) plus the 
+> cumulative number
+>      of whole seconds specified by modulo_time_base and, if present,
+>      time_code of Group_of_VideoObjectPlane() fields.  
+>    - In the case of interlaced video, a VOP will consist of 
+> lines from two
+>      fields, and the timestamp will indicate the composition 
+> time of the
+>      first field.
+> (   - When multiple VOPs are carried in the same RTP packet, 
+> the timestamp
+>      indicates the earliest of the composition time within 
+> the VOPs carried
+>      in the RTP packet.)
+> Young-Kwon's proposed modification of the above bullet:
+>    - When multiple VOPs are carried in the same RTP packet, 
+> the timestamp
+>      indicates the earliest of the composition time<<s>> 
+> within the VOPs
+> carried in
+>      the RTP packet.  Timestamp information of the rest of 
+> <<the>> VOPs are
+> derived from
+>      the timestamp fields in the VOP header (modulo_time_base and
+>      vop_time_increment)<<, or from the temporal_reference field
+>      in the case of short video header>>.
+>    - If the RTP packet contains only configuration information and/or
+>      Group_of_VideoObjectPlane() fields, the composition 
+> time of the next
+>      VOP in the coding order is used.
+>    - If the RTP packet contains only visual_object_sequence_end_code
+>      information, the composition time of the immediately 
+> preceding VOP in
+>      the coding order is used.
+> 
+> Perhaps someone knows better wording for the first bullet.  
+> I used SHALL in
+> that paragraph to prevent accumulation of round-off error in 
+> the sequence
+> of time stamps, and SHOULD to allow for other picture clock 
+> frequencies
+> than the one specified in the MPEG-4 visual semantics.  More 
+> detail about
+> this difference in MPEG-4 short header and H.263 semantics follows:
+> 
+> Quoting ISO/IEC 14496-2:1999, clause 6.3.5.2:  "temporal 
+> reference:  This
+> is an 8-bit number which can have 256 possible values.  It 
+> is formed by
+> incrementing its value in the previously transmitted
+> video_plane_with_short_header() by one plus the number of 
+> non-transmitted
+> pictures (at 30000/1001 Hz) since the previously transmitted 
+> picture.  The
+> arithmetic is performed with only the eight LSBs."
+> 
+> In contrast the TR field in the H.263 spec allows different 
+> clocks than
+> 29.97 Hz:  "the value of TR is formed by incrementing its 
+> value in the
+> temporally-previous reference picture header by one plus the 
+> number of
+> skipped or non-reference pictures at the picture clock 
+> frequency since the
+> previously transmitted one.  The interpretation of TR 
+> depends upon the
+> active picture clock frequency. ..."  Negotiation of the 
+> picture clock
+> frequency is done by external means, I believe.
+> 
+> I suspect that fixing the H.263 TR frequency in the MPEG-4
+> temporal_reference semantics causes problems particularly 
+> for use of short-
+> header on 25 Hz material in Europe.

MPEG-4's short header mode corresponds with the H.263 "baseline"
profile of operation.  The baseline form only supports one picture
clock frequency -- 30000/1001 Hz.  The extra flexibility of
supporting other picture clock frequencies is an optional
feature that was added in version 2.  MPEG-4 does not support
compatibility for any optional H.263 optional features (even for the
Advanced Prediction mode, which would have been a very easy one
to specify since all the tools are in MPEG-4 in an identical form).
It only supports the baseline form.  The baseline form only has one
picture clock frequency.

So the MPEG-4 form of short header temporal reference corresponds to
the old definition in H.263 version 1.

The H.263 picture clock frequency is specified within the bitstream
syntax.  It is not something negotiated only externally.  The
standard picture clock frequency of 30000/1001 Hz is used unless
the following is indicated:
  1) Bits 6-8 of PTYPE are set to '111' (indicating that PLUSPTYPE
     is present), and
  2) The UFEP parameter of PLUSPTYPE is '001' (indicating that OPPTYPE
     is present), and
  2) Bit 4 of OPPTYPE is set to '1'
In which case a "Custom Picture Clock Frequency Code (CPCFC)" is
present that specifies the picture clock frequency.

MPEG-4 does not allow bits 6-8 of PTYPE to be '111', so
it does not allow its short header mode to use custom picture
clock frequencies.

The reason that 25 Hz was not originally supported in baseline mode
is the famous/infamous "CIF compromise" originating in H.261 which
created a picture size that favored the 625/50 world but used it
with a picture clock frequency that favored the 525/60 world.
This created a single video format for use world-wide so that
videoconferencing systems made in all countries would be able to
communicate with each other seamlessly.

So the intent of the compromise was that there should not be
separate flavors of H.261 or H.263 for the 50 Hz and 60 Hz worlds.
Everyone should be able to communicate equally well with everyone.

Today there is somewhat less concern over requiring a decoder to be
able to rescale and retime video data than there was in the early
'90s when that compromise was struck.  So when H.263 went up for
its second (H.263+) revision completed in Sept. '97, more flexible
picture format support was added.  Myself and Samson Cheung (then
of Compression Labs, Inc.) were the primary contributors to adding
that flexibility in the second version.

+> 
+> As I noted in the first sentence, an alternative to describing how to
+> compute timestamp in short header mode is simply to change 
+> MAY to SHOULD or
+> SHALL regarding use of H.263 payload format for short header mode.

Perhaps "SHALL" is the appropriate thing unless there is some alternative
that also provides interoperability.

It should probably also include something like "(e.g., RFC 2190 or RFC
2429)", as
those are the two packetization formats that have been defined that can
carry
H.263 video.

+> 
+> Best regards,
+> Kris Huber
+> Sorenson Laboratories, Inc.
+> 
+> -----Original Message-----
+> From: Young-Kwon LIM [mailto:[email-masked]]
+> Sent: Friday, September 08, 2000 1:08 AM
+> To: Franceschini Guido; Koji Imura; IETF AVT; Colin Perkins; 
+> 'Yoshihiro
+> Kikuchi'; Varesio Andrea
+> Cc: 4on2andIP-sys
+> Subject: Re: IETF AVT last call on draft-ietf-avt-rtp-mpeg4-es-03.txt
+> 
+> 
+> Dear Guido and all,
+> 
+> > > 
+> > > It would be better to add a description to clarify this. 
+> How about
+> adding
+> > > the following description in the timestamp definition in 
+> section 3.1?
+> > > 
+> > > - When multiple VOPs are carried in the same RTP packet, 
+> the timestamp
+> > > indicates the earliest of the composition time within 
+> the VOPs carried
+> in
+> > > the RTP packet. Timestamp information of the rest of 
+> VOPs are derived
+> from
+> > > the timestamp fields in the VOP header (modulo_time_base and
+> > > vop_time_increment).?
+> > > 
+> > Are you sure you want to possibly indicate the CTS of the 
+> second VOP in a
+> > packet ?
+> > This seems to me quite difficult to manage
+> > I would better do either of the following:
+> > - always indicate the CTS of the first VOP in the packet, 
+> even if the
+> second
+> > VOP will be presented first
+> > - disallow concatenation of VOPs whose decoding order 
+> differs from the
+> > composition order
+> 
+> "modulo_time_base" and "vop_time_increment" are the syntax 
+> elements of VOP
+> header invented for the purpose of carrying relative timing 
+> information.
+> So, I don't see any problem to use those information to 
+> calcute CTS. When
+> the receiver get a RTP packet with multiple VOPs, the CTS of 
+> first VOP can
+> be derived from the RTP time stmap and CTS of successive VOPs can be
+> calculated by using time stamp and two syntax elements in VOP header.
+> Calculating timing information of each VOP is nothing new to 
+> the decoder.
+> 
+> > 
+> > I am also doubtful about allowing variable frame rate 
+> videos to take
+> > advantage of such concatenation of VOPs in the same RTP packet.
+> > IMHO it would be safer to add some more constraints in the 
+> specification
+> > (i.e. prohibit VOP concatenation in case of variable frame 
+> rates), instead
+> > of adding the complexity of extracting TS information 
+> through cooperation
+> of
+> > different layers.
+> > In other words: VOP concatenation has pros and cons.
+> > Pros: optimizes bandwidth consuption
+> > Cons: deteriorates packet loss resilience; increases implementation
+> > complexity.
+> > I therefore deduce that VOP concatenation is nice but not 
+> that wonderful,
+> > and I would suggest to allow VOP concatenation only when 
+> this is trivial
+> and
+> > does not require significant complexity in the receivers 
+> implementations.
+> > 
+> 
+> I don't think there is additional complexity. MPEG-4 video 
+> decoder always
+> calculate the time difference of current VOP from the last 
+> one.  So, when
+> the decoder push the reconstructed data to composition buffer, it can
+> easily send a presentation time information to the compositor. It is
+> nothing to the implementation.
+> 
+> Sincerely,
+> Young.
+>

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Sat, 9 Sep 2000 14:06:15 +0900

Dear Kris, Gary, all,

I hope this message is not too late...

Thank you Kris for your clarification. I will incorporate your proposed
description for clarification in the Internet-Draft.

> that also provides interoperability.

I think "SHALL" is too strong because there are definitions related to short
video header mode in other places of the Internet-Draft. The change may
cause interoperability problem with other payload format.

> those are the two packetization formats that have been defined that can

Thank you for your suggestion for your editorial update.

Best regards,
Yoshihiro

----- Original Message -----
From: Gary Sullivan <[email-masked]>
To: 'Kris Huber' <[email-masked]>; Young-Kwon LIM
<[email-masked]>; Franceschini Guido <[email-masked]>;
Koji Imura <[email-masked]>; IETF AVT <[email-masked]>; Colin
Perkins <[email-masked]>; 'Yoshihiro Kikuchi' <[email-masked]>;
Varesio Andrea <[email-masked]>
Cc: 4on2andIP-sys <[email-masked]>
Sent: Saturday, September 09, 2000 6:44 AM
Subject: RE: IETF AVT last call on draft-ietf-avt-rtp-mpeg4-es-03.txt - ti
mestamp of short header VOP