Subject: RE: IETF AVT last call on draft-ietf-avt-rtp-mpeg4-es-03.txt

From: Franceschini Guido <[email-masked]>
Date: Thu, 07 Sep 2000 10:20:19 +0200

Dear Kikuchi-Sun,

> ----------
> From: 	Yoshihiro Kikuchi[SMTP:[email-masked]]
> Sent: 	Thursday, September 07, 2000 7:40 AM
> To: 	Koji Imura; IETF AVT; Colin Perkins
> Cc: 	4on2andIP-sys; Yoshihiro Kikuchi
> Subject: 	Re: IETF AVT last call on draft-ietf-avt-rtp-mpeg4-es-03.txt
> 
> Dear Imura-san,
> 
> Thank you very much for the comment.
> 
> > I reviewed the draft and had a question.
> > In the section 3.1, there is a description about the marker bit and
> > time stamp for the MPEG-4 visual.
> > According to this description, it is possible to include some VOPs
> > in one RTP packet.
> > In such a case, the only time stamp for first VOP is included in the
> > RTP header, but there is no time information for rest of VOPs in the
> > same RTP packet.
> > The temporal interval between VOPs is variable in MPEG-4 visual
> > So it is difficult to estimate the time stamp of the other VOPs.
> >
> > If there is a system that presentation/composition time is decided
> > by the RTP time stamp, how do the system estimate the time stamp
> > of rest of the VOPs?
> >
> Since MPEG-4 Video stream contains timing information in the VOP header
> fields (modulo_time_base and vop_time_increment), the timestamp can be
> derived from these fields inside the RTP payload.
> 
> It would be better to add a description to clarify this. How about adding
> the following description in the timestamp definition in section 3.1?
> 
> - When multiple VOPs are carried in the same RTP packet, the timestamp
> indicates the earliest of the composition time within the VOPs carried in
> the RTP packet. Timestamp information of the rest of VOPs are derived from
> the timestamp fields in the VOP header (modulo_time_base and
> vop_time_increment).?
> 
Are you sure you want to possibly indicate the CTS of the second VOP in a
packet ?
This seems to me quite difficult to manage
I would better do either of the following:
- always indicate the CTS of the first VOP in the packet, even if the second
VOP will be presented first
- disallow concatenation of VOPs whose decoding order differs from the
composition order

I am also doubtful about allowing variable frame rate videos to take
advantage of such concatenation of VOPs in the same RTP packet.
IMHO it would be safer to add some more constraints in the specification
(i.e. prohibit VOP concatenation in case of variable frame rates), instead
of adding the complexity of extracting TS information through cooperation of
different layers.
In other words: VOP concatenation has pros and cons.
Pros: optimizes bandwidth consuption
Cons: deteriorates packet loss resilience; increases implementation
complexity.
I therefore deduce that VOP concatenation is nice but not that wonderful,
and I would suggest to allow VOP concatenation only when this is trivial and
does not require significant complexity in the receivers implementations.

Bye
Guido

> Best regards,
> Yoshihiro
> 
> ----
>         Yoshihiro Kikuchi
> 
> E-mail: [email-masked]
> Corporate Research and Development Center
> TOSHIBA Corporation
> TEL: +81 +44 549 2288    FAX: +81 +44 520 1267
> 
> 
>

---

From: "Y. Matsui" <[email-masked]>
Date: Thu, 7 Sep 2000 18:06:39 +0900

Dear Guido, Kikuchi-san and all,

I agree on Guido's opinion that concatenation of VOPs is
allowed only when constant frame rate, that is defined when
the fixed_vop_rate field of the VOL header is set to 1.

Best regards,
Yoshinori Matsui
Matsushita Electric Industrial Co., LTD.

----- Original Message -----
From: Franceschini Guido <[email-masked]>
To: Koji Imura <[email-masked]>; IETF AVT <[email-masked]>; Colin Perkins <[email-masked]>; 'Yoshihiro Kikuchi'
<[email-masked]>; Varesio Andrea <[email-masked]>
Cc: 4on2andIP-sys <[email-masked]>
Sent: Thursday, September 07, 2000 5:20 PM
Subject: RE: IETF AVT last call on draft-ietf-avt-rtp-mpeg4-es-03.txt

---

From: "Y. Matsui" <[email-masked]>
Date: Fri, 8 Sep 2000 15:00:55 +0900

Dear Guido, Kikuchi-san, all,

> - disallow concatenation of VOPs whose decoding order differs from the

I would prefer the latter one.  I think it should be added to the current
draft-ietf-avt-rtp-mpeg4-es-03 document.

> does not require significant complexity in the receivers implementations.

How about the following change ?

[Original sentense in 1.1 MPEG-4 Visual RTP Payload Format]
The fragmentation rule recommends not to map more than one VOP in an
RTP packet so that RTP timestamp uniquely indicates the VOP time
framing. On the other hand, MPEG-4 video may generate VOPs of very
small size, in cases with a not coded VOP containing only VOP header or
an arbitrary shaped VOP with a small number. To reduce the overhead for
such cases, the fragmentation rule permits concatenating multiple VOPs in
an RTP packet when the VOP size is small.

[My proposed modification]
MPEG-4 video may generate VOPs of very
small size, in cases with a not coded VOP containing only VOP header or
an arbitrary shaped VOP with a small number. To reduce the overhead for
such cases, the fragmentation rule permits concatenating multiple VOPs in
an RTP packet when the VOP size is small.  However, for the variable
frame rate video, the fragmentation rule recommends not to map more
than one VOP in an RTP packet so that RTP timestamp uniquely indicates
the VOP time framing.

Is this sentense enough for prohibiting the mapping of the multiple
VOPs in a packet when variable frame rate video ?
If there is no any objections on above sentense, I would like to propose
to change the current draft as this.

Best regards,
Yoshinori Matsui

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Fri, 8 Sep 2000 15:17:57 +0900

Dear Guido, Matsui-san, Imura-san, all,

> - disallow concatenation of VOPs whose decoding order differs from the
I agree with Guido. I support the latter one becase it is a safer soluiton,
if there is no objection.

> I therefore deduce that VOP concatenation is nice but not that wonderful,

Therefore, fragmentation rule (4) in section 3.2 is defined that multiple
VOPs SHOULD NOT be concatenated. The concatenation is an exception rule only
when VOP size is small.

   (4) Two or more VOPs SHOULD be fragmented into different RTP packets so
   that one RTP packet consists of the data bytes associated with a unique
   presentation time (that is indicated in the timestamp field in the RTP
   packet header), with the exception that multiple VOPs MAY be carried
   within one RTP packet if the size of the VOPs is small.

> does not require significant complexity in the receivers implementations.
The complexity may be huge if we need to check every RTP packets. However,
because of the marker bit and fragmentation rule definitions, we need check
only RTP packets satisfying the following both two conditions:
- Marker bit equal to one, and
  (Because marker bit in section 3.1 is defined that it is set to one when
multiple VOP is concatenated.)
- The beginning of RTP payload is not a start code.
  (Because the fragmentation rule is defined that a start code is placed
always at the beginning of RTP payload, if exists.)

I would like to propose adding a note to describe this for clarify, so that
implementer can avoid unnecessary operation.

Best regards,
Yoshihiro

----
        Yoshihiro Kikuchi

---

From: "Young-Kwon LIM" <[email-masked]>
Date: Fri, 8 Sep 2000 16:08:10 +0900

Dear Guido and all,

> - disallow concatenation of VOPs whose decoding order differs from the

"modulo_time_base" and "vop_time_increment" are the syntax elements of VOP header invented for the purpose of carrying relative timing information. So, I don't see any problem to use those information to calcute CTS. When the receiver get a RTP packet with multiple VOPs, the CTS of first VOP can be derived from the RTP time stmap and CTS of successive VOPs can be calculated by using time stamp and two syntax elements in VOP header. Calculating timing information of each VOP is nothing new to the decoder.

> does not require significant complexity in the receivers implementations.

I don't think there is additional complexity. MPEG-4 video decoder always calculate the time difference of current VOP from the last one.  So, when the decoder push the reconstructed data to composition buffer, it can easily send a presentation time information to the compositor. It is nothing to the implementation.

Sincerely,
Young.

---

From: "Young-Kwon LIM" <[email-masked]>
Date: Fri, 8 Sep 2000 17:38:51 +0900

Dear Kikuchi-san,

>    within one RTP packet if the size of the VOPs is small.

I'm OK with this because this allows concatenation of VOPs. For more clarification I'd like to add "When multiple VOPs are carried in one RTP payload, the presentation time of the VOPs after the first one may be calculated by the decoder." 

BTW, where is a sentence prohibiting concatenation of VOPs with different decoding order?

Sincerely,
Young.

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Fri, 8 Sep 2000 21:35:51 +0900

Dear Young-Kwon, Guido, Matsui-san, Imura-san, all,

It seems to be better to list all description related the multiple VOPs
concatenation in the "current" draft text posted on Sep. 5 here in this mail
for the better understandings.

1. Introduction
   The fragmentation rule recommends not to map more than one VOP in an RTP
   packet so that RTP timestamp uniquely indicates the VOP time framing. On
   the other hand, MPEG-4 video may generate VOPs of very small size, in
   cases with a not coded VOP containing only VOP header or an arbitrary
   shaped VOP with a small number. To reduce the overhead for such cases,
   the fragmentation rule permits concatenating multiple VOPs in an RTP
   packet when the VOP size is small.

3.1 Use of RTP header fields for MPEG-4 Visual
   Marker (M) bit: The marker bit is set to one to indicate the last RTP
   packet (or only RTP packet) of a VOP. When multiple VOPs are carried in
   the same RTP packet, the marker bit is set to 1.

   Timestamp:
   - When multiple VOPs are carried in the same RTP packet, the timestamp
     indicates the earliest of the composition time within the VOPs carried
     in the RTP packet.

3.2 Fragmentation of MPEG-4 Visual bitstream

I think it is better to add cross-references (see section ..., definition of
....) for each description above in the draft so that every readers of the
draft can understand this set of rule easily.

Young-Kwon wrote:

> I don't think there is additional complexity. MPEG-4 video decoder always
calculate the time difference of current VOP from the last one.  So, when
the decoder push the reconstructed data to composition buffer, it can easily
send a presentation time information to the compositor. It is nothing to the
implementation.

> I'm OK with this because this allows concatenation of VOPs.

Oh, now there are agreements on the current description. As usual, if there
is any other strong disagreements, we should stick with the current text.
(except for editorial changes and clarifications)

Please look not only sentences I posted but a set of rules. It is defined
that multiple VOPs SHOULD NOT be concatenated so as not to force huge amout
of complexity. The concatenation is an exception rule only when VOP size is
small.

>For more clarification I'd like to add "When multiple VOPs are carried in
one RTP payload, the presentation time of the VOPs after the first one may
be calculated by the decoder."
Agreed.

> BTW, where is a sentence prohibiting concatenation of VOPs with different
decoding order?

There is no sentence now in the draft but only in mails exchanged.

Following is the list of proposed changes by mails on this issue for
editorial update and clarifications but not technical.

1) Add cross-cross references to each descriptions.
2) Add the following sentence to the timestamp definition
    "Timestamp information of the rest of VOPs are derived from the
timestamp fields in the VOP header (modulo_time_base and
vop_time_increment)."
3) Add the following description as a note to fragmentation rule (4)
    "When multiple VOPs are carried in one RTP payload, the presentation
time of the VOPs after the first one may be calculated by the decoder."
    "This operation is necessary only for RTP packets satisfying the
following both two conditions:
- Marker bit equal to one, and
  (Because marker bit in section 3.1 is defined that it is set to one when
multiple VOP is concatenated.)
- The beginning of RTP payload is not a start code.
  (Because the fragmentation rule is defined that a start code is placed
always at the beginning of RTP payload, if exists.)"

If there is no further objection, only changes reflected to the next version
of I/D will be these descriptions for editorial update and clarifications.

Best regards,
Yoshihiro

----
        Yoshihiro Kikuchi