Subject: RE: 2 Adelaide drafts ahead

From: Yoshihiro Kikuchi <[email-masked]>
Date: Thu, 16 Mar 2000 19:32:12 +0900

Dear Paul and all,

>preparing the AVT group meeting in Adelaide and
>pursuing now our MPEG-4/RTP activities within
>the framework of the European IST project SoNG
>we would like to draw your attention to
>
>1- the fact that our draft
>
>"RTP Payload Format for MPEG-4 with
>Scaleable & Flexible Error Resiliency",
>(so-called 'GP' in short)
>
>is under revision.  Listening somewhat to the
>"complexity argument against us", we simplified a bit
>the existing draft (we dropped the fragmentation
>capability).
>
>2- the fact that a new draft
>
>- "RTP payload format for MPEG-4 Visual Advanced Profiles"
>is also being produced as a specialization exercise of
>our 'GP' towards visual and also as a "critique" of
>"draft-jnb-mpeg4av-rtp-00.txt ". Covering mostly the
>so-called simple visual profile and omitting a series of
>other important video profiles, we have to disagree
>with its arguments for a simple payload in general.
>

I read your Internet Drafts, but I cannot understand issues you are
trying to raise.

---  draft-gc-avt-mpeg4visual-00.txt ---

   The decoding of
   arbitrary shape VOP requires the dimensions of its bounding
   rectangle, its horizontal and vertical spatial position, as well as
   the shape coding type.  This information is respectively provided
   by the parameters "vop_width", "vop_height",
   "vop_horizontal_mc_spatial_ref" and "vop_vertical_mc_spatial_ref",
   "vop_shape_coding_type" in the VOP header.  The parameter
   "change_conv_ratio_disable" is also needed to be able to decode
   properly the video packet.  The "vop_constant_alpha parameter" as
   well as the "vop_constant_alpha_value" (if vop_constant_alpha==1),
   scaling factor applied to the decoded VOP before display need also
   protection.

The issue on the protection of the VOP header fields for the arbitrary
shaped objects was discussed one year ago(!) at the MPEG Seoul meeting.
The conclusion was to correct the HEC syntax to support the VOP header
for the arbitrary shape, and the corrected syntax has been adopted in
the
Corrigendum of MPEG-4 Visual(14496-2/DCOR1). All necessary information
in the VOP header for the decoding of the arbitrary shaped objects are
supported by the HEC.

   Depending  on  the  different  visual  profiles,  different  sets  of
   parameters will be present in the header of the VideoObjectPlane().
   In  the  simple  profiles,  essential  VOP  header  parameters  are:
   vop_coding_type,  modulo_time_base,  marker_bit,  vop_time_increment,
   fcodes when the VOP_coding_type is P or B, VOP_reduced_resolution if
   reduced_resolution_vop_enable is equal to 1.

You are correct in the meaning that all these information are supported
by HEC. But  VOP_reduced_resolution is NOT for the Simple Profile but
for the ARTS profile.

   Let us now consider the simple scalable profile supporting temporal
   and spatial scalability. Scalable or layered coding is very
   interesting for rate or congestion control on the Internet,
   especially in multicast. A key parameter in order to be able to
   decode a VOP in an enhancement layer is "ref_select_code" which
   signals the VOP that has been taken as a reference for the
   prediction.

   When scalability is applied on arbitrary shape objects, extra
   parameters need to be protected. For the shape decoding these
   parameters are "load_backward_shape", "backward_shape_width",
   "backward_shape_height", "backward_shape_horizontal_mc_spatial_ref",
   "backward_shape_vertical_mc_spatial_ref", "load_forward_shape",
   "forward_shape_width", "forward_shape_height",
   "forward_shape_horizontal_mc_spatial_ref",
   "forward_shape_vertical_mc_spatial_ref".  Another important
   parameter is "background_composition" which signals the usage of
   background composition in conjunction with scalability.

As you can see in Table G-2 in Annex G, any error resilience
tools(including video packet) are NOT supported for the enhancement
layer of the scalability. This is because the information above are not
supported in the HEC.
Even if the VOP header information above are protected by the RTP or
whatever, how you can used it for the error resilient decoding without
video packet? It is impossible to decode a middle part of a VOP without
video packet if the beginning part of the VOP is lost.

   The solution recommended in draft-jnb-mpeg4av-rtp-01.txt, when using
   the  combined  mode,  consists  in  transporting  this  configuration
   information in separate RTP packets, and in possibly repeating the
   corresponding RTP packets periodically if needed for protection
   purposes.  However, this vital information being restricted to a few
   bytes, to transport it in separate RTP packets leads to unnecessary
   overhead.

The configuration information is NOT restricted to a few bytes. It is
about 30 to 40 bytes for the typical case of the Simple Profile. When
the quantization matrix is transported by setting load_intra_quant_mat
and load_nonintra_quant_mat to 1 in Core, Main and other profiles, this
length will go up to about 200 bytes.

   More efficient transport can be reached by grouping this
   data with elementary stream data inside packets.  The same remark
   applies to the Group_of_VideoObjectPlane() entry point and to its
   corresponding header or configuration information.

It is possible to group the configuration information, GOV and the
subsequent VOP into one RTP packet by the fragmentation rule of the
MPEG-4 Visual RTP payload format. (See section 3.2 of
draft-ietf-avt-rtp-mpeg4-es-00.txt) This fragmentation rule is flexible,
and you can select either grouping or no grouping of the configuration
information or GOV adequately, depends on the bitrate, packet-loss rate,
etc.

   Depending on the visual profiles, different sets of key parameters
   will be present in the header of the VideoObjectPlane(), as
   described above.  Key parameters for the simple scalable, main and
   core profiles are not covered by the error resilience tools defined
   in MPEG-4.

I fully disagree with this as I wrote above.

   Five profiles have been defined for natural video content [3]:

Not only five profiles of MPEG-4 Visual Version 1 you raised, HEC
supports all Video coding tools having video packet for natural video
profiles in both Version 1 and Version 2. As you can see in the video
packet header syntax of Version 2 FDAM(14496-2/FDAM1), NEWPRED(ARTS
Profile), DRC(ARTS Profile), GMC(ACE Profile) are also supported by the
HEC.

I should disagree with the last sentence of bellow in
draft-ietf-avt-mpeg4streams-00.txt:

   A set of error resilience tools has been defined in the MPEG-4 vis-
   ual syntax in order to recover corrupted headers [5]. In particular,
   the VideoObjectPlane data is structured in video packets, the entry
   point being defined by the function video_packet_header(), and
   delimited by resync_markers. Basic configuration parameters can be
   inserted in the packet header.  However, this concerns only
   parameters used in the simple visual profile, many parameters
   essential in the simple scalable, main and core profiles are not
   covered by this mechanism [5].

Best regards,
Yoshihiro

----
        Yoshihiro Kikuchi

E-mail: [email-masked]
Corporate Research and Development Center
TOSHIBA Corporation
TEL: +81 +44 549 2288    FAX: +81 +44 520 1267

---

From: Yoshihiro Kikuchi <[email-masked]>
Date: Fri, 17 Mar 2000 12:29:55 +0900

Dear Paul and all,

>We have checked the output documents of the MPEG Seoul meeting (March
>99), and
>could not find any document addressing the above issues.
>
Which document did you check?
You can find the revised HEC syntax for the arbitrary shape at item 1 in
the WD1 of the corrigendum(N2693). This syntax has been adopted in the
ISO/IEC 14496-2/DCOR1(N2919). You can find the syntax in item 12 of the
DCOR1.

>We have been raising these points since november 98 (IETF in Orlando),
>we have raised them on the 4onIP list, many times during the year 1999,
>but these points have never been acknowledged.
>
>Yoshihiro, you were yourself, on the 6th of March 99,
>writing on the 4onIP list the following mail
>arguing in the same direction, and recognizing the
>limitations of the HEC mechanism for profiles other than the simple
>profile:
>

(snip)

Yes, I surely wrote this mail. If I remember correctly, there were
suggestions by Katto-san and Carsten to revised the HEC syntax to
support the shape coding.  I posted a contribution to the MPEG Seoul
meeting regarding this issue(m4480). The issue was discussed at the
meeting, and the conclusion was to revise the HEC syntax as suggested
and adopt it in the Corrigendum of MPEG-4 Visual so that this HEC
functionality can be used for all MPEG-4 Visual compliant codecs and all
kind of networks. So the problem I raised in my very old mail has been
solved. This is what I wrote in my previous mail.

>We have looked for the 14496-2/DCOR1 corrigendum and finally found this
>document as an output document of the MELBOURNE meeting in OCTOBER
>1999 (and not as an output from the Seoul meeting in march 99)..
>This corrigendum has apparently been prepared at the same time as the
>draft-jnb-mpeg4av-rtp-00.txt, and been discussed in October 1999 in the
>MPEG4 video group.
>
No. The corrigendum of 14496-2 was initiated at the March 1999 MPEG
meeting more than half year before Melbourne, and WD1 of the corrigendum
including the revised HEC syntax was published in March as I wrote
above. It has been integrated in the reference software (ISO/IEC
14496-5) before the MPEG July 1999 meeting. It is also before Melbourne.

>There were two ways for correcting these limitations,
>either by modifying the
>visual syntax, either by proposing to add protection in the
>encapsulation process.
>You apparently proposed the first solution, while we were and are still
>defending the second approach.  The second proposal being
>more flexible in the case of pre-encoded streams for
>streaming applications.
>
>Given all the discussions that had taken place between
>the MPEG-4 and the IETF
>on these questions between november 98 and october 99, we would have
>expected to post this corrigendum on the 4onIp reflector.

If I was not so kind enough to post the WD of the corrigendum, I'm sorry
for it. But I'm not sure if I, just as one member of MPEG, can post the
official output document from MPEG to the open reflector.

>At least, this would have allowed to discuss the pro's
>and con's on the two possible ways to solve the problem,
>and to find in a consensual way the
>best solution.
>
I understand there were discussion on the reflector before March as I
wrote above. And
there is a conclusion.

>The lack of notion of video packet and the lack of protection of the
>essential parameters for the enhancement layers in the
>context of scalable  coding is a problem.
>This means not being able to use the scalable features in
>a context of packet losses (since the enhancement layer is
>unlikely to be decodable, in case of losses),  although one
>of the interests of scalability is its amenability for
>congestion control on IP. It is especially essential for
>congestion and rate control when you have pre-encoded streams
>(streaming applications) or in multicast scenarios.
>
My understanding on the packet loss resiliency of the scalable coding is
that the scaleable coding itself has resiliency in the meaning that we
can use base layer for decoding even when enhancement layer is corrupt.
So, the base layer should be protected well. This is why MPEG-4 error
resiliency tools(video packet and others) are supported in the base
layer. But the resiliency required for the enhancement layer is
relatively low.
I remember that there were such discussions in MPEG before defining
Table G-1.

>So, with respect to this last point, it comes again to
>several possible solutions:
>One additional corrigendum to the video syntax, for the
>enhancement layers, or/and relying on an additional mechanism
>in the encapsulation process, which we again support.
>
I have to ask other MPEG-4 video experts, but is this corrigendum really
possible? It requires not only syntax changes, but also changes on the
video decoding process. All video coding operation using prediction from
the surrounding macroblocks, such as motion vector prediction and AC/DC
prediction, will be affected by this.

>> about 30 to 40 bytes for the typical case of the Simple Profile.
>
>30 to 40 bytes are a few bytes, compared to likely MTU sizes.
>Experiments made by people from Berkeley on congestion control
>assume MTU sizes of 1500 bytes.
>
Then, you can put the configuration information with a subsequent VOP
(or video packet(s) in the VOP) into a single RTP packet. The
fragmentation rules in section 3.2 of draft-ietf-avt-rtp-mpeg4-es-00 are
designed to be able to do so.
I hope you read carefully the specifications in the I/D.

Best regards,
Yoshihiro

----
        Yoshihiro Kikuchi