Subject: Re: Updated I/D for MPEG-4 Audio/Visual RTP payload format

From: Stephan Wenger <[email-masked]>
Date: Thu, 01 Jun 2000 14:47:37 +0200

Dear Shigeru, group,

Thanks for your reply.

>At first, we would like to describe only MPEG-4 specific things in this 
>draft.
>As FIR is a common technique for all video coding, other common document
>seems to be appropriate.

Ok, so you propose to leave "common" feedback mechanism like FIR to
some other place, and put only "MPEG-4" specific feedback stuff into
your payload spec.  I'm not sure that this is the right thing to do, for three
reasons: 1) unavailability of a 'general' feedback spec, 2) previous
payload specs contents and 3) the difficulty to define "common feedback
mechanisms".

Regarding 1)
When we discussed RFC2429, covering H.263+, we had already adopted
reference picture selection (NEWPRED), due to work of your company (OKI).
Early Internet drafts of RFC2429 suggested at least two different ways
to handle NEWPRED feedback, but were deleted from the drafts because
of scaling problems.
Several people though of coming up with a 'unified feedback solution', but,
until now, nothing happened.  This might be partly lack of interest, and partly
the difficulty of the problem.  Consequently, when adopting your draft
unchanged, there would not be a standardized way to perform FIR and
similar in IETF environments such as SIP.  The only way to respond to
packet losses, beyond responding on RTCP receiver reports, would be
to use NEWPRED.  Correct me if I'm wrong here.

Regarding 2)
There are currently two categories of RTP payload specs with respect to the
feedback problem.  Category one defines RTCP feedback, and category
two does not.  There are, to the best of my knowledge, no payload specs
proposed or accepted that follow your idea to include only a few feedback
mechanisms specific to a media coding and not in include other known
ones for the same media coding.  Certainly, not so important ones like FIR.

Regarding 3)
FIR is common to all video coding?  Well, people not using compression
(BT565) will disagree.  Audio folk will disagree.  Motion JPEG proponents will
also disagree.  On the other hand, NEWPRED is unique to MPEG-4?  You
know that H.263+, available as a standard since 1998, includes NEWPRED
as well, so NEWPRED is NOT unique to MPEG-4.  A strong argument to
include NEWPRED into a unified RTCP feedback RFC?  Furthermore, there
are mechanisms like Slices out there, which are available in many, but not all,
predictive video coding standards.  Are they candidates for a 'common' feedback
doc, or should they go into a RTCP section of a specific payload?

>Second, new proposal just before the last call is not good. We largely 
>discussed
>the issues of RTCP for NEWPRED since Adelade meeting, and revised text.
>It also takes much time to discuss FIR, however, we have enough time.

With this I disagree strongly.

First let me say, that I'm not on a mission to prevent MPEG-4 ES packetization
to move forward.  It's not my plan to artificially delay the process, and 
if it would be,
then I'm sure that the our chairs would be able to move forward with the draft
regardless of my concerns.  If people in the ITU-T want to prevent MPEG-4
from happen in H.323, then there are other fori more appropriate to achieve 
that
goal.
Having said this, however, I have to emphasize that a) we are talking about a
Internet Draft, b) that you asked for comments, and c) that I see an 
architectural
flaw here.  I'm aware that additional delay may cause some delay in ITU-T
Q.13/16 and Q.14/16 work (on H.245 for example) as well (meaning that
possible the adoption of MPEG-4 visual/audio to H.323 would be delayed
somewhat).  That could be a reason for us (me and you guys) to work harder,
but not to let a draft pass that contains known architectural problems.

Can you please inform this group about your timing constraints?  Why the rush?

The way to move forward:

I see three ways to move forward from now
1) Someone convinces me that I should shut up and let things go.  Probably I'm
   stickling to principles that are not the general principles of AVT.  Anyone
   besides the proponents of the draft (outside the MPEG core)?
2) Delete section 5 (the rest seems to be, apart from language/editorial 
problems,
   ok, at least to me).  Work together with interested parties, including 
myself, on
   a unified feedback solution, in a separate RFC or a set of separate RFCs.
3) Leave section 5 and bring it into such a shape that it can incorporate other
   feedback schemes, including FIR and SLice Re-transmission at least.  I'm
   still working on my changes for that.

Stephan

---

From: "Shigeru Fukunaga" <[email-masked]>
Date: Fri, 2 Jun 2000 08:43:49 +0900

Dear Stephan and all,

Thank you for your comments.
I would like to try to answer your questions.

> to handle NEWPRED feedback, but were deleted from the drafts because

Sure. I was one of the proponents of NEWPRED to H.263+/AnnexN.

> to use NEWPRED.  Correct me if I'm wrong here.

I don't understand the correct reason why the backchannel of RTP was dropped
two years ago. So we could not propose another solution for H.263+ RTP.
At least I understand that backchannel over RTP got storongly objection, and 
many persons preferred to RTCP.

Threrefore I proposed upstream RTCP for MPEG-4 case.

> also disagree.  

I mean the video codings using inter frame coding, such as H.261/3 and 
MPEG-1/2/4.

> On the other hand, NEWPRED is unique to MPEG-4?  You
> as well, so NEWPRED is NOT unique to MPEG-4.  

That' right. Only H.263 and MPEG-4 have a NEWPRED technique.

> A strong argument to

This issue was discussed at Adelaid. The conclusion was that the RTCP of
NEWPRED was accepted to be included in MPEG-4 RTP/RTCP doc. 
And if the common feedback draft will be made in the future, the specification of 
RTCP of NEWPRED in MPEG-4 will be considered as a base idea.

> Can you please inform this group about your timing constraints?  Why the rush?

We would like to formally use MPEG-4 on H.323 in our products as soon as 
possible. As the next version of H.323 will be decided in Nov. meeting of ITU-T/SG16, 
we would like our proposed RTP format to be RFC before the meeting.

>    a unified feedback solution, in a separate RFC or a set of separate RFCs.

In this case, we cannot formally use NEWPRED of MPEG-4 on H.323 for at least 
2 years, until the next H.323 will be decided in 2002 or later.
We would like to disagree with this case.

>    still working on my changes for that.

FIR and re-transmission can cause new congestions, because the amount of
bits is much increased during congestion. This is much against to the IETF
policy of congestion control. We have to solve this issue, if you want to include 
FIR and re-transmission into our draft.

Best Regards,
Shigeru 
--
   Shigeru Fukunaga ([email-masked])
   Oki Electric Industry Co., LTD.
   Tel. +81 6 6949 5101; Fax. +81 6 6949 5108

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Mon, 5 Jun 2000 13:28:32 +0900

Dear Stephan, Fukunaga-san, all,

Thank you very much for your reply.

I'm one of the authors of the I/D, but I myself is not a NEWPRED proponent.
I'm writing here just as an editor.

There are two things discussed in the thread: (1) Stephan's proposal to add
the FIR description to the I/D, and (2) How to treat NEWPRED RTCP part
(section 5) of the I/D. These two issues are related, but I'm afraid these
are too much mixed up throughout the discussion.

(1) FIR (Stephan's proposal)
I also think that it may have some value to have a feedback mechanism that
covers all MPEG-4 Visual Profiles but not only ARTS Profile. However, we
need to know exactly what the proposal is to add any technical change on the
I/D. We heard only a rough idea on Stephan's proposal. We also need to
discuss whether Stephan's proposal matches best to the I/D, or some others.
These requires precise description on the proposal, and probably requires
further discussions.

I think it is hard to take the third way proposed in Stephan's latest mail
to add the FIR description to the I/D at this very late stage. It would be
nice to have a discussion in the future.

(2) NEWPRED RTCP (section 5)
It is true that NEWPRED RTCP covers only ARTS Profile. It would be desired
to have feedback mechanism that covers all MPEG-4 Visual Profiles. This does
need further discussions and thus impossible to add it to the I/D, but could
be nice to have such scheme in a separate I/D.

I don't want to prevent the future discussions for such feedback mechanism,
Stephan's or whatever, that can be used combined with the RTP part of the
I/D. To clarify this, I would like to propose adding some descriptions to
the I/D that show the followings.

- The RTCP described in section 5 is OPTIONAL. The use of this RTCP is
limited to NEWPRED in ARTS Profile.
- For profiles other than ARTS, some feedback mechanisms other than NEWPRED
may be defined in separate documents and may be used combined with the
MPEG-4 Visual RTP part of the I/D.

Comments are welcome.

Best regards,
Yoshihiro

----- Original Message -----
From: Stephan Wenger <[email-masked]>
To: Shigeru Fukunaga <[email-masked]>; Yoshihiro Kikuchi
<[email-masked]>; 4on2andIP-sys
<[email-masked]>; IETF AVT <[email-masked]>
Sent: Thursday, June 01, 2000 9:47 PM
Subject: Re: Updated I/D for MPEG-4 Audio/Visual RTP payload format

---

From: "Dr. Injong Rhee" <[email-masked]>
Date: Fri, 16 Jun 2000 18:48:49 -0400

Hi All,

I would to add some comments and express some concerns over the current draft.

First,
as Stephan pointed out, we need to clean up the language. Many parts are
 written in a very confusing manner. I will list a subset
of things that I found confusing below. Read on....

Second, this draft is only specific to newpred, but it is still called
profile for mpeg-4 audio/visiual streams. With this title, I suggest
adding other ways of accomplishing the same affair, in particular, other
feedback mechanisms such as retransmission and even for forward error
correction.

Third, it seems to me that feedback mechanisms, especially the
types of feedback that this draft specifies (NACK or ACK), are
very generic and can be used in many places. They may not
be specific for newpred only. Wouldn't it be a good idea to
delete section 5, and discuss a more general unified
feedback mechansim. I believe there was some drafts on
retransmission and feedbacks some time ago. Can we try to
extend on that? I am willing to work together on this issue.

Fourth, discussion on congestion control in Section 5.3 is not
convincing. It seems that the authors are confused about the
issues of loss recovery with those of flow control. They are
orthogonal issues.  For instance, it seems that the authors
believe retx causes more congestion. This is not the case.
retx is a recovery technique, but not a flow control technique.
Instead, you should argue one technique gives lower or higher
bit overhead or not -- which is a totally different ball
game.

Fifth, Does retransmission have more bit overhead than NEWPRED?
It seems that the authors believe that. I don't believe that
this is a commonly accepted statement because I don't see
any evidence or proof for it in any other literature they
publish...need more extensive testing to say retx is worse
than newpred. On the contrary, my experiments suggest that
retx and newpred use similar amount of bit overhead -- of
course this depends on the amount of motion present in
the video input. Inaddition, why does retx cause lower
coding efficiency????

Sixth, the authors need to say that in order for newpred to work,
newpred needs real-time encoder. Thus, it MAY not be suitable for
streaming. For instance, in section 6.1, in the applications which use this
type of media type, the authors need to clarify this point.
How can you design internet email applications in a scalable manner
with real-time encoders which need to adapt reference frames
on the fly depending on the ack from the player????

I believe that this document needs more work. I will be glad to help if
requested.

Regards,
Injong

-- 
Injong Rhee, Department of Computer Science
North Carolina State University, Raleigh, NC 27695
Home page: http://www.csc.ncsu.edu/faculty/rhee
Email: [email-masked], Phone: 919-515-3305, Fax: 919-515-7925

---

From: Stephan Wenger <[email-masked]>
Date: Sat, 17 Jun 2000 01:44:36 +0200

Hi All,

some comments on the procedural issues, especially on
ITU-T vs. IETF timing issues (and how we can cope with
then and still get a reasonable feedback mechanism for
MPEG-4).

The main reason for the rush with the MPEG-4 ES packetization
draft is to meet the decision deadline of two new versions of
ITU-T recommendations that take advantage of MPEG-4 and Newpred.
In order to keep the packetization scheme in those standards, a
RFC number has to be issued by that time.  It's a hard deadline --
if the draft, even after passing any last calls, is still in the RFC editor's
queue, SG.16 will take the mechanism out.  The next SG16 meeting
is some time this fall (October?  I'll look it up ad soon as I'm back in
my office).  That means that there is an additional delay for having
MPEG-4 in H.323 systems when IETF/AVT cannot meet the deadline,
which, in turn, is the reason for the rush with the draft.

References to MPEG-4 packetization occur in the drafts of H.225.0v.3
and H.245 v.7 (was it 7?  Something fairly high).

ITU-Recommendations of the H.323 series continue to evolve.  New
versions of H.245 are decided almost every study group meeting.  The
update schedule on H.225.0 is much slower.

The interesting thing now is, that adding the MPEG-4 ES packetization
without Newpred concerns both H.225.0 and H.245, whereas, based
on the language the drafts, the added Newpred features would concern
only H.245.  Maybe the proponents of Newpred want to comment on that.

If I'm right with this assessment, then we can delete the controversial
section 5 out of the I-D and move forward quickly, in the hope to get
over with it in time for the H.225.0 decision in fall.  Simultaneously, we
can work on a feedback solution in an additional I-D (either MPEG-4
specific or not).  This would be added later, but, due to the quick update
schedule of H.245, it would probably less time then two years.

Stephan

---

From: Stephan Wenger <[email-masked]>
Date: Sat, 17 Jun 2000 01:19:06 +0200

Hi all,

Sorry for my long silence.

>the video input. Inaddition, why does retx cause lower

Newpred typically uses MORE bits than a well designed re-transmission
algorithm.  Re-transmission actually restores the signal completely
in the most efficient way, because only those bits are conveyed which
were lost before.  But, and that is the critical point, re-transmission
induces substantial delay, whereas Newpred does not.

Newpred's advantage is the low number bits that are added in case
compared to other source coding mechanisms such as Intra macroblock
refresh, which is, similar to Newpred, also neutral to the delay.

This just for clarification.

Stephan

---

From: "Dr. Injong Rhee" <[email-masked]>
Date: Sat, 17 Jun 2000 09:13:55 -0400

> refresh, which is, similar to Newpred, also neutral to the delay.

Stephan,

I am afraid I have to disagree on this. :-)
Retx can help even with delays. Retransmitted
packets can be used to restore displayed reference frames and these
restore reference frames can be used by later frames as reference
 to stop error
propagation. The same effect as NEWPRED can be achieved. Retx will
take the same delay as NEWPRED and gives the similar benefit (sometimes
with much lower bit overhead)
Additional advantages of retx  are that it can be used for
*multicast* and also for
*streaming* without requiring re-encoding to change reference frames....

you can find more details about it
from an upcoming jsac paper by me and my student on this:

Error recovery for Interactive video transmission over the internet.
JSAC, Vol 18. No. 6 June 2000.

If there is an interest, I can send you the draft electronically.

Regards,
Injong

---

From: "Gunnar Hellstrom" <[email-masked]>
Date: Sat, 17 Jun 2000 20:03:03 +0200

Stephan,

I was in the same situation in February, in an SG16 meeting with a document
for decision, and still no RFC for my RTP text packetization. A scaring
experience, but with a happy end that time.  The IETF approval came the same
day as the final plenary of SG16, and the decision could go forward.

There is another mechanism published in the RTP profile (RFC 1890). It is to
just register the payload with IANA and register that ITU is the managing
body of the payload description. Then include it in your ITU Recommendation.

I do not know your situation exactly. For me, I am glad that I selected to
go the IETF way, because the resulting RFC 2793 is then usable both for
H.323 and SIP text conversation purposes. We can have Total Conversation
with no transcoding between text in any multimedia protocol.

But it was frightening..

Regards
Gunnar Hellström
______________________________________
Gunnar Hellström
Omnitor AB
Alsnögatan 7, 4tr
S-116 41 Stockholm
Sweden

Mob +46 708 204 288
Tel +46 8 556 002 03
Fax +46 8 556 002 06
Video +46 8 556 002 05
Txt (All kinds) +46 8 556 002 05
E-mail [email-masked]
WWW:     http://www.omnitor.se

> -----Original Message-----
> From: Stephan Wenger [mailto:[email-masked]]
> Sent: Saturday, June 17, 2000 1:45 AM
> To: Dr. Injong Rhee; Shigeru Fukunaga; Yoshihiro Kikuchi; 4on2andIP-sys;
> IETF AVT

---

From: "Shigeru Fukunaga" <[email-masked]>
Date: Mon, 19 Jun 2000 10:52:05 +0900

Dear Rhee and all, 

Thank you for your comment.

> of things that I found confusing below. Read on....

We cleaned up the text and will release the next version.

> feedback mechanisms such as retransmission and even for forward error

This draft concerned with MPEG-4 audio and visual. Not only for feedback 
channel. The description of feedback channel is a subsection of this draft.

> extend on that? I am willing to work together on this issue.

The target of this draft is only MPEG-4 tools, which syntax and semantics
have been defined in ISO/IEC 14496.
In MPEG-4 audio, vidual, and systems, there are some tools using feedback
channel. We would like to define the RTCP only for these MPEG-4 feedback
tools.
For the moment, only NEWPRED is the candidate to use this new RTCP,
because we got no request from other feedback tools.

Again, we would like to discuss about only tools defined in MPEG-4 
standard. I understand that re-transmission and intra refresh is good 
techniques for packet loss. These tools are workable not only for MPEG-4
but also H.26x and MPEG-x. I think it is better that these tools will be 
discuss in the generic RTCP. The RTCP definition of our draft will not 
prevent the feture discussion of the generic feedback tools.

> bit overhead or not -- which is a totally different ball

I'm sorry our text in Section 5.3 made confusion. If you think 
some text is not appropriate, we can remove them or revise them ASAP.

We made our text according to the results in Adelaid meeting.
In Adelaid meeting, the chairman of AVT WG clearly indicated that
re-transmission may cause another congestion. So all proposals
of re-transimission were required to include the good congestion control.

This is the problem of re-transmission. Not of our draft. If you want,
we can remove the text to indicate the features of re-transmission from 
our draft.

> the video input. Inaddition, why does retx cause lower

We don't want to compare the NEWPRED with other refresh tools now.
These technical comparisons have been done in ISO/MPEG meeting 
for many years. ISO selected NEWPRED into MPEG-4 visual, and we 
would like to use the NEWPRED over IP. That' all reason we propose RTCP
in this draft.

We don't want to discuss about the details of re-transmissoin.
We will revise our text not to refer about re-transmission in the next version.

> on the fly depending on the ack from the player????

Yes, you are right. NEWPRED needs the real-time encoder.
The definition of mpeg4-newpred-upstream-message is defined for
real-time communications using SDP. Not for e-mail apprications.
So, we will add some explanation in the next version.

Best Regards,
Shigeru

---

From: "Shigeru Fukunaga" <[email-masked]>
Date: Mon, 19 Jun 2000 10:59:08 +0900

Dear Stephan and all,

Thank you for your clarification.

> induces substantial delay, whereas Newpred does not.

I didn't know that a well designed re-transmission does not use large bits.
So we will remove text about re-transimssion from our draft.

Best Regards,
Shigeru

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Mon, 19 Jun 2000 11:10:21 +0900

Dear Rhee, Stepha all,

Thank you very much for all who sent us comments.

Although we need a little more time to make a complete answer to all
comments, here is a quick reply.

- The authors are now working on the upgrade of the English quality. The
text will be ready in a few days.

- The RTP part of the I/D is NOT specific to NEWPRED. ONLY RTCP (section 5)
is specific to NEWPRED. Several companies have done cross-verification tests
(4 companies for Visual RTP and 2 companies for Audio RTP) WITHOUT NEWPRED.

- A new retransmission scheme other than NEWPRED may be valuable, but it
needs further discussions. I feel very hard to add new retransmission
scheme, or any other new technical items, now without having enough
discussions.

- I agree that we need to discuss now how to treat NEWPRED RTCP part
(section 5) of the I/D. Not only the proponent's voice, but also others from
the reflector should be considered to make a decision.

- Stephan's proposal on the procedure to defer only NEWPED RTCP will be
taken into account for the discussion, although the procedure needs a
confirmation. Any ITU-T experts?

Best regards,
Yoshihiro

----- Original Message -----
From: Dr. Injong Rhee <[email-masked]>
To: Stephan Wenger <[email-masked]>; Shigeru Fukunaga
<[email-masked]>; Yoshihiro Kikuchi <[email-masked]>;
4on2andIP-sys <[email-masked]>; IETF AVT
<[email-masked]>
Sent: Saturday, June 17, 2000 7:48 AM
Subject: Re: Updated I/D for MPEG-4 Audio/Visual RTP payload format

---

From: "Shigeru Fukunaga" <[email-masked]>
Date: Mon, 19 Jun 2000 11:39:29 +0900

Dear Stephan and all,

> only H.245.  Maybe the proponents of Newpred want to comment on that.

As Kikuchi-san said, this procedure needs a confirmation from ITU-T.

There seems no space to indicate the RFC number in H.245. We have 
already propose to add the backchannel information for H.323 into H.245
at May. In this proposal, we could not indicate the RFC number in H.245. 
Therefore I think we will also have to add the text to indicate the RFC number
of the RTCP in H.225.0. So we have to rush the last call WITH RTCP.

If we decide to remove the RTCP section, we will have to wait the next 
decision of H.225.0 in 2002. This schedule does not match to our company's
products.

Would you please point out, if there is misunderstanding?

> schedule of H.245, it would probably less time then two years.

I think the RTCP definition in our draft will not prevent to discuss the other
feedback tools in the future at all. Would you please tell us what is the 
problem you will get, if our draft includes RTCP definition?

Best Regards,
Shigeru

---

From: Hideaki Kimata <[email-masked]>
Date: Mon, 19 Jun 2000 12:37:59 +0900

Dear Dr.Injong, and all,

I believe section 5 should remain in this ID.
Because NEWPRED backward channel messages are definitly different from
other feedback messages from following reasons.

(1) These messages can not be used for generic use, because they include
MPEG-4 specific syntaxs and semantics. And their names are "NP_NACK" and
"NP_ACK", for distinguishing from other generic messages.

(2) These messages are only transmitted between Advanced Real-Time
Simple (ARTS) profile terminals. This profile and level indication can
be transmitted in this ID "MPEG-4 Audio/Visual RTP payload format", at
least in IETF way.

>From these reasons, NEWPRED backward channels messages are meaningful
only ARTS profile in MPEG-4. That is why the format for these messages
should exist in text for MPEG-4 payload format.

Best regards,
Hideaki Kimata

On Fri, 16 Jun 2000 18:48:49 -0400

>extend on that? I am willing to work together on this issue.

-----------------------------------------
HIDEAKI KIMATA
 NTT Cyber Space Laboratories
Email: [email-masked]
Phone: +81 468 59 3124
Fax  : +81 468 59 2829
-----------------------------------------

---

From: "Shigeru Fukunaga" <[email-masked]>
Date: Tue, 20 Jun 2000 17:17:51 +0900

Dear Stephan, Dr. Rhee, and all,

I revised the RTCP text again from the latest version which I sent yesterday.
At section 5.4, I revised text back to the original expression.
Would you please check the RTCP part of our ID?

As Kikuchi-san reported this morning, we decided the deadline at the middle 
of this week. If there is any problem in the RTCP part, would you please
indicate it as soon as possible?

Best Regards,
Shigeru

----
5. RTCP Packetization of MPEG-4 upstream messages

This section specifies the usage of particular RTCP packets to carry 
the upstream messages generated by MPEG-4 decoders to advise 
corresponding MPEG-4 encoder, using the MPEG-4 Audio/Visual upstream 
messaging functionalities. This particular RTCP is called "RTCP-MP4U" 
in the following. In the current specification, NEWPRED in the MPEG-4 
Visual Advance Real Time Simple (ARTS) Profile[4] is the only tool 
which uses this RTCP-MP4U payload specification. The RTCP-MP4U packet 
SHALL ONLY be used when it is indicated by some out of band means 
that the corresponding MPEG-4 Visual codec is compliant with the ARTS 
profile and it is indicated in the configuration information of the 
MPEG-4 visual bitstream that the NEWPRED tool is enabled 
(newpred_enable is set to 1). 

5.1. Abstract of NEWPRED in the ARTS profile

The NEWPRED tool in the MPEG-4 Visual ARTS profile is an error 
resilience oriented tool using the upstream messages from the decoder 
to the encoder.  As the inter-frame coding is used in the MPEG-4 
Visual, the image degradation resulting from a packet loss will be 
propagated toall predicted frames following the damaged frame.  In 
order to prevent this type of temporal error propagation, a complying 
encoder may switch reference frames of the inter-frame coding, 
according to the information available in the received NEWPRED 
RTCP-MP4U.  As the correctly decoded frame is used as the reference 
frame, the error propagation is refreshed.

As the intra refresh is not used, the coding efficiency can be kept 
comparatively high, although using older reference frames then the 
most recent one does add bits due to less efficient prediction (e.g. 
through longer motion vectors due to a longer time interval for 
prediction). And the NEWPRED can achieve the faster error recovery 
than the intra refresh as well.

There are two types of upstream messages; acknowledged message 
(NP_ACK) and non-acknowledged message (NP_NACK).  The syntax and 
semantics of both type of upstream messages are defined in MPEG-4[4].  
NP_ACK and/or NP_NACK messages are transmitted on the RTCP-MP4U 
packets in the NEWPRED.  The selecting methods of reference frames 
depend on the kind of used messages.

5.2. RTCP-MP4U packets for low delay

The RTCP-MP4U SHALL be treated as a completely different kind of RTCP 
traffic, not following timing rules associated with normal RTCP 
information. The performance of MPEG-4 tools using upstream messages 
(such as NEWPRED) is sensitive to the delay of the upstream message 
and does not require the full reliability of the upstream message. 
Therefore, it is more effective to send the MPEG-4 upstream message 
packets as soon as possible, i.e. as soon as a packet loss is 
detected, without adding any random delays. 

5.3.  Congestion control

In the cases of the demand type of intra refresh, the number of bits 
during congestion is greater than that in the error free condition. 
This may cause more congestion.  While in the NEWPRED, as the 
intra-frame coding is not used, the bit increase is much lower than 
that of the intra refresh even in the case of packet loss.  NEWPRED 
is less of a burden for the congestion.

The amount of traffic necessary to convey NEWPRED RTCP-MP4U depends 
on the strategy of the reference frame selection by the encoder, the 
type of upstream messages sent by the decoder, frame rate, and 
network conditions.  In order to avoid the congestion, the amount of 
upstream message packets should be small. In the NEWPRED, the 
decoder can control the amount of the upstream message packets by 
not sending some upstream messages; For example, in the case that 
the NP_NACK messages are mainly used to select the reference frames 
in the encoder, the decoder may not send the NP_ACK messages even if 
it receives downstream data.  On the other hand, in the case that 
the NP_ACK messages are mainly used in the encoder, the decoder may 
not send the NP_NACK messages. The amount of the upstream messages 
is at most 5% (normally about 1%) of the visual downstream data. 

Especially the amount of NP_ACK messages is decreased in the case of 
packet loss.  Therefore the NP_ACK message has no additional burden 
on the congestion.  On the other hand, NP_NACK messages 
corresponding to the lost packets are usually sent after the 
congestion, because the decoder detects the packet loss after it 
receives the next downstream packet. Again the NP_NACK message has 
little additional burden on the congestion.

To reduce the number of RTCP-MP4U packets, multiple upstream 
messages can be concatenated in the payload of one RTCP-MP4U 
packet.  In this case, it is desirable to send these concatenated 
messages as soon as possible.

The RTCP-MP4U transmission interval depends on the interval of 
decoding the visual downstream data.  Both the receiving interval 
of the visual RTP packet and the decoding time for each packet 
data have some jitter associated with them.  Therefore the 
RTCP-MP4U transmission interval has some jitter by itself.  It is 
effective for the congestion control, and there is no need to add 
any random delays.  This means that the amount of jitter is enough 
to avoid another congestion in the case of unicast.

5.4. Limited to Unicast

Although NEWPRED can work in multicast when the number of decoders 
is small, in order to avoid additional congestion, the NEWPRED over 
RTP/RTCP SHALL NOT be used in multicast.

5.5. Relations with SR and RR

The RTCP-MP4U packets SHALL be treated as a completely different 
kind of packets from the normal RTCP packets such as SR, RR and so 
on.

For example, if the RTCP-MP4U packets is included in the 
calculation of RTCP sending interval, the RR packets should be 
generated in the timing of the RTCP-MP4U packets.  In this case, 
the interval of the RR packets would be smaller than 5 seconds, 
and the number of the normal RTCP packets is much increased. This 
is bad from the view point of the congestion. 

Therefore all RTCP-MP4U packets SHALL be ignored when analyzing 
the information in the sender and receiver reports (SR and RR), 
and only normal RTCP packets SHALL be used.  

Multiple RTCP-MP4U packets can be concatenated without any 
intervening separators to form a compound RTCP packet.  The 
normal compound RTCP packet SHOULD start with SR or RR packets. 
However in the case of compound RTCP-MP4U packet, other normal 
RTCP packets SHALL NOT be included, and only RTCP-MP4U packets 
SHALL be included in one compound RTCP-MP4U packet.

5.6.  MPEG-4 Visual upstream message packets definition

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |V=2|P|   UMT   | PT=RTCP_MP4U  |           length              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                              SSRC                             |
    +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
    |       MPEG-4 Upstream Messages Payload (byte aligned)         |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :             padding           |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

version (V): 2 bits
Identifies the version of RTP, which is the same in RTCP 
packets as in RTP data packets. 

padding (P): 1 bit
If the padding bit is set, this RTCP-MP4U packet contains 
some additional padding octets at the end which are not 
part of the control information. The last octet of the 
padding is a count of how many padding octets should be 
ignored. In the case several upstream messages are mapped 
onto one RTCP-MP4U packet, padding should only be required 
on the last individual message.

upstream message type (UMT): 5 bits
Identifies the type of the MPEG-4 upstream messages.
0:  forbidden
1:  MPEG-4 Visual NEWPRED in the ARTS Profile
2-63: reserved
In this internet-draft, only the NEWPRED in the ARTS profile 
is assigned as the candidate of the UMT for the moment.  Some 
other MPEG-4 Audio/Visual applications using the upstream 
messages may be assigned in the future. 

packet type (PT): 8 bits
The value of the packet type (PT) identifier is the constant 
RTCP_MP4U (TBD). 

SSRC: 32 bits
SSRC is the synchronization source identifier for the sender 
of this packet. 

MPEG-4 Upstream Message Payload: variable
The syntax and semantics of the MPEG-4 upstream messages are 
defined in the ISO/IEC 14496-2/3[4][5]. All messages are byte 
aligned. Normally one message is mapped onto one RTCP packet, 
and several messages with same UMT could be continuously 
mapped onto one RTCP packet.  One message SHALL NOT be 
fragmented into different RTCP packets.

6. MIME type registration for MPEG-4 Audio/Visual streams

...

6.1 MIME type registration for MPEG-4 Visual

...

mpeg4-newpred-upstream-message: A boolean number to indicate 
the receiver capability of sending the upstream message of 
NEWPRED in MPEG-4 video. The upstream messages are delivered 
on the particular RTCP packets which are described in Section 
5. This capability exist when and only when the 
"profile-level-id" is 145, 146, 147 or 148 (Advance Real Time 
Simple Profile/Level 1, 2, 3 or 4). And this parameter is set 
to '1' only for the real-time communication with the real-time 
encoder. Therefore this parameter is usually set to '0' for 
the e-mail applications or streaming applications without 
real-time encoder.

Example usages for these parameters are:
- MPEG-4 Visual Core Profile/Level 2:
Content-type: video/mp4v; profile-level-id=34

- MPEG-4 Visual Advanced Real Time Simple Profile/Level 1, 
upstream message is not used:
Content-type: video/mp4v; profile-level-id=145; 
mpeg4-newpred-upstream-message=0

...

---

From: Stephan Wenger <[email-masked]>
Date: Sat, 17 Jun 2000 15:52:52 +0200

Oops.  I stay corrected.

Stephan

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 21 Jun 2000 02:24:30 +0200

Dear all,

I'm sorry for replying a bit late.  I was on travel to VCIP in
Australia, where I arrived just about now.

I certainly see improvements in the language below concerning
the RTCP feedback.  There may be still a few places where the
language could be improved further, but this is a minor issue
compared to my general architectural concerns, which are not
addressed in the documents below, and which therefore lead me
to keep my concerns.

Just briefly, I believe that it is inconsistent with current payload
specs to include RTCP-based feedback only for NEWPRED and
not for other known feedback schemes.  I understand the rationale
the proponents follow, but I don't feel it compelling.

And, I continue to believe that we can add a well designed,
architectural consistent feedback scheme for MPEG-4 into H.323
after finalization of the ES payload spec, without a delay of
several years.

Stephan

---

From: [email-masked]
Date: Tue, 20 Jun 2000 20:49:12 -0400 (EDT)

Hi All,

I've been away for a few days to attend a conf in west coast; srry
about the delay responding to you. I think I glanced over the
document. I am afraid I have to agree with Stephen. I will give
more detailed comments once I get to NC tmr. For those who
expressed interests in receiving my paper, I will try to
send you the paper as soon as I come back to my office.

Thanks
Injong

---

From: Colin Perkins <[email-masked]>
Date: Thu, 22 Jun 2000 21:39:52 -0400

>the proponents follow, but I don't feel it compelling.

I have to agree with Stephan on this. We clearly do not have consensus on
the current approach, which makes me unwilling to push the current spec
forward as it stands.

>after finalization of the ES payload spec, without a delay of

Or, at least, we can derive an ES payload format which allows for a range
of backchannel techniques. Consider the means by which we have added RTP-
based FEC/reliability to a number of payload formats, where the format is
independent of the FEC scheme (e.g. RFC 2198, 2733). I'm not convinced
there is a need to closely couple the backchannel to the payload format.

Colin

---

From: Hideaki Kimata <[email-masked]>
Date: Mon, 26 Jun 2000 19:34:56 +0900

Dear Colin and Stephan, and all

As I pointed out in previous Email, NEWPRED is one tool of ARTS profile
in MPEG-4. And NEWPRED RTCP messages are only transmitted between ARTS
profile terminals.

This means the usage of these messages is not definitly generic, and
setup procedure should be also different from other backward channel
messages. I mean the setup should be after receiving profile and level
indication of MPEG-4.

I believe this fact makes objection for NEWPRED RTCP meaningless.

Best regards,
Hideaki Kimata

On Thu, 22 Jun 2000 21:39:52 -0400

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Mon, 26 Jun 2000 19:58:31 +0900

Dear Colin, all,

I agree with Colin.

I understand that there are concerns to include NEWPRED RTCP supporting only
ARTS profile. On the other hand, I don't think there is a good reason that
other backchannel schemes must be rushed into the I/D. I fully support
Colin's comment of not to couple backchannel to the payload format.

I would like to propose moving NEWPRED RTCP to another I/D and not to
include any backchannel scheme in the MPEG-4 Audio/Visual RTP payload format
document.

Best regards,
Yoshihiro

----- Original Message -----
From: Colin Perkins <[email-masked]>
To: Stephan Wenger <[email-masked]>
Cc: Shigeru Fukunaga <[email-masked]>; Dr. Injong Rhee
<[email-masked]>; Yoshihiro Kikuchi <[email-masked]>;
4on2andIP-sys <[email-masked]>; IETF AVT
<[email-masked]>
Sent: Friday, June 23, 2000 10:39 AM
Subject: Re: Updated I/D for MPEG-4 Audio/Visual RTP payload format

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 26 Jun 2000 13:23:23 +0200

Hi all,

 > in MPEG-4. And NEWPRED RTCP messages are only transmitted between
 > ARTS profile terminals.

 > I believe this fact makes objection for NEWPRED RTCP meaningless.

No.  I believe not.  Sorry.

The definition of exactly one (of many possible and known) feedback
mechanisms is the very point of my objection.  This is regardless of
whether that mechanism is specific to MPEG-4 (which it is not, as
H.263 has NEWPRED, too), or to a specific MPEG-4 profile.

I started working on a general video feedback draft, which covers both
conventional feedback messages such as FIR, and special, low delay,
point-to-point feedback for things such as NEWPRED.  A rough draft
should show up on rem-conf within a week.

Stephan

---

From: Christine Guillemot <[email-masked]>
Date: Tue, 27 Jun 2000 11:26:11 +0200

Dear all,

Sorry for the very late reaction to the exchange
of mails, due to traveling over the last two weeks.

I would like first to say that I support the idea of
removing the section on NEWPRED RTCP as
announced by Kikuchi, sharing completely the
concerns of Stephan and Injong on the related
issues.

I would like to add the following questions/comments:

1-  I understood from earlier e-mail/MPEG4-IETF phone
meeting discussions that to be MPEG-4 compliant,
terminals would have to support the sync layer, because
the ESI is not normative.

Therefore, what would be the status of
the current draft with respect to MPEG-4 compliance
issues, since it says:

"These RTP payload formats enable to carry MPEG-4
Audio/Visual streams without using the synchronization
and stream management functionality of MPEG-4 Systems
[6]. Such RTP payload format would be used within systems
where their own stream management functionality is provided
and thus such functionality in MPEG-4 Systems is not necessary.
H.323 terminals are an example of such systems. MPEG-4
Audio/Visual streams are not managed by
MPEG-4 Systems Object Descriptors  but by H.245. The streams
are directly mapped onto RTP packets without using the
synchronization functionality of MPEG-4 Systems."

2- It seems to me that some of the fragmentation rules do
not apply to scalable streams (or at least to enhancement
layers of scalable streams), e.g. below:
----
   (5) A single video packet SHOULD NOT be split into a plurality of RTP
   packets. The size of a video packet SHOULD be adjusted in such a way that
   the resulting RTP packet is not larger than the path-MTU. A video packet
   MAY be split into a plurality of RTP packets when the size of the video
   packet is large.
----
since - unless recent changes - enhancement layers do not
support the notion of video packets.  If this is
the case, how the scalable profiles can be really used
in a loss-prone environment?.

3- Overall, the draft seems to target extremely specific
applications - relying on real-time encoders -. I would like
to have some indications on how it can be used for
streaming applications.

Indeed, considering the video part, protection of headers
rely on the usage of the HEC field.
A network adaptive usage of this field would require the
implementation of a video stream parser (i.e. a part of a
decoder), in order to incorporate inside the video packets
the important data that needs to be protected. Is it really
what we want?

Best regards,
Christine

---

From: Colin Perkins <[email-masked]>
Date: Tue, 27 Jun 2000 18:24:24 -0400

>H.263 has NEWPRED, too), or to a specific MPEG-4 profile.

Right. I would like the payload format for elementary streams to be usable
with a range of different backchannel and error protection schemes. It is
hard to do this if the format includes a specific reference to NEWPRED. I
have no problem with NEWPRED being one option for this, but we have to
allow the possibility of other schemes, and I think this requires that the
NEWPRED support is broken out into a separate document.

Colin

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Wed, 28 Jun 2000 23:28:04 +0900

Dear Chiristine, all,

Let me try to answer the second and third questions.

> in a loss-prone environment?.
I'm not sure if this question is relevant to an RTP payload format spec. or
something more general. It's true that the video packet is not supported in
the enhancement layer. (It is supported in the base layer.) But this is the
specification of the ISO/IEC 14496-2 itself but not specific to RTP payload
formats.

There was a discussion inside MPEG when defining enhancement layer syntax
which does not support video packet. The understanding of the group was that
the scalable video coding scheme itself has loss resiliency. It is possible
to decode using only the base layer stream even when the enhancement layer
is corrupt. Therefore, the base layer should be protected well but not so
much for the enhancement layer. This is why MPEG-4 error resiliency tools
(video packet and others) are supported in the base layer.

> to have some indications on how it can be used for
I think network oriented delivery shemes used in the existing streaming
frameworks can also be applied to this RTP payload format.

One way is to prepare a pare of streams with a variety of encoding
parameters, and select one of them when transporting a stream depend on the
network environment. This is the way used widely in streaming frameworks,
and I believe this works properly also for MPEG-4 audio/video.

> the important data that needs to be protected. Is it really
I'm sorry but I cannot understand what you are trying to address. Are you
trying to do something like an adjustment of an RTP packet size depend on
the network environment?

Best regards,
Yoshihiro
----
        Yoshihiro Kikuchi

E-mail: [email-masked]
Corporate Research and Development Center
TOSHIBA Corporation
TEL: +81 +44 549 2288    FAX: +81 +44 520 1267

---

From: Christine Guillemot <[email-masked]>
Date: Wed, 28 Jun 2000 19:07:09 +0200

Dear Yoshihiro,

> specification of the ISO/IEC 14496-2 itself but not specific to RTP payload

Since the comment refers to a fragmentation rule of the videostream, a rule
which is part of the payload format spec., i
would think the question to be relevant to the payload format spec.

> parameters,

How many streams for the same content to be storedon the server?

> and select one of them when transporting a stream depend on the

Definitely not! simply pointing out that relying entirely on theHEC field for
protection of header information may not be that suited
in the case of streaming applications.

Regards,
Christine

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Thu, 29 Jun 2000 19:05:01 +0900

Dear Christine, all,

> > >    packets. The size of a video packet SHOULD be adjusted in such a
way
> > >    MAY be split into a plurality of RTP packets when the size of the
video

Christine is correct in the meaning that the video packet size cannot be
adjusted for the enhancement layer stream where the video packet is not
supported.

To be more precise, I would like to propose the following change on the
fragmentation rule:

(5) A single video packet SHOULD NOT be split into a plurality of RTP
packets, except for the enhancement layer of the scalable streams where the
video packet is not supported. The size of a video packet SHOULD be adjusted
in such a way that the resulting RTP packet is not larger than the path-MTU
in this case. A video packet MAY be split into a plurality of RTP packets
when the size of the video packet is large.

Best regards,
Yoshihiro

----
        Yoshihiro Kikuchi