Subject: Re: [AVT] New I-D on JVT/H.26L packetization

From: Stephan Wenger <[email-masked]>
Date: Wed, 06 Mar 2002 03:00:14 +0100

Folks,

Philippe's EMail is rather long, and mine is even longer.  I
apologize in advance.

At 02:43 PM 3/5/2002 +0100, [email-masked] wrote:
>  I dont see the lack of final status for NAL as a problem, on the contrary
>I beleive that making sure that the (RTP) transport is independent of
>the NAL internals is a key design issue.

Well, the remark regarding the final status of the NAL is intended
to make people in AVT aware that there actually is a chance (for
them) to influence the NAL design, if they wish.  In other words,
what we try is to start the standardization process of an RTP
specification early so that we can still fix serious bugs in the
media coding itself, if such bugs exist.  In that regard, I consider
network-unfriendlyness a bug. That is in contrast to most previous
video packetization projects, where the poor payload designers had
to response to the various shortcomings of media codecs not
designed for a packet environment.  That is what my remark was
all about.

I disagree with the Philippe's general approach of hiding the
internals of the media from the transport.  This is the old DMIF
approach, which is, IMHO, a failure.  An RTP payload specification
can well be aware of media specifics -- that is its very nature.  The
abstraction happens at the RTP level itself.

>in this respect, you state that:
>
>"The RTP payload specification is designed to be
>    unaware of the bit string in the NALP payload."
>
>but you dont state explicitely what is the situation for NALP headers,
>rather you write:
>
>  "JVT  Video-aware network elements such as Gateways can perform many
>    operations by handling only those byte buffers"
>
>but is this in the scope of a RTP payload format ?

The designs of gateways that look into the payload is not within the
scope of the payload specification, although entities defined in RTP
such as mixers can certainly take advantage of the knowledge of the
semantics of the NALP type and the first couple of bytes of the
payload (containing the Slice/Partition headers).  One example: an
(intelligent) RTP mixer that receives a DPB/DPC NALP without the
associated DPA NALP could simply throw out those DPB/DPC NALPs
and thus reduce the network load (congestion control, ..., the whole nine
yards).  This is a task that is very simple, but requires some media
awareness. Less intelligent RTP mixers can simply transmit whatever they
get, without looking at the payload.

I have to admit that drawing the precise borderline between the
payload spec (under IETF control) and the NAL spec (under JVT
control) is difficult, and I'm open for suggestions.  But I would not
like to see procedural arguments in this discussion.  Certainly I
don't like to see a "mandate" discussion" to be used to promote or
discourage a very basic design philosophy of H.26L, namely that
H.26L DOES know about transports and IS transport aware.

>Fine, but why would you specify that in the RTP payload ?
>same for the NALP type byte, why this table of value?

You need to know the NALP type in entities such as mixers in
order to unevenly protect those NALPs which need protection,
and transport the rest as best effort.  I can show you simulation
data in which such an uneven protection scheme has a tremendous
impact on the reproduced picture quality.

Clearly, the draft states that the NALP types are reproduced for
convenience only, and that people should look at the normative
reference for up-to-date information.  More information about the
payload never hurts in an RTP specification.

>  I am -as you are- concerned about using the RFC2250 timestamp policy.

The time stamp issue is a very difficult topic.  Plain fact is:

a) in H.26L you don't necessarily know the time at which the content
of a certain packet is displayed (my understanding of the word
presentation timestamp).  It is possible to send data that gets
decoded, but gets NEVER displayed.  And I can show you scenarios
where such a (at the first glance weird) scheme is actually useful.

b) It is possible to use "interleaving schemes" (has little to do with
the interleaving of other RTP payload specs) that cover tens of seconds
or minutes, maybe even hours, between media packets that belong to
the same picture.  Again, useful applications and not academic examples.
Meaning, there is no way that we could assume that those interleaved
packets fit together into the jitter buffer.

>moreover each fragment (slice) objectively HAS a presentation time stamp
>which is the CTS (composition time stamp) of the corresponding Access Unit 
>(picture)...

I don't understand this sentence.  Why is the presentation time stamp
the same as the composition time stamp?  I don't think I took the
sentence out of context, but if I did, I apologize in advance.

>the fact that you want to send fragments a long time before they are to be 
>used
>is a separate (buffer management, etc) issue ...

No, it is not a buffer management issue, neither if you understand the
word "buffer'" as the RTP jitter buffer, nor if you see it as the bit buffer of
the video codec.  All arriving packets that are addressed to a reference
buffer other than the one used for P-prediction are decoded as they arrive.
They are just not displayed, but rather go to a reference buffer for
potential future use.  In H26L you don't collect slices, until you have a
complete picture.  You process them as they come in.

Folks, please note that the Enhanced Reference Picture Selection
feature that is necessary to support those (at the first glance weird)
functionalities this is a fundamentally new thing, available only in H.263++
and in H.26L, but no where else.  For many of you who are familiar with
traditional predictive coding schemes it will probably require a learning
process. I will spend time during the presentation of the draft in Minneapolis
to give you a better understanding because the current documents of
the JVT group are not at all tutorial in nature.

Best regards
Stephan

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Colin Perkins <[email-masked]>
Date: Wed, 06 Mar 2002 21:37:50 -0500

A couple of minor comments inline...

>abstraction happens at the RTP level itself.

Correct. Indeed, it is common for RTP payload formats to be aware of the
media specifics, and this is what makes the good RTP payload formats so
robust to the wierdness of the network.

>where such a (at the first glance weird) scheme is actually useful.

That shouldn't be an issue, since RTP makes no comment on when the media
data is displayed. We do need to figure out how the RTP timestamp relates
to the (various?) H.26L timelines.

>complete picture.  You process them as they come in.

Again, remember that RTP makes no reference to buffer management either in
the codec or the jitter buffer. Is this necessarily something the payload
format needs to discuss?

>the JVT group are not at all tutorial in nature.

Cheers,
Colin

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 07 Mar 2002 12:58:00 +0100

Colin,

I think there is need for some clarification on your comment.

>to the (various?) H.26L timelines.

You are saying that RTP makes no comment on when the media is displayed, my interpretation of that sentence is: As RTP timestamp represents the sampling instant it does not explicit express a display time, however the display time is often implicit given by the RTP timestamp. Otherwise your comment can easily be miss interpreted by people not knowing RTP very well.

I agree that it is important to decide how to map the H.26L timeline to the RTP timestamp. I believe it is also important to avoid ambiguities, like duplicating timeline information in both RTP header and bit stream. This has happened in H.263 and creates problems when for example a RTSP server pause a video stream for 5 seconds and then resume. In that case the RTP timestamp will increase by 5 seconds but the temporal reference in the bit stream does not reflect this. 

Regards 

Magnus Westerlund 

Multimedia Technologies, Ericsson Research ERA/T/VA
----------------------------------------------------------------------
Ericsson Radio Systems AB  | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]

---

From: Colin Perkins <[email-masked]>
Date: Thu, 07 Mar 2002 08:53:45 -0500

--> Magnus Westerlund writes:
>You are saying that RTP makes no comment on when the media is displayed,
>my interpretation of that sentence is: As RTP timestamp represents the
>sampling instant it does not explicit express a display time, however the
>display time is often implicit given by the RTP timestamp. Otherwise your
>comment can easily be miss interpreted by people not knowing RTP very well.

That's a fair clarification, yes. It also means that mapping presentation
time to RTP time is not necessarily appropriate, for example, although it
is done for some codecs.

>I agree that it is important to decide how to map the H.26L timeline to
>the RTP timestamp. I believe it is also important to avoid ambiguities,
>like duplicating timeline information in both RTP header and bit stream.
>This has happened in H.263 and creates problems when for example a RTSP
>server pause a video stream for 5 seconds and then resume. In that case the
>RTP timestamp will increase by 5 seconds but the temporal reference in the
>bit stream does not reflect this. 
>
>Regards 
>Magnus Westerlund 

Cheers,
Colin

---

From: Stephan Wenger <[email-masked]>
Date: Thu, 07 Mar 2002 17:20:05 +0100

Dear Magnus,

At 12:58 PM 3/7/2002 +0100, you wrote:

>server pause a video stream for 5 seconds and then resume. In that case 
>the RTP timestamp will increase by 5 seconds but the temporal reference in 
>the bit stream does not reflect this.

I absolutely agree with your requirement.

The problem you have is that you don't use H.263 in the environment it was 
designed for.  A real-time H.263 encoder could "pause" a video stream for a 
couple of seconds and continue to increase the TR, hence keeping 
synchronization with the RTP timestamp.  But when you have pre-encoded 
video, you can't, without patching the bit stream and (more critically) 
messing up reference picture selection information (H.263 Annex N/U).  That 
is why H.26L (precisely, its IP/RTP NAL) does not have a TR in the sense of 
H.263, but rather a thing called Picture ID that is used primarily for the 
association of slices and reference buffers to pictures.

Stephan