Subject: [AVTCORE] Clarifying the H.264 RTP payload format specification's text on when to set the RTP "M" bit

From: Ross Finlayson <[email-masked]>
Date: Wed, 31 Jul 2013 20:18:56 +0200

As I noted in Wednesday morning's AVTCORE session, the proposed H.265 RTP payload format specification is not as clear as it could be about how a RTP sender knows when to set the "M" bit on an outgoing RTP packet.

Currently, the text says:

   Marker bit (M): 1 bit

      Set for the last packet of the access unit indicated by the RTP
      timestamp, in line with the normal use of the M bit in video
      formats, to allow an efficient playout buffer handling.  Decoders
      can use this bit as an early indication of the last packet of an
      access unit.

There is nothing inherently wrong with this.  However (because we're not in the MPEG world here :-) it's not sufficient to fully specify only how the receivers of RTP packets should behave; we should also try to specify, as clearly as possible, how the *senders* of RTP packets should behave.  We should try to give at least some guidance about when the NAL unit that's being sent ends an "access unit" because - for many implementations - it is not immediately obvious.

If I were amending the H.264 payload format specification - which of course I'm not - then I would, hypothetically, add a paragraph something like the following:

----------
Unfortunately the contents of a NAL unit, alone, does not tell a RTP sender implementation whether or not the NAL unit ends an access unit.  Instead, the implementation can obtain this information separately, from the encoder.  If, however, this information is not directly available from the encoder (e.g., because the implementation is sending data that consists solely of a sequence of pre-encoded NAL units), then it must instead inspect the next NAL unit, to determine whether or not the current NAL unit ends an access unit.  The following rule can be used:
    The current NAL unit ends an access unit if it is a VCL NAL unit (i.e., if its "nal_unit_type" is in the range [1..5]), and if either:
	- the next NAL unit is not a VCL NAL unit, or
	- either the current NAL unit's "nal_unit_type" or the next NAL unit's "nal_unit_type" - but not both - is 5 (i.e., "Coded slice of an IDR picture"), or
	- the next NAL unit's "nal_ref_idc" field differs from the current NAL unit's "nal_ref_idc" field, with one of them being equal to zero, or
	- both the current and next NAL units begin with slice headers, and their "frame_num" fields differ, or
	- both the current and next NAL units begin with slice headers, and their "pic_parameter_set_id" fields differ, or
	- both the current and next NAL units begin with slice headers, and their "field_pic_flag" fields differ, or
	- both the current and next NAL units begin with slice headers, and their "bottom_field_flag" fields differ, or
	- both the current and next NAL units' "nal_ref_idc" field is 5, and the "idr_pic_id" fields differ.
----------

(I derived this rule from Section 7.4.1.2.4 - "Detection of the first VCL NAL unit of a primary coded picture" - of the H.264 specification.  Perhaps it could be simplified?)

I suggest that an analogous paragraph be added to the H.265 RTP payload format specification.  (Or alternatively, instead of writing a detailed rule like this, perhaps a reference to an appropriate part of the H.265 specification might be made instead (can we do this in a IETF RFC?).)

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 31 Jul 2013 23:58:22 +0000

Hi Ross,

If your application interfaces with an encoder or systems layer (container file format, transport stream, CDN stream, etc.) that does not indicate frame boundaries nor any timing information, then how are you setting the RTP timestamp? That seems like a much more fundamental issue than the marker bit.

If you are willing to detect the last NALU of a frame by waiting for the first VCL NALU of the next frame, that is very simple in both H.264 and H.265. The first bit after the VCL NALU header is set if it's the first VCL NALU of a frame. This works in H.265 because the slice header explicitly has such a flag. It works in H.264 because the slice header starts with the macroblock address which will be 0 at the start of the frame (assuming no ASO/FMO), and 0 encodes to 1 under ue(v) encoding rules. Keep in mind the NALU header is 2 bytes in H.265 vs. 1 byte in H.264, so the slice header offsets are different.

If you can't wait for the first NALU of the next frame, it is much harder to detect the last NALU of a frame without full VCL bitstream parsing, unless you use some heuristics which may be simpler but less reliable. This is part of the reason why the marker bit in video payloads indicates end of frame, because it may be very difficult for RTP receivers to detect without deep parsing otherwise. Note that RTP receivers are technically forbidden to rely on the marker bit. But many implementations detect whether it is reliable, then use it as an optimization to minimize latency if reliable.

I don't think the draft needs any more text beyond what it currently says.

Regards,
Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross Finlayson
Sent: Wednesday, July 31, 2013 2:19 PM
To: [email-masked]
Subject: [AVTCORE] Clarifying the H.264 RTP payload format specification's text on when to set the RTP "M" bit

   Marker bit (M): 1 bit

----------

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/