RTCP Messages for Point Cloud Prioritization
Internet-Draft draft-engelbart-avtcore-rtcp-point-cloud-roi-00
Mathis Engelbart
Technical University Munich
mathis.engelbart@tum.de
Jörg Ott
Technical University Munich
ott@in.tum.de
Lukasz Kondrad
Nokia Technologies
lukasz.kondrad@nokia.com
25 September 2024
Web and Internet Transport
Audio/Video Transport Core Maintenance
RTP
RTCP
ROI
Point Cloud
This document specifies RTCP messages and RTP header extensions for exchanging parameters of real-time streamed point clouds. A sender can notify receivers of the currently applied parameters, such as selected regions, and their parameters, such as the respective resolutions and included point attributes. A receiver can request updates to the same parameters using RTCP feedback messages.
About This Document
The latest revision of this draft can be found at https://mengelbart.github.io/draft-engelbart-avtcore-rtcp-point-cloud-roi/draft-engelbart-avtcore-rtcp-point-cloud-roi.html. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-engelbart-avtcore-rtcp-point-cloud-roi/.
Discussion of this document takes place on the Audio/Video Transport Core Maintenance Working Group mailing list (mailto:avt@ietf.org), which is archived at https://mailarchive.ietf.org/arch/browse/avt/. Subscribe at https://www.ietf.org/mailman/listinfo/avt/.
Source for this draft and an issue tracker can be found at https://github.com/mengelbart/draft-engelbart-avtcore-rtcp-point-cloud-roi.
Introduction
A point cloud is a set of data points in a three-dimensional coordinate system where each point is defined by its three coordinates. Point clouds can represent three-dimensional environments, such as a vehicle's surroundings. Each point in a point cloud may optionally be associated with additional attributes. Attributes can, for example, be colors or reflectance of objects in the scene. Sequences of point clouds can be generated by sensors such as Lidar ("light detection and ranging"), Radar ("radio detection and ranging"), or a multi-camera setup. Due to the high number of points in a scene, the bandwidth requirements of transmitting point clouds over a network are often higher than those of streaming video. In video streaming, efficient codecs are used to reduce the bandwidth requirement. Similar codecs are being developed for point clouds. However, when streaming point cloud data, consumers of the data usually aren't equally interested in each point. For further processing, focusing on some regions of a point cloud stream is often sufficient, allowing the producer of a point cloud sequence to filter out points of lower interest to further reduce the bandwidth requirements and prioritize bandwidth usage for points of higher importance. When selecting the regions to prioritize and deciding which points can be excluded from transmission, producers and consumers need a mechanism to signal a) which regions of a scene are currently being prioritized and b) request updates to prioritize different regions in the future. This document describes such a mechanism for real-time transmission of point clouds building on the RTP Control protocol RTCP, which is part of the Real-time Transport Protocol RTP. Additionally, this document provides RTP header extensions for senders to inform receivers about the currently applied parameters. The information might be useful for receivers in determining if an RTCP message requesting an update was received and acted upon by the sender.
In Section 4, this document first defines a set of shared encoding schemes to represent point cloud parameters. Section 5 and Section 6 define RTCP messages and RTP header extensions, respectively. The RTCP messages and RTP header extensions reference the encoding scheme in Section 4. Section 7 defines the SDP parameters required to negotiate the usage of the presented RTCP messages and header extensions.
Conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
Definitions and Abbreviations
- TODO: Add definitions or remove section
Region and Parameter Encodings
This section introduces encodings for point cloud regions and their parameters, such as the included attributes and the level of detail. It also provides an encoding for the definition of a viewport, including a dynamically adaptable precision.
The encodings described in the following subsections are reused in RTCP feedback messages and RTP header extension as described in Section 5 and Section 6.
Octree Encoding of Point Cloud Regions
This section defines an encoding that is reused in RTCP message types to index regions in a point cloud. The encoding uses a recursive octree encoding to signal the presence of certain regions of a point cloud. The encoding can be used to set parameters for the present regions, e.g., receivers can signal individual priority of every region or sets of attributes to include in every region.
The root node in every tree is a single byte where every bit indicates whether a child node of an octant is present. Child nodes are appended in pre-order traversal order. A zero-byte encodes a leaf node. The order of octants by the signs of the points of the X-, Y-, and Z-coordinates in each octant is given in Table 1.
Regions in the octree are considered present when a leaf node encoding that region is present. For example, a single zero-byte encodes a single leaf node, and the root region covering the complete space is present. An encoded value of the two bytes 0x4000 would indicate that only the octant with X<0, Y>=0, and Z>=0 is present.
Octant to bit mapping order with bit 0 being the most significant bit and bit 7 being the least significant bit.
Bit | X | Y | Z
0 | + | + | +
1 | - | + | +
2 | - | - | +
3 | + | - | +
4 | + | + | -
5 | - | + | -
6 | - | - | -
7 | + | - | -
The octree encoding can be used in absolute and relative forms. In the absolute form, the bounding box of the octree is implicitly defined by the space covered by the point cloud-generating source. In the relative form, the bounding box can be explicitly defined by setting the min and max values of the X-, Y-, and Z-coordinates. The coordinates are prefixed before the octree encoding as four-byte integers each, as shown in Figure 1.
Relative octree encoding using an explicitly defined bounding box

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min X                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min Y                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min Z                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max X                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max Y                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max Z                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | octree encoding                                             ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Region-Dependent Attributes Encoding
Attributes are additional values optionally associated with every point in a point cloud. The available attributes depend on the context of an application and thus need to be configured out-of-band. The attribute encoding described in this section requires mapping attributes to a bitmask value. Section 7 describes one option to negotiate the mapping when using SDP.
To signal the presence of attributes per region, senders and receivers can use the octree encoding presented in Section 4.1. For every region presented in the octree encoding, N bytes will be used to indicate the presence of attributes by setting the bits of the bitmask of the respective attributes to true. The size of N depends on the number of attributes and is implicitly given by the smallest number of bytes that can represent the largest bitmask negotiated during signaling.
Bitmask values can be shared among attributes to allow signaling of attribute sets that always occur in combination.
Format of RTCP Feedback Messages
This document describes RTCP message types that can be used to signal interest in the prioritization of regions and their parameters. Receivers can signal an interest in receiving a region with a higher priority. Different regions can be requested in different resolutions or with different sets of attributes included. Additionally, receivers can request a viewport precision update.
A sender can acknowledge a parameter update using the RTP header extensions described in Section 6. Receivers should not retransmit the same request multiple times to avoid unnecessary overhead, but if the receiver can assume that a request was lost, it may retransmit the request. The request should not be retransmitted earlier than at least one RTT after the first request was transmitted.
The following sections define the available message types in detail. All RTCP feedback messages use the common header format shown in Figure 2. The first eight bytes of the header follow the format of RTCP message headers defined in [RFC3550]. The common header is followed by an additional byte consisting of flags describing the payload.
The payload of the RTCP messages following the header contains one or more of the parameter encodings in the following order:
1. Absolute/Relative Octree Encoding
2. (optional) Priority
3. (optional) Attribute Encoding
4. (optional) Level-of-Detail Encoding
- Note: alternative form: <Header><{abs,rel}-octree>[<priority>][<attributes>][<level-of-detail>]
- TODO: In addition to the encodings already described here, one could come up with additional parameter requests such as viewport precision, and sender position and possibly others.
The semantics of the different payload formats are explained in the following subsections.
Common RTCP feedback message header

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |V=2|P|   FMT   |       PT      |          length               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  SSRC of packet sender                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Res. |R|P|A|L| Payload                                      ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The fields V, P, SSRC and length are used as defined in the RTP specification [RFC3550]. The respective meaning is summarized below:
version (V): 2 bits
This field identifies the RTP version. The current version is 2.
padding (P): 1 bit
If set, the padding bit indicates that the packet contains additional padding octets at the end that are not part of the control information but are included in the length field.
Feedback message type (FMT): 5 bits
The feedback message type is XX (TODO: Use correct value).
Payload type (PT): 8 bits
The RTCP packet type is PSFB (206).
Relative (R): 1 bit
This field distinguishes between absolute and relative region requests (see Section 4.1. If the bit is set, the message contains a relative octree encoding and the minimum and maximum fields described in Figure 1 are present.
Priority: (P): 1 bit
If set, the RTCP message contains a priority request and the octree encoding is followed by one byte indicating the requested priority for each region encoded as leaf node in the octree encoding.
Attributes (A): 1 bit
If this bit is set, the RTCP message contains an attribute request and the octree encoding is followed by an attribute encoding for every region encoded as leaf node in the octree encdoing.
Level-of-Detail (L): 1 bit
If this bit is set, the RTCP message contains a level-of-detail request and the octree encoding (or the attribute encoding, if it is present) is followed by a level-of-detail encoding for every region encoded as leaf node in the octree encoding.
Region Requests
A receiver can use region of Interest signaling to indicate interest in some areas of a point cloud, and a sender can react by prioritizing the regions of interest when allocating bandwidth.
The region interest request uses the standard header defined in Figure 2.
Priority Requests
Receivers can append a priority encoding to the octree to signal fine-grained priorities per region. A priority is a value encoded as a single byte where a higher value indicates a higher priority. A priority request contains a priority byte for every region encoded as a leaf node in the preceding octree encoding.
Attribute Requests
By setting the header Flag A to 1, the receiver can include an attribute encoding as described in Section 4.2 to the feedback message to request attribute sets for every region encoded as a leaf node of the octree. The attribute encoding length depends on the number of negotiated attributes and their associated bitmask values. A single byte can indicate up to eight attributes or attribute sets. For example, if the two attributes, color, and reflectance, are negotiated with bitmask values of 0x01 and 0x02, one byte will be appended to the octree for every leaf node where the bits 0x01 and 0x02 are set to one for every region, which should include these attributes.
TODO: Region-Dependen Resolution or Level-of-Detail Request
- TODO: Using the same mechanism as for attrbiutes, we can signal resolution per region, but we need to define how to express resolution.
Examples
An example encoding of a relative octree encoding followed by attribute and level-of-detail encodings:
A RTCP feedback message containing a relative octree encoding, attribute encoding and level-of-detail encoding.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |V=2|P|   FMT   |       PT      |          length               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  SSRC of packet sender                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Res. |1|0|1|1| Payload                                      ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min X                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min Y                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min Z                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max X                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max Y                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max Z                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | octree encoding                                             ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | attribute encoding                                          ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | level-of-detail encdoing                                    ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Format of RTP Header Extensions
RTP senders use RTP header extensions to acknowledge the applied parameters to the receiver. The parameters chosen by the sender may differ from the ones requested by the receiver. The header extension uses the two-byte header form defined in [RFC8285]. The payload of the header extension element uses the same format as the RTCP messages defined in Section 5. Each extension element begins with the same one-byte header (Figure 4) followed by an octree encoding and optional priority, attribute, or level-of-detail encodings.
Common RTP header extension payload header

    0
    0 1 2 3 4 5 6 7
   +-+-+-+-+-+-+-+-+
   |  Res. |R|P|A|L|
   +-+-+-+-+-+-+-+-+

SDP Parameters
This section defines SDP parameters for negotiating usage of the RTCP messages and RTP header extension described in this document.
RTCP Feedback Messages
The rtcp-fb attribute is extended to indicate the capability to send or receive the RTCP feedback defined in this document. This document adds a new parameter "oerr" to the "ccm" feedback value defined in [RFC5104]. The "oerr" (Octree Encoded Region Request) parameter indicates support for the RTCP feedback message defined in Section 5.
rtcp-fb-ccm-param =/ SP "oerr" ; Octree Encoded Region Request
RTP Header Extensions
The URI for indicating support for the RTP Header extension described in Section 6 is "urn:ietf:params:rtp-hdrext:octree-region".
IANA Considerations
RTCP Feedback Message
The following value are requested to be registered as FMT value in the "FMT Values for PSFB Payload Types" Registry:
Name:
OERR
Long Name:
Octree Encoded Region Request
Value:
TODO
Reference:
TODO: This document
RTP Header Extension
The following URI is requested to be added to the RTP Compact Header Extensions registry:
Extension URI:
urn:ietf:params:rtp-hdrext:octree-region
Description:
Octree Encoded Region Information
Contact:
mathis.engelbart@gmail.com
Reference:
TODO: This document
Security Considerations
TODO
RFC Editor Considerations
Note to RFC Editor: This section may be removed after carrying out all the instructions of this section.
TODO: Consider
Normative References
[RFC2119]

Key words for use in RFCs to Indicate Requirement Levels
S. Bradner
March 1997
BCP 14
RFC 2119
DOI 10.17487/RFC2119
[RFC8174]

Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words
B. Leiba
May 2017
BCP 14
RFC 8174
DOI 10.17487/RFC8174
[RFC3550]

RTP: A Transport Protocol for Real-Time Applications
H. Schulzrinne
S. Casner
R. Frederick
V. Jacobson
July 2003
STD 64
RFC 3550
DOI 10.17487/RFC3550
[RFC8285]

A General Mechanism for RTP Header Extensions
D. Singer
H. Desineni
R. Even, Ed.
October 2017
RFC 8285
DOI 10.17487/RFC8285
[RFC5104]

Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)
S. Wenger
U. Chandra
M. Westerlund
B. Burman
February 2008
RFC 5104
DOI 10.17487/RFC5104
Acknowledgments
TODO acknowledge.