RTP Payload Format for sub-codestream latency JPEG 2000 streaming

6.2. Transmitting multi-tile images as multiple single-tile images

A sequence of multi-tile images can be transmitted by splitting it into multiple sequences of single-tile images, where:¶

• each sequence of single-tile images corresponds to a unique tile of the multi-tile image;¶
• each sequence of single-tile images is transmitted in a separate RTP stream;¶
• the coordinates of each single-tile image are expressed using the coordinate system of the multi-tile image; and¶
• the bitstreams of each of each single-tile image are identical to the corresponding bitstreams in the multi-tile image.¶

Such sequences of single-tile images are identified using the tile media type parameter specified at Section 9.2.¶

7.4. PTSTAMP field

A sender SHOULD set P = 1, but only if it can generate PTSTAMP accurately.¶

PTSTAMP can be derived from the same clock that is used to produce the 32-bit timestamp field in the RTP fixed header. Specifically, a sender maintains, at least conceptually, a 32-bit counter that is incremented by a 90kHz clock. The counter is sampled at the point when each RTP Packet is or SHOULD be at least notionally transmitted and the 12 LSBs of the sample are stored in the PTSTAMP field.¶

If P = 1, then the transmission time TOFF (as defined at Section 5.3) for two consecutive RTP packets with identical timestamp fields MUST NOT differ by more than 4095.¶

7.6. Extra information

The sender MUST set the value of XTRAC to 0.¶

Future edition of this specification can permit other values.¶

7.7. Reserved values

The sender MUST set reserved values to 0.¶

Future edition of this specification can specify other values such that these values can be ignored by receivers that conform to this specification.¶

7.8. Extension values

A sender MUST NOT use an extension value.¶

7.9. Code-block caching

This section applies only if C = 1.¶

A sender can improve bandwidth efficiency by only occasionally transmitting code-blocks corresponding to static portions of the video and otherwise transmitting empty code-blocks. When C = 1, and as described at Section 8.7, a receiver maintains a simple cache of previously received code-blocks, which it uses to replace empty code-blocks.¶

A sender alone determines which and when code-blocks are replaced with empty code-blocks.¶

The sender cannot however determine with certainty the state of the receiver's cache: some code-blocks might have been lost in transit, the sender doesn't know exactly when the receiver started processing the stream, etc.¶

A code-block is empty if:¶

• it does not contribute code-bytes as specified in the parent JPEG 2000 packet header; or¶
• if the code-block conforms to [jpeg2000-15], contains an HT cleanup segment and the first two bytes of the Magsgn byte-stream are between 0xFF80 and 0xFF8F.¶

NOTE: the last condition allows the encoder to insert padding bytes to achieve a constant bit rate even when a code-block does not contribute code-bytes, as suggested at [jpeg2000-15], F.4.¶

8.1. PTSTAMP

Receivers can use PTSTAMP values to accelerate sender clock recovery since PTSTAMP typically updates more regularly than timestamp.¶

8.3. RES

The JPEG 2000 coding process decomposes an image using a sequence of discrete wavelet transforms (DWT) stages.¶

Table 2 illustrates the case where each DWT stage consists of both horizontal and vertical transforms, which is the only mode supported in [jpeg2000-1]. The first stage transforms the image into (i) the image at half-resolution (LL1 sub-bands) and (ii) residual high-frequency data (HH1, LH1, HL1 sub-bands). The second stage transforms the image at half-resolution (LL1 sub-bands) into the image at quarter resolution (LL2 sub-bands) and residual high-frequency data (HH2, LH2, HL2 sub-bands). This process is repeated N_L times, where N_L is the number of decomposition levels as defined in the COD and COC marker segments of the codestream.¶

The decoding process reconstructs the image by reversing the coding process, starting with the lowest resolution image stored in the codestream (LL_NL).¶

As a result, it is possible to reconstruct a lower resolution of the image by stopping the decoding process at a selected stage. For example, in order to reconstruct the image at quarter resolution (LL2), only sub-bands with index greater than 2, e.g., HL3, LH3, HH3, HL4, LH4, HH4, etc., are necessary. In other words, a receiver that wishes to reconstruct an image at quarter resolution could discard all packets where RES >= 6 since those packets can only contribute to HL1, LH1, HH1, HL2, LH2 and HH2 sub-bands.¶

In the case where all DWT stages consist of both horizontal and vertical transforms, the maximum decodable resolution is reduced by a factor of 2^{7 - N} if all Body Packets where RES > N are discarded.¶

Table 3 illustrates the case where some of DWT stage consist of only horizontal transforms, as specified at Annex F of [jpeg2000-2].¶

A receiver can therefore discard all Body Packets where RES is greater than some threshold value if it is interested in decoding an image with its resolution reduced by a factor determined by the threshold value, as illustrated in Table 2 and Table 3.¶

8.4. Extra information

The receiver MUST accept values XTRAC other than 0 and MUST ignore the value of XTRAB, whose length is given by XTRAC.¶

Future edition of this specification can specify XTRAB contents such that this content can be ignored by receivers that conform to this specification.¶

8.5. Reserved values

The receiver MUST ignore the value of reserved values.¶

8.6. Extension values

The receiver MUST discard an RTP packet that contains any extension value.¶

8.7. Code-block caching

This section applies only if C = 1.¶

When C = 1, and as specified in Section 7.9, the sender can improve bandwidth efficiency by only occasionally transmitting code-blocks corresponding to static portions of the video and otherwise transmitting empty code-blocks, as defined at Section 7.9.¶

When decoding a codestream, and for each code-block in the codestream:¶

• if the code-block in the codestream is empty, the receiver MUST replace it with a matching code-block from the cache, if one exists; or¶
• if the code-block in the codestream is not empty, the receiver MUST replace any matching code-block from the cache with the code-block in the codestream.¶

Two code-blocks are matching if the following characteristics are identical for both: spatial coordinates, resolution level, component, sub-band and value of the TP field of the parent RTP packet.¶

9.2. Definition

Type name

video¶

Subtype name

jpeg2000-scl¶

Required parameters

None¶

Optional parameters

pixel

Specifies the pixel format used by the video sequence.¶

The parameter MUST be a URI-reference as specified in [RFC3986].¶

If the parameter is a relative-ref as specified in [RFC3986], then it MUST be equal to one of the pixel formats specified in Table 5 and the RTP header and payload MUST conform with the characteristics of that pixel format.¶

If the parameter is not a relative-ref, the specification of the pixel format is left to the application that defined the URI.¶

If the parameter is not specified, the pixel format is unspecified.¶

sample

Specifies the format of the samples in each component of the codestream.¶

The parameter MUST be a URI-reference as specified in [RFC3986].¶

If the parameter is a relative-ref as specified in [RFC3986], then it MUST be equal to one of the formats specified in Appendix C and the stream MUST conform with the characteristics of that format.¶

If the parameter is not a relative-ref, the specification of the sample format is left to the application that defined the URI.¶

If the parameter is not specified, the sample format is unspecified.¶

width

Maximum width in pixels of each image. Integer between 0 and 4,294,967,295.¶

The parameter MUST be a sequence of 1 or more digits.¶

If the parameter is not specified, the maximum width is unspecified.¶

height

Maximum height in pixels of each image. Integer between 0 and 4,294,967,295.¶

The parameter MUST be a sequence of 1 or more digits.¶

If the parameter is not specified, the maximum height is unspecified.¶

signal

Specifies the sequence of image types.¶

The parameter MUST be a URI-reference as specified in [RFC3986].¶

If the parameter is a relative-ref as specified in [RFC3986], then it MUST be equal to one of the signal formats specified in Appendix B and the image sequence MUST conform to that signal format.¶

If the parameter is not a relative-ref, the specification of the pixel format is left to the application that defined the URI.¶

If the parameter is not specified, the stream consists of an arbitrary sequence of image types.¶

caps

The parameters contains a list of sets of constraints to which the stream conforms, with each set of constraints identified using an absolute-URI defined by an application.¶

The parameter MUST conform to the uri-list syntax expressed using ABNF ([RFC5234]):¶

  uri-list = absolute-URI *(";" absolute-URI)

¶

Each absolute-URI MUST NOT contain any ";" character.¶

The application that defines the absolute-URI MUST associate it with a set of constraints to which the stream conforms. Such constraints can, for example, include the maximum height and width of images.¶

If the parameter is not specified, constraints, beyond those specified in this document, are unspecified.¶

tile

The parameter MUST conform to the tile syntax expressed using ABNF ([RFC5234]):¶

tile        = tile-index "&" image-siz
tile-index  = %x31-39 *%x30-39
image-siz   = 1*HEXDIG

¶

If the tile parameter is present, each image MUST correspond to one tile of a multi-tile image, as defined in Section 6.2.¶

tile-index is the index of the tile in the multi-tile image.¶

image-siz contains the SIZ marker segment parameters of the multi-tile image, encoded as a case insensitive hexadecimal string.¶

The SIZ parameters of each single-tile image MUST conform to the following:¶

• Xsiz MUST be equal to the smaller of (i) the coordinate of the right edge of tile index tile-index in the multi-tile image and (ii) Xsiz of the multi-tile image.¶
• Ysiz MUST be equal to the smaller of (i) the coordinate of the bottom edge of tile index tile-index in the multi-tile image and (ii) Ysiz of the multi-tile image.¶
• XOsiz MUST be equal to the larger of (i) the coordinate of the left edge of tile index tile-index in the multi-tile image and (ii) XOsiz of the multi-tile image.¶
• YOsiz MUST be equal to the larger of (i) the coordinate of the top edge of tile index tile-index in the multi-tile image and (ii) YOsiz of the multi-tile image.¶
• XTOsiz MUST be equal to the coordinate of the left edge of tile index tile-index in the multi-tile image.¶
• YTOsiz MUST be equal to the coordinate of the top edge of tile index tile-index in the multi-tile image.¶
• All other parameters MUST be equal to that in the multi-tile image.¶

Figure 4 illustrates an example where a multi-tile image that consists of two tiles is transmitted as two single-tile images (images 1 and 2). Figure 5 and Table 4 describe the tile and SIZ parameter values, respectively.¶

(a) Multi-tile image

<----------Xsiz---------->
<----> XTOSiz
+------------------------+
|    <---Xtsiz---->      |
|    xxxxxxxxxxxxxxxxxxxxxxxxxxx
|    x T0         x T1   |     x
|    x     #######x#######     x
|    x     #......x......#     x
|    x     #......x......#     x
+----xxxxxxxxxxxxxxxxxxxxxxxxxxx
<---XOsiz-->

(b) Single-tile image 1

<-------Xsiz------>     +--+
<----> XTOSiz           |  | Reference grid
+-----------------+     +--+
|    <---Xtsiz---->
|    xxxxxxxxxxxxxx     xxxx
|    x T0         x     x  x Tile area
|    x     #######x     xxxx
|    x     #......x
|    x     #......x     ####
+----xxxxxxxxxxxxxx     #..# Image area
<---XOsiz-->            ####

(a) Single-tile image 2

<----------Xsiz---------->
<-----XTOSiz------>
+------------------------+
|                        |
|                 xxxxxxxxxxxxx
|                 x T0   |    x
|                 x#######    x
|                 x......#    x
|                 x......#    x
+-----------------xxxxxxxxxxxxx
<-------XOsiz-----X---Xtsiz--->

NOTE: '\' line wrapping per RFC 8792

tile=0&00000190000000c8000000c800000000000000\
c8000000c800000064000000000001080000

Table 4: Selected SIZ parameters for a two-tile image and two corresponding single-time images.

SIZ parameter	Two-tile image	Image where tile-index = 1	Image where tile-index = 2
XSiz	400	300	400
YSiz	200	200	200
XTSiz	200	200	200
YTSiz	200	200	200
XOSiz	200	200	300
YOSiz	0	0	0
XTOSiz	100	100	300
YTOSiz	0	0	0

cache

The value of the parameter MUST be either false or true.¶

If the parameter is true, the field C MAY be 0 or 1; otherwise the field C MUST be 0.¶

If the parameter is not specified, then the parameter is equal to false.¶

Encoding considerations

This media type is framed and binary, see Section 4.8 of [RFC6838].¶

Security considerations

See Section 12.¶

Interoperability considerations

The RTP stream is a sequence of JPEG 2000 images. An implementation that conforms to the family of JPEG 2000 standards can decode and attempt to display each image.¶

Published specification

This document¶

Applications that use this media type

video streaming and communication¶

Person and email address to contact for further information

Pierre-Anthony Lemieux <pal@sandflow.com>¶

Intended usage

COMMON¶

Restrictions on Usage

This media type depends on RTP framing, and hence is only defined for use with RTP as specified at [RFC3550]. Transport within other framing protocols is not defined at the time.¶

Author

Pierre-Anthony Lemieux ¶

Change controller

IETF Audio/Video Transport Core Maintenance Working Group delegated from the IESG.¶

D.1. Introduction

This Appendix summarizes substantive changes across revisions of this specification. This summary is informative and not intended to be exhaustive.¶

D.2. Changes from draft-ietf-avtcore-rtp-j2k-scl-01

• Allow multi-tile images in a single stream, in addition to allowing multi-tile images to be transmitted as multiple single-tile streams, as specified at Section 6.2.¶
• Fix incorrect TRANS values at Table 5.¶