Subject: [Cellar] AV1 seeking

From: Steve Lhomme <[email-masked]>
Date: Sat, 14 Jul 2018 08:47:45 +0200

Hi,

Following Andreas' feedback on the AV1 thread it seems there's
something that needs to be looked deeper in AV1. There is a kind of
RAP (Random Access Point) that is actually a not visible frame but
that is a keyframe and a seekpoint (definition of RAP). That frame is
actually displayed later (maybe sometimes it isn't) using a dummy
frame (Frame Header OBU but not a Frame OBU). Sounds good so far.

The problem is that between the keyframe and its visible sibling,
there can be other frames. They may depend on frame before the RAP.
This is grey area in the specs where it's up to the decoder to chose
how to properly (or not) display them.

Here's an example stream
a b [C d] e f g h i [c] j k l
C is the non visible keyframe which (from what I understand) always
have another frame in the Temporal Unit, d in this case. And [c] is
displayed later in the stream.

If a user wants to seek to frame 'f', the normal seeking would start
decoding at 'C'. But f may depend on frame 'a' (which hopefully is a
keyframe otherwise we need the frames referenced by 'a'). Also this is
very much the concept of B frames since 'c' is displayed after 'f' but
with a different name.

Using BlockReference we can actually know where to seek for these
particular frames to get all the frames they need. But most people use
SimpleBlock (or we could forbid its use for such streams) and it would
mean referencing all frames in the Cues which is not a good idea.

In MP4 they have [initial_presentation_delay_minus_one] in the
CodecPrivate. I did not understand it so far because it's not found in
the AV1 spec. But it seems to guarantee that to read frame 'f' you
need to decode X frame before that one. In our case that would be 5 to
have at least a decoded.

IMO this is a bad design because it mixes the container with the codec
(something done in the past in ogg with bad results). Seeking should
not ask the codec how it should be done. Also that means to mux such
an MP4 you'd need to scan the whole source first to verify the value
is correct or that information needs to be given to the muxer by the
encoder/packetizer (this information is not found in the stream).

We have the option of not caring as it's allowed to leave it in grey
area by the AV1 spec. Or we can think of a proper "container"
solution. ie storing this information in the TrackInfo but not in the
CodecPrivate. We already have a CodecDelay but IMO it doesn't fit the
bill. Each frame has the timestamp modified by it. Which is not the
case here. We may need a SeekingDelay so that looking for a seek point
takes this amount in account.

-- 
Steve Lhomme
Matroska association Chairman

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sat, 14 Jul 2018 13:31:11 -0700

> need to decode X frame before that one. In our case that would be 5 to

You want to search for InitialPresentationDelay in the AV1 
specification. It is derived from the decoder model info if it is 
available (see decode_process()).

---

From: Andreas Rheinhardt <[email-masked]>
Date: Sun, 15 Jul 2018 22:19:00 +0000

Hello,

> mean referencing all frames in the Cues which is not a good idea.
I actually have already made some proposals that allow the demuxer to
know where to seek to. Here are two old ones and a new one (c)) which is
my new favourite:

a) One could reference the recovery points and the keyframe (i.e.
non-delayed) RAPs in the cues. Both would be normal cue entries. The
recovery points would have to be put into a `Block` in a `Blockgroup`
and said `Blockgroup` must have a `ReferenceBlock` that points to the
delayed RAP.
Pro: Normally the delayed RAP and the recovery point end up in the same
cluster and so hopefully the delayed RAP can be quickly accessed because
it is still cached. Also this approach doesn't rely on anything
deprecated in Matroska or unavailable in Webm. It can be generalized to
the other gradual decoder refresh scenarios.* It has very low overhead.
Con: Depending on the demuxer implementation this might result in two
seeks; it is also not really very backward-friendly: If one has a
delayed RAP at t_0, the recovery point at t_1 and the next RAP at t_2
with t_0<t_1<t_2 and a user wants to seek to somewhere between t_1 and
t_2, a player that doesn't support this kind of seeking will seek to t_1
and either refuse to decode said frame, because it isn't a keyframe, (in
which case it probably proceeds to t_2 and starts decoding there) or
will decode resulting in corrupted output.

b) One could reference both recovery points and keyframes. The
`CuePoint` (actually CueTrackPositions) for recovery points would
contain `CueReference` containing `CueRefTime` (whose value would be the
timestamp of the (Simple)Block containing the delayed RAP) and `CueCluster`.
Pro: Only one seek is necessary. Can be generalized (actually it is
already general) to the general case of gradual decoder refresh. Low
overhead.
Con: Uses elements that are deprecated in Matroska and unavailable in
Webm. A demuxer that doesn't handle these uncommon elements will have
the same problem as in a).

c) To seek adaequately the demuxer doesn't need to know the complete
reference structure. Actually, experience has shown that this
information is both unneeded and undesired at the container level. The
only things that are needed are the timestamp and position of the RAP
and the timestamp of the block from which point on the output is
considered acceptable.
Therefore I propose to create a new element (child of
`CueTrackPositions`) called `DecoderConvergence`/`OutputGood` or
whatever (I don't have a good name for it -- suggestions welcome!); this
element (a signed (variable sized) integer) would contain the timestamp
offset (in TimestampScale units) relative to the CuePoint`s
corresponding `CueTime` from which on the output of the decoder is
considered good/acceptable when one starts decoding at the referenced
block. No default value and if this element is not present then nothing
can be inferred regarding the point from which on the output will be
acceptable. (We shouldn't simply make the element mandatory with default
value 0, because there are already intra-decoder-refresh H.264 files and
muxers (like mkvmerge) have simply referenced the frames that contain a
recovery point SEI even though outputting said frame is unacceptable
because they are damaged (when decoding began at said frame).)
The AV1 usage is clear: The delayed RAPs that are later output at
recovery points get referenced and the offset of the accompanying
recovery point ends up in the new element. Similar for
intra-decoder-refresh scenarios.
There is also another usage: It can be used for H.265's decodable
leading (RADL) pictures: This is like open-gop, but with the difference
that the frames that follow the keyframe in decoding order and precede
it in display order don't reference anything that precedes the keyframe
in decoding order so that these frames can be correctly decoded. This
means that you can effectively use the "golden" keyframe as a suitable
reference for more frames. This is why the element is signed.
(This also shows that `DecoderConvergence` is not a fitting name. As has
been said: Suggestions welcome!)

Pro: Low overhead. Backward compatible with demuxers that don't
understand the element (as long as they ignore it as they should).
General solution. Only one seek is necessary.
Con: Requires a new element, in particular we need to talk to the Webm
guys; doesn't solve the problem for files without cues.

*: In this case the RAP where decoding starts might not be one of the
frames that the frame where the output is good directly references; yet
it indirectly references it, because the direct references reference it
(or the references of the references; you get the idea). Nevertheless
one should add a `ReferenceBlock` pointing to said RAP (and this is the
only needed `ReferenceBlock`). But I have to add that this is a
deviation from the current understanding of the `ReferenceBlock` element.

> need to decode X frame before that one. In our case that would be 5 to
You completely misunderstood this field. It is the equivalent of the
max_num_reorder_frames value from H.264. Let me explain it in MPEG
terminology (with b-frames) as you probably have way more experience
with this: Consider a decoder that can only decode one frame per unit of
time and a stream like this (left to right is decoding order; the
numbers are presentation order):
I0 P2 B1 ...
If one displayed the leading I frame immediately after decoding it, one
would not have the right frame to display at time 1, because at that
time only I0 and P2 has been decoded, not B1. Therefore one has to
decode I0 and P2 before one outputs the first frame and and
max_num_reorder_frames would be 1. The typical b-pyramid would require
to decode the first three frames before the display of the first frame
and max_num_reorder_frames would be 2.
The [initial_presentation_delay_minus_one] is the AV1 analogue of this.
This number is not an upper bound for the amount of temporal units
between delayed RAP and recovery point/for the amount of frames shared
between a GOP. Just look at this example:
MPEG example
I0 P5 B1 B2 B3 B4 I10 B6 B7 B8 B9
AV1 example (I kept the MPEG-naming with P and B to make it easier
comparable to the above; furthermore the pointer *x denotes that a frame
is showable and x without * is a frame header that outputs *x via
show_existing_frame;  square brackets are the delimiters of temporal
units; I10 is the delayed RAP frame)
[I0] [*P5 B1] [B2] [B3] [B4] [P5] [*I10 B6] [B7] [B8] [B9] [I10] ...
This stream can have [initial_presentation_delay_minus_one] equal to 1,
yet in order to seek to [I10] one has to decode the temporal unit [*I10
B6] (or at least the decodable keyframe in it) which is four temporal
units in front of [I10].

> takes this amount in account.
yes, this is completely different than CodecDelay. There is a Matroska
field that could be used for this: SeekPreRoll. But there are two
problems with this: This value is valid for the whole track, whereas the
amount of time between delayed RAP and recovery point can vary. And of
course the needed value is generally unknown at the beginning of the
muxing process. Not to mention the fact that actually SeekPreRoll is
valid if one starts decoding at an arbitrary frame, not necessarily a
(delayed) keyframe. Therefore a valid SeekPreRoll element for the track
would have to be so large that it would be impractical.

- Andreas Rheinhardt

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 16 Jul 2018 08:33:55 +0200

Ah yes, thanks. I overlooked that section (E.4.7) (for some reason the
PDF searching doesn't catch everything so I'll you the github repo
source).

The field definition also says:
"If not signaled then initial_display_delay_minus_1[ i ] =
BufferPoolMaxSize - 1."
and
"it is indicated by a VBI slot that points to this frame buffer.
BufferPoolMaxSize is equal to 10."

So if I understand correctly by default the value of that field is
assumed to be 9 if [initial_display_delay_present_flag] is set but not
the individual values for each Operating Point. And it can go up to 15
(4 bits) meaning 16 frames are needed in the decoding buffer.

2018-07-14 22:31 GMT+02:00 Timothy B. Terriberry <[email-masked]>:
> You want to search for InitialPresentationDelay in the AV1 specification. It
> is derived from the decoder model info if it is available (see
> decode_process()).

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 16 Jul 2018 10:01:47 +0200

Hi,

> This stream can have [initial_presentation_delay_minus_one] equal to 1,

A value of 1 means 2 frame delay. That would mean B6 needs *I10 and P5 ?

> B6] (or at least the decodable keyframe in it) which is four temporal

That seems more like it.

In the Frame presentation timing paragraph (E.4.7) it says:

InitialPresentationDelay =  Removal [ initial_display_delay_minus_1 ]
+ TimeToDecode [ initial_display_delay_minus_1 ]

and

PresentationTime[ 0 ] = InitialPresentationDelay
PresentationTime[ j ] = InitialPresentationDelay + (
frame_presentation_time[ j ] - frame_presentation_time[ 0 ] ) * DispCT

or in constant bitrate mode

PresentationTime[ 0 ] = InitialPresentationDelay
PresentationTime[ j ] = PresentationTime[ j - 1 ] + (
num_ticks_per_picture_minus_1 + 1 ) * DispCT

And our `Block` timestamp derives directly from this
[PresentationTime]. The delay is carried over every frame. So it does
seem like we need to globally shift these timestamps for the Track.
Possibly with `CodecDelay`.

Basically each frame has its timestamp on which this delay has to be
added (can't be negative). While `CodecDelay` is a positive value that
needs to be substracted from the timestamps. It is available in WebM,
because Opus needs it.

Let's assume CodecDelay = InitialPresentationDelay

The `Block` timestamps would be stored like this:
Block[ 0 ] = 0
Block[ 1 ] = frame_presentation_time[ 1 ] * DispCT
Block[ 2 ] = frame_presentation_time[ 2 ] * DispCT
...

And on output of the demuxer we would get:
Block[ 0 ] = -InitialPresentationDelay
Block[ 1 ] = -InitialPresentationDelay + frame_presentation_time[ 1 ] * DispCT
Block[ 2 ] = -InitialPresentationDelay + frame_presentation_time[ 2 ] * DispCT
...

`CodecDelay` cannot be used here. But it's very close to what we need.

It's not the SeekPreRoll either because the actual frame timestamps of
each frame is affected, not just when seeking (although it may be
needed for delayed RAP).

We also need to figure out whether the Blocks need to be stored with
or without the delay. If that's with the delay then we don't even need
to care about it. The frames will just not start at 0 but that's
already the case in the original stream.

But that may just be the tricky part here, the reference point. The
PresentationTime[ 0 ] doesn't start at 0 because some frames were
needed to decode before getting actual usable data out of the decoder.
But this is really the first frame to display so it would actually be
0 on the output of the demuxer. So it does seem exactly like what
CodecDelay does:
"CodecDelay is The codec-built-in delay in nanoseconds. This value
must be subtracted from each block timestamp in order to get the
actual timestamp."

The `Block` timestamp would be PresentationTime[ 0 ] which isn't 0.
But on the output Block[ 0 ] should really give 0. So we would have
something like this in the file:
Block[ 0 ] = InitialPresentationDelay
Block[ 1 ] = InitialPresentationDelay + frame_presentation_time[ 1 ] * DispCT
Block[ 2 ] = InitialPresentationDelay + frame_presentation_time[ 2 ] * DispCT
...

And on the output of the demuxer we would have the following, when
`CodecDelay` is [InitialPresentationDelay]:
Block[ 0 ] = 0
Block[ 1 ] = frame_presentation_time[ 1 ] * DispCT
Block[ 2 ] = frame_presentation_time[ 2 ] * DispCT
...

It does look like what the internal AV1 display delay is trying to achieve.

The `CodecDelay` also has this note on proper muxing:
The value SHOULD be small so the muxing of tracks with the same actual
timestamp are in the same Cluster.

Because muxing is done based on the stored `Block` timestamp. It
usually doesn't take in account CodecDelay (but it could/should).

So I'll update my AV1 codec mapping saying `CodecDelay` is
[InitialPresentationDelay] and how to use it.

---

From: Andreas Rheinhardt <[email-masked]>
Date: Tue, 17 Jul 2018 13:23:00 +0000

Hello,

> A value of 1 means 2 frame delay. That would mean B6 needs *I10 and P5 ?
No, absolutely not. This field tells you nothing at all about seeking
and in particular it does not say that "2 frame delay" implies that it
is sufficient to seek to a frame two frames before said frame and start
decoing there to get the (undamaged) desired frame. But judging from
your next comment, you have already realized this on your own. The frame
delay field is there to solve a problem that is completely orthogonal to
the seeking problem, hence it is actually wrong to discuss it under the
label "[Cellar] AV1 seeking". I will nevertheless reply to your latest
change of the current proposal in this email.

> [InitialPresentationDelay] and how to use it.
1. Good that you thought about the initial presentation delay. I have
personally experienced how unfortunate it is that the H.264
packetization doesn't provide a way to indicate the necessary number of
reorder frames (see e.g. <https://github.com/FFMS/ffms2/issues/301>) and
I have talked to Moritz Bunkus about using the `MinCache` value to store
the number of reorder frames. (Btw: The "reference pseudo-cache system"
this element talks about seems to be undefined.)

2. Nevertheless I think that your proposal is totally wrong:

a) The semantics of `CodecDelay` are unclear: Although the current
wording only speaks about a delay that should be subtracted from the
timestamps, the muxers are actually using this value to map the Opus
preskip to a Matroska field, i.e. they are treating it as if it would
not only be a delay that should be subtracted from the timestamps, but
as if it also signaled `DiscardPadding` at the beginning. There was a
discussion in May 2016 in which everyone agreed that `CodecDelay`
included discarding samples at the beginning and the following proposal
for an alternative wording was made (see
<https://mailarchive.ietf.org/arch/msg/cellar/ATGyypffoo9DuIFGUpeRSfzTVqE>):
"CodecDelay is the duration of the codec-built-in delay in nanoseconds.
The decoded frames from the beginning of a stream should be discarded
until CodecDelay duration has passed, and CodecDelay must be subtracted
from each block timestamp in order to get the presentation timestamp.
The value should be small so the muxing of tracks with the same
presentation timestamp are in the same Cluster."
This has not been agreed on not because the discard-part was
controversial per se, but because it was unclear how exactly to convert
this ns value to precise samples given that they use different timebases.

Notice that one thing has not been brought up during the linked
discussion: What exactly will be discarded? Is it always the first
`CodecDelay` output or is it only the output which has a negative
timestamp after the timestamp shift specified by `CodecDelay` has been
applied? This is important if the lowest timestamp is >0 (before
shifting by `CodecDelay`). In the second case one would have to
additionaly use a `DiscardPadding` element at the beginning to signal
that the appropriate number of samples should be discarded. No muxer I
know of currently does this, i.e. they work as if not only the samples
that end up with a negative timestamp after the shift were discarded.
And if we keep this logic then the first frames will be decoded, but
discarded under your proposal; if we don't keep it, then some files will
have been wrongly muxed (i.e. then current muxers haven't signalled the
Opus `PreSkip` correctly at all and people that relied on this
interpretation are screwed).

b) The elementary streams of other codecs also has such an offset (See
e.g. equation (C-12) in the H.264 standard.), but we don't store it for
them either. And it works.

c) The time in Annex E is obviously a dts given that it starts at zero
when decoding begins. (Here I consider the arrival of the data in a
buffer as part of the decoding process.) But Matroska uses pts. The only
thing that your proposal does is signalling the pts to dts difference.
But only for the very first picture. This is enough so that one can play
the track smoothly from the beginning; but it is not enough in general.

Let me explain: Nothing guarantees you that the pts to dts offset for
the first frame is the pts to dts offset for any further keyframes (even
when we restrict this to the keyframe RAP and not the delayed RAPs;
after all, if one discards from a CVS all OBUs in front of a keyframe
RAP, one still has a valid CVS (provided that the `Sequence Header OBU`
is still available)). That's because it may depend on
[buffer_removal_time] which needs to be signalled for every frame. Now
the obvious solution for this would be to use the highest pts to dts
offset for any RAP (or only for every keyframe RAP; I actually don't
know if one would have to use the offset between pts and dts for the
delayed RAPs, too). This has the downside that you would have to do the
muxing in two passes; after all, if you notice that you need to use an
increased value of `CodecDelay`, you would have to rewrite not only the
track header, but the timestamps written so far as well if you tried to
mux in one pass.
Here is a situation where the above scenario might be realistic:
You have a vfr track that actually consists of several sections that are
cfr (but when combined are nevertheless vfr). The beginning is (say)
50p, the rest is 25p (this really happens: if the movie is shot in 25p,
but the title and end credits are overlayed as 50p, then the result is
effectively 50p at the beginning and end and 25p in the middle; I have
several TV broadcasts like that). Then it is quite likely that the
individual 50p frames are smaller (coded size) so that they can be
transmitted faster and so one can use a lower ScheduledRemovalTiming for
the DFGs corresponding to the 50p frames. This means that the Removal
times (in decoding schedule mode) will be smaller as well. Hence the
InitialPresentationDelay for decoding from the very first frame is
smaller than the InitialPresentationDelay for decoding from one of the
later RAPs.
For this reason decoders would probably rely on the
[initial_display_delay_minus_1] values (that are luckily not stripped
away). But then there is no point in using `CodecDelay` at all as this
value can also be used at the beginning.

d) There are additional complications:

i) It would complicate muxers that would have to implement the
procedures described in E.4.

ii) It breaks compability with mp4/ISOBMFF because they only store the
initial_presentation_delay_minus_one value (storing it is recommended,
but not required) and (that's a SHOULD) they should set the
[timing_info_present] flag to zero which means that the bitstream itself
wouldn't contain an explicitly signalled presentation delay any more.
One would have to recalculate the `CodecDelay` value, but there is a
problem with this approach: Appendix E.3 knows two modes: Resource
availability mode and the decoding schedule mode. The latter needs the
buffer_removal_time information from the frame header, but they SHOULD
be stripped away during muxing into mp4, so they are likely not
available and the calculation can't be performed at all. The former has
the prerequisite that it needs to be crf ([equal_picture_interval] equal
to 1). Given that the timing information has been discarded,
[equal_picture_interval] is generally unavailable in mp4, but we can
still check whether it is crf. And what if it isn't? What `CodecDelay`
value does one use?

iii) It is not possible every time even when muxing from an AV1
elementary stream: A bitstream can contain
[initial_display_delay_minus_1] even when it doesn't signal explicit
timing info or when it doesn't signal the decoder_model_info or the
operating_parameters_info or if it doesn't signal the buffer removal time.
In my experience with H.264, it was enough to explicitly signal the
[max_num_reorder_frames] for correct playback; it was not necessary to
also add the SEI messages necessary for conformance testing of the
track. (This is also how x264 works by default: No unnessary SEIs, but
the [max_num_reorder_frames] is set.) I actually wouldn't be surprised
if an x264-like AV1 encoder would use the same defaults: Signal
[initial_display_delay_minus_1], don't signal the buffer_removal_time
(too many wasted bits for no gain) and don't signal the operating
parameters.

iv) Of course the `CodecDelay` would also depend on the operating point
(if scalability is used). I have to admit this could be easily
rectified: Simply calculate the `CodecDelay` for every operating point
and use the maximum.

v) Purely Matroska-wise, there are further complications: Splitting such
a file would mean that the muxer has to use the correct `CodecDelay`
value for the second file. This is in general a different value than the
value for the first file, but the value can't be calculated because the
timing information has been discarded from the bitstream.
Alternatively appending files where two tracks that are to be appended
to one another have different `CodecDelay` values also requires changes;
if the second file has the higher `CodecDelay` value, one would have to
increase the `CodecDelay` for the whole output track to that value and
this entails to shift the timestamps of the first file, too.
There probably are more complications further down the road.

3. Here is a counterproposal: We more or less copy how mp4 does it. I.e.
we ignore InitialPresentationDelay when determining the timestamps and
we add a field that is equivalent to their
[initial_presentation_delay_minus_one].
Honstely I don't even know what they exactly mean by "display model
verification algorithm". Is it the procedure described in annex E? If
so, they can run this algorithm only if the video is cfr (resource
availability mode) or the buffer_removal_time is given in the frame
headers (decoding schedule mode), but the latter SHOULD have been
striped away, so that leaves a crf requirement. (Or they could have kept
the relevant paramters from the elementary stream (maybe not in the mp4
file, but somewhere else).)
Anyway, for our purposes it will be enough to use the maximum of all
[initial_display_delay_minus_1] values if said value is present for
every operating point that occurs in the file. It is one of the fields
which can't change mid-way in a CVS so it works well for one-pass muxers.
The easiest way would be probably to add a byte to the beginning of
`CodecPrivate` (whose definition would have to be changed). How about
the MSB bit being the "initial_presentation_delay_present" bit, then
three reserved bits (set to zero, may hypothetically be used to indicate
a new version of the AV1 packetization in Matroska if we wished to keep
the AV1 CodecID) and then the `initial_presentation_delay_minus_one`,
coded on 4 bits.
If someone wonders why I propose to add such a field even when the
information is in the 'Sequence Header OBU` (after all, this is not
stripped away): Because it actually needn't be in the `Sequence Header
OBU`! E.g. think of an AV1 bitstream inside mp4 where the `Sequence
Header OBU` delay field doesn't exist, but where the  initial delay is
signalled on the container level. Muxing from mp4 to Matroska would
loose this information when it is not kept somewhere. Furthermore, just
as a muxer can actually count the number of reorder frames necessary for
H.264, it can probably do the equivalent with the initial presentation
delay for AV1 (this is not to say that this should be required or even a
SHOULD) and then it can put the value into this field (rewriting a field
in the track header is possible even for a one-pass muxer).

4. How helpful is this buffer removal timing information actually? Is it
needed for streaming? I don't think so, because then the mp4 guys (one
of the main authors of the mp4 packetization works for Netflix) would
not have recommended to strip it away. But maybe conformance testing
should be mentioned as another reason not to strip the [timing_info]
away. (I would only keep the timing_info parameters for a crf track in
order to have the real framerate stored, but I would discard the
decoder_model_info stuff.)

- Andreas Rheinhardt

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 18 Jul 2018 14:22:30 +0200

> 2. Nevertheless I think that your proposal is totally wrong:

It's may be wrong but it's not *totally* wrong.

> presentation timestamp are in the same Cluster."

OK, something needs to be done here. It seems clear that the data
dropping is needed for Opus. It's not entirely clear for AV1. Since
the DRAP is delayed it may have data before the actual keyframe is
shown. We could decide that it should not be shown and so it would be
like Opus (after clarifying the dropping of CodecDelay).

I added an issue so we don't forget to clarify this:
https://github.com/Matroska-Org/matroska-specification/issues/242

> `CodecDelay` output or is it only the output which has a negative

As you quoted the CodecDelay definition and the change proposal you
know it applies to all timestamps in the track.

> them either. And it works.

I'm not well versed in other codecs. I can tell you about a lot issues
with hardware decoder that fail after seeking though. So maybe it
works most of the time but it's not clear cut.

> value can also be used at the beginning.

It depends if CodecDelay always means dropping the amount of
CodecDelay before rendering data. In this case it doesn't correspond
to the [InitialPresentationDelay]. But if it doesn't (and it could be
managed by a separate flag) then it can make sense.

I believe the values are added in MP4 so that seeking happens
smoothly, ie you know how much you need to buffer when seeking. It may
also be a way for adaptive streaming to tell how many frames need to
be grabbed in advance when switching to another bitrate/resolution. It
would be an odd way to do it though.

I don't agree that it's the dts though. Because the dts would be
strictly incrementing. But the DRAP is in fact decoded many frames
before being displayed (in the normal usage case).

> procedures described in E.4.

Yes. This is also the case for MP4 to find the relevant
"initial_presentation_delay_minus_one" value.

> but not required) and (that's a SHOULD) they should set the

I don't see where it says it's a SHOULD. Only the pseudo-algorithm
"SHALL" be applied to the data and still be a valid bitstream.

It's not a MUST either so it may be optional, but it's not clear.

> rectified: Simply calculate the `CodecDelay` for every operating point

If it's [InitialPresentationDelay] then it's implied. If it's based on
something else like [initial_display_delay_minus_1] then it needs to
be the maximum, yes.

> timing information has been discarded from the bitstream.

No. At least in the case of Opus, splitting a file keeps the same
`CodecDelay` value. Splitting is just like seeking in this case.

> to one another have different `CodecDelay` values also requires changes;

You can't append tracks with a different CodecPrivate. And if it's the
same I would assume the CodecDelay is the same (either in Opus or
AV1).

> which can't change mid-way in a CVS so it works well for one-pass muxers.

I still don't understand this part in MP4. This is information that is
already in the Sequence Header OBU that is (but not always) following
this data. And in our case the Sequence Header OBU is mandatory so no
need to copy this information again.

> the AV1 CodecID) and then the `initial_presentation_delay_minus_one`,

No, we're not going to change the interpretation of a field that's
already widely used (WebM). We can add an extra element with a default
value that matches what is done with Opus. For example whether the
CodecDelay means discarding that amount of data after a discontinuity
(start or seek). For the record VLC doesn't discard this amount of
data on output of the decoder. I'm not sure avformat/avcodec is
capable of doing that either.

> loose this information when it is not kept somewhere. Furthermore, just

It wouldn't be lost. To mux in Matroska you'd need to recover the
Sequence Header OBU from the MP4 to put it in the CodecPrivate (it's
an OBU that preceeds the first keyframe).

> in the track header is possible even for a one-pass muxer).

I think we went full circle. Do we need to do anything at all ?

The initial problem is with DRAP and seeking, right ? The Block with a
"keyframe" data is marked as a keyframe whether it's displayed or not.
So the real problem is what to do with the frames before the keyframe
is displayed. The spec is blurry on what to do with frames in between
(big Note in 7.6.3: "the preceding frames are implementation
defined"). And it's likely not a choice to be made at the container
level. So maybe we just don't care.

> decoder_model_info stuff.)

Thanks for taking the time for all these detailed talks. It really
helps to have a foolproof solution in the end.

> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 18 Jul 2018 15:42:52 +0200

I asked Cyril Concolato to join the conversation so we can discuss the
differences between our current mapping and the ISOBMFF/MP4 one he is
working on.

IMO there are two main differences for now:

- initial_presentation_delay_present /
initial_presentation_delay_minus_one which we may or may not store
(and where does it come from)

- the configOBU that may be completely empty, whereas in our case we
have exactly one Sequence Header OBU and the ones in the Segment must
be very close to that one.

---

From: Cyril Concolato <[email-masked]>
Date: Wed, 18 Jul 2018 16:59:35 +0200

> differences between our current mapping and the ISOBMFF/MP4 one he is
Thanks Steve.

Some comments regarding the previous messages I read in the archives of
this list:

You wrote:
"Here's an example stream

with a different name."

[CC] In this configuration, to seek to frame 'f' you need to start decoding
at frame 'a' (visible Key Frame). To seek to frame 'k' you need to decode
only 'C' and then 'c', and 'j'.

"In MP4 they have [initial_presentation_delay_minus_one] in the
CodecPrivate. I did not understand it so far because it's not found in
the AV1 spec. But it seems to guarantee that to read frame 'f' you
need to decode X frame before that one. In our case that would be 5 to
have at least a decoded."
[CC] "initial_presentation_delay_minus_one" in the AV1-ISOBMFF is related
to the "initial_display_delay" in the AV1 codec spec. They express the same
delay but in different units: the latter expresses it in number of decoded
frames, while the former expresses it in number of presented frames. The
reason for this difference is that an application may not be able to know
when a frame is not decoded it only sees when it is output.
That said, this delay is not at all related to the seeking feature. It
tells you how many frames have to be decoded (or output) to guarantee
smooth playback if the decoder decodes at the decode rate of the level
indicated in the bitstream. It may be useful for low latency applications.

"IMO this is a bad design because it mixes the container with the codec
(something done in the past in ogg with bad results). Seeking should
not ask the codec how it should be done. Also that means to mux such
an MP4 you'd need to scan the whole source first to verify the value
is correct or that information needs to be given to the muxer by the
encoder/packetizer (this information is not found in the stream)."
[CC] I don't understand what this means.

> (and where does it come from)
[CC] Hope it's clearer.

> be very close to that one.
[CC] The intent of the AV1-ISOBMFF spec is that the Sequence Header OBU has
to be present in-band and may be present in the configOBU and if present
shall be identical. In your spec, I don't understand what "very close
means". AV1 does not allow variation of Sequence Header within a CVS.

HTH,
Cyril

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 18 Jul 2018 17:36:09 +0200

Hi Cyril,

Thanks for clarifying some things.

2018-07-18 16:59 GMT+02:00 Cyril Concolato <[email-masked]>:
> Some comments regarding the previous messages I read in the archives of this
> list:
>
> [CC] "initial_presentation_delay_minus_one" in the AV1-ISOBMFF is related to
> the "initial_display_delay" in the AV1 codec spec. They express the same
> That said, this delay is not at all related to the seeking feature. It tells
> you how many frames have to be decoded (or output) to guarantee smooth
> playback if the decoder decodes at the decode rate of the level indicated in
> the bitstream. It may be useful for low latency applications.

OK but it is a duplicate value from the Sequence Header OBU may (or
may not) be found in the configOBU. Maybe it would be worth having a
specific atom just for that ? And you have either this atom or the one
with a Sequence Header OBU but not both.

There are many [initial_display_delay_minus_1] values in a single
Sequence Header OBU. So I assume the value stored in MP4 is the
maximum value ? IMO it should be said in the document.

Also in what cases a Sequence Header OBU would not be found in the
configOBU ? IMO it should be specified. We may have the same
requirement but it seems better to me to always have one. It helps
knowing the profile of the codec to initiate the decoder instance.
Otherwise we need to wait for the actual data when playback has
already started (along with audio and subtitles).

> [CC] I don't understand what this means.

Ogg is notorious for tying the container and the codec data. You can't
know the timestamp from just reading ogg, you need to know the codec
and analyse the data inside. So if seeking in AV1-in-ISOBMFF required
the demuxer to parse the "initial_presentation_delay_minus_one" data
would be a bit similar. But as you explained, it's not the case.

>> (and where does it come from)

Yes

> means". AV1 does not allow variation of Sequence Header within a CVS.

"Very close" was in my email but that's not the wording in the
MKV/WebM mapping. In that document there is this CVS documentation:

"A Coded Video Sequence is a sequence of video frames where the
contents of [sequence_header_obu] must be bit-identical for all the
Sequence Header OBUs found in the bitstream before Matroska
encapsulation except for the contents of [operating_parameters_info]."

[operating_parameters_info] is allowed to change in a CVS in our case.
This is the same as found in the AV1 specification 7.5 Ordering of
OBUs:

"Sequence header OBUs may appear in any order within a coded video
sequence. Within a particular coded video sequence, the contents of
sequence_header_obu must be bit-identical each time the sequence
header appears except for the contents of operating_parameters_info."

I suppose by "shall be identical" that's the change that is allowed.
But is there are reason not to put a Sequence Header OBU in the
AV1CodecConfigurationRecord ? In Matroska we advise to strip the
similar ones from the bitstream to save space so that's a direct
benefit of putting it there. But there's also a benefit for playback
(a similar feature is missing from VP9 and causes hardware detection
issues in VLC).

> Cyril

Thanks again.

---

From: Andreas Rheinhardt <[email-masked]>
Date: Thu, 19 Jul 2018 08:46:00 +0000

Hello Cyril,

thanks for joining us here.

> indicated in the bitstream. It may be useful for low latency applications.
That confirms what I have been thinking all along. But nevertheless I
have some questions (My interest stems from the fact that I am in favour
of including a field indicating how many frames need to be decoded in
advance):

1. What exactly is the "display model verification algorithm" that the
AV1-ISOBMFF speaks about? Does this just mean that the bitstream so
constructed should be tested whether it fulfills the requirements of
annex E of the AV1 specification?

2. If yes, then how can one test this precisely?

a) There are two operating modes in annex E.3: Resource availability
mode and decoding schedule mode. The latter mode needs
buffer_removal_time values for the check; but these values SHOULD have
been removed from the bitstream when muxing into ISOBMFF so this can in
general not be checked with the ISOBMFF file alone. Is it intended that
the muxer checks this during muxing when the original file with the
original values are still available or not?
Or do you just test this in resource availability mode. For the resource
availability mode equal_picture_interval equal should have been set to 1
(i.e. crf video), but E.3.1 also says "If the parameters listed above
are not specified by the bitstream, the parameters necessary to input
into this model can be signaled by the application or some other means."
I have not found that this crf requirement is in any way necessary. Are
you simply using the timestamps derived from the ISOBMFF file (even if
vfr) and use them to check in resource availability mode?

b) If one has verified that one can use a value of k for
initial_presentation_delay_minus_one, is this value also automatically
applicable for starting the presentation at any RAP (or at least any
keyframe/non-delayed RAP because -- after all, discarding the samples in
front of a keyframe RAP should give a conformant bitstream)? The answer
is of course tautologically true if "display model verification
algorithm" includes checking presentation from every RAP onwards, but
the current spec proposal talks only about constructing a (singular)
hypothetical bitstream consisting of all the OBUs in the sample entry
followed by all OBUs in all the samples.
The reason I am asking this are bitstreams like this one (everything
inside square brackets forms a temporal unit; &x means that a showable
frame x is output via the show_existing_frames mechanism (if I am not
mistaken, then it is allowed to output frames more than once -- and it
might even make sense to do this at the beginning if there are several
static frames at the beginning which sometimes happens with logos/black
frames)):
[A] [B] [&B] [C] [D E] [&D]
A is a keyframe RAP, B an ordinary frame that is allowed to be output
more than once. And it is output more than once. C is another
non-delayed keyframe. One can start presentation as soon as one has A
decoded (assuming the decoder can decode one frame during the
presentation time of one frame):
When one reaches the temporal unit [B], one has already decoded B. One
can decode C while B is shown.
When one reaches [&B], one has of course B still available. When one
shows B a second time, one can decode D. This effectively means that
when one starts at the very beginning, one enters the second CVS with
two frames already decoded (as if one started decoding at C with
initial_display_delay_minus_1 set to 1) and therefore one can use
temporal units like [D E] with more than one frame to decode.
In the above bitstream, one could use
initial_presentation_delay_minus_one equal to zero for presentation from
the beginning, but for decoding from C onwards one would need a value of 1.

3. a) The ISOBMFF specs contain the clause: "set the first
initial_display_delay_minus_1 field of each Sequence Header OBU to the
number of frames contained in the first
initial_presentation_delay_minus_one + 1 samples,"
Is this really intended? Shouldn't it read: "set the first
initial_display_delay_minus_1 field of each Sequence Header OBU to the
number of frames - 1 contained in the first
initial_presentation_delay_minus_one + 1 samples,"
After all initial_display_delay_minus_1 includes a minus 1, too; without
the minus 1 one would have to decode the first frame of the sample
number initial_presentation_delay_minus_one + 1 (zero-based), too,
before the presentation can start.

b) If one has a compliant AV1 bitstream where the Sequence Headers all
include the initial_display_delay_minus_1 for every operating point,
then one can use the maximum of all these initial_display_delay_minus_1
as initial_presentation_delay_minus_one if my above corrections turns
out to be right and is accepted? (If it's not accepted, then one could
even use the first the maximum - 1, couldn't one?)

- Andreas Rheinhardt