Subject: [Cellar] AV1 mapping update

From: Steve Lhomme <[email-masked]>
Date: Mon, 9 Jul 2018 12:24:17 +0200

I updated the AV1 mapping to clean a few sentences.

Since we allow stripping the Sequence Header OBU from the stream when
it's equal to the CodecPrivate one, we need to add it back to the
bitstream for compliance. At least when seeking on keyframes. So I
added a section to explain that.

IMO that's an extra feature of the CodecPrivate that it's meant to be
added to the bistream as-is. And in this case on startup and when
seeking. I wonder if we should add an element next to the CodecPrivate
to describe that. Because in this case it's not entirely opaque to the
demuxer. Or maybe it's implied by the CodecID and is up to the decoder
to use it how it's supposed to be (in this case detecting keyframes
and possibly adding back the Sequence Header OBU).

https://github.com/Matroska-Org/matroska-specification/blob/av1-mappin/codec/av1.md
and the list of changes can be found here
https://github.com/Matroska-Org/matroska-specification/commits/av1-mappin/codec/av1.md

Let me know what you think so we can settle this spec for good.

-- 
Steve Lhomme
Matroska association Chairman

---

From: Andreas Rheinhardt <[email-masked]>
Date: Wed, 11 Jul 2018 13:47:00 +0000

> https://github.com/Matroska-Org/matroska-specification/commits/av1-mappin/codec/av1.md
1. Whether `DisplayWidth` and `DisplayHeight` needs to be written
actually depends on the value of `DisplayUnit`.

2. You forgot `OBU_PADDING` in the list of OBU types that mustn't be in
the `CodecPrivate`. (Either that or your sentence that only
`OBU_SEQUENCE_HEADER` and `OBU_METADATA` are currently allowed in the
`CodecPrivate` should be changed.)

3. "They SHOULD have the [obu_has_size_field] set to 1 except for the
last OBU in the sample, for which [obu_has_size_field] MAY be set to 0,
in which case it is assumed to fill the remaining of the sample."
"The OBUs in the Block MUST follow the [Low Overhead Bitstream Format
syntax]."
The first sentence leaves the possibility that [obu_has_size_field] is 0
for OBUs other than the last OBU of a block (only a SHOULD). And the
requirement in the second sentence actually makes MUST out of the SHOULD
in the first sentence (making this part of the first sentence redundant)
and contradicts/voids the MAY part of the first sentence. In other
words, the two sentences should be merged to something like: "The `OBUs`
in the block must follow the `Low Overhead Bitstream Format` (in which
[obu_has_size_field] MUST be equal to one for every OBU) for every `OBU`
with the possible exception of the very last `OBU` in which
[obu_has_size_field] MAY be set to 0, in which case the `OBU` is assumed
to consist of the remainder of the block."

4. "ReferenceBlocks inside a BlockGroup MUST reference frames according
to the [ref_frame_idx] values of frame that is neither a KEYFRAME nor an
INTRA_ONLY_FRAME.": The problem with this sentence is that
[ref_frame_idx] needn't be present. It depends upon
[frame_refs_short_signaling] and [show_existing_frame]. If one uses a
Block inside a Blockgroup and if [show_exsting_frame] equals one one
should reference the block that contained the showable frame that is now
output (and that this should be the only `ReferenceBlock` written). In
case of [frame_refs_short_signaling] == 1 the obvious candidates for
`ReferenceBlocks` are the blocks containing the `last_frame_idx` and
`gold_frame_idx` that are explicitly signalled. If I am not mistaken,
then there are also other reference frames that are not explicitly
signalled, but computed. I don't know if we should really write a
`ReferenceBlock` entry for every reference as the current proposal seems
to imply. This would be quite a bit of overhead for no gain (and
furthermore, it would complicate muxers that would have to compute the
references that are not explicitly signalled in case that
[frame_refs_short_signaling] is 1). One `ReferenceBlock` would be enough
to distinguish keyframes from non-keyframes.
By the way: If a temporal unit contains multiple frames with references,
whose references should end up as `ReferenceBlocks`? Or may the muxer
choose some?

5. AV1 may use spatial scalability and/or temporal scalability. What do
we make of these? They are currently not forbidden if I am not mistaken,
but if e.g. the spatial dimensions of different layers disagree, the
`PixelWidth` and `PixelHeight` values can't be true for all layers.
Matroska seems to be missing some features here.

6. Depending on [frame_size_override_flag] there is even the possibility
that the size of the frames differs even without scalability (if I am
not mistaken). Should this be allowed?

7. Then there is another thing with keyframes and cues (for this point
it is always presumed that the relevant sequence header OBUs are
available regardless of whether this is done in-band or via CodecPrivate):
a) The proposal currently does not take into account that key frames
reset the decoder when they are output, not when they are decoded. A key
frame needn't be immediately output; if it is (i.e. [show_frame]
equaling 1), it is called a "key frame random access point" in section
7.6 of the standard and is the equivalent of an IDR frame in H.264.
Everything's fine here. But a key frame can also be declared a
showable_frame (but only if [show_frame] equals 0) and output later via
the show_existing_frame mechanism. This is similar to an open GOP in
other codecs (but in contrast to them, the block that contains the coded
keyframe doesn't have the same timestamp (pts) as the first frame that
can be output after a seek). The coded key frame with [show_frame] equal
to zero is called a delayed random access point and a key frame
dependent recovery point is a frame where a key frame with
[showable_frame] equal to 1 is output via the show_existing_frame
mechanism. If one starts decoding at the delayed random access point,
all the output frames up to but not including the key frame dependent
recovery point can depend both on the delayed random access point frame
and on other earlier frames so that these frames can't be correctly
decoded in general. But all the frames from the key frame dependent
recovery point onwards can be correctly decoded if one starts decoding
at the delayed random access point (because the decoder is reset after
displaying the key frame). If one starts decoding at the key frame
dependent recovery point, one doesn't have the key frame that should be
shown via the show_existing_frames mechanism at all, so that this frame
is simply not a real key frame.
b) But although a key frame dependent recovery point is not a "real" key
frame, it has the same [frame_type] as the frame that is output, i.e.
its [frame_type] is KEY_FRAME. According to our current proposal this
would mean that it should be treated as a keyframe in Matroska which is
obviously wrong.
c) Marking a delayed random access point as keyframe deviates from the
way that flag has been traditionally understood: If one starts decoding
at this point, one doesn't get the frame that should be output for the
temporal unit containing the delayed random access point. But I
nevertheless think that these are the right keyframes, because they are
the points at which random access has to begin when there aren't key
frame random access points available; this also means that one can split
the stream at this point and the second part will still play so that a
muxer like mkvmerge needn't be rewritten too much.
d) A consequence of this is that a `Blockgroup` containing a delayed
random access point mustn't contain a `ReferenceBlock` (although the
actual frame that is output for that temporal unit very likely uses
other reference frames than the key frame that is contained in the same
temporal unit).
e) Yes, this proposal means that it is impossible to tell from Matroska
alone (well, from the block structure that is; see f) for a way for
which one could put this information into the Cues) whether it is a key
frame random access point or a delayed random access point. One will
have to decode it (or parse deeper) to know.
f) This also leads to problems with seeking: If one simply added a
CuePoint for the keyframe (i.e. for the delayed random access point) and
the user wants to seek to a point between the delayed random access
point (inclusive) and the dependent recovery point (exclusive) and the
player used the cues to seek to the nearest keyframe in front of the
desired point, then decoding at the point referenced in the cues would
not yield the desired frame (it would be either corrupted or not output
at all). Therefore I think it is best to add a CuePoint for every key
frame random access point and every key frame dependent recovery point.
The CuePoint for the key frame random access point would be an ordinary
CuePoint as usual. But the CuePoint for the key frame dependent recovery
point wouldn't be (my favourite is iv) (and if I were allowed to play
God it would be i))):
i) A comprehensive way of doing it is this: The CueTime would be the
timestamp of the block containing the dependent recovery point; it would
include a CueTrackPositions for the video track we are talking about
that contains the right CueTrack, the CueClusterPosition containing the
position of the dependent recovery point block and a CueReference with
CueRefTime and CueRefCluster, both corresponding to the valus of the
delayed random access point. This proposal has several downsides: It
uses Cue elements that are deprecated in Matroska and not part of Webm.
So this would require a quite nontrivial change in both projects. (Btw:
If one does this, one should add a default value for `CueRefCluster`: It
should be the same as `CueClusterPosition` as both blocks that we are
talking about will probably end up in the same cluster anyway.)
ii) One uses the CueTime of the dependent recovery point, but the
position of the Cluster of the delayed access point (and
`CueRelativePosition` (if used) should also point to this block).
Pro: It only uses elements that are supported by both Matroska and Webm.
Furthermore, the specs only say that `CueClusterPosition` should point
to the cluster containing the "required block; they don't explicitly say
that said block needs to have the same timestamp as `CueTime`.
Contra: How does a demuxer know from which block onwards it should feed
the data to the decoder? It might use the `CueRelativePosition`, but
probably a lot of demuxers would simply read the cluster until they come
to the block with timestamp `CueTime` (i.e. they interpret the specs so
that the "required block" is the block with the timestamp `CueTime`) and
then they would either deliver this to the decoder or conclude that the
file is damaged (because the block they found is no keyframe).
iii) The last is the same as i) with the difference that `CueRefCluster`
is omitted. It is also incompatible with current Webm, but at least it
has the advantage that it doesn't use any currently deprecated elements
of Matroska. One could add a requirement that the delayed random access
point and the dependent recovery point need to be in the same cluster
and then omitting `CueRefCluster` is not a problem any more.
iv) And then there is the possibility of creating a normal CuePoint for
the dependent recovery point, writing the dependent recovery point as a
Block in a Blockgroup with exactly one ReferenceBlock which points to
the delayed access point block and let the demuxer seek backwards from
the dependent recovery point to the delayed access point.
Pro: Would only use things that are already supported by Matroska and
Webm. It would also not be AV1 specific. The demuxer doesn't need to
know anything about AV1, everything is signalled at the container level.
Contra: Demuxers would have to be adapted not to expect any more that
only keyframes are referenced in the cues. They would also have to be
adapted to actually make use of the value of `ReferenceBlock` and seek
backwards. This also implies more seeks, but this should be quite
limited when one puts both the delayed random access point and the
dependent recovery point in the same cluster -- hopefully the data is
still cached. (Maybe one should add a SHOULD clause that says that both
blocks should be in the same cluster.)
g) Of course there are two easy alternative solutions:
i) Restrict the type of AV1 that is allowed in Matroska even further so
that all key frames are of key frame random access type. (This could
exclude quite a lot of AV1 and therefore I recommend not doing so.)
ii) Create cues as usual, i.e. reference every delayed random access
point, and don't care about the fact that seeking will be partially
broken in this case.
h) It should be noted that exactly the same situation exists with
periodic intra refresh in general. There was a short discussion on the
Matroska developer mailing list in April 2011, but nothing came out of
it. Every solution I outlined here for AV1 is also applicable for this case.

> and possibly adding back the Sequence Header OBU).
8. I think we can relax the requirements on the existence of in-band
sequence header OBUs a bit: If a keyframe (i.e. a key random access
point or a delayed random access point, not a dependent recovery point)
uses the same sequence header OBU as in the CodecPrivate (including the
same operating_parameters_info), then the sequence header OBU needn't be
prepended to the block with the keyframe, because seeking already works
without it provided one always adds the sequence header OBU from the
CodecPrivate back in the bitstream on seeking. For example, consider the
following scenario:
One has an elementary stream that uses two different sequence header
OBUs A and B that only differ in the operating_parameters_info. The
first three keyframes use A, between the third and the B is contained in
a temporal unit between the third and the fourth keyframe. Between the
sixth and the seventh keyframe is a temporal unit containing sequence
header A again. Then a muxer that wants to put this elementary stream
into Matroska may put A in the CodecPrivate can strip the very first
occurence of A away; it must leave B inside the temporal unit that it
was in (so that a player that plays the file linearly is notified about
the change) and has to make sure that keyframes #4 to #6 contain
sequence header B (so that one has the correct sequence header when
seeking to said keyframes). It mustn't strip A between the sixth and the
seventh keyframe away (so that a player that plays the file linearly
notices the change of sequence header), but it needn't preprend
keyframes #7 and following with A.
That way one can save a few bytes.
This is consistent with an interpretation of the CodecPrivate as the
default extradata/header (but it is not necessarily a truly global
header). Before `CueCodecState` was deprecated it had a default value of
0 that mandated that one should look in the CodecPrivate for the
CodecState upon seking. So the only thing specific to AV1 in this case
is the "as-is" part; that one should reset the decoder to whatever
initialization information is contained in the CodecPrivate upon seeking
is nothing new.

9. Image that future AV1 encoders would find out that changing the
sequence header (with a change of the CVS) enables better compression.
What would we do in this case? Simply lift the restriction of one track
= one CVS even when this means that some players that can't cope with
changing coded video sequences won't know in advance whether they can
play the files at all? I'm asking because we don't include a version
field in the CodecPrivate (in contrast to how it is done in mp4 with avcC).

> Let me know what you think so we can settle this spec for good.
I think we are not even close to settling this for good.

- Andreas Rheinhardt

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 12 Jul 2018 11:54:29 +0200

Hi Andreas,

Thanks for your detailed feedback.

> actually depends on the value of `DisplayUnit`.

True, I will mention that.

> `CodecPrivate` should be changed.)

Indeed. The MP4 is a bit fuzzy on what is allowed. It's Sequence
Header and Metadata only if they apply to all samples. But anything
that is is valid to put before a sync point is OK. But in our case
it's better to be more strict. Remuxing from MP4 might require some
cleaning...

> to consist of the remainder of the block."

Indeed there's a contradiction here. If we use MUST (can't be must
lowercase) on [Low Overhead Bitstream Format] then the
[obu_has_size_field] MUST be 1.

On MP4 for the CodecPrivate the [obu_has_size_field] MUST be 1. But in
the Blocks it can be 0:

"Each OBU SHALL have the obu_has_size_field set to 1 except for the
last OBU in the sample, for which obu_has_size_field MAY be set to 0,
in which case it is assumed to fill the remaining of the sample"

I think we should mimic that. I'll rephrase it.

> references that are not explicitly signalled in case that

This is how `ReferenceBlock` is supposed to be used. So a muxer that
has no idea of any codec can cut a file and keep the relevant
references. So they all have to be there. It's one of the reasons
SimpleBlock was added, to simplify things a little (and reduce
overhead).

> whose references should end up as `ReferenceBlocks`? Or may the muxer

I think a Temporal Unit can only have one (visible frame). I don't
know if golden frames can have extra references. But the BlockGroup
should contain all frames needed to decode this frame, that includes
all the frames in the Block (even if not visible).

> Matroska seems to be missing some features here.

Our spec says that the Sequence Header OBU should be valid for all
frames. That can't be used for spatial scalability. We don't support
that mode for now.

It may technically possible to add different sizes in BlockAddition.

> not mistaken). Should this be allowed?

It's not restricted by our spec. But then it's up to the codec to
handle, not the container. It's used with a SWITCH frame. The MP4 spec
don't make any special case for that either.

(please add spacing between your paragraph, it's hard to read this big
block of text)
> would mean that it should be treated as a keyframe in Matroska which is

That's not how I understand it. Here's section 7.6.3:

"Informally, the requirement for decoder conformance is that decoding
can start at any key frame random access point or delayed random
access point."

And 7.6.2:

"delayed random access point is defined as being a frame:
• with frame_type equal to KEY_FRAME
• with show_frame equal to 0
• that is contained in a temporal unit that also contains a sequence header OBU"

So as long as we seek on frames of type KEY_FRAME we should be able to
seek. Wether it's a visible frame or not.

But because this is a bit loose in the 7.6.3 section they add this:

"To support the different modes of operation, a conformant decoder is
required to be able to decode bitstreams consisting of:
• a temporal unit containing a delayed random access point
• immediately followed by a temporal unit containing the associated
key frame dependent recovery point"

So the invisible KEY_FRAME should be immediately followed by the
recovery point data. So effectively it will work. There's also a note
that if it's not followed immediately then what is done with the
intermediate frames is implementation dependent. I don't think we need
to care too much.

> muxer like mkvmerge needn't be rewritten too much.

Yes, IMO this is the correct way.

> other reference frames than the key frame that is contained in the same

This won't happen if it's a proper random access point, ie it doesn't
need past frames to start decoding. If it's not then it's not a RAP
and then it can/should have ReferenceBlock.

> have to decode it (or parse deeper) to know.

No, this is independent of the codec. Also Cues can target a frames
that can't be seeked to directly but that's beside the point. S
SimpleBlock marked keyframe or BlockGroup with no ReferenceBlock can
be seeked to directly and that's why they equal Random Access Points
as defined in AV1 (and other codecs).

> point wouldn't be (my favourite is iv) (and if I were allowed to play

Cues are quite loose. It would be possible to do Cues only for frames
that are not delayed RAP and that's valid. IMO it's fine to reference
the delayed RAP. But they are RAP so it's legal to seek there.

The AV1 specs have this to say about this tricky case:

"Note:In practice, decoder implementations are expected to be able to
start decoding bitstreams from a delayed random access point when the
intermediate temporal units are still present. The decoder should
correctly produce all output frames from the next key frame or key
frame dependent recovery point onwards, while the preceding frames are
implementation defined. For example: a streaming decoder may choose to
decode and display all frames even when the reference frames are not
available (tolerating some errors in the output), a low latency
decoder may choose to decode and display all frames that are
guaranteed to be correct (e.g. an inter frame that only uses inter
prediction from the delayed random access point), a media player
decoder may choose to decode and display only frames starting from a
key frame or key frame dependent recovery point (guaranteeing smooth
playback once display starts)."

It's not up to the container to solve this.

> keyframes #7 and following with A.

That works but then we need to tell in the specs that following a
Sequence Header MUST not be stripped if the previous Sequence Header
was not bit identical to the one in CodecPrivate.

IMO it's valid to output A and then B before the rest of the data in
frame #4. That should be done if it's a keyframe. But if it's not a
keyframe the demuxer/decoder doesn't know it has to prepend the
CodecPrivate there. That could be a problem. Even though it doesn't
make much sense to change Sequence Header data before a non keyframe
(RAP).

So maybe your proposal should be added. We'll gain a bit of weight but
it's safer.

> field in the CodecPrivate (in contrast to how it is done in mp4 with avcC).

Another CodecID should be used if the restrictions/format is defined.
It's safer than reusing one with different data and hoping the system
using the old one paid attention to the version bit that was always
the same anyway.

> I think we are not even close to settling this for good.

\o/

>
> - Andreas Rheinhardt
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 12 Jul 2018 11:58:17 +0200

New update after the points from Andreas were addressed:

---

From: Andreas Rheinhardt <[email-masked]>
Date: Thu, 12 Jul 2018 18:20:00 +0000

Hello,

> True, I will mention that.
The "Notes" still only cover the case of `DisplayUnit` indicating pixels.

> I think we should mimic that. I'll rephrase it.

The current version is:
"The OBUs in the Block follow the [Low Overhead Bitstream Format
syntax]. They SHOULD have the [obu_has_size_field] set to 1 except for
the last OBU in the sample, for which [obu_has_size_field] MAY be set to
0, in which case it is assumed to fill the remaining of the sample."

If one interprets the first sentence as meaning "The OBUs in the Block
MUST follow the [Low Overhead Bitstream Format syntax]", then given that
this syntax mandates [obu_has_size_field] to be equal to 1 the first
part of the second sentence is redundant (given that MUST is stronger
than SHOULD) and the second part is again in contradiction to/voided by
the first sentence because the first sentence doesn't allow
"[obu_has_size_field]" set to zero at all.
If one interprets the first sentence as not conveying a MUST, then it is
allowed (albeit strongly discouraged) to use [obe_has_size_field] equal
to 0 for an OBU that is not the last OBU in the sample. This is not what
we want, isn't it? How about:
"The OBUs in the `Block` MUST follow the [Low Overhead Bitstream Format
syntax] with the possible exception of the last OBU of a `Block` for
which [obu_has_size_field] MAY be set to 0, in which case it is assumed
to fill the remainder of the `Block`."

> SimpleBlock was added, to simplify things a little (and reduce
Actually a muxer can cut a file if it just knows the keyframes, the
decoding order and the display order. It doesn't need to have complete
information about reference frames. After all, one can cut files that
exclusively use `SimpleBlocks`.
(If one wants to cut the beginning away, one can cut according to the
keyframes; and at the end one can cut every block with timestamp >t_0
away if there is no block with timestamp >t_0 that precedes a block with
timestamp <=t_0 in coding/storage order.)

> frames. That can't be used for spatial scalability. We don't support
Then this should be explicitly stated in the codec mapping. And I also
fail to see why the fact that the Sequence Header OBU should be valid
for all frames should be incompatible with spatial scalability (after
all, in my reading of the spec the various share the same Sequence
Header OBU).

> seek. Wether it's a visible frame or not.
a) According to the Uncompressed Header Syntax, the [frame_type] of a
frame that is output via the [show_existing_frame] mechanism is the
[frame_type] of the [showable] frame that is output. This implies that a
key frame dependent recovery point (KFDRP from now on) has [frame_type]
KEY_FRAME. And this means that a KFDRP is a keyframe according to the
codec mapping (actually it is only heavily implied to be a keyframe,
because the current codec mapping does not require to label any frame a
keyframe).

b) The first passage you cited says that decoding can start at key frame
random access points (KFRAP from now on) or delayed random access points
(DRAP from now on). It does not say that one can seek to frames of type
KFDRP. And these frames are of type KEY_FRAME, too.

> intermediate frames is implementation dependent. I don't think we need
I don't think we should assume that the DRAP block is immediately
followed by a KFDRP (and I don't see how the assumption that there are
no temporal units between DRAP and KFDRP simplifies anything for us
container guys). That's just the only place where being able to decode
is absolutely required. But just read the note (that you actually quote
below) and you will see that they expect more:

"In practice, decoder implementations are expected to be able to start
decoding bitstreams from a delayed random access point when the
intermediate temporal units are still present. The decoder should
correctly produce all output frames from the next key frame or key frame
dependent recovery point onwards, while the preceding frames are
implementation defined."

So it's reasonable to assume that there will be temporal units between
DRAP and KFDRP.

And I think we should care about such stuff and in particular make sure
that seeking works well (because when there are non-KFRAP keyframes, AV1
is very much like intra decoder refresh when it comes to seeking and
intra decoder refresh seeking currently doesn't work well because the
cues only contain the timestamps of the frames where one should start
decoding in order to output frames, but not the timestamp of the first
frame that is undamaged after one has started decoding; this is a
problem when one wants to seek to one of the frames inbetween these two
frames).
> and then it can/should have ReferenceBlock.
The DRAP frame itself (being coded intra) won't reference other frames,
but the other frames (including the shown frame) in the temporal unit
containing said DRAP will of course reference earlier frames (the H.264
equivalent of this scenario is an open gop and the other frames in this
case are B-frames shared between two GOPs (i.e. the B-frames that
precede the open GOP's keyframe in display order and follow it in coding
order) -- and they of course reference both the keyframe and earlier
frames, that is after all the whole point of having an open GOP). So it
does happen although it is a random access point; it does happen,
because it is just a delayed random access point.

> as defined in AV1 (and other codecs).
Given that your reply began with "No" I presume that you believe that
you refuted me, but I don't see that you have. After all, how can one
tell from the Matroska layer alone whether a `Block` is a KFRAP or a DRAP?

(Of course I know that Cues can also target frames that can't be seeked
to directly -- after all, my proposal below is about adding Cues for
such frames.)

> the delayed RAP. But they are RAP so it's legal to seek there.
Cues are loose, but we can (and I think we should) add a SHOULD clause
that describes what cues muxers are expected to produce. And referencing
the delayed RAP brings the problem that the cues only contain where to
start decoding, but not from when on the output is undamaged/valid.
> It's not up to the container to solve this.
It is not up to the container to decide whether the possibly damaged
frames between the DRAP and the KFDRP should be displayed, but it very
much is up to the container to make it as easy as possible to find the
data needed for random access. And so the cues should answer the
question "I want to play from point t_0 on. Where do I have to seek to?"

> make much sense to change Sequence Header data before a non keyframe
Why should the demuxer ever want to prepend the CodecPrivate in front of
a non-keyframe? This doesn't make sense to me; after all, one should
only seek to keyframes and only add the CodecPrivate in front of a
keyframe if one has just seeked to said keyframe, not if one just
encounters a keyframe (after the seek one just uses whatever Sequence
Header one encounters in-band (if any)).

> So maybe your proposal should be added. We'll gain a bit of weight but
My wording would be:
"Upon seeking to a keyframe the player/demuxer MUST prepend the `Block`
with the `Sequence Header OBU` contained in the `CodecPrivate`.
A muxer MUST make sure that the correct `Sequence Header OBU` is in
force both during linear access and also after seeking to a keyframe
`Block`. So in particular a keyframe `Block` where a `Sequence Header
OBU` that is not bit-identical to the one in the `CodecPrivate` is in
force for the decoding of the first frame contained in said `Block` MUST
contain the `Sequence Header OBU` that is in force for the decoding of
the first frame contained in said `Block` in front of the first frame
contained in said `Block`."

One could also relax this a bit and only make the correct linear access
a MUST and the rest a SHOULD. This might be useful for applications
where seeking isn't desired (although it really should be included even
for those scenarios to support resuming playback after a transmission
error).

And maybe one should add a clause like "Seeking to non-keyframes is
undefined and not recommended."

And if one wants to one can add a clause recommending to strip out
unnecessary `Sequence Header OBUs`

>> I think we are not even close to settling this for good.
I honestly don't know what this means.

- Andreas Rheinhardt

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 13 Jul 2018 09:24:18 +0200

Hi,

> The "Notes" still only cover the case of `DisplayUnit` indicating pixels.

If they are not in pixels then the values should not be taken from the
codec anyway. The general Matroska rules apply. IMO we are not going
to repeat here the way DisplayWidth/Height has to be interpreted.

> to fill the remainder of the `Block`."

MUST is wrong IMO if you add an exception. Then it should be a SHOULD
and explain the cases where it shouldn't. I left it out of the first
sentence on purpose because the "normative" SHOULD is on the next
sentence and the one that should apply.

> exclusively use `SimpleBlocks`.

I'm not sure it's true for modern codec where the referenced frame may
be older than the previous keyframes. Also the ReferenceBlock allows
picking only the frames necessary to render that particular frame,
regardless of the internals of the codec.

> all, in my reading of the spec the various share the same Sequence

I didn't fully understand how the spatial scalability works. If the
same Sequence Header OBU supports it then we can support it. After all
the PixelWidth/Height use the "maximum" width/height. But internally
it may be less.

> because the current codec mapping does not require to label any frame a

In "6.8.2 Uncompressed header semantics" it says:

"If obu_type is equal to OBU_FRAME, it is a requirement of bitstream
conformance that show_existing_frame is equal to 0."

So the KFDRP which has show_existing_frame=1 is not an OBU_FRAME. I
think it's in an OBU of type OBU_FRAME_HEADER. Also a `Block` is one
Temporal Unit which has at least one shown OBU frame (exactly 1 if
spatial scalability is not used). It's not entirely clear if an OBU of
type OBU_FRAME_HEADER is a frame or not. We could add the requirement
for our keyframe definition that the OBU with the keyframe flag must
be of type OBU_FRAME. Currently we say:

A `SimpleBlock` MUST only be marked as a Keyframe if the first `Frame
Header OBU` in the `Block` has a __[frame_type]__ of `KEY_FRAME`

This is actually the opposite of what we want... It should not be a
`Frame Header OBU` but a `Frame OBU`. I think the Frame Header OBU is
a frame placeholder for the show_existing_frame flag. It's missing the
Tile Group that contains the actual data. I'll fix that in my next
push.

> KFDRP. And these frames are of type KEY_FRAME, too.

I think this is exactly what Random Access Point means. You can seek
there and start decoding safely. And that's exactly when a Block must
be a keyframe in Matroska and not any other time. And for that I
prefer to use Chapter 7.6.

> tell from the Matroska layer alone whether a `Block` is a KFRAP or a DRAP?

A RAP is exactly what we need, no matter the time. If there are
glitches because it's a DRAP it's fine according to the specs. But
that's were proper seek should be done.

> question "I want to play from point t_0 on. Where do I have to seek to?"

In Matroska to the CuePoint with no CueReference with the closest time
before point_t_0.

In AV1 to the RAP with the closest time before point_t_0.

> Header one encounters in-band (if any)).

Because they can be omitted. But given we changed the way we omit it
so that changes always go back to the "reference" one it's not needed
anymore.

The AV1 specs say a RAP must contains a Sequence Header OBU. In our
case it may be omitted but it's up to the codec/demuxer to add it back
in the stream when seeking.

> for those scenarios to support resuming playback after a transmission

OK, I'll try to add something for seeking.

> I honestly don't know what this means.

Raised arms.

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 13 Jul 2018 09:26:35 +0200

> OK, I'll try to add something for seeking.

Actually we already have it in the Segment Restrictions:

Given a `Sequence Header OBU` can be omitted from a `Block` if
__[decoder_model_info_present_flag]__ is 0 and it is bit identical to
the one found in `CodecPrivate`, when seeking to a keyframe, that
omitted `Sequence Header OBU` MUST be added back to the bitstream for
compliance with the Random Access Decoding section of the [AV1
Specifiations](#av1-specifications).

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sat, 14 Jul 2018 13:33:59 -0700

> frames between the DRAP and the KFDRP should be displayed, but it very

I would argue that this does have implications for file trimming. The 
Matroska mapping is allowed to place more requirements on an 
implementation than the AV1 specification. So there is a choice to make. 
Just considering the default (no further restrictions) or requiring one 
of the three example behaviors in the spec has the following implications:

If you choose to leave the handling of frames between the DRAP and the 
KFDRP implementation-defined, then someone cutting a stream has no idea 
how they will be handled. So the only safe thing to do is to strip them 
all. This adds some minor complexity burden to the remuxer, but 
guarantees that all such useless data will get stripped by a correct 
implementation.

Choosing to require decoding and display even when the reference frames 
are not available would produce a similar result (since packet loss 
concealment is not normative).

Choosing to require decoding and display of all frames that are 
guaranteed to be correct might allow a remuxer to cut at slightly more 
places without including an entire extra keyframe interval, but given 
that such frames are not required to be consecutive, this would get 
pretty complicated to implement (but at least it is optional: you can 
always fall back to stripping all such frames as above).

Choosing to require that only frames starting from the KFDRP be decoded 
and displayed means that a remuxer does not need to strip such frames to 
guarantee consistent output from all players, allowing it to be slightly 
simpler, at the cost of allowing it to include some data that is useless.

None of this helps you for seeking, because you don't have a choice 
about which frames to include, so one must rely on the player not to 
display garbage (in the case that references are missing) or stuttering 
output (in the case that only some frames between the DRAP and KFDRP are 
correctly decodable).

---

From: Andreas Rheinhardt <[email-masked]>
Date: Sat, 14 Jul 2018 21:55:00 +0000

Hello,

what I have to say about the "seeking with delayed random access points"
topic will be said in a separate email as a reply to your other email.
Here is the rest:

1. "The [timing_info_present_flag] of the Sequence Header OBU SHOULD be
0. Even when it is 1 the presentation time of the Frame Header OBUs in
Blocks should be discarded. In other words, only the timestamps given by
the Matroska container MUST be used."

a) This clause should not be in the section about the `CodecPrivate` as
it pertains to all `Sequence Header OBUs`, not just the one in the
`CodecPrivate`.

b) I don't see a reason why the bitstream should be cleaned of these
values. This isn't done for H.264 or other codecs either.

c) If you want to keep this recommendation in, then we should merge it
with the recommendation to discard [temporal_point_info] (after all,
setting [timing_info_present_flag] to 0 automatically does the same with
[decoder_model_info_present_flag] and then [[temporal_point_info]]
mustn't be present anyway) that uses [frame_presentation_time]. It's
probably good to adapt the mp4 version:
"The presentation times of AV1 samples are given by the Matroska
container. The [timing_info_present_flag] in the `Sequence Header OBU`
(in the `CodecPrivate` or in the bitstream) SHOULD be set to 0. If set
to 1, the [timing_info] structure of the `Sequence Header OBU`, the
[frame_presentation_time] and [buffer_removal_time] fields of the `Frame
Header OBUs`, if present, SHALL be ignored for the purpose of timed
processing of the Matroska file."

2.

> sentence and the one that should apply.

My interpretation of my proposal is: "If the exception doesn't apply,
then it is a MUST (a real MUST), otherwise the exception (with its MAY)
applies." I thought this is what is really wanted: After all, there used
to be the sentence "The OBUs in the `Block` MUST follow the __[Low
Overhead Bitstream Format syntax]__." And the commit message of the
commit that changed this part to the way it currently is reads: "av1:
the low overhead format must be used in Blocks except for the last OBU"

Anyway, given that my earlier proposal can apparently be misunderstood
in ways I didn't imagine let me rephrase it:
"If an OBU is not the last OBU in a `Block`, it MUST follow the [Low
Overhead Bitstream Format syntax] (i.e. it MUST have
[obu_has_size_field] set to 1); the last OBU in a `Block` MAY have
[obu_has_size_field] set to 0, in which case it is assumed to fill the
remainder of the `Block`."

I can't think of any ambiguity in the above wording.

3.

> If by cut a file one means to specify an interval of the (output) movie
that should be preserved, then I have to correct myself: The criteria I
mentioned only work if the keyframes have the property that any block
with a timestamp bigger than the timestamp of the keyframe can be
correctly decoded if one starts decoding from the keyframe on. If not,
one also needs the information from which frame on the output is ok. One
doesn't need every reference for every frame for that, one could also
convey this information with one of the ways I proposed.

4.

> the PixelWidth/Height use the "maximum" width/height. But internally
It uses the same `Sequence Header OBU`. But it is not confined to
something internal to the codec. Different spatial layers can have
different output resolutions; in fact even without this the frame
dimensions may change from frame to frame. And if I am not mistaken,
then this is not something internal to the codec: The output pictures
(that have to be scaled by the renderer) can have varying dimensions,
too. Currently the segment restrictions contain the following clause:

"Matroska doesn't allow dynamic changes within a codec for the whole
Segment. The parameters that should not change for a video Track are the
dimensions and the CodecPrivate."

I take this as meaning that all output frames must have the same
dimension and I think (this point is not clearly stated) it is meant to
be that said output dimensions coincide with what must be put into
`DisplayWidth` and `DisplayHeight` so that one actually has the added
restriction for AV1 in Matroska that every output frame has the same
width as [max_frame_width_minus_1]+1 (similar for height). But what I'd
like to know is how much of AV1 will probably be excluded from Matroska
by these requirements? Could we contact some of the codec designers and
ask them about their opinions on this? (Honestly, "we" probably means
"you" as you are the chairman of Matroska.) And maybe other things where
we might have questions for them. (We should collect the questions first.)

5.

> Specifiations](#av1-specifications).

No, we haven't. But before I come to this here are two more points that
I would like to mention first:

a) [decoder_model_info_present_flag] controls more than just whether
[operating_parameters_info] is present: It also influences the presence
of [temporal_point_info] which is AV1's way of storing the timestamps
(possibly VFR) of the video. The real condition whether the `Sequence
Header OBUs` of a CVS can change is whether
[decoder_model_present_for_this_op[i]] is 1 for an operating point. But
in my proposal it doesn't matter anymore anyway, as the way of
specifying when a `Sequence Header OBU` has to be present is entirely
independent of the one track = one CVS restriction (it would work just
as well if this restriction were dropped).

b) The current proposal mandates that one track can only consist of one
(subset of a) CVS in a suboptimal manner (if it mandates it at all): It
says:

"Matroska doesn't allow dynamic changes within a codec for the whole
Segment. The parameters that should not change for a video Track are the
dimensions and the CodecPrivate."

The `CodecPrivate` can't change in a valid Matroska track anyway,
because it can only have one. Therefore this requirement is empty and
the only requirement of one track = one CVS is in the next sentence:

"The first Sequence Header OBU of a CVS is stored in the CodecPrivate of
a Track. So this AV1 Track has the same requirements as the CVS."

Here the "so" is currently a non-sequitur.
Better wording:

"Matroska doesn't allow dynamic changes within a `Track` for the whole
`Segment`. Therefore the `Sequence Header OBUs` of the `Track` (both the
one in the `CodecPrivate` as well as the in-band `Sequence Header OBUs`)
MUST adhere to the restriction that the `Sequence Header OBUs` of a
`CVS` must fulfill: Their content MUST be bit-identical each time a
`Sequence Header OBU` appears except for the contents of
[operating_parameters_info]. Furthermore the dimensions of all output
frames MUST be equal."

c) Here is what the proposal currently has to say about `Sequence Header
OBUs`:
"Sequence Header OBUs SHOULD be omitted when they are bit-identical to
the one found in CodecPrivate and [decoder_model_info_present_flag] is 0
and the previous Sequence Header OBUs in the bistream was also
bit-identical to the one found in CodecPrivate. They can be kept when
encryption constraints require it."

"The first Sequence Header OBU of a CVS is stored in the CodecPrivate of
a Track. So this AV1 Track has the same requirements as the CVS."

"If the [decoder_model_info_present_flag] of this Sequence Header OBU is
set to 1 then each keyframe Block MUST contain a Sequence Header OBU
before the Frame Header OBUs."

"Given a Sequence Header OBU can be omitted from a Block if
[decoder_model_info_present_flag] is 0 and it is bit identical to the
one found in CodecPrivate, when seeking to a keyframe, that omitted
Sequence Header OBU MUST be added back to the bitstream for compliance
with the Random Access Decoding section of the AV1 Specifiations."

"A SimpleBlock MUST be marked as a Keyframe only if the first Frame OBU
in the Block has a [frame_type] of KEY_FRAME and the SimpleBlock
contains a Sequence Header OBU or if the Sequence Header OBU is
correctly omitted (see above)."

"A Block inside a BlockGroup MUST use ReferenceBlock elements if the
first Frame OBU in the Block has a [frame_type] other than KEY_FRAME or
the Block doesn't contain a Sequence Header OBU when it should not be
omitted."

Clause 1 and clause 4 only apply when [decoder_model_info_present_flag]
is 0. In this case (by our requirements of each track only containing a
single CVS) all `Sequence Header OBUs` must be bit-identical; in
particular, the part of clause 1 dealing with the previous `Sequence
Header OBU` is unnecessary because it is automatically fulfilled.

Clause 3 directly contradicts your claim that the current proposal
already allows to strip away some of the `Sequence Header OBUs` when
they are bit-identical to the one in the `CodecPrivate` and redundant
for linear access.

Clause 2 is actually an unnecessary restriction. It might be that the
very first `Sequence Header OBU` is rare and that one could save more
bytes when putting a different one in the `CodecPrivate`. In any case I
fail to see why the first `Sequence Header OBU` should be treated
specially in the standard; that it allows for simpler muxers will of
course make most muxer put the first `Sequence Header OBU` in the
`CodecPrivate`, no doubt about that. But if one requires this, then
splitting AV1 tracks in Matroska will be more complicated: One has to
change the `CodecPrivate` to what is the `Sequence Header OBU` used by
the first frame.

Even ignoring this I regard the whole requirements as confusing. The
requirements should not be stated in terms of
[decoder_model_info_present_flag] (or
[decoder_model_present_for_this_op[i]]); instead one should state them
as a corollary of the general principle that the necessary codec
initialisation data/extradata must be available to the decoder during
linear access as well as after random access (to a keyframe) and not
hide this fact behind some conditions involving the aforementioned
flags. It's much more intelligible this way. And it encourages muxers
not to hard-code to treat the cases of [decoder_model_info_present_flag]
being 0 or 1 (or [decoder_model_present_for_this_op[i]] being 0 for all
applicable i or not) separately, but to use a system that works even if
multiple CVS were allowed in the future.

d) Here is a proposal for said section in case we make it a MUST that
keyframes must have the correct `Sequence Header OBU` (whether because
it is there or because it is prepended from the `CodecPrivate`):

"Upon seeking to a random access point (RAP) or starting playback from
the beginning the player/demuxer MUST prepend the `Block` where decoding
starts with the `Sequence Header OBU` contained in the `CodecPrivate`.
Afterwards playback proceeds normally including consuming any `Sequence
Header OBUs` that are found in the bitstream.
Seeking to a non-RAP is undefined and not recommended.

A muxer MUST make sure that when using a conformant demuxer/player the
correct `Sequence Header OBU` is active both during linear access and
also after seeking to a random access `Block`. In particular, if a
`Sequence Header OBU` that differs from the `Sequence Header OBU` in the
`CodecPrivate` is active during consuming of the first `Frame Header
OBU` of a random access point sample, then said sample MUST contain said
`Sequence Header OBU` in front of the first `Frame Header OBU`.

The muxer MAY omit (strip away and discard) `Sequence Header OBUs`
provided the above criteria are still fulfilled afterwards.

Note: A `Sequence Header OBUs` that fulfills any of these criteria can
be omitted without changing compliance to the above criteria:

If, after potentially stripping away all `Temporal Delimiter OBUs` there
are more than one `Sequence Header OBUs` immediately after each other,
then all but the last of these `Sequence Header OBUs` can be omitted.

A `Sequence Header OBU` that differs from the `Sequence Header OBU` in
the `CodecPrivate` can be omitted if there was a preceding `Sequence
Header OBU` in the bitstream, if the preceding `Sequence Header OBU` was
bit-identical to the current `Sequence Header OBU` and if the current
`Sequence Header OBU` is not the first OBU after a `Temporal Delimiter
OBU` that starts a temporal unit whose corresponding `Block` is a random
access point.

A `Sequence Header OBU` that is bit-identical to the `Sequence Header
OBU` in the `CodecPrivate` can be omitted if there was no preceding
`Sequence Header OBU` or if there was a preceding `Sequence Header OBU`
and the previous `Sequence Header OBU` was bit-identical to the current
`Sequence Header OBU`."

If this is adopted, clauses 2-4 from c) above should be omitted and
clauses 5 and 6 can be adapted, too.

That's it from me today.

- Andreas Rheinhardt

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sat, 14 Jul 2018 18:19:11 -0700

> (in the `CodecPrivate` or in the bitstream) SHOULD be set to 0. If set

I'll ask the usual question: when is it reasonable to violate this SHOULD?

> ask them about their opinions on this? (Honestly, "we" probably means

Hi, I am one of the codec designers. I think the expectation is that 
DisplayWidth and DisplayHeight will remain constant for a coded video 
sequence (and these should match [max_frame_width_minus_1] + 1 and 
[max_frame_height_minus_1] + 1). If the output dimensions of a frame 
change, it should be scaled to the display resolution before being 
displayed. Spatial scalability depends on this (at least if you ever 
intend to drop some of the layers, and if you don't then there is no 
point to using scalability). Even without spatial scalability, it is 
sometimes desirable to reduce the coded resolution to maintain quality 
while fitting within the instantaneous channel bandwidth. That mostly 
applies to live/low-latency streams, but I think an important use case 
for Matroska is to be able to capture such streams for long-term storage 
(for example, via WebRTC and the MediaStream Recording API in web browsers).

> [operating_parameters_info]. Furthermore the dimensions of all output

As such, I disagree with this part (as an individual).

> correctly omitted (see above)."

This is mathematically correct (assuming you really mean "only if" and 
not "if and only if), but I think it could be easily misread. Perhaps 
"MUST NOT... unless..."?

> Seeking to a non-RAP is undefined and not recommended.

RECOMMENDED is an RFC 2119 keyword (but "NOT RECOMMENDED" is not 
explicitly listed as one). You might want to rephrase to avoid confusion.

---

From: Andreas Rheinhardt <[email-masked]>
Date: Sun, 15 Jul 2018 17:27:00 +0000

Hello,

> I'll ask the usual question: when is it reasonable to violate this SHOULD?
every timestamp in Matroska is a multiple of 1 ns (this number is
hard-coded) so that NTSC timings can't be exactly represented on the
container level: E.g. the default duration field for 24/1.001fps content
would indicate 41708333ns, although it is 41708333.33333333...ns. So by
preserving the bitstream information it is possible to increase the
certainity that 41708333ns actually means 24/1.001 fps.

> (for example, via WebRTC and the MediaStream Recording API in web
> browsers).>
a) I don't like restrictions on the content that can be put into
Matroska either. The reason why I put it in is because this is (in my
understanding) currently an absolute requirement of Matroska
(`PixelWidth` and `PixelHeight` are mandatory and valid for every
frame). Given that decoders will have to support these varying output
dimensions anyway (it is a mandatory part of AV1, isn't it?), it would
be possible to drop this requirement, but for this the specifications of
both Matroska and Webm would have to be changed. I'm not sure if
everyone is ok with this.

b) Just to be sure:  "Normally" the size of an output frame is given by
[max_frame_height_minus_1]+1 and [max_frame_width_minus_1]+1, although
part 7.18 that deals with the output process says that the width is
given by `UpscaledWidth`. The reason for this is that for intra frames,
key frames and some inter frames the initial `FrameWidth` (directly
inferred from [max_frame_width_minus_1] or [frame_width_minus_1]) is
saved to `UpscaledWidth` and then `FrameWidth` gets overwritten with a
smaller value (that is the value that is used during much of the
decoding process, but only internally, so that we can ignore it). Is
that correct? (I have to admit I thought for a time that the output
dimensions also depend on the superres parameters and that they can be
even bigger than [max_frame_height/width_minus_1]+1.)

> not "if and only if), but I think it could be easily misread. Perhaps
I agree that this should be changed and your proposal is certainly an
improvement. However this is not the only thing in this definition that
needs to be looked at (see 4. below).

> explicitly listed as one). You might want to rephrase to avoid confusion.
Ok: "Seeking to a non-RAP is undefined and discouraged." (We also need
to explicitly add what a RAP is.)

And because you are one of the AV1 designers, here are a few questions:

1. Section 7.5 includes the following:
"If scalability is not being used (OperatingPointIdc equal to 0), then
all frames are part of the operating point. The following constraints
must hold:

    The first frame header must have frame_type equal to KEY_FRAME and
show_frame equal to 1.

    Each temporal unit must have exactly one shown frame.

If scalability is being used (OperatingPointIdc not equal to 0), then
only a subset of frames are part of the operating point. For each
operating point, the following constraints must hold:

    The first frame header that will be decoded must have frame_type
equal to KEY_FRAME and show_frame equal to 1.

    Every layer that has a coded frame in a temporal unit must have
exactly one shown frame that is the last frame of that layer in the
temporal unit."

I am wondering whether the the last two requirements only apply if
scalability is being used (OperatingPointIdc not equal to 0) or in any
case. The actual reason why I am asking myself this is that in case that
scalability is not being used there is no restriction that a shown frame
is the last frame in that temporal unit. I don't see a reason why there
should be another frame afterwards in the same temporal unit, but is it
actually allowed?

2. Would it be legal to code a frame with [show_frame] 0,
[showable_frame] 1 followed (perhaps immediately) by a frame header in
the same temporal unit that has [show_existing_frame] set to 1 and
outputs the frame just coded? If yes, is there any reason to ever use
it? I don't see any. According to the definition, this temporal unit
contains both delayed random access point and a key frame dependent
access point and according to 7.6.3 a decoder is not required to be able
to handle such a situation for random access. But obviously it is as
good as a key frame random access point.

3. Would it be legal to have a temporal unit that contains a frame that
is not shown (but might be declared as showable) followed by a keyframe
with [show_frame] equal to 1? I'm asking because both the current
proposal as well as the mp4/ISOBMFF definition of sync sample require
the keyframe to be the first frame in the temporal unit.

4. You have only criticized the "MUST only" wording of the definition of
keyframe simpleblocks. You said nothing about the requirement that it is
the first frame that needs to be of type KEY_FRAME. The standard doesn't
require this and I fail to see why this should be needed. Moreover, in
contrast to 2. and 3. there seems to be a usecase for this: Think about
a situation where all reference frames available before/at the start of
the temporal unit that contains a delayed RAP constitute a very good set
of reference frames for a frame that is to be shown between the output
frame of the temporal unit that contains the delayed RAP and the output
frame of the temporal unit that contains the dependent keyframe recovery
point, but where the combination of the dependent RAP frame + all but
one of the reference frames available at the start of the temporal unit
that contains the delayed RAP frame constitute a worse set of reference
frames (regardless of which of the initial reference frames gets
overwritten by the delayed RAP frame). Then it may make sense to first
code this frame as a showable frame using all the reference frames and
then code the delayed RAP frame followed by the frame that is actually
output for the temporal unit containing the delayed RAP. Is my usecase
sound and should the "first"-requirement be dropped?

5. What do you think of the restriction of a Matroska track to a (subset
of) a CVS?

- Andreas Rheinhardt

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 16 Jul 2018 08:11:37 -0700

> certainity that 41708333ns actually means 24/1.001 fps.

This seems like a good enough reason that it should probably be 
explicitly described in the draft (especially if you want any of that to 
work in practice). If that's the only reason then okay, but if there are 
a lot of other reasons, then SHOULD may be too strong.

As an aside the timestamp situation still makes me sad.

> everyone is ok with this.

I don't think this is too burdensome. A system that requires a constant 
pixel size for the decoded frame can put an upscaler between the 
underlying AV1 decoder implementation and the input to that system, and 
pretend that rescaling is part of the AV1 decoder black box 
(implementing an upscaler is vastly simpler than implementing a whole 
video codec). The fact that there are different dimensions in the AV1 
frame header shouldn't be something it has to worry about.

> that correct? (I have to admit I thought for a time that the output

I believe that is correct. The UpscaledWidth allows the codec to 
partially do what I was describing for real-time/low-latency streams 
internally, but still run some of the loop filters at the upscaled 
resolution, but due to hardware decoder complexity concerns it can only 
rescale in the horizontal direction, so there's a limited operating 
space where that is sufficient.

> should be another frame afterwards in the same temporal unit, but is it

I don't know. I agree it's ambiguous as written. I'll ask someone and 
get back to you.

> it? I don't see any. According to the definition, this temporal unit

I think it is legal. I don't know why you would ever do it, but it 
wouldn't cause any great harm. Does it complicate anything to allow it 
any more than DRAPs complicate things already?

> the keyframe to be the first frame in the temporal unit.

I think this is legal. I also think this would be entirely wasteful. The 
key frame would reset all of the reference buffers, so the non-shown 
frame would never contribute anything to a shown frame.

> the first frame that needs to be of type KEY_FRAME. The standard doesn't

Please assume that if I don't comment on something, it may just be 
because I have not thought it all of the way through (you might not be 
wrong to assume that about things I do comment on, also).

> sound and should the "first"-requirement be dropped?

I agree that, because a DRAP does not immediately reset all of the 
reference buffers, it may not be wasteful to precede it by a non-shown 
frame, and because you will (presumably) consume at least one reference 
buffer slot to store the DRAP, you cannot in general re-order them.

It gets more complicated when you consider spatial scalability. It is 
possible for you to have a RAP for a frame other than the base layer, so 
you may actually have one or more shown frames in the same temporal unit 
(for different spatial layers) before a (delayed or even non-delayed) 
RAP, and again be unable to re-order them, and now whether you can start 
display at that frame or a following frame can depend on which operating 
point you are trying to use.

In particular, the problem is that the AV1 specification defines a RAP 
in terms of a *frame*, and not in terms of a temporal unit. Because 
Matroska defines its blocks as containing a temporal unit, and RAPs in 
terms of a block, you will have to define the details of when a temporal 
unit should be considered a RAP (or DRAP, KFDRP, etc.) based on the 
frames contained inside of it.

I think it may have been a mistake to do things this way in the AV1 
specification, since the idea was always for temporal units to be the 
unit of muxing in any container, but I expect it was done this way out 
of expediency to avoid having to think through all of these cases. We 
have to think through them now.

As I said in another mail, I think it may be okay to impose additional 
restrictions on top of the AV1 specification, especially if it makes 
everyone else's lives easier. This may be one such case. While the 
situations described above may confer some marginal advantage, and I can 
imagine an encoder that could take advantage of them, I don't think 
there would be a big loss by forbidding them.

I do think it would be a very good idea to have the *same* restrictions 
(if any) for Matroska and MP4. If something is technically allowed by 
the AV1 specification but you can't actually put that into any 
container, then in practical terms it might as well not be allowed. But 
if we allow some things in Matroska but don't allow them in MP4, or vice 
versa, then you just have an interoperability mess.

> 5. What do you think of the restriction of a Matroska track to a (subset

If you look back to my mail on June 30th, I thought I was the one who 
suggested it in the first place.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 16 Jul 2018 11:37:32 -0700

> I don't know. I agree it's ambiguous as written. I'll ask someone and 

I asked the person whom I think wrote that text, Peter de Rivaz, who said:

"I believe this is an oversight.  I believe the intention is that the 
shown frame should be the last frame in all cases."

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 18 Jul 2018 11:41:32 +0200

Hi Tim,

2018-07-16 17:11 GMT+02:00 Timothy B. Terriberry <[email-masked]>:
> This seems like a good enough reason that it should probably be explicitly
> described in the draft (especially if you want any of that to work in
> practice). If that's the only reason then okay, but if there are a lot of
> other reasons, then SHOULD may be too strong.

In any cases the container time will apply. It's fine to "resample" a
24fps to display it in 1fps (slow motion mode) without changing the
coded data. So it doesn't really matter how the source was encoded.
(I'm not sure decoders do temporal adjustments, at least when
interlacing is not involved).

> As an aside the timestamp situation still makes me sad.

All of us :( We still haven't come up with a good backward compatible
solution. For future Matroska versions it's certainly doable.

> I think it is legal. I don't know why you would ever do it, but it wouldn't
> cause any great harm. Does it complicate anything to allow it any more than
> DRAPs complicate things already?

No, it's just not optimal. It may happen in cases where some frames
that were in between were dropped (or some capture system that would
only keep keyframes from a source stream).

> possible for you to have a RAP for a frame other than the base layer, so you
> may actually have one or more shown frames in the same temporal unit (for
> different spatial layers) before a (delayed or even non-delayed) RAP, and
> again be unable to re-order them, and now whether you can start display at
> that frame or a following frame can depend on which operating point you are
> trying to use.
>
> In particular, the problem is that the AV1 specification defines a RAP in
> terms of a *frame*, and not in terms of a temporal unit. Because Matroska
> defines its blocks as containing a temporal unit, and RAPs in terms of a
> block, you will have to define the details of when a temporal unit should be
> considered a RAP (or DRAP, KFDRP, etc.) based on the frames contained inside
> of it.
>
> specification, since the idea was always for temporal units to be the unit
> of muxing in any container, but I expect it was done this way out of
> expediency to avoid having to think through all of these cases. We have to
> think through them now.

Sadly it's too late to fix this. And on our side we cannot just assume
people will just do "the right thing" if the spec allows different/odd
ways.

> imagine an encoder that could take advantage of them, I don't think there
> would be a big loss by forbidding them.

I agree. We could also have a different CodecID for a more loose/less
seek friendly version.

> I do think it would be a very good idea to have the *same* restrictions (if
> any) for Matroska and MP4. If something is technically allowed by the AV1
> specification but you can't actually put that into any container, then in
> practical terms it might as well not be allowed. But if we allow some things
> in Matroska but don't allow them in MP4, or vice versa, then you just have
> an interoperability mess.

I agree. That's why I had a few picks there to see how things are
done. There doesn't seem to be much restrictions on the bitstream
itself (other that the ones I already copied like the low overhead
bitstream format). But that also means if we want to add new
restrictions we need to have them on the MP4 side as well.