Subject: Re: [Cellar] AV1 mapping Matroska

From: Andreas Rheinhardt <[email-masked]>
Date: Fri, 29 Jun 2018 22:09:00 +0000

Hello,

I have only just started reading the specification, but I already have
some comments:

1. Your proposal says that DisplayHeight and DisplayWidth must be set so
that they agree with what is said in the CodecPrivate/in the bitstream
so that one can't use the Matroska fields anymore to override what is
set in the bitstream. If I am not mistaken, then AV1 would be unique in
this regard among all the video codecs that are allowed in Matroska. I
don't see a reason for this at all and propose that there shouldn't be a
special treatment for DisplayHeight and DisplayWidth in AV1.

2. Two sequence header OBUs with the same [seq_profile],
[high_bitdepth], [seq_level_idx] and [color_range] can still differ in
the [BitDepth] derived from them due to [twelve_bit]. Therefore I wonder
if this is the right restriction. Wouldn't it be better if it is
demanded that the derived [BitDepth] coincide for every sequence header OBU?

3. Given that AV1 seems to only use one Sequence Header OBU at a time
couldn't one use the CodecState and CueCodecState elements to designate
where the currently active Sequence Header OBU can be found so that one
doesn't need to repeat the Sequence Header with every keyframe?
(Btw: There are currently conflicting recommendations regarding the
existence of Sequence Header OBU if there is actually only one Sequence
Header OBU. On the one hand, they should be omitted, on the other hand
every block marked as keyframe should start with a Sequence Header OBU.)

Andreas Rheinhardt/mkver

PS: This is my first mail to this list and I hope that it will be
presented as a reply to Steve Lhomme's proposed AV1 mapping in Matroska,
but given the fact that I only subscribed today I can't simply reply,
but had to explicitly set the "In-Reply-To" field. Hope it works.

---

From: Steve Lhomme <[email-masked]>
Date: Sat, 30 Jun 2018 09:44:13 +0200

Hi,

> special treatment for DisplayHeight and DisplayWidth in AV1.

You're right. This is a mistake on my part. Instead of MUST it's a
SHOULD. Because Matroska allows to override the codec internal values
to display this differently, including changing the aspect ratio or
forcing physical dimensions. I will change that.

> demanded that the derived [BitDepth] coincide for every sequence header OBU?

It depends if it influences the output of the decoder or the decoder
that can be used. My main concern here is that we have the necessary
information to setup the decoder to start playback and be done with
the duration of the Segment. Otherwise that means the CodecPrivate is
sufficient.

There's a fine line between being too restrictive and forcing multiple
Segment for a simple internal change in the codec. So we don't want to
restrict too many field from the Sequence Header. But on the other
hand we want to ensure a Segment is consistent, especially regarding
seeking.

That being said, you're right [high_bitdepth] and [twelve_bit] are the
ones needed to make up the actual [BitDepth] so I will change that.

> doesn't need to repeat the Sequence Header with every keyframe?

Yes, that would be a good use for when seeking is used. But these
elements were ruled out as deprecated because they were of no use
until now. I know VLC doesn't read them for example.

> every block marked as keyframe should start with a Sequence Header OBU.)

You mean in my document or the AV1 specs ? My understanding is that
the AV1 bitstream allow Sequence Header OBUs to be found within the
stream. And doesn't mention which elements are allowed to change or
not. Maybe there will be profiles for such restrictions. It won't be
stored in the bitstream (which is frozen) but could be extra data we
put in the CodecPrivate.

> but had to explicitly set the "In-Reply-To" field. Hope it works.

It worked perfectly. Thanks for taking the time to review this !

> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Sat, 30 Jun 2018 09:49:57 +0200

I updated my pull request with these 2 changes.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sat, 30 Jun 2018 06:54:21 -0700

> forcing physical dimensions. I will change that.

Keep in mind that "SHOULD" tends to have a somewhat stronger force in an 
IETF specification than in normal English. If you're going to use SHOULD 
and not MUST, it is a good idea to explain when someone might reasonably 
violate that SHOULD (if you can't because you can't think of a reason, 
then you want MUST; if you can't because there are too many reasons, 
then you want MAY).

> not. Maybe there will be profiles for such restrictions. It won't be

See Section 7.5 "Ordering of OBUs":

"Sequence header OBUs may appear in any order within a coded video 
sequence. Within a particular coded video sequence, the contents of 
sequence_header_obu must be bit-identical each time the sequence header 
appears except for the contents of operating_parameters_info. A new 
coded video sequence is required if the sequence header parameters change."

So the question you want to answer is how what is stored in a Matroska 
file maps to a "coded video sequence". My opinion (as an individual) is 
that a single EBML Document should correspond to a single coded video 
sequence. That means that nothing in the sequence header can change 
(except for the operating_parameters_info).

---

From: Luca Barbato <[email-masked]>
Date: Sat, 30 Jun 2018 17:10:03 +0200

> (except for the operating_parameters_info).

+1

I had to plan for sudden codec reconfiguration because it MAY happen,
but I'd rather have it clearly signaled at demuxer level when possible.

lu

---

From: Andreas Rheinhardt <[email-masked]>
Date: Sat, 30 Jun 2018 21:02:00 +0000

Hello,

1.
> until now. I know VLC doesn't read them for example.
Are you sure that it is deprecated? I couldn't find anything that says
this and the current version of the Codec Specs contains this: "When the
Initialisation is updated within a track then that updated
Initialisation data MUST be written into the CodecState Element of the
first Cluster to require it." (Btw: This requirement is not fulfilled
for the way H.264 is commonly muxed into Matroska.) Your draft also
includes nothing that indicates that CodecState is deprecated.

Anyway, given that this element is not part of WebM means that it can't
be used in Matroska either if one (sensibly) wants only one codec
mapping for both.
> You mean in my document or the AV1 specs ?
In your document. I have already made a PR for this. Timothy B.
Terriberry has misunderstood what I had in mind: At one place the draft
said that every keyframe should have a sequence header OBU and at
another place it said that there shouldn't be in-band sequence header
OBUs if all of them coincide. This is contradictory in case all of the
sequence header OBUs coincide.

2. How does one signal the difference between a real KEY_FRAME and an
INTRA_ONLY_FRAME (that isn't a valid random access point) when using
blocks (that don't have a keyframe flag)? According to your draft, the
relevant `BlockGroup` won't have any `ReferenceBlocks` so that both
frame types are indistinguishable at the container level when using
`BlockGroup`s (if the muxer knows how to handle AV1 it should of course
not reference an INTRA_ONLY_FRAME in the cues, but it would really be
advantageous to be able to infer this without resorting to cues which
are optional anyway and unavailable for e.g. live-streaming).

3. There is something wrong with the invisible flag:
a) According to 7.5. every temporal unit contains at least one frame for
which [show_frame] || [show_existing_frame] equals 1, i.e. for each
Matroska block there is an output frame (even if said output frame is
only from a frame buffer and not from data directly contained in the
Matroska block). This implies that actually no block should have the
invisible flag bit set. And if it is set, it actually means that the
frame that should normally be output due to said frame should not be
displayed (regardless of whether the frame that would normally be output
is one of the already existing ([show_existing_frame] == 1) frames or
not). Otherwise we'd be breaking the semantics of the invisible flag
(and therefore make it impossible to hide individual frames of an
already encoded AV1 track).
b) If one nevertheless wants to map the invisible flag to properties of
the track, it IMO shouldn't be done the way it is currently proposed:
i) It is possible that [show_frame] equals 1 (meaning the frame should
be immediately be output) and [showable_frame] is 1 (namely if
[frame_type] is not KEY_FRAME). In your draft the invisible bit should
be set to 1 for such a `Block` meaning that the frame should not be
displayed although it should be immediately output. This is obviously wrong.
ii) Apart from that there is the problem that a temporal unit can
contain more than one frame.
iii) A frame might also be displayed later if [showable_frame] equals 1.
So the closest to a invisible frame seems to be a frame with
[show_frame] and [showable_frame] equal to zero. So a temporal unit that
contains only frames with [show_frame] and [showable_frame] equal to 0
seems to be a sensible choice to merit the invisible flag. (Such a
temporal unit must contain a frame with [show_existing_frame] equal to 1.)

4. There is currently no hard requirement that only real random access
points get flagged as keyframes in Matroska/WebM. The current wording is
only a "SHOULD". Is this really intended?

5. Given that changing extradata are not that uncommon in H.264 I don't
deem it a good idea to restrict Matroska to one coded video sequence.
This might end up excluding a lot of content from being muxed into
Matroska (and it makes it much harder to append different tracks).

Andreas Rheinhardt