Subject: Re: [Cellar] AV1 mapping Matroska

From: Steve Lhomme <[email-masked]>
Date: Mon, 2 Jul 2018 09:38:53 +0200

2018-06-30 15:54 GMT+02:00 Timothy B. Terriberry <[email-masked]>:
> Steve Lhomme wrote:
>>
>> You're right. This is a mistake on my part. Instead of MUST it's a
>> SHOULD. Because Matroska allows to override the codec internal values
>> to display this differently, including changing the aspect ratio or
>> forcing physical dimensions. I will change that.
>
>
> Keep in mind that "SHOULD" tends to have a somewhat stronger force in an
> IETF specification than in normal English. If you're going to use SHOULD and
> not MUST, it is a good idea to explain when someone might reasonably violate
> that SHOULD (if you can't because you can't think of a reason, then you want
> MUST; if you can't because there are too many reasons, then you want MAY).

I changed the DisplayWidth/DisplayHeight mapping to MAY because there
is no strong requirement needed. PixelWidth/PixelHeight should be
fixed for the whole Segment and are mandatory. But the display ones
can be edited even by a user to change the aspect ratio. Using MAY
just means that the default values should come from those bits
mentioned.

>> You mean in my document or the AV1 specs ? My understanding is that
>> the AV1 bitstream allow Sequence Header OBUs to be found within the
>> stream. And doesn't mention which elements are allowed to change or
>> not. Maybe there will be profiles for such restrictions. It won't be
>
>
> See Section 7.5 "Ordering of OBUs":
>
> "Sequence header OBUs may appear in any order within a coded video sequence.
> Within a particular coded video sequence, the contents of
> sequence_header_obu must be bit-identical each time the sequence header
> appears except for the contents of operating_parameters_info. A new coded
> video sequence is required if the sequence header parameters change."

Ah I kinda missed that part.

> So the question you want to answer is how what is stored in a Matroska file
> maps to a "coded video sequence". My opinion (as an individual) is that a
> single EBML Document should correspond to a single coded video sequence.

That would be a Segment in Matroska.

> That means that nothing in the sequence header can change (except for the
> operating_parameters_info).

Section 7.5 starts with this:

"A bitstream conforming to this specification consists of one or more
coded video sequences."

So there can be bitstreams with various coded video sequences, which
parameters may change as the text right after the one you pasted
suggests:

"A new coded video sequence is required if the sequence header
parameters change. Any sequence header in a bitstream which changes
the parameters must be contained in a temporal unit with temporal_id
equal to zero."

My understanding is that an AV1 bitstream can change a lot of
parameters for a whole stream. A coded video sequence is strict but so
much that people might tend to use more than one even in simple files
(although it would probably lose some of the codec efficiency). So
either we allow more than one coded video sequence (the direction I
took) and so we need to define what parameters must not change for all
the Sequence Header OBUs found. Or we only allow one coded video
sequence per Segment. It simplifies things a lot, but I fear we'll end
up with more than one Segment per "file" which is not what most people
use nowadays, even if it's legal to do so.

I'd like more input on this on what they expect the best practices for
"coded video sequence" will be. It's not obvious to me from just
reading the specs.

Looking at the term "coded video sequence" (CVS) in other codecs
(H.264 and H.265) it seems it's a common term. And for those codec one
Segment correspond to one CVS, with the parameters of that CVS stored
in the CodecPrivate (SPS + PPS for H.264 for example). So we should
probably go in that simple way.

>
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Moritz Bunkus <[email-masked]>
Date: Mon, 02 Jul 2018 09:47:45 +0200

Hey,

> Looking at the term "coded video sequence" (CVS) in other codecs (H.264
> and H.265) it seems it's a common term. And for those codec one Segment
> correspond to one CVS, with the parameters of that CVS stored in the
> CodecPrivate (SPS + PPS for H.264 for example). So we should probably go
> in that simple way.

I have quite a lot of h.264 samples here, mainly M2TS from DVB, where
SPS/PPS change mid-stream, often multiple times. For such files the first
occurrences of SPS/PPS make up the AvcC in CodecPrivate, but all key frames
are still prefixed with the currently active SPS/PPS — because that's the
only way to signal that stuff has actually changed.

Requiring a new segment each time those parameters change would make for
quite a bad experience, mainly due to two reasons:

1. There's no index listing where each segment starts making seeking inside
   multi-segment files all but impossible.
2. Software support for multi-segment files is pretty much non-existent.

I'm very pessimistic 2 would ever change, even if we mandated it in the AV1
mapping.

m.

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 2 Jul 2018 10:40:49 +0200

2018-06-30 23:02 GMT+02:00 Andreas Rheinhardt
<[email-masked]>:
> Hello,
>
> 1.
> Steve Lhomme:
>>> 3. Given that AV1 seems to only use one Sequence Header OBU at a time
>>> couldn't one use the CodecState and CueCodecState elements to designate
>>> where the currently active Sequence Header OBU can be found so that one
>>> doesn't need to repeat the Sequence Header with every keyframe?
>>
>> Yes, that would be a good use for when seeking is used. But these
>> elements were ruled out as deprecated because they were of no use
>> until now. I know VLC doesn't read them for example.
> Are you sure that it is deprecated? I couldn't find anything that says

No, you're right it's not. Only CueRefCodecState is.

> this and the current version of the Codec Specs contains this: "When the
> Initialisation is updated within a track then that updated
> Initialisation data MUST be written into the CodecState Element of the
> first Cluster to require it." (Btw: This requirement is not fulfilled
> for the way H.264 is commonly muxed into Matroska.) Your draft also
> includes nothing that indicates that CodecState is deprecated.
>
> Anyway, given that this element is not part of WebM means that it can't
> be used in Matroska either if one (sensibly) wants only one codec
> mapping for both.

Correct.

>>> (Btw: There are currently conflicting recommendations regarding the
>>> existence of Sequence Header OBU if there is actually only one Sequence
>>> Header OBU. On the one hand, they should be omitted, on the other hand
>>> every block marked as keyframe should start with a Sequence Header OBU.)
>>
>> You mean in my document or the AV1 specs ?
> In your document. I have already made a PR for this. Timothy B.
> Terriberry has misunderstood what I had in mind: At one place the draft
> said that every keyframe should have a sequence header OBU and at
> another place it said that there shouldn't be in-band sequence header
> OBUs if all of them coincide. This is contradictory in case all of the
> sequence header OBUs coincide.

OK I will try to clarify this.

> 2. How does one signal the difference between a real KEY_FRAME and an
> INTRA_ONLY_FRAME (that isn't a valid random access point) when using
> blocks (that don't have a keyframe flag)? According to your draft, the
> relevant `BlockGroup` won't have any `ReferenceBlocks` so that both
> frame types are indistinguishable at the container level when using
> `BlockGroup`s (if the muxer knows how to handle AV1 it should of course
> not reference an INTRA_ONLY_FRAME in the cues, but it would really be
> advantageous to be able to infer this without resorting to cues which
> are optional anyway and unavailable for e.g. live-streaming).

Indeed, there's an issue here. I think we had a similar discussion in
the past and we concluded that a ReferenceBlock with a value of 0 can
be used. Meaning it's referencing itself. I should add that to the
document.

> 3. There is something wrong with the invisible flag:
> a) According to 7.5. every temporal unit contains at least one frame for
> which [show_frame] || [show_existing_frame] equals 1, i.e. for each
> Matroska block there is an output frame (even if said output frame is
> only from a frame buffer and not from data directly contained in the
> Matroska block). This implies that actually no block should have the
> invisible flag bit set. And if it is set, it actually means that the
> frame that should normally be output due to said frame should not be
> displayed (regardless of whether the frame that would normally be output
> is one of the already existing ([show_existing_frame] == 1) frames or
> not). Otherwise we'd be breaking the semantics of the invisible flag
> (and therefore make it impossible to hide individual frames of an
> already encoded AV1 track).

Indeed, it's either " Each temporal unit must have exactly one shown
frame." or "Every layer that has a coded frame in a temporal unit must
have exactly one shown frame that is the last frame of that layer in
the temporal unit."

So each temporal unit has at least one visible frame. And since we map
1 Temporal Unit = 1 Block we don't have to extract the invisible part.

> b) If one nevertheless wants to map the invisible flag to properties of
> the track, it IMO shouldn't be done the way it is currently proposed:
> i) It is possible that [show_frame] equals 1 (meaning the frame should
> be immediately be output) and [showable_frame] is 1 (namely if
> [frame_type] is not KEY_FRAME). In your draft the invisible bit should
> be set to 1 for such a `Block` meaning that the frame should not be
> displayed although it should be immediately output. This is obviously wrong.
> ii) Apart from that there is the problem that a temporal unit can
> contain more than one frame.
> iii) A frame might also be displayed later if [showable_frame] equals 1.
> So the closest to a invisible frame seems to be a frame with
> [show_frame] and [showable_frame] equal to zero. So a temporal unit that
> contains only frames with [show_frame] and [showable_frame] equal to 0
> seems to be a sensible choice to merit the invisible flag. (Such a
> temporal unit must contain a frame with [show_existing_frame] equal to 1.)
>
> 4. There is currently no hard requirement that only real random access
> points get flagged as keyframes in Matroska/WebM. The current wording is
> only a "SHOULD". Is this really intended?

I'll double check on that.

> 5. Given that changing extradata are not that uncommon in H.264 I don't
> deem it a good idea to restrict Matroska to one coded video sequence.
> This might end up excluding a lot of content from being muxed into
> Matroska (and it makes it much harder to append different tracks).

I'm looking at how it's officially handled in MP4. It seems they have
different modes, but the "avc1" only has the SPS and PPS in the
extradata and not in the stream. Also they have this note:

"NOTE 1 It is recommended that when several parameter sets are used
and parameter set updating is desired, a separate parameter set
elementary stream be used"

Meaning a single file should not have changing parameters others that
the one expressed in the header.

Thanks a lot for your feedback !

> Andreas Rheinhardt

---

From: Michael Niedermayer <[email-masked]>
Date: Mon, 2 Jul 2018 12:34:55 +0200

Hi

> only way to signal that stuff has actually changed.

IMHO All SPS/PPS should be in the "global header" (CodecPrivate) which the
global header applies to, not just the first.
This way an application can know all resolutions, aspect ratios and so on
from just the header without the need to scan through all random access
points.

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Those who are too smart to engage in politics are punished by being
governed by those who are dumber. -- Plato

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 2 Jul 2018 12:51:40 +0200

> from just the header without the need to scan through all random access

My understanding is that it's how it's done in MP4:

"Each AVC sample entry, which contains the AVC video stream decoder
specific information, includes a group of SPSs and PPSs. This group of
parameter sets functions much like a codebook. Each parameter set has
an identifier, and each slice references the parameter set it was
coded against using the parameter set's identifier."

I have to check but I don't think Sequence Headers OBUs have
identifiers like that in AV1.

By the way, this is one mapping for V_AV1. There could be other AV1
mapping in Matroska where there is no CodecPrivate and there could be
multiple coded video sequences in it. But it would require the use of
CodecState and CueCodecState which is currently not an option for
WebM. So for now I will concentrate on a single coded video sequence.
And if needed in the future we can do a different one.

---

From: Jerome Martinez <[email-masked]>
Date: Mon, 2 Jul 2018 13:14:02 +0200

>> global header applies to, not just the first.

Not possible with e.g. real time streaming (you don't know in advance 
the next SPS/PPS)

> identifiers like that in AV1.

Not in AV1 (I just updated my own code for AV1  1.0.0 spec, still not 
there).
But it actually does not answer the issue because in the example there 
is usually still only the "set id" of 0 (the new SPS replaces the old 
one, not a new number).

Actually IIRC in HEVC (same for AVC) there are explicitly 2 different 
codec identifiers:
- "hvc1" is for *all* SPS/PPS in the global header (CodecPrivate), and 
there is no SPS/PPS in the stream itself
- "hev1" is for *no* SPS/PPS in the global header (CodecPrivate), and 
key frames are prefixed with the currently active SPS/PPS
(I personally don't see the need of 2 different identifiers, as it quick 
to see if SPS/PPS is in the file, but there is surely a reason)

The issue is not only for AV1, but for all formats permitting to have 
the init in the stream itself (AVC, HEVC, AV1...).

IMO we should not restrict to only one possibility in Matroska spec, as 
they are for different needs (non real time vs real time), whatever is 
the format (AVC, HEVC, AV1...).

> And if needed in the future we can do a different one.

If WebM decides that real time AV1 is for them, they may decide to 
discard CodecPrivate usage and put sequence header in the "Block" (as 
they currently do with up to date AOM build)
So restriction will not be there in practice.

Jérôme

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 2 Jul 2018 13:27:46 +0200

2018-07-02 13:14 GMT+02:00 Jerome Martinez <[email-masked]>:
> Not possible with e.g. real time streaming (you don't know in advance the
> next SPS/PPS)
>
> But it actually does not answer the issue because in the example there is
> usually still only the "set id" of 0 (the new SPS replaces the old one, not
> a new number).
>
> Actually IIRC in HEVC (same for AVC) there are explicitly 2 different codec
> identifiers:
> - "hvc1" is for *all* SPS/PPS in the global header (CodecPrivate), and there
> is no SPS/PPS in the stream itself
> - "hev1" is for *no* SPS/PPS in the global header (CodecPrivate), and key
> frames are prefixed with the currently active SPS/PPS
> (I personally don't see the need of 2 different identifiers, as it quick to
> see if SPS/PPS is in the file, but there is surely a reason)
>
> The issue is not only for AV1, but for all formats permitting to have the
> init in the stream itself (AVC, HEVC, AV1...).
>
> IMO we should not restrict to only one possibility in Matroska spec, as they
> are for different needs (non real time vs real time), whatever is the format
> (AVC, HEVC, AV1...).
>
>
>
> If WebM decides that real time AV1 is for them, they may decide to discard
> CodecPrivate usage and put sequence header in the "Block" (as they currently
> do with up to date AOM build)

Seeking in such a file would not be possible. I'm not sure it's
desirable even when doing adaptive streaming with short sequences. We
will need CodecState and CueCodecState for that. We could define such
a case (with a different codec ID) for Matroska and WebM could pick
that from there.

Also while working on the case where the Sequence Header OBU doesn't
change, I noticed an issue. The Sequence Header OBU can still change
the [operating_parameters_info] during the lifetime of the stream
(paragraph 7.5 pointed out by Tim). That means even applying the
stricter restrictions of the Codec Video Sequence in AV1 means we'd
still need CodecState and CueCodecState for seeking to work. This
change is only possible in a specific decoder model (Annex E) when
[decoder_model_info_present_flag] is 1. When it's 0 then the
[operating_parameters_info] doesn't apply. So we should probably
restrict the single Sequence Header OBU to streams where
[decoder_model_info_present_flag] is 0.

---

From: Hendrik Leppkes <[email-masked]>
Date: Mon, 2 Jul 2018 13:35:00 +0200

> [decoder_model_info_present_flag] is 0.

In this particular case, doesn't the stream already include in-band
Sequence Header OBUs, and judging from the requirement of some
parameters being allowed to change, I would wager that decoders would
be setup to handle them as well.
So all this comes down to is setting up Cues properly to point to
clean seek points with a Sequence Header + Keyframe, much like a H264
stream with minor SPS/PPS changes would be muxed right now.

Now if an encoder would produce a stream that uses multiple different
Sequence Headers but doesn't regularly repeat the sequence header (ie.
with keyframes), then such a stream would be unseekable in whatever
medium, and that would be the encoders fault.

(PS: re-posted to the list, can someone fix this list to set reply-to headers?)

- Hendrik

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 2 Jul 2018 13:54:37 +0200

> stream with minor SPS/PPS changes would be muxed right now.

Ah yes, good point. In Paragraph 7.5 there is this:

"Note:There is not a requirement that every temporal unit with a key
frame also contains a sequence header, just that the sequence header
has been sent before the first key frame. However, note that temporal
units without sequence header OBUs are not considered to be random
access points."

So we can assume that differing Sequence Headers OBUs will be found
in-band. And must be bit-identical to the first one in the CVS except
for [operating_parameters_info].

It may also mean that our keyframe flag may need to be refined. And by
"temporal unit" I think it means "temporal unit with a key frame"
wihout a sequence header OBU are not RAP. In practice I don't think
there will not be keyframes where the sequence header OBU is not in
that temporal unit.

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 2 Jul 2018 15:46:06 +0200

I updated the document with the following changes:

- the Sequence Header OBU in CodecPrivate has the same restrictions as
a coded video sequence in AV1

- if all Sequence Header OBU in Blocks are going to be the same
(decoder_model_info_present_flag =0) they can be omitted from the
Blocks.

- if decoder_model_info_present_flag =1 in the CodecPrivate then all
keyframes MUST have a Sequence Header OBU

- the low overhead syntax is a MUST

- set the keyframe flag/ReferenceBlock for valid random access points
(not intra frame).

- remove the Inisible bit mention (it's actually supposed to be used
for prerolling when editing files).

- DisplayWidth/Height are just hints of how to get the value

- typos (from mkver)

https://github.com/Matroska-Org/matroska-specification/blob/av1-mappin/codec/av1.md

---

From: Michael Niedermayer <[email-masked]>
Date: Mon, 2 Jul 2018 21:07:01 +0200

> Not possible with e.g. real time streaming (you don't know in advance the

In cases where the SPS/PPS can change and it is not known when the global
header needs to be produced (that is for example real time streaming).
it follows that there should be no global header or it should not contain 
any SPS/PPS. But only things which are known for the whole stream.

I would argue this is kind of the definition of a global header.
It applies globally to the whole stream. 
Storing only one out of several PPS/SPS in the header 
(which was mentioned above) feels  rather incorrect to me. And any
software which makes decissions based on SPS data like resolution and
aspect ratio could benefit from knowing if it needs to scan the whole
stream for SPS changes or if it can trust the first or global header

[...]

If you drop bombs on a foreign country and kill a hundred thousand
innocent people, expect your government to call the consequence
"unprovoked inhuman terrorist attacks" and use it to justify dropping
more bombs and killing more people. The technology changed, the idea is old.

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 9 Jul 2018 10:17:40 +0200

> any SPS/PPS. But only things which are known for the whole stream.

In Matroska this can be achieved by starting a new Segment when the
codec parameters change. That has an impact on all tracks at the same
time so it may not be good/friendly in all cases. We could describe
this scenario in the specs when a `PrevUID` should be used to specify
the new Segment is a continuation of the previous segment. Actually
even the NextUID could be generated for each Segment and written when
it starts. Technically it would be a correct segment hard-linking but
in the context of live streaming (unknown duration per segment).