Subject: [Cellar] AV1 init

From: Steve Lhomme <[email-masked]>
Date: Thu, 19 Jul 2018 06:29:31 +0200

Hi,

As the CodecDelay seems to be unnecessary, we're getting close to
finalizing the AV1 mapping for Matroska and WebM. It would be
beneficial to everyone that Matroska and MP4 (ISOBMFF) share as much
as possible so it's easy to transmux from one to the other.

The initial_presentation_delay_present we may not deal with at all or
we should have a specific Matroska element that can be used by all
codecs. So it doesn't really go into the AV1 codec mapping (more an
MKV/ISOBMFF compatibility document).

The other outstanding issue is the configOBU in MP4 that can be empty.
In our case we mandate that the Sequence Header OBU that represent the
Coded Video Sequence be put in there.

Because it's not mandatory in MP4 it's fair to say people will not
bother too much writing it. In fact the only currently available
encoder doesn't export it, one has to parse the bitstream to find it.
See this ffmpeg thread [1] which led to this "fix" request in libaom
[2]. When that is done, libaom should be able to output WebM that
follow our current mapping [3].

We have the choice of make the Sequence Header mandatory in
CodecPrivate (MUST) make it highly recommended (SHOULD), just
suggested (MAY) or not put it at all.

IMO it should always be there. Because as stated above, if not people
will likely not write it. And players will have to handle the case
where it's not there.

And the reason why it should be there is for playback reasons, so that
the right decoder can be selected before having to parse the data.
There's a reason H264/H265 have profiles, it's so that you can tell if
you can decode something or not and how. In many cases you get
hardware decoding and in other cases not. This will likely be the same
for AV1, for example there might never be hardware decoders doing 12
bits, that doesn't mean hardware decoding should never be used.

In the case of avcodec, deciding if HW is going to be used needs to be
done before starting the decoder, before parsing any codec data.
Because you need to chose whether 1 thread should be used (a
limitation of HW decoders on Windows and Linux at least) or multiple.
And you can't switch once the codec is initialized. That situation is
still not too bad. If one has to use completely different code for HW
decoding (like Intel's MediaSDK API) and for SW decoding (libaom) it
gets worse. You'd need to parse the codec data before deciding which
one to use. That also requires an extra buffering step at the
beginning to probe the codec data and not lose these data (still need
to be fed to the codec once it's initialized) (while the interleaved
audio is going through the decoder already, especially in WebM where
audio should come before video).

We also have the counter example of not having the codec
characteristics known in advance with VP9. Some GPU are capable of
decoding 8 bits VP9 but not 10 bits. In VLC (and other apps using
libavcodec) we can't decide whether to use 1 thread or many before
intializing the decoder. So we use many which is the case that works
best. But when the HW decoder is used it's from multiple threads which
is not how it's designed to be used, has performance issues and
sometimes crashes.

If we make the Sequence Header OBU mandatory in the CodecPrivate we
avoid all these issues on the playback side. It puts the trouble of
extracting that data on the muxer/encoder side. So I think it's the
right way for Matroska/WebM and also the right way for ISOBMFF, so I
cc Cyril in this thread.

[1] http://ffmpeg.org/pipermail/ffmpeg-devel/2018-July/232084.html
[2] https://bugs.chromium.org/p/aomedia/issues/detail?id=2012
[3] https://bugs.chromium.org/p/aomedia/issues/detail?id=2027

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 19 Jul 2018 12:21:26 +0200

> bits, that doesn't mean hardware decoding should never be used.

Reading the libaom code it seems that it does not support profile 2
https://aomedia.googlesource.com/aom/+/master/av1/decoder/obu.c#205

That's also some information that would be good to have before
starting using that decoder.

---

From: Andreas Rheinhardt <[email-masked]>
Date: Thu, 19 Jul 2018 13:03:00 +0000

Hello,

I agree with everything you said regarding the availability of the
necessary data for codec initialization in the `CodecPrivate`. But there
are some other points:

1. The new proposal says that `DisplayHeight` and `DisplayWidth` are not
mandatory. This is news to me as the website (outdated, I know) says
that they are. That's why they can have default values. That the default
values only apply if `DisplayUnit` is 0 (its default value) doesn't
change this IMO. (Or is it possible to not signal an aspect ratio with
using `DisplayUnit` 4?)

2. The current version still contains this: "The Block timestamp is
translated from the [PresentationTime], including the
[InitialPresentationDelay], which is then substracted by the CodecDelay.
Audio/Video/Subtitle interleaving SHOULD take the CodecDelay in account
so that Blocks with similar timestamps (adjusted with the CodecDelay)
are close to each other in a Cluster."
I think you want to discard this.

3. The current proposal does allow to have a `CodecPrivate` that
consists of multiple OBUs none of which has [obu_has_size_field] set to
1. This is the same situation as for the blocks. Then the commit message
read as if you only wanted to allow this (not having
[obu_has_size_field] set to 1) for the very last OBU of a `Block`. If
this is really what is wanted then the current wording doesn't convey
the current intention and should be changed. I already made a proposal
(in my email from July 14th) for this for the Block:
"If an OBU is not the last OBU in a `Block`, it MUST follow the [Low
Overhead Bitstream Format syntax] (i.e. it MUST have
[obu_has_size_field] set to 1); the last OBU in a `Block` MAY have
[obu_has_size_field] set to 0, in which case it is assumed to fill the
remainder of the `Block`."
Of course, if it is intended to allow [obu_has_size_field] set to zero
everywhere, then the above proposal can be ignored.

4.

Steve Lhomme:
> I think we really should deal with the initial delay; actually, knowing
how many frames should be decoded before the first one can be output
should be considered part of the initialization data. But if you want to
create a new element for this (and not put it into the CodecPrivate),
then I have to tell you that the way this delay is counted is
codec-dependent: H.264 has max_num_reorder_frames and the important
thing is that this does not translate 1-1 to `Blocks`, but that
complementary field pairs are only counted as 1, whole frames are
counted as 1 and unpaired fields are counted as 1 whereas complementary
field pairs are stored as two `Blocks` in Matroska. So I don't think a
good way to do is in a codec-agnostic manner exists. But given that
adding this to existing codecs via a modification of the `CodecPrivate`
is an absolute no-no because it breaks everything, not putting it here
in the `CodecPrivate` seems to be the right choice. But the semantics of
this new field would have to be codec-dependent, i.e. they would have to
be part of every codec mapping for every codec that can benefit from
such a field. Ideally, this new field would already be used in the first
version of the AV1 codec mapping.

5. As you are saying that we are getting close: What about my proposal
regarding when the `Sequence Header OBU` needs to be present in the
bitstream (see my email from July 14th for this)? Should I prepare a PR
for this? And what do you think about my proposal c) from my email from
July the 16th regarding the introduction of a new child of
`CueTrackPositions` to solve the seeking problem?

- Andreas Rheinhardt

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 19 Jul 2018 15:24:04 +0200

> change this IMO. (Or is it possible to not signal an aspect ratio with

If you are referring to this page, it doesn't say it's mandatory:
https://matroska.org/technical/specs/index.html

> I think you want to discard this.

Indeed, well spotted.

> everywhere, then the above proposal can be ignored.

This is an issue in ISOBMFF as well
https://github.com/AOMediaCodec/av1-isobmff/issues/26

There it is in the sample (Block for us) but it's tricky to enforce
this requirement on the size mode when the original bitstream was in a
different format. That means the muxers has to be ready to rewrite
OBUs and they won't even be restored to their original format on
playback (since they should be prepended on keyframes).

IMO any solution has its drawback. But I would lean towards keeping
the original format (so the current wording is OK). The main goal of
the Sequence Header OBU in the `CodecPrivate` is to find the profile
and we don't need to know the size for that, it's at the beggining the
OBU data (the size parsing still need to be handled but that's the
case anyway if the SH_OBU is the only one).

> version of the AV1 codec mapping.

I added a version number in case we need to explain things more or add
more things. It would be OK to add a new element for cases were this
would be needed. And the way it is done would be discussed if/when we
introduce this element. Although we never needed it so far so I'm not
sure there's some real benefit. As for the value it would be ok to
count frames/fields rather than durations.

> bitstream (see my email from July 14th for this)? Should I prepare a PR

Yes, a PR is always good. Although I update the document many times a
day so it's still a moving target.

> `CueTrackPositions` to solve the seeking problem?

IMO a new element for seeking is not an option for now, at least
because we can't guarantee it will be implemented in WebM.

> - Andreas Rheinhardt
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 19 Jul 2018 15:25:31 +0200

BTW I added an issue on the ISOBMFF spec to ask for the Sequence
Header OBU in the codec extradata be made mandatory:
https://github.com/AOMediaCodec/av1-isobmff/issues/46

---

From: Andreas Rheinhardt <[email-masked]>
Date: Thu, 19 Jul 2018 14:09:00 +0000

Hello,

> https://matroska.org/technical/specs/index.html
true. I could have sworn it was, but apparently I was wrong.>
> case anyway if the SH_OBU is the only one).
Ok. Then that is settled as well.
> day so it's still a moving target.
Ok, I will try to make it this evening.
> because we can't guarantee it will be implemented in WebM.
I don't think that a satisfactory solution exists without new elements.
And given that Webm is a member of the Alliance for Open Media I don't
think that they will resist if we add a new element which is very
benificial for AV1. Who is actually in charge of the Webm bitstream
format? Maybe we should simply informally ask him. Maybe he is even a
member of this mailing list.

- Andreas Rheinhardt

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 20 Jul 2018 09:48:27 +0200

> member of this mailing list.

I think Frank Galligan is the one to contact.

---

From: Andreas Rheinhardt <[email-masked]>
Date: Fri, 20 Jul 2018 23:07:00 +0000

Hello,

> day so it's still a moving target.
Done: https://github.com/Matroska-Org/matroska-specification/pull/247
I also converted the clause "The parameters that should not change for a
video Track are the dimensions..." to "Furthermore the dimensions of all
output frames MUST be equal", although I am not sure if we should make
this hard requirement or not. But I did not want to ambush you with such
a change buried in a big PR.

- Andreas Rheinhardt