Subject: [Cellar] AV1 mapping Matroska

From: Steve Lhomme <[email-masked]>
Date: Tue, 26 Jun 2018 17:42:36 +0200

Hi,

I drafted a document to specify how AV1 should be stored within Matroska.

You can find the document here:
https://github.com/Matroska-Org/matroska-specification/blob/av1-mappin/codec/av1.md

It may start as a basis for how we define (extensively) other codecs.

Let me know if it's readable for people with no knowledge of AV1 and
for people with knowledge of AV1.

Some parts marked inside {} are not finished yet. Especially the
encryption parts.

For reference the MP4 mapping draft can be found here
https://aomediacodec.github.io/av1-isobmff/

PS: it seems AV1 has new "standard" values for matrix coefficients and
Primaries that we could borrow for Matroska.

-- 
Steve Lhomme
Matroska association Chairman

---

From: Jerome Martinez <[email-masked]>
Date: Wed, 27 Jun 2018 10:32:03 +0200

Hi,

> for people with knowledge of AV1.

It is readable, at least for me, as a person who implemented quickly the 
parsing of the AV1 sequence headers.

> https://aomediacodec.github.io/av1-isobmff/

For me it is important to use same semantics that in ISO BMFF, easier 
for encoders and decoders.

> Primaries that we could borrow for Matroska.

Nothing new, they use up to date H.265 values (which are de facto 
already in use when muxing H.265, with such values, in Matroska).

Jérôme

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 27 Jun 2018 10:38:04 +0200

2018-06-27 10:32 GMT+02:00 Jerome Martinez <[email-masked]>:
> For me it is important to use same semantics that in ISO BMFF, easier for
> encoders and decoders.

Yes. One of the things the currently differ is in CodecPrivate I
didn't prefix with the presentation_delay bits. It sounds like this is
something we should parse and adjust our timestamps with. It's either
TrackOffset which is currently marked as deprecated for all versions
(ie never used) or CodecDelay which is an unsigned value and MUST be
substracted to the Block timestamp. Except it seems the AV1
presentation delay should be ADDED to the timestamps. I will dig
deeper but we may have to tweak CodecDelay to make it signed.

> Nothing new, they use up to date H.265 values (which are de facto already in
> use when muxing H.265, with such values, in Matroska).

Ah, I looked at matroska.org not the RFC specs (yet).

>
> Jérôme
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Moritz Bunkus <[email-masked]>
Date: Wed, 27 Jun 2018 10:58:52 +0200

Hey,

I've only had time to skim over it. It's definitely understandable.

For me the most important thing is to have the mapping in Matroska and WebM
be identical. As the reference encoder already produces WebM files, I
highly suggest we communicate with their developers regarding the
mapping. I cannot find a document for AV1/WebM bindings at the moment,
though…

> I will dig deeper but we may have to tweak CodecDelay to make it signed.

You cannot change the type of an existing element. It would potentially
break the interpretation of existing files created with the old
semantics. Let's take the value 144, which as an unsigned EBML integer
would be stored as 0x81 0x90. If you read that back as a signed EBML
integer, you'd get the value -128.

Kind regards,
mosu

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 27 Jun 2018 11:30:40 +0200

> mapping. I cannot find a document for AV1/WebM bindings at the moment,

Ah I didn't know that. Now I can see the code. Especially webmenc.c
with a TODO for Frank Galligan in there.

It way for some Wip In Progress code, because I have been appointed by
the powers that be at AOM to do the proper/official mapping.

At least this code doesn't write any CodecPrivate which could work,
except, as for VP9 we wouldn't know the decoding profile just from
reading the TrackEntry which is a major problem to select the proper
hardware decoder. And the MP4 mapping also integrates a kind of
CodecProvate with the Sequence Header OBU in there.

> integer, you'd get the value -128.

Given a codec delay sounds like some time the codec needs when fed
data before it outputs data. So I have hope it matches and I did the
math wrong.

In any case I don't know if we should include the flags like MP4 does.
Because it seems very MP4 specific (and it's a draft and may go away).
Unlike the rest of their "Codec Private" it should not be fed to the
decoder (the OBUs we have can be fed as-is to the decoder).

---

From: Moritz Bunkus <[email-masked]>
Date: Wed, 27 Jun 2018 11:45:10 +0200

Hey,

> the powers that be at AOM to do the proper/official mapping.

That's good to hear. For both Matroska and WebM?

I fear that the existence of the WebM output mode of aomenc may result in
its output becoming the de facto standard; therefore we should get the
official mapping done quickly.

Kind regards,
mosu

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 27 Jun 2018 12:28:43 +0200

> That's good to hear. For both Matroska and WebM?

Yes. They should not differ.

> official mapping done quickly.

If/when this mapping is approved I might be able to fix the code in
libaom. This code is mostly a proof of concept than an actual lib. But
for now that's what people will use.

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 27 Jun 2018 13:31:19 +0200

> decoder (the OBUs we have can be fed as-is to the decoder).

In page 20 of https://aomediacodec.github.io/av1-spec/av1-spec.pdf
there's a pseudo-code on how to handle the
[initial_display_delay_present_flag]. The values to use may differ for
each "operating point" (An operating point specifies which spatial and
temporal layers should be decoded).

It doesn't say clearly in the MP4 draft but the delay (in frame count)
means it should be the maximum delay of all operating points. Which
also means it will add too much delay for some operating points. So
IMO it's not good.

An hypotethical delay in Matroska would also use something similar,
but again it won't be a correct delay in some cases. It's really an
internal codec thing. So I think we should not handle it at all or say
that [initial_display_delay_present_flag] SHOULD be 0.

Let alone the fact that it's expressed in number of frames that may
not all have the same duration.

And finally in MP4 it starts with these bits that are mostly copied
from the Sequence Header OBU that just right after. So IMO it's
unnecessary in that case, adds more data and is prone to errors...

---

From: Thomas Daede <[email-masked]>
Date: Fri, 29 Jun 2018 17:42:19 -0700

> for now that's what people will use.

I can help with this too. Indeed, the webm output code in libaom was
just inherited from VP9, and is not supposed to be the final mapping. We
will want to change it as soon as possible to prevent files created with
the incomplete mapping entering the wild.