Subject: [Cellar] review of codec_specs.md for defining use of encodings in Matroska

From: Dave Rice <[email-masked]>
Date: Sat, 24 Sep 2016 18:05:55 -0400

Hi all,

Here’s a review of the codec_specs.md page in the matroska-specification repository. I also compared the Codec IDs listed in this page to a collection of sample files from the Internet Archive to get an idea on how much this page and real use overlap.

Currently there is a page that documents how certain encodings are stored in Matroska. The original document is on the matroska site at https://matroska.org/technical/specs/codecid/index.html and there is a representation in markdown in the matroska-specification repository at https://github.com/Matroska-Org/matroska-specification/blob/gh-pages/codec_specs.md. There’s a related version on Haali’s site at http://haali.su/mkv/codecs.pdf.

Within this page three values are used to define Matroska’s support for an encoding:

1. Codec ID
2. Name
3. Description

The Codec ID appears to have a form to it but it is not defined though should be.

The description appears to consistently.

The terms ‘private data’ and the `CodecPrivate Element` seems to be referenced in the same way. Are these the same things? If so, they should be made consistent.

The phrase "The private data is void.” is used several times in the document. Does this mean that the `CodecPrivate Element` is stored but as an `Empty Element`? Could the `CodecPrivate Element` just not be present if not needed?

The tables of codecs has some odd nested that breaks the table in markdown. I propose that the data be rearranged into a simple table.

Are there encodings that Matroska supports that are not in this list? 

This Codec ID is listed as ”supported later” but is it working now?
Codec ID: V_MJPEG
Sample file: http://archive.org/download/2VideoMP4MPEG44096x216063625KbpsHiRez/34_Video_MKV_(mjpeg)________640x480______3166Kbps__________________.mkv

Codec ID: V_VP8
There are many files around with this Codec ID. Supported?
Sample file: http://archive.org/download/20140825071_201408/2014-08-25-070-1.mkv

Codec ID: V_MPEGH/ISO/HEVC
Even more of these files are around but not in the spec. Supported?
Sample file: http://archive.org/download/1A71zv37dr/1A71zv37dr.mkv

Codec ID: V_VP9
Also popular, but unlisted. Supported?
Sample file: http://archive.org/download/yt-adonfjv/Super Mario 64 Disk Version 64DD - Powerfull Mario song-MGeGIaJr8gc.mkv

Codec ID: V_MPEG4/ISO/AVC
This is possibly the most widely used Codec ID in Matroska, but unlisted. Supported?
Sample: http://archive.org/download/00013862/0001-3862.mkv

Codec ID: A_AAC
This is very popular as a Matroska audio codec, but it deviates from the forms listed in codec_specs.md
Sample file: http://archive.org/download/150069/150069.mkv

Codec ID: A_OPUS
Opus is common in Matroska but unlisted here. Supported?
Sample; http://archive.org/download/JigwallysNesTheGreatWaldoSearchIn0052.41/greatwaldosearch-tas-jigwally.mkv

Codec ID: A_EAC3
More rare, but there are around. Unlisted in specs.
Sample file: http://archive.org/download/Barcelona1VS1EspanyolPenaltiesAndCrowning720pHDBarcelonaHD.blogspot.com/Barcelona 1 VS 1 Espanyol  penalties and crowning 720pHD Barcelona-HD.blogspot.com.mkv

Codec ID: A_TRUEHD
Unlisted but found in real world. Supported?
Sample file: http://archive.org/download/mad.max.fury.road.2015.1080p.bluray.hevc.truehd.7.1.atmos.absinth/Mad.Max.Fury.Road.2015.1080p.BluRay.TrueHD.7.1.ATMOS-Absinth.mkv

Note that for many of the Codec IDs noted that are not in the Matroska specification, they are in the webm specification at http://www.webmproject.org/docs/container/: such as V_VP8, V_VP9, A_OPUS. Are these valid in Matroska as well? A discussion on adding these was started at https://lists.matroska.org/pipermail/matroska-devel/2015-October/004852.html but did not resolve.

Other questions:
Should the Matroska specification provide enough info for a user to independently define support for an encoding in Matroska? Or should codec support somehow be ‘registered’ via Matroska and/or CELLAR?

Can the structure of a Codec mapping be more clearly defined, so that a user could define their own support and understand if it is correct or not?

For a Matroska validation that finds something like V_MPEG4/ISO/AVC in Matroska. Should it be considered as unsupported (most MKV files!)?

Best Regards,
Dave Rice

---

From: "Sebastian G. <bastik>" <[email-masked]>
Date: Sun, 25 Sep 2016 08:46:19 +0200

> The Codec ID appears to have a form to it but it is not defined though should be.

The obvious pattern to me is that a Codec ID starts with an V for a
video codec, with an A for a audio codec and with an S for subtitles
(and B for button). Then the Codec ID includes the name in some way. For
subtitles it states if it is a text or image based subtitle, though not
for all of them.

> The tables of codecs has some odd nested that breaks the table in markdown. I propose that the data be rearranged into a simple table.

I assume that the separation of video, audio, subtitles and other codec
related stuff will be possible with a simple table, in which case I
would be OK with replacing the nested table with a simple one.

> Are there encodings that Matroska supports that are not in this list? 

Yes, I think so. VP9 and OPUS should be already supported, but are not
listed. There might be even more.

> Sample file: http://archive.org/download/20140825071_201408/2014-08-25-070-1.mkv

I think it is the basis for WebM, so either it had been supported before
or Google would have added support for it to WebM.

> Sample file: http://archive.org/download/yt-adonfjv/Super Mario 64 Disk Version 64DD - Powerfull Mario song-MGeGIaJr8gc.mkv

The same that applies to WebM I think.

> Sample; http://archive.org/download/JigwallysNesTheGreatWaldoSearchIn0052.41/greatwaldosearch-tas-jigwally.mkv

There have been changes to Matroska in order to support pre-roll for
OPUS. There is a related email, but I can't find the list thread
discussing changes to Matroska.[0]

> Note that for many of the Codec IDs noted that are not in the Matroska specification, they are in the webm specification at http://www.webmproject.org/docs/container/: such as V_VP8, V_VP9, A_OPUS.…

A question that I see is: What about Codec ID other people used and got
popular? Should they just be included in the codec spec? And even if
they go against some naming scheme?

> Should the Matroska specification provide enough info for a user to independently define support for an encoding in Matroska? Or should codec support somehow be ‘registered’ via Matroska and/or CEL…

I quote the codec specs: "[...] to list those codecs that are currently
supported in matroska (or will be supported soon), and therfore need a
well defined codec ID so that all developers supporting matroska will
use the same ID."

(There is a spelling mistake in 'therfore')

I'm not sure if people come up with the same name independently for
every possible new format, even if the spec contains detailed
information on how to derive them.

Well, arguably the quoted section did not turn out well, as you show
examples where there are no defined IDs, but there are ones that are used.

However, to me it still appears to be better to have a governing body to
ensure that in the long run, if not from the beginning of the usage of
an ID, that only one Codec ID will be used.

> Can the structure of a Codec mapping be more clearly defined, so that a user could define their own support and understand if it is correct or not?

Certainly for the sake of clarity, to increase understanding and avoid
errors or mismatches some information on how it is supposed to work
should be included.

> For a Matroska validation that finds something like V_MPEG4/ISO/AVC in Matroska. Should it be considered as unsupported (most MKV files!)?

For the sake of supporting the current state, there could be a primary
Codec ID and a secondary Codec ID. While a muxer that complies with the
specs would write the primary ID, a demuxer could support both. The file
would remain valid in the sense that a player would not refuse it.

A validator could point out that an .mk? contains a secondary Codec ID,
that better should be replaced with the primary Codec ID.

Tools that would aim to bring files in a (more) compliant state would
replace the secondary ID with the primary ID, unless the user would
choose not to, because his player would not understand the primary ID.

> Dave Rice

Best Regards,
Sebastian

[0]
https://lists.matroska.org/pipermail/matroska-devel/2013-September/004573.html

---

From: Dave Rice <[email-masked]>
Date: Sun, 25 Sep 2016 09:34:24 -0400

> On Sep 25, 2016, at 2:46 AM, Sebastian G. <bastik> <[email-masked]> wrote:
> 

But the identifiers are not complete. For instance the TrackType Element, https://matroska.org/technical/specs/index.html#TrackType, also lists `complex`, `logo`, and `control` track types. Would encodings of these types have different letter prefixes in Codec ID.

Off topic, but are `complex` track types for encodings such as DV where audio, video, timecode, captions, and metadata information is all arranged within a single encoding?

> The same that applies to WebM I think.

Yes it would be good if it were feasible to remux a valid webm to a valid Matroska and still consider the encoding as supported. I suggest that first the manner in which Matroska encoding support is written is reviewed and clarified and then consider adopting the encoding support from the webm specification.

> they go against some naming scheme?

Presently there is no naming scheme to comply with or go against. I suggest one be written according to the forms of existing supported Codec ID (as you described a bit above). With a Codec ID expression and more clarity in how a encoding support is expressed, it would be easier to adopt externally defined Codec IDs.

> (There is a spelling mistake in 'therfore’)

Also I think “supported soon” should be removed from the draft and possibly moved to branches or discussion on the list instead.

> an ID, that only one Codec ID will be used.

V_MS/VFW/FOURCC provides a good workaround anyhow as it allows a lot of encodings to be used with fourccs in AVI style. Still it sounds messy to recommend that users create their own Codec IDs without some registry (similar issue to official vs user TagNames).

But supposing the Matroska RFC is finalized. How would a new Codec ID be registered? Update the Matroska RFC with an addedum or extended Codec ID part of the specification? Or create a new RFC for the use of encoding in Matroska as a separate document? IIUC the latter option is how MXF registers new encodings, by new documents.

> choose not to, because his player would not understand the primary ID.

I think that makes sense, to have some Codec ID aliases that should not be written but could be read to be interpreted with an official Codec ID.

> https://lists.matroska.org/pipermail/matroska-devel/2013-September/004573.html

Dave

---

From: "Sebastian G. <bastik>" <[email-masked]>
Date: Sun, 25 Sep 2016 18:00:24 +0200

> Yes it would be good if it were feasible to remux a valid webm to a valid Matroska and still consider the encoding as supported. I suggest that first the manner in which Matroska encoding support i…

+1 to both of that.

> Presently there is no naming scheme to comply with or go against. I suggest one be written according to the forms of existing supported Codec ID (as you described a bit above). With a Codec ID expr…

+1

> Also I think “supported soon” should be removed from the draft and possibly moved to branches or discussion on the list instead.

+1 The document seems to have been written a long time ago. It has seen
some updates, but some stuff is supported and not listed. It appears to
be good if the document would list what is supported.

> But supposing the Matroska RFC is finalized. How would a new Codec ID be registered? Update the Matroska RFC with an addedum or extended Codec ID part of the specification? Or create a new RFC for…

There are multiple possibilities as you line out.

1) Include the codec IDs in the main RFC. If I understand RFCs correctly
they are not updated/changed, but extended or replaced by other RFCs.

2) Have the Matroska RFC contain what makes a Matroska container a
Matroska container, how to create, how to read and how to change them.
And in the same specs it is listed how the Codec ID system works. Then
the codec ID list stays independent and can be updated whenever needed.

2a) Have the Matroska RFC contain anything from 2) besides how the codec
ID system works. Everything related to the codecs would go to the codecs
specs.

This list might not end here, there are other ways to achieve this.

I am unsure to what the best option would be. If one wants to support
OPUS and VP9 (as example) and has to read the main RFC, how many more
documents would have to be read?

if 1) the follow-up RFC defining support for OPUS and VP9, plus the RFC
for format changes, if lucky that would be one document more, in the
worst case one for OPUS, one for VP9.

if 2) and 2a) the RFCs for possible format changes and the document
listing the supported IDs. That would make one in the best case, two if
there actually have been changes. For the codecs ID that does not get
worser it is just one document.

At first it appears to stay more centralized with version 2) or 2a), but
what if the main specs of Matroska have to be updated for a new codec,
because it needs new elements? Wouldn't it be better to have one RFC
defining the changes to Matroska and the newly introduced codec ID?

Therefore I am not sure what the is the more reasonable approach here.

> I think that makes sense, to have some Codec ID aliases that should not be written but could be read to be interpreted with an official Codec ID.

Well, at least for those 'false' Codec IDs that have been established.
It is bad if developers would depend on the RFC too closely and reject
files that they would not have rejected otherwise.

Best Regards,
Sebastian

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 25 Sep 2016 21:07:11 +0200

Le 25/09/2016 à 00:05, Dave Rice a écrit :
> [...]

They are same thing, as described: "CodecPrivate element: Private data 
only known to the codec." but right, maybe we need to add more 
information about it.

> The phrase "The private data is void.” is used several times in the document. Does this mean that the `CodecPrivate Element` is stored but as an `Empty Element`? Could the `CodecPrivate Element` ju…

Not present if not needed is the implemented way. Maybe rephrasing the 
sentence would make it more explicit (e.g. "no CodecPrivate Element" 
instead, for stream format not needing configuration bytes).

> [...]

First, the list of supported formats and their respective definition 
should be in a separate document (updated more often than the core spec).
And yes, each new format should be registered, because there is an 
unique codec name namespace. Maybe defining "X_" prefix as private (but 
mime RFC now discourages use of "x-", so maybe not a good idea).
Now, the question is where to define CodecPrivate. e.g. for FFV1 we 
already defined it in the FFV1 spec but we can not update AVC specs for 
that.

> [...]

Matroska-centric validation tools should not validate raw streams, that 
includes CodecPrivate part. This is the role of a more general 
validation tool (a tool described as "validation of Matroska files" 
would not validate the CodecPrivate part, a tool described as 
"validation of Matroska files + FFV1 and FLAC" would use the description 
of how FFV1 and FLAC are registered)

---

From: Dave Rice <[email-masked]>
Date: Sun, 25 Sep 2016 17:37:30 -0400

> On Sep 25, 2016, at 3:07 PM, Jerome Martinez <[email-masked]> wrote:
> 
> They are same thing, as described: "CodecPrivate element: Private data only known to the codec." but right, maybe we need to add more information about it.

If CodecPrivate is referenced by nearly every definition of an encoding in Matroska, perhaps we should split the ‘Description’ into ‘CodecPrivate Usage’ and ‘Description’.

> Not present if not needed is the implemented way. Maybe rephrasing the sentence would make it more explicit (e.g. "no CodecPrivate Element" instead, for stream format not needing configuration bytes).
> 
>> [...]
>> 
>> Other questions:
> First, the list of supported formats and their respective definition should be in a separate document (updated more often than the core spec).

+1

For the chairs, if we are splitting an intention of one document (Matroska) into several documents (Matroska, Matroska Encoding Registry, Matroska Metadata Registry) do we need to update the charter?

> And yes, each new format should be registered, because there is an unique codec name namespace. Maybe defining "X_" prefix as private (but mime RFC now discourages use of "x-", so maybe not a good idea).

+1

> Now, the question is where to define CodecPrivate. e.g. for FFV1 we already defined it in the FFV1 spec but we can not update AVC specs for that.
> 
>> [...]
>> 
> Matroska-centric validation tools should not validate raw streams, that includes CodecPrivate part. This is the role of a more general validation tool (a tool described as "validation of Matroska files" would not validate the CodecPrivate part, a tool described as "validation of Matroska files + FFV1 and FLAC" would use the description of how FFV1 and FLAC are registered)

There are still some tests permitted by only the Matroska specification. For instance with a revision we could make it clear which Codec ID MUST or MUST NOT have an associated `CodecPrivate Element`.

Dave Rice

---

From: Tessa Fallon <[email-masked]>
Date: Sun, 25 Sep 2016 21:00:30 -0400

Re: multiple documents, I am not 100% sure and defer to those with more
experience at the IETF to indicate if there is precedent; if it is not part
of the core spec, then it/they could be published as informational drafts.
However, before that I would ask for group consensus to ensure that this
fragmentation is absolutely necessary, and that the implications are fully
outlined and understood.

Also, I'd ask for it to be made clear why, if the items in question are not
part of the core spec, they merit publication under the CELLAR charter, as
we'd be adding two more documents to our already substantial list of
documents to be published.

Best

Tessa

On Sep 25, 2016 17:37, "Dave Rice" <[email-masked]> wrote:

> >> The terms ‘private data’ and the `CodecPrivate Element` seems to be
> referenced in the same way. Are these the same things? If so, they should
> be made consistent.
> >
> only known to the codec." but right, maybe we need to add more information
> about it.
>
> If CodecPrivate is referenced by nearly every definition of an encoding in
> Matroska, perhaps we should split the ‘Description’ into ‘CodecPrivate
> Usage’ and ‘Description’.
>
> >> The phrase "The private data is void.” is used several times in the
> document. Does this mean that the `CodecPrivate Element` is stored but as
> an `Empty Element`? Could the `CodecPrivate Element` just not be present if
> not needed?
> >
> >> Should the Matroska specification provide enough info for a user to
> independently define support for an encoding in Matroska? Or should codec
> support somehow be ‘registered’ via Matroska and/or CELLAR?
> >
> For the chairs, if we are splitting an intention of one document
> (Matroska) into several documents (Matroska, Matroska Encoding Registry,
> Matroska Metadata Registry) do we need to update the charter?
>
> >> For a Matroska validation that finds something like V_MPEG4/ISO/AVC in
> Matroska. Should it be considered as unsupported (most MKV files!)?
> >
> includes CodecPrivate part. This is the role of a more general validation
> tool (a tool described as "validation of Matroska files" would not validate
> the CodecPrivate part, a tool described as "validation of Matroska files +
> FFV1 and FLAC" would use the description of how FFV1 and FLAC are
> registered)
>
> There are still some tests permitted by only the Matroska specification.
> For instance with a revision we could make it clear which Codec ID MUST or
> MUST NOT have an associated `CodecPrivate Element`.
>
> Dave Rice
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>