Subject: [Cellar] EBML RFC changed semantics of "default values" and therefore a lot of existing Matroska files

From: Moritz Bunkus <[email-masked]>
Date: Fri, 05 Feb 2021 21:07:11 +0100

Hey,

I've posted this as an issue[1] in the EBML specification repo, but I feel
it'll get more attention and more input here; hence reposting.

TLDR: existing Matroska files where elements such as the track language
haven't been written need to be interpreted differently according to the
new EBML RFC (before the RFC: track language missing means it's "eng";
after the RFC: missing means it isn't known).

I've verified using the Wayback Machine[2] that right before the website
was updated to match the released RFC the semantics of default values was
as follows:

> Default Values
>
> The default value of an element is assumed when not present in the data
> stream. It is assumed only in the scope of its upper-element (for example
> Language in the scope of the Track element). If the upper element is not
> present or assumed, then the element cannot be assumed.

Note that there is no distinction made for whether an element is mandatory
or not.

Now the RFC turns this into the following:

> 6.1.  Data Size Format
>
> …
> If an Empty Element has a default value declared, then the EBML Reader
> MUST interpret the value of the Empty Element as the default value.

and

> 11.1.6.4.  minOccurs
>
> …
> EBML Elements with "minOccurs" set to "1" that also have a default value
> (see Section 11.1.6.8) declared are not REQUIRED to be stored but are
> REQUIRED to be interpreted; see Section 11.1.19.

and the whole section 11.1.19, which only talks about mandatory elements.

So to recap:

* Before the RFC all elements with default values would not have be written
  to the stream.
* After the RFC non-mandatory elements are suddenly handled differently.

This is not just theoretical. Pretty much all Matroska readers out there
treat mandatory and non-mandatory elements the same wrt. default
values. Most popular example is the track language element which is
non-mandatory with a default value of `eng`. Software such as MKVToolNix,
VLC, ffmpeg, mkclean all use the value `eng` if the element isn't present.

In fact, our own published libEBML doesn't handle writing such elements
like the RFC describes either; instead it handles them according to the old
semantics. Here[3] is the issue I filed for it.

The RFC retroactively changes the meaning of nearly all files ever
created. No matter what we might think about how default values should be
handled, such a radical, retroactive change is unacceptable in my
opinion. Unfortunately I didn't realize this during the RFC process, so
yeah, mea culpa — I fucked this up, too.

mkver remarked that this can be remedied somewhat by changing the Matroska
specification for the affected elements and simply making them mandatory,
thereby bringing their semantics back to what they were before the RFC.

However, that won't work for frozen versions of the Matroska specs
(v1–3). Also: it is hella confusing for developers who were used to
mandatory elements working one way for 18 years and now suddenly having to
change their code.

For example, here[4] is me getting confused about the new semantics in our
recent discussion about the semantics of the new track header elements
('hearing impaired' flag etc.).

On the flip side I cannot really think of an advantage of treating
non-mandatory elements with default values differently than mandatory
elements with default values.

So… am I way off base here? Did I misunderstand the RFC and the semantics
haven't actually changed? Did they change, and do we keep the semantics as
they are now? How do we communicate the change to the wider developer
community?

If not: how would we go about fixing such an issue (probably by… doing
another RFC obsoleting the existing one… this was my first RFC, not
familiar with the right process)?

I'm thankful for any kind of input & advice on this.

Kind regards,
mosu

[1]  https://github.com/ietf-wg-cellar/ebml-specification/issues/394
[2]  https://web.archive.org/web/20190630173942/https://matroska.org/technical/specs/notes.html
[3]  https://github.com/Matroska-Org/libebml/issues/73
[4]  https://github.com/ietf-wg-cellar/matroska-specification/issues/449