Subject: [Cellar] test4.mkv and EBML Elements with Unknown Size

From: Dave Rice <[email-masked]>
Date: Wed, 30 Dec 2015 19:09:45 -0500

Hi Steve and Moritz,

The Matroska Test file suite is at http://matroska.org/downloads/test_w1.html. This includes test4.mkv which is an example of a live stream recording. However the file doesn’t appear to adhere to the EBML specs. It the file written wrong or are we missing something in the EBML specification?

Within this file is this extract from the beginning of the Segment element to the end of the Element ID of the Info Element.

18 53 80 67 FF 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 15 49 A9 66

I understand that 0x18538067 is the Element ID of the Segment Element. 

The 0xFF represents the case where as the EBML draft spec states, "An Element Data Size with all VINT_DATA bits set to one is reserved as an indicator that the size of the Element is unknown.”

Next, between the Element Data Size of the Segment Element and the beginning of the Info Element there is 134 bytes of 0x0A! In this case 0x0A can not be a valid start of an Element ID since for this file the EBMLMaxIDLength is 4 (via the default). Thus the Segment Master-element has child data that is not an EBML Element.

Is the Segment Element here invalid since it contains immediate child data that is not a part of a child Element or should the EBML draft specification be updated to permit mystery data (sounds dangerous and complex)? Is there any documentation existing on this scenario? Should a parser keep reading from byte to byte until finding a valid Element ID again?

Dave Rice

---

From: Moritz Bunkus <[email-masked]>
Date: Thu, 31 Dec 2015 09:33:24 +0100

Hey,

> The Matroska Test file suite is at
> http://matroska.org/downloads/test_w1.html. This includes test4.mkv
> which is an example of a live stream recording. However the file
> doesn’t appear to adhere to the EBML specs. It the file written wrong
> or are we missing something in the EBML specification?
>
> Within this file is this extract from the beginning of the Segment
> element to the end of the Element ID of the Info Element.
>
> 18 53 80 67 FF 0A 0A 0A…

Uhm… this is indeed invalid. A master must only contain child elements,
not additional data. To put it differently: all child elements of a
master element must cover the whole space occupied by the master without
any gaps in between.

I honestly don't know that file was created. libmatroska contains
programs to create a couple of those files in its "tests" sub-directory,
but test4 is not among them. Judging from the download page you've
linked it's possible that the file was created with mkclean – in that
case I would say that mkclean produces non-compliant files.

Steve?

> The 0xFF represents the case where as the EBML draft spec states, "An
> Element Data Size with all VINT_DATA bits set to one is reserved as an
> indicator that the size of the Element is unknown.”

Correct.

> Next, between the Element Data Size of the Segment Element and the
> beginning of the Info Element there is 134 bytes of 0x0A! In this case
> 0x0A can not be a valid start of an Element ID since for this file the
> EBMLMaxIDLength is 4 (via the default). Thus the Segment
> Master-element has child data that is not an EBML Element.

Correct.

Like I said, invalid.

I definitely do not want to adjust the specs to allow arbitrary data in
arbitrary places that a parser has to be able to skip over.

Kind regards,
mosu

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 31 Dec 2015 11:51:30 +0100

Le 31/12/2015 09:33, Moritz Bunkus a écrit :
> A master must only contain child elements, not additional data. To put 
> it differently: all child elements of a master element must cover the 
> whole space occupied by the master without any gaps in between

I have seen a bunch of files (I'll try to find them again, in order to 
see the writing library) with a NULL byte between some elements.

Looking at the files I have right now, I found a file written by:
Writing application                      : mkvmerge v2.6.0 ('Kelly watch 
the Stars') built on Mar 24 2009 15:23:17
Writing library                          : libebml v0.7.7 + libmatroska 
v0.8.1
(so an "reference" application?)
with a NULL padding (1 byte 0x00 between "SeekHead" element and "Info" 
element).

I guess this is due to the fact that a "void" element needs at least 2 
bytes, so not possible to put one here.
should it be considered as not conform?
Maybe we should consider an exception to the rule "A master must only 
contain child elements, not additional data" with a single NULL byte as 
valid data and considered as a tiny "Void" element.

Hex dump (from "SeekHead" to "Info", see the 00 before 15 49 A9 66) :
11 4D 9B 74 BD 4D BB 8C 53 AB 84 16 54 AE 6B 53 AC 82 10 A5 4D BB 8D 53 
AB 84 11 4D 9B 74 53 AC 83 C5 50 27 4D BB 8D 53 AB 84 1C 53 BB 6B 53 AC 
83 C5 4E D3 4D BB 8B 53 AB 84 15 49 A9 66 53 AC 81 43 00 15 49 A9 66

---

From: Moritz Bunkus <[email-masked]>
Date: Thu, 31 Dec 2015 11:58:24 +0100

Hey,

> Writing application                      : mkvmerge v2.6.0 ('Kelly watch the
> Stars') built on Mar 24 2009 15:23:17

It's quite possible that older versions of mkvmerge (and definitely of
mkvpropedit the old GUI's header and chapter editors) did not handle
this case properly. Those were bugs, and today's versions should be
pretty great wrt. to this particular issue.

> bytes, so not possible to put one here.

Correct, but one can usually do and what mkvpropedit & the GUI's editors
do is to move the following element's ID and size element one byte to
the front. Additionally they extend the move element's size portion by
one byte (e.g. writing 0x40 0x04 for the size instead of 0x84).

> should it be considered as not conform?

Yes.

> contain child elements, not additional data" with a single NULL byte
> as valid data and considered as a tiny "Void" element.

I don't like exceptions. Just because it's a bit harder to get it right
doesn't mean it should be allowed.

The more exceptions there are the more conforming parsers have to
test. Yes, I know and like the principle of being liberal in what you
read & understand but conservative & strict in what you write. But that
also implies that we should encourage muxers to write good files;
allowing sloppy things like one zero byte between elements does the
opposite.

Kind regards,
mosu

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 31 Dec 2015 12:11:21 +0100

Le 31/12/2015 11:58, Moritz Bunkus a écrit :
> I don't like exceptions.

Me neither. but in that case, the "reference" muxer had this bug, so 
there are some files in the wild with such thing.

> Just because it's a bit harder to get it right doesn't mean it should 
> be allowed. The more exceptions there are the more conforming parsers 
> have to test. Yes, I know and like the principle of being liberal in 
> what you read & understand but conservative & strict in what you 
> write. But that also implies that we should encourage muxers to write 
> good files; allowing sloppy things like one zero byte between elements 
> does the opposite.

We got the same kind of issue with FFV1, and I wrote that:

bits_per_raw_sample:
value     bits for each luma and chroma sample
0     reserved*
Other     the actual bits for each luma and chroma sample
* Encoders MUST NOT store bits_per_raw_sample = 0
Decoders SHOULD accept and interpret bits_per_raw_sample = 0 as 8.

(this is due to previous reference encoder writing 0 instead of 8)
Maybe we could use the same policy for the RFC of Matroska v1 to v3 (or 
v1 only? I don't know, the file I get is v1 but I don't know when this 
bug was removed from the library), and explicitly forbid it in future v4 
only, and validators could provide a warning instead of an error for 
such files.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Thu, 31 Dec 2015 04:52:18 -0800

> also implies that we should encourage muxers to write good files;

I encourage people to read 
<https://tools.ietf.org/html/draft-thomson-postel-was-wrong>, although 
as an expired individual draft I do not claim this represents the 
official position of anyone at the IETF beyond its author.

---

From: Moritz Bunkus <[email-masked]>
Date: Thu, 31 Dec 2015 14:03:53 +0100

Hey,

thanks for pointing this out. "Be as strict as you can be" is roughly
what I had in mind.

Jerome pointed out that earlier files which contained e.g. that one-byte
gap were created by the reference muxer, mkvmerge, and that it would be
detrimental to consider such files invalid. That's why he proposed such
an exception.

However, the same reasoning could be applied to all the bugs mkvmerge
had in the past. Where would that end? What about bugs where mkvmerge
clearly violated the specs and where interpreting the data according to
the bug is incompatible with interpreting it according to the specs?

This cannot be solved properly, hence my vote for "no exception".

It's also clear that you cannot judge files by our upcoming Matroska RFC
specs that had been created before that RFC has been ratified. This
should probably be signalled somehow, for example by using the Matroska
version number. As v4 is already used in reality (even though we haven't
finalized it yet) bumping this to v5 for the RFC would be an option.

Kind regards,
mosu

---

From: "Sebastian G. <bastik>" <[email-masked]>
Date: Thu, 31 Dec 2015 16:20:35 +0100

31.12.2015, 11:58 Moritz Bunkus:

Retrospectively I now know where the trouble with test4.mkv may have
originated from.

Back in 2011 I throw the test suit against some players on the Windows
platform. None of the players could play the file. Most just did
nothing, VLC even crashed. Two started playing, but just for 2-5
seconds. Not even with Haalis' Media Splitter the players would play the
file.

I suggested to take a closer look at the test suit to at least one
players development team. I am unaware if anything got changed on how
the player would handle test4.mkv.

>> should it be considered as not conform?

mkvalidator is expected to complain, I assume. This would be true for
all tools that check if a file conforms to the specification.

Is mkclean expected to correct this?

> allowing sloppy things like one zero byte between elements does the

It does not seem to be correct and there does not seem to be a benefit
to it, so I agree to not having an exception.

I am under the impression that demuxers will not stop handling files
with that error. Well at least not for 'old' files, because it would
harm backward compatibility. Having them not handle (or have them raise
a flag) v5 matroska files with that error would be fine on the other hand.

Best Regards,
Sebastian