Subject: [Cellar] Defining EBML Schema in the EBML spec & mandatory/default

From: Dave Rice <[email-masked]>
Date: Sat, 5 Dec 2015 02:07:16 -0500

Hi all, but particularly calling out Michael Bradshaw and Steve Lhomme for a review,

I just submitted a pull request to the EBML Specification here <https://github.com/Matroska-Org/ebml-specification/pull/40>  The intent is to offering a description within the EBML Specification regarding to how make a machine-readable definition of an EBML Document Type. This is a bit analogous to saying how to make something like an XML Schema for the Matroska or webM types of EBML Documents.

The current machine-readable spec of Matroska is available here <https://github.com/Matroska-Org/foundation-source/blob/3414f292bb5b1ac5ee2fd99976b7274fd81e48ee/spectool/specdata.xml>  I hope to soon propose changes to this to align it more so with XML Schema practices (i.e. use attributes like minOccurs/maxOccurs instead of the current mandatory/multiple attributes as well as nesting element hierarchically). For now I think it would be best to start with this pull request to start a definition of how to express specifications for EBML DocTypes and get it in alignment with that current Matroska implementation of that.

In this commit <https://github.com/MediaArea/ebml-specification/commit/c16d8943b8b9308ccbf5e4c172677e67ecbfddec>  I removed the `EBML Template` section which was a list of attributes to use to define an EBML Element with very brief descriptions. This is replaced by a few paragraphs to introduce the concept of an EBML Schema and then a table of attributes to define EBML Elements and the definitions of those attributes. The table is a little hard to read in the pull request but there is a rendered copy of the table here <https://github.com/MediaArea/ebml-specification/blob/c16d8943b8b9308ccbf5e4c172677e67ecbfddec/specification.markdown#ebml-schema-element-attributes>.

In another commit <https://github.com/MediaArea/ebml-specification/commit/4d9606b886fbd92f6a7a994d122b375a46a43325> I added a more detailed description of the associated requirements of EBML Element usage that are conveyed by various combinations of the default and mandatory element. Since this topic has been long discussed and a source of confusion, I’d like to be precise here. There seem to be three factors involved regarding if a mandatory element should be written to a file or not (is there a default value declared?, is the value equal to the default?, and is the parent element used?). I eventually ended up making a lookup table <https://github.com/MediaArea/ebml-specification/blob/4d9606b886fbd92f6a7a994d122b375a46a43325/specification.markdown#note-on-the-use-of-mandatory-and-default-attributes-to-define-ebml-elements> to say if the Element MUST be written or not.

Feedback requested.

Thanks much,
Dave Rice

---

From: Michael Bradshaw <[email-masked]>
Date: Sat, 5 Dec 2015 09:24:58 -0800

Overall I like it. I've added some questions/nits to the GitHub pull
request.

On Fri, Dec 4, 2015 at 11:07 PM, Dave Rice <[email-masked]> wrote:

> I just submitted a pull request to the EBML Specification here
> <https://github.com/Matroska-Org/ebml-specification/pull/40>  The intent
> is to offering a description within the EBML Specification regarding to how
> make a machine-readable definition of an EBML Document Type. This is a bit
> analogous to saying how to make something like an XML Schema for the
> Matroska or webM types of EBML Documents.
>
> The current machine-readable spec of Matroska is available here
> <https://github.com/Matroska-Org/foundation-source/blob/3414f292bb5b1ac5ee2fd99976b7274fd81e48ee/spectool/specdata.xml>
> I hope to soon propose changes to this to align it more so with XML Schema
> practices (i.e. use attributes like minOccurs/maxOccurs instead of the
> current mandatory/multiple attributes as well as nesting element
> hierarchically). For now I think it would be best to start with this pull
> request to start a definition of how to express specifications for EBML
> DocTypes and get it in alignment with that current Matroska implementation
> of that.
>
> In this commit
> <https://github.com/MediaArea/ebml-specification/commit/c16d8943b8b9308ccbf5e4c172677e67ecbfddec>
> I removed the `EBML Template` section which was a list of attributes to use
> to define an EBML Element with very brief descriptions. This is replaced by
> a few paragraphs to introduce the concept of an EBML Schema and then a
> table of attributes to define EBML Elements and the definitions of those
> attributes. The table is a little hard to read in the pull request but
> there is a rendered copy of the table here
> <https://github.com/MediaArea/ebml-specification/blob/c16d8943b8b9308ccbf5e4c172677e67ecbfddec/specification.markdown#ebml-schema-element-attributes>
> .
>
> In another commit
> <https://github.com/MediaArea/ebml-specification/commit/4d9606b886fbd92f6a7a994d122b375a46a43325> I
> added a more detailed description of the associated requirements of EBML
> Element usage that are conveyed by various combinations of the default and
> mandatory element. Since this topic has been long discussed and a source of
> confusion, I’d like to be precise here. There seem to be three factors
> involved regarding if a mandatory element should be written to a file or
> not (is there a default value declared?, is the value equal to the
> default?, and is the parent element used?). I eventually ended up making a
> lookup table
> <https://github.com/MediaArea/ebml-specification/blob/4d9606b886fbd92f6a7a994d122b375a46a43325/specification.markdown#note-on-the-use-of-mandatory-and-default-attributes-to-define-ebml-elements> to
> say if the Element MUST be written or not.
>
> Feedback requested.
>
> Thanks much,
> Dave Rice
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>
>

---

From: Dave Rice <[email-masked]>
Date: Sat, 5 Dec 2015 13:58:42 -0500

Hi Michael,

> On Dec 5, 2015, at 12:24 PM, Michael Bradshaw <[email-masked]> wrote:
> 
> Overall I like it. I've added some questions/nits to the GitHub pull request.

Thanks for finding the typos.I asked this question via the discussion on the pull request <https://github.com/Matroska-Org/ebml-specification/pull/40>, but should ask here to:

The specdata.xml <https://github.com/Matroska-Org/foundation-source/blob/3414f292bb5b1ac5ee2fd99976b7274fd81e48ee/spectool/specdata.xml> file is the only precedent expression of an EBML Schema that I know of. Is there a machine readable expression of webm anywhere? Or any other EBML Document Type?

Considering your points about nailing down the definition of range, there are only a few range values in use in specdata.xml, “> 0”, “>0”, “0-1”, “1-254”, and “not 0”, I made some notes of a definition of these in a pull request comment <https://github.com/Matroska-Org/ebml-specification/pull/40#discussion_r46761573>, but propose that ultimately we move the definition of `range` towards the XML Schema equivalent of `restriction` <http://www.w3schools.com/Xml/schema_facets.asp>.

Regarding the conflicts of the spec regarding the meaning of Empty Elements versus how the default value applies to an Empty Element, I remember this discussion but forgot about it while drafting. I’ll add this in shortly, it will be nice to finally have rules about default/mandatory/empty all centralized as the discussions and clues have been widespread.

Kind Regards,
Dave Rice

> On Fri, Dec 4, 2015 at 11:07 PM, Dave Rice <[email-masked] <mailto:[email-masked]>> wrote:
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/cellar <https://www.ietf.org/mailman/listinfo/cellar>
> 
>

---

From: Michael Bradshaw <[email-masked]>
Date: Mon, 7 Dec 2015 09:37:42 -0800

On Sat, Dec 5, 2015 at 10:58 AM, Dave Rice <[email-masked]> wrote:
>
> The specdata.xml
> <https://github.com/Matroska-Org/foundation-source/blob/3414f292bb5b1ac5ee2fd99976b7274fd81e48ee/spectool/specdata.xml> file is
> the only precedent expression of an EBML Schema that I know of. Is there a
> machine readable expression of webm anywhere? Or any other EBML Document
> Type?
>

Not that I know of (there certainly isn't anything normative like that for
WebM). I'm really looking forward to a standardized machine readable form,
because it'll help keep the WebM and MKV specs consistent. Right now
they've fallen out of sync (i.e. WebM marks several elements as supported,
like BlockVirtual, BlockAdditions, MaxBlockAdditionID, etc that aren't
marked as supported in WebM within the MKV spec).

Considering your points about nailing down the definition of range, there
> are only a few range values in use in specdata.xml, “> 0”, “>0”, “0-1”,
> “1-254”, and “not 0”, I made some notes of a definition of these in a pull
> request comment
> <https://github.com/Matroska-Org/ebml-specification/pull/40#discussion_r46761573>,
> but propose that ultimately we move the definition of `range` towards the
> XML Schema equivalent of `restriction`
> <http://www.w3schools.com/Xml/schema_facets.asp>.
>

Something along the lines of XML Schema restriction would probably be nice,
because then you could use it for strings and the like.

> Regarding the conflicts of the spec regarding the meaning of Empty
> Elements versus how the default value applies to an Empty Element, I
> remember this discussion but forgot about it while drafting. I’ll add this
> in shortly, it will be nice to finally have rules about
> default/mandatory/empty all centralized as the discussions and clues have
> been widespread.
>

Great, looking forward to that. I'll review that when it's up.

---

From: Dave Rice <[email-masked]>
Date: Mon, 7 Dec 2015 13:04:57 -0500

Hi Michael,

> On Dec 7, 2015, at 12:37 PM, Michael Bradshaw <[email-masked]> wrote:
> 
> On Sat, Dec 5, 2015 at 10:58 AM, Dave Rice <[email-masked] <mailto:[email-masked]>> wrote:
> Not that I know of (there certainly isn't anything normative like that for WebM). I'm really looking forward to a standardized machine readable form, because it'll help keep the WebM and MKV specs consistent. Right now they've fallen out of sync (i.e. WebM marks several elements as supported, like BlockVirtual, BlockAdditions, MaxBlockAdditionID, etc that aren't marked as supported in WebM within the MKV spec).
> 
> Something along the lines of XML Schema restriction would probably be nice, because then you could use it for strings and the like.

One caution I have here is that the current specdata.xml file is used to create libraries that are used to build mkvalidator. I think possibly a different specdata.xml could be used to create a validator from mkvalidator code for another EBML format. However, mkvalidator likely does not support much outside of the current `range` values. If we use something broader like XML Schema's restriction we could end up making EBML Schemas that can not be used within mkvalidator. On the other hand, support for string restrictions would allow us to have a machine-readable test on Matroska tags, whereas we 

If we standardize EBML Schema, I think it's practically essential to have a utility that can use the EBML Schema to validate an EBML Document. To do that there is mkvalidator and inside the PreForma project there is work underway to build an EBML validator using a combination of MediaTrace and XSL.

Cc'ing matroska-devel on this, but I like to ask:
If EBML Schema is extended to support more comprehensive expressions that specdata.xml currently supports, are there volunteers to update mkvalidator to support the new EBML Schema? I can volunteer to update data2spec to produce a similar output but feasibly our work on EBML Schema may mean that it then continues unpredicted expressions.

An alternate could be to say that EBML Schema has X features but that the Matroska specification adds T constraints onto the implementation of the schema, so that the only Matroska EBML Schema is semantically equivalent to specdata.xml

> Great, looking forward to that. I'll review that when it's up.

I'll probably do this by adding another paragraph and then another column that that lookup table that I started.
Best Regards,
Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 8 Dec 2015 10:15:36 +0100

2015-12-07 18:37 GMT+01:00 Michael Bradshaw <[email-masked]>:
>> The specdata.xml file is the only precedent expression of an EBML Schema
>> that I know of. Is there a machine readable expression of webm anywhere? Or
>> any other EBML Document Type?
>
>
> because it'll help keep the WebM and MKV specs consistent. Right now they've
> fallen out of sync (i.e. WebM marks several elements as supported, like
> BlockVirtual, BlockAdditions, MaxBlockAdditionID, etc that aren't marked as
> supported in WebM within the MKV spec).

We might take this in consideration in the XML Schema definition: how
to we define different formats based on the same core of fields. That
applies to Matroska versions as well.

I do not have an answer to that.

>> request comment, but propose that ultimately we move the definition of
>> `range` towards the XML Schema equivalent of `restriction`.
>
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 8 Dec 2015 10:20:06 +0100

2015-12-07 19:04 GMT+01:00 Dave Rice <[email-masked]>:
> One caution I have here is that the current specdata.xml file is used to
> create libraries that are used to build mkvalidator. I think possibly a
> different specdata.xml could be used to create a validator from mkvalidator
> code for another EBML format. However, mkvalidator likely does not support
> much outside of the current `range` values. If we use something broader like
> XML Schema's restriction we could end up making EBML Schemas that can not be
> used within mkvalidator. On the other hand, support for string restrictions
> would allow us to have a machine-readable test on Matroska tags, whereas we
>
> If we standardize EBML Schema, I think it's practically essential to have a
> utility that can use the EBML Schema to validate an EBML Document. To do
> that there is mkvalidator and inside the PreForma project there is work
> underway to build an EBML validator using a combination of MediaTrace and
> XSL.

If we go the XSLT way, I suppose a more flexible validator could be
written using that and be easily updated on the fly, compared to a
compile program like mkvalidator.

> If EBML Schema is extended to support more comprehensive expressions that
> specdata.xml currently supports, are there volunteers to update mkvalidator
> to support the new EBML Schema? I can volunteer to update data2spec to
> produce a similar output but feasibly our work on EBML Schema may mean that
> it then continues unpredicted expressions.

I will continue maintaining mkvalidator, until another validator
supports all the same error/warning detections and profiles (live,
webm, divx).

> An alternate could be to say that EBML Schema has X features but that the
> Matroska specification adds T constraints onto the implementation of the
> schema, so that the only Matroska EBML Schema is semantically equivalent to
> specdata.xml
>

---

From: Dave Rice <[email-masked]>
Date: Tue, 8 Dec 2015 08:17:04 -0500

> On Dec 8, 2015, at 4:20 AM, Steve Lhomme <[email-masked]> wrote:
> 

[…]

> supports all the same error/warning detections and profiles (live,

Thanks Steve.

I realize that another approach we could use is to convert EBML to XML and then validate the resulting XML with an XML Schema that defines an EBML format. This would means that we could use any XML validator instead of necessarily needing a custom EBML validator. There is already an EBML<->XML converter here https://github.com/vi/mkvparse/. An EBML validator could work like this:

mkv2xml < test.mkv > output.xml
xmllint --noout --schema output.xml

I note that mkvalidator has some tests that are analogous to XML validation against an XML Schema, but some are rules more similar to Schematron. I’ll have to see if these rules could possibly to stored properly in an XML Schema to verify if we’d need a second schematron-like test.

Ideas? re: Updating mkvalidator to validate an EBML Dcoument validator against an EBML Schema. Or. Using existing XML validation tools with an EBML->XML conversion?
Dave

---

From: Dave Rice <[email-masked]>
Date: Thu, 10 Dec 2015 08:02:31 -0500

Hi all,

> On Dec 8, 2015, at 8:17 AM, Dave Rice <[email-masked]> wrote:
> 
> 

This conversation continued in GitHub on the pull request here https://github.com/Matroska-Org/ebml-specification/pull/40. Moritz merged the PR though some issues remain for another one, such as defining how to express EBML Element defaults when the default value may be a string, integer, or reference to another Element’s value.

The new merge adds a section on defining an EBML Schema, https://github.com/Matroska-Org/ebml-specification/blob/master/specification.markdown#ebml-schema. The current in-use EBML Schema is specdata.xml which is here https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml. Now that we have a definition for specdata.xml, I suggest we draft a new PR that updates the EBML Schema definition to bring it into as much alignment with XML Schema’s specification as we can. Alongside this i can update the XSL that converts the EBML Schema into spec.xml which is used for documentation in the matroska.org website and as a definition for use in mkvalidator.

I think the EBML Specification is getting close to complete. There are some open issues here, https://github.com/Matroska-Org/ebml-specification/issues, some of which relate to missing pieces of the spec and sections needed to bring the spec moreso towards RFC style (adding a Security Consideration section). Volunteers?

Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 17 Dec 2015 09:07:53 +0100

> I note that mkvalidator has some tests that are analogous to XML validation against an XML Schema, but some are rules more similar to Schematron. I’ll have to see if these rules could possibly to s…

I think there can be more than one validator. One using XML is
certainly useful. On the other hand, given the tight binary nature of
EBML, it may produce very large files. But it will probably be limited
to basic EBML. For example for Matroska it would be tricky to verify
Cues refer to proper Clusters/frames without being able to seek in the
XML file or keep everything in memory. The C based validator is being
"smart" about this and keeps things in memory depending on the mode
it's being used. It also uses the tight in-memory representation of an
EBML element. Whereas an XML parser will likely use a lot more memory
for each possible named element.

> Ideas? re: Updating mkvalidator to validate an EBML Dcoument validator against an EBML Schema. Or. Using existing XML validation tools with an EBML->XML conversion?

That would be a nice goal. But as described above, some further checks
may not be possible with such an approach.