Subject: Re: [Cellar] [Matroska-devel] EBML Schema

From: Dave Rice <[email-masked]>
Date: Tue, 1 Dec 2015 10:40:22 -0500

Hi,

> On Nov 9, 2015, at 1:19 PM, Dave Rice <[email-masked]> wrote:
> 
> Hi all,
> 
>> On Oct 3, 2015, at 9:46 AM, Steve Lhomme <[email-masked] <mailto:[email-masked]>> wrote:
>> On Aug 28, 2015 17:00, "Dave Rice" <[email-masked] <mailto:[email-masked]>> wrote:
>> >
>> > Hi,
>> >
>> >> On Aug 28, 2015, at 2:50 AM, Moritz Bunkus <[email-masked] <mailto:[email-masked]>> wrote:
>> >>
>> >> Hey,
>> >>
>> >> I have no objections, however I don't know a lot about XML schemas in
>> >> the first place (neither about DTDs, to be honest).
>> >
>> >
>> > Honestly, I know a lot more about XML Schemas than I do about DTDs. As wikipedia mentions at https://en.wikipedia.org/wiki/Document_type_definition <https://en.wikipedia.org/wiki/Document_type_definition>, DTDs have largely been superseded by XML Schemas. And at this point I think that XML Schemas may be a more familiar analogy to use.
>> >
>> > I think XML Schemas also share more in common with specdata.xml than DTDs do. Schemas use the <element> node and have maxOccurs and minOccurs attributes (specdata has semantically the same thing with mandatory and multiple), they both have a similar declaration of element type, element name and element description. Actually I think a semantically equivalent version of specdata.xml could be written as an XML Schema.
>> >
>> > XML Schemas also offer a few advantages for machine readable expressions; for instance XML Schemas can mandate a particular pattern or regex for a value. 
>> >
>> >>> I propose the specdata.xml file here
>> >>> https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml <https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml>
>> >>> <https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml <https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml>>
>> >>> is a good basis for the consideration of an EBML Schema. From what I
>> >>> can see, specdata.xml is an expression of the EBML + Matroska
>> >>> specifications to support automated creation of documentation, but the
>> >>> structure of this already shares a lot of similarity to XML Schemas.
>> >>
>> >>
>> >> For both documentation (e.g. the table on the matroska.org <http://matroska.org/> specs page is
>> >> generated from this file) and code (libMatroska's class hierarchy is
>> >> generated automatically from this file) actually.
>> >
>> >
>> > Does specdata.xml play a role in mkvalidate? I'm thinking of the potential to have an ebmlvalidator where you can provide the EBML Schema to validate particular EBML docType.
>> 
>> Well the parsing code is generated from the XML file, so in a way, yes. But it's not parsed "live".
>> 
>> >>> Is there a preference in handling the standardization of Matroska:
>> >>> documenting it in a similar fashion to our work in the EBML spec or to
>> >>> define what an EBML Schema is and consider matroska an expression of
>> >>> it?
>> >>
>> >>
>> >> I'm not sure whether or not I understand the implications. But my gut
>> >> feeling is that having a definition for an EBML Schema would benefit
>> >> other formats than Matroska, too, therefore the latter seems the way to
>> >> go.
>> >
>> >
>> > I have the same feeling:
>> > - document EBML as a specification that includes rules for defining a docType in the form of an EBML Schema
>> > - write an EBML Schema (updated specdata.xml) for Matroska and maybe webM
>> >
>> >>> Are some changes to specdata.xml acceptable? Such as a filename change
>> >>> or changing the name of the <table> element of some attributes?
>> >>
>> >>
>> >> Well, like I said above the specdata.xml is used for generating both
>> >> documentation and code. Both should stay viable. If changes to it are
>> >> made then the accompanying tools must be updated as well.
>> >>
>> >>> Neither the current EBML specs nor the specdata.xml specifically refer
>> >>> to the hierarchical arrangement of the elements, but this could be
>> >>> presumed by their ordering. For instance, could any level 3 element be
>> >>> a child of any level 2 Master-element? I presume not, but I don't
>> >>> think it's clear anywhere what parent-child relationships are
>> >>> feasible. Possibly specdata.xml and/or the EBML Schema Definition
>> >>> could define the relationship between levels of related elements
>> >>> similar to how an XML Schema (XSD) does.
>> >>
>> >>
>> >> So far it is understood that an element not marked as a global element
>> >> must only occur as a child of its parent. Its parent is the last element
>> >> located before the child element in the specdata file with a lower level
>> >> than the child element. Or something like that.
>> >
>> >
>> > This will need some documentation. That's how I've understood the mkv spec as well but the definition for how an EBML Schema works should be explicit about this.
>> 
> Any more opinion about how to go about (or if to go about) modifying specdata.xml towards becoming an expression of a to-be-defined EBML Schema for matroska and webm? As a summary of proposed changes to specdata.xml
> 
> - change to XML Schema conventions where relevant:
> 		- use maxOccurs attribute instead of the current Multiple attribute.
> 		- use minOccurs attribute instead of the current Mandatory attribute.
> 		- move documentation of elements to a sub-element (allows for possible internationalization in the schema and better semantics)
> - arrange elements in hierarchical form to indicate parent-child relationships (rather than the current practices where all elements are defined at the same level, and you have to parse back in elements to the one with the lower-numbered level attribute to find the parent)
> 
> A draft of specdata.xml with these changes is at https://gist.github.com/dericed/f0a4bb0e7dc635ed1347 <https://gist.github.com/dericed/f0a4bb0e7dc635ed1347>. I can continue to work on this and send back changes for advice/approval but if I do so is there someone who could later update the tools that use specdata.xml so that newly-defined EBML Schemas may later to be into use?

I’m preparing a pull request on specdata.xml but want to update the utilities in spectool at the same time so that spec2data, data2lib, and data2spec still function properly. I’m having trouble getting the spectool utilities to build properly so that I can test them. I was able to build coremake but not sure where to go from here. I’ve read spec2data and have an idea of how it works and am thinking that I could reproduce the spec2data workflow with xsl and then we could have a make file which uses xsltproc to convert an EBML Schema into the Drupal table and library files as needed. Any advice on which route to take: continue trying with getting spectool utilities to build or redo the utilities in xsl?

Best Regards,
Dave Rice

> btw I'm cc'ing the newly-established CELLAR listserv which focuses on work on the EBML and Matroska specification. If you are interested in these topics please considering subscribing at https://www.ietf.org/mailman/listinfo/cellar <https://www.ietf.org/mailman/listinfo/cellar>.
> 
> Best Regards,
> Dave Rice
> 
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Moritz Bunkus <[email-masked]>
Date: Tue, 1 Dec 2015 17:01:03 +0100

Hey,

> I’m preparing a pull request on specdata.xml but want to update the
> utilities in spectool at the same time so that spec2data, data2lib,
> and data2spec still function properly. I’m having trouble getting the
> spectool utilities to build properly so that I can test them.

All of that is Steve's code and I don't know a lot about coremake and
the assorted build process (and there's zero documentation), but I do
manage to get it to compile most of the time ;)

From a fresh checkout of the foundation repository[1] in ~/foundation:

--------------------

1. Build and install coremake:

cd ~/foundationcorec/tools/coremake
make
make install

Unfortunately coremake has /usr/local/share/coremake hardcoded on Linux
for looking up its include files; therefore the "make install" is
actually really necessary.

2. Configure the rest of the project with coremake:

cd ~/foundation
coremake gcc_linux_x64

3. Compile the tools (mkvtree, mkvalidator, mkclean; optional):

cd ~/foundation
make

4. Compile the programs in spectool:

cd ~/foundation
make -C spectool

5. Use the programs:

# This creates spec.xml from specdata.xml. spec.xml is basically the
# HTML table you see on www.matroska.org/technical/specs/:

cd ~/foundation/spectools
../release/gcc_linux_x64/data2spec

--------------------

> I was able to build coremake but not sure where to go from here.

The whole foundation repo could use a serious rewrite of its build
system. Steve had reasons why he created coremake, but the result is
very, very complicated to use. If you (or anyone else) want to give such
a rewrite a try I'd highly welcome it.

> Any advice on which route to take: continue trying with getting
> spectool utilities to build or redo the utilities in xsl?

I'd prefer established technologies like XSLT in this case as it allows
you to use a plethora of proven tools like xsltproc or any other XSLT
processor (Saxon?). You can (and should) still build the tools with the
instructions above so that you verify stuff works :)

Kind regards,
mosu

[1] https://github.com/Matroska-Org/foundation-source/

---

From: Dave Rice <[email-masked]>
Date: Thu, 3 Dec 2015 18:50:29 -0500

Thanks Moritz,
> On Dec 1, 2015, at 11:01 AM, Moritz Bunkus <[email-masked]> wrote:
> 
> Hey,
> 
> ——————————

These instructions worked well. To help the next person who happens along, I converted your notes into a basic README which I sent as a pull request here: https://github.com/Matroska-Org/foundation-source/pull/6. 

> a rewrite a try I'd highly welcome it.

I am interested in the foundation repo since it is the home of the Matroska instance of the basis of the conceptual EBML Schema (specdata.xml). To work on moving the current state of specdata.xml towards a more standardized EBML Schema expression I should synchronize the work on specdata.xml to the spectools so that the foundation repo doesn’t break because of IETF work.

The rest of the foundation repo may be out of scope to the charter. There is obviously a lot of documentation on Matroska use on matroska.org but that appears to be based in your Drupal site rather than this repo.

I think we still need a good method to handle linking pull requests, comments, and discussion upon the Matroska documentation. Drupal makes this a bit difficult. Should we move some of this documentation to GitHub as well and have a similar process as spec.xml with the other documentation (i.e. build it from a repo and copy/paste it to Drupal). Or potentially we could move some Matroska documentation from Drupal to GitHub gh-pages as done with the EBML Specification website (formerly on sourceforge).

> instructions above so that you verify stuff works :)

This is verified. If XSLT is welcome I certainly think it’s much more familiar.
Thanks,
Dave Rice

---

From: Ashley Blewer <[email-masked]>
Date: Fri, 4 Dec 2015 22:34:06 -0500

I want to second encouragement for a move off of a platform that doesn't
support open source collaboration (Drupal site) and into something that
does (GitHub Readme and/or gh-pages). I'm also happy to volunteer the time
& effort to making this happen.

My best,
Ashley Blewer

On Thu, Dec 3, 2015 at 6:50 PM, Dave Rice <[email-masked]> wrote:

> These instructions worked well. To help the next person who happens along,
> I converted your notes into a basic README which I sent as a pull request
> here: https://github.com/Matroska-Org/foundation-source/pull/6.
>
> I am interested in the foundation repo since it is the home of the
> Matroska instance of the basis of the conceptual EBML Schema
> (specdata.xml). To work on moving the current state of specdata.xml towards
> a more standardized EBML Schema expression I should synchronize the work on
> specdata.xml to the spectools so that the foundation repo doesn’t break
> because of IETF work.
>
> The rest of the foundation repo may be out of scope to the charter. There
> is obviously a lot of documentation on Matroska use on matroska.org but
> that appears to be based in your Drupal site rather than this repo.
>
> I think we still need a good method to handle linking pull requests,
> comments, and discussion upon the Matroska documentation. Drupal makes this
> a bit difficult. Should we move some of this documentation to GitHub as
> well and have a similar process as spec.xml with the other documentation
> (i.e. build it from a repo and copy/paste it to Drupal). Or potentially we
> could move some Matroska documentation from Drupal to GitHub gh-pages as
> done with the EBML Specification website (formerly on sourceforge).
>
> Matroska-devel mailing list
> [email-masked]
> http://lists.matroska.org/cgi-bin/mailman/listinfo/matroska-devel
> Read Matroska-Devel on GMane:
> http://dir.gmane.org/gmane.comp.multimedia.matroska.devel
>

---

From: Moritz Bunkus <[email-masked]>
Date: Sat, 5 Dec 2015 09:38:47 +0100

Hey,

> does (GitHub Readme and/or gh-pages).

I'm perfectly fine with this.

Kind regards,
mosu

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 8 Dec 2015 10:12:14 +0100

> The rest of the foundation repo may be out of scope to the charter. There is obviously a lot of documentation on Matroska use on matroska.org but that appears to be based in your Drupal site rather…

If you're referring to the documentation in the HTML tables on
matroska.org, they are taken from the specdata.xml after the HTML is
generated, not the other way around.

> I think we still need a good method to handle linking pull requests, comments, and discussion upon the Matroska documentation. Drupal makes this a bit difficult. Should we move some of this documen…

I agree, it will be much easier to do things on Github.

> This is verified. If XSLT is welcome I certainly think it’s much more familiar.

Not to me, but it shouldn't too hard to adapt the spec tools to match
field name changes at the same time. If XSLT can also generate the
C/C++ code matching the semantics, then it's great. That should make
it easier for people to create their own EBML format and play with the
code while defining it.

-- 
Steve Lhomme
Matroska association Chairman