Subject: [Cellar] Matroska Elements to support frame side data

From: Dave Rice <[email-masked]>
Date: Thu, 1 Nov 2018 10:03:44 -0400

Hi all,

I’m looking for thoughts on pull request https://github.com/Matroska-Org/matroska-specification/pull/276 <https://github.com/Matroska-Org/matroska-specification/pull/276> for Matroska which addresses an issue described at https://github.com/Matroska-Org/matroska-specification/issues/270 <https://github.com/Matroska-Org/matroska-specification/issues/270>. This would add a new Master Element called “Metadata” within BlockGroup and the Metadata Element would contain a MetadataName which labels the content of either a MetadataString or MetadataBinary Element (this is a bit similar to the TagName/TagString/TagBinary set of elements). These elements would allow information to be associated directly with the contents of the Block.

I also considered Michael Niedermayer’s work in the nut specification at http://ffmpeg.org/~michael/nut.txt <http://ffmpeg.org/~michael/nut.txt> where a structure is defined to store side metadata per frame, see the section on "sm_data / side_data / meta_data”.

Here are a few use cases where I think this structure could be helpful:

=== timecode
We’ve made many starts and stops at defining how to store timecode in Matroska but haven’t found consensus or resolution on those attempts. With the Metadata Element I think a timecode value could be stored such as

Cluster/
	Timestamp=01:00:00.000
	BlockGroup/
		Block/
			...
		Metadata/
			MetadataName=“timecode"
			MetadataString="01:23:45;12”

The Metadata Element would here would be encoded as 0xD297D38874696D65636F6465D48B30313A32333A34353B3132 which adds 25 bytes per frame (about 2MB per hour for PAL).

Currently the decklink input device in libavdevice can pass along timecode as frame side metadata (see https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417 <https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417>) but Matroska offers no method to store such side data.

=== DPX -> MKV -> DPX lossless remuxing
When converting a DPX image sequence to FFV1/MKV the image data is losslessly stored but the DPX headers contain lots of information specific to DPX which is lost. The RawCooked project has an approach where that DPX header data is stored in Matroska attachments and then recalled and used when RawCooked is reverting from Matroska back to DPX (so the resulting DPX bitstream is identical as the input, similar to how flac can do this with wav->flac->wav. With this proposal the DPX header data could be stored within the Metadata Elements of the block group, such as:

Cluster/
	Timestamp=01:00:00.000
	BlockGroup/
		Block/
			...
		Metadata/
			MetadataName=“dpx_header"
			MetadataBinary=0x58504453…

Comments welcome.
Dave Rice

---

From: Tobias Rapp <[email-masked]>
Date: Fri, 2 Nov 2018 10:31:47 +0100

Hi Dave,

On 01.11.2018 15:03, Dave Rice wrote:
> [...]
> We’ve made many starts and stops at defining how to store timecode in 
> Matroska but haven’t found consensus or resolution on those attempts. 
> With the Metadata Element I think a timecode value could be stored such as
> 
> Cluster/
> Timestamp=01:00:00.000
> BlockGroup/
> Block/
> ...
> Metadata/
> MetadataName=“timecode"

while I think that frame side data can be useful for the DPX scenario 
mentioned later, I see problems with storing timecode information that 
way. Shall timecode be attached to the video or audio track? Shall there 
be some sub-classification for VITC/LTC/other timecode sources?

Also I would prefer the timecode value to be stored as a number instead 
of a string representation, for the same reason that the Cluster 
Timestamp element uses a numerical representation: It makes processing a 
lot easier.

Best regards,
Tobias

---

From: Dave Rice <[email-masked]>
Date: Sat, 3 Nov 2018 19:43:25 -0400

> On Nov 2, 2018, at 5:31 AM, Tobias Rapp <[email-masked]> wrote:
> 
> Hi Dave,
> 
> while I think that frame side data can be useful for the DPX scenario mentioned later, I see problems with storing timecode information that way. Shall timecode be attached to the video or audio track? Shall there be some sub-classification for VITC/LTC/other timecode sources?
> 
> Also I would prefer the timecode value to be stored as a number instead of a string representation, for the same reason that the Cluster Timestamp element uses a numerical representation: It makes processing a lot easier.

If the structure is sufficient then perhaps is better discussed in how to define the MetadataName values. For instance, we could define:
MetadataName=“timecode.s314m”
where MetadataBinary is set to a binary value as defined by SMPTE ST 314M-2005 Sec 4.4.2.2.1 "Time code pack (TC)”.

In other thoughts on this suggestion, I think it could make it difficult to easily understand if a file has a particular type of side data. For instance if only a few Clusters somewhere in the Segment contain a certain type of side data, it would require parsing every Cluster to know what types of side data are available. This uncertainly wouldn’t be the same issue if the side data was itself a Track.

Dave

---

From: Tobias Rapp <[email-masked]>
Date: Mon, 5 Nov 2018 10:07:11 +0100

> where MetadataBinary is set to a binary value as defined by SMPTE ST 314M-2005 Sec 4.4.2.2.1 "Time code pack (TC)”.

I should clarify: Storing the timecode value as a _single_ number makes 
processing a lot easier. For example when you export a sub-range of your 
archived video file and have to shift timecode offset in the output 
file. Or when you change the video frame-rate for some reason.

Both, the "01:23:45;12" timecode string and its binary representation 
are using some display-oriented format with mixed timebases 
(hours/minutes/seconds use a fixed timebase, framecount uses another 
timebase depending on the media). In our ingest system we use a separate 
XML file to store VITC/LTC values and first tried to store the SMPTE TC 
string but then switched to storing a microseconds number together with 
some hint for presentation "these values shall be rendered at 29.97 fps 
with drop-frame counting".

> In other thoughts on this suggestion, I think it could make it difficult to easily understand if a file has a particular type of side data. For instance if only a few Clusters somewhere in the Segm…

I would agree here.

Best regards,
Tobias

---

From: Moritz Bunkus <[email-masked]>
Date: Mon, 05 Nov 2018 10:29:02 +0100

Hey,

> In other thoughts on this suggestion, I think it could make it difficult
> to easily understand if a file has a particular type of side data. For
> instance if only a few Clusters somewhere in the Segment contain a
> certain type of side data, it would require parsing every Cluster to know
> what types of side data are available. This uncertainly wouldn’t be the
> same issue if the side data was itself a Track.

It's not entirely necessary to use a full track for side data. We can simply
signal the presence of side data in the track headers and refer to it from
the side data in the block groups. This would also mean we only have to
store the string identifying the side data type once (in the track headers)
instead of in each block.

(I'll use "BlockMetadata" as the basis for all element names
here. Initially I proposed "FrameMetadata", but "BlockMetadata" is fine
with me, too.)

For example:

Tracks
+- TrackEntry
 +- TrackBlockMetadata (Master)
  +- TrackBlockMetadataType (String, required)
  +- TrackBlockMetadataID (Unsigned Integer, required)

…

Cluster
+- BlockGroup
 +- BlockMetadata (Master)
  +- BlockMetadataID (Unsigned Integer, required, refers to existing
     TrackBlockMetadataID in track headers)
  +- BlockMetadataString (Unicode String, optional)
  +- BlockMetadataBinary (Binary, optional)
  +- BlockMetadataUInteger (Unsigned Integer, optional)
  +- BlockMetadataSInteger (Signed Integer, optional)
  +- BlockMetadataFloat (Float, optional)

with the restriction that exactly one of (BlockMetadataString,
BlockMetadataBinary, BlockMetadataUInteger, BlockMetadataSInteger,
BlockMetadataFloat) must exist.

Advantages as I see them:

• Less overhead (no repeated string parsing required)
• Quicker parsing (no repeated string parsing required)
• Presence of meta data is known upfront
• Not using a full-blown track for meta data would alleviate the need to
  specify how all those track and block features (e.g. BlockDuration,
  TrackDefaultDuration…) apply to a "meta data track".

Kind regards,
mosu

---

From: Jerome Martinez <[email-masked]>
Date: Mon, 5 Nov 2018 12:10:16 +0100

On 05/11/2018 10:07, Tobias Rapp wrote:
> I should clarify: Storing the timecode value as a _single_ number 
> makes processing a lot easier. [...]

It adds another complexity: you need to transport somewhere else the 
configuration of this time code value (DP/NDP, frame rate).
which is not always available, if I do e.g. realtime transcoding from 
SDI and catch SMPTE ST 12, I don't have immediately the frame rate (I 
need to wait for frame number going back to 0) so I would have to wait 
up to 1 second before writing the Matroska header if frame rate info is 
in track header.
For reference MP4/MOV or MXF time codes use numbered values, SMPTE ST 
12, SDTI in MXF or GXF use a format which is more like a string time 
code. strict conversion between time code format (especially keeping the 
user bytes) is not always doable.

In my opinion we should not debate here of the best time code format (it 
would be in a metadata format registry, and both string time code and 
numbered time code may exist), just permit any of them so I like the 
proposal from Moritz, especially because I would use it too for 
RAWcooked the same way as with time code configuration in the track 
header for DP/NDP and frame rate info (if I store some RAWcooked 
configuration in the track header, I can improve a lot the comrpession 
of RAWcooked metadata).

---

From: Jerome Martinez <[email-masked]>
Date: Mon, 5 Nov 2018 12:20:09 +0100

> instead of in each block.

It would permit more efficient storage, but IMO it should be optional 
only, because if we do real time transcoding to MKV we don't have always 
all sidecar info. I have in mind some A/V content with ancillary data 
coming only after few minutes, without any change in A/V part.

> here. Initially I proposed "FrameMetadata", but "BlockMetadata" is fine

I prefer also "BlockMetadata" due to here is located the element.

>    +- TrackBlockMetadataID (Unsigned Integer, required)

Please add:
  +- TrackBlockMetadataPrivate (Binary, optional)

Would have the same usage as CodecPrivate but for metadata.

>       TrackBlockMetadataID in track headers)

Here I suggest to keep something like current PR "BlockMetadataName" and 
the restriction that  that exactly one of (BlockMetadataID ,
BlockMetadataName) must exist.

This would be even more complex but IMO more versatile.

---

From: Moritz Bunkus <[email-masked]>
Date: Mon, 05 Nov 2018 12:30:03 +0100

Hey,

>  +- TrackBlockMetadataPrivate (Binary, optional)

Sure, that's useful.

> BlockMetadataName) must exist.

Sure. I wasn't trying to write down the full specs; just enough to have
something to discuss. That restriction is already present in Dave's PR, and
I'm all for keeping it.

Kind regards,
mosu

---

From: Tobias Rapp <[email-masked]>
Date: Mon, 5 Nov 2018 16:41:16 +0100

> configuration of this time code value (DP/NDP, frame rate).

I would say that the complexity is just shifted from read to write side. 
It depends on the assumed balance of write and read (archive vs. reuse, 
ingest vs. outgest) whether to see this as adding or removing complexity.

> up to 1 second before writing the Matroska header if frame rate info is 

You could reserve some space in the header to write the frame-rate 
information later.

> configuration in the track header, I can improve a lot the comrpession 
Indeed the timecode format is of second interest (even though 
implementation of multiple formats also adds complexity).

The idea of adding general metadata for each frame is nice, and storing 
DPX tags into it is fine. But I wouldn't store timecode that way, 
regardless of the serialization format. When you want to seek to a 
specific timecode position you have to read all frame metadata in 
advance. If timecode is stored separately, possibly as an own track, it 
allows you to do the mapping from timecode to timestamp first and then 
use the cue table to perform the locate operation efficiently.

Regards,
Tobias

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 11 Nov 2018 16:40:36 +0100

I would go with the Track way as well. Primarily because storing a
string (which pretty much never changes) in each Block is a huge
waste.

Add extra data per Block is already supported using BlockAdditions.
There's already BlockAddID which correspond to Moritz' BlockMetadataID
and BlockAdditional which correspond to BlockMetadataString /
BlockMetadataBinary / BlockMetadataUInteger / BlockMetadataSInteger /
BlockMetadataFloat. It's like a codec where the CodecID defines how
the data in the binary blob should be interpreted.

The current system states that the additions are left to
interpretation to the codec. It was originally designed to hold the
lossless complement to lossy versions of Musepack. So in that case
it's really meant to be passed to the codec. I think we can expand
this system with keeping this default behaviour by default (albeit not
used anywhere) and have different ones on demand.
There's also an AlphaMode that also uses BlockAdditions to store the
alpha track. Which pretty much no info on how to do it....

As noted timecode may be a separate track (as originally intended) if
it is not related to the video frames (ie the timestamps doesn't
match).

This would look like this:
- Musepack lossless complement:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 1
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 1
(same as BlockAddID) (default)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName:
"complement" (default)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType: 0
(Codec Complement data) (default)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 1
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
lossless part interpreted by the codec

- Alpha layer:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 2
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 2
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "alpha"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType: 1
(Alpha layer data)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 2
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
alpha mask to apply on the video track

- RawCooked DPX data
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "rawcooked"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x1234567 (rawcooked identifier)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
DPX data defined by RawCooked

- Timecode storing
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "timecode"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x890ABCD (SMPTE TC identifier, can be another ID for different kind
of timecode)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
Timecode storage

That means the alpha mode would not be backward compatible with
existing files, because it requires non default values. But I don't
think anyone ever used this improperly defined feature.

The value of MaxBlockAdditionID is kept low on purpose. BlockAddID 1
was always for codec complement and thus 2 for the AlphaMode. But we
don't need to go much higher that than now that we have a mapping. If
there are Timecode AND Rawcooked it would be like this:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 4
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "rawcooked"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x1234567 (rawcooked identifier)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 4
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "timecode"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x890ABCD (SMPTE TC identifier, can be another ID for different kind
of timecode)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
DPX data defined by RawCooked
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 4
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
Timecode storage

Le lun. 5 nov. 2018 à 10:29, Moritz Bunkus
<moritz=[email-masked]> a écrit :
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Dave Rice <[email-masked]>
Date: Sun, 11 Nov 2018 18:15:30 -0500

Hi Steve,

Initially I looked at the BlockAdditions Element as a possibility to store side data, but the definition didn’t seem to allow it. Since the specification defers to the encoding ("Interpreted by the codec as it wishes”) there doesn’t seem to be any way for the Matroska specification to define a BlockAdditionalID value (except for 0 which is reserved as a reference to the main block). 

However, this approach works for me if others accept the break in reverse compatibility.

Several other definitions would require updates. Currently we have:
BlockAdditions: "Contain additional blocks to complete the main one. An EBML parser that has no knowledge of the Block structure could still see and use/skip these data.”
 BlockAddID: "An ID to identify the BlockAdditional level.” 
 BlockAdditional: "Interpreted by the codec as it wishes (using the BlockAddID)."
AlphaMode: "Alpha Video Mode. Presence of this Element indicates that the BlockAdditional Element could contain Alpha data.”

Notes about what definition updates would be needed:
BlockAdditions’s definition would have to be rewritten as the data wouldn’t necessarily “complete” the main block but might alternatively supplement it or describe it.
BlockAddID should include an enumerated list of integers and reference a registry on what each value means.
BlockAdditional’s definition would have to be updated, since only in some cases would the BlockAddition content be interpreted by the codec.
AlphaMode’s definition seems to imply that one of the BlockAdditional Elements contains alpha but gives no way to identify which one, so this is the largest reverse compatibility issue. Are there Matroska demuxers which would try to use timecode or rawcooked data as if it was alpha data?

So IIUC BlockAddID=1 is reserved for the context of the associated codec mapping (as done in A_WAVPACK4). Then we reserve BlockAddID=2 for alpha and 0 is reserved since BlockAdditionID uses 0 to reference the main block. Then any other type of data for storage in BlockAdditional would use the next available unsigned integer from 3 and would require an BlockAdditionMapping. I suggest that BlockAddId = 0, 1 and 2 would not require a BlockAdditionMapping since the specification would reserve them for particular purpose.

But if we already define BlockAddId for 0, 1, and 2 in the specification then why not continue and reserve 3 for rawcooked and 4 for timecode and so on. That would avoid the need for storing that data in BlockAdditionMapping, eliminate the need for new BlockAdditionMapping elements, and then demuxers could understand if they can use that data by simply checking the BlockAddId rather than looking up the corresponding value in BlockAddIDName.

Also what is the need for MaxBlockAdditionID? And what is the reason to keep BlockAdditionID low, simply to keep it to a single byte?

Dave

> On Nov 11, 2018, at 10:40 AM, Steve Lhomme <[email-masked]> wrote:
>

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 12 Nov 2018 16:38:15 +0100

Just as I double check this, it turns out WebM has support for the 
AlphaMode and all the BlockAddition that goes with it.
https://www.webmproject.org/docs/container/

I have no idea how this can even work, except it must be tied to VP8 and 
VP9 internals. So it is also extra data added per frame. I'd be 
interested in a sample making use of that.

They don't even support MaxBlockAdditionID, so technically the value is 
0 for them. I just hope they don't use BlockAddId 0, it is a forbidden 
value. If not it's probably a value of 1 but not 2.

I found this sample which uses a value of 1 
https://examples.phaser.io/assets/video/alpha-webm.webm

So we can assume 1 is really for "codec interpreted data" and all the 
other are freely interpreted.

In addition of 
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName and 
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType we may 
also add a 
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDExtraData which 
could hold extra information for the given type. For example the type of 
timecode format for the given format.

MaxBlockAdditionID is there to signal that there will be extra data in the header of the file, since there was no other way of knowing. It's not used in WebM but they have AlphaMode (which we will need to keep as well).

As for hardcoding BlockAddIDType in the specs we can do it. But we need to allow more values anyway.

---

From: Luca Barbato <[email-masked]>
Date: Mon, 12 Nov 2018 18:21:39 +0100

> coming only after few minutes, without any change in A/V part.

This should warrant for a stand alone track maybe?

lu

---

From: Michael Richardson <[email-masked]>
Date: Mon, 12 Nov 2018 12:49:35 -0500

> https://www.webmproject.org/docs/container/

Is there some implication for Matroska? 

btw: IANA (Michelle) and I had some discussion about how to reserve DocType
names for WebM properly, and I think we have a solution.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 18 Nov 2018 13:07:24 +0100

> Is there some implication for Matroska?

That means at least we can't deprecate it and we have to make sure
what we do in the future is compatible with that. We don't want to
separate the specs. From a "container" perspective it's just a pity
that it's can't be "contained" in a BlockAdditionMapping. But that's
the only issue.

> names for WebM properly, and I think we have a solution.

OK, feel free to share it ;)

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 18 Nov 2018 15:03:13 +0100

I did a Pull Request with my proposed additions to handle
BlockAdditional data in a meaningful way:
https://github.com/Matroska-Org/matroska-specification/pull/287

I added Timecode and RAWCooked values but IMO they should be in a
separate document (since the binary extra data will evolve over time,
independently of the Matroska specs). We will need a IANA registry for
each BlockAddIDType value.

The value 0 (Codec Complement) may be explained further.

---

From: Michael Richardson <[email-masked]>
Date: Sun, 18 Nov 2018 14:58:54 -0500

> OK, feel free to share it ;)

Michelle was going to propose the right text so that the EBML document would
correctly reserve the DocType "WebM" and "Matroska" names in the First Come
First Served DocType Registry.

Both are to be entered into the Registry under IESG ownership/control.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Bradshaw <[email-masked]>
Date: Tue, 20 Nov 2018 11:10:30 -0800

> value. If not it's probably a value of 1 but not 2.

Good catch that MaxBlockAdditionID is unsupported in WebM. I'll try to
rectify that.

To clarify, alpha in WebM uses BlockAddID==1. It would be nice to
standardize that within CELLAR. If I can help in any way, please let me
know.