Subject: [Cellar] Ancillary data in Matroska

From: Jerome Martinez <[email-masked]>
Date: Tue, 27 Dec 2016 21:22:39 +0100

We need additional codec to be supported in Matroska in order to be able 
to transwrap in a lossless manner content from older containers to 
Matroska, especially time codes (all the codecs defined in the patch can 
contain 1 or more time codes)
The global idea is to be able to transwrap e.g. MOV (from Apple), MXF 
(from SMPTE), GXF (from Grass Valley, also known as SMPTE ST 360 and 
SMPTE RDD 14, found in some broadcaster archives), LXF (from 
Leitch/Harris, found in some broadcaster archives) without losing 
metadata in some sidecar non visible streams.

After agreeing on the format, we will have to update tools e.g. FFmpeg 
in order to support such lossless transwrap.

I added an "Ancillary" section with several mappings in the attached 
patch, please comment.

Comments:

The frequent remark is "Why do you need so many time code formats, 
choose one and stop using the others": in the real world, people 
consider that one format is the best one, but not the same format as 
others, and they use only this time code for their work and don't plan 
to switch. You can have 10+ time codes in a file and each one is 
important for someone, as well as the sidecar information transported at 
the same time (never same between each format). There is no consensus 
about that for years, and I don't think we could achieve a consensus 
from the different actors (lot of them are not interested by our work at 
IETF) so the idea here is to be able to move from one container to 
another one without losing any metadata, absolutely not deciding about a 
reference time code (and other metadata) format, as Matroska does for 
e.g. video (Matroska does not say that VP9 should be used instead of 
AVC, it supports both).

"N_" prefix arbitrary used (2nd letter of "Ancillary" because first 
letter is already used)

N_QUICKTIME: is a pure copy of V_QUICKTIME and A_QUICKTIME already 
supported in Matroska, N_ would be used which would be used for e.g. 
tmcd codec (QuickTime time code). Exactly same principles as video and 
audio. Note that classic usage is to store only the first frame content 
and other content is computed from it, so a player would have to read 
the first frame even if there is a direct seek request to another place; 
this could be avoided with some "hack" in the track header, could be the 
next step after this one is accepted.

VBI and ANC: used to transport time codes (LTC, VITC, ATC...), Recording 
Information, bar and pan/scan data, captions (North American CEA-608, 
CEA-708, European and Australian WSS/Teletext, Japanese ARIB B37...), 
camera acquisition dynamic metadata, Audio Metadata, Film Transfer and 
Video Production Information (in theory, I never saw it)...
The "perfect" solution would be to decode VBI and ANC, and put each 
format in its own specification, but it would be an huge task and we 
still would have to store opaque content (content is present in the 
VBI/ANC but we ignore all from it right now), so it is better to start 
by the beginning and we consider all opaque without trying to decode the 
data, as do other containers.

N_STRIPPEDTIMECODE: the Cluster part will contain no data, but despite 
of that it must be a complete standalone track, with Track Id ans so on, 
as it is a standalone track in the source container.

N_SDTICP: nearly 1:1 copy of SMPTE ST 385 elements. In order to have all 
data and being able to have only 1 Matroska block for all STDI-CP 
elements, I kept also the 2 last bytes of the KLV "Key" and the KLV 
"Length" in addition of the KLV "Value". Note that something weird, this 
is a standalone track in source container but the difference with other 
tracks (including ANC or VBI) is that this stream has no track header so 
no track number, and Matroska specs makes TrackNumber mandatory, I 
wonder what should be used during transwrap (fake number?)

TODO: support of GXF non stripped time codes

Jérôme

From 7057c252c2438e1c4cdbff8f8f97dd0a10c3d8d0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?J=C3=A9r=C3=B4me=20Martinez?= <[email-masked]>
Date: Tue, 27 Dec 2016 21:14:38 +0100
Subject: [PATCH] Embedding of ancillary data in Matroska

---
 codec_specs.md | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/codec_specs.md b/codec_specs.md
index 3fde112..fa9073c 100644
--- a/codec_specs.md
+++ b/codec_specs.md
@@ -384,3 +384,35 @@ For each line containing the timestamp and file position data is read from the a
 **Codec ID:** B_VOBBTN  
 **Codec Name:** VobBtn Buttons   
 **Description:** Based on [MPEG/VOB PCI packets](http://dvd.sourceforge.net/dvdinfo/pci_pkt.html). The file contains a header consisting of the string "butonDVD" followed by the width and height in pixels (16 bits integer each) and 4 reserved bytes. The rest is full [PCI packets](http://dvd.sourceforge.net/dvdinfo/pci_pkt.html).
+
+## Ancillary
+
+  
+**Codec ID:** N_VBI  
+**Codec Name:** Vertical Blanking Interval  
+**Description:** used in SDTV, see https://en.wikipedia.org/wiki/Vertical_blanking_interval for more information. Each Matroska block contains a SMPTE ST 436 VBI Frame Element.  
+
+  
+**Codec ID:** N_ANC
+**Codec Name:** Vertical or Horizontal Ancillary data
+**Description:** used in HDTV, see https://en.wikipedia.org/wiki/Ancillary_data for more information. Each Matroska block contains a SMPTE ST 436 VBI Frame Element.
+
+  
+**Codec ID:** N_QUICKTIME  
+**Codec Name:** Generic or null handlers (non video/audio/text handlers) taken from QuickTime(TM) files  
+**Description:** Several codecs are stored in QuickTime with a generic handler type ("gmhd") or a null handler type ("nmhd"), this CodeCID is intended for non visible streams e.g. Time code. The CodecPrivate contains all additional data that is stored in the 'stsd' (sample description) atom in the QuickTime file **after** the mandatory video descriptor structure (starting with the size and FourCC fields). For an explanation of the QuickTime file format read [QuickTime File Format Specification](https://developer.apple.com/library/mac/documentation/QuickTime/QTFF/QTFFPreface/qtffPreface.html).
+
+  
+**Codec ID:** N_STRIPPEDTIMECODE  
+**Codec Name:** Stripped time code  
+**Description:** track containing no data in the Cluster elements, as all is in the track header: the CodecPrivate element contains, in big endian: 4 bytes Rate numerator, 4 bytes rate denominator, 4 bytes count of consecutive time codes / frames (the "duration", 0 if unknown), 1 byte time code drop frame flag (0 is no, 1 is yes, others are reserved), 4 byte the start time code (in frames). Note: this permits to map a MXF time code track or a GXF stripped time code track to Matroska.  
+
+  
+**Codec ID:** N_SDTICP1  
+**Codec Name:** Serial Digital Transport Interface Content Package version 1 (SMPTE ST 326)  
+**Description:** Track containing SDTI content e.g. time codes, picture metadata and so on. Matroska block contains 1 or more of the following sequence: 1 byte of identifier (1=System, 2=Package, 3=Picture, 4=Sound, 5=Data, 6=Control), BER long-form encoding of the SDTI element (typicaly 0x83 then 3 bytes corresponding the the size in big endian), the SDTI element content.  
+
+  
+**Codec ID:** N_SYSTEMSCHEME1  
+**Codec Name:** System Scheme 1 (SMPTE ST 405)  
+**Description:** Track containing System Scheme content e.g. time codes, frame rte info and so on. Matroska block contains a complete System Scheme 1 element .  
-- 
2.6.4.windows.1