Subject: [Cellar] Data stream support in Matroska

From: Kieran O Leary <[email-masked]>
Date: Sun, 27 Dec 2015 20:03:06 +0000

Hello,

Matroska does not currently support data tracks. From my experience working
in a moving image archive, a lot of quicktime files from Final Cut Pro can
end up with a timecode data track. I can't remember off the top of my head
if AVID MC does this as well.

The timecode always seems to exist as metadata in the actual AV streams so
the information isn't lost when the data track doesn't exist. It would be
nice if matroska supported it, though. It would be preferable to maintain
all streams within the original file when migrating to the matroska
container.

If I'm converting a v210 quicktime with multiple audio tracks to ffv1.mkv
with ffmpeg, I end up using `-map 0 -dn` (`-map 0:a -map 0:v` would also
work, but an extra condition would have to be scripted if some embedded
subtitles existed in some files).

Best regards,

Kieran O'Leary
IFI Irish Film Archive

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 27 Dec 2015 21:17:07 +0100

On 27/12/2015 21:03, Kieran O Leary wrote:
> Hello,
>
> Matroska does not currently support data tracks. From my experience 
> working in a moving image archive, a lot of quicktime files from Final 
> Cut Pro can end up with a timecode data track. I can't remember off 
> the top of my head if AVID MC does this as well.
>
> The timecode always seems to exist as metadata in the actual AV 
> streams so the information isn't lost when the data track doesn't exist.

I wonder how you see the time code in your transcoded Matroska file, I 
see not place for it (and v210 raw stream can not contain any time code)
As far as I know, the time code information is lost during QuickTime to 
Matroska remuxing.

> It would be nice if matroska supported it, though. It would be 
> preferable to maintain all streams within the original file when 
> migrating to the matroska container.

"All" is always difficult to do, because containers do not have the same 
capabilities (especially tags).
And QuickTime time codes are very specific (very tied to QuickTime 
container format).
But I agree that having ancillary data (time code or any other ancillary 
stream) in Matroska may be useful for handling more cases.

>
> If I'm converting a v210 quicktime with multiple audio tracks to 
> ffv1.mkv with ffmpeg, I end up using `-map 0 -dn` (`-map 0:a -map 0:v` 
> would also work, but an extra condition would have to be scripted if 
> some embedded subtitles existed in some files).
>
>
> Best regards,
>
> Kieran O'Leary
> IFI Irish Film Archive
>
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Kieran O Leary <[email-masked]>
Date: Sun, 27 Dec 2015 20:43:33 +0000

On Sun, Dec 27, 2015 at 8:17 PM, Jerome Martinez <[email-masked]>
wrote:

> working in a moving image archive, a lot of quicktime files from Final Cut
> Pro can end up with a timecode data track. I can't remember off the top of
> my head if AVID MC does this as well.
>
> the information isn't lost when the data track doesn't exist.
>
>
> I wonder how you see the time code in your transcoded Matroska file, I see
> not place for it (and v210 raw stream can not contain any time code)

Hi Jerome,

Thanks for getting back so quick.

The only test files I have to hand are here:
https://archive.org/details/test20100622_v210_and_ffv1

The v210.mov from that link has a timecode track (will post full ffmpeg
output of both files at the bottom of the email just to be safe):
    Stream #0:6(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
      reel_name       : DHC0034_1984.0040_pmaster
      timecode        : 00:30:40:13

This timecode info also appears in the video stream:
  Stream #0:0(eng): Video: v210 (v210 / 0x30313276),
yuv422p10le(bt470bg/smpte240m/unknown), 720x486, 222820 kb/s, 29.85 fps,
29.97 tbr, 2997 tbn, 2997 tbc (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
      encoder         : Uncompressed 10 bit
      timecode        : 00:30:40:13
    Stream #0:1(eng): Subtitle: eia_608 (c608 / 0x38303663), 640x480, 4
kb/s (default)

I transcoded the file to ffv1.mkv and the timecode info is preserved as
part of the video stream:
    Stream #0:4(eng): Video: ffv1 (FFV1 / 0x31564646), yuv422p10le,
720x486, SAR 1:1 DAR 40:27, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      CREATION_TIME   : 2010-05-25 22:47:06
      LANGUAGE        : eng
      HANDLER_NAME    : Apple Alias Data Handler
      TIMECODE        : 00:30:40:13
      ENCODER         : Lavc56.60.100 ffv1
      DURATION        : 00:00:01.001000000

My command line was  ffmpeg -i
'/home/kieranjol/Downloads/DHC36.v210.pcm.mov' -c:v ffv1 -level 3 -g 1 -t 1
-c:a copy -map 0:a -map 0:v '/home/kieranjol/Downloads/DHC36.v210.pcm.mkv'

Here's the ffmpeg -i  from the v210 original, and the ffv1.mkv will follow.

ffmpeg -i '/home/kieranjol/Downloads/DHC36.v210.pcm.mov'
ffmpeg version 2.8.1 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04)
  configuration: --prefix=/home/kieranjol/.linuxbrew/Cellar/ffmpeg/2.8.1
--enable-shared --enable-pthreads --enable-gpl --enable-version3
--enable-hardcoded-tables --enable-avresample --cc=/usr/bin/gcc-4.8
--host-cflags='-Os -w -pipe -march=core2'
--host-ldflags='-L/home/kieranjol/.linuxbrew/lib
-Wl,-rpath,/home/kieranjol/.linuxbrew/lib' --enable-libx264
--enable-libmp3lame --enable-libvo-aacenc --enable-libxvid
--enable-libfreetype --enable-ffplay --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from
'/home/kieranjol/Downloads/DHC36.v210.pcm.mov':
  Metadata:
    creation_time   : 2010-05-25 22:46:56
    timecode        : 00:30:40:13
  Duration: 00:00:09.91, start: 0.040375, bitrate: 230674 kb/s
    Stream #0:0(eng): Video: v210 (v210 / 0x30313276),
yuv422p10le(bt470bg/smpte240m/unknown), 720x486, 222820 kb/s, 29.85 fps,
29.97 tbr, 2997 tbn, 2997 tbc (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
      encoder         : Uncompressed 10 bit
      timecode        : 00:30:40:13
    Stream #0:1(eng): Subtitle: eia_608 (c608 / 0x38303663), 640x480, 4
kb/s (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
    Stream #0:2(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, mono,
s32 (24 bit), 1152 kb/s (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
    Stream #0:3(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, mono,
s32 (24 bit), 1152 kb/s (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
    Stream #0:4(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, mono,
s32 (24 bit), 1152 kb/s (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
    Stream #0:5(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, mono,
s32 (24 bit), 1152 kb/s (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
    Stream #0:6(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2010-05-25 22:47:06
      handler_name    : Apple Alias Data Handler
      reel_name       : DHC0034_1984.0040_pmaster
      timecode        : 00:30:40:13
At least one output file must be specified

THE FFV1.MKV:
 ffmpeg -i '/home/kieranjol/Downloads/DHC36.v210.pcm.mkv'
ffmpeg version 2.8.1 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04)
  configuration: --prefix=/home/kieranjol/.linuxbrew/Cellar/ffmpeg/2.8.1
--enable-shared --enable-pthreads --enable-gpl --enable-version3
--enable-hardcoded-tables --enable-avresample --cc=/usr/bin/gcc-4.8
--host-cflags='-Os -w -pipe -march=core2'
--host-ldflags='-L/home/kieranjol/.linuxbrew/lib
-Wl,-rpath,/home/kieranjol/.linuxbrew/lib' --enable-libx264
--enable-libmp3lame --enable-libvo-aacenc --enable-libxvid
--enable-libfreetype --enable-ffplay --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Guessed Channel Layout for  Input Stream #0.0 : mono
Guessed Channel Layout for  Input Stream #0.1 : mono
Guessed Channel Layout for  Input Stream #0.2 : mono
Guessed Channel Layout for  Input Stream #0.3 : mono
Input #0, matroska,webm, from
'/home/kieranjol/Downloads/DHC36.v210.pcm.mkv':
  Metadata:
    TIMECODE        : 00:30:40:13
    ENCODER         : Lavf56.40.101
  Duration: 00:00:01.01, start: 0.000000, bitrate: 96188 kb/s
    Stream #0:0(eng): Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit),
1152 kb/s (default)
    Metadata:
      CREATION_TIME   : 2010-05-25 22:47:06
      LANGUAGE        : eng
      HANDLER_NAME    : Apple Alias Data Handler
      DURATION        : 00:00:01.006000000
    Stream #0:1(eng): Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit),
1152 kb/s (default)
    Metadata:
      CREATION_TIME   : 2010-05-25 22:47:06
      LANGUAGE        : eng
      HANDLER_NAME    : Apple Alias Data Handler
      DURATION        : 00:00:01.006000000
    Stream #0:2(eng): Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit),
1152 kb/s (default)
    Metadata:
      CREATION_TIME   : 2010-05-25 22:47:06
      LANGUAGE        : eng
      HANDLER_NAME    : Apple Alias Data Handler
      DURATION        : 00:00:01.006000000
    Stream #0:3(eng): Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit),
1152 kb/s (default)
    Metadata:
      CREATION_TIME   : 2010-05-25 22:47:06
      LANGUAGE        : eng
      HANDLER_NAME    : Apple Alias Data Handler
      DURATION        : 00:00:01.006000000
    Stream #0:4(eng): Video: ffv1 (FFV1 / 0x31564646), yuv422p10le,
720x486, SAR 1:1 DAR 40:27, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      CREATION_TIME   : 2010-05-25 22:47:06
      LANGUAGE        : eng
      HANDLER_NAME    : Apple Alias Data Handler
      TIMECODE        : 00:30:40:13
      ENCODER         : Lavc56.60.100 ffv1
      DURATION        : 00:00:01.001000000
At least one output file must be specified

Best regards,

Kieran.

---

From: Kieran O Leary <[email-masked]>
Date: Sun, 27 Dec 2015 21:15:07 +0000

P.S

It shows up as TIMECODE in Mediainfo in the video section of the ffv1.mkv.
It doesn't appear anywhere else (not in General)
it only shows up in ffprobe in the following format (using -show_streams)
[STREAM]
index=4
codec_name=ffv1
codec_long_name=FFmpeg video codec #1
profile=unknown
codec_type=video
codec_time_base=1/1000
codec_tag_string=FFV1
codec_tag=0x31564646
width=720
height=486
coded_width=720
coded_height=486
has_b_frames=0
sample_aspect_ratio=1:1
display_aspect_ratio=40:27
pix_fmt=yuv422p10le
level=-99
color_range=N/A
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=unspecified
timecode=N/A
refs=1
id=N/A
r_frame_rate=2997/100
avg_frame_rate=2997/100
time_base=1/1000
start_pts=0
start_time=0.000000
duration_ts=N/A
duration=N/A
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=10
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
TAG:CREATION_TIME=2010-05-25 22:47:06
TAG:LANGUAGE=eng
TAG:HANDLER_NAME=Apple Alias Data Handler
TAG:TIMECODE=00:30:40:13
TAG:ENCODER=Lavc56.60.100 ffv1
TAG:DURATION=00:00:01.001000000
[/STREAM]

---

From: Moritz Bunkus <[email-masked]>
Date: Sun, 27 Dec 2015 22:23:21 +0100

Hey,

> TAG:DURATION=00:00:01.001000000

Well. If it's a tag then a data track wouldn't be a good fit for such
information anyway.

Matroska differentiates between information that is timestamped and
needs synchronization on the one hand and general information applying
to the whole file on the other.

These tags output by ffmpeg seem to be information of the latter
kind. They don't differ from frame to frame, they don't have to be
interleaved with other frames in order to avoid seeking.

Therefore the best way to deal with those is to store them as tags in
Matroska. I don't think that it should be ffmpeg's place to convert such
information to tags automatically (or at least not by default, maybe
with an option), though.

Such a conversion would be easy to achieve outside of ffmpeg with some
scripting. First convert as you've already done. Next run ffprobe,
extract the lines starting with "TAG:", create a Matroska tags XML file
for it and use mkvpropedit to attach those tags to the converted file.

Kind regards,
mosu

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 27 Dec 2015 22:34:48 +0100

On 27/12/2015 21:43, Kieran O Leary wrote:
>
> I transcoded the file to ffv1.mkv and the timecode info is preserved 
> as part of the video stream:
> '/home/kieranjol/Downloads/DHC36.v210.pcm.mov' -c:v ffv1 -level 3 -g 1 
> -t 1 -c:a copy -map 0:a -map 0:v 
> '/home/kieranjol/Downloads/DHC36.v210.pcm.mkv'

I see, FFmpeg transforms the time code track to a piece of metadata.

I don't think this is something "standard" (for Matroska) and I see an 
issue: it works only for an unique timecode value. if your time code 
track contains more than 1 time code value (QuickTime has 2 "modes" for 
time codes: 1/ "striped", contains an unique time code value, the one of 
the first frame, and timecode value of other frames are computed from 
the time code of the first frame + frame count; 2/ not "striped", 
contains one time code value per frame, so it can handle 
discontinuities), I think you lose other time codes values (I'll test 
with such file later next week).
We also lose the time code frame rate (it may look like weird, but 
sometimes the time code framerate is not same as the video framerate)

So current method is a (not bad) workaround, but nothing safe for 
LossLess Archiving and Realtime transmission.

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 27 Dec 2015 22:46:40 +0100

> interleaved with other frames in order to avoid seeking.

In that case, yes, because the time code track is "stripped" (QuickTime 
wording).
It is not always the case:
- QuickTime time code can contain one time code value per frame, it can 
be interleaved with video (time code packet 0 with frame 0 time code 
value, video packet 0, time code packet 1 with frame 1 time code value, 
video packet 1...) or all in one block (one unique packet with all time 
code values in one shot, the decoder/demuxer must keep all time codes in 
RAM for sending the right timecode at the right time)
- More generally speaking, time codes can be transported by other means 
than QuickTime time code tracks, e.g. Ancillary data 
https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1364-3-201510-I!!PDF-E.pdf 
, there is one Ancillary data packet per corresponding video frame, so 
interleaved.

so such data content could be "information that is timestamped and needs 
synchronization" and we need to specify a CodecID for both QuickTime non 
stripped time code tracks and for BT.1364 tracks if we want to fully 
keep such data during remuxing.

> for it and use mkvpropedit to attach those tags to the converted file.

But TIMECODE is not a "standard" tag for the moment, and it may be 
something important for some people to be sure to retrieve such 
information for sure.

---

From: Kieran O Leary <[email-masked]>
Date: Sun, 27 Dec 2015 22:08:20 +0000

It seems like it's a tag in the v210 stream as well, or is ffmpeg
generating this tag based on the timecode track? It seems like in my case,
the timecode track is ignored and the timecode tag from the v210 is carried
over as a tag to the ffv1 stream in the .mkv?
So it seems like the tag is carried over from the v210 stream to the the
ffv1 stream, ignoring the data timecode track?

Just FYI, here's the ffprobe of the quicktime timecode track, followed by
the quicktime v210 output:
[STREAM]
index=6
codec_name=unknown
codec_long_name=unknown
profile=unknown
codec_type=data
codec_time_base=1/30
codec_tag_string=tmcd
codec_tag=0x64636d74
id=N/A
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/2997
start_pts=-121
start_time=-0.040374
duration_ts=29700
duration=9.909910
bit_rate=3
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=1
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
TAG:creation_time=2010-05-25 22:47:06
TAG:language=eng
TAG:handler_name=Apple Alias Data Handler
TAG:reel_name=DHC0034_1984.0040_pmaster
TAG:timecode=00:30:40:13
[/STREAM]

[STREAM]
index=0
codec_name=v210
codec_long_name=Uncompressed 4:2:2 10-bit
profile=unknown
codec_type=video
codec_time_base=1/2997
codec_tag_string=v210
codec_tag=0x30313276
width=720
height=486
coded_width=720
coded_height=486
has_b_frames=0
sample_aspect_ratio=0:1
display_aspect_ratio=0:1
pix_fmt=yuv422p10le
level=-99
color_range=N/A
color_space=bt470bg
color_transfer=unknown
color_primaries=smpte240m
chroma_location=unspecified
timecode=N/A
refs=1
id=N/A
r_frame_rate=2997/100
avg_frame_rate=893106/29921
time_base=1/2997
start_pts=-121
start_time=-0.040374
duration_ts=29921
duration=9.983650
bit_rate=222820111
max_bit_rate=N/A
bits_per_raw_sample=10
nb_frames=298
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
TAG:creation_time=2010-05-25 22:47:06
TAG:language=eng
TAG:handler_name=Apple Alias Data Handler
TAG:encoder=Uncompressed 10 bit
TAG:timecode=00:30:40:13
[/STREAM]