Subject: Re: [Cellar] [Matroska-devel] Depth offsets for subtitles in case of 3D MVC tracks in MKV files

From: madshi <[email-masked]>
Date: Sun, 28 Feb 2016 23:12:34 +0100

2016-02-14 15:50 GMT+01:00 Steve Lhomme <[email-masked]>:
> 2016-01-26 14:39 GMT+01:00 Nithin Mathew Kurien <[email-masked]>:
>> Hi,
>>
>> It's like this. Suppose the MVC video track has 8 frames. Also suppose it
>> has 5 Offset Metadata Sequences embedded in it at the time of encoding.
>> (Sequence entries are in pixels, positive if in front of the screen, and
>> negative if behind.)
>>
>> Frame  1  2  3  4  5  6  7  8
>> Seq1    5  1 -8 -2 -9  1 -3 -1
>> Seq2    8  1  0  8 -4 -5  4 -1
>> Seq3   -2  8 -3 -5  7  5 -5 -4
>> Seq4    9  0  0 -5 -2  1 -4 -1
>> Seq5   -9  3 -2 -4 -6  0 -3  6
>>
>> If the M2TS container has 2 subtitle tracks P1, P2 and 1 menu track I1, then
>> a mapping can be defined in the M2TS container like this:
>> P1 : Seq1
>> P2 : Seq3
>> I1  : Seq4
>
> OK, so it seems it's both in the header and in the track (the GOP is
> part of the stream, I think).

Yes. As Nithin Mathew explained (thanks!), there are SEI 3D depth info
blocks inside of the video stream. We don't need to do anything about them.
But there's additional data in the Blu-Ray playlist file(s) which
assigns specific
subtitle tracks to an "array index" in the SEI 3D depth info block.

In the meanwhile nevcairiel and I have already added full support for all
this 3D depth offset stuff to LAV Splitter/Video decoder and madVR, when
playing back original Blu-Rays. But for Matroska there's no full solution
possible yet, because the information in the playlist cannot be properly
stored yet.

It would be great if we could get an official solution for this soon, because
we'd like to implement it "now", and we have the MakeMKV devs on board
for this now, too.

Basically I think what we need is one additional item for each subtitle
track which simply assigns one SEI 3D depth array index to the subtitle
track. E.g. something like "stereo_subtitle_offset_id", or whatever. The
name is not important to me. Possible values I think are between 0..31
for 3D Blu-Rays.

One additional piece of information would be which array indexes of the
SEI 3D depth info block are assigned to IG (interactive graphics). This
could be used by media players or video renderers to decide at which
3D depth to draw the user interface (GUI/OSD). FWIW, it could be none,
one or multiple array indexes. So the information field in the MKV header
would have to be some sort of list/array. Logically this info would
probably belong to the video track. This is less important than the
subtitle assignment, though. But would still be nice to have.

Best regards, madshi.

---

From: Hendrik Leppkes <[email-masked]>
Date: Mon, 29 Feb 2016 09:22:41 +0100

On Mon, Feb 29, 2016 at 8:51 AM, Steve Lhomme <[email-masked]> wrote:
>
> If we want the general case to work we may introduce an element at the
> same level as CodecState that defines the Z-order for the Block. That
> means it would have to translate the lookup index in the SEI to the
> actual z-order value (if I understand correctly how it works). We need
> to define if it's like the CodecState, ie it only changes from scene
> to scene, not Block by Block. So it would be a State that needs to be
> remembered when needed and only written when it changes.
>
> What is the range of depth allowed in Blu-Rays ? Are they discrete
> value or floating values ?
>

The "depth" is defined in discrete pixel values in the BD SEI, ie. the
offset to move it left/right for each eye, so it doesn't have a direct
meaning of "depth", but just instructions on how to render it.
Note that the way this is implemented on Blu-rays, the depth can
change from frame to frame. Every GOP has a new SEI block which
includes one offset for every frame in the SEI, so its controlled on a
frame level, not a scene/GOP level.

(( This time sent to the list as well. ))

- Hendrik

---

From: madshi <[email-masked]>
Date: Mon, 29 Feb 2016 09:54:55 +0100

> Ah, interresting. We could still use a z-order value, because
> that's what 3D is about.

The problem is that a 3D frame usually has different depths in
different sections of the image. Each subtitle track has its own
position where it's drawn, and its own x offset (-> depth) assigned
to it, and depending on which section of the video frame the
subtitles are drawn over, the depth could be widely different for
different subtitle tracks (at least theoretically).

Practically that means we don't really have one depth information
per frame (or block), which works for all subtitles and OSD/GUI.
Instead we have a different depth information for each separate
subtitle track per video frame, and one (or multiple or none) more
depth information for OSD/GUI depth per frame.

> So I think we should go with the CodecState approach. Meaning
> we need a specific codec ID for such Blu-Ray subtitles. We
> should not write the CodecState if the values are the same as
> the time it was previously written.

CodecState is per Cluster, though, not per Track, so I'm not sure
which exact information you'd want to store there? (Unless I'm
misunderstanding the MKV cluster/track structure.)

I think we need to store an array index per subtitle track in order
to properly cover the depth information contained for subtitles in
3D Blu-Ray.

I suppose if you wanted to make this totally codec independent,
one other option would be to store the depth (or x offset) directly
as a property of the subtitle track, but if we do that, then depth
could only change once for each subtitle image, I think, which
is too limited to properly capture what 3D Blu-Ray can do, because
with 3D Blu-Ray, subtitles can have a different depth per subtitle
track *and* per video frame.

Best regards, madshi.

---

From: madshi <[email-masked]>
Date: Mon, 29 Feb 2016 10:14:12 +0100

2016-02-29 10:04 GMT+01:00 Moritz Bunkus via Matroska-devel
<[email-masked]>:
>> CodecState is per Cluster, though, not per Track,
>
> This is wrong. CodecState is a child element of BlockGroup. It's effect
> as it's been meant to be understood so far is to replace CodecPrivate
> for that track from that BlockGroup onwards (until another BlockGroup
> with CodecState comes along). So it _could_ be used for changing Z-depth
> values for individual subtitle frames. However, as it's supposed to
> replace CodecPrivate I don't think it would be a good fit for Z-depth
> data as Z-depth data is additional data that changes on a frame-by-frame
> basis; it doesn't invalidate the existing CodecPrivate data.
>
> CodecState was meant for situations in which e.g. a whole new set of
> SEIs/PPSs come along.
>
> I'd prefer new elements, or maybe BlockAddition.

Ok, thanks.

I think the key thing that needs to be decided first is which of the two
following approaches we want to use:

1) Do we want to store *ALL* 3D subtitle depth information we need
into the MKV header structures? If we do that, we need to be able
to store (up to) one depth information per subtitle track per video frame.

2) Do we want to make use of the SEI 3D depth information? In that
case all we need to store in addition to that is one "int" per subtitle
track, for the whole MKV file!

Best regards, madshi.

---

From: Hendrik Leppkes <[email-masked]>
Date: Mon, 29 Feb 2016 10:26:50 +0100

> to store (up to) one depth information per subtitle track per video frame.

This is the problem that occured to me just now as well - there is
typically more depth information than subtitle frames.
BDs store the depth information based on video frames, not subtitle
frames, so any subtitle frame can change its depth for each video
frame to react to changes in the video.

Which means that storing such information inside the subtitle track is
probably going to be rather problematic.

> track, for the whole MKV file!

This would certainly be the easier option, but it is highly specific
to H264 MVC SEI's, so I can see how they might be reluctant to do
that.

-  Hendrik

---

From: "Sebastian G. <bastik>" <[email-masked]>
Date: Sun, 13 Mar 2016 10:26:20 +0100

29.02.2016, 08:51 Steve Lhomme:
> 2016-02-28 23:12 GMT+01:00 madshi <[email-masked]>:
> That seems to be heavily relying on the Blu-ray specs. I'd favor a
> solution that is more generic and doesn't have to go through "codec
> specific" lookup tables.

In general I also prefer a solution that is independent from Blu-Ray.

> Otherwise if the codec used for Blu-Rays subtitles is already
> specific, we could just use the CodecState to store the SEI value when
> it changes.
> https://matroska.org/technical/specs/index.html#CodecState
> 
> That means custom made text subtitles will not work properly with 3D
> Blu-ray content because it doesn't use the same codec.

Practically it seems better to have it as close as possible to Blu-Ray
to make it easier to support it for hardware players that already
support 3D Blu-Rays.