Subject: [Cellar] Film: color handling - container or codec?

From: "Peter B." <[email-masked]>
Date: Thu, 15 Dec 2016 10:16:57 +0100

Dear all,

Due to recent demands, I would like to follow up on the issue of color
handling.
Especially the features required for properly handling digital film
material (e.g. from DPX).

There have been threads addressing this already:

"FFV1 Version 4" (2016-01-05)
https://mailarchive.ietf.org/arch/msg/cellar/RFYBaQSuoULr8sIw9RfHwnfZKvQ

"Colour Format proposal" (2016-02-16)
https://mailarchive.ietf.org/arch/msg/cellar/A3DsbrRMsPeps1WKRNJOZQRxYy4

In order to avoid getting lost in detail (which I'm an expert in!), it
might be a good starting point to agree on *where* to store color
information:

=======================
container or codec - or both?
=======================

All 3 versions have their pros and cons, but I think having it in both
causes more trouble than does good (see below).

Here's my current point of view:

1) Container only
=======================
PROs:
  * Independent of codec used
  * Already existing color-handling in MKV
  * Higher chance of non-ffmpeg applications supporting it.

CONs:
  * Quote Niedermayer: "if its not stored at codec level and only
supported by one of several containers it would have a high chance of
being lost"
  * Restricts to a very limited choice of containers

2) Codec only
=======================
PROs:
  * Independent which container is used.
  * Higher chance of keeping the information throughout the material's
lifecycle
  * Fresh start possible (no legacy compatibility to consider with FFV1)

CONs:
  * If container already has *some* color-handling, there might be an
information conflict (=interoperability issues)
  * Less chance of non-ffmpeg applications supporting it, compared to
container.

3) Both
=======================
All PROs of both individual options, *but* with a very very high chance
of interpretation- and interoperability issues between different
applications and scenarios, depending on whether an application
prioritizes data in container or codec.

Doesn't matter if they are identical, but I've seen
container/codec-mismatching metadata (e.g. aspect-ratio, etc) in
real-life, that I do believe this would be a real issue.

If all this is a no-brainer and has an absolutely obvious answer, then
I'm happy!

Thank you very much in advance,
Pb

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 15 Dec 2016 12:04:45 +0100

Le 15/12/2016 à 10:16, Peter B. a écrit :
> [...]

You already have mismatches do handle, this would not be new here.
So from my point of view having them in both the container and the codec 
is the only choice if you want to keep such metadata during transcoding 
/ transwrapping in a safe manner in all cases.
e.g. someone changes of container from MOV to Matroska without 
re-encoding, the MOV metadata may be lost (classic) but the codec 
metadata would be kept.
e.g. some codecs (raw video, current FFV1) has not this metadata, we 
need to support container values.

For the DPX issue, a first step could be to adapt FFmpeg in order to 
correctly read and map the DPX values in the already defined Matroska 
metadata (with maybe some "upgrade" because some DPX values may not be 
in the Matroska lists), then we add also a metadata part in the FFV1 
global header (update of FFV1 v3 or in new v4, it would depend on if v3 
can be expanded, to be studied).

Jérôme

---

From: Luca Barbato <[email-masked]>
Date: Thu, 15 Dec 2016 13:13:01 +0100

On 15/12/2016 10:16, Peter B. wrote:
> 3) Both ======================= 

> real-life, that I do believe this would be a real issue.

If you care about preserving the data during processing having it in
both places gives you a better chance to preserve it.

If you focus about conversion, you would enjoy having your master data
containing the information at both level so you do not need software
that's particularly careful in moving the data from the container-level
to the codec level while streamcopying to containers less rich.

In my opinion, from the safest to the least safe:

- both places

  Easier to spot inconsistencies

  Easier to derive content that's correct even if the target
  container is less than optimal.

- codec only

  Requires to have a decoder to get some information instead of
  just looking at the container.

- container only

  If you remux to the wrong container you have to guess the information
  you lost.

In general you do not want to rely on a single tool being correct and
you'd rather want to have a mean to validate tools that is independent.

lu

---

From: "Peter B." <[email-masked]>
Date: Thu, 15 Dec 2016 13:19:42 +0100

> You already have mismatches do handle, this would not be new here.

I know, but I thought it would be nice to avoid adding new ones :)

> So from my point of view having them in both the container and the
> codec is the only choice if you want to keep such metadata during
> transcoding / transwrapping in a safe manner in all cases.

Indeed.

Yet, I would like to avoid scenarios where one has to live with wrong
metadata, because there is no tool to correct it without requiring
re-encoding. If it's supported by both (in an easily compatible way) and
there are ways to fix wrong metadata, I'd be happy!

> global header (update of FFV1 v3 or in new v4, it would depend on if
> v3 can be expanded, to be studied).

As I understand it, LUTs would enable the decoder to handle any mapping
properly.
Whereas a predefined "mapping-referred-to-by-index" would mean that
not-so-popular mappings would more likely be unsupported in different
applications.

I'm not an expert on this, but I would be *very much* in favor of
embedded LUTs, as this seems the most self-sustaining way.

Any downside of embedded LUTs?

Cheers!
Pb

---

From: "Peter B." <[email-masked]>
Date: Thu, 15 Dec 2016 13:26:57 +0100

> you'd rather want to have a mean to validate tools that is independent.

Absolutely on your side.
If both (container+codec) contain the same value, it's great to have it
in both.
But once container and codec don't "agree" for some reason, things get ugly.

How is this currently dealt with with existing metadata that exists in
both container+codec?

Thanksalot,
Pb

---

From: Luca Barbato <[email-masked]>
Date: Thu, 15 Dec 2016 13:36:51 +0100

On 15/12/2016 13:26, Peter B. wrote:
>> In general you do not want to rely on a single tool being correct
>> and you'd rather want to have a mean to validate tools that is
>> independent.
> 
> Absolutely on your side. If both (container+codec) contain the same
> value, it's great to have it in both. But once container and codec
> don't "agree" for some reason, things get ugly.

I'd call it inconsistent, but it is sort of ok since you easily can spot
it and amend it.

> How is this currently dealt with with existing metadata that exists
> in both container+codec?

Usually if you have the information from the codec that's the one to
prioritize.

lu

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 15 Dec 2016 13:59:47 +0100

> How is this currently dealt with with existing metadata that exists in

You get all behaviors; depending of the software you use.
Yes, it is ugly, but it is the less awful method (it is better to have 2 
different values with different behavior than no value).

As Luca, I argue for prioritizing the one from the codec because en 
encoder uses to be more near from the source and anyone can create a 
(buggy) muxer without a lot of skills, but I know some cases where 
people use the container for "correcting" the bad value from the wrongly 
configured digitalizing hardware (instead of patching the hardware, due 
to different reasons e.g. vendor not willing to patch), so the one from 
the codec is not always the real one.
Unfortunately, no way to know for sure which one is correct when we have 
2 different values, this is a manual check.
FYI, MediaInfo shows both values when a mismatch is detected, so you can 
know that there is a mismatch and you can see the 2 values.

---

From: Reto Kromer <[email-masked]>
Date: Thu, 15 Dec 2016 14:34:23 +0100

Luca Barbato wrote:

>If you care about preserving the data during processing
>having it in both places gives you a better chance to
>preserve it.

I agree with the idea to have the information twice: in the
container and in the codec. In case of differences, I
suggest to consider the codec's data over the container's.

Best regards, Reto

---

From: [email-masked]
Date: Thu, 15 Dec 2016 10:31:24 -0500

> On Dec 15, 2016, at 8:34 AM, Reto Kromer <[email-masked]> wrote:
> 
> Luca Barbato wrote:
> 

If the data is intended to or offered to be stored redundantly in both the container (Matroska) and encoding (FFV1), then perhaps FFV1 should adopt something semantically identical to the Colour Element of Matroska. If keeping the Colour of Matroska and FFV1 semantically identical is agreeable then we could consider the need for LUT storage by first adding it to the Colour Element of Matroska.

To store such Colour data in FFV1, would Colour data be best stored in Parameters. I see other data that regards the presentation (such as sar_num and sar_den) stored in the Slice Header, but I don't think it's feasible for Colour to vary within a frame per slice.

Dave Rice