Subject: [art] Fwd: [media-types] Need guidance on overloading an existing mime type parameter vs. defining a new parameter

From: Alexey Melnikov <[email-masked]>
Date: Wed, 4 Oct 2017 16:50:47 +0100

Hi,

Any comments on Ali's question?

Thank you,
Alexey

-------- Forwarded Message --------
Subject: 	[media-types] Need guidance on overloading an existing mime 
type parameter vs. defining a new parameter
Date: 	Thu, 14 Sep 2017 12:25:05 +0300
From: 	Ali C. Begen <[email-masked]>
To: 	[email-masked]
CC: 	Joey Parrish <[email-masked]>, Christian Timmerer (ITEC) 
<[email-masked]>, Alexey Melnikov 
<[email-masked]>

Hi there,

Alexey M. suggested that I wrote my question to this list. I hope to get 
some advice on something we are dealing with in MPEG.

The core discussion is here:
https://github.com/w3c/encrypted-media/issues/400 
<https://github.com/w3c/encrypted-media/issues/400>

(At the bottom, there is the text we presented in the last mpeg meeting)

I personally think overloading the codecs parameter to indicate every 
media future (the encryption scheme in this specific discussion but the 
main problem is broader) is not a good idea just for the sake of using 
it as implicit signaling.

Admittedly, defining a new parameter is not perfect, either but maybe we 
can find a way to make it work nicely.

What are your thoughts on this? Comments are welcome.

Thanks.
-acbegen

m41135 (Joint contribution to July 2017 meeting by Comcast, Google and 
AAU/Bitmovin)

Title: MIME Type Parameters for Signaling Encryption and Other Media 
Features

1) Introduction

RFC 6381 specifies parameters that are used with various MIME types or 
type/subtype combinations to allow for unambiguous specification of the 
codecs employed by the media formats contained within, or the profile(s) 
of the overall container format. However, the MIME type parameters do 
not indicate the type of encryption. This could create a problem for the 
receivers when the encapsulating environment is not capable of declaring 
separately that the content is protected (such as in the MPD for DASH) 
since in that case the receivers cannot infer whether they can decode 
the media without first fetching and parsing the media tracks.

ISO/IEC 14496-12 defect report (N16618) reported this issue, and the 
revised defect report (N16785) proposed an annex that defined certain 
rules on how to use MIME type parameters for ISOBMFF and derived 
specifications.

The proposed annex in N16785 specifies the following:
If the 4CC of a sample entry indicates protected content (e.g., ‘encv’, 
‘enca’, ‘enct’ etc.), the ‘codecs’ parameter value for that track is any 
the following values, separated by “.”:
* the 4CC indicating protected content (e.g., ‘encv’);
* the scheme-type (e.g., ‘cenc’);
* the original_format (e.g., ‘avc1’), followed by any sub-parameters as 
defined for that original format

As a result, the following values would be possible for the ‘codecs’ 
parameter:
encv.cbcs.avc1.402567
or
encv.cenc.avc1.402567

The motivation here is that when a receiver reads such a value, it can 
determine whether it can decode the media (i.e., whether it supports the 
encryption mode). If it does not understand the value, by definition it 
will ignore the value and will not fetch the media to avoid waste of 
(server, network or time) resources.

While the motivation for this approach seems just, overloading the 
‘codecs’ parameter value has several drawbacks and may lead us to more 
problems in the future.

2) Proposal

A cleaner and more future-proof approach is to put the additional values 
out of the ‘codecs’ parameter. For example, to indicate the encryption 
mode for the following existing media
video/mp4; codecs=”avc1.640028”
we could rather say
video/mp4; codecs=”avc1.640028”; encscheme=”cbcs”
or
video/mp4; codecs=”avc1.640028”; encscheme=”cenc”

Indeed, the older receivers who do not understand the ‘encsheme’ 
parameter will ignore it and may unnecessarily fetch the media (this is 
a behavior by design). However, we need to worry more about the newer 
clients who will understand this and other newly-defined parameters 
since very likely the usage will be dominated by such newer clients.
Note that on the respective Github discussion page [1], other issues 
have also been listed for using the ‘codecs’ parameter to indicate the 
encryption mode. They are:
* The protection scheme is more related to the container than the codec. 
So, it should not be signaled as a part of the ‘codecs’ parameter.
* Codec strings are currently container-independent, and the same string 
can be used with multiple containers. However, the protection scheme is 
not container-independent and this complicates the decoder APIs.
* Codec strings are more likely to be passed around into deeper layers 
of a system than container. Implementations would need to strip this 
prefix before doing so.
* Strings for some codecs are already quite long.
While the above uses the encryption mode as the example, the same 
approach equally applies to other media features. For example, a media 
could be HDR or non-HDR, supports (or not) WCG, could be omnidirectional 
video, etc. One should not signal these types of features in the 
‘codecs’ parameter.

It is recommended that the MPEG community reaches an agreement on these 
principles at this meeting and the work is coordinated between the IETF, 
W3C and MPEG to implement the necessary changes as an update to RFC 6381.

3) References
[1] https://github.com/w3c/encrypted-media/issues/400#issue

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Oct 2017 14:01:00 -0400

Alexey and Ali,

(Small warning -- the first part of this note may not be
applicable to the current case but is, I think, important for
background and to understand the conclusion in context.   Also,
this note is a personal opinion.  While I had a role in defining
the media type and associated registration mechanisms, I have
not consulted my co-authors, I do not follow the ietf-types
list, and a lot of what follows is not in the documents.)

A media type and its parameters should be as precise as possible
and applications should be able to rely on whatever those things
tell them.  I don't understand MPEG, much less this particular
case, and haven't read RFC 6381 in the last half-dozen years if
ever, but, if an early implementation of the file format looks
at the media type and parameters, it should be able to figure
out whether it can interpret the file and interpret it in the
same way that a contemporary rendering implementation would.
If that means looking at a parameter and saying "don't
understand this, maybe you need an updated implementation", that
is just fine and produces a clear error message, while putting
out gibberish or failing midway through the file is less
desirable.  

One way to look at the whole media type idea is that it is
intended to prevent or eliminate the need for heuristics in
determining what should be used to render or interpret a file
and how it should do it.  If retroactively interpreting a
parameter (or even a primary type) puts implementations back in
the business  of trying to guess whether they can interpret the
file or not, that is a violation of at least the spirit of the
idea.   

What makes this a bit complicated is that there can be some
non-heuristic cases.  For example, if a file format contains a
well-defined version number that is always in the same place
regardless of versions and there are clear rules about what an
implementa6ion is supposed to do when an unrecognized version is
encountered, having a media type that is version independent and
either no version parameter or a version parameter that is
purely advisory (with the one in the file overriding) may be
perfectly reasonable.

Now, from what I understand from Ali's note, if the codecs
parameter is precisely enough defined and implementations simply
reject a file whose associated codec name they don't understand,
then the odds are low that serious harm would be caused by
overloading codec names to also include encryption type
information.  That makes it mostly a matter of taste.   However,
multiplying parameter values to incorporate essentially
orthogonal types of information is rarely a good idea and, as
Ali's note sort of points out, it gets worse, exponentially
worse, with each new type of information that is added in.  So
my sense of good taste and instinct match Ali's and the proposal
-- it is probably better to go through the pain of adding one or
more parameters now than to overload "codecs".   

On the other hand, if I were told that most implementations just
ignore parameters they don't understand even while they don't
accept unknown parameter values for parameters they do
recognize, I might well hold my nose, suggest that overloading
the parameter is the least bad solution for the present, and
that people should start looking carefully at updating RFC 6381
and current practice to be sure that the next logical extension
to come along doesn't dig this hole even deeper.

best,
    john

--On Wednesday, October 4, 2017 16:50 +0100 Alexey Melnikov
<[email-masked]> wrote:

> Subject: 	[media-types] Need guidance on overloading an
> existing mime type parameter vs. defining a new parameter
> CC: 	Joey Parrish <[email-masked]>, Christian Timmerer
> (ITEC) <[email-masked]>, Alexey
> Melnikov <[email-masked]>
> 
> 
> 
> Hi there,
> 
> Alexey M. suggested that I wrote my question to this list. I
> hope to get some advice on something we are dealing with in
> MPEG.
> 
> I personally think overloading the codecs parameter to
> indicate every media future (the encryption scheme in this
> specific discussion but the main problem is broader) is not a
> good idea just for the sake of using it as implicit signaling.
> 
> Admittedly, defining a new parameter is not perfect, either
> but maybe we can find a way to make it work nicely.
> 
> m41135 (Joint contribution to July 2017 meeting by Comcast,
> Google and AAU/Bitmovin)
> 
> Title: MIME Type Parameters for Signaling Encryption and Other
> Media Features
> 
> 1) Introduction
> 
> RFC 6381 specifies parameters that are used with various MIME
> types or type/subtype combinations to allow for unambiguous
> specification of the codecs employed by the media formats
> contained within, or the profile(s) of the overall container
> format. However, the MIME type parameters do not indicate the
> type of encryption. This could create a problem for the
> receivers when the encapsulating environment is not capable of
> declaring separately that the content is protected (such as in
> the MPD for DASH) since in that case the receivers cannot
> infer whether they can decode the media without first fetching
> and parsing the media tracks.
> 
> ISO/IEC 14496-12 defect report (N16618) reported this issue,
> and the revised defect report (N16785) proposed an annex that
> defined certain rules on how to use MIME type parameters for
> ISOBMFF and derived specifications.
> 
> If the 4CC of a sample entry indicates protected content
> (e.g., 'encv', 'enca', 'enct' etc.), the
> 'codecs' parameter value for that track is any the
> following values, separated by ".":
> * the 4CC indicating protected content (e.g., 'encv');
> * the scheme-type (e.g., 'cenc');
> * the original_format (e.g., 'avc1'), followed by any
> sub-parameters as defined for that original format
> 
> As a result, the following values would be possible for the
> 'codecs' parameter:
> encv.cbcs.avc1.402567
> or
> encv.cenc.avc1.402567
> 
> The motivation here is that when a receiver reads such a
> value, it can determine whether it can decode the media (i.e.,
> whether it supports the encryption mode). If it does not
> understand the value, by definition it will ignore the value
> and will not fetch the media to avoid waste of (server,
> network or time) resources.
> 
> While the motivation for this approach seems just, overloading
> the 'codecs' parameter value has several drawbacks and may
> lead us to more problems in the future.
> 
> 2) Proposal
> 
> A cleaner and more future-proof approach is to put the
> additional values out of the 'codecs' parameter. For
> example, to indicate the encryption mode for the following
> existing media
> video/mp4; codecs="avc1.640028"
> we could rather say
> video/mp4; codecs="avc1.640028"; encscheme="cbcs"
> or
> video/mp4; codecs="avc1.640028"; encscheme="cenc"
> 
> Indeed, the older receivers who do not understand the
> 'encsheme' parameter will ignore it and may unnecessarily
> fetch the media (this is a behavior by design). However, we
> need to worry more about the newer clients who will understand
> this and other newly-defined parameters since very likely the
> usage will be dominated by such newer clients.
> Note that on the respective Github discussion page [1], other
> issues have also been listed for using the 'codecs'
> parameter to indicate the encryption mode. They are:
> * The protection scheme is more related to the container than
> the codec. So, it should not be signaled as a part of the
> 'codecs' parameter.
> * Codec strings are currently container-independent, and the
> same string can be used with multiple containers. However, the
> protection scheme is not container-independent and this
> complicates the decoder APIs.
> * Codec strings are more likely to be passed around into
> deeper layers of a system than container. Implementations
> would need to strip this prefix before doing so.
> While the above uses the encryption mode as the example, the
> same approach equally applies to other media features. For
> example, a media could be HDR or non-HDR, supports (or not)
> WCG, could be omnidirectional video, etc. One should not
> signal these types of features in the 'codecs' parameter.
> 
> It is recommended that the MPEG community reaches an agreement
> on these principles at this meeting and the work is
> coordinated between the IETF, W3C and MPEG to implement the
> necessary changes as an update to RFC 6381.
> 
> 3) References

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 04 Oct 2017 21:49:10 -0400

> If the 4CC of a sample entry indicates protected content (e.g., 'encv', 
> 'enca', 'enct' etc.), the 'codecs' parameter value for that track is any 
> the following values, separated by ".":
> * the original_format (e.g., 'avc1'), followed by any sub-parameters as 
> As a result, the following values would be possible for the 'codecs' 
> parameter:
> encv.cbcs.avc1.402567
> or
> encv.cenc.avc1.402567

I favor John's observation:

    However, multiplying parameter values to incorporate essentially
    orthogonal types of information is rarely a good idea and, as Ali's
    note sort of points out, it gets worse, exponentially worse, with
    each new type of information that is added in.  So my sense of good
    taste and instinct match Ali's and the proposal -- it is probably
    better to go through the pain of adding one or more parameters now
    than to overload "codecs".

It seems to me that there are interacting factors of
upward-compatibility and priority:  I assume that the higher-priority
goal is to achieve communication when possible, and it is a
lower-priority goal to prevent the resource-wasting situation of setting
up a media stream, only to discover once media is flowing that the the
receiver can't decode the media stream.

Adopting the N16785 proposal assumes that (1) New receivers will be
rolled out fairly quickly that can interpret the composite codec values,
and (2) Senders will know this and start generating the composite codec
values.  Because in order to avoid unnecessary media failures, senders
will not generate composite codec values until all receivers in the
ecosystem they are deployed in upgraded.  And as long as senders don't
generate composite codec values, none of the benefits of this proposal
can be realized.  So benefits can only be obtained at the end of the
roll-out of updated senders and receivers.

OTOH, once the IETF defines "encscheme" (or any other new parameter),
senders can start using it, because sending it does not cause
unnecessary failures.  And receivers can implement it without causing
upward-compatibility problems, as non-updated senders won't send that
parameter.  The benefits of avoiding useless media set-ups grow as
approximately the product of the adoption level among senders and the
adoption level among receivers.  Unnecessary media failures are avoided
completely.  Deployment can proceed among senders and receivers at
whatever pace is convenient.

Dale