Subject: Re: [Cellar] 360/VR videos in Matroska

From: Steve Lhomme <[email-masked]>
Date: Sun, 4 Sep 2016 14:46:00 +0200

2016-08-30 18:40 GMT+02:00 Michael Bradshaw <[email-masked]>:
> Matroska currently doesn't have a way to specify projection information for
> 360/VR videos. Google's Spherical Video V2 RFC draft was just updated a few
> days ago to include some new elements specifying projection and viewer pose
> information[1].
>
> These proposed elements haven't made their way over to the WebM
> specification (and I haven't asked the WebM team if they intend to
> incorporate them), but I think it raises a good question about Matroska
> support for these kinds of things.
>
> I don't know of any past work in specifying projection information in
> Matroska. Assuming there hasn't been any, is there interest in adding it
> (I'm assuming the answer is still yes, from those I asked in Berlin)? Do the

Since I'm currently adding 360° support in VLC the answer is a
definite YES. So far we only have MP4 sources. For now we only support
some XML embedded in an MP4 UUID XML360 box. The XML is similar to the
one defined here:
https://github.com/google/spatial-media/blob/master/docs/spherical-video-rfc.md#user-content-appendix-1---global-metadata-sample

There's already hardware that saves directly to this format.

> elements from the Spherical Video V2 RFC draft look like good candidates?

It's a good start ;)

I think the 0.0 default value should be 0x0p+1.

Also I think the ProjectionType value 0 should be None corresponding
to the current situation when the field doesn't exist.

Are the Yaw/Pitch/Roll used in practice ?;

> (For the record, I'm just breaking the ice here and getting the conversation
> started; this isn't a proposal to adopt the new elements just quite yet)
>
> [1]:
> https://github.com/google/spatial-media/blob/master/docs/spherical-video-v2-rfc.md#webm-matroska
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 4 Sep 2016 15:09:49 +0200

> to the current situation when the field doesn't exist.

Never mind, Rectangular (0) is actually the regular/flat projection.

---

From: Aaron Colwell <[email-masked]>
Date: Tue, 06 Sep 2016 14:43:12 +0000

Comments inline...

> > I think the 0.0 default value should be 0x0p+1.

I don't really understand this suggestion. Could you please clarify. Sorry
if this may be obvious to others, but I'm new to this list. :)

> > Are the Yaw/Pitch/Roll used in practice ?;

Yes. They are particularly helpful when you want to use multiple
directionally-biased projections of the same scene. A simple, but not
ideal, example is using the increased pixel density at the equirect poles
as a "directional projection". You could create multiple transcodes of a
scene where different yaw/pitch/roll combinations move different parts of
the sphere to the poles so you could have increased resolution in that
direction. Then a player could switch between these in a DASH-like manner
based on where the viewer is looking. Equirect is not a particularly great
projection for this, but it gets the idea across. The yaw/pitch/roll fields
help the client map these different transcodes into a single coordinate
space.

Aaron

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 6 Sep 2016 18:09:59 +0200

> if this may be obvious to others, but I'm new to this list. :)

We try to use a more standardized way to express floats. Obviously for
0.0 there's not problem but for other values it's better expressed in
a non ambiguous hexademical format.

See this section
https://github.com/Matroska-Org/ebml-specification/blob/master/specification.markdown#textual-expression-of-floats

> directionally-biased projections of the same scene. A simple, but not ideal,
> example is using the increased pixel density at the equirect poles as a
> "directional projection". You could create multiple transcodes of a scene
> where different yaw/pitch/roll combinations move different parts of the
> sphere to the poles so you could have increased resolution in that

OK, got it.

---

From: Nithin Mathew Kurien <[email-masked]>
Date: Tue, 6 Sep 2016 21:41:38 +0530

On Tue, Sep 6, 2016 at 8:13 PM, Aaron Colwell <[email-masked]> wrote:

>> > https://github.com/google/spatial-media/blob/master/
>> docs/spherical-video-rfc.md#user-content-appendix-1---
>> global-metadata-sample
>> >

This refers to the representation of floats in EBML as hexadecimal
constants for better precision, according to the C11 notation (
https://github.com/Matroska-Org/ebml-specification/blob/master/specification.markdown#textual-expression-of-floats).
This is a very recent feature, adopted a few weeks ago by CELLAR. :)