Subject: Re: [Cellar] Mapping of MOV clean aperture values to Matroska

From: Steve Lhomme <[email-masked]>
Date: Sun, 2 Dec 2018 16:44:35 +0100

Hello,

For VLC the values are read but they are likely lost along the way to
the display by the decoder or the video output. You may try this file
to check if your video output in VLC supports cropping:
http://v2v.cc/~j/theora_testsuite/offset_test.ogv

This illustrates the issue with cropping at the container level (and
ffmpeg) the container may have some values set but some codecs also
have such info and may overwrite this value. And there's not really
any way to tell who is write. It may depend on the codec and the
container. In Matroska we assume that we override what the codec says.
But I don't think that's how it's handled in ffmpeg. Also there's the
question of what these values apply to: on the decoded surface, which
may include padding on the right/bottom or the area that's supposed to
be visible. In Matroska the "PixelWidth/PixelHeight" correspond to the
visible area, on which we apply extra cropping. (this is currently not
written anywhere).

As for the MOV atom, is there are spec somewhere ? Or is it clear that
it's some cropping ? Is it in the supposedly visible area or a
supposedly padded area (which may not correspond in ffmpeg to the
padding Quicktime would use) ?
Le mar. 27 nov. 2018 à 18:44, Kieran O Leary
<[email-masked]> a écrit :
>
> Hey,
>
> Thanks for replying:
>
> On Sun, 25 Nov 2018, 20:20 Dave Rice <[email-masked] wrote:
>>
>> Hi Kieran,
>>
>> > On Nov 23, 2018, at 4:56 AM, Kieran O Leary <[email-masked]> wrote:
>> >
>> > Hi,
>> >
>> > Would I be correct in saying that ideally, Clean Aperture values in a
>> > MOV/MPEG-4 file should map to the PixelCrop element in Matroska?
>>
>> Yes, they conceptually do the same thing but I’m a little confused about how to use the vert/horiz offset fractions of the clap atom to convert to the pixel crop atom. I tried in both QuickTime 7 and X, and while QuickTime X uses the cleanAperature w/h fractions to present a cropped image, QuickTime 7 doesn’t but appears to adjust the aspect ratio according to those values. QuickTime X does seem impacted by non-zero offset values not in a way that I consider coherent. Do you have any examples of un-centered aperatures? I was hoping that IMX files would, does the ones I have don’t use any aperature.
>
>
> I don't think I've ever seen non-centered.. but it sounds like Matroska is capable of a mapping,but the actual quicktime implementation sounds inconsistent? I guess this could be specified explicitly by an archivist who will hopefully be able to determine the intended rendering of the original QuickTime file..
> Also it looks like here's yet another related ticket:
> https://trac.ffmpeg.org/ticket/4489
>
>>
>> > I'm asking about this in regards to this ffmpeg ticket -
>> > https://trac.ffmpeg.org/ticket/7437
>> > As it appears that FFmpeg does not currently read the values in the
>> > clap atom and it also does not map them to Matorksa files when
>> > remuxing. It seems to make the most sense to map them to PixelCrop,
>> > but I'd like to hear it from folks more knowledgable than me.
>>
>> Seems reasonable to me, but I’m not exactly certain what the mapping would be.
>
>
> I was considering  doing some experiments with some common PAL scenarios ( crop to 703x576 with a PAR of 59:54, or crop to 704x576 with PAR of 12:11) and using mkvpropedit to insert the metadata. I'll have to look beyond VLC to other players that can display these crops and aspect ratio conversions for the moment.
>
>>
>> > Another associated issue with the display of these kinds of values is
>> > with VLC as it appears that
>> >
>> > 1) PixelCrop values in Matroska are not supported by VLC at the moment
>> > - https://trac.videolan.org/vlc/ticket/21192
>> > 2) Clean aperture values are not supported by VLC at the moment
>> > https://trac.videolan.org/vlc/ticket/21179
>>
>> Sounds like these tickets are in my priority order as well.
>
>
> Cheers,
>
> Kieran O'Leary,
>
> Irish Film Institute
>>
>> Dave Rice
>>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Kieran O Leary <[email-masked]>
Date: Sun, 2 Dec 2018 16:17:57 +0000

Hi Steve,

On Sun, Dec 2, 2018 at 3:44 PM Steve Lhomme <[email-masked]> wrote:

> That's pretty cool. ffplay and mpv seem to display the full image, but VLC
displays a crop. At Teasdale in my home Ubuntu 18.04 desktop.

> container.

I"m mostly familiar with uncompressed video within MOV, which is then
compressed to FFV1/MKV, and in these cases you would only have values in
the container. I hadn't thought of the other codecs that could have
stream-level cropping.

> In Matroska we assume that we override what the codec says.
> it's some cropping ?

I think this is as good of a spec that is freely available - the 'clap
section'
https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html#//apple_ref/doc/uid/TP40000939-CH205-125850

> Is it in the supposedly visible area or a

Unfortunately I don't fully understand what you mean here by supposedly
visible and padded. PAL video has padded black pillarboxing (i have seen
expections to this) that is not intended to be viewed. These clean aperture
values specify the area that is to be displayed to the viewer. These values
are also essential to generating the correct aspect ratio. If the 720x576
pixels is transformed with a PAR of the PAL standard - 59:54, then you get
786x576 aka 1.364:1, but if the 59:54 value is applied with the clean
aperture values taken into consideration, then you will get the correct 4:3
aspect ratio.

Best,

Kieran O'Leary,
Irish FIlm Institute.

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 2 Dec 2018 17:26:00 +0100

On 02/12/2018 16:44, Steve Lhomme wrote:
> [...] some codecs also
> have such info and may overwrite this value.

Which one?
e.g. AVC crop values are more for handling widths/heights not multiple 
of the macroblock size, and output from the decoder is the one cropped, 
then the clean aperture from the container is applied (BTW, this is 
sometimes not what the user wants :(,  I got complains that I show the 
clean aperture size and not the encoded size in MediaInfo).

> [...]

https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFChap2/qtff2.html

"The clean aperture defines the part of the encoded pixels to be 
displayed. The pixel aspect ratio is the aspect ratio of the encoded 
pixels. Conceptually the encoded pixels are decompressed, stretched (or 
shrunk) based on the pixel aspect ratio, and extra pixels are cropped 
off according to the clean aperture."

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 3 Dec 2018 08:45:37 +0100

Le dim. 2 déc. 2018 à 17:18, Kieran O Leary <[email-masked]> a écrit :
> That's pretty cool. ffplay and mpv seem to display the full image, but VLC displays a crop. At Teasdale in my home Ubuntu 18..04 desktop.

This is one tricky corner case that we don't fully support everywhere.
But we're supposed to.

> I"m mostly familiar with uncompressed video within MOV, which is then compressed to FFV1/MKV, and in these cases you would only have values in the container. I hadn't thought of the other codecs that could have stream-level cropping.

There's Theora at least (see above) probably some modern MPEG ones as
well since the "blocks" have to be some power of 2 sizes but the main
source may not be.

> I think this is as good of a spec that is freely available - the 'clap section' https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html#//apple_ref/doc/uid/TP40000939-CH205-125850

OK, so we should support it in VLC (which we don't) and that seems to
map more or less to Matroska values.

> Unfortunately I don't fully understand what you mean here by supposedly visible and padded. PAL video has padded black pillarboxing (i have seen expections to this) that is not intended to be viewed. These clean aperture values specify the area that is to be displayed to the viewer. These values are also essential to generating the correct aspect ratio. If the 720x576 pixels is transformed with a PAR of the PAL standard - 59:54, then you get 786x576 aka 1.364:1, but if the 59:54 value is applied with the clean aperture values taken into consideration, then you will get the correct 4:3 aspect ratio..

As said above modern codecs use Blocks of 8x8, 16x16, etc to encode
pictures. There's codec padding if your have a source of 3x3 for
example. But to use SIMD (fast assembler code) you may want to align
to 32 pixels (to use AVX-2 for example) that's purely a decoder
optimization and depends on the implementation. But that's extra
padding.

libavcodec (in ffmpeg) requires a padding of 32 on all planes even if
the source doesn't require it. That means it forces a first layer of
padding/cropping. But the container/codec one should be applied
nonetheless. It's probably not done here.

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 3 Dec 2018 08:50:15 +0100

> map more or less to Matroska values.

I see you submitted a feature request already, excellent
https://trac.videolan.org/vlc/ticket/21179