Subject: Re: [Cellar] AV1 seeking

From: Cyril Concolato <[email-masked]>
Date: Thu, 2 Aug 2018 10:48:27 -0700

Sorry for the delay in responding. See more comments below.

On Wed, Jul 18, 2018 at 8:36 AM Steve Lhomme <[email-masked]> wrote:

> Hi Cyril,
>
> Thanks for clarifying some things.
>
> 2018-07-18 16:59 GMT+02:00 Cyril Concolato <[email-masked]>:
> >
> > On Wed, Jul 18, 2018 at 3:43 PM Steve Lhomme <[email-masked]>
> wrote:
> >>
> >> I asked Cyril Concolato to join the conversation so we can discuss the
> >> differences between our current mapping and the ISOBMFF/MP4 one he is
> >> working on.
> >
> > Thanks Steve.
> >
> > Some comments regarding the previous messages I read in the archives of
> this
> > list:
> >
> > "In MP4 they have [initial_presentation_delay_minus_one] in the
> > CodecPrivate. I did not understand it so far because it's not found in
> > the AV1 spec. But it seems to guarantee that to read frame 'f' you
> > need to decode X frame before that one. In our case that would be 5 to
> > have at least a decoded."
> > [CC] "initial_presentation_delay_minus_one" in the AV1-ISOBMFF is
> related to
> > the "initial_display_delay" in the AV1 codec spec. They express the same
> > delay but in different units: the latter expresses it in number of
> decoded
> > frames, while the former expresses it in number of presented frames. The
> > reason for this difference is that an application may not be able to know
> > when a frame is not decoded it only sees when it is output.
> > That said, this delay is not at all related to the seeking feature. It
> tells
> > you how many frames have to be decoded (or output) to guarantee smooth
> > playback if the decoder decodes at the decode rate of the level
> indicated in
> > the bitstream. It may be useful for low latency applications.
>
> OK but it is a duplicate value from the Sequence Header OBU may (or
> may not) be found in the configOBU.

[CC] This is not exactly a duplicate as it is expressed with a different
unit but yes they should convey the same information.

> Maybe it would be worth having a
> specific atom just for that ? And you have either this atom or the one
> with a Sequence Header OBU but not both.
>
[CC] The current syntax already allows that without boxing it. But I do
think there is a value for a player to have the file format one even if the
Sequence Header one is present, given that they are in different units.
Otherwise, the player has to do the conversion.

>
> There are many [initial_display_delay_minus_1] values in a single
> Sequence Header OBU. So I assume the value stored in MP4 is the
> maximum value ? IMO it should be said in the document.
>
[CC] There are multiple values if scalability is used, which is not covered
specifically by the current spec. Plus, unless it has changed in the base
AV1 spec, the first value is assumed to be the recommended one. But I agree
this could be clarified in the spec.

>
> Also in what cases a Sequence Header OBU would not be found in the
> configOBU ?

[CC] Just like for AVC you have avc3 or HEVC hev1. In Live Scenarios, you
might not know in advance the exact configuration you are going to produce,
yet you want to generate the initialization segment.

> IMO it should be specified. We may have the same
> requirement but it seems better to me to always have one. It helps
> knowing the profile of the codec to initiate the decoder instance.
>
[CC] In the case of local file playback, you have the first sample so you
have the codec. In the case of Adaptive Streaming, you have 'codecs'
parameter that can help.

> Otherwise we need to wait for the actual data when playback has
> already started (along with audio and subtitles).
>
[CC] Not necessarily if you have a manifest or a MIME type.

Cyril

>
> > "IMO this is a bad design because it mixes the container with the codec
> > (something done in the past in ogg with bad results). Seeking should
> > not ask the codec how it should be done. Also that means to mux such
> > an MP4 you'd need to scan the whole source first to verify the value
> > is correct or that information needs to be given to the muxer by the
> > encoder/packetizer (this information is not found in the stream)."
> > [CC] I don't understand what this means.
>
> Ogg is notorious for tying the container and the codec data. You can't
> know the timestamp from just reading ogg, you need to know the codec
> and analyse the data inside. So if seeking in AV1-in-ISOBMFF required
> the demuxer to parse the "initial_presentation_delay_minus_one" data
> would be a bit similar. But as you explained, it's not the case.
>
> >>
> >>
> >> IMO there are two main differences for now:
> >>
> >> - initial_presentation_delay_present /
> >> initial_presentation_delay_minus_one which we may or may not store
> >> (and where does it come from)
> >
> > [CC] Hope it's clearer.
>
> Yes
>
> >> - the configOBU that may be completely empty, whereas in our case we
> >> have exactly one Sequence Header OBU and the ones in the Segment must
> >> be very close to that one.
> >
> > [CC] The intent of the AV1-ISOBMFF spec is that the Sequence Header OBU
> has
> > to be present in-band and may be present in the configOBU and if present
> > shall be identical. In your spec, I don't understand what "very close
> > means". AV1 does not allow variation of Sequence Header within a CVS.
>
> "Very close" was in my email but that's not the wording in the
> MKV/WebM mapping. In that document there is this CVS documentation:
>
> "A Coded Video Sequence is a sequence of video frames where the
> contents of [sequence_header_obu] must be bit-identical for all the
> Sequence Header OBUs found in the bitstream before Matroska
> encapsulation except for the contents of [operating_parameters_info]."
>
> [operating_parameters_info] is allowed to change in a CVS in our case.
> This is the same as found in the AV1 specification 7.5 Ordering of
> OBUs:
>
> "Sequence header OBUs may appear in any order within a coded video
> sequence. Within a particular coded video sequence, the contents of
> sequence_header_obu must be bit-identical each time the sequence
> header appears except for the contents of operating_parameters_info."
>
> I suppose by "shall be identical" that's the change that is allowed.
> But is there are reason not to put a Sequence Header OBU in the
> AV1CodecConfigurationRecord ? In Matroska we advise to strip the
> similar ones from the bitstream to save space so that's a direct
> benefit of putting it there. But there's also a benefit for playback
> (a similar feature is missing from VP9 and causes hardware detection
> issues in VLC).
>
>
> > HTH,
> > Cyril
>
> Thanks again.
>
> >>
> >> --
> >> Steve Lhomme
> >> Matroska association Chairman
> >>
> >> _______________________________________________
> >> Cellar mailing list
> >> [email-masked]
> >> https://www.ietf.org/mailman/listinfo/cellar
>
>
>
> --
> Steve Lhomme
> Matroska association Chairman
>

---

From: Cyril Concolato <[email-masked]>
Date: Thu, 2 Aug 2018 14:31:20 -0700

Hi Andreas,

Thank you for your detailed, helpful comments. See below for my responses.

On Thu, Jul 19, 2018 at 1:47 AM Andreas Rheinhardt <
[email-masked]> wrote:

> Hello Cyril,
>
> thanks for joining us here.
>
> Cyril Concolato:
> > [CC] "initial_presentation_delay_minus_one" in the AV1-ISOBMFF is related
> > to the "initial_display_delay" in the AV1 codec spec. They express the
> same
> > tells you how many frames have to be decoded (or output) to guarantee
> > smooth playback if the decoder decodes at the decode rate of the level
> > indicated in the bitstream. It may be useful for low latency
> applications.
> >
> That confirms what I have been thinking all along. But nevertheless I
> have some questions (My interest stems from the fact that I am in favour
> of including a field indicating how many frames need to be decoded in
> advance):
>
> 1. What exactly is the "display model verification algorithm" that the
> AV1-ISOBMFF speaks about? Does this just mean that the bitstream so
> constructed should be tested whether it fulfills the requirements of
> annex E of the AV1 specification?
>
[CC] That text was written when the base AV1 spec was still moving and
Annex E did not exist. Yes, this is related to the decoder model, but not
the full model with its buffer constraints, just the display one. The idea
is that if you set a value smaller than 10 (max theoretical bound) you can
guarantee that a decoder that decodes at the limit of the decoding rate
given in the level will be able to present all the frames as intended, at
their presentation time.

>
> 2. If yes, then how can one test this precisely?
>
[CC] There were discussions in AOM about extending the software for that
but I'm not sure of the latest status.

>
> a) There are two operating modes in annex E.3: Resource availability
> mode and decoding schedule mode. The latter mode needs
> buffer_removal_time values for the check; but these values SHOULD have
> been removed from the bitstream when muxing into ISOBMFF so this can in
> general not be checked with the ISOBMFF file alone. Is it intended that
> the muxer checks this during muxing when the original file with the
> original values are still available or not?
> Or do you just test this in resource availability mode. For the resource
> availability mode equal_picture_interval equal should have been set to 1
> (i.e. crf video), but E.3.1 also says "If the parameters listed above
> are not specified by the bitstream, the parameters necessary to input
> into this model can be signaled by the application or some other means."
> I have not found that this crf requirement is in any way necessary. Are
> you simply using the timestamps derived from the ISOBMFF file (even if
> vfr) and use them to check in resource availability mode?
>
[CC] These are good points. Thank you for raising them. Again, initially
there was only the equivalent of the "resource availability mode". This
should be clarified. Also, as indicated in the AV1-ISOBMFF, the timing of
frames is given by the ISOBMFF structures not by inband values:
"The presentation times of AV1 samples are given by the ISOBMFF structures.
The timing_info_present_flag in the Sequence Header OBU (in the configOBUs
field or in the associated samples) SHOULD be set to 0. If set to 1, the
timing_info structure of the Sequence Header OBU, the
frame_presentation_time and buffer_removal_time fields of the Frame Header
OBUs, if present, SHALL be ignored for the purpose of timed processing of
the ISOBMFF file."

and this is true for validating the value of initial_presentation_delay as
indicated:
"set the frame_presentation_time field of the frame header of each
presentable frame such that it matches the presentation time difference
between the sample carrying this frame and the previous sample (if it
exists, 0 otherwise)".

I will try to make that clearer.

>
> b) If one has verified that one can use a value of k for
> initial_presentation_delay_minus_one, is this value also automatically
> applicable for starting the presentation at any RAP (or at least any
> keyframe/non-delayed RAP because -- after all, discarding the samples in
> front of a keyframe RAP should give a conformant bitstream)? The answer
> is of course tautologically true if "display model verification
> algorithm" includes checking presentation from every RAP onwards, but
> the current spec proposal talks only about constructing a (singular)
> hypothetical bitstream consisting of all the OBUs in the sample entry
> followed by all OBUs in all the samples.
> The reason I am asking this are bitstreams like this one (everything
> inside square brackets forms a temporal unit; &x means that a showable
> frame x is output via the show_existing_frames mechanism (if I am not
> mistaken, then it is allowed to output frames more than once -- and it
> might even make sense to do this at the beginning if there are several
> static frames at the beginning which sometimes happens with logos/black
> frames)):
> [A] [B] [&B] [C] [D E] [&D]
> A is a keyframe RAP, B an ordinary frame that is allowed to be output
> more than once. And it is output more than once. C is another
> non-delayed keyframe. One can start presentation as soon as one has A
> decoded (assuming the decoder can decode one frame during the
> presentation time of one frame):
> When one reaches the temporal unit [B], one has already decoded B. One
> can decode C while B is shown.
> When one reaches [&B], one has of course B still available. When one
> shows B a second time, one can decode D. This effectively means that
> when one starts at the very beginning, one enters the second CVS with
> two frames already decoded (as if one started decoding at C with
> initial_display_delay_minus_1 set to 1) and therefore one can use
> temporal units like [D E] with more than one frame to decode.
> In the above bitstream, one could use
> initial_presentation_delay_minus_one equal to zero for presentation from
> the beginning, but for decoding from C onwards one would need a value of 1.
>
[CC] The AV1 spec has changed at the last minute regarding the ability to
output a given frame multiple times. The published version allows it only
for non-key-frames. That said, your analysis seems right (as B is a
non-keyframe). As you note, the current text mentions constructing a single
bitstream, but that might not be useful enough because it does not give you
any information about what to do when starting at a subsequent RAP. I see 2
options: a) the text could be changed to apply the procedure at every RAP
leading to the max value over all CVS being used but that might introduce
latency in generating the file or in generating multiple sample entries
(one per set of CVS with a different value); or b) instead of signaling
this delay in the sample entry, we need to be able to associate a value
with each CVS start (i.e. sample groups). Thoughts?

>
> 3. a) The ISOBMFF specs contain the clause: "set the first
> initial_display_delay_minus_1 field of each Sequence Header OBU to the
> number of frames contained in the first
> initial_presentation_delay_minus_one + 1 samples,"
> Is this really intended? Shouldn't it read: "set the first
> initial_display_delay_minus_1 field of each Sequence Header OBU to the
> number of frames - 1 contained in the first
> initial_presentation_delay_minus_one + 1 samples,"
> After all initial_display_delay_minus_1 includes a minus 1, too; without
> the minus 1 one would have to decode the first frame of the sample
> number initial_presentation_delay_minus_one + 1 (zero-based), too,
> before the presentation can start.
>
[CC] Good catch!
Let's make sure we agree on some examples:
a) all TU are made of a single frame with show_frame=1 and all frames take
the same amount of time to be decoded and the timing is CFR. In this case,
there is a guarantee that you can present all frames at their presentation
time after the first frame has been decoded. So
initial_display_delay_minus_1 = 0. Given that all TU are made of 1 frame,
initial_presentation_delay_minus_one = 0.
b) There is one non-showable frame in the sequence and one TU has 2 frames.
So initial_display_delay_minus_1 = 1.
If the first TU contains 2 frames, then you can set
initial_presentation_delay_minus_one = 0.
If the second TU contains 2 frames, then you can set
initial_presentation_delay_minus_one = 1 (the number of frames in the first
2 TUs will be 3, so initial_display_delay_minus_1 will hypothetically be
set to 2 which is greater than the correct value (1), so this is fine).
if another TU contains the 2 frames, then you can set
initial_presentation_delay_minus_one = 1 (and in this case the hypothetical
value of initial_display_delay_minus_1 will match the minimum theoretical
value).
Do we agree?

> b) If one has a compliant AV1 bitstream where the Sequence Headers all
> include the initial_display_delay_minus_1 for every operating point,
>
[CC] First, they don't need to syntactically include it, we can assume the
values are known.
Then, as indicated in my previous response, the current text assumes only 1
operating point and if not, takes the first one.

then one can use the maximum of all these initial_display_delay_minus_1
> as initial_presentation_delay_minus_one if my above corrections turns
> out to be right and is accepted? (If it's not accepted, then one could
> even use the first the maximum - 1, couldn't one?)
>
[CC] I think there are two points here:
- the variation across CVSs: I agree taking the maximum would work but as
indicated above this could introduce latency and I'm wondering if signaling
at the CVS level is not preferable
- the value of initial_presentation_delay_minus_one depends on the value of
initial_display_delay_minus_1 but also on the distribution of the frames in
TUs.
In the following cases, initial_display_delay_minus_1 is constant but
initial_presentation_delay_minus_one differs
[A] [b c d] [e] [f] [&c] [&b]: initial_display_delay_minus_1 = 2 because
once c has been decoded you can display A while decoding d and the rest
will be decoded/presented correctly. initial_presentation_delay_minus_one =
1 because c is decoded when the decoding TU [b c d] outputs d.
[A] [b] [c] [d e f] [&d] [&e]: initial_display_delay_minus_1 = 2 because
once c has been decoded, you present A while decoding d, present b while
decoding e and present c while decoding f and the rest will follow, but
initial_presentation_delay_minus_one = 2 because c is decoded in the 3rd
TU.

Cyril

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 6 Aug 2018 09:02:05 +0200

> Sorry for the delay in responding. See more comments below.

No problem,

> have the codec. In the case of Adaptive Streaming, you have 'codecs'

But that's the problem. At least with libavcodec it selects the
hardware or software decoder after the first frame is parsed. But at
this point you already set the number of threads it can use (usually a
lot), except this is incompatible with the hardware decoders (there
are some workarounds but it still lead to HW performance issues). This
can be avoided by knowing the profile before libavcodec is
initialized. This is possible for all known codec except for VP9 which
doesn't have extradata, and we can't know if we need to use the 8 bits
decoder or the 10 bits decoder.

I would like to avoid this as much as possible. Especially since 12
bits HW support might not be widespread. But at least I know each bit
depth will be a different decoder in the hardware.

> [CC] Not necessarily if you have a manifest or a MIME type.

Relying on metadata outside a file to play a file is usually not a
good idea (which one has precedence when both information are found
and not equal). But I understand it's a common use case for adaptive
streaming. It just doesn't work for every other use case.