Subject: Re: [AVT] RFC3984bis - profile & level in offer/answer

From: "Allison, Art" <[email-masked]>
Date: Thu, 11 Sep 2008 12:18:52 -0400

Folks,
I am a mere observer here... But I wish to inform you that ATSC is in
the process of defining delivery of a special profile of AVC
(constrained by use of many such flags since profile and level were
simply not enough). We are using RTP over UDP for a broadcast delivery
system that enables addition of SVC supplementary signals.
And of course we have no negotiation possible as it is a one-way
broadcast service, so we must signal what <is> being sent.
  
The less that is explicitly defined in the RFCs, the more we have to
define.  It seems to me that other SDOs around the world will each
define different ways to signal constraints if there is not a common
way, especially for pre-encoded content.

We determined that Baseline was inadequate if not problematical to
signal - but that is a discussion we need not go into.

The more that can be common, the better; and we urge serious
consideration of Ye-Kui's request.

Art

Art Allison

Director Advanced Engineering, Science and Technology
 
National Association of Broadcasters
1771 N Street NW
Washington, DC 20036
Phone  202 429 5418 
Fax  202 775 4981
www.nab.org

Advocacy  Education  Innovation

|-----Original Message-----
|From: [email-masked] [mailto:[email-masked]] On 
|Behalf Of Roni Even
|Sent: Thursday, September 11, 2008 9:35 AM
|To: [email-masked]; [email-masked]
|Subject: Re: [AVT] RFC3984bis - profile & level in offer/answer
|
|YK,
|The sender can offer both Main and Baseline. The text may 
|clarify the issue of the flags letting the receiver decide how 
|to respond to the offer.
|Roni 
|
|-----Original Message-----
|From: [email-masked] [mailto:[email-masked]]
|Sent: Thursday, September 11, 2008 7:54 AM
|To: [email-masked]; [email-masked]
|Subject: RE: [AVT] RFC3984bis - profile & level in offer/answer
|
|
|Roni,
|
|I don't have a strong opinion here. Just to point out that 
|requiring the
|flags to be matched (except for level 1b) has no problem for live
|encoding, but for pre-encoded contents such a requirement may
|essentially reduce interoperability. For example, a Baseline encoder
|when it wants to generate a stream that is also decodable by a Main
|decoder, it typically mark the bitstream with Baseline and set flag1,
|while a Main encoder when it wants to generate a stream that is also
|decodable by a Baseline decoder, it typically mark the bitstream with
|Main and set flag0. 
|
|BR, YK 
|
|>-----Original Message-----
|>From: ext Roni Even [mailto:[email-masked]] 
|>Sent: 10 September, 2008 22:54
|>To: Wang Ye-Kui (Nokia-NRC/Tampere); [email-masked]
|>Subject: RE: [AVT] RFC3984bis - profile & level in offer/answer
|>
|>YK,
|>My preference is to mandate profile match and level can be 
|>downgraded based
|>on the hierarchy.
|>I assume that 1b should be in the hierarchy between level 1 and 1.1
|>As for the flags my view is that they should be matched.
|>
|>I think that H.264 compliance point is based on defined 
|>profiles. And the
|>levels gives hierarchy inside a specific profile.
|>I think that will make it simple enough. Note that there are more H264
|>parameters defined and offer enough challenge. 
|>Roni Even
|>
|>
|>-----Original Message-----
|>From: [email-masked] [mailto:[email-masked]] On Behalf Of
|>[email-masked]
|>Sent: Wednesday, September 10, 2008 10:30 PM
|>To: [email-masked]
|>Subject: [AVT] RFC3984bis - profile & level in offer/answer
|>
|>
|>Folks,
|>
|>I feel that this issue should be publicly discussed. 
|>
|>The media type parameter profile-level-id includes three bytes,
|>profile_idc (8 bits), some flags and reserved bits (8 bits), and
|>level_idc (8 bits). It has been discussed that level 1b is signaled in
|>an ad-hoc manner, such that a lower value of level_idc does not always
|>mean a lower level. Actually, there is also an issue for 
|profile, which
|>has not been mentioned as far as I know. 
|>
|>For example, a bitstream with profile_idc=66 (Baseline profile) and
|>constraint_set1_flag (the second bit of the middle byte)=1 can also be
|>marked with profile_idc=77 (Main profile) and 
|constraint_set0_flag (the
|>first flag)=1, and a Baseline profile decoder of a particular 
|level can
|>decode any bitstream with constraint_set0_flag=1 with a level 
|>not higher
|>than the particular level. This means the following in SDP 
|>offer/answer:
|>
|>- If the direction attribute is sendrecv, two different 
|>profile-level-id
|>values 42401E and 4D801E in the offer and the answer should be
|>considered equivalent. In this case, if other symmetric parameters
|>match, the offer/answer should be considered matched. Just to be more
|>complicated (but accurate :-), the second byte in the first 
|>value 42401E
|>can be replaced with any value with the second bit
|>(constraint_set1_flag) equal to 1, and the second byte in the second
|>value 4D801E can be replaced with any value with the first bit
|>(constraint_set0_flag) equal to 1, and the offer/answer are still
|>matching. 
|>
|>- If the offer has direction attribute sendonly and
|>profile-level-id=42XX1E (XX is any value in the range of 00-FF,
|>inclusive), and the answer has direction attribute recvonly and
|>profile-level-id=YYZZ1E (YY is any value in the range of 00-FF,
|>inclusive, and ZZ is any value with the first bit equal to 
|1), then the
|>offer/answer should be considered matched.
|>
|>And there are many other different combinations of profiles and
|>constraint flags to be considered to make the list complete. RFC 3984
|>considers all these cases as un-matched. 
|>
|>So, opinion requested: should this issue be addressed in the 
|bis draft?
|>
|>BR, YK
|>
|>_______________________________________________
|>Audio/Video Transport Working Group
|>[email-masked]
|>https://www.ietf.org/mailman/listinfo/avt
|>
|>
|
|_______________________________________________
|Audio/Video Transport Working Group
[email-masked]
|https://www.ietf.org/mailman/listinfo/avt
|
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Randell Jesup <[email-masked]>
Date: Thu, 11 Sep 2008 16:49:08 -0400

"Allison, Art" <[email-masked]> writes:
>simply not enough).

Please describe the needs and constraints.  A separate RFC can always be
done, but if we can roll it in here (and it's well-enough "baked" for
inclusion) that saves everyone hassle (and may improve interoperability).

> We are using RTP over UDP for a broadcast delivery

Please define "addition of SVC supplementary signals".  Do you mean
enhancement streams?

>broadcast service, so we must signal what <is> being sent.

That's fine, RFC 3984 and most others like it work for "declarative" uses
as well (non-offer/answer use).  We just have to make sure we think
those through - for example, 3984(bis) specifies in sendonly mode that
a bunch of parameters are declaring the properties of the stream instead
of what the offerer can receive.

>signal - but that is a discussion we need not go into.

Actually, it would help us to know the rationale, and the reasons for
rejection of that, and what it is you plan to provide, and how it would
be used.

It may be fine (depending on the usage) for you to specify it's main
profile, if the decoders really need to decode main in order to use this.

>consideration of Ye-Kui's request.

So, supposedly if constraint_set0 is set, then a baseline decoder can
decode it, and if constraint_set1 is set, then a main decoder can decode
it, both regardless of the profile marked.  

(Ok, so this is a mildly hokey design - why have the profile in the stream
at all when you effectively have a bitfield of "type X decoder can decode
this stream"?)

So, given that, the design in 3984 which locks profile and all constraint
bits (including the 4 reserved ones) to be an exact match is probably not
a good design.

It makes some sense for "live" encoders, in that those can be used to
initialize them.  But it's a real problem for requesting a pre-encoded
stream, since the requester only knows what they can decode, not how the
stream is encoded.

>From rfc 3984:
       If the profile-level-id parameter is used to indicate properties of
       a NAL unit stream, it indicates the profile and level that a decoder
       has to support in order to comply with [1] when it decodes the
       stream.  The profile-iop byte indicates whether the NAL unit stream
       also obeys all constraints of the indicated profiles as follows.  If
       bit 7 (the most significant bit), bit 6, or bit 5 of profile-iop is
       equal to 1, all constraints of the Baseline profile, the Main
       profile, or the Extended profile, respectively, are obeyed in the
       NAL unit stream.

       If the profile-level-id parameter is used for capability exchange or
       session setup procedure, it indicates the profile that the codec
       supports and the highest level supported for the signaled profile.
       The profile-iop byte indicates whether the codec has additional
       limitations whereby only the common subset of the algorithmic
       features and limitations of the profiles signaled with the
       profile-iop byte and of the profile indicated by profile_idc is
       supported by the codec.  For example, if a codec supports only the
       common subset of the coding tools of the Baseline profile and the
       Main profile at level 2.1 and below, the profile-level-id becomes
       42E015, in which 42 stands for the Baseline profile, E0 indicates
       that only the common subset for all profiles is supported, and 15
       indicates level 2.1.

           Informative note: Capability exchange and session setup
           procedures should provide means to list the capabilities for
           each supported codec profile separately.  For example, the
           one-of-N codec selection procedure of the SDP Offer/Answer model
           can be used (section 10.2 of [7]).
 
So in declarative use (first para), it talks about the properties of the
encoded stream.  You support them or not.  It *appears* that a decoder
can ignore "profile" (largely?) and just look at the "constraint" bits.

In negotiated or "interactive" use (though possibly one-way), it indicates
profile supported and highest level at that profile.  As I read it, though
(and perhaps I'm wrong; it's tough to parse) the profile_iop byte has a
subtly different meaning here: it means the codec (encoder/decoder)
supports *only* the common subset of the profile and the constraint bits.
Note that "common subset" is somewhat loosely defined, given that
Main is not a pure superset of Baseline (for example, no FMO in main).

So "Main plus constraint_set0 and constraint_set1" is more constrained than
pure Baseline - none of the added Main features can be used, AND none of
the Baseline-only features (like FMO).  Also, IF I read it right (without
the H.264 spec), "Main plus constraint_set0 and constraint_set1" should be
*identical* to "Baseline plus constraint_set0 and constraint_set1", except
for SPS NAL units will have a different profile in them.

Actually, it appears that there's never a reason to set constraint_set1 and
constraint_set0 together.  Main + constraint_set0 should be identical to
Baseline + constraint_set1, and either with both set is also identical.
Correct?

Example: I can decode baseline, so I request 42801e (baseline, baseline
constraints, level 3.0).  The encoded video was encoded as Main, main
constraints, baseline constraints, level 3.0 (4dc01e).  That video *should*
be decodable by me as-is.  3984 doesn't allow the answerer to specify the
actual profile-level-id in the pre-encoded stream.  It could 
a) lie and say 42801e but actually send 4dc01e (ouch - affects
   sprop-parameter-sets too).
b) modify it on the fly: reply with 42801e and modify each SPS NAL (and
   sprop-parameter-sets if given) to say that -- might be tough, especially
   if the data is already in RTP packets and you have to deal with
   FU-A/B's/STAPS/etc.
c) Don't ever pre-encode something with "Main" profile if the
   constraint_set0 bit is set; encode it as Baseline and add the
   constraint_set1 bit.  That simplifies things.

>>For example, a bitstream with profile_idc=66 (Baseline profile) and
>>constraint_set1_flag (the second bit of the middle byte)=1 can also be
>>marked with profile_idc=77 (Main profile) and constraint_set0_flag (the
>>first flag)=1, and a Baseline profile decoder of a particular level can
>>decode any bitstream with constraint_set0_flag=1 with a level not higher
>>than the particular level.

So, it seems like we have a significant disconnect between the people who
wrote 3984 and what capabilities H.264 provides.  (Though I'd like to know
how Baseline and Main H.264 decoders when presented with those streams will
react!  (Including SPS (Sequence Parameter Set) NALs with a "higher" profile.)  

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: <[email-masked]>
Date: Fri, 12 Sep 2008 10:25:34 +0300

>it, both regardless of the profile marked.  

Yes.

>at all when you effectively have a bitfield of "type X decoder 

Correct for Baseline, Main, Extended and the new Mutiview High profiles.
Historical reasons. There are other profiles without such a bitfield.
Note also that a Baseline bitstream (with profile_idc=66) may have flag0
equal to either 0 or 1.

>bits (including the 4 reserved ones) to be an exact match is 

I agree.  

>can ignore "profile" (largely?) and just look at the "constraint" bits.

Correct for Baseline, Main, Extended and the new Mutiview High profiles,
which have the correspoinding bit.

>profile supported and highest level at that profile.  As I 

Is this piece of semantics in 3984 absolutely correct? My impression is
that you may actually offer or answer a lower level than what the codec
can do. 

>read it, though

"Common subset" is a clearly defined in H.264/AVC, for all profiles. For
example, the common subset of Baseline and Main is equal to Baseline
excluding FMO, ASO and Redundant Slices.

>the Baseline-only features (like FMO).  

Correct.

> Also, IF I read it right (without

Yes, *identical* from decoding capability point of view. Note again that
"Main plus constraint_set0 and constraint_set1" is *identical* to "Main
plus constraint_set0", as "constraint_set1" may or may not be set for
Main bitstreams. Similar for Baseline. 

>Baseline + constraint_set1, and either with both set is also identical.

Correct. See above. 

>be decodable by me as-is.  

Correct. If the decoder is really conforming to H.264/AVC, it *must* be
able to decode the bitstream as is, otherwise the decoder cannot be
claimed conforming to Baseline profile @ Level 3.0. 

>3984 doesn't allow the answerer to 
>specify the
>actual profile-level-id in the pre-encoded stream. 

If the interpretation of any profile-level-id, when used as capability
indication, is support of certain coding tools (i.e. the common subset)
at certain level, than the answerer can specify in the answer either the
actually value of profile_level_id or an equivalent profile_level_id,
the the mapping of profile_level_id values to "common subset at certain
level" is multiple-to-one, not one-to-one. 

> It could 

This sounds unacceptable. 

>   constraint_set1 bit.  That simplifies things.

But this is just the other side of the coin - consider when if the Main
decoder requests a bitstream in a similar situation.  

>react!  (Including SPS (Sequence Parameter Set) NALs with a 

If decoders are really conforming to H.264/AVC, then they must be ble to
decode such bitstreams as they are, otherwise the decoder cannot be
claimed conforming. But of course decoders (or people) may make false
claims

BR, YK

---

From: "Roni Even" <[email-masked]>
Date: Fri, 12 Sep 2008 11:00:08 +0300

Hi,
>(Ok, so this is a mildly hokey design - why have the profile in the 
>stream at all when you effectively have a bitfield of "type X decoder 
>can decode this stream"?)

This is not fully correct according to H.264 which defines compliance to a
profile, If you state the you support a profile and want to be H.264
compliance you must support the profile regardless of the constrain flags.
So if you support baseline you support baseline, you can receive baseline
with constrain flags0 set but what you support is baseline and not main. 
This was discussed also in JVT.

Roni

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Friday, September 12, 2008 10:26 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [AVT] RFC3984bis - profile & level in offer/answer

Yes.

I agree.  

Correct.

Correct. See above. 

BR, YK