Subject: RE: [MMUSIC] Offer/Answer model does not work for H.264 parameters

From: "Jae-Hwan Kim" <[email-masked]>
Date: Fri, 26 Mar 2004 02:22:05 +0900

Hi Magnus,
Question for you. How to provide multiple capability list? 
One possible choice we can use is to make multiple 'm='line in the SDP.
The second is something like RTSP transport headerfield.
Or... do you have any idea you already considering?

Thanks in advance and Best Regards,
Jae-Hwan

> -----Original Message-----
> From: Magnus Westerlund [mailto:[email-masked]] 
> Sent: Thursday, March 25, 2004 6:57 PM
> To: mmusic (E-mail)
> Cc: Stephan Wenger; Thomas Stockhammer; Dave Singer; Miska Hannuksela
> Subject: [MMUSIC] Offer/Answer model does not work for H.264 
> parameters
> 
> 
> Hi,
> 
> In the effort to finalize H.264 we authors very trying to 
> fine tune the 
> Offer/Answer section the RTP payload format for H.264. Latest 
> published 
> version is:
> 
> http://www.ietf.org/internet-drafts/draft-ietf-avt-rtp-h264-04.txt
> 
> When doing this we have found a serious problem with the H.264 
> parameters and the negotiation model in offer/answer (O/A). 
> The problem 
> is that H.264 has 5 parameters that are declarative and set 
> by the media 
> sender. These parameters are:
> - max-don-diff
> - init-buf-time
> - parameter-sets
> - interleaving-depth
> - deint-buf-size
> 
> More information on them can be found in the last version of the RTP 
> payload format. Some of them can also be used for capability 
> declaration, but for that functionality there is no problem. 
> The problem 
> is when they are used to describe a media stream that will be sent to 
> the receiver. For example parameter-sets is parameter 
> carrying decoder 
> configuration information. This information is determined by 
> the encoder 
> implementation and configuration, prior or during encoding 
> (in case of 
> pre-encoded material). The whole H.264 coder is built on giving the 
> encoder the freedom to set the parameter-sets, and it can freely 
> determine these parameters within the limitation set by the 
> profile and 
> level in use.
> 
> This clashes with the O/A model that is based on that the receiver 
> declares what it can receive. For a media encoder and its 
> payload format 
> it means that all media parameters that one specifies are 
> either a fixed 
> configuration on how the information I will receive must look 
> like, or a 
> capability declaration. An example of the first case is the 
> AMR payload 
> format, where a receiver would specify that the payload format 
> configuration used are octet-aligned mode with interleaving. 
> A case of 
> the second is the H.264 "max-br" parameter which defines the maximum 
> bit-rate that a receiver is capable of handling, thus puts 
> restriction 
> on what the sender is allowed to use.
> 
> In the case of the 5 parameters above they all work in the opposite 
> direction. The sender determines and declares the 
> configuration that it 
> will use. As a complication the possible values for these 
> parameters are 
> restricted through receiver declared capabilities. In the case of the 
> parameter-sets the limitation is based on video codec profile 
> and level 
> as declared by the receivers "profile-level-id" parameter. In 
> the case 
> of the other parameters they are restricted by the capability 
> version of 
> the two parameters "interleaving-depth", and "deint-buf-size".
> 
> To get this two work in a full fledged way, the negotiation 
> taking place 
> between a H.264 sender and receiver would be the following.
> 1. Receiver declares its capabilities to sender.
> 2. Sender declares its configuration that fits within the receivers 
> capabilities.
> 
> However I don't see how this would be possible to get to work in O/A. 
> Any suggestion how to get this to work is greatly appreciated.
> 
> One possible solution that will give partial functionality 
> would be the 
> following.
> 
> Offerer declares its own capabilities on what it will receive. At the 
> same time it assumes symmetric configuration and declares the 
> parameters 
> it will use to send that falls within its own receiver capabilities. 
> This will require splitting the "interleaving-depth", and 
> "deint-buf-size" two different parameters, one for 
> capabilities and one 
> for declaration. The offerer will have to make a leap of faith as it 
> must construct the sender configuration prior to knowing the 
> receivers 
> capabilities.
> 
> The answerer receiver the offer. If its capabilities matches those of 
> the sender it can declare its receiving capabilities equal to those 
> capabilities. Then it also declares its sending parameters that fits 
> within the offerer's capabilities. If the sender does not 
> have matching 
> capabilities it must remove that media alternative. To avoid total 
> rejection of a stream the offerer can include multiple payload types 
> with different levels of capabilities.
> 
> When the offerer receives the answer it checks which of its 
> offers the 
> answerer accepted and what parameters the answerer will use to send.
> 
> This will put certain restriction on how to handle the 
> payload types and 
> their numbers. Also the risk of rejection will be high if the offerer 
> and answerer are not well matched, or the offerer goes to 
> considerable 
> effort of constructing multiple well structured payload types.
> 
> Any more problems with the above proposal?
> 
> 
> Cheers
> 
> Magnus Westerlund
> 
> Multimedia Technologies, Ericsson Research EAB/TVA/A
> ----------------------------------------------------------------------
> Ericsson AB                | Phone +46 8 4048287
> Torshamsgatan 23           | Fax   +46 8 7575550
> S-164 80 Stockholm, Sweden | mailto: [email-masked]
> 
> 
> 
> This communication is confidential and intended solely for 
> the addressee(s). Any unauthorized review, use, disclosure or 
> distribution is prohibited. If you believe this message has 
> been sent to you in error, please notify the sender by 
> replying to this transmission and delete the message without 
> disclosing it. Thank you.
> 
> E-mail including attachments is susceptible to data 
> corruption, interruption, unauthorized amendment, tampering 
> and viruses, and we only send and receive e-mails on the 
> basis that we are not liable for any such corruption, 
> interception, amendment, tampering or viruses or any 
> consequences thereof.
> 
> 
> _______________________________________________
> mmusic mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/mmusic
> 

_______________________________________________
mmusic mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/mmusic

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 26 Mar 2004 11:10:24 +0100

HI Jae-Hwan,

Multiple capability lists are accomplished easiest to my knowledge by 
including multiple RTP payload types. For example on payload type is a 
base-line with single-nalu mode payload format that will be possible to 
use by any receiver that implements H.264 with RTP transport. Then one 
can create another payload type which has higher requirements on 
receiver and senders and uses other modes of the payload format.

Having multiple "m=" lines is a bad idea in my perspective for this.

Cheers

Magnus

-- 

Magnus Westerlund

This communication is confidential and intended solely for the addressee(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you believe this message has been sent to you in error, please notify the sender by replying to this transmission and delete the message without disclosing it. Thank you.

E-mail including attachments is susceptible to data corruption, interruption, unauthorized amendment, tampering and viruses, and we only send and receive e-mails on the basis that we are not liable for any such corruption, interception, amendment, tampering or viruses or any consequences thereof.

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 26 Mar 2004 13:55:52 +0100

Hi,

I thought about the issue and there is a reason why one could need to 
have multiple media lines: to be able to declare asymmetric sending and 
receiving capabilities. This is best illustrated by the following 
example. It also clarifies a bit about what restrictions will need to be 
applied to make the negotiation mechanism work.

Example:
Below the offerer gives two sets of configurations. The parameter-sets 
are based on the offered profile and level. Notice that I am not very 
strict with parameter names or if levels exist. it is simply an example.

Offer:
PT: parameters
m= 97 98
97: profile-level:extended, level=3; interleaved-mode; parameter-sets=O-1
98: profile-level:baseline,level=1; single-nalu-mode; parameter-sets=O-2

Parameter set O-2 would be used by the offerer to send a stream in 
accordance with the declared PT 98 from the offerer to the answerer.

If the answerer would return something like this:
PT: parameters
m= 100 101
101: profile-level:extended, level=5; interleaved-mode; parameter-set=A-1
100: profile-level:baseline,level=1; single-nalu-mode; parameter-set=A-2

Then we have a problem:
How to know if A-1, is a sender declaration belonging to payload type 97 
or 98. This could be resolved in two ways:

1. Restrict payload type numbers to be the same, and use these to match.

2. Restrict the payload type declaration to be exactly the same, so one 
can use the parameters string to match that it belongs to the 
corresponding declaration.

3. Encode the senders parameter with what payload type is belongs to.

There is a warning in RFC 3264 that due to H.323 there is certain cases 
where the PTs must change. This would make 1 in that case impossible.

2. will probably work better, as it would allow the answerer to add 
additional payload types for further capabilities. Please note that 
there is a requirement in O/A that at least one PT in a answer is from 
the offer. This solution will require more difficult comparisons.

3. This will make it impossible to have only a single round of offer and 
answer, as the initial offerer would need to update his offer to match 
the PTs that the answerer declares. Therefore I don't see this a viable 
solution.

If one applies alternative 2 above an answer would need to look like this.

The answerer would return something like this:
PT: parameters
m= 101 102 100
101: profile-level:extended, level=5; interleaved-mode; parameter-set=A-1
100: profile-level:baseline,level=1; single-nalu-mode; parameter-set=A-2
102: profile-level:extended, level=3; interleaved-mode; parameter-set=A-3

The problem with this is that the offerer would need to update its offer 
and include a configuration in accordance with the Answers PT 101 in its 
offer. Thus requiring it to do another round of O/A.

It also has the problem with that the Offer can't declare its sending 
parameters without also accepting to receive them, unless there is a 
send only stream. For send and receive streams it will not be possible 
to have non-symmetric receiving capabilities between the end-points.

The solution as I see to this problem would require the offerer and 
answer to use at least two media streams. Otherwise an offerer would 
need to declare the video stream as two separate media sessions, one 
with sendonly, another with recvonly.

Feedback is greatly appreciated.

Cheers

Magnus Westerlund