Subject: Re: [AVT] spatial-resolution parameter for RFC 3984bis

From: "Ingemar Johansson S" <[email-masked]>
Date: Mon, 28 Jul 2008 12:03:40 +0200

Hi

Given the discussion below I really wonder if it is not a better alternative to specifiy preferred image sizes using a dedicated (generic) SDP attribute. 
Even though this is solved in some way in 3984bis we still have to provide with a solution for MPEG-4 or H.263 later on.
I drafted a proposed solution for an "a=imageattr" attribute some time ago that may be useful
http://tools.ietf.org/id/draft-johansson-mmusic-image-attributes-01.txt
The draft provides with a solution that may work but lacks a requirement spec to outline exactly what the attribute aims for. Also extra work is needed to avoid conflicts with related payload format parameters.

Regards
Ingemar

=============================
To: <Ye-Kui.Wang at nokia.com> 
Subject: Re: [AVT] spatial-resolution parameter for RFC 3984bis 
From: Randell Jesup <rjesup at wgate.com> 
Date: Mon, 14 Jul 2008 09:35:18 -0400 
Cc: roni.even at polycom.co.il, avt at ietf.org 

<Ye-Kui.Wang at nokia.com> writes:
>The resolution must be restricted by the level part of profile-level-id
>and max-fs. Therefore, if a configuration is agreed (or agreeable), then
>the answerer must request a resolution under the constraint, and the
>offerer must accept the request as long as it meets the restriction.
>This is similar as sprop-parameter-sets, as long as it is complaint with
>the agreed (or agreeable) configuration, the other side must accept it. 

I would suggest instead (and did in the thread back in March): that as a
receiver property, it is the *preferred* maximum frame size.  The receiver
is required to decode and handle any size up to the size implied by the
level (and max-fs if given), but the receiver prefers this size (for
example, it may support 1920x1080, but only have an LCD display that can
display 1200x720, so there's no *need* to send it anything higher (and the
encoder may prefer to avoid encoding more macroblocks, especially in an MCU
that shares resources)).  In other cases, the sender may have the stream
already encoded in different resolutions, and this can help select the best
one to send to save encoding (and bandwidth) resources.

There is no requirement that the encoder scale, but if it wants to it
can.

It also (if I remember) provides aspect-ratio information.

>Would it work this way?
>
>BR, YK 
>
>>-----Original Message-----
>>From: ext Even, Roni [mailto:roni.even at polycom.co.il] 
>>Sent: 14 July, 2008 10:38
>>To: Wang Ye-Kui (Nokia-NRC/Tampere); avt at ietf.org
>>Subject: RE: [AVT] spatial-resolution parameter for RFC 3984bis
>>
>>YK,
>>Just to clarify,
>>The text on the receiver side says that this is the requested
>>resolution, it does not address issue of negotiation, what will the
>>sender do if it cannot send this resolution, should it reject the
>>stream?.
>>If this is for negotiation do you except to do a new offer 
>>answer if the
>>receiver wants a new resolution?
>>
>>Roni
>>
>>> -----Original Message-----
>>> From: avt-bounces at ietf.org [mailto:avt-bounces at ietf.org] On Behalf Of
>>> Ye-Kui.Wang at nokia.com
>>> Sent: Sunday, July 13, 2008 7:57 PM
>>> To: avt at ietf.org
>>> Subject: [AVT] spatial-resolution parameter for RFC 3984bis
>>> 
>>> 
>>> Concluded from the discussions in the email threads listed below, I
>>> added a new parameter, spatial-resolution, to the RFC 3984bis draft
>>> (which will be submitted by tomorrow):
>>> 
>>> http://www.ietf.org/mail-archive/web/avt/current/msg09425.html
>>> http://www.ietf.org/mail-archive/web/avt/current/msg09650.html
>>> 
>>> The parameter is specified as follows (basically as Randell suggested
>>> in
>>> http://www.ietf.org/mail-archive/web/avt/current/msg09429.html).
>>> 
>>> spatial-resolution:
>>> This parameter MAY be used to indicate the maximum spatial resolution
>>> of
>>> a NAL unit stream or the preferred spatial resolution of a receiver.
>>> The value is a base16 [6] (hexadecimal) representation of the width
>>and
>>> height of the spatial resolution, in pixels, separated by a comma.
>>> 
>>> Usage in SDP offer/answer:
>>> - When "a=sendrecv" or no direction attribute is used,
>>> "spatial-resolution" indicates both the maximum spatial resolution of
>>> the stream sent by the declaring entity and the preferred spatial
>>> resolution for receiving a stream.
>>> - When "a=sendonly", "spatial-resolution" indicates the maximum
>>spatial
>>> resolution of the outgoing stream.
>>> - When "a=recvonly", "spatial-resolution" indicates the preferred
>>> spatial resolution for receiving a stream.
>>> 
>>> Comments are welcome.
>>> 
>>> BR, YK
>>> _______________________________________________
>>> Audio/Video Transport Working Group
>>> avt at ietf.org
>>> https://www.ietf.org/mailman/listinfo/avt
>>
>_______________________________________________
>Audio/Video Transport Working Group
>avt at ietf.org
>https://www.ietf.org/mailman/listinfo/avt

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
rjesup at wgate.com
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

******************************************* 
Ingemar Johansson 
Senior Research Engineer, IETF "nethead" 
EAB/TVP - Multimedia Technologies 
Ericsson Research Ericsson AB 
Box 920 S-971 28 Luleå, Sweden 
Tel: +46 (0)8 4043042 
ECN: 850-43042 
ECC: 850-43074 
Mobile: +46 (0)730 783289 
******************************************* 
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: <[email-masked]>
Date: Mon, 28 Jul 2008 13:11:22 +0300

Interesting! I just wrote the following email, planned to send to the MMUSIC mailing list. Now this comes only to AVT. Feel free to forward it to MMUSIC (don't know whether it is allowed to cross posting) or let me to send that, if needed. 

After a reading of this draft (draft-johansson-mmusic-image-attributes), I think the sprop-spatial-resolution parameter in the RFC3984bis can go. It is completely covered by imageattr. 

A couple of comments to draft-johansson-mmusic-image-attributes. 

- It would be helpful to introduce the methods in payload format specs for H.261, H.263 and H.264. In particular, for the one I am familiar with (i.e. H.264), I am not sure how exactly image size together with frame rate is negotiated. 

- Re the following paragraph, I am not sure which parameters are "similar features" and MUST NOT be specified in imageattr. Basically, H.264 sequence parameter set (SPS)may contain all information relating to what may be included in imageattr according to the curent draft. For example, the VUI part of SPS may even contain frame rate related synatx elements. 
   o  Conflict with sprop-parameter-set: H.264 defines the sprop-
      parameter-set which may define features that are defined in this
      draft.  To avoid conflicts, sessions that use the imageattr
      attribute MUST NOT specify similar features in sprop-parameter-
      set.

BR, YK

>From: ext Ingemar Johansson S 
>[mailto:[email-masked]] 
>Sent: Monday, July 28, 2008 1:04 PM
>To: [email-masked]
>Cc: Wang Ye-Kui (Nokia-NRC/Tampere); Even, Roni; Randell Jesup
>Given the discussion below I really wonder if it is not a 
>better alternative to specifiy preferred image sizes using a 
>dedicated (generic) SDP attribute. 
>Even though this is solved in some way in 3984bis we still 
>have to provide with a solution for MPEG-4 or H.263 later on.
>The draft provides with a solution that may work but lacks a 
>requirement spec to outline exactly what the attribute aims 
>for. Also extra work is needed to avoid conflicts with related 
>payload format parameters.
>
>Regards
>Ingemar
>  
>
>>and max-fs. Therefore, if a configuration is agreed (or agreeable), 
>>then the answerer must request a resolution under the constraint, and 
>>the offerer must accept the request as long as it meets the 
>restriction.
>>This is similar as sprop-parameter-sets, as long as it is complaint 
>>with the agreed (or agreeable) configuration, the other side 
>must accept it.
>
>I would suggest instead (and did in the thread back in March): 
>that as a receiver property, it is the *preferred* maximum 
>frame size.  The receiver is required to decode and handle any 
>size up to the size implied by the level (and max-fs if 
>given), but the receiver prefers this size (for example, it 
>may support 1920x1080, but only have an LCD display that can 
>display 1200x720, so there's no *need* to send it anything 
>higher (and the encoder may prefer to avoid encoding more 
>macroblocks, especially in an MCU that shares resources)).  In 
>other cases, the sender may have the stream already encoded in 
>different resolutions, and this can help select the best one 
>to send to save encoding (and bandwidth) resources.
>
>There is no requirement that the encoder scale, but if it 
>wants to it can.
>
>>>If this is for negotiation do you except to do a new offer answer if 
>>>the receiver wants a new resolution?
>>>
>>>Roni
>>>
>>>> From: avt-bounces at ietf.org [mailto:avt-bounces at ietf.org] On 
>>>> Behalf Of Ye-Kui.Wang at nokia.com
>>>> The parameter is specified as follows (basically as Randell 
>>>> suggested in 
>>>> This parameter MAY be used to indicate the maximum spatial 
>>>> resolution of a NAL unit stream or the preferred spatial 
>resolution 
>>>> of a receiver.
>>>> "spatial-resolution" indicates both the maximum spatial resolution 
>>>> of the stream sent by the declaring entity and the 
>preferred spatial 
>Randell Jesup, Worldgate (developers of the Ojo videophone), 
>ex-Amiga OS team rjesup at wgate.com "The fetters imposed on 
>liberty at home have ever been forged out of the weapons

---

From: "Even, Roni" <[email-masked]>
Date: Mon, 28 Jul 2008 13:31:31 +0300

Hi,
I think that we have to be clear about the usage of the parameter

1. Announce sending of the specific resolution - just resolution 
2. Request a specific resolution - Receiver wants a specific resolution
3. Negotiate available resolutions - The offer says, I can do CIF, QCIF, 1080P,... here it may dependent in H.263 and H.261 on the frame rate while in H.264 the limit is in the macroblocks per second which will specify the maximum frame rate. The reason for this offer is because it offers the operation points that the offerer can have. The level parameter in H.263 and H.264 gives a maximum but does not imply support for any resolution that the answer would like to receive that falls within the range of the level.

I think the img attribute address 1 and 2, I am not sure about 3

Roni Even

> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, July 28, 2008 1:11 PM
> To: [email-masked]; [email-masked]
> Cc: Even, Roni; [email-masked]
> Interesting! I just wrote the following email, planned to send to the
> MMUSIC mailing list. Now this comes only to AVT. Feel free to forward
> it to MMUSIC (don't know whether it is allowed to cross posting) or let
> me to send that, if needed.
> 
> 
> After a reading of this draft (draft-johansson-mmusic-image-
> attributes), I think the sprop-spatial-resolution parameter in the
> RFC3984bis can go. It is completely covered by imageattr.
> 
> - It would be helpful to introduce the methods in payload format specs
> for H.261, H.263 and H.264. In particular, for the one I am familiar
> with (i.e. H.264), I am not sure how exactly image size together with
> frame rate is negotiated.
> 
> - Re the following paragraph, I am not sure which parameters are
> "similar features" and MUST NOT be specified in imageattr. Basically,
> H.264 sequence parameter set (SPS)may contain all information relating
> to what may be included in imageattr according to the curent draft. For
> example, the VUI part of SPS may even contain frame rate related synatx
> elements.

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Mon, 28 Jul 2008 15:01:04 +0200

Hi

My response below, also a crosspost to mmusic, hope that this not too upsetting. 

To answer Roni's and Ye-Kui's comments I believe that the image attribute can be used for 3. However, looking in perspective I am not longer sure that the possibility to specify a framerate range makes sense as it does not connect  too wellto reality when it comes to e.g H.264 which talks about macroblocks per second. The conclusion is then that the framerate (or MBs) is determined from the fmtp line.
This means that if the imageattr specifies a large offered range the frame rate is limited by what is possible for the given preferred resolution. 

As an example, consider H.263 with the fmtp line
 CIF=4;QCIF=3;SQCIF=2;
the imageattr may then be specified as
a=imageattr:97 [x=[128:8:352],y=[96:8:288],par=[1.2,1.3]]
If the receiver prefers the resolution 344x288 (slightly less than CIF) the maximum frame rate would be 30/(1.0001*4) ~ 7.5Hz. 
If the prefered resolution is 160x128 (slightly less than QCIF) then the max frame rate is 30/(1.0001*3) Hz and so on.
 
For the H.264 case the frame rate depending of the preferred image size would be limited according to the maximum framerate repending on the max MBs and the resolution given in table A-6 in the H.264 spec. For instance for level 2 an image size of 344x280 which is less than CIF but larger than QVGA would yield a max frame rate of 30Hz.

Would this distincion between framerate (or MBs) and preferred image size work?

/Ingemar   

> From: Even, Roni [mailto:[email-masked]] 
> Sent: den 28 juli 2008 11:32
> To: [email-masked]; Ingemar Johansson S; [email-masked]
> Cc: [email-masked]
> 1. Announce sending of the specific resolution - just 
> resolution 2. Request a specific resolution - Receiver wants 
> a specific resolution 3. Negotiate available resolutions - 
> The offer says, I can do CIF, QCIF, 1080P,... here it may 
> dependent in H.263 and H.261 on the frame rate while in H.264 
> the limit is in the macroblocks per second which will specify 
> the maximum frame rate. The reason for this offer is because 
> it offers the operation points that the offerer can have. The 
> level parameter in H.263 and H.264 gives a maximum but does 
> not imply support for any resolution that the answer would 
> like to receive that falls within the range of the level.
> 
> > it to MMUSIC (don't know whether it is allowed to cross posting) or 
> > let me to send that, if needed.
> > 
> > 
> > to what may be included in imageattr according to the curent draft. 
> > For example, the VUI part of SPS may even contain frame 
> rate related 
> > synatx elements.
> > >Given the discussion below I really wonder if it is not a better 
> > >alternative to specifiy preferred image sizes using a dedicated 
> > >(generic) SDP attribute.
> > >Even though this is solved in some way in 3984bis we still have to 
> > >provide with a solution for MPEG-4 or H.263 later on.
> > >requirement spec to outline exactly what the attribute 
> aims for. Also 
> > >extra work is needed to avoid conflicts with related 
> payload format 
> > >parameters.
> > >
> > >Regards
> > >Ingemar
> > >
> > >
> > >>then the answerer must request a resolution under the constraint, 
> > >>and the offerer must accept the request as long as it meets the
> > >restriction.
> > >that as a receiver property, it is the *preferred* maximum frame 
> > >size.  The receiver is required to decode and handle any 
> size up to 
> > >the size implied by the level (and max-fs if given), but 
> the receiver 
> > >prefers this size (for example, it may support 1920x1080, but only 
> > >have an LCD display that can display 1200x720, so there's 
> no *need* 
> > >to send it anything higher (and the encoder may prefer to avoid 
> > >encoding more macroblocks, especially in an MCU that shares 
> > >resources)).  In other cases, the sender may have the 
> stream already 
> > >encoded in different resolutions, and this can help select 
> the best 
> wants to it 
> > >>>If this is for negotiation do you except to do a new 
> offer answer 
> > >>>if the receiver wants a new resolution?
> > >>>
> > >>>Roni
> > >>>
> > >>>> "spatial-resolution" indicates both the maximum spatial 
> > >>>> resolution of the stream sent by the declaring entity and the
> > >preferred spatial
> > >>>> Audio/Video Transport Working Group avt at ietf.org 
> > >Randell Jesup, Worldgate (developers of the Ojo 
> videophone), ex-Amiga 
> > >OS team rjesup at wgate.com "The fetters imposed on 
> liberty at home 
> > >have ever been forged out of the weapons provided for 
> defence against 
> > >real, pretended, or imaginary dangers from abroad."

---

From: Randell Jesup <[email-masked]>
Date: Mon, 28 Jul 2008 11:49:38 -0400

"Ingemar Johansson S" <[email-masked]> writes:
>To answer Roni's and Ye-Kui's comments I believe that the image attribute
>can be used for 3. However, looking in perspective I am not longer sure
>that the possibility to specify a framerate range makes sense as it does
>not connect  too wellto reality when it comes to e.g H.264 which talks
>about macroblocks per second. The conclusion is then that the framerate
>(or MBs) is determined from the fmtp line.
>This means that if the imageattr specifies a large offered range the frame
>rate is limited by what is possible for the given preferred resolution. 
>

So far as I know (and I didn't recheck), H.263 only allows for resolutions
specified in the fmtp (though QCIF and CIF are I think required).  If you
want 344x288, you need to add a CUSTOM resolution to the fmtp line.  And
the mpi must be an integer in the fmtp.  Now, if both sides use this, then
those restrictions could be relaxed.

>For the H.264 case the frame rate depending of the preferred image size
>would be limited according to the maximum framerate repending on the max
>MBs and the resolution given in table A-6 in the H.264 spec. For instance
>for level 2 an image size of 344x280 which is less than CIF but larger
>than QVGA would yield a max frame rate of 30Hz.

My suggestion would be for this to express preferences within the normal
negotiated resolution and framerate space.  i.e for 264 the level and
max-fs and max-mbps would determine the *maximum* frame and framerate
supported, but this would be used to decide what to actually send.

See my original email below -- look for:
    "I would suggest instead (and did in the thread back in March)"

>> > >>The resolution must be restricted by the level part of
>> > >>profile-level-id and max-fs. Therefore, if a configuration is agreed
>> > >>(or agreeable), then the answerer must request a resolution under
>> > >>the constraint, and the offerer must accept the request as long as
>> > >>it meets the restriction.  This is similar as sprop-parameter-sets,
>> > >>as long as it is complaint with the agreed (or agreeable)
>> > >>configuration, the other side must accept it.
>> > >
>> > >I would suggest instead (and did in the thread back in March): that
>> > >as a receiver property, it is the *preferred* maximum frame size.
>> > >The receiver is required to decode and handle any size up to the size
>> > >implied by the level (and max-fs if given), but the receiver prefers
>> > >this size (for example, it may support 1920x1080, but only have an
>> > >LCD display that can display 1200x720, so there's no *need* to send
>> > >it anything higher (and the encoder may prefer to avoid encoding more
>> > >macroblocks, especially in an MCU that shares resources)).  In other
>> > >cases, the sender may have the stream already encoded in different
>> > >resolutions, and this can help select the best one to send to save
>> > >encoding (and bandwidth) resources.
>> > >

[email-masked]