Subject: Re: [AVT] SVC open issues - sprop-spatial-resolution

From: Colin Perkins <[email-masked]>
Date: Tue, 3 Jun 2008 13:51:05 +0100

There are also a payload format specific image size parameters  
defined in RFC 4175.

Colin

On 1 Jun 2008, at 09:51, Ingemar Johansson S wrote:
> Hi
> FYI I submitted an "a=imageattr" draft to MMUSIC
> as a response to input document S4-080145
> (http://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_48/Docs/S4-080145.zip)
>
> The link to the draft is
> http://tools.ietf.org/id/draft-johansson-mmusic-image- 
> attributes-00.txt
> ================
> Abstract
>
>    This draft proposes a new generic session setup attribute to  
> make it
>    possible to negotiate different image attributes such as image  
> size.
>    A possible use case is to make it possible for a e.g a low-end  
> hand-
>    held terminal to display video without the need to resample the
>    image, something that may consume large amounts of memory and
>    processing power.
> ================
>
> Regards
> Ingemar
>
>
>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On
>> Behalf Of [email-masked]
>> Sent: den 30 maj 2008 21:00
>> To: [email-masked]
>> Subject: avt Digest, Vol 49, Issue 20
>>
>> Send avt mailing list submissions to
>> 	[email-masked]
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	https://www.ietf.org/mailman/listinfo/avt
>> or, via email, send a message with subject or body 'help' to
>> 	[email-masked]
>>
>> You can reach the person managing the list at
>> 	[email-masked]
>>
>> When replying, please edit your Subject line so it is more
>> specific than "Re: Contents of avt digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: SVC open issues - sprop-spatial-resolution
>>       ([email-masked])
>>
>>
>> --------------------------------------------------------------------- 
>> -
>>
>> Message: 1
>> Date: Fri, 30 May 2008 11:12:42 +0300
>> From: <[email-masked]>
>> Subject: Re: [AVT] SVC open issues - sprop-spatial-resolution
>> To: <[email-masked]>, <[email-masked]>
>> Cc: [email-masked], [email-masked]
>> Message-ID:
>> 	<[email-masked]>
>> Content-Type: text/plain;	charset="us-ascii"
>>
>>
>> I have studied earlier email threads (including Roni's suggestion in
>> 2002) Randell mentioned below, as well as the following related
>> documents:
>> - The a=framesize defined in 3GPP PSS
>> (http://www.3gpp.org/ftp/Specs/archive/26_series/ 
>> 26.234/26234-750.zip)
>> - The newly consented ITU-T H.241video submode negotiation
>> (in ITU-T document T05-SG16-080422-TD-PLEN-0490!!ZIP-E.zip)
>> - The 3GPP Feature or Study Item: MTSI Video - Dynamic Rate
>> Adaptation/Signalling of Image Size
>> (http://www.3gpp.org/ftp/Specs/html-info/GanttChart-Level-2.ht
> m#bm34041)
>> , and the input document S4-080145
>> (http://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_48/Docs/ 
>> S4-080145.zip)
>>
>> Summary:
>> - The purpose of having the sprop-spatial-resolution
>> parameter for SVC use is different from any other
>> earlier-mentioned purposes. Herein, the parameter is only
>> used to indicate the property of an SVC bitstream or a subset
>> thereof, and may be used in negotiation of a subset of an SVC
>> bitstream to be transmited from the sender to the receiver.
>> - The sprop-spatial-resolution parameter is closest, in
>> semantics, to the PSS a=framesize, which is defined as
>> "a=framesize:<payload type
>> number> <width>-<height>". Having <payload type number> is not needed
>> for the SVC use, but it may not harm.
>>
>> Example:
>> -  In unicast streaming, the client gets an SDP (e.g. by
>> sending RTSP DESCRIBE method to the server and sever sends
>> back the SDP) for an SVC bitstream containing multiple
>> operation points, each being assoicated with a spatial
>> resolution, frame rate, and bit rate, among others. The
>> client then choose one of the operation points and tells the
>> server its choice. Then the server sends the bitstream subset
>> for the chosen operation point.
>>
>> Conclusion:
>> - An SDP parameter having similar semantics as
>> sprop-spatial-resolution or the PSS a=framesize is needed.
>>
>> The way to go:
>> - Either to have a generic SDP parameter defined somewhere
>> else or to have a SVC specific defintion as it is. If there
>> is no way to do the former quickly, then let's leave the
>> definition in the SVC draft.
>>
>> BR, YK
>>
>>
>>> -----Original Message-----
>>> From: ext Randell Jesup [mailto:[email-masked]]
>>> Sent: 14 April, 2008 17:33
>>> To: Wang Ye-Kui (Nokia-NRC/Tampere)
>>> Cc: [email-masked]; [email-masked]
>>> Subject: Re: [AVT] SVC open issues - sprop-spatial-resolution
>>>
>>> Everyone interested in this discussion should review the
>> last time we
>>> discussed this in 2004 (note - the thread index links seem
>> to take you
>>> to 2007; prev and next work):
>>>
>>> http://www.ietf.org/mail-archive/web/mmusic/current/msg03137.html
>>> and
>>> http://www.ietf.org/mail-archive/web/avt/current/msg04871.html
>>>
>>> Note: the discussion goes on for a while.
>>>
>>> Also, Roni's suggestion from 2002:
>>> http://www.ietf.org/mail-archive/web/mmusic/current/msg00649.html
>>>
>>> Also, from RFC 4566:
>>>      a=framerate:<frame rate>
>>>
>>>         This gives the maximum video frame rate in
>> frames/sec.  It is
>>>         intended as a recommendation for the encoding of video data.
>>>         Decimal representations of fractional values using the
>>> notation
>>>         "<integer>.<fraction>" are allowed.  It is a media-level
>>>         attribute, defined only for video media, and it is not
>>>         dependent on charset.
>>>
>>> Note that this "is intended as a recommendation for the encoding".
>>> I.e. that does not say "you cannot encode at a higher rate"; it says
>>> "I'd like you to encode at this rate (or lower)".  Also, this is
>>> media-level, as it says, so it doesn't help with
>> codec-specific issues;
>>> it maps better to output-device ability (or ability of
>> whatever codec
>>> offered is fastest).
>>>
>>> "Even, Roni" <[email-masked]> writes:
>>>> I am not sure what the requirements for this parameter are.
>>>>
>>>> There is no generic resolution parameter defined. There are
>>> specific ones
>>>> in H.263 and H.261 and they list the supported resolution in
>>> relations to
>>>> the frame rate.
>>>
>>> A generic solution would be better, or a specific solution
>> that could
>>> be generalized later.  See previous discussions.  I should note some
>>> devices use weird resolutions, like 352x240 (kinda 1/2 CIF,
>> 1/2 QVGA)
>>> or unusual layouts like 144x176 (portrait mode).  :-)
>>>
>>>> In RFC 3984 the receiver capabilities max-mbps and max-fs
>> allows the
>>>> receiver to signal that it supports higher macro blocks per
>> second or
>>>> frame size in macro blocks than the signaled level
>> according to table
>>>> A-1.
>>>
>>> Those can define decoder ability, but not display ability.
>> Note that a
>>> level 3 decoder could decode QCIF (99 macroblocks) at very
>> high frame
>>> rates, but typically could not display it any faster than
>> the refresh
>>> rate of the physical display device (30/50/60Hz, typically),
>> so higher
>>> frame rates would be wasted.
>>>
>>> Maximum display ability is better specified by a=framerate,
>> per above,
>>> on a media level.
>>>
>>> I should note that a device which can support max-fs of X
>> and max-mbps
>>> of Y cannot necessarily support them at the same time.  At
>> max-fs, the
>>> decoder might need to use an expensive output scaler, or to decode a
>>> maximum framesize it might have to kick out some data from cache,
>>> hurting the max-mbps.  (For example, at SQCIF the current frame and
>>> reference frame(s) might all fit in L2 or L3 cache, enabling a very
>>> high max-mbps, while at VGA they may not fit, slowing it down a  
>>> lot.)
>>>
>>> How much that matters in practice is another question - I
>> suspect most
>>> don't run that close to the limit.
>>>
>>> <[email-masked]> writes:
>>>> Let me try to explain with an example. Assume that a
>> streaming server
>>>> offers a scalable bitstream containing three operation points with
>>>> different resolutions, e.g. QCIF, CIF and 4CIF. In the SDP,
>> it should
>>>> include multiple m lines, each operation point should be
>>> associated with
>>>> one m line. Layer dependency is inidcated according to the MMUSIC
>>>> decoding dependency draft. Beside layer dependency
>>> information, for each
>>>> of the operation point, other important information such as
>> bitrate,
>>>> frame rate and spatial resolution should also be included. My poor
>>>> understanding to SDP is that it provides means to indicated
>> bitrate,
>>>> frame rate but not spatial resolution - and I don't why SDP
>> does not
>>>> include this parameter.
>>>
>>> It's been discussed (per above), but never handled on a
>> generic level.
>>>
>>>> With RFC 3984 it is possible to indicate the spatial
>> resolution of a
>>>> stream using the sequence parameter set included in one
>> OPTIONAL media
>>>> type parameter sprop-parameter-sets, but there is no other
>> means as I
>>>> know. However, in SVC, this sprop-parameter-sets for the 4CIF
>>> stream in
>>>> the example above may contain sequence parameters associated with
>>>> different spatial resolutions. Furthermore, it is not
>>> convenient to dig
>>>> out the spatial resolution values buried in the
>> variable-length coded
>>>> parameter sets in sprop-parameter-sets.
>>>
>>> Exactly; parsing them out of the parameter sets is a non-starter.
>>>
>>> --
>>> Randell Jesup, Worldgate (developers of the Ojo videophone),
>> ex-Amiga
>>> OS team [email-masked] "The fetters imposed on liberty at
>> home have
>>> ever been forged out of the weapons provided for defence
>> against real,
>>> pretended, or imaginary dangers from abroad."
>>> 		- James Madison, 4th US president (1751-1836)
>>>
>>>
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> Audio/Video Transport Working Group
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/avt
>>
>>
>> End of avt Digest, Vol 49, Issue 20
>> ***********************************
>>
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www.ietf.org/mailman/listinfo/avt

-- 
Colin Perkins
http://csperkins.org/

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: "Even, Roni" <[email-masked]>
Date: Tue, 3 Jun 2008 16:11:46 +0300

Colin,
I sent comments to the MMUSIC mailing list on Ingemar draft. 
A general issue is if we need to have a general SDP parameter or codec
specific like the ones in RFC 4175 or the H.263 and H.261 payload
specifications.
Another issue is if the image size is a standalone parameter or is it
also important to have the Frame rate (or Macro blocks per second in
H.264) relating to the image size (This is a processing issue)
On the geometry size having just an image size without aspect ratio (In
H.264) may cause geometric distortion
Roni

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Colin Perkins
> Sent: Tuesday, June 03, 2008 3:51 PM
> To: Ingemar Johansson S
> Cc: [email-masked]; [email-masked]
--------------------------------------------------------------------

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Tue, 3 Jun 2008 21:18:49 +0200

Hi

Thanks Roni and Colin for the comments. I skimmed through RFC4175 and my
immediate reaction is that the parameters width and height applies to
the encoding. The proposed SDP attribute draft tries to separate the two
as the encoder may only be able to encode certain sizes with a limited
granularity.

Issue 1 below:
It is true that cropping of the image may cause a problem but I would
believe that the example below is a bit extreme. A more likely example
is maybe that the image is cropped 30%, which may indeed cause problems
with missing heads etc. One solution is that the sender can see what the
other party receives for instance in a small window but he consequences
must probably be studied more.

Issue 2 below:
Geometric distorsion is a problem, in an early drop of this draft I had
aspect ratio as an open issue but I removed it for some reason. I guess
aspect ratio is a bit tricky as it is a rational number. Perhaps one
could steal something from H.241 here. 

Besides video I can see other uses such a shared whiteboard
applications. In a office environment there is often problem with the
sender in e.g a sametime session that use a way to large screen. In the
other end the screen is resampled, something that often renders the
image useless. If the sender image size is adjusted to fit the receivers
the image will be much better as e.g the vector graphics and truetype
faces are scaled without distorsion. OK we may have scalability issue
here.

BTW!
What working group does this discussion belong to ?, it seems like it is
a mix of payload format and session setup. I cross-posted to avt this
time (promise I will not do it again).
 
Regards
Ingemar

> From: Even, Roni [mailto:[email-masked]] 
> Sent: den 3 juni 2008 13:52
> To: [email-masked]
> Cc: Ingemar Johansson S
> Subject: comments on draft-johansson-mmusic-image-attributes-00
> 
> Hi,
> I read the draft and have some comments
> 
> 1. On section 5 issue 2 you suggest that the encoder will 
> crop the image, this may lead to having an irrelevant part of 
> the image. For example cropping a QCIF size from a HD camera 
> may give a small top left part of the image with no content 
> that may be in the middle.
> 
> 2. My understanding is that the size negotiation is to enable 
> a decoder with low processing power to decode without any 
> scaling requirements.
> For H.264 and future codecs this may still lead to geometric 
> distortion due to  different picture and sample aspect 
> ratios. Example 4:3 or 16:9.
> To have a full solution you should be able to declare your 
> preferred aspect ratio. Such work was approved last month in 
> ITU-T H.241
> 
> Roni Even
> 
> 
>