Subject: [MMUSIC] imageattr and codec parameter sets (with embedded image size info)

From: Kyunghun Jung <[email-masked]>
Date: Thu, 31 Jul 2008 05:37:31 +0000 (GMT)

Dear Mr.
Johannson, Mr. Even Roni, Mr. Randell Jesup and other experts of MMUSIC/AVT:

The draft of imageattr currently does not introduce the codec parameter set
issues, which also have image size information.

For example, sprop-parameter-sets contains pic_width_in_mbs_minus1
and pic_height_in_map_units_minus_1.

Let me summarize.

- profile and level specify the maximum image size

- imageattr specify the image sizes available from
sender

- config/sprop-parameters-sets specify the image
size to be encoded (but not considered in session setup)

Therefore I asked your opinions on the forming of SDP offer and suggested
three (crudely drafted) examples of

imageattr coupled with config/sprop-parameter-sets. In fact, one form showed
a possibility that covers multiple levels.

If imageattr is limited to a level, then the following form might be necessary
to match the sizes in imageattr with three

sets of config with different image size. My question
was whether this form could be acceptable?

a=imageattr:99 [x=x1,y=y1] [x=x2,y=y2] [x=x3,y=y3]

a=fmtp:99 profile-level-id=8

a=fmtp:99 config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_1

a=fmtp:99 config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_2

a=fmtp:99 config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_3

Or do we need to send config after the image size is decided?
This will increase the session setup delay.

We are fine with either one imageattr for one profile/level or one imageattr
for multiple profile/levels.

But each possibility of handling parameter sets will influence the format
of imageattr. Therefore we asked

experts in IETF to clarify this issue.

Let me answer a question from Mr. Randell Jesup.

Um, a significant problem here.  There seems to be a requirement that the
negotiation is one-directional.  In this
case, the offerer provides NO indication about what he prefers to receive, just
what he *can* send.  When the answerer
responds "picking" a resolution, what does the answerer send?  The implication is that it's the same as what
they picked to receive, which may be seriously sub-optimal.

-> q value, as in Example 3 of the draft, will offer the priority
among offered sizes.

Sincerely yours,

Kyunghun Jung

------- Original Message -------
Sender : Ingemar Johansson S<[email-masked]>
Date : 2008-07-30 18:23 (GMT+09:00)
Title : RE: RE: [MMUSIC] [AVT] spatial-resolution parameter for RFC 3984bis

Hi

If I
understand your email correctly we will, with the constructed SDP's below, have
the potential issue that imageattr may offer combinations (x and y resoluions)
that are not allowed for a given profile level ID. The solution I see is that
the profile levels provides with an upper (maybe also a lower?)
limit.

For instance
if we have some (really wierd) imageattr like

a=imageattr:97 [x=[128:16:1920], y=[96:16:1280],par=[1.1
1.3]]

and the
profile levels only supports a resolution up to 352x288 , the latter limitation
will apply which means that the imageattr should infact be interpreted
as

a=imageattr:97 [x=[128:16:352], y=[96:16:288],par=[1.1
1.3]]

This is not
written anywhere in the draft and I don't right now know if this will hold
in the general case, please comment.

/Ingemar

From: Kyunghun Jung
[mailto:[email-masked]]
Sent: den 30 juli 2008
00:54
To: Even, Roni
Cc: Randell Jesup; Ingemar Johansson S;
[email-masked]; [email-masked]; [email-masked]
Subject: Re: RE:
[MMUSIC] [AVT] spatial-resolution parameter for RFC
3984bis

Thank you Mr. Even Roni for the information.

We would like to ask experts in MMUSIC and AVT to pay a special
interest to the relationship

between imageattr and config/sprop-parameter-set. How to stack the set of
information in the SDP offer

will influence the session setup.

For example, suppose imageattr specifies image sizes over multiple levels.
The following form might be necessary.

m=video 49154 RTP/AVPF 99

b=AS:48

b=RS:0

b=RR:2500

a=rtpmap:99 MP4V-ES/90000

a=imageattr:99 [x=x1,y=y1] [x=x2,y=y2] [x=x3,y=y3]

a=fmtp:99 profile-level-id=a; \
config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_1

a=fmtp:99 profile-level-id=b; \

config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_2

a=fmtp:99 profile-level-id=c; \

config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_3

In case imageattr specifies image sizes within a single level.

a=imageattr:99 [x=x1,y=y1] [x=x2,y=y2] [x=x3,y=y3]

a=fmtp:99 profile-level-id=8

a=fmtp:99 config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_1

a=fmtp:99 config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_2

a=fmtp:99 config=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_3

The above forms are only basic possibilities
and more systematic design and suggestion are

requested from MMUSIC and AVT experts.

Have a nice meeting at Dublin!

Kyunghun Jung

_______________________________________________
mmusic mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mmusic

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Thu, 31 Jul 2008 11:56:30 +0200

Hi
 
My assumption here is that: 
while the profile-level-id is the limiting factor for imageattr 
the pic_width_in_mbs_minus_1 and pic_height_in_map_units_minus_1 in
sprop-parameter-sets should not be used if imageattr is used.
This is expressed in the first bullet in the section 5 in the draft.
 
That said, I am not 100% sure that I adressed your issue below.
 
As regards to the "q value" coment below, unfortunately it does not
address Randell's issue below.
 
Regards
/Ingemar

________________________________

	From: Kyunghun Jung [mailto:[email-masked]] 
	Sent: den 31 juli 2008 06:38
	To: Ingemar Johansson S
	Cc: Even, Roni; Randell Jesup; [email-masked]; [email-masked]
	Subject: imageattr and codec parameter sets (with embedded image
size info)

	Dear Mr. Johannson, Mr. Even Roni, Mr. Randell Jesup and other
experts of MMUSIC/AVT: 

	The draft of imageattr currently does not introduce the codec
parameter set issues, which also have image size information.

	For example, sprop-parameter-sets contains
pic_width_in_mbs_minus1 and pic_height_in_map_units_minus_1.  

	Let me summarize.

	    - imageattr specify the image sizes available from sender

	    - config/sprop-parameters-sets specify the image size to be
encoded (but not considered in session setup)

	Therefore I asked your opinions on the forming of SDP offer and
suggested three (crudely drafted) examples of

	imageattr coupled with config/sprop-parameter-sets. In fact, one
form showed a possibility that covers multiple levels.

	If imageattr is limited to a level, then the following form
might be necessary to match the sizes in imageattr with three

	sets of config with different image size. My question was
whether this form could be acceptable?

		a=fmtp:99

		a=fmtp:99

		a=fmtp:99

	We are fine with either one imageattr for one profile/level or
one imageattr for multiple profile/levels.

	But each possibility of handling parameter sets will influence
the format of imageattr. Therefore we asked

	Um, a significant problem here.  There seems to be a requirement
that the negotiation is one-directional.  In this case, the offerer
provides NO indication about what he prefers to receive, just what he
*can* send.  When the answerer responds "picking" a resolution, what
does the answerer send?  The implication is that it's the same as what
they picked to receive, which may be seriously sub-optimal.

	-> q value, as in Example 3 of the draft, will offer the
priority among offered sizes.

	Sincerely yours,

	Kyunghun Jung

	------- Original Message -------
	Sender : Ingemar Johansson S<[email-masked]>
	Date : 2008-07-30 18:23 (GMT+09:00)
	Title : RE: RE: [MMUSIC] [AVT] spatial-resolution parameter for
RFC 3984bis

	Hi 

	If I understand your email correctly we will, with the
constructed SDP's below, have the potential issue that imageattr may
offer combinations (x and y resoluions) that are not allowed for a given
profile level ID. The solution I see is that the profile levels provides
with an upper (maybe also a lower?) limit. 

	For instance if we have some (really wierd) imageattr like 

	a=imageattr:97 [x=[128:16:1920], y=[96:16:1280],par=[1.1 1.3]] 

	and the profile levels only supports a resolution up to 352x288
, the latter limitation will apply which means that the imageattr should
infact be interpreted as 
	a=imageattr:97 [x=[128:16:352], y=[96:16:288],par=[1.1 1.3]]

	This is not written anywhere in the draft and I don't right now
know if this will hold in the general case, please comment. 

	/Ingemar 

________________________________

	From: Kyunghun Jung [mailto:[email-masked]] 
	Sent: den 30 juli 2008 00:54
	To: Even, Roni
	Cc: Randell Jesup; Ingemar Johansson S; [email-masked];
[email-masked]; [email-masked]
	Subject: Re: RE: [MMUSIC] [AVT] spatial-resolution parameter for
RFC 3984bis

		 We would like to ask experts in MMUSIC and AVT to pay a
special interest to the relationship

		between imageattr and config/sprop-parameter-set. How to
stack the set of information in the SDP offer

		For example, suppose imageattr specifies image sizes
over multiple levels. The following form might be necessary.

		b=AS:48

		b=RS:0

		b=RR:2500

		In case imageattr specifies image sizes within a single
level.

		a=fmtp:99

		a=fmtp:99

		a=fmtp:99

		The above forms are only basic possibilities and more
systematic design and suggestion are 

		Kyunghun Jung

---

From: Randell Jesup <[email-masked]>
Date: Thu, 31 Jul 2008 09:34:35 -0400

Kyunghun Jung <[email-masked]> writes:
>The draft of imageattr currently does not introduce the codec parameter
>set issues, which also have image size information.
>
>For example, sprop-parameter-sets contains pic_width_in_mbs_minus1 and
>pic_height_in_map_units_minus_1.
>
>Let me summarize.
>
> - config/sprop-parameters-sets specify the image size to be encoded
>(but not considered in session setup)

Note: if it's a specific, small list of resolutions, the imageattr values
an offerer is willing to send can all be included in the list of parameter
sets in the fmtp.

If the range of selected sizes is large or unspecified ("*") in the offer,
either a second offer/answer will be required, or in-band parameter set
transmission, or both.  See discussions of RFC 3984bis on this list
recently.

>imageattr coupled with config/sprop-parameter-sets. In fact, one form
>showed a possibility that covers multiple levels.
>
>If imageattr is limited to a level, then the following form might be
>necessary to match the sizes in imageattr with three
>
>sets of config with different image size. My question was whether this
>form could be acceptable?
>
>
>

If payload 99 is H264, then this is invalid - only a single fmtp line
is allowed, and all the parameter sets are specified as a single
sprop-parameter-sets entry in the fmtp line.

>Or do we need to send config after the image size is decided? This will
>increase the session setup delay.

If the list of resolutions is large, then a second offer/answer or in-band
transmission (which is subject to loss) will be needed.  One option is to
start with in-band transmission (with some duplicates, since parameter sets
are normally short and can be merged with other NALs using STAP packets),
while at the same time starting a second offer/answer to provide guaranteed
reception.  There is no "I need the parameter set" codec-control message in
rtcp-fb, just FIR (requesting IDR, and commonly used to recover from packet
loss) and NACK, etc, so there's no easy way for the receiver to ask for the
parameter sets to be resent if they're lost.

>>negotiation is one-directional.  In this case, the offerer provides NO
>>indication about what he prefers to receive, just what he *can* send. 
>>When the answerer responds "picking" a resolution, what does the answerer
>>send?  The implication is that it's the same as what they picked to
>>receive, which may be seriously sub-optimal.
>
>-> q value, as in Example 3 of the draft, will offer the priority among
>offered sizes.

You didn't understand my comment.  My issue is with the requirement in the
current draft to use the same resolution in both directions for a given m=
line (RTP session).

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: Randell Jesup <[email-masked]>
Date: Thu, 31 Jul 2008 09:38:18 -0400

>This is expressed in the first bullet in the section 5 in the draft.

That's not a practical proposal - the parameter sets are fed to the
decoders directly, and affect how much buffer space is reserved in the
codec for each frame, for example.  Also, you can't know if the answer will
understand imageattr.

---

From: "Ingemar Johansson S" <[email-masked]>
Date: Thu, 31 Jul 2008 15:56:47 +0200

Hi

With the disclaimer that I am not an implementer: 
Even though sprop-paramteter-sets (or MPEG4 config) string on the fmtp
looks horrible, it is nontheless possible to parse by a SPD parser that
has the ambition to understand imageattr. But if this capability does
not exist in the parser I would guess that imageattr can only be
ignored. 
I hope that the next update of the draft will cover these issues better.

/Ingemar

> -----Original Message-----
> From: Randell Jesup [mailto:[email-masked]] 
> Sent: den 31 juli 2008 14:38
> To: Ingemar Johansson S
> Cc: [email-masked]; Even, Roni; [email-masked]; 
> [email-masked]
> Subject: Re: imageattr and codec parameter sets (with 
> embedded image size info)
> 
> >while the profile-level-id is the limiting factor for imageattr the 
> >pic_width_in_mbs_minus_1 and pic_height_in_map_units_minus_1 in 
> That's not a practical proposal - the parameter sets are fed 
> to the decoders directly, and affect how much buffer space is 
> reserved in the codec for each frame, for example.  Also, you 
> can't know if the answer will understand imageattr.
> 
> --
> Randell Jesup, Worldgate (developers of the Ojo videophone), 
> ex-Amiga OS team [email-masked] "The fetters imposed on 
> liberty at home have ever been forged out of the weapons