Subject: [AVT] SSRC use in single- and multi-session transmission for SVC payload draft

From: Alex Eleftheriadis <[email-masked]>
Date: Thu, 10 Jul 2008 15:17:19 +0300

Greetings to all.

During today's editing session for draft-ietf-avt-rtp-svc-13, we  
(myself, Ye-Kui Wang, Thomas Schierl, and Roni Even) were not sure  
about the consensus reached regarding SSRC use.

The draft currently says (based on discussions in the last SVC design  
team conference call) that in single-session transmission (one RTP  
session carries the entire SVC stream) the different layers are all  
using the same SSRC. In multi-session transmission, it says that the  
same SSRC is used in all RTP sessions (aligning with RFC 3550/layered  
multicast, see, e.g., Section 8.3).

Some of us recalled this being discussed in the last AVT meeting, but  
were not sure about the conclusion reached.

Is there any reason for changing the above rules? Specifically for  
the single-session transmission, using different SSRCs would  
presumably allow selective elimination of layers without looking at  
the RTP packet payload. But I personally am not sure that such a  
system would make sense. A MANE would be SVC-aware, and can use  
information specifically placed in the RTP payload to do this.

Comments appreciated. Thanks.

--Alex

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: [email-masked]
Date: Fri, 11 Jul 2008 10:58:24 +0200

Hi,

In my understanding of SVC usages, the two main cases are:

a unicast transport where all the layers a user can receive and
decode are in a single RTP session. I am not sure if the MANE apply to
this scenario, but the server (or the source) can run an adaptation
function that select and optimize the NALU to encapsulate in the RTP
session. Using the same SSRC for all layers in the session is not a
problem, because a user can distinguish layers by the dynamic PT as
described in the SDP.

a multicast transport where the user select one or more IP
address to adapt the received rate at its access link or its terminal
properties. This case is more sensitive, because some receivers can
decode only the AVC-base layer that must be compliant with rfc 3984.
Using the same SSRC is not a problem, because the receiver can also use
the dynamic PT to reorder the NALUs in AUs

So for me  the same SSRC can run efficiently, and it makes sense
because the layers come from a single source (same encoder).

However, SVC is still a very new concept and may be some different
scenarios could arise, as using several transmission channels. For
instance some layers in a unicast channel on others in a  multicast
channel.

If it is possible to validate that 1) recovering NALUs from the same AU
and 2) reordering NALUs in a AU are the main function the receiver
needs to perform, then the transport ought to cover  most of the
imaginable user cases.

Regards,

Gérard BABONNEAU

Alex Eleftheriadis a écrit :

Greetings to all.

Is there any reason for changing the above rules? Specifically for the
single-session transmission, using different SSRCs would presumably
allow selective elimination of layers without looking at the RTP packet
payload. But I personally am not sure that such a system would make
sense. A MANE would be SVC-aware, and can use information specifically
placed in the RTP payload to do this.

--Alex

[email-masked]

---

From: Alex Eleftheriadis <[email-masked]>
Date: Fri, 11 Jul 2008 13:09:29 +0300

On Jul 11, 2008, at 11:58 AM, [email-masked] wrote:

> decode are in a single RTP session. I am not sure if the MANE apply  
> to this scenario, but the server (or the source) can run an  
> adaptation function that select and optimize the NALU to  
> encapsulate in the RTP session. Using the same SSRC for all layers  
> in the session is not a problem, because a user can distinguish  
> layers by the dynamic PT as described in the SDP.
> a multicast transport where the user select one or more IP address  
> to adapt the received rate at its access link or its terminal  
> decode only the AVC-base layer that must be compliant with rfc  
> 3984. Using the same SSRC is not a problem, because the receiver  
> can also use the dynamic PT to reorder the NALUs in AUs

Exactly, this correctly maintains the intended use of the SSRC.

Regarding the PT, the current draft is (to my understanding)  
implicitly assuming that the SVC stream is on a single PT. Mutliple  
PTs may be offered in an SDP Offer/Answer, but each one of them  
corresponds to a complete subset of SVC layers (i.e., each PT is not  
a distinct layer). I think that having each layer on a separate PT  
would break a lot of assumptions in SDP/RTP, but I may be wrong.

RFC3550 btw explicitly says not to use PT for multiplexing separate  
media streams, but it is true that in this case it is actually one  
stream.

> If it is possible to validate that 1) recovering NALUs from the  
> same AU and 2) reordering NALUs in a AU are the main function the  
> receiver needs to perform, then the transport ought to cover  most  
> of the  imaginable user cases.
>

The draft provides two mechanisms for performing decoding order  
recovery, i.e., doing both 1 and 2. The case of combined unicast and  
multicast transmission is not explicitly mentioned in the draft, but  
there is nothing prohibiting one from doing so as far as I can see,  
assuming it is supported by the signaling infrastructure. Another use  
case is when SVC streams from multiple users are combined in a single  
unicast connection. All these cases are fully supported by the  
current draft.

Thanks.

--Alex

>> The draft currently says (based on discussions in the last SVC  
>> design team conference call) that in single-session transmission  
>> (one RTP session carries the entire SVC stream) the different  
>> layers are all using the same SSRC. In multi-session transmission,  
>> it says that the same SSRC is used in all RTP sessions (aligning  
>> with RFC 3550/layered multicast, see, e.g., Section 8.3).
>>
>> Some of us recalled this being discussed in the last AVT meeting,  
>> but were not sure about the conclusion reached.
>>
>> presumably allow selective elimination of layers without looking  
>> at the RTP packet payload. But I personally am not sure that such  
>> a system would make sense. A MANE would be SVC-aware, and can use

---

From: Thomas Schierl <[email-masked]>
Date: Fri, 11 Jul 2008 12:42:36 +0200

Alex Eleftheriadis wrote:

On Jul 11, 2008, at 11:58 AM, [email-masked]
wrote:

Hi,

a unicast transport where all the layers a user can receive
and decode are in a single RTP session. I am not sure if the MANE apply
to this scenario, but the server (or the source) can run an adaptation
function that select and optimize the NALU to encapsulate in the RTP
session. Using the same SSRC for all layers in the session is not a
problem, because a user can distinguish layers by the dynamic PT as
described in the SDP.

Regarding the PT, the current draft is (to my understanding)
implicitly assuming that the SVC stream is on a single PT. Mutliple PTs
may be offered in an SDP Offer/Answer, but each one of them corresponds
to a complete subset of SVC layers (i.e., each PT is not a distinct
layer). I think that having each layer on a separate PT would break a
lot of assumptions in SDP/RTP, but I may be wrong.

RFC3550 btw explicitly says not to use PT for multiplexing
separate media streams, but it is true that in this case it is actually
one stream.

I think we already decided at Montreal meeting in 2006, that PT
multiplexing is not preferable for differentiating between different
SVC layers within the same RTP session.

When following the resolutions from the Prague meeting, SSRC
multiplexing was removed just for missing use cases. The original use
case was adaption of SRTP protected streams without being in the
security context. Observing this from today, I could still imagine a
use case and that is the transport of multiple layers with it's own
transport parameters. The point is here, that someone may have to open
many ports with session multiplexing to bring all the layers through ,
e.g. a firewall. With SSRC multiplexing this could be reduced to just
one port.

Thomas

--
Thomas Schierl
--------------
Fraunhofer HHI

---

From: <[email-masked]>
Date: Fri, 11 Jul 2008 14:25:32 +0300

Hi All,

I studied this issue (see the previous AVT meeting minutes and the
following email thread starting with
http://www.ietf.org/mail-archive/web/avt/current/msg09399.html),
particularly regarding downgrading of Level and the use of
sprop-parameter-sets in offer/answer, and concluded that the following
solution should be considered for both H.264/AVC video and SVC video.
Comments appreciated!

Summary of the solution:
- Level downgrade allowed
- Both out-of-band and in-band transport of parameter sets allowed
- No extra restriction to sprop-parameter-sets in SDP but it must be
compliant with other parameters in the SDP message (e.g.
profile-level-id). In other words, the purpose for inclusion of
sprop-parameter-sets in SDP serves only for one purpose - out-of-band
transport of parameter sets

Following is an example. 

- The offerer offers Baseline profile, level 3.0 with the
Non-Interleaved packetization mode
- The answerer accepts but would downgrade level to 2.0. In addition,
the answerer sends the parameter sets to use in the bitstream it will
send (therefore the answerer does not need to store
sprop-parameter-sets=<base-64 data#0>)
- The offerer agrees and sends the parameter sets to use in the
bitstream it will send (therefore the sender stores
sprop-parameter-sets=<base-64 data#1> to be used in decoding the
incoming bitstream)
- After receiving SDP#2, the answerer stores
sprop-parameter-sets=<base-64 data#2> to be used in decoding the
incoming bitstream

Offerer -> Answerer SDP#0: 
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0
  packetization-mode=1; 
  sprop-parameter-sets=<base-64 data#0> //Optional

Answerer -> Offerer SDP#1:
m=video 49170 RTP/AVP 97
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42A014; //Baseline profile, Level 2.0
  packetization-mode=1; 
  sprop-parameter-sets=<base-64 data#1>

Offerer -> Answerer SDP#2:
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000
a=fmtp:97 profile-level-id=42A014; //Baseline profile, Level 2.0
  packetization-mode=1;
  sprop-parameter-sets=<base-64 data#2>

BR, YK

---

From: Thomas Schierl <[email-masked]>
Date: Fri, 11 Jul 2008 15:54:43 +0200

Further, we found at the Prague 2007 meeting, that a MANE's job is not 
significantly reduced by using SSRC multiplexing. That time, the MANE 
was assumed to be an RTP mixer, i.e. it terminates a session and sets up 
a new session for the adapted packet streams. This for sure enhances the 
complexity of such a device but removes problems as SSRC collisions. 
Those days and also these days, this arguments seem to hold, more or 
less. See also discussion on the avt mailing list on 
draft-lennox-avt-rtp-layered-encoding-timestamps-00.txt:

On 04 July 2008 Colin Perkins wrote:
> On 12 Jun 2008, at 22:12, Jonathan Lennox wrote:
>> Colin Perkins wrote:
>>> I have no particular objection to this proposal in itself, but I
>>> believe that using the same SSRC across multiple layers is
>>> undesirable, and when different SSRC values are used for each layer,
>>> this proposal offers no benefit.
>>>
>>> While using the same SSRC for each layer is simpler for some case, it
>>> makes multiplexing several layers onto the same session difficult
>>> (the suggestion below to demultiplex based on the RTP payload type
>>> doesn't work). Since managing multiple UDP ports is difficult, I
>>> would expect that multiplexing several layers onto the same port,
>>> with media-aware RTP translators dropping layers at network
>>> boundaries to perform adaptation, to be a common use case, and not
>>> something we should break.
>>
>> The question is if there's any benefit to multiplexing layers into an
>> RTP session by placing them in separate RTP streams, as opposed to doing
>> so in a payload-specific manner within a single stream.  It's certainly
>> the case that for H.264 SVC, SSRC-based layer multiplexing within a
>> session was present for a few revisions of the payload draft, but was
>> dropped for lack of a convincing use case.
>>
>> The compelling reason for splitting a media stream across multiple RTP
>> streams is when you actually want different transport-layer behavior for
>> the streams, for example when the layers are distributed over different
>> multicast groups.  In this case the streams obviously have to go into
>> different RTP sessions, and RFC 3550 SSRC-alignment applies.
>>
>> The case of adaptation at network boundaries is certainly a common use
>> case, but a media-aware network element can operate (in RTP terminology)
>> as a mixer, rather than as a translator, and drop layers from within an
>> RTP stream (fixing up sequence numbers to compensate).  Since it's
>> media-aware, it necessarily understands the payload format.  This avoids
>> problems of stream association and SSRC collision, especially if you're
>> also sending multiple distinct media sources over the session.
>
>
> Sure, at the expense of more complexity in the adaptation device (both 
> in terms of computation, and specification since this is a media 
> specific, rather than a general mechanism).

Thomas

-- 
Thomas Schierl
--------------
Fraunhofer HHI

Thomas Schierl wrote:
>> <mailto:[email-masked]> wrote:
>>
>>> Hi,
>>>
>>>     * a unicast transport where all the layers a user can receive
>>>       and decode are in a single RTP session. I am not sure if the
>>>       MANE apply to this scenario, but the server (or the source)
>>>       can run an adaptation function that select and optimize the
>>>       NALU to encapsulate in the RTP session. Using the same SSRC
>>>       for all layers in the session is not a problem, because a user
>>>       can distinguish layers by the dynamic PT as described in the SDP.
>>>     * a multicast transport where the user select one or more IP
>>>       address to adapt the received rate at its access link or its
>>>       terminal properties. This case is more sensitive, because some
>>>       receivers can decode only the AVC-base layer that must be
>>>       compliant with rfc 3984. Using the same SSRC is not a problem,
>>>       because the receiver can also use the dynamic PT to reorder
>>>       the NALUs in AUs
>>>
> ------------------------------------------------------------------------
>

---

From: Alex Eleftheriadis <[email-masked]>
Date: Fri, 11 Jul 2008 18:02:21 +0300

Thanks for the pointer, I somehow missed this exchange. It is very  
much to the point, and - as far as I am concerned - it appears to  
pretty much settle the issue.

--Alex

On Jul 11, 2008, at 4:54 PM, Thomas Schierl wrote:

> Further, we found at the Prague 2007 meeting, that a MANE's job is  
> not significantly reduced by using SSRC multiplexing. That time,  
> the MANE was assumed to be an RTP mixer, i.e. it terminates a  
> session and sets up a new session for the adapted packet streams.  
> This for sure enhances the complexity of such a device but removes  
> problems as SSRC collisions. Those days and also these days, this  
> arguments seem to hold, more or less. See also discussion on the  
> avt mailing list on draft-lennox-avt-rtp-layered-encoding- 
> timestamps-00.txt:
>
>> Sure, at the expense of more complexity in the adaptation device  
>> (both in terms of computation, and specification since this is a  
>> media specific, rather than a general mechanism).
>
>
> Thomas
>
> -- 
> Thomas Schierl
> --------------
> Fraunhofer HHI
>
>
>
>
> Thomas Schierl wrote:
>>> implicitly assuming that the SVC stream is on a single PT.  
>>> Mutliple PTs may be offered in an SDP Offer/Answer, but each one  
>>> of them corresponds to a complete subset of SVC layers (i.e.,  
>>> each PT is not a distinct layer). I think that having each layer  
>>> on a separate PT would break a lot of assumptions in SDP/RTP, but  
>>> I may be wrong.
>>>
>>> separate media streams, but it is true that in this case it is  
>>> actually one stream.
>> multiplexing is not preferable for differentiating between  
>> different SVC layers within the same RTP session.
>>
>> multiplexing was removed just for missing use cases. The original  
>> use case was adaption of SRTP protected streams without being in  
>> the security context. Observing this from today, I could still  
>> imagine a use case and that is the transport of multiple layers  
>> with it's own transport parameters. The point is here, that  
>> someone may have to open many ports with session multiplexing to  
>> bring all the layers through , e.g. a firewall. With SSRC  
>> multiplexing this could be reduced to just one port.
>>
>> Thomas
>>
>> -- 
>> Thomas Schierl
>> --------------
>> Fraunhofer HHI
>>
>>
>>
>>
>>

---

From: Randell Jesup <[email-masked]>
Date: Fri, 11 Jul 2008 11:46:22 -0400

Alex Eleftheriadis <[email-masked]> writes:

>Thanks for the pointer, I somehow missed this exchange. It is very  much to
>the point, and - as far as I am concerned - it appears to  pretty much
>settle the issue.
>
>--Alex
>
>> complexity of such a device but removes  problems as SSRC
>> collisions. Those days and also these days, this  arguments seem to hold,
>> more or less. See also discussion on the  avt mailing list on

This all seems reasonable to me.  The one question remaining for me was
Thomas' point about number of ports (especially when combined with ICE).

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: [email-masked]
Date: Fri, 11 Jul 2008 17:48:47 +0200

Hi thomas,

In multicast, a receiver can manage the received rate by receiving more
or less multicast flow from several IP multicast addresses (by mean of
IGMP requests). With SSRC multiplexing, it receive the full rate from
the alone IP adress, even if its access link has not enough bandwidth.
Then adaptation is impossible and may be the user cannot receive the
video at all. If firewall is a true constraint in networks, they are
however configurable.

Regards,

Gérard BABONNEAU

Thomas Schierl a écrit :

I think we already decided at Montreal meeting in 2006, that
PT
multiplexing is not preferable for differentiating between different
SVC layers within the same RTP session.

Thomas

--
Thomas Schierl
--------------
Fraunhofer HHI

--

BABONNEAU Gerard RD-MAPS-REN

REN/MAPS/DPC

Tel : +33 299124008

Email:
[email-masked]

R&D Rennes 4, rue du Clos-Courtel - BP 91226 - 35512 Cesson-Sevigne
Cedex - France

Mailer ThunderBird sous Mandriva

Installez simplement votre poste de travail Linux avec
http://web2000.rd.francetelecom.fr

---

From: Randell Jesup <[email-masked]>
Date: Fri, 11 Jul 2008 12:25:48 -0400

>transport of parameter sets

Note that this means you can't offer (in the same PT) different levels,
at least not with sprop-parameter-sets.  The details for offers and
responses without sprop-parameter-sets should also be locked down.

My suggestion: if either side doesn't have parameter-sets, then I'd suggest
both sides will need to use in-band paramsets, since you can't be sure the
person who answered without paramsets will use the ones in the SDP.  If
you're willing to ignore implementations that blew off sprop-parameter-sets
in 3984, then you can say that in-band paramsets will only NEED to be sent
by the party who sent the SDP without paramsets.

If in-band paramsets are to be blessed in 3984bis, then it needs
recommendations (or at least discussion of) when they need to be sent.

>sprop-parameter-sets=<base-64 data#0>)

My analysis:

At this point, the answerer can send (using the (level 2) paramsets in the
answer).  The offerer can not send unless it does so with the paramsets in
the answer, or using in-band paramsets with indexes that were not in either
offer or answer (so the answerer won't try to decode the packets if the
in-band paramsets are lost).

If the offerer is ok with the answerer's paramsets, or is using in-band
(and doesn't feel it needs to verify reception), no further negotiation is
needed. 

If the offerer wants different paramsets, or to validate reception of
in-band paramsets, a second round of offer/answer is needed.

>sprop-parameter-sets=<base-64 data#1> to be used in decoding the

The revised offer SHOULD include the paramsets that the answerer inserted,
unless those are in some manner unacceptable to the offerer.  The offerer 
SHOULD NOT offer a higher level than the answerer responded with.

>sprop-parameter-sets=<base-64 data#2> to be used in decoding the

The answerer must answer with an SDP, just as it did in the first
offer/answer exchange.  The answer is under the same rules as the first
answer.  Note that if a paramset changes, a new index should be used since
video streams are already being sent usign the old paramsets.

>  sprop-parameter-sets=<base-64 data#2>

There is no offer/answer/confirm in offer/answer.  :-) There's
offer/answer, and then a new offer/answer to replace the original
(re-INVITE in SIP).

---

From: "Even, Roni" <[email-masked]>
Date: Fri, 11 Jul 2008 19:32:30 +0300

Hi,
Looking at this email, I think we also need to look at SVC usage in
multipoint cases where you will get layers from different source based
on spatial layering, this will need different SSRCs
Roni

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Thomas Schierl
> Sent: Friday, July 11, 2008 4:55 PM
> To: Alex Eleftheriadis
> Cc: Jonathan Lennox; Xavier Marjou; avt; [email-masked]
> Subject: Re: [AVT] SSRC use in single- and multi-session transmission
> for SVCpayload draft
> 
> >> streams is when you actually want different transport-layer
behavior
> for
> >> as a mixer, rather than as a translator, and drop layers from
within
> an
> (both
use
> > e.g. a firewall. With SSRC multiplexing this could be reduced to
just
---------------------------------------------------------------------
> ---

---

From: <[email-masked]>
Date: Fri, 11 Jul 2008 21:26:37 +0300

Hi Randell,

Thanks a lot for your comments. See inline.

>From: ext Randell Jesup [mailto:[email-masked]] 
>Sent: 11 July, 2008 19:26
>To: Wang Ye-Kui (Nokia-NRC/Tampere)
>Cc: [email-masked]
>Subject: Re: [AVT] SDP offer/answer for the 3984bis and SVC 
>payload drafts
>

A few points:
- The Level in profile-level-id is the maximum level the offerer offers,
all lower levels are automatically supported. Therefore, this does mean
offering of different levels.
- sprop-parameter-sets being included in SDP is only used for
out-of-band transport. As long as the two endpoints agree with the
configuration (profile-level-id, packetization-mode, and, if required by
packetization-mode, sprop-deint-buf-req), one side must accept any
parameter sets complaint with other parameters in the SDP from the other
side. This also implies that the two sides are free to use different
parameter sets for their encodings. 
- The sender of an SDP including sprop-parameter-sets is willing to use
those included parameter sets for its encoding (by the encoder there),
while the receiver of an SDP including sprop-parameter-sets will use
those included parameter sets for its decoding (by the decoder) once the
configuration is agreed. In any side, parameter sets for encoding and
parameter sets for decoding are independent from each other, as the
encoder and the decoder in one side are independent. 

>person who answered without paramsets will use the ones in the SDP.  If

I think the following flexibility is allowed by RFC 3984 (let me know if
I am wrong). Even during negotiation (through offer/answer)
sprop-parameter-sets was involved, one endpoint can still send in-band
parameter sets. This indicates that the two sides can use different
parameter sets for their encodings anyway. 

Therefore, why the two sides should be aligned whether to send a subset
of parameter sets out-of-band?

One side must accept the parameter sets in SDP from the other side if
the configuration is accepted. 

>recommendations (or at least discussion of) when they need to be sent.

3984 already includes some recommendations on in-band sending of
parameter. Of course if more is needed they should be included in the
bis draft. 

I stop writing further comments for the following, as most of them are
dependent on how out-of-band parameter sets are used by the SDP sender
and the SDP receiver, except for the education on general rule of
offer/answer, which I accept with thanks (obviously I have a lot to
learn in this area). 

Let's find out a common place here first to talk the same language :-)

BR, YK

---

From: <[email-masked]>
Date: Fri, 11 Jul 2008 21:32:44 +0300

Hi Roni,

Do you mean a kind of distributed coding wherein an endpoint receives
different layers of a same SVC bitstream encoded by different senders?
Or do you mean different senders send different bitstreams that are
spatially composed by the receiver?

BR, YK 

>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of ext Even, Roni
>Sent: 11 July, 2008 19:33
>To: Thomas Schierl; Alex Eleftheriadis
>Subject: Re: [AVT] SSRC use in single- and multi-session 
>transmission forSVCpayload draft
>
>Hi,
>> >>> While using the same SSRC for each layer is simpler for 
>some case,
>> it
>> >>>       can distinguish layers by the dynamic PT as 
>described in the
>> SDP.
>> > many ports with session multiplexing to bring all the 
>layers through
>> ,

---

From: Randell Jesup <[email-masked]>
Date: Sat, 12 Jul 2008 00:54:12 -0400

>offering of different levels.

Yes, but your comment was that sprop-parameter-sets needs to be "compliant"
with profile-level-id.  As mentioned at IETF in Philly, if you have to
provide parameter sets for every possible level at or below the one in
profile-level-id, the set (and SDP) could get very large.  If you don't
provide them, then it's problematic for the answerer to downgrade the level
unless you either do a second offer/answer, or send them in-band (with the
attendent possible-loss issues).

>parameter sets for their encodings. 

Yup.  Though the current 3984 requires that the answerer include the
paramset(s) from the offer in the answer, if I remember (which is a real
problem with level downgrade).

>encoder and the decoder in one side are independent. 

Modulo the requirement above, and that on a level downgrade, the offerer
MUST NOT use parameter sets from a level above that of the answer.  

>>My suggestion: if either side doesn't have parameter-sets, then I'd
>>suggest both sides will need to use in-band paramsets, since you can't be
>>sure the person who answered without paramsets will use the ones in the
>>SDP.  If
>

You can send them, though RFC 3984 recommends against it, and you must
send them realizing that they may be lost.  Any actual change in parameter
sets may take a while or even forever to be received, especially if there's
no working RTCP (CCM/etc) back-channel to report loss of sync.

>of parameter sets out-of-band?

I don't understand the question.

>the configuration is accepted. 

Not in the case of level downgrade, however (see above).

>>you're willing to ignore implementations that blew off
>>sprop-parameter-sets in 3984, then you can say that in-band paramsets
>>will only NEED to be sent by the party who sent the SDP without
>>paramsets.
>>

My memory was that 3984 basically said "don't do it", but maybe there was
more there somewhere else.  I realize current common practice is to (re)send
paramsets on every IDR, which (in the common packetization-mode 0) yields a
bunch of extra packets.  The point of sprop-parameter-sets was to avoid the
need to do so, and also to avoid "unlucky" stream-startup losses from
delaying decoding of the stream.

>offer/answer, which I accept with thanks (obviously I have a lot to

My analysis below was mostly about how level-downgrade would work (or
not work) using the rules you gave.  Also, offer-answer usually discourages
a second round, though ICE uses a second round to lock down the transport,
and some recommendations for SIP/VoIP suggest having the offerer can lock
down the payload (if the answerer accepts more than one PT) by a second
round.

>>At this point, the answerer can send (using the (level 2) paramsets in
>>the answer).  The offerer can not send unless it does so with the
>>paramsets in the answer, or using in-band paramsets with indexes that
>>were not in either offer or answer (so the answerer won't try to decode
>>the packets if the in-band paramsets are lost).
>>
>>(and doesn't feel it needs to verify reception), no further negotiation
>>is needed.
>>
>>The revised offer SHOULD include the paramsets that the answerer
>>inserted, unless those are in some manner unacceptable to the offerer.
>>The offerer SHOULD NOT offer a higher level than the answerer responded
>>with.
>>

---

From: <[email-masked]>
Date: Sat, 12 Jul 2008 11:58:50 +0300

I guess we are now basically on the same ground. So here are some
examples again how the offer/answer should be done according to my
understanding. Also, see inline. 

Any comments are welcome, and comments on which direction we should go
for the two drafts (or only the 3984bis draft and then let the SVC draft
refer to this) would be greatly appreciated. 

Case 1: Original offer accepted without level downgrade (one round
needed), with out-of-band parameter sets transport. Note that
sprop-parameter-sets=<base-64 data#0> is basically independent of
sprop-parameter-sets=<base-64 data#1>. The former is to be used by the
encoder of the offerer and the decoder of the answerer. The latter is to
be used by the encoder of the answerer and the decoder of the offerer.
Requiring them (including both sequence parameter sets and picture
parameter sets) to be identical or the former to be a subset of the
latter (as in the current RFC 3984) seriously limits independent
encoding optimizations by the independent encoders to the basically
independent video contents. 

Offer SDP#0: 
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0
  packetization-mode=1;
  sprop-parameter-sets=<base-64 data#0> 

Answer SDP#1: 
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000

Case 2: Original offer accepted without level downgrade (one round
needed), with in-band parameter sets transport

Offer SDP#0: 
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000

Answer SDP#1: 
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000

Case 3: Original offer accepted with level downgrade (two rounds
needed), with out-of-band parameter sets transport. Note that
sprop-parameter-sets=<base-64 data#0> contains level_id indicating Level
3.0, therefore cannot be used as the answerer wants Level 2.0 and does
not need to be stored by the answerer. The sprop-parameter-sets=<base-64
data#2> is to be used by the encoder of the offerer and the decoder of
the answerer. The sprop-parameter-sets=<base-64 data#1> is to be used by
the encoder of the answerer and the decoder of the offerer. Again,
requiring them (including both sequence parameter sets and picture
parameter sets) to be identical or the former to be a subset of the
latter (as in the current RFC 3984) seriously limits independent
encoding optimizations by the independent encoders to the basically
independent video contents. 

Offer SDP#0: 
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0
  packetization-mode=1;
  sprop-parameter-sets=<base-64 data#0> 

Answer SDP#1:
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0
  packetization-mode=1;
  sprop-parameter-sets=<base-64 data#1>

Offer SDP#2:
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000

Answer SDP#3: //SDP#3 is identical to SPD#1
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0
  packetization-mode=1;
  sprop-parameter-sets=<base-64 data#1>

Case 4: Original offer accepted with level downgrade (one round needed),
with in-band parameter sets transport

Offer SDP#0: 
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000

Answer SDP#1: 
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0
  packetization-mode=1;

>attendent possible-loss issues).

See above. For case 3, it is certainly possible to include in the
original offer SDP multiple payload types, each corresponding to a
level. But that is not ideal as you said that the SDP would be large. 

>problem with level downgrade).

You are right - that limit should be removed. Along with the relaxation
of that limit is that the parameter-add gets useless (it should always
be equal to 1 or not present).

>MUST NOT use parameter sets from a level above that of the answer.  

Sure. 

>no working RTCP (CCM/etc) back-channel to report loss of sync.

You can send redundant copies of parameter sets, or use FEC, etc. I
think this has been mentioned in RFC 3984 for in-band transmission of
parameter sets. 

>I don't understand the question.

The logic is here (but not important now). My understanding of the
intention of requiring the two sides to be aligned in using of
out-of-band parameter sets transmission is to mandate all the four codec
parts (encoder/decoder of the two sides) to use the same set of
parameter sets. However, as in-band transport of parameter sets is not
disallowed by language like "SHALL NOT" or "MUST NOT", that intention
would fail anyway. Therefore, there is no point to require the two sides
to be aligned in using of out-of-band parameter sets transmission. 

>Not in the case of level downgrade, however (see above).

It looks that we have a different understanding of level downgrade. My
understanding is that when the offerer offers one Level, all the lower
levels are automatically offered, i.e. no need to offer multiple payload
types. Your understanding seems to be mine in addition to that the
answerer should accept the parameter sets originally offered. 

>more there somewhere else.  I realize current common practice 

I guess you mean the use of the language "RECOMMENDED" or "SHOULD" in
the spec. For in-band transmission of parameter sets, the spec mentioned
the following in section 8.4: "If in-band signaling of parameter sets is
used, the sender SHOULD take the error characteristics into account and
use mechanisms to provide a high probability for delivering the
parameter sets correctly.  Mechanisms that increase the probability for
a correct reception include packet repetition, FEC, and retransmission."

>is to (re)send

Even out-of-band transmission of parameter sets is not used, it is not a
good idea for many applications (e.g. streaming, conversational) to do
this "common practice" - only broadcast applications require this. 

>down the payload (if the answerer accepts more than one PT) by a second

Thanks again for these knowledge. As you can see, to allow for both
level-downgrade and out-of-band transmission of parameter sets, it seems
unavoidable to do a second-round. 

BR, YK

---

From: <[email-masked]>
Date: Sat, 12 Jul 2008 18:02:50 +0300

Just an idea to save the last unspecified NAL unit type value (31) for
future extension. Comments are welcome!

In the latest SVC draft, NAL unit type value 31 is used for the
Non-Interleaved Multi-Time Aggregation Packet (NI-MTAP), to allow for
aggregation of NAL units from more than one access unit but their
transmission order is still identical to the decoding order. 

The idea is to re-use the existing MTAP16 - NAL unit type 26, and, if
needed, MTAP24 - NAL unit type 27 specified in RFC 3984, with
backward-compatible revisions. 

There are two alternatives. 

Alternative 1: 
The semantics of MTAP16 and, if needed, MTAP24, are constrained as
follows. When they are used in non-interleaved transmission (i.e. the
session packetization mode is the Non-Interleaved mode) for payload type
H264-SVC, the DON related fields MUST be set such that the transmission
order of the aggregated NAL units is identical  to the decoding order.
Certainly, this cannot be used for the base layer or a subset therein
using payload type H264. Compared to using NI-MTAP, a few more bytes are
needed for the DON related fields, with the advantage that the last
unspecified NAL unit type value (31) still remains there for future
extension. 

Alternative 2: 
The syntax of MTAP16 and, if needed, MTAP24, are changed as follows. The
DON related fields are changed to be optionally present. When MTAP16 and
MTAP24 packets are used in non-interleaved transmission (i.e. the
session packetization mode is the Non-Interleaved mode) for payload type
H264-SVC, the DON related fields MUST NOT be present (this makes the
structure essentially the same as NI-MTAP). The semantics are further
restricted the same as above (and as for NI-MTAP), i.e. the transmission
order of the aggregated NAL units is identical  to the decoding order.
Again, this cannot be used for the base layer or a subset therein using
payload type H264. Compared to using NI-MTAP, now even the a few more
bytes needed for the DON related fields are saved, with the net
advantage that the last unspecified NAL unit type value (31) still
remains there for future extension. 

Certainly, the second one is preferred. 

Furthermore, for both alternatives, the MTAPs can be further allowed to
include FUs, such that aggregation of FUs and using of PACSI NAL units
with FUs can be enabled in a backward-compatible way. Actually this FU
aggregation capability can also be extended to STAPs. 

BR, YK

---

From: Thomas Schierl <[email-masked]>
Date: Sat, 12 Jul 2008 22:35:33 +0200

Hi Ye-Kui,

For "Alternative 2", I would be interested in how the MTAP packets
without DON (equal to NI-MTAP of the current SVC draft) would be
integrated into RFC3984bis? I guess these packets are not usable with
'H264'.

Thomas

[email-masked] wrote:

Alternative 1:

Alternative 2:

BR, YK

--

____________________________________________________
Thomas Schierl, Dipl.-Ing.
Fraunhofer HHI
Senior Research Engineer
Image Processing Dept. / Multimedia Transport
Einsteinufer 37, 10587 Berlin, Germany
PHONE: +49 30 31002 227, FAX: +49 30 392 7200, WEB: ip.hhi.de
EMAIL: [email-masked], SKYPE: thomas.schierl

---

From: <[email-masked]>
Date: Sat, 12 Jul 2008 23:58:21 +0300

Hi Thomas,
 
You are right that the change cannot be integrated into 3984bis as it is
not applicable to 'H246'. Therefore, if people agree the change should
only stay in the SVC draft. 
 
BR, YK

________________________________

	From: [email-masked] [mailto:[email-masked]] On
Behalf Of ext Thomas Schierl
	Sent: 12 July, 2008 23:36
	To: Wang Ye-Kui (Nokia-NRC/Tampere)
	Cc: [email-masked]
	Subject: Re: [AVT] NI-MTAP packet type in the SVC payload draft

	Hi Ye-Kui,
	
	For "Alternative 2", I would be interested in how the MTAP
packets without DON (equal to NI-MTAP of the current SVC draft) would be
integrated into RFC3984bis? I guess these packets are not usable with
'H264'.
	
	Thomas

		Just an idea to save the last unspecified NAL unit type
value (31) for
		future extension. Comments are welcome!
		
		In the latest SVC draft, NAL unit type value 31 is used
for the
		Non-Interleaved Multi-Time Aggregation Packet (NI-MTAP),
to allow for
		aggregation of NAL units from more than one access unit
but their
		transmission order is still identical to the decoding
order. 
		
		The idea is to re-use the existing MTAP16 - NAL unit
type 26, and, if
		needed, MTAP24 - NAL unit type 27 specified in RFC 3984,
with
		backward-compatible revisions. 

		Alternative 1: 
		The semantics of MTAP16 and, if needed, MTAP24, are
constrained as
		follows. When they are used in non-interleaved
transmission (i.e. the
		session packetization mode is the Non-Interleaved mode)
for payload type
		H264-SVC, the DON related fields MUST be set such that
the transmission
		order of the aggregated NAL units is identical  to the
decoding order.
		Certainly, this cannot be used for the base layer or a
subset therein
		using payload type H264. Compared to using NI-MTAP, a
few more bytes are
		needed for the DON related fields, with the advantage
that the last
		unspecified NAL unit type value (31) still remains there
for future
		extension. 
		
		Alternative 2: 
		The syntax of MTAP16 and, if needed, MTAP24, are changed
as follows. The
		DON related fields are changed to be optionally present.
When MTAP16 and
		MTAP24 packets are used in non-interleaved transmission
(i.e. the
		session packetization mode is the Non-Interleaved mode)
for payload type
		H264-SVC, the DON related fields MUST NOT be present
(this makes the
		structure essentially the same as NI-MTAP). The
semantics are further
		restricted the same as above (and as for NI-MTAP), i.e.
the transmission
		order of the aggregated NAL units is identical  to the
decoding order.
		Again, this cannot be used for the base layer or a
subset therein using
		payload type H264. Compared to using NI-MTAP, now even
the a few more
		bytes needed for the DON related fields are saved, with
the net
		advantage that the last unspecified NAL unit type value
(31) still
		remains there for future extension. 

		Furthermore, for both alternatives, the MTAPs can be
further allowed to
		include FUs, such that aggregation of FUs and using of
PACSI NAL units
		with FUs can be enabled in a backward-compatible way.
Actually this FU
		aggregation capability can also be extended to STAPs. 
		
		BR, YK

	--

---

From: "Even, Roni" <[email-masked]>
Date: Sun, 13 Jul 2008 10:05:56 +0300

YK,
I think both are feasible under the control of a conference bridge that
may instruct the receiver which incoming streams to use to compose the
displayed layout.
Roni

> From: [email-masked] [mailto:[email-masked]]
> Sent: Friday, July 11, 2008 9:33 PM
> To: Even, Roni; [email-masked]; [email-masked]
> Cc: [email-masked]; [email-masked];
[email-masked];
> [email-masked]
> Subject: RE: [AVT] SSRC use in single- and multi-session transmission
> forSVCpayload draft
> 
> 
> Hi Roni,
> 
> transmission
> >> was assumed to be an RTP mixer, i.e. it terminates a session and
> sets
> >> up
> >> a new session for the adapted packet streams. This for sure
enhances
> >> the
collisions.
> >> >> The question is if there's any benefit to multiplexing layers
> into
> >> an
> >> >> RTP session by placing them in separate RTP streams, as opposed
> to
> >> doing
> >> >> session was present for a few revisions of the payload draft,
but
> >> was
> >> >> The compelling reason for splitting a media stream across
> multiple
> >> RTP
> >> >> The case of adaptation at network boundaries is certainly a
> common
> >> use
> >> >> media-aware, it necessarily understands the payload format.
This
> >> avoids

---

From: <[email-masked]>
Date: Sun, 13 Jul 2008 13:09:28 +0300

Roni,

Here is my understanding. For the first case (distributed soruce coding
for SVC), I cannot find a good use case for it. For the second case
(spatial mixing), it seems to be out of the scope of the SVC draft. 

BR, YK 

>From: ext Even, Roni [mailto:[email-masked]] 
>Sent: Sunday, July 13, 2008 10:06 AM
>To: Wang Ye-Kui (Nokia-NRC/Tampere); [email-masked]; [email-masked]
>[email-masked]; [email-masked]
>Subject: RE: [AVT] SSRC use in single- and multi-session 
>I think both are feasible under the control of a conference 
>bridge that may instruct the receiver which incoming streams 
>to use to compose the displayed layout.
>Roni
>
>Behalf Of 
>> >ext Even, Roni
>> >on spatial layering, this will need different SSRCs Roni
>> >
>> >> >> multicast groups.  In this case the streams obviously 
>have to go
>> >> into
>> >> >> problems of stream association and SSRC collision, 
>especially if
>> >> you're
>> >> >>>       for all layers in the session is not a problem, 
>because a
>> >> user
>> >> >>>       terminal properties. This case is more 
>sensitive, because
>> >> some
>> >> >> corresponds to a complete subset of SVC layers (i.e., 
>each PT is
>> >not
>> >> > transport parameters. The point is here, that someone 
>may have to
>> >> open
>> >> > Audio/Video Transport Working Group [email-masked] 
>> >> Audio/Video Transport Working Group [email-masked]

---

From: <[email-masked]>
Date: Sun, 13 Jul 2008 19:56:59 +0300

Concluded from the discussions in the email threads listed below, I
added a new parameter, spatial-resolution, to the RFC 3984bis draft
(which will be submitted by tomorrow): 
 
http://www.ietf.org/mail-archive/web/avt/current/msg09425.html
http://www.ietf.org/mail-archive/web/avt/current/msg09650.html

The parameter is specified as follows (basically as Randell suggested in
http://www.ietf.org/mail-archive/web/avt/current/msg09429.html). 

spatial-resolution:
This parameter MAY be used to indicate the maximum spatial resolution of
a NAL unit stream or the preferred spatial resolution of a receiver.
The value is a base16 [6] (hexadecimal) representation of the width and
height of the spatial resolution, in pixels, separated by a comma. 

Usage in SDP offer/answer: 
- When "a=sendrecv" or no direction attribute is used,
"spatial-resolution" indicates both the maximum spatial resolution of
the stream sent by the declaring entity and the preferred spatial
resolution for receiving a stream. 
- When "a=sendonly", "spatial-resolution" indicates the maximum spatial
resolution of the outgoing stream.
- When "a=recvonly", "spatial-resolution" indicates the preferred
spatial resolution for receiving a stream.

Comments are welcome.

BR, YK

---

From: Alex Eleftheriadis <[email-masked]>
Date: Mon, 14 Jul 2008 02:49:25 +0300

> (spatial mixing), it seems to be out of the scope of the SVC draft.

I agree on the first one, but for the second one I don't believe it's  
outside the scope of the draft in the sense that the draft should  
(and I believe does) allow for this, even if it is not explicitly  
described (as many other possible application-level functionalities).

Getting back to the original question, this appears to be orthogonal  
to SSRC use. A conference bridge is certainly media-aware.

PS: I point out that, as of upcoming version 13, the draft will have  
no scope section :-)

--Alex

---

From: Alex Eleftheriadis <[email-masked]>
Date: Mon, 14 Jul 2008 05:49:37 +0300

This also relates to an ommission in ver. 12 Section 6.10.1 (CS-DON)  
where there is nothing specified for CS-DON and NI-MTAPs, as pointed  
out by Mike. In the NI-TC multi-session transmission mode, we need to  
specify how CS-DON number are calculated within NI-MTAPs. Similarly  
the NI-C/NI-TC packetization (ver. 12 Section 7.2.1) rules need to  
specify what happens in NI-MTAPs.  Finally, the PACSI section (6.8)  
is silent about NI-MTAPs, and so is the NI-MTAP section about  
PACSI... In fact, the PACSI section is also silent about MTAPs (it  
assumes there is just one PACSI for the whole aggregation packet -  
how useful is that??).

In the ver. 12 design I presume the NI-MTAP NI-TC mode packetization  
rule would be to - at least - start with a PACSI, as with STAP-As,  
include the DONC field in the NI-TC mode, and derive CS-DON numbers  
as usual.

With your proposal, I presume that the rule would be:

- NI-T

Remove DOND from MTAP16 aggregation unit, keeping NALU unit size and  
TS_offset, and remove DONB from MTAP16 header. MTAP24 does not appear  
useful here.

- NI-TC

Keep DOND in aggregation unit and DONB in MTAP16. Use MTAP's DONB/ 
DONDs for CS-DON, enforce ordering per decoding order, set PACSI T=0,  
and have the PACSI DON (as specified by the MTAP header) be equal to  
the CS-DON of the next non-PACSI NAL unit. The PACSI here is, in fact  
optional.

  I am in favor of re-using a NAL unit type and thus preserving #31.   
However, this will become yet another example of how unbalanced the   
current design is, in that PACSIs are required in NI-TC mode for all  
packet types when there is a NALU-time change, they must be the first  
uni in an STAP-A, and are not needed in this new MTAP. Here the PACSI  
is used as a DON mechanism. But PACSIs are supposed to be used to  
indicate layer information, not access units, so there should be  
several within an access unit... We allow empty PACSIs (for the NI-T  
mode). Yet PACSIs are, in general, optional... :-)

It is strange to me that we still consider them optional. If we made  
them mandatory this will streamline the packetization rules, minimize  
the difference between the multi-session transmission modes to just  
saying which fields in the PACSI and MTAPng (the new MTAP) are  
present, and that's it. Plus ensure that scalability works in the  
network as advertized.

Anyway. For the MTAPng concept, I am afraid we will have to wait for  
the post-Dublin version to get it integrated if accepted. Right now  
there is just too much to do to (*try to*) get a solid version 13 by  
tomorrow' deadline.

--Alex

On Jul 12, 2008, at 11:58 PM, <[email-masked]> wrote:

> Hi Thomas,
>
> You are right that the change cannot be integrated into 3984bis as  
> it is not applicable to 'H246'. Therefore, if people agree the  
> change should only stay in the SVC draft.
>
> BR, YK
>
> From: [email-masked] [mailto:[email-masked]] On Behalf  
> Of ext Thomas Schierl
> integrated into RFC3984bis? I guess these packets are not usable  
> with 'H264'.
>
> Thomas
>

---

From: <[email-masked]>
Date: Mon, 14 Jul 2008 09:39:41 +0300

Hi Alex,
 
Very good analysis and design rules - I basically agree all what you
said. 
 
BR, YK

	From: [email-masked] [mailto:[email-masked]] On
Behalf Of ext Alex Eleftheriadis
	Sent: 14 July, 2008 05:50
	To: Wang Ye-Kui (Nokia-NRC/Tampere)
	Cc: [email-masked]; [email-masked]
	Subject: Re: [AVT] NI-MTAP packet type in the SVC payload draft

	This also relates to an ommission in ver. 12 Section 6.10.1
(CS-DON) where there is nothing specified for CS-DON and NI-MTAPs, as
pointed out by Mike. In the NI-TC multi-session transmission mode, we
need to specify how CS-DON number are calculated within NI-MTAPs.
Similarly the NI-C/NI-TC packetization (ver. 12 Section 7.2.1) rules
need to specify what happens in NI-MTAPs.  Finally, the PACSI section
(6.8) is silent about NI-MTAPs, and so is the NI-MTAP section about
PACSI... In fact, the PACSI section is also silent about MTAPs (it
assumes there is just one PACSI for the whole aggregation packet - how
useful is that??). 

	In the ver. 12 design I presume the NI-MTAP NI-TC mode
packetization rule  would be to - at least - start with a PACSI, as with
STAP-As, include the DONC field in the NI-TC mode, and derive CS-DON
numbers as usual. 

	- NI-T

	Remove DOND from MTAP16 aggregation unit, keeping NALU unit size
and TS_offset, and remove DONB from MTAP16 header. MTAP24 does not
appear useful here. 

	- NI-TC

	Keep DOND in aggregation unit and DONB in MTAP16. Use MTAP's
DONB/DONDs for CS-DON, enforce ordering per decoding order, set PACSI
T=0, and have the PACSI DON (as specified by the MTAP header) be equal
to the CS-DON of the next non-PACSI NAL unit. The PACSI here is, in fact
optional. 

	 I am in favor of re-using a NAL unit type and thus preserving
#31.  However, this will become yet another example of how unbalanced
the  current design is, in that PACSIs are required in NI-TC mode for
all packet types when there is a NALU-time change, they must be the
first uni in an STAP-A, and are not needed in this new MTAP. Here the
PACSI is used as a DON mechanism. But PACSIs are supposed to be used to
indicate layer information, not access units, so there should be several
within an access unit... We allow empty PACSIs (for the NI-T mode). Yet
PACSIs are, in general, optional... :-)  

	It is strange to me that we still consider them optional. If we
made them mandatory this will streamline the packetization rules,
minimize the difference between the multi-session transmission modes to
just saying which fields in the PACSI and MTAPng (the new MTAP) are
present, and that's it. Plus ensure that scalability works in the
network as advertized.

	Anyway. For the MTAPng concept, I am afraid we will have to wait
for the post-Dublin version to get it integrated if accepted. Right now
there is just too much to do to (*try to*) get a solid version 13 by
tomorrow' deadline. 

	--Alex

		Hi Thomas,
		 
		You are right that the change cannot be integrated into
3984bis as it is not applicable to 'H246'. Therefore, if people agree
the change should only stay in the SVC draft. 
		 
		BR, YK

			From: [email-masked]
[mailto:[email-masked]] On Behalf Of ext Thomas Schierl
			Sent: 12 July, 2008 23:36
			To: Wang Ye-Kui (Nokia-NRC/Tampere)
			Cc: [email-masked]
			Subject: Re: [AVT] NI-MTAP packet type in the
SVC payload draft

			Hi Ye-Kui,
			
			For "Alternative 2", I would be interested in
how the MTAP packets without DON (equal to NI-MTAP of the current SVC
draft) would be integrated into RFC3984bis? I guess these packets are
not usable with 'H264'.
			
			Thomas

				Just an idea to save the last
unspecified NAL unit type value (31) for
				future extension. Comments are welcome!
				
				In the latest SVC draft, NAL unit type
value 31 is used for the
				Non-Interleaved Multi-Time Aggregation
Packet (NI-MTAP), to allow for
				aggregation of NAL units from more than
one access unit but their
				transmission order is still identical to
the decoding order. 
				
				The idea is to re-use the existing
MTAP16 - NAL unit type 26, and, if
				needed, MTAP24 - NAL unit type 27
specified in RFC 3984, with
				backward-compatible revisions. 

				Alternative 1: 
				The semantics of MTAP16 and, if needed,
MTAP24, are constrained as
				follows. When they are used in
non-interleaved transmission (i.e. the
				session packetization mode is the
Non-Interleaved mode) for payload type
				H264-SVC, the DON related fields MUST be
set such that the transmission
				order of the aggregated NAL units is
identical  to the decoding order.
				Certainly, this cannot be used for the
base layer or a subset therein
				using payload type H264. Compared to
using NI-MTAP, a few more bytes are
				needed for the DON related fields, with
the advantage that the last
				unspecified NAL unit type value (31)
still remains there for future
				extension. 
				
				Alternative 2: 
				The syntax of MTAP16 and, if needed,
MTAP24, are changed as follows. The
				DON related fields are changed to be
optionally present. When MTAP16 and
				MTAP24 packets are used in
non-interleaved transmission (i.e. the
				session packetization mode is the
Non-Interleaved mode) for payload type
				H264-SVC, the DON related fields MUST
NOT be present (this makes the
				structure essentially the same as
NI-MTAP). The semantics are further
				restricted the same as above (and as for
NI-MTAP), i.e. the transmission
				order of the aggregated NAL units is
identical  to the decoding order.
				Again, this cannot be used for the base
layer or a subset therein using
				payload type H264. Compared to using
NI-MTAP, now even the a few more
				bytes needed for the DON related fields
are saved, with the net
				advantage that the last unspecified NAL
unit type value (31) still
				remains there for future extension. 

				Furthermore, for both alternatives, the
MTAPs can be further allowed to
				include FUs, such that aggregation of
FUs and using of PACSI NAL units
				with FUs can be enabled in a
backward-compatible way. Actually this FU
				aggregation capability can also be
extended to STAPs. 
				
				BR, YK

				  

			-- 

____________________________________________________
			Thomas Schierl, Dipl.-Ing.
			Fraunhofer HHI
			Senior Research Engineer
			Image Processing Dept. / Multimedia Transport
			Einsteinufer 37, 10587 Berlin, Germany
			PHONE: +49 30 31002 227, FAX: +49 30 392 7200,
WEB: ip.hhi.de
			EMAIL: [email-masked], SKYPE:
thomas.schierl

---

From: "Even, Roni" <[email-masked]>
Date: Mon, 14 Jul 2008 10:30:25 +0300

YK,
I thought that the parameter is only to specify the sender information.
I do not think that this parameter as a receiver request is well
specified. 
Roni

> [email-masked]
> Sent: Sunday, July 13, 2008 7:57 PM
> To: [email-masked]
> Subject: [AVT] spatial-resolution parameter for RFC 3984bis
> 
>

---

From: "Even, Roni" <[email-masked]>
Date: Mon, 14 Jul 2008 10:38:18 +0300

YK,
Just to clarify,
The text on the receiver side says that this is the requested
resolution, it does not address issue of negotiation, what will the
sender do if it cannot send this resolution, should it reject the
stream?.
If this is for negotiation do you except to do a new offer answer if the
receiver wants a new resolution?

Roni

---

From: <[email-masked]>
Date: Mon, 14 Jul 2008 10:54:59 +0300

Hi Roni,

The resolution must be restricted by the level part of profile-level-id
and max-fs. Therefore, if a configuration is agreed (or agreeable), then
the answerer must request a resolution under the constraint, and the
offerer must accept the request as long as it meets the restriction.
This is similar as sprop-parameter-sets, as long as it is complaint with
the agreed (or agreeable) configuration, the other side must accept it. 

Would it work this way?

BR, YK 

>Sent: 14 July, 2008 10:38
>To: Wang Ye-Kui (Nokia-NRC/Tampere); [email-masked]
>Subject: RE: [AVT] spatial-resolution parameter for RFC 3984bis
>
>YK,
>Just to clarify,

---

From: "Even, Roni" <[email-masked]>
Date: Mon, 14 Jul 2008 10:59:17 +0300

YK,
Having an upper restriction is OK but you still do not address the issue
of what if the sender can not supply it.
For example let's say that the request is for a resolution of 100x50,
what will the sender do if it can not send such resolution, should it
reject the request, should it send 90x40 or 110x50 if it can?
Roni

> Sent: Monday, July 14, 2008 10:55 AM
> To: Even, Roni; [email-masked]
> Of
> >> The parameter is specified as follows (basically as Randell
> suggested
> >> in
> >> This parameter MAY be used to indicate the maximum spatial
> resolution
> >> of

---

From: <[email-masked]>
Date: Mon, 14 Jul 2008 11:41:01 +0300

Roni,

If the requested resolution (100x50) is supported by the Level, and the
offerer agreed or offered the Level, then the offerer MUST accept and
send the resolution. There is no reason that a sender does not support a
certain resolution complaint to a level the sender supports, right? And
as it is als restricted that the answerer MUST only request a resolution
complaint to the agreed or offered Level (or any lower Level), it is
then impossible that such a requested resolution cannot be supported by
the offerer. 

If the answerer requests a resolution of a Level that the offerer cannot
support, the offerer has to reject anyway as it cannot support the Level
at the first step. 

BR, YK 

>Sent: 14 July, 2008 10:59
>> >> The value is a base16 [6] (hexadecimal) representation of 
>the width
>> >and

---

From: "Even, Roni" <[email-masked]>
Date: Mon, 14 Jul 2008 11:44:48 +0300

YK,
According to H.264 the encoder does not need to scale, this is up to the
receiver, so there is no requirement for the sender to support a
specific resolution even if it is within the level!!!!
Roni

> Sent: Monday, July 14, 2008 11:41 AM
> Behalf
> >> >> "spatial-resolution" indicates both the maximum spatial
> resolution
> >> of

---

From: <[email-masked]>
Date: Mon, 14 Jul 2008 11:53:58 +0300

Roni,

But why is this related to encoder scaling? The encoder can just encode
a bitstream with the resolution requested, without scaling. If the
encoder supports Level 2.0, then it automatically supports all
resolutions for which the number of MBs is not greater than that for CIF
(352x288), without scaling. 

BR, YK 

>Sent: 14 July, 2008 11:45
>> send the resolution. There is no reason that a sender does 
>not support
>> a
>> >> and max-fs. Therefore, if a configuration is agreed (or 
>agreeable),
>> >> then
>> >> This is similar as sprop-parameter-sets, as long as it is 
>complaint
>> >> with
>> >> the agreed (or agreeable) configuration, the other side 
>must accept
>> >it.
>> >>
>> >> Would it work this way?
>> >>
>> >> BR, YK
>> >>

---

From: "Even, Roni" <[email-masked]>
Date: Mon, 14 Jul 2008 12:00:44 +0300

YK,
This is not required from the encoder to prepare a specific resolution
or to support it. It only need to support a resolution within the level
and not any resolution within the level.
Roni 

> Sent: Monday, July 14, 2008 11:54 AM
> >> then impossible that such a requested resolution cannot be
supported
> >by
> >> >> >> - When "a=sendonly", "spatial-resolution" indicates the
> maximum
> >> >> >spatial

---

From: <[email-masked]>
Date: Mon, 14 Jul 2008 12:05:08 +0300

Roni,

Now I understand. Yes, requirement to encoder is different requirement
to decoder. Thanks (also to Gary in private email exchanges)!

OK, let me then think how the parameter should be used in negotiation
with this new context. 

BR, YK 

>Sent: 14 July, 2008 12:01
>> >> If the requested resolution (100x50) is supported by the 
>Level, and
>> >the
>> >> certain resolution complaint to a level the sender 
>supports, right?
>> >And
>> >> >> the answerer must request a resolution under the 
>constraint, and
>> >the
>> >> >> >resolution, it does not address issue of negotiation, 
>what will
>> >the

---

From: "Even, Roni" <[email-masked]>
Date: Mon, 14 Jul 2008 12:08:12 +0300

YK,
This is why we have all the text in H.241 about the resolution, the
request can be for exact resolution and the encoder can reject it or to
a close resolution and it specifies what it means
Roni

> Sent: Monday, July 14, 2008 12:05 PM
> >> resolutions for which the number of MBs is not greater than that
for
> >> CIF
> >> >> offerer agreed or offered the Level, then the offerer MUST
accept
> >> >> complaint to the agreed or offered Level (or any lower Level),
it
> >is
> >> >> If the answerer requests a resolution of a Level that the
offerer
> >> >> cannot
> >> >> support, the offerer has to reject anyway as it cannot support
> the
> >> >> Level
> >> >> at the first step.
> >> >>
> >> >> BR, YK
> >> >>
> >> >> >> >sender do if it cannot send this resolution, should it
reject
> >> >> >> >> height of the spatial resolution, in pixels, separated by
a
> >> >> comma.

---

From: Thomas Schierl <[email-masked]>
Date: Mon, 14 Jul 2008 11:53:02 +0200

Hi Alex, Hi Ye-Kui,

To me, it seems really dangerous
to parse a packet with the same type depending on
out-of-band signalling and not depending on any information within the
packet.

What is about empty MTAP packets in case re-using MTAP16?

For the NI-C mode there would be a mixture of DON fields
present either in the PACSI or in the DON fields of the MTAP packet.

And last but not least, I do not understand why PACSI should become
explicitly mandatory? Do you mean for all modes including the NI-T
mode? They are already
mandatory
for NI-C and NI-TC mode since the DON fields are required! For NI-T
they are not required!

Thomas

Hi Alex,

Very good analysis and design
rules - I basically agree all what you said.

BR, YK

From:
[email-masked] [mailto:[email-masked]] On Behalf Of ext
Alex Eleftheriadis

Subject: Re: [AVT] NI-MTAP packet type in the SVC payload
draft

This also relates to an ommission in ver. 12 Section 6.10.1 (CS-DON)
where there is nothing specified for CS-DON and NI-MTAPs, as pointed
out by Mike. In the NI-TC multi-session transmission mode, we need to
specify how CS-DON number are calculated within NI-MTAPs. Similarly the
NI-C/NI-TC packetization (ver. 12 Section 7.2.1) rules need to specify
what happens in NI-MTAPs.  Finally, the PACSI section (6.8) is silent
about NI-MTAPs, and so is the NI-MTAP section about PACSI... In fact,
the PACSI section is also silent about MTAPs (it assumes there is just
one PACSI for the whole aggregation packet - how useful is that??).

In the ver. 12 design I presume the NI-MTAP NI-TC mode
packetization rule  would be to - at least - start
with a PACSI, as with STAP-As, include the DONC field in the NI-TC
mode, and derive CS-DON numbers as usual.

- NI-T

Remove DOND from MTAP16 aggregation unit, keeping NALU unit
size and TS_offset, and remove DONB from MTAP16 header. MTAP24 does not
appear useful here.

- NI-TC

Keep DOND in aggregation unit and DONB in MTAP16. Use MTAP's
DONB/DONDs for CS-DON, enforce ordering per decoding order, set PACSI
T=0, and have the PACSI DON (as specified by the MTAP header) be equal
to the CS-DON of the next non-PACSI NAL unit. The PACSI here is, in
fact optional.

It is strange to me that we still consider them optional. If
we made them mandatory this will streamline the packetization rules,
minimize the difference between the multi-session transmission modes to
just saying which fields in the PACSI and MTAPng (the new MTAP) are
present, and that's it. Plus ensure that scalability works in the
network as advertized.

Anyway. For the MTAPng concept, I am afraid we will have to
wait for the post-Dublin version to get it integrated if accepted.
Right now there is just too much to do to (*try to*) get a solid
version 13 by tomorrow' deadline.

--Alex

On Jul 12, 2008, at 11:58 PM, <[email-masked]>
wrote:

Hi Thomas,

You are right that the change
cannot be integrated into 3984bis
as it is not applicable to 'H246'. Therefore, if people agree the
change should only stay in the SVC draft.

BR, YK

From:
[email-masked] [mailto:[email-masked]] On
Behalf Of ext Thomas Schierl

Cc: [email-masked]

Subject: Re: [AVT] NI-MTAP packet type in the SVC
payload draft

Hi Ye-Kui,

Thomas

Alternative 1:

Alternative 2:

BR, YK

--

[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Mon, 14 Jul 2008 09:35:18 -0400

>the agreed (or agreeable) configuration, the other side must accept it. 

I would suggest instead (and did in the thread back in March): that as a
receiver property, it is the *preferred* maximum frame size.  The receiver
is required to decode and handle any size up to the size implied by the
level (and max-fs if given), but the receiver prefers this size (for
example, it may support 1920x1080, but only have an LCD display that can
display 1200x720, so there's no *need* to send it anything higher (and the
encoder may prefer to avoid encoding more macroblocks, especially in an MCU
that shares resources)).  In other cases, the sender may have the stream
already encoded in different resolutions, and this can help select the best
one to send to save encoding (and bandwidth) resources.

There is no requirement that the encoder scale, but if it wants to it
can.

It also (if I remember) provides aspect-ratio information.

---

From: <[email-masked]>
Date: Mon, 14 Jul 2008 20:06:09 +0300

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

	Title           : RTP payload format for G.718 speech/audio
	Author(s)       : Ari Lakaniemi, Ye-Kui Wang
	Filename        : draft-lakaniemi-avt-rtp-evbr-02.txt
	Pages           : 32
	Date            : 2008-07-14

This document specifies the Real-Time Transport Protocol (RTP) payload
format for the Embedded Variable Bit-Rate (EV-VBR) speech/audio codec,
specified in ITU-T G.718. A media type registration for this RTP payload
format is also included. 

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-lakaniemi-avt-rtp-evbr-02.txt

Comments are welcome!

BR, YK

---

From: <[email-masked]>
Date: Mon, 14 Jul 2008 20:10:58 +0300

Comments are welcome. See the last two sections for changes to RFC 3984
and open issues. 

http://www.ietf.org/internet-drafts/draft-wang-avt-rfc3984bis-01.txt

BR, YK

---

From: Dave Singer <[email-masked]>
Date: Mon, 14 Jul 2008 11:39:59 -0700

I think you mean pixel dimensions, don't you?  Resolution is 
something like 72 dots-per-inch.  Dimensions is something like 100 
pixels wide.

-- 
David Singer
Apple/QuickTime

---

From: <[email-masked]>
Date: Tue, 15 Jul 2008 00:17:18 +0300

Yes. People use different terms. Maybe we should we what you suggested,
"pixel dimensions", it is even clearer than "frame size".

BR, YK 

>From: ext Dave Singer [mailto:[email-masked]] 
>Sent: 14 July, 2008 21:40
>Subject: Re: [AVT] spatial-resolution parameter for RFC 3984bis
>
>suggested in
>resolution of
>maximum spatial

---

From: <[email-masked]>
Date: Tue, 15 Jul 2008 13:03:02 +0300

http://www.ietf.org/internet-drafts/draft-hannuksela-avt-rtp-svc-01.txt

Abstract 

   This memo describes two alternative methods for decoding order 
   recovery of the Network Abstraction Layer (NAL) units carried in 
   multiple RTP sessions for Scalable Video Coding (SVC), which is 
   defined in Annex G of the ITU-T Recommendation H.264 video codec that

   is technically identical to Amendment 3 of ISO/IEC International 
   Standard 14496-10.  The methods apply when non-interleaved 
   transmission of NAL units using the Single NAL Unit packetization 
   mode or the Non-Interleaved packetization mode defined in RFC 3984 is

   in use. 

Comments are welcome, as usual.

BR, YK

---

From: Alex Eleftheriadis <[email-masked]>
Date: Wed, 16 Jul 2008 03:34:15 +0300

On Jul 14, 2008, at 12:53 PM, Thomas Schierl wrote:

> Hi Alex, Hi Ye-Kui,
>
> To me, it seems really dangerous to parse a packet with the same  
> type depending on out-of-band signalling and not depending on any  
> information within the packet.
>

A very valid point.

> What is about empty MTAP packets in case re-using MTAP16?

I don't understand the question/comment.

> For the NI-C mode there would be a mixture of DON fields present  
> either in the PACSI or in the DON fields of the MTAP packet.
>

True. This is yet another example of mixing things in the current  
design. It is strange that the design of MTAPs was done exclusively  
for the Interleaved mode in 3984. It is also strange that one more NAL  
unit type was taken to expand the timestamp field, when a flag could  
do the job in the MTAP16 design.

I guess the NI-MTAP is the missing MTAP (MTAP16 with no DON) in the  
3984 Table 4. Maybe this time we should add a flag there (8-bit  
subtype) for future extensions. This way we can keep adding NAL unit  
subtypes at will. In actual fact PACSI could live in that space as well.

Is anyone btw currently using MTAPs in the field? I am not aware of  
any implementation. I think only single NAL units and STAP-As are  
used. Could this give us an opportunity to fix them in 3984bis?

> mode? They are already mandatory for NI-C and NI-TC mode since the  
> DON fields are required! For NI-T they are not required!
>

Yes, but remember though that an empty PACSI is one of the 3 ways to  
create "filler" NALs to do decoding order recovery in the NI-T mode,  
so they MAY be used in some coding cases as well. It's this arbitrary  
use/appearance of PACSIs that bothers me.

It just seems to me that running SVC without PACSI you are making it  
look like a single-layer codec at the transport level. In fact RFC3984  
suffices for its transport if we just added the new NAL unit types to  
its list and the Scalablity Information SEI message in the signaling.  
Maybe we should have a mode without PACSI at all (like NI-T but for  
single-session transmission only) and require PACSIs everywhere in all  
other modes (SST w/ PACSI + the 3 MST modes). This would satisfy also  
those who want to save the bits. Is there a use case for sending PACSI  
for some NAL units and not for others (except when used exclusively  
for decoding order recovery)?

At the very minimum we should enable multiple PACSIs for MTAPs and NI- 
MTAPs.

--Alex

>> Very good analysis and design rules - I basically agree all what  
>> you said.
>>
>> BR, YK
>>
>> Of ext Alex Eleftheriadis
>> This also relates to an ommission in ver. 12 Section 6.10.1 (CS- 
>> DON) where there is nothing specified for CS-DON and NI-MTAPs, as  
>> pointed out by Mike. In the NI-TC multi-session transmission mode,  
>> we need to specify how CS-DON number are calculated within NI- 
>> MTAPs. Similarly the NI-C/NI-TC packetization (ver. 12 Section  
>> 7.2.1) rules need to specify what happens in NI-MTAPs.  Finally,  
>> the PACSI section (6.8) is silent about NI-MTAPs, and so is the NI- 
>> MTAP section about PACSI... In fact, the PACSI section is also  
>> silent about MTAPs (it assumes there is just one PACSI for the  
>> whole aggregation packet - how useful is that??).
>>
>> packetization rule  would be to - at least - start with a PACSI, as  
>> with STAP-As, include the DONC field in the NI-TC mode, and derive  
>> CS-DON numbers as usual.
>>
>> DONDs for CS-DON, enforce ordering per decoding order, set PACSI  
>> T=0, and have the PACSI DON (as specified by the MTAP header) be  
>> equal to the CS-DON of the next non-PACSI NAL unit. The PACSI here  
>> is, in fact optional.
>>
>> #31.  However, this will become yet another example of how  
>> unbalanced the  current design is, in that PACSIs are required in  
>> NI-TC mode for all packet types when there is a NALU-time change,  
>> they must be the first uni in an STAP-A, and are not needed in this  
>> new MTAP. Here the PACSI is used as a DON mechanism. But PACSIs are  
>> supposed to be used to indicate layer information, not access  
>> units, so there should be several within an access unit... We allow  
>> empty PACSIs (for the NI-T mode). Yet PACSIs are, in general,  
>> optional... :-)
>>
>> minimize the difference between the multi-session transmission  
>> modes to just saying which fields in the PACSI and MTAPng (the new  
>> MTAP) are present, and that's it. Plus ensure that scalability  
>> works in the network as advertized.
>>
>> for the post-Dublin version to get it integrated if accepted. Right  
>> now there is just too much to do to (*try to*) get a solid version  
>> 13 by tomorrow' deadline.
>>
>> --Alex
>>

---

From: Thomas Schierl <[email-masked]>
Date: Wed, 16 Jul 2008 09:39:04 +0200

Hi Alex,

>> What is about empty MTAP packets in case re-using MTAP16?

The definition of empty MTAP16 packets would have to go in RFC3984.

> design. It is strange that the design of MTAPs was done exclusively for 
> the Interleaved mode in 3984. It is also strange that one more NAL unit 
> type was taken to expand the timestamp field, when a flag could do the 
> job in the MTAP16 design.

Yes, this is strange. Those days, nobody was thinking of extending H.264
payload. We should ask if anybody is using MTAP24 packets. If not, we
should free this packet type in RFC3984 and make the time stamp length
switchable. Btw, I have implemented MTAPs for interleaved and
prioritized transmission of H.264, but we never put that into a product.

> 
> I guess the NI-MTAP is the missing MTAP (MTAP16 with no DON) in the 3984 
> Table 4. Maybe this time we should add a flag there (8-bit subtype) for 
> future extensions. This way we can keep adding NAL unit subtypes at 
> will. In actual fact PACSI could live in that space as well. 

This sounds reasonable.

> 
> Is anyone btw currently using MTAPs in the field? I am not aware of any 
> implementation. I think only single NAL units and STAP-As are used. 
> Could this give us an opportunity to fix them in 3984bis? 

See comment above.

>> mode? They are already mandatory for NI-C and NI-TC mode since the DON 
>> fields are required! For NI-T they are not required!
>>  
> 
> create "filler" NALs to do decoding order recovery in the NI-T mode, so 
> they MAY be used in some coding cases as well. It's this arbitrary 

Then we remove the "empty PACSI case" and just allow empty NI-MTAPs and 
the "filler" NAL units already present in the bitstream.

> for some NAL units and not for others (except when used exclusively for 
> decoding order recovery)? 

I do not understand this! Why does it look as a single layer codec 
without PACSI???

Where is PACSI useful for NI-T? Let us assume you want to have a
knowledge about the importance of the content in the payload, this was
the original use case for PACSI. For STAPs or single NAL unit packets
you can just have a look into the first VCL NAL unit and since the NAL 
units have to be by definition in decoding order within STAP-A packets 
and the more important VCL-NAL units precede the less important ones in 
decoding order within the same access unit, I guess you will find the 
SVC header information you are looking for right in the first VCL NAL 
unit in the aggregation packet.
Sure, this is not vaild if there are non-VCL NAL units preceding
those VLC NAL units, but the MANE has to anyway have a deep look into
the payload, if not using MST. So, why not beside parsing all the packet 
types also having a look into NAL units in the payload? So, I guess 
there is not need for PACSI at all in the NI-T/MST mode.

Further, we could easily fix that non-VCLs excepting Prefix NAL units
are contained in their own packets in case PACSI is not used.

I further propose to add an SDP paramter which indicates that PACSI is 
not present in the session.

> 
> At the very minimum we should enable multiple PACSIs for MTAPs and 
> NI-MTAPs. 

Yes, if used.

Thomas

>>>     *From:* [email-masked] [mailto:[email-masked]] *On
>>>     Behalf Of *ext Alex Eleftheriadis
>>>     *Sent:* 14 July, 2008 05:50
>>>     *To:* Wang Ye-Kui (Nokia-NRC/Tampere)
>>>     *Cc:* [email-masked]; [email-masked]
>>>     *Subject:* Re: [AVT] NI-MTAP packet type in the SVC payload draft
>>>
>>>     (CS-DON) where there is nothing specified for CS-DON and
>>>     NI-MTAPs, as pointed out by Mike. In the NI-TC multi-session
>>>     transmission mode, we need to specify how CS-DON number are
>>>     calculated within NI-MTAPs. Similarly the NI-C/NI-TC
>>>     packetization (ver. 12 Section 7.2.1) rules need to specify what
>>>     happens in NI-MTAPs.  Finally, the PACSI section (6.8) is silent
>>>     about NI-MTAPs, and so is the NI-MTAP section about PACSI... In
>>>     fact, the PACSI section is also silent about MTAPs (it assumes
>>>     there is just one PACSI for the whole aggregation packet - how
>>>     useful is that??).
>>>
>>>     packetization rule  would be to - at least - start with a PACSI,
>>>     as with STAP-As, include the DONC field in the NI-TC mode, and
>>>     derive CS-DON numbers as usual. 
>>>
>>>     and TS_offset, and remove DONB from MTAP16 header. MTAP24 does
>>>     not appear useful here. 
>>>
>>>     - NI-TC
>>>
>>>     DONB/DONDs for CS-DON, enforce ordering per decoding order, set
>>>     PACSI T=0, and have the PACSI DON (as specified by the MTAP
>>>     header) be equal to the CS-DON of the next non-PACSI NAL unit.
>>>     The PACSI here is, in fact optional. 
>>>
>>>     they must be the first uni in an STAP-A, and are not needed in
>>>     this new MTAP. Here the PACSI is used as a DON mechanism. But
>>>     PACSIs are supposed to be used to indicate layer information, not
>>>     access units, so there should be several within an access unit...
>>>     We allow empty PACSIs (for the NI-T mode). Yet PACSIs are, in
>>>     general, optional... :-)  
>>>
>>>     modes to just saying which fields in the PACSI and MTAPng (the
>>>     new MTAP) are present, and that's it. Plus ensure that
>>>     scalability works in the network as advertized.
>>>
>>>     for the post-Dublin version to get it integrated if accepted.
>>>     On Jul 12, 2008, at 11:58 PM, <[email-masked]
>>>     <mailto:[email-masked]>> wrote:
>>>
>>>>     Hi Thomas,
>>>>      
>>>>     You are right that the change cannot be integrated into 3984bis
>>>>     as it is not applicable to 'H246'. Therefore, if people agree
>>>>         *From:* [email-masked] [mailto:[email-masked]]
>>>>         *On Behalf Of *ext Thomas Schierl
>>>>         *Sent:* 12 July, 2008 23:36
>>>>         *To:* Wang Ye-Kui (Nokia-NRC/Tampere)
>>>>         *Cc:* [email-masked] <mailto:[email-masked]>
>>>>         *Subject:* Re: [AVT] NI-MTAP packet type in the SVC payload
>>>>         draft
>>>>
>>>>         Hi Ye-Kui,
>>>>
>>>>         packets without DON (equal to NI-MTAP of the current SVC
>>>>         draft) would be integrated into RFC3984bis? I guess these
>>>>         packets are not usable with 'H264'.
>>>>
>>>>         Thomas
>>>>
>>>>     [email-masked] <mailto:[email-masked]>

---

From: Colin Perkins <[email-masked]>
Date: Wed, 16 Jul 2008 18:06:08 +0100

> with RFC 3550/layered multicast, see, e.g., Section 8.3).

The rules for single SSRC use in RFC 3550 section 8.3 were defined on  
the assumption that layered codecs consist of a monotonic set of  
layers, with every participant always being subscribed to a single  
base layer. If that assumption is violated, then SSRC collisions can  
be missed, and the rules don't work (one must then treat each layer  
independently, using different SSRCs and synchronising using the RTCP  
CNAME).

The current SVC codec does seem to fit the assumptions, so the single  
SSRC rules will work. Can we guarantee that will always be the case  
with future versions of the codec? What about other H.264 extensions,  
such as MVC, and its future variants? What about other layered or  
multi-view codecs, such as EVBR audio?

If we're going to have RTP payload formats for several different  
layered and multi-view codecs, then we should try to ensure they all  
handle the SSRCs in the same way, otherwise implementation and  
specification complexity is going to get out of hand. Using the  
single SSRC rules from RFC 3550 section 8.3 is a short-term win on  
complexity for SVC, but limits what we can do for future codecs. Are  
we comfortable with that limitation?

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Wed, 16 Jul 2008 23:08:24 +0300

SVC allows flexible layer dependency. For example, if there three
layers, l0, l1 and l2. Then it is possible that l2 does not depend on
l1. It is also possible that all the three layers are independently
coded. I guess this does not fit the "monotonic" requirement.

Similar flexibility is allowed in MVC too.

EVBR has a simple layer dependency, layer N+1 in the bitstream always
directly depend on the next lower layer (layer N).

BR, YK 

>Behalf Of ext Colin Perkins
>Sent: 16 July, 2008 20:06
>To: Alex Eleftheriadis
>Cc: avt
>transmission forSVC payload draft
>

---

From: Alex Eleftheriadis <[email-masked]>
Date: Thu, 17 Jul 2008 12:41:13 +0300

Ye-Kui, you did not make a comment though on if single SSRC or  
multiple SSRCs should be used.

Although SVC does allow considerable flexibility in inter-layer  
prediction, making some layers independently decodable, this does not  
mean that from a presentation point of view you don't need the base  
layer (e.g, in temporal scalability). Since coding efficiency is very  
poor without inter-layer prediction, it's hard to imagine use cases  
for this (that make engineering sense). Do we have any?

I don't know enough about MVC to comment. Speaking about future  
codecs, multiple-desciption coding clearly does not fit the model, but  
I am not sure if there is any probability of seeing one in this forum  
for video.

I don't have a strong opinion against or for; I just want the matter  
settled so that we can finish the draft (with simplicity at the top of  
the list of desirable characteristics).

PS: Note that the requirement is not monotonic layer dependency, but  
that all layers have a common "core (base)" to get the RTCP BYE, a  
weaker constraint.

--Alex

---

From: Randell Jesup <[email-masked]>
Date: Mon, 21 Jul 2008 11:04:05 -0400

>independent video contents.

Note: currently, RFC 3984 requires ("MUST") that the answer have paramset
#0 included in paramset #1:

 o  The "sprop-parameter-sets" parameter is used as described above.
      In addition, an answerer MUST maintain all parameter sets received
      in the offer in its answer.  Depending on the value of the
      "parameter-add" parameter, different rules apply: If "parameter-
      add" is false (0), the answer MUST NOT add any additional
      parameter sets.  If "parameter-add" is true (1), the answerer, in
      its answer, MAY add additional parameter sets to the "sprop-
      parameter-sets" parameter.  The answerer MUST also, independent of
      the value of "parameter-add", accept to receive a video stream
      using the sprop-parameter-sets it declared in the answer.

         Informative note: care must be taken when parameter sets are
         added not to cause overwriting of already transmitted parameter
         sets by using conflicting parameter set identifiers.

So, paramset #1 is always identical or a superset of paramset #0.
That doesn't mean that the answerer has to use parameter indexes from
paramset #0 (it could just use parameter sets added in #1, unless
parameter-add was 0).

I understand why you want to make them independent, and I don't understand
why the text above was put into 3984 - but it was.  (Perhaps to make "early
media" possible, before an answer had been received and processed by the
offerer (for example, in SIP the answer (200 OK or provisional 183 w/ SDP)
might be lost or delayed, and thus media from the answerer may arrive ahead
of it)).

>a=fmtp:98 profile-level-id=42A01E; //Baseline profile, Level 3.0

There are some sub-cases here, where one side provides parameter sets and
the other side doesn't.  My suggestion is that if you (as the offerer)
provide paramsets and the answer doesn't, assume they were NOT installed
and send them in-band.  When answering an offer with no paramsets, you can
include them or not, but don't assume they'll be installed if you send them
- send in-band paramsets as well.

>  sprop-parameter-sets=<base-64 data#1>

Note that Answer SDP #3 doesn't *have* to be identical to SDP#1, and
for a number or reasons may well not be - especially if SDP#2 isn't
identical to SDP#1.

This case needs to list what happens in-between answer SDP#1 and offer SDP#2
and answer SDP#3.  Who can send media, what does it look like, etc.

Also: a second round of offer/answer would not be *needed* (though is
allowed of course) if the original offer included a parameter-set for level
2.0.  However, that would require the answerer to be able to look into the
paramsets and figure out if there are acceptable ones there (and install
those).  We may not want the negotiation code to require that level of
knowledge/communication with the H.264 decoder.  In some unusual cases (raw
NAL message recording, decomposed receive) there may be no decoder
available to query, so smarts would have to be added to the negotiator.

>a=fmtp:98 profile-level-id=42A014; //Baseline profile, Level 2.0

Same comments as above about in-band.

>
>>>A few points:
>>>- The Level in profile-level-id is the maximum level the 
>>>offerer offers, all lower levels are automatically supported. Therefore,
>>>this does mean offering of different levels.
>>
>>Yes, but your comment was that sprop-parameter-sets needs to be
>>"compliant" with profile-level-id.  As mentioned at IETF in Philly, if
>>you have to provide parameter sets for every possible level at or below
>>the one in profile-level-id, the set (and SDP) could get very large.  If
>>you don't provide them, then it's problematic for the answerer to
>>downgrade the level unless you either do a second offer/answer, or send
>>them in-band (with the attendent possible-loss issues).
>

Right.  We need to realize that we are buying into a virtually required
second exchange on level downgrade.  This should be run past
offer/answer and mmusic folks for comment!

>be equal to 1 or not present).

Note that some devices currently include a parameter-add=0.  The revised
spec should at least speak to that.  (We include it, but if it's ignored we'll
handle that, so we're ok.  I'm not sure anyone else is using it.  Is
anyone?)

>>>>suggest both sides will need to use in-band paramsets, since you can't
>>>>be sure the person who answered without paramsets will use the ones in
>>>>the SDP.  If
>>>
>>send them realizing that they may be lost.  Any actual change in
>>parameter sets may take a while or even forever to be received,
>>especially if there's no working RTCP (CCM/etc) back-channel to report
>>loss of sync.
>

For reference (people commenting on this should re-read 8.4 anyways):

   8.4
   ...
   If in-band signaling of parameter sets is used, the sender SHOULD take
   the error characteristics into account and use mechanisms to provide a
   high probability for delivering the parameter sets correctly.
   Mechanisms that increase the probability for a correct reception include
   packet repetition, FEC, and retransmission.

   ...
   -  Adding or modifying parameter sets by using both principles B and
      C in the same RTP session may lead to inconsistencies of the
      parameter sets because of the lack of synchronization between the
      control and the RTP channel.  Therefore, principles B and C MUST
      NOT both be used in the same session unless sufficient
      synchronization can be provided.

(B is out-of-band update, reliable, and C is in-band (normally
unreliable).  A is out-of-band before the call, reliable.)

   In some scenarios (e.g., when only the subset of this payload format
   specification corresponding to H.241 is used), it is not possible to
   employ out-of-band parameter set transmission.  In this case,
   parameter sets have to be transmitted in-band.  Here, the
   synchronization with the non-parameter-set-data in the bitstream is
   implicit, but the possibility of a loss has to be taken into account.
   The loss probability should be reduced using the mechanisms discussed
   above.

>From 8.4:
   To allow for maximum flexibility and best performance from the H.264
   coder, it is recommended, if possible, to allow any sender to add its
   own parameter sets to be used in a session.  Setting the "parameter-
   add" parameter to false should only be done in cases where the
   session topology prevents a participant to add its own parameter
   sets.

This implies to me that there are topologies (multicast?  large
conferences?  Translators?  It isn't clear) where it is *NOT* reasonable to
have each sender use their own preferred parameter set.  We need to know
what those topologies are, and why parameter-add (and that paragraph) were
added in the first place.

>answerer should accept the parameter sets originally offered. 

I meant that on level downgrade, the answerer *can't* accept the parameter
sets in the offer that use a higher level.

>a correct reception include packet repetition, FEC, and retransmission."

Actually, I was referring to this:

   It is necessary to implement principles A and B within a session
   control protocol.  SIP and SDP can be used as described in the SDP
   Offer/Answer model and in the previous sections of this memo.  This
   section contains guidelines on how principles A and B must be
   implemented within session control protocols.  It is independent of
   the particular protocol used.  Principle C is supported by the RTP
   payload format defined in this specification.

   The picture and sequence parameter set NALUs SHOULD NOT be
   transmitted in the RTP payload unless reliable transport is provided
   for RTP, as a loss of a parameter set of either type will likely
   prevent decoding of a considerable portion of the corresponding RTP
   stream.  Thus, the transmission of parameter sets using a reliable
   session control protocol (i.e., usage of principle A or B above) is
   RECOMMENDED.

The language is "SHOULD NOT unless reliable transport is used for RTP"
(paraphrased).   Agreed, it does not say "MUST NOT" (and I think in
H.241 that A and B aren't available, so C must be used).

>>is to (re)send paramsets on every IDR, which (in the common
>>packetization-mode 0) yields a bunch of extra packets.  The point of
>>sprop-parameter-sets was to avoid the need to do so, and also to avoid
>>"unlucky" stream-startup losses from delaying decoding of the stream.
>

Unfortunately, this is common practice - and while I've never read H.241,
my understanding is that it doesn't allow for out-of-band parameter sets at
all (which is probably why so many videoconferencing endpoints don't
implement sprop-parameter-sets).

>>My analysis below was mostly about how level-downgrade would work (or not
>>work) using the rules you gave.  Also, offer-answer usually discourages a
>>second round, though ICE uses a second round to lock down the transport,

---

From: "Even, Roni" <[email-masked]>
Date: Mon, 21 Jul 2008 19:15:24 +0300

Hi,
One of the issues with the parameter set has to do with multipoint voice
activated video switch. In this scenario the MCU switch the video
source. Since the switch may occur when the receivers do not have a
synchronization point (Both IDR picture and parameter sets) in H.323 we
do not send parameter sets out of band and require that the fast update
will cause the sending of the IDR picture and the parameter sets.
Currently in my view RFC 3984 is broken in this case. There is no
requirement to send the parameter sets for fast update. Currently all
conference participants may use the same parameter sets numbers which
will make it difficult on the receiver to keep parameter sets with the
same numbers without knowing how to correlate them to a specific source,
this can be addressed by having the MCU assign the parameter sets
numbers or by relating the parameter sets in the SDP to the SSRC or
doing like H.241 requiring the sending of parameter sets with IDR on
fast update (In this case video conferencing will not need the sprop
support)

Roni Even

> Randell Jesup
> Sent: Monday, July 21, 2008 6:04 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [AVT] SDP offer/answer for the 3984bis and SVC payload
> drafts
> 
> >not need to be stored by the answerer. The
sprop-parameter-sets=<base-
> 64
> >>>>be sure the person who answered without paramsets will use the
ones
>    the error characteristics into account and use mechanisms to
provide
> >>My memory was that 3984 basically said "don't do it", but maybe
there
> >Even out-of-band transmission of parameter sets is not used, it is
not
> provided for defence against real, pretended, or imaginary dangers
from
> abroad."

---

From: Randell Jesup <[email-masked]>
Date: Mon, 21 Jul 2008 12:50:47 -0400

"Even, Roni" <[email-masked]> writes:
>requirement to send the parameter sets for fast update.

In this case, the MCU is acting as a video multiplexer - forwarding but not
modifying the streams.  Recipients should see the SSRC (and source/SDES)
change on a switch by the MCU.  If the new source can have conflicting
parameter sets in what it sends, then the recipient MUST NOT try to decode
the new video source until it has the correct paramsets installed.  This
may be why they suggest avoiding potential conflicting paramsets.

Note that in-band doesn't help here (since any in-band packet can be lost,
even with FEC and retransmission, etc) unless the decoder throws away all
parameter sets on any source change, and goes black/static until it gets a
set of paramsets and an IDR.  If the paramsets are *known* to be compatible
(which is probably the point of paramater-add=0), then it can start
decoding immediately (if it doesn't mind odd "predator-mode" results), and
any IDR (without paramsets) or other form of refresh will fix the video.

>Currently all conference participants may use the same parameter sets
>numbers which will make it difficult on the receiver to keep parameter
>sets with the same numbers without knowing how to correlate them to a
>specific source, this can be addressed by having the MCU assign the
>parameter sets numbers or by relating the parameter sets in the SDP to the
>SSRC or doing like H.241 requiring the sending of parameter sets with IDR
>on fast update (In this case video conferencing will not need the sprop
>support)

If we define lack of parameter sets to mean in-band is required for both
sides (and live with possible loss), then this use case/topology could use
that (by having the MCU/multiplexer use SDP's without
sprop-parameter-sets).  In the case of a "dumb" multiplexer, which isn't
modifying or re-encoding the streams, it's hard to support anything else,
unless the MCU insists on a parameter-set (via parameter-add=0, and using
re-INVITE if the initial invite came from the endpoint).

What about other topologies?  My apologies; I didn't really follow the
topology draft during the discussion of CCM.

---

From: "Even, Roni" <[email-masked]>
Date: Tue, 22 Jul 2008 10:07:54 +0300

Randell,
The problem is that the endpoint has signaling connection only to the
MCU, and if he has to forward the sprop parameter sets in the signaling
there is no relation to the SSRC.
Example, the MCU establish connections with three EPs and get their
sprop parameter sets in the offer, all of them use same numbers. Now
what does it send back in the answer (Change the numbers? Add relation
to SSRC? All this is not defined). Does it mean that before doing video
switch the MCU needs to do a re-invite to send new parameter sets.

Roni

> From: Randell Jesup [mailto:[email-masked]]
> Sent: Monday, July 21, 2008 7:51 PM
> To: Even, Roni
> Cc: [email-masked]; [email-masked]
> In this case, the MCU is acting as a video multiplexer - forwarding
but
> >parameter sets numbers or by relating the parameter sets in the SDP
to
from

---

From: Colin Perkins <[email-masked]>
Date: Sun, 27 Jul 2008 16:41:15 +0100

> http://www.ietf.org/internet-drafts/draft-lakaniemi-avt-rtp- 

This generally looks fine. The use of an RTCP APP packet for feedback  
has previously been flagged as a problem.

I notice that section 3.6 defines a new mechanism for clock  
synchronisation between sessions, claiming that synchronisation based  
on the RTCP SR is too slow. It's not clear that's true, especially if  
RTP/AVPF is used, but if it is, then a general mechanism should be  
defined for rapid synchronisation, not something payload format  
specific.

Section 3.3 would benefit from a reference to RFC 5117, and some  
discussion that the middleboxes considered are RTP translators which  
must follow the rules in RFC 3550 section 7.

Regards,
-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Sun, 27 Jul 2008 23:59:43 +0300

Hi Colin,

Thanks for your review and comments. 

>
>This generally looks fine. The use of an RTCP APP packet for 
>feedback has previously been flagged as a problem.
>

Yes. As it reads, the text existing in the draft serves for stating what
types of information may be needed. As at this meeting it was decided
(by you chairs) not to fully present this but have one slide in chairs'
update, I will start a discussion regarding this after this meeting. 

>synchronisation between sessions, claiming that 
>synchronisation based on the RTCP SR is too slow. It's not 
>clear that's true, especially if RTP/AVPF is used, but if it 
>is, then a general mechanism should be defined for rapid 
>synchronisation, not something payload format specific.
>

Agreed - as the same has been discussed in the SVC context. Will check
earlier dicussions and study whether anything in a general level is
needed. 

>Section 3.3 would benefit from a reference to RFC 5117, and 
>some discussion that the middleboxes considered are RTP 
>translators which must follow the rules in RFC 3550 section 7.
>

Thanks - will include in the next version. 

BR, YK

---

From: <[email-masked]>
Date: Mon, 28 Jul 2008 14:02:50 +0300

Hi Roni,

Here is my understanding regarding MCU switching of the video bitstream
for a receiver between different sources. This may well have been
discussed earlier - please forgive me it that is the case. Let's use
Fig. 7 in RFC 5117 for example.

  Shortcut name: Topo-Video-switch-MCU

         +---+      +------------+      +---+
         | A |------| Multipoint |------| B |
         +---+      |  Control   |      +---+
                    |   Unit     |
         +---+      |   (MCU)    |      +---+
         | C |------|            |------| D |
         +---+      +------------+      +---+

      Figure 7 - Point to Multipoint Using a Video Switching MCU

Assume that each endpoint may receive the video bitstream from one of
any other endpoints, switched by the MCU, and out-of-band transmission
of parameter sets is used. 

In the negotiation process, the MCU carries out separate SDP
offer/answer processes with A, B, C and D. Even though there is no
source specific informtation in SDP messages during the SDP offer/answer
processes, the MCU should still be able to know from whom it has
received and SDP (I guess SIP, e.g example, should provide this
funcationality). Therefore, the MCU stores the different copies of SDP,
including parameter sets, from different endpoints and marks them with
the assoicated endpoints.

The answer to the following question decides whether it is possible to
do out-out-band transmission of parameter sets (without caring of ID
value conflict across endpoints) with the above topology: Is it possible
for one endpoint (e.g. endpoint B) to know which other endpoint (e.g.
endpoint A) an SDP parameter (e.g. sprop-parameter-sets) is from, per
existing standards? For example, if the endpoints use different payload
types, then the answer is yes. Is there any other means (e.g. through
SIP) to know this (with existing standards)?

If the answer is yes to the first question in the above paragraph, then
each endpoint stores all different copies of parameter sets associated
with different endpoint points. For example, endpoint A stores all
copies of parameter sets associated with A, B, C and D, respectively.
The set of parameter sets assoicated with A is used for encoding by
endpoint A. The set of parameter sets assoicated with B is used to
decode incoming bitstreams orignated from B. As source switching can
only happen at IDR picture positions, the receiver implementation of A
can simply feed all the parameter sets assoicated with B to the decocer
of A at the first access unit a switching from any other endpoint to
endpoint B occurs.

At least from the video coding standard point of view, I see no problem
with the above process. Please comment what is the answer to the
important question above. 

If the answer is clearly no, then we need to find out another way to
improve this aspect in RFC 3984.

BR, YK

>Sent: Tuesday, July 22, 2008 10:08 AM
>To: Randell Jesup
>Cc: Wang Ye-Kui (Nokia-NRC/Tampere); [email-masked]
>Subject: RE: [AVT] SDP offer/answer for the 3984bis and SVC
>payload drafts
>
>Randell,
>The problem is that the endpoint has signaling connection only
>to the MCU, and if he has to forward the sprop parameter sets
>in the signaling there is no relation to the SSRC.
>Example, the MCU establish connections with three EPs and get
>their sprop parameter sets in the offer, all of them use same
>numbers. Now what does it send back in the answer (Change the
>numbers? Add relation to SSRC? All this is not defined). Does
>it mean that before doing video switch the MCU needs to do a
>re-invite to send new parameter sets.
>
>Roni
>
>> parameter sets in what it sends, then the recipient MUST NOT try to
>> decode the new video source until it has the correct paramsets
>> installed.
>> This
>> Note that in-band doesn't help here (since any in-band packet can be
>> lost, even with FEC and retransmission, etc) unless the
>decoder throws
>> away all parameter sets on any source change, and goes black/static
>> until it gets a set of paramsets and an IDR.  If the paramsets are
>> *known* to be compatible (which is probably the point of
>> paramater-add=0), then it can start decoding immediately (if it
>> doesn't mind odd "predator-mode" results), and any IDR (without
>> paramsets) or other form of refresh will fix the video.
>>
>> >SSRC or doing like H.241 requiring the sending of parameter
>sets with
>> IDR
>> If we define lack of parameter sets to mean in-band is required for
>> both sides (and live with possible loss), then this use
>case/topology
>> could use that (by having the MCU/multiplexer use SDP's without
>> sprop-parameter-sets).  In the case of a "dumb" multiplexer, which
>> isn't modifying or re-encoding the streams, it's hard to support
>> anything else, unless the MCU insists on a parameter-set (via
>> parameter-add=0, and using re-INVITE if the initial invite came from
>> the endpoint).
>>
>> Randell Jesup, Worldgate (developers of the Ojo videophone),
>ex-Amiga
>> OS team [email-masked] "The fetters imposed on liberty at
>home have
>> ever been forged out of the weapons provided for defence
>against real,
>> pretended, or imaginary dangers
>from
>> abroad."