Subject: [AVT] AVC NALUs RTP packetisation

From: Nikola Sprljan <[email-masked]>
Date: Tue, 27 Mar 2007 17:33:13 +0100

Hi all,

I have a question related to RTP packetisation of the AVC NAL units.
I consider scenario where pre-encoded content is being packetised, and
where AVC NAL units may be larger than MTU permits.

In my understanding, the only way to signal SVC parameters, i.e.,
exactly which layer a particular AVC NALU belongs to, is to send this
AVC NALU in an aggregation unit, together with another NALU conveying
these parameters - either a PACSI NALU or the corresponding suffix NALU.
Does that imply that AVC NALUs cannot be fragmented at the application
layer at all and that fragmentation will be done only at the IP
layer? In that case, what to do in the case when an AVC NALU is larger
than the IPv4 packet limit (64kB)? I understand that this is not likely
to happen, but it should be covered if permitted by the specifications.

I am aware that this question has been already raised:
[http://www1.ietf.org/mail-archive/web/avt/current/msg07009.html], and
the given answer
[http://www1.ietf.org/mail-archive/web/avt/current/msg07011.html] just
confirmed that there was no solution at that moment. Looking at the
latest draft, it seems that's still the case?

Please, could someone comment on this - is this still planned to be
resolved in the future versions of the draft?

Nikola Sprljan

-- 
*Dr. Nikola Sprljan*
/Research Engineer/
*MITSUBISHI ELECTRIC INFORMATION TECHNOLOGY CENTRE EUROPE B.V*
*VISUAL INFORMATION LABORATORY*
20, Frederick Sanger Road
The Surrey Research Park
Guildford, Surrey GU2 7YD
/UK Registered Branch BR 003158/
*DDI Telephone: +44 1483 885538*
Tel: +44 1483 885800   Fax: +44 1483 579107

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Randell Jesup <[email-masked]>
Date: Tue, 27 Mar 2007 13:34:51 -0400

>to happen, but it should be covered if permitted by the specifications.

I'm not sure how this interacts with SVC (since I haven't followed it), but
in non-SVC use you'd use packetization-mode 1, which allows fragmentation
(via FU-A)'s.  You may also be able to use mode 2 and FU-A's and FU-B's.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: <[email-masked]>
Date: Tue, 27 Mar 2007 23:14:08 +0300

In SVC context, it is also possible to use FU's as Randell said below,
although it is not possible to include the PACSI or suffix or prefix
(newly introduced to the SVC draft in January) NALU in packets
containing FU's. Currently there is no plan to support containing of
additional NALU in FU packets. The main reason is still that there is no
compelling use case requiring the support. 

In case one NALU is larger than MTU size but smaller than 64kB, the
packetizer can use FU's to avoid lower network layer fragmentation.
Alternatively, aggregation packet containing the SVC specific parameters
can be used. In this case, after fragmentation is done at lower network
layer, the SVC specific parameters would be present in the first
fragmentation, which is most important among all the fragments
originated from the aggregation packet. 

BR, YK

>-----Original Message-----
>From: ext Randell Jesup [mailto:[email-masked]] 
>Sent: Tuesday, March 27, 2007 8:35 PM
>To: Nikola Sprljan
>Cc: [email-masked]
>Subject: Re: [AVT] AVC NALUs RTP packetisation
>
>>layer at all and that fragmentation will be done only at the 
>IP layer? 
>>In that case, what to do in the case when an AVC NALU is larger than 
>>the IPv4 packet limit (64kB)? I understand that this is not likely to 
>>happen, but it should be covered if permitted by the specifications.
>
>I'm not sure how this interacts with SVC (since I haven't 
>followed it), but in non-SVC use you'd use packetization-mode 
>1, which allows fragmentation (via FU-A)'s.  You may also be 
>able to use mode 2 and FU-A's and FU-B's.
>
>--
>Randell Jesup, Worldgate (developers of the Ojo videophone), 
>ex-Amiga OS team [email-masked] "The fetters imposed on 
>liberty at home have ever been forged out of the weapons

---

From: Nikola Sprljan <[email-masked]>
Date: Wed, 28 Mar 2007 11:38:29 +0100

Ye-Kui,

Thank you for your reply. Please find my comments inline.

[email-masked] wrote:
> containing FU's. 

I see now that I omitted something that made my question
imprecise/incorrect - when I said "AVC NALUs cannot be fragmented at the
application layer", I actually wanted to say "AVC NALUs *aggregated* in
a packet together with a NALU conveying the SVC-specific parameters
cannot be fragmented above the UDP/IP level". It just seems to me that
AVC NALUs that are not in aggregation units are not very useful in SVC
context, and that this implies that fragmentation at the RTP level is
never going to be used for AVC NALUs in SVC context. Is that correct?

> Currently there is no plan to support containing of

Since I think there is sometimes a confusion with what is done at which
communication level, I will summarise briefly my understanding here:
*Application level* - deals with slices; if FMO is used, SVC encoder
sets the maximum byte length limit for slices.
*RTP level* - deals with NALUs; NALU packetisation modes are used, i.e.,
selection of fragmentation and aggregation modes. Fragmentation modes
must be used for slices larger than 64kB to divide them into segments
(FUs) that will fit into packets sent over IPv4.
*UDP/IP (or any lower than RTP) level* - deals with RTP packets;
fragmentation of RTP packets that exceed the MTU limit.

Also, maybe it is worth noting that although I say I need "SVC-specific
parameters for AVC NALUs", actually I only need the information on
temporal level - to be able to decide whether to forward a packet
containing an AVC NALU to a particular client. And using a PACSI or a
suffix NALU seems to me is the only way to achieve that.

Since I consider pre-encoded content here, the cases for packetisation
of AVC NALUs in SVC context are then the following:

1. aggregated AVC NALU + SVC-specific parameters <=MTU,
including overheads -> Can be put into an aggregation NALU without any
fragmentation.
2. aggregated AVC NALU + SVC-specific parameters >MTU and <=64kB,
including overheads -> Can be put into an aggregation NALU, with
fragmentation done on UDP/IP level.
3. aggregated AVC NALU + SVC-specific parameters >64kB,
including overheads -> Since this has to be fragmented at the RTP level,
SVC parameters cannot be sent and therefore this pre-encoded bit-stream
is not supported for sending over IPv4.

I guess when you say "there is no compelling use case", that actually
means that the case 3. is not likely to happen and is thus is not
supported? If that is so, should it be explicitly stated in the
packetisation rules?

> packetizer can use FU's to avoid lower network layer fragmentation.

Only for SVC NALUs, since for AVC NALUs I must use aggregation units.

> originated from the aggregation packet. 

I think this covers only case 2.

Best regards,

Nikola

---

From: <[email-masked]>
Date: Thu, 29 Mar 2007 00:30:20 +0300

Hi Nikola,

Thanks for your comments. I basically agree with those that I did not
response in below. Please see inline. 

>From: ext Nikola Sprljan [mailto:[email-masked]] 
>Sent: Wednesday, March 28, 2007 1:38 PM
>To: Wang Ye-Kui (Nokia-NRC/Tampere)
>Cc: [email-masked]; [email-masked]
>imprecise/incorrect - when I said "AVC NALUs cannot be 
>fragmented at the application layer", I actually wanted to say 
>"AVC NALUs *aggregated* in a packet together with a NALU 
>conveying the SVC-specific parameters cannot be fragmented 
>above the UDP/IP level". It just seems to me that AVC NALUs 
>that are not in aggregation units are not very useful in SVC 
>context, and that this implies that fragmentation at the RTP 
>level is never going to be used for AVC NALUs in SVC context. 
>Is that correct?
>

AVC NALUs are useful and actually important in SVC context. Note that
"AVC NALUs in SVC context" does not mean that each RTP packet containing
an AVC NALU or part thereof must contain the SVC-specific parameters.
See further discussions of the three cases in below.

>additional NALU in 
>> FU packets. The main reason is still that there is no compelling use 
>> case requiring the support.
>
>Since I think there is sometimes a confusion with what is done 
>at which communication level, I will summarise briefly my 
>understanding here:
>*Application level* - deals with slices; if FMO is used, SVC 
>encoder sets the maximum byte length limit for slices.

The encoder can always sets the maximum byte size limit for slices,
regardless of whether FMO is in use. 

>*RTP level* - deals with NALUs; NALU packetisation modes are 
>used, i.e., selection of fragmentation and aggregation modes. 
>Fragmentation modes must be used for slices larger than 64kB 
>to divide them into segments
>*UDP/IP (or any lower than RTP) level* - deals with RTP 
>packets; fragmentation of RTP packets that exceed the MTU limit.
>
>Also, maybe it is worth noting that although I say I need 
>"SVC-specific parameters for AVC NALUs", actually I only need 
>the information on temporal level - to be able to decide 
>whether to forward a packet containing an AVC NALU to a 
>particular client. And using a PACSI or a suffix NALU seems to 
>me is the only way to achieve that.
>
>Since I consider pre-encoded content here, the cases for 
>packetisation of AVC NALUs in SVC context are then the following:
>
>including overheads -> Can be put into an aggregation NALU 
>without any fragmentation.
>2. aggregated AVC NALU + SVC-specific parameters >MTU and 
><=64kB, including overheads -> Can be put into an aggregation 
>NALU, with fragmentation done on UDP/IP level.
>including overheads -> Since this has to be fragmented at the 
>RTP level, SVC parameters cannot be sent and therefore this 
>pre-encoded bit-stream is not supported for sending over IPv4.
>

Cases 1 and 2 are correct. For case 3, the SVC specific parameters (e.g.
in the prefix NAL unit) can be sent in their own RTP packet, and the big
AVC NALU, fragmented in RTP level, is sent in the subsequent RTP
packets. Yes, there will be no information of temporal_level or
priority_id in these packets. If a MANE would like to do something
intelligent to these packets, it can buffer the previous packet and
obtain the required information.

>I guess when you say "there is no compelling use case", that 
>actually means that the case 3. is not likely to happen and is 
>thus is not supported? If that is so, should it be explicitly 
>stated in the packetisation rules?
>

This is part of the reason. Another part is that a MANE can buffer up to
a couple of RTP packets to get required information for some
not-very-common cases. The reason why we did not put such things to the
packetization rules was, we think handling of case 3 above, for example,
seems to be obvious to people who would implement RTP packetizer. But I
think it would not hurt if we do. So text proposals are welcome. 

BR, YK

>> Alternatively, aggregation packet containing the SVC specific 
>> parameters can be used. In this case, after fragmentation is done at 
>> lower network layer, the SVC specific parameters would be present in 
>> the first fragmentation, which is most important among all the 
>> fragments originated from the aggregation packet.
>
>>>> exactly which layer a particular AVC NALU belongs to, is to send 
>>>> this AVC NALU in an aggregation unit, together with another NALU 
>>>> conveying these parameters - either a PACSI NALU or the 
>>>> corresponding
>>> suffix NALU.
>>>> the IPv4 packet limit (64kB)? I understand that this is not likely 
>>> I'm not sure how this interacts with SVC (since I haven't followed 
>>> it), but in non-SVC use you'd use packetization-mode 1, 
>which allows 
>>> fragmentation (via FU-A)'s.  You may also be able to use mode 2 and 
>>> FU-A's and FU-B's.
>>>
>>> --
>>> Randell Jesup, Worldgate (developers of the Ojo 
>videophone), ex-Amiga 
>>> OS team [email-masked] "The fetters imposed on liberty at 
>home have 
>>> ever been forged out of the weapons provided for defence against 
>>> real, pretended, or imaginary dangers from abroad."

---

From: Nikola Sprljan <[email-masked]>
Date: Fri, 30 Mar 2007 11:38:23 +0100

Ye-Kui,

Thank you for replying - I see now that a solution would be to buffer up
the RTP packets and keep them in a MANE until they are not needed any
more - the decision on forwarding/dropping is postponed until more
information on this packet becomes available. I was actually trying to
avoid solutions based on buffering at the RTP level as my intention was
to perform adaptation solely on a per-RTP packet basis, with all SVC
layers transmitted in a single RTP session.

One could argue that buffering is required at least for the case when
RTP packets are carrying fragmentation units, so they are not being
forwarded until it is made sure that all the fragments originating from
the same NALU have been received (I am not considering FGS here). But if
we are already at this level of complexity, and since a MANE can be made
aware of inter-layer dependencies, then why not observing decoding
dependencies between all received packets (i.e. their decoding
dependency to the lost packets) when deciding which ones can be safely
dropped? What I am trying say is - since MANEs will potentially have to
deal with lots of incoming streams, I would prefer having an option of
just forwarding or dropping the required packets, without employing any
"smart" repacketisation / thinning / retransmission techniques (I don't
consider parsing and reassembling the aggregation units to be "smart"),
and let the next node in the transmission chain deals with packet
losses. This could be seen as some kind of a "light" mode of MANE's
operation.

Basically, if one wants to avoid buffering packets in a MANE, the
following has to be ensured:
-fragmentation is not used at the RTP level;
-aggregation units are used for packetisation of AVC NALUs, along with
NALUs containing SVC-specific parameters in the same RTP packet.
However, since there always exists a possibility that I will encounter
some pre-encoded content that has slices larger than 64kB, as a
consequence I need to support FUs and implement buffering in the MANE.

Now, I know this has been discussed many times, but I am still of an
opinion that SSRC multiplexing could be helpful - especially in this
"no-buffer" scenario, as it would enable immediate forwarding/dropping
of RTP packets that would otherwise need to be buffered. Perhaps I am
just leaning towards having something close to a "Magic Device from
Heaven", but I also have a feeling that this could be useful for others
as well - e.g. it's still mentioned as a possibility in the current
version of draft-lennox-mmusic-sdp-source-attributes.

I guess what I'd like to ask this time is - is SSRC multiplexing gone
for good, or there's still a chance it could be reconsidered?

Best regards,

Nikola

>> I think this covers only case 2.
[cut]