Subject: RE: [AVT] I-D ACTION:draft-ietf-avt-rtp-svc-03.txt

From: <[email-masked]>
Date: Sun, 2 Dec 2007 03:18:47 +0200

See my comments inline. 

BR, YK

>-----Original Message-----
>From: ext Thomas Wiegand [mailto:[email-masked]] 
>Sent: Friday, November 30, 2007 10:19 PM
>To: IETF AVT WG
>Cc: Wang Ye-Kui (Nokia-NRC/Tampere); Jonathan Lennox; Thomas Schierl
>Subject: Re: [AVT] I-D ACTION:draft-ietf-avt-rtp-svc-03.txt 
>
>Let me make a few remarks about the CL-DON vs. no-CL-DON 
>discussion, which leads to the proposal to remove the CL-DON approach.
>
>The question that needs to be answered is, how can the 
>decoding order of the SVC bitstream be established when 
>temporal scalability is used over multiple RTP sessions.

In addition to temporal scalability, we solve the problem for spatial
scalability and quality/SNR scalability. Cases where different
scalabilities are combined should also be taken care of. 

>
>ASSUMPTIONS
>
>1) Multiple RTP sessions for SVC transport:
>* We need to consider the generic case of transporting SVC in 2 (or
>more) RTP sessions, one of which is the H.264 base layer RTP 
>sessions and the other(s) is (are) SVC enhancement layer RTP 
>session(s).
>* Streaming using layered multicast and with or w/o 
>interleaving is one application, while low-delay communication 
>without interleaving is another application that could be 
>served by this setup. (interleaving = decoding order is not 
>equal to sequence numbers)
>
>2) H.264 base layer:
>* When interleaving is used, the H.264 base layer RTP session 
>conforming to RFC 3984 shall be operated using packetization mode 2.
>* When no interleaving is used, the H.264 base layer RTP 
>session conforming to RFC 3984 can be operated using 
>packetization modes 0, 1 or 2.
>
>3) SVC enhancement layer:
>* When interleaving is used, the SVC enhancement layer RTP 
>session(s) shall be operated using packetization mode 2.
>* When no interleaving is used, the SVC enhancement layer RTP
>session(s) can be operated using packetization modes 0, 1, or 
>2 ( according to the new draft)
>
>4) PACSI:
>* When packetization mode 0 is used, PACSI cannot be present.
>* When packetization modes 1 or 2 are used, PACSI may be 
>present; for 3984-compliant devices, it would hopefully be 
>ignored, but could be used by the SVC device.
>
>5) CL-DON is part of the PACSI (according to the new draft).
>
>
>CONCLUSIONS
>
>A) When temporal scalability using multiple RTP sessions is 
>used in combination with packetization mode 0 anywhere, PACSI 
>and with that CL- DON is not present and the CL-DON solution 
>is not working for establishing decoding order. Therefore, the 
>no-CL-DON solution MUST be specified in the draft.
>

This conclusion may hold only when there is good deployment
"multicast-capable mode-0 only" legacy RFC 3984 receivers. So far the
indication is no to this from the discussions in this mailing list. 

A big issue has been mentioned earlier for using mode 0 for the base
layer session, that is the prefix NAL unit problem. Prefix NAL unit is
of small size, and it must be present for each coded slice NAL unit in
the base layer stream. Using mode 0 mandates that each prefix NAL unit
is in its own packet. That basically doubles the number of packets for
the same media stream. 

>B) When interleaving is used in all RTP sessions, mode 2 must 
>be used in all RTP sessions and the CL-DON solution is not 
>needed since DONs are present and can be used the same way as 
>CL-DONs for establishing decoding order.
>
>This solution also works if the H.264 base layer RTP session 
>can use mode 2 without interleaving. Then, mode 2 can be used 
>in the SVC enhancement layer RTP sessions.
>
>C) CL-DON is potentially only useful for the remaining case, i.e.
>* when packetization mode 1 in the H.264 base layer RTP 
>session (for legacy reasons) is used in combination with
>* packetization modes 1 or 2 in the SVC enhancement layer RTP 
>sessions and
>* temporal scalability (i.e. having non-aligned access units).
>This is a very special case.
>
>This case can also be covered by the no-CL-DON approach, where 
>the dummy NAL units can be merged into the same packet with 
>other non- dummy NAL units causing very little overhead. This 
>actually causes less overhead compared to the CL-DON overhead 
>in the PACSI (occurring in all RTP sessions).
>
>Since the dummy NAL units are not in separate packets, the 
>error resilience issue is mitigated.
>

"non-algined access units" can easily happen in any case, not only in
temporal scalability case. For example, when temporal scalability is
used in combination with other scalabilities. For another example, even
when temporal scalability is not in use while only spatial and/or
quality scalability is in use, packet losses can easily lead to
"non-aligned access units". 

Furthermore, I don't know how exactly dummy NAL units should be
generated to solve the problem when there are "non-aligned access
units". Could you please clarify? For example, how to add dummy NAL
units to the bitstream with the following scalability structure?

- Two temporal scalable levels, 15 Hz and 30 Hz
- Two spatial resolution, QCIF and CIF
- Each spatial resolution has a medium-granularity scalability (MGS)
quality enhancement. 

A section of the bitstream without dummy NAL units is as follows:

CIF (DID=1, QID=1)	:    I  B  P  B
CIF (DID=1, QID=0)	:    I  B  P  B
QCIF (DID=0, QID=1)	:    I  B  P  B
QCIF (DID=0, QID=0)	:    I  B  P  B
                                  TID: 0  1  0  1

Assume that there are different groups of receivers require operation
points QCIF 15, QCIF 30, CIF 15, and CIF 30, respectively, all without
MGS enhancements. 

>Hence, the CL-DON approach is not needed and should be 
>removed, as the additional option is not justified.
>
>
>Best Regards,
>Thomas
>
>---
>Thomas Wiegand, Dr.-Ing.
>Head, Image Communication Group
>Image Processing Department
>Fraunhofer Inst. for Telecommunications - HHI Einsteinufer 37, 
>D-10587 Berlin, Germany
>Phone: +49 30 31002 617,
>Mobile: +49 172 381 3814
>http: iphome.hhi.de/wiegand
>---
>
>
>
>
>
>

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Thomas Wiegand <[email-masked]>
Date: Sun, 2 Dec 2007 11:23:13 +0100

Ye-Kui,

replies in-line. The conclusion remains the same: CL-DON is an option  
that is not needed and should not be supported.

Best Regards,
Thomas

> scalabilities are combined should also be taken care of.

The problem, however, only occurs when a combination with temporal  
scalability is present.

Actually, good that you mention this: the no-CL-DON approach is even  
more efficient when different scalabilities are combined, since the  
need for dummy NAL units is even further reduced.

>> 5) CL-DON is part of the PACSI (according to the new draft).

Not seeing a comment here means to me that you agree with my  
assumptions.

> indication is no to this from the discussions in this mailing list.

You agreed with my assumption 1) above. Streaming with layered  
multicast is just one application.

Multiple RTP sessions maybe used also for low-delay communications. It  
was pointed out multiple times (also on this reflector) that many  
devices that decode the H.264 base layer decode only mode 0. Backwards- 
compatible communication with these devices is of high importance to a  
number of companies. Hence, it needs to be supported.

> is in its own packet. That basically doubles the number of packets for

When assuming 45 Bytes per packet and 30 frames/s, this prefix NAL  
unit stream is roughly 10 kbit/s.

Not every device is a mobile phone and your multicast streaming  
application is not the only application for SVC !

For higher rate applications, these 10 kbit/s are not nice to have if  
not used but also not that important. The problem is acknowledged but  
at this point unavoidable.

>> in the SVC enhancement layer RTP sessions.

I then also assume that conclusion B is agreed.

What is left is the case described in conclusion C below. BTW, could  
you please mention those cases where you are forced to use  
packetization mode 1 for the H.264 base layer RTP session? If there  
are no such cases, you could always use mode 2 in base and enhancement  
layer and be done with the problem.

> "non-aligned access units".

I am assuming that this discussion is restricted to the case of  
packetization mode 1 in the H.264 base layer RTP session. (All other  
cases are covered above)

The constraint that is enforced is that each SVC enhancement RTP  
session contains a superset of the access units of the base layer. The  
superset may be larger or equal to the set of access units of the H. 
264 base layer RTP session.

The dummy NAL units are inserted to fulfill the above constraint. That  
is a very simple rule that can is easily applied for your complicated  
example below.

Hence, the CL-DON solution is not needed.

Best Regards,
Thomas

---

From: Colin Perkins <[email-masked]>
Date: Sat, 1 Dec 2007 18:36:20 -0500

On 27 Nov 2007, at 13:56, <[email-masked]> <Ye- 
[email-masked]> wrote:
...
>> 4.  On the issue of how to do session multiplexing:  it's
>> clear that DON or CL-DON mode is necessary if any of the
>> sessions being multiplexed (including, potentially, the H.264
>> base session) is using packetization mode 2.  However, I think
>> that the alternate (no CL-DON) mode is sufficient if every
>> session is using packetization mode 0 or 1, subject to some
>> minor restrictions.  (I'm fairly sure the restrictions in
>> table I of section 7 can be loosened somewhat; in particular,
>> I think point b. is too strong.)
>
> As you know, this is the part we authors have not yet been able to  
> agree
> with each other. The "no CL-DON mode" has the following disadvantages:
>
> - To meet the requirement I.a in section 7, dummy NAL units (with only
> NAL unit header and slice header) are required to be present in the
> bitstream to handle some cases. This requires the RTP encasulator  
> to be
> able to create dummy NAL units, which is a burden to the  
> implementation
> for a streaming server that deals with pre-encoded contents without  
> such
> dummy NAL units.
>
> - The process is non-deterministic when packet loss exists, and the
> performance of the process could be extremely low, because- for a
> certain non-base session, even a single packet loss happens, even to a
> discardable packet (e.g. containing SEI NAL units or redundant slices
> only), data of this and higher sessions are discarded unitl the next
> sync point. See "In case of packet loss, the essential connection
> between RTP sessions and packets cannot be kept, for that reason, we
> propose the reduce the reordering process only up to the session which
> the packet loss contains.  After finding a S0 Synchronization  
> point, the
> reordering can be applied without restrictions.."
>
> - The process requires the synchronizaiton of RTP timestamps  
> between all
> the RTP sessions conveying the same SVC bitstream. It is also possible
> to reply on RTCP based timestamp synchronization. However, in this  
> case,
> the receiver must wait for the reception of the first RTCPs from  
> all the
> sessions.

That's a general limitation of RTP, and not something you should try  
to fix in this payload format. The payload format should use the  
standard RTP mechanisms to synchronise streams - i.e. use RTCP SR  
information, not matching timestamps - allowing standard RTP receiver  
code to synchronise the streams, and pass them to the layered decoder  
at the appropriate time.

If initial synchronisation delay due to the RTCP timing rules is an  
issue, then it should be fixed in a payload format independent  
manner, not as part of SVC.

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Sat, 1 Dec 2007 18:50:38 -0500

On 30 Nov 2007, at 04:59, <[email-masked]> <Ye- 
[email-masked]> wrote:
>>> I am not sure whether those are of the same essense as the  
>>> context here. However, at least layered multicast of scalable  
>>> audio is the same. For  that, it seems that Colin has some  
>>> concern with RTP timestamp synchonization therein, see the  
>>> attached email.
>>
>> The algorithm works with RTP timestamps as well as with media time  
>> stamps derived by using RTCP.
>
> It is also possible to reply on RTCP based timestamp synchronization.
> However, in this case, the receiver must wait for the reception of the
> first RTCPs from all the sessions to start the de-packetization  
> process.

For unicast sessions, the first RTCP packet MAY be sent immediately  
(RFC 3550 section 6.2), which should reduce the impact of waiting for  
RTCP. Using AVPF will likely help for group scenarios.

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Sat, 1 Dec 2007 18:36:20 -0500

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Sat, 1 Dec 2007 18:50:38 -0500

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Sun, 2 Dec 2007 20:54:25 +0200

Hi Thomas,

To me, still the no-CL-DON mode is not needed because 

1) There is no real deployment of "multicast-capable mode 0 only" legacy
RFC 3984 receivers. 
2) It is flawed due to a) the reqirement of timely synchronization of
streams from different sessions; b) the requirement of dummy NAL units -
and it remained unclear whether this is always viable; c) each small
prefix NAL unit in its own packet; d) requirement of discarding of
received NAL units when packet losses exist; and e) the text complexity
is high - lot of text needed to clearly describe the entire process. 

See also my responses inline below, where there are also more details
for supporting the above.

BR, YK

>Sent: Sunday, December 02, 2007 12:23 PM
>To: Wang Ye-Kui (Nokia-NRC/Tampere)
>Cc: [email-masked]; [email-masked]; [email-masked]
>replies in-line. The conclusion remains the same: CL-DON is an 
>option that is not needed and should not be supported.
>
>Best Regards,
>Thomas
>
>discussion, 
>>> which leads to the proposal to remove the CL-DON approach.
>>>
>decoding order 
>>> of the SVC bitstream be established when temporal 
>scalability is used 
>>> over multiple RTP sessions.
>>
>The problem, however, only occurs when a combination with 
>temporal scalability is present.
>

This depends on which problem you are referring to. The general problem
we are trying to solve is decoding order recovery of NAL units conveyed
in multiple sessions. For the branch problem of the no-CL-DON mode
requiring dummy NAL units, yes that occurs when temporal scalability is
present, solely or in combitation with othe scalabilities. 

>Actually, good that you mention this: the no-CL-DON approach 
>is even more efficient when different scalabilities are 
>combined, since the need for dummy NAL units is even further reduced.
>

I don't see why the need for dummy NAL units is even further reduced. It
seems to me that sometimes adding dummy NAL units would not work. I may
be wrong, but I think this can be clarified if you tell how exactly to
add dummy NAL units, including giving details on how to set values of
the fields in the dummy NAL units, e.g. for the example I gave earlier,
which is present in the bottom of this email. 

>>> more) RTP sessions, one of which is the H.264 base layer 
>RTP sessions 
>>> and the other(s) is (are) SVC enhancement layer RTP session(s).
>>> * Streaming using layered multicast and with or w/o interleaving is 
>>> one application, while low-delay communication without interleaving 
>>> is another application that could be served by this setup. 
>>> (interleaving = decoding order is not equal to sequence numbers)
>>>
>>> 2) H.264 base layer:
>>> * When no interleaving is used, the H.264 base layer RTP session 
>>> conforming to RFC 3984 can be operated using packetization 
>modes 0, 1 
>>> or 2.
>>>
>present; for 
>>> 3984-compliant devices, it would hopefully be ignored, but could be 
>>> used by the SVC device.
>>>

Yes, you assumptions are correct to me. 

>
>>> CONCLUSIONS
>>>
>>> A) When temporal scalability using multiple RTP sessions is used in 
>>> combination with packetization mode 0 anywhere, PACSI and with that 
>>> CL- DON is not present and the CL-DON solution is not working for 
>>> establishing decoding order. Therefore, the no-CL-DON solution MUST 
>>> be specified in the draft.
>>
>Multiple RTP sessions maybe used also for low-delay 
>communications. It was pointed out multiple times (also on 
>this reflector) that many devices that decode the H.264 base 
>layer decode only mode 0. Backwards- compatible communication 
>with these devices is of high importance to a number of 
>companies. Hence, it needs to be supported.
>

I think earlier discussions on this reflector indicate that there are
nowadays non-trivial number of mode-0-only receivers for point-to-point
use (videophones), but not "multicast-capable mode 0 only" legacy
dervices. For the two types of systems we are currently aware of that
will likely deploy the layered multicast in question, namely DVB-H like
system and 3GPP MBMS like system, mode 1 is required, and the MBMS even
requires mode 2. 

>When assuming 45 Bytes per packet and 30 frames/s, this prefix 
>NAL unit stream is roughly 10 kbit/s.
>

Yes, the additional birate itself is not big, but the additional number
of packets, which is big, also matters in transmission and packets
handling. 

>Not every device is a mobile phone and your multicast 
>streaming application is not the only application for SVC !
>

I agree that mobile multicast streaming application is not the only
application for SVC. 

>For higher rate applications, these 10 kbit/s are not nice to 
>have if not used but also not that important. The problem is 
>acknowledged but at this point unavoidable.
>

This can be avoided to by not using mode 0 for the session(s) conveying
the full base layer. 

>
>>> B) When interleaving is used in all RTP sessions, mode 2 
>must be used 
>>> in all RTP sessions and the CL-DON solution is not needed 
>since DONs 
>>> are present and can be used the same way as CL-DONs for 
>establishing 
>>> decoding order.
>>>
>>> This solution also works if the H.264 base layer RTP 
>session can use 
>>> mode 2 without interleaving. Then, mode 2 can be used in the SVC 
>>> enhancement layer RTP sessions.
>

Yes. But note that in the latest draft the so-called CL-DON mode
includes also the case when all the sessions use mode 2. 

>What is left is the case described in conclusion C below. BTW, 
>could you please mention those cases where you are forced to 
>use packetization mode 1 for the H.264 base layer RTP session? 

For legacy or future receivers that implement mode 1 but not mode 2 with
multicast capability. 

>If there are no such cases, you could always use mode 2 in 
>base and enhancement layer and be done with the problem.
>
>
>session (for 
>>> legacy reasons) is used in combination with
>>> This case can also be covered by the no-CL-DON approach, where the 
>>> dummy NAL units can be merged into the same packet with other non- 
>>> dummy NAL units causing very little overhead. This actually causes 
>>> less overhead compared to the CL-DON overhead in the PACSI 
>(occurring 
>>> in all RTP sessions).
>>>
>>> Since the dummy NAL units are not in separate packets, the error 
>>> resilience issue is mitigated.
>>
>> used in combination with other scalabilities. For another example, 
>> even when temporal scalability is not in use while only 
>spatial and/or 
>I am assuming that this discussion is restricted to the case 
>of packetization mode 1 in the H.264 base layer RTP session. 
>(All other cases are covered above)
>

No. We are discussing the problem of "non-aligned access units" when the
no-CL-DON mode is in use. There is no problem herein if the CL-DON mode
is in use. For the no-CL-DON mode, when "non-aligned access units" are
available in the original video stream then someone (the encoder or the
RTP packetizer) needs to inject dummy NAL units, which needs
understanding of slice header in addition to NAL unit header. When the
"non-aligned access units" occur due to packet losses, then all these
"non-aligned access units" and even more need to be discarded. 

>The constraint that is enforced is that each SVC enhancement 
>RTP session contains a superset of the access units of the 
>base layer. The superset may be larger or equal to the set of 
>access units of the H. 

Yes, I understand what this restriction is. 

>
>The dummy NAL units are inserted to fulfill the above 
>constraint. That is a very simple rule that can is easily 
>applied for your complicated example below.
>

As I mentioned earlier, could you please tell how exactly to add dummy
NAL units, including how to set the values of the fields in the dummy
NAL units. I think it would be clear enough for understanding if you do
this using the example below. I requested this because I think that
adding dummy NAL units may be not viable in some cases. 

>>> Hence, the CL-DON approach is not needed and should be removed, as 
>>> the additional option is not justified.
>>>
>>>
>>> Best Regards,
>>> Thomas
>>>
>>> ---
>>> Thomas Wiegand, Dr.-Ing.

---

From: Thomas Wiegand <[email-masked]>
Date: Tue, 4 Dec 2007 00:49:44 +0100

Ye-Kui

You are just repeating yor old arguments and not add anything new to  
the discussion. I see my previous conclusions still being valid.

Best Regards,
Thomas

On Dec 2, 2007, at 7:54 PM, <[email-masked]> <[email-masked] 
 > wrote:

---

From: <[email-masked]>
Date: Tue, 4 Dec 2007 03:20:48 +0200

Thomas,

I have not seen my arguments being invalidated yet - while you have
drawn a conclusion without considering the arguments - and they are even
old ones as you said. At least, I think you should tell how exactly to
add dummy NAL units for folks (including myself) to understand whether
that is always viable. Other parts I guess people in the list would
understand after careful reading. 

BR, YK 

>Sent: Tuesday, December 04, 2007 1:50 AM
>Cc: [email-masked]; [email-masked]
>You are just repeating yor old arguments and not add anything 
>new to the discussion. I see my previous conclusions still being valid.
>
>Best Regards,
>Thomas
>
>
>
>On Dec 2, 2007, at 7:54 PM, <[email-masked]> 
><[email-masked]  > wrote:
>
>>
>> Hi Thomas,
>>
>> streams from different sessions; b) the requirement of dummy 
>NAL units 
>> - and it remained unclear whether this is always viable; c) 
>each small 
>> received NAL units when packet losses exist; and e) the text 
>> complexity is high - lot of text needed to clearly describe 
>the entire 
>> process.
>>
>> This depends on which problem you are referring to. The general 
>> problem we are trying to solve is decoding order recovery of 
>NAL units 
>> conveyed in multiple sessions. For the branch problem of the 
>no-CL-DON 
>> mode requiring dummy NAL units, yes that occurs when temporal 
>> scalability is present, solely or in combitation with othe 
>> scalabilities.
>>
>> I don't see why the need for dummy NAL units is even further 
>reduced. 
>> It seems to me that sometimes adding dummy NAL units would 
>not work. I 
>> may be wrong, but I think this can be clarified if you tell how 
>> exactly to add dummy NAL units, including giving details on 
>how to set 
>> values of the fields in the dummy NAL units, e.g. for the example I 
>> gave earlier, which is present in the bottom of this email.
>>
>>>>> ASSUMPTIONS
>>>>>
>interleaving is 
>>> Multiple RTP sessions maybe used also for low-delay communications. 
>>> It was pointed out multiple times (also on this reflector) 
>that many 
>>> devices that decode the H.264 base layer decode only mode 0. 
>>> Backwards- compatible communication with these devices is of high 
>>> importance to a number of companies. Hence, it needs to be 
>supported.
>>>
>>
>> nowadays non-trivial number of mode-0-only receivers for point-to- 
>> point use (videophones), but not "multicast-capable mode 0 only" 
>> legacy dervices. For the two types of systems we are currently aware 
>> of that will likely deploy the layered multicast in question, namely 
>> DVB-H like system and 3GPP MBMS like system, mode 1 is required, and 
>> the MBMS even requires mode 2.
>>
>> Yes, the additional birate itself is not big, but the additional 
>> number of packets, which is big, also matters in transmission and 
>> packets handling.
>>
>> This can be avoided to by not using mode 0 for the session(s) 
>> conveying the full base layer.
>>
>>>
>> No. We are discussing the problem of "non-aligned access 
>units" when  
>> the
>> NAL units. I think it would be clear enough for 
>understanding if you  
>> do