Subject: [AVT] I-D ACTION:draft-ietf-avt-rtp-svc-03.txt

From: [email-masked]
Date: Tue, 20 Nov 2007 15:15:01 -0500

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.
This draft is a work item of the Audio/Video Transport Working Group of the IETF.

	Title		: RTP Payload Format for SVC Video
	Author(s)	: S. Wenger, et al.
	Filename	: draft-ietf-avt-rtp-svc-03.txt
	Pages		: 0
	Date		: 2007-11-20
	
This memo describes an RTP payload format for scalable video coding
   (SVC) defined in_Annex G of the ITU-T Recommendation H.264 video
   codec which is technically identical to Amendment 3 of ISO/IEC
   International Standard 14496-10.  The RTP payload format allows for
   packetization of one or more Network Abstraction Layer (NAL) units,
   produced by the video encoder, in each RTP packet payload.  The
   payload format has wide applicability, such as low bit-rate
   conversational, Internet video streaming, or high bit-rate
   entertainment quality video.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-avt-rtp-svc-03.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-ietf-avt-rtp-svc-03.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-avt-rtp-svc-03.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: <[email-masked]>
Date: Wed, 21 Nov 2007 00:16:44 +0200

Hi Colleagues,

In this version, we have worked on all the missing parts, including SDP parameters, semantics and their usage, and security considerations. 

Another major change is the addition to enable decoding order recovery of data units transported in multiple RTP sessions using layered multicast, without the requirement of interleaved mode. For this purpose, two alternative solutions are included. The first is based on cross-layer decoding order number (CL-DON), as specified in 8.1.2, which is close to RFC 3984 when interleaved mode is in use. The second uses timestamp etc., as specified in 8.1.1. The first process has been agreed by the editors, but having the second process in addition has not been agreed by the editors. The editors request the AVT WG to make a decision in this regard. 

There are also numerous smaller changes. In addition, we have addressed all the comments sent to the reflector or in private, but one from Magnus, which will be addressed in the next version. We also noticed that the current version does not display well for semantics of many of the media type parameters. We are sorry for that and will fix it in the next version. 

We thank a lot to Roni, Colin, Thomas Rusert, and Magnus for their reviews and valuable comments to the drafts. Summary of our responses to the comments are as follows:

Roni's comments (sent to AVT mailing list on 20 July 2007):
===========================================================
1# - done - single NAL unit mode allowed now
2# - done - single NAL unit mode allowed now
3# - done - comment incorrect, but corresponding text removed 
4# - no action - the definition matches well with the one in the SVC spec
5# - resolved by the changes in Section 7 (Packetization Rules)
6# - done - removed "The following applies in addition"
7# - done - addressed as part of the signaling thing
8#  - no action - the new NAL unit type of PACSI does not has direct relationship with MDfH
9# - done -clarification improved in the last paragraph of §3. 
10# - done - comment incorrect, but text clarified better
11# - done - addressed by the new mechanisms of NAL unit order recovery for layered multicast
12# - done - an informative note added
13# - done - all parameters are now listed and semantics given 
14# - done - Thomas Schierl made clarifications in email 
15# - done - the media subtype was defined as "H.264-SVC", now changed to "H264-SVC", to align with RFC 3984 that uses "H264" not "H.264"
16# - done - changed "MIME" to "media type" or "media", instead of the proposed "payload type"
 
Colin' comment (sent to the AVT mailing list on 31 July 2007):
===========================================================
1# - done - added references to RFC 3450 and a research paper in the congestion control section
 
Thomas Rusert's comments (sent to the editors in private on 31 August 2007):
===========================================================
1# - done - defintion of "reference base picture" added
2# - done - "three" changed to "'11' (in binary form)"
3# - done - 'temporal_level' changed to 'temporal_id'
4# - no action - priority_id based adaption is typically for integer number of layers 
5# - done - number of bytes in PACSI corrected
6# - done - D bit (discardable_flag) semantics changed
7# - done - remaining FGS stuff to be removed
 
Magnus' comments (sent to the AVT mailing list on 31 August 2007)
===========================================================
1#- done - mentioned that a new media subtype name is defined, and solved the "single NAL unit mode deprecation" issue
2# - done - parameter sets introduction text updated 
3# - done - semantics of R and RR updated
4# - done - removed "identical PT" from the defintion of RTP packet stream
5# - done - removed "The following applies in addition"
6# - done - meaning of "protecting" clarified in §6.4
7# - done - single NAL unit mode allowed now
8# - done - addressed as part of the signaling thing and decoding order recovery for layered multicast
9# - no action - old defintion of the F bit is OK
10# - done - defintion of "anchor layer representation" added
11# - done - the flag for the temporal scalable layer switching point has been used to make CL-DON optional in the PACSI NAL unit
12# - done - semantics of PACSI fields expressed that apply only to the NALUs within the current packet
13#  - done - justifications of the PACSI fields added as informative notes after the semantics
14# - done - media subtype name changed to "H264-SVC or H264"
15# - done - addressed as part of the signaling thing and decoding order recovery for layered multicast
16# - done - added references of base16 and base64
17# - done - changed to representaion of sprop-leyer-ids from base64 to base16
18# - done - offer/answer model added
19# - to be addressed in the next version
20# - done - security considerations updated
21# - done - remaining FGS stuff removed 

BR, YK 

>-----Original Message-----
>From: ext [email-masked] [mailto:[email-masked]] 
>Sent: Tuesday, November 20, 2007 10:15 PM
>To: [email-masked]
>Cc: [email-masked]
>Subject: [AVT] I-D ACTION:draft-ietf-avt-rtp-svc-03.txt 
>
>A New Internet-Draft is available from the on-line 
>Internet-Drafts directories.
>To remove yourself from the I-D Announcement list, send a 
>message to [email-masked] with the word 
>unsubscribe in the body of the message. 
>Internet-Drafts are also available by anonymous FTP. Login 
>with the username "anonymous" and a password of your e-mail 
>address. After logging in, type "cd internet-drafts" and then 
>http://www.ietf.org/shadow.html or 
>ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>Below is the data which will enable a MIME compliant mail 
>reader implementation to automatically retrieve the ASCII 
>version of the Internet-Draft.
>

---

From: "Jonathan Lennox" <[email-masked]>
Date: Tue, 27 Nov 2007 11:18:00 -0500

[email-masked] writes:
> Hi Colleagues,
>
> In this version, we have worked on all the missing parts, including 
> SDP parameters, semantics and their usage, and security
considerations.

Hi, Ye-Kui, all,

I read over this draft, and I have several comments.

1.  It looks like this new revision now not only registers the media
type video/H264-SVC, but also introduces normative changes to the
definition of video/H264 from RFC 3984.  This seems problematic.  If
need be, I understand that it may be useful to define new format
parameters for video/H264 (though I don't really understand why they're
needed), but it looks like this draft does more than this, changing the
definition of existing parameters.

I think that video/H264 should remain defined by 3984, with only
backward-compatible updates if needed, and this document should only
define video/H264-SVC.

2.  Relatedly, I'm not sure why it's now necessary to support
packetization mode 0 for SVC.  As Roni pointed out, it's clearly
necessary (for compatibility reasons) to support the use case where
session multiplexing uses mode 0 for the H264 base layer, but that's
sent as video/H264.  Requiring that higher layers, or non-multiplexed
sessions, sent as video/H264-SVC use packetization mode 1 or 2 does not
seem like a hardship to implementers, whereas using packetization mode 0
for SVC means that you can't aggregate prefix NAL units with the slices
they prefix, and you can't use PACSI at all.

3. I'm wondering why PACSI now has optional parts, since it itself is
optional.  This includes the bit flags (A, B, C, S, and E) and the
TL0PICIDX related fields, which were fixed in the previous version that
was agreed to with JVT.  Especially, the X bit doesn't make much sense
to me, since it doesn't even save any bytes; it just switches several of
the other bits from being defined to being undefined.

4.  On the issue of how to do session multiplexing:  it's clear that DON
or CL-DON mode is necessary if any of the sessions being multiplexed
(including, potentially, the H.264 base session) is using packetization
mode 2.  However, I think that the alternate (no CL-DON) mode is
sufficient if every session is using packetization mode 0 or 1, subject
to some minor restrictions.  (I'm fairly sure the restrictions in table
I of section 7 can be loosened somewhat; in particular, I think point b.
is too strong.) 

Also, it looks like the CL-DON mode can't work if the base H.264 layer
is using packetization mode 0, i.e. Roni's use case, since it requires
the use of PACSIs in STAP-A packets.  I think this is a strong
motivation for maintaining the no-CL-DON mode.

5.  The definition of Session Multiplexing implies that only one source
is sent in each RTP session.  In principle, Session Multiplexing
(splitting one source over multiple sessions) could happen in parallel
with sending multiple sources over each RTP session.  This is just a
wording issue I think.

(I've also heard some confusion, when discussing this draft, about the
term "Session Multiplexing", since it's really more of an inverse
multiplexing.  Perhaps some terminology could be cleaned up?)

Thanks!

("Note well" disclaimer: Vidyo (formerly Layered Media) has prospective
IPR related to this draft.  Our IPR disclosure is at
<https://datatracker.ietf.org/ipr/832/>.)

-- 
Jonathan Lennox
Vidyo, Inc (formerly Layered Media)
[email-masked]

---

From: <[email-masked]>
Date: Tue, 27 Nov 2007 20:56:29 +0200

Hi Jonathan,

Thanks for reading the draft and your comments. See inline, please. 

>1.  It looks like this new revision now not only registers the 
>media type video/H264-SVC, but also introduces normative 
>changes to the definition of video/H264 from RFC 3984.  This 
>seems problematic.  If need be, I understand that it may be 
>useful to define new format parameters for video/H264 (though 
>I don't really understand why they're needed), but it looks 
>like this draft does more than this, changing the definition 
>of existing parameters.
>
>I think that video/H264 should remain defined by 3984, with 
>only backward-compatible updates if needed, and this document 
>should only define video/H264-SVC.
>

Our intention is the same as you said: video/H264 remains defined by RFC
3984, and updates, e.g. to the semantics of the media type parameters,
are backward-compatible.

I guess the paragraph "Media subtype name:  H264-SVC or H264" caused
part of confusion. I have tried to explain the intention in the
subsequent paragraph. I would appreciate if you could suggest better
wording in case you think improvement herein is needed. 

Another part of the confusion should come from the changes to the
semantics of some existing media type parameters, e.g.
sprop-deint-buf-req. The change compared to RFC 3984 is as follows. In
RFC 3984, the parameter MUST be present when the packetization mode 2 is
in use, otherwise it MUST NOT be present. In this draft, the parameter
MUST be present when the packetization mode 2 is in use or the RTP
session depends on any other RTP sessions in layered multicast use.
Since layered multicast (for temporal scalable layers) was not
considered at all in RFC 3984, and RFC 3984 as such does not work for
layered multicast. Therefore, we think that the change is backward
compatible. Other changes of the semantics to existing media type
parameters are the same. 

Please let me know if you have found other places that you thought not
backward compatible. 

>packetization mode 0 for SVC.  As Roni pointed out, it's 
>clearly necessary (for compatibility reasons) to support the 
>use case where session multiplexing uses mode 0 for the H264 
>base layer, but that's sent as video/H264.  Requiring that 
>higher layers, or non-multiplexed sessions, sent as 
>video/H264-SVC use packetization mode 1 or 2 does not seem 
>like a hardship to implementers, whereas using packetization 
>mode 0 for SVC means that you can't aggregate prefix NAL units 
>with the slices they prefix, and you can't use PACSI at all.
>

I agree with you on all these points. Again, our intention was trying to
resolve people's (including Roni's) concerns expressed earlier regarding
requiring the support of packetization mode 0 for backward compatible
reason. Because of the same latter reasons you mentioned above, we
specified the requirement of mode 1 support. What would be a better
wording of these in your opinion?

>
>3. I'm wondering why PACSI now has optional parts, since it 
>itself is optional.  This includes the bit flags (A, B, C, S, 
>and E) and the TL0PICIDX related fields, which were fixed in 
>the previous version that was agreed to with JVT.  Especially, 
>the X bit doesn't make much sense to me, since it doesn't even 
>save any bytes; it just switches several of the other bits 
>from being defined to being undefined.
>

Reasons to have them optional are as follows. JVT has recently agreed
(during the October 2007 JVT meeting) a syntax change to the TL0PICIDX
SEI message, by adding a new syntax element, which should be added to
the PACSI NAL unit (i.e. IDRPICID). This one is another 16-bit field. We
therefore decided to make all these optional fields, as they are not
used in many use cases. 

The X bit does not save bits, but it enables RTP packetizers to avoid
deriving the values of A, P, C, S, and E flags if the application does
not want to use them but it still wants to use other parts of PACSI. 

>
>4.  On the issue of how to do session multiplexing:  it's 
>clear that DON or CL-DON mode is necessary if any of the 
>sessions being multiplexed (including, potentially, the H.264 
>base session) is using packetization mode 2.  However, I think 
>that the alternate (no CL-DON) mode is sufficient if every 
>session is using packetization mode 0 or 1, subject to some 
>minor restrictions.  (I'm fairly sure the restrictions in 
>table I of section 7 can be loosened somewhat; in particular, 
>I think point b.
>is too strong.) 
>

As you know, this is the part we authors have not yet been able to agree
with each other. The "no CL-DON mode" has the following disadvantages:

- To meet the requirement I.a in section 7, dummy NAL units (with only
NAL unit header and slice header) are required to be present in the
bitstream to handle some cases. This requires the RTP encasulator to be
able to create dummy NAL units, which is a burden to the implementation
for a streaming server that deals with pre-encoded contents without such
dummy NAL units.

- The process is non-deterministic when packet loss exists, and the
performance of the process could be extremely low, because- for a
certain non-base session, even a single packet loss happens, even to a
discardable packet (e.g. containing SEI NAL units or redundant slices
only), data of this and higher sessions are discarded unitl the next
sync point. See "In case of packet loss, the essential connection
between RTP sessions and packets cannot be kept, for that reason, we
propose the reduce the reordering process only up to the session which
the packet loss contains.  After finding a S0 Synchronization point, the
reordering can be applied without restrictions.." 

- The process requires the synchronizaiton of RTP timestamps between all
the RTP sessions conveying the same SVC bitstream. It is also possible
to reply on RTCP based timestamp synchronization. However, in this case,
the receiver must wait for the reception of the first RTCPs from all the
sessions. 

- The process is more complex than the CL-DON mode. The CL-DON mode is
similar as the RFC 3984 de-interleaving process in complexity. The "no
CL-DON mode" requires first to de-packetize each RTP session separately,
then apply the new process using RTP timestamps etc. 

>Also, it looks like the CL-DON mode can't work if the base 
>H.264 layer is using packetization mode 0, i.e. Roni's use 
>case, since it requires the use of PACSIs in STAP-A packets.  
>I think this is a strong motivation for maintaining the no-CL-DON mode.
>

You are right. The CL-DON mode requires that receivers involved in
layered multicast implement at least the packetization mode 1. We think
this is OK because we don't see a layered multicast use case where
receivers implement only packetization mode 0, even they intends to
receive only the H.264/AVC base layer. Do you see such a use case?

>
>5.  The definition of Session Multiplexing implies that only 
>one source is sent in each RTP session.  In principle, Session 
>Multiplexing (splitting one source over multiple sessions) 
>could happen in parallel with sending multiple sources over 
>each RTP session.  This is just a wording issue I think.
>
>(I've also heard some confusion, when discussing this draft, 
>about the term "Session Multiplexing", since it's really more 
>of an inverse multiplexing.  Perhaps some terminology could be 
>cleaned up?)
>

I think you are right and clean-up in this regard is needed. 

BR, YK

>
>Thanks!
>
>("Note well" disclaimer: Vidyo (formerly Layered Media) has 
>prospective IPR related to this draft.  Our IPR disclosure is at

---

From: "Thomas Schierl" <[email-masked]>
Date: Tue, 27 Nov 2007 12:20:26 -0800

Hi Ye-Kui, Hi Jonathan, all

Please find my comments below.

Thomas Schierl
--------------
Fraunhofer HHI

> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, November 27, 2007 10:56 AM

> for a streaming server that deals with pre-encoded contents without

First, the insertion of dummy NAL units is only required, when using plain
temporal scalability. For the use case we have in mind (spatial scalability
over a wireless broadcast/multicast channel) this feature is negligible. I
think that will be also the case for most of the other scenarios.
Second, I do not think that the streaming server is required to produce
these NAL units. This can be easily done by the encoder. For pre-encoded
content, these days' streaming server typically rely on hinting tracks for
generating the RTP packets, thus even in this special case I see no extra
burden for the server.

> reordering can be applied without restrictions.."

The process is absolutely deterministic, if packet loss occurs.
Since the informative section 8.1.1. only gives a simple process
description, it does not aim to present the whole solution. This is due to
the implementer and I think the process can of course deal with single
packet loss, without discarding packets as drastically as currently
described in the draft. I do not think that there are any normative parts in
the non-DON re-ordering process, it's really simple. At most, a more detail
description may be a part for an informative annex. I could accept that.

See, what other people think about this.

> then apply the new process using RTP timestamps etc.

It may be a little bit more tricky than implementing the re-ordering based
on DON, but I do not see a drastically increased algorithm complexity.

---

From: "Even, Roni" <[email-masked]>
Date: Wed, 28 Nov 2007 11:31:43 +0200

Hi,
I would like to clarify my view about the SVCs draft and RFC 3984.

According to H.264 amendment 3 and the liaison from JVT, SVC is not a
new codec but an extension to H.264 through 3 new profiles. I could
argue that as such you could specify it not as a new media subtype but
also as new profiles (Scalable Baseline, Scalable High, and Scalable
High Intra profiles) for H264 media subtype. This may take out some of
the confusion.

As H264 profiles or extensions I would like to see the interoperability
with RFC 3984 enabling a common mode as specified in RFC 3984 to be
supported in the document. For example like specified in the SVC
document for the optional profile and level parameters which said if it
is not given then you assume baseline at level 1. 
I would expect an implementation that supports H264-SVC to also support
the mandatory modes of RFC 3984. This means for example that H264 and
H264-SVC with no optional parameters have the same payload or that H264
Baseline at level 3 and H264-svc baseline at level 3 are the same. This
is clear to me from the current parameters defined.

Since this draft defines new packetization I think it is good to enable
systems that only support RFC 3984 packetization and mandatory modes
(like single NAL unit per packet) to get the base layer and use it. Of
course they will need to make changes but those are in the RTP layer
which may be simpler. 

BTW: most current implementations in video conferencing are using H.323
and they expect the parameter sets in band and not in the signaling.

Regards
Roni Even

---

From: "Jonathan Lennox" <[email-masked]>
Date: Wed, 28 Nov 2007 11:50:03 -0500

Ye-Kui Wang wrote:

> Our intention is the same as you said: video/H264 remains defined by 
> RFC 3984, and updates, e.g. to the semantics of the media type 
> parameters, are backward-compatible.
> 

The media type registration of video/H264-SVC should probably be in one
section, and then a separate section can define extensions to the
existing registration of video/H264.

> RFC 3984, the parameter MUST be present when the packetization mode 2 
> is in use, otherwise it MUST NOT be present. In this draft, the 
> parameter MUST be present when the packetization mode 2 is in use or 
> the RTP session depends on any other RTP sessions in layered multicast
use.
> compatible. Other changes of the semantics to existing media type 

Well, an SDP fmtp for H264 containing
"packetization-mode=1;sprop-deint-buf-req=<whatever>" is invalid
according to 3984:

       sprop-deint-buf-req:
                        This parameter MUST NOT be present when
                        packetization-mode is not present or the value
                        of packetization-mode is equal to 0 or 1.  It
                        MUST be present when the value of
                        packetization-mode is equal to 2.

If the usage we're concerned about is one where a legacy endpoint can
ignore/reject the H264-SVC enhancement streams but successfully receive
the base H264 stream, then we have to make sure that the legacy endpoint
won't reject the H264 stream because of what it believes to be an
illegal combination of parameters.

3984 says that receivers MUST ignore unspecified parameters, so if new
semantics need to be defined for CL-DON mode, adding new parameters is
fine.  We just can't change the semantics of existing ones.

> backward compatible.

It's very hard to tell due to the formatting problems in the parameter
descriptions.  I take it most of this text is copied verbatim from 3984.
Can you review what's different in them?

> >sessions, sent as video/H264-SVC use packetization mode 1 or 2 does 
> >not seem like a hardship to implementers, whereas using packetization

> >mode 0 for SVC means that you can't aggregate prefix NAL units with 
> >the slices they prefix, and you can't use PACSI at all.
> 
> I agree with you on all these points. Again, our intention was trying 
> to resolve people's (including Roni's) concerns expressed earlier 
> regarding requiring the support of packetization mode 0 for backward 
> compatible reason. Because of the same latter reasons you mentioned 
> above, we specified the requirement of mode 1 support. What would be a

> better wording of these in your opinion?

For backward compatibility, I'd suggest that multiplexed sessions SHOULD
support packetization mode 0 for a base video/H264 stream.

The question is whether packetization mode 0 should be allowed for a
video/H264-SVC stream.  My initial thought was no, but I'm reviewing
this in light of Roni's most recent comment.

> >TL0PICIDX related fields, which were fixed in the previous version 
> >that was agreed to with JVT.  Especially, the X bit doesn't make much

> >sense to me, since it doesn't even save any bytes; it just switches 
> >several of the other bits from being defined to being undefined.
> 

> the PACSI NAL unit (i.e. IDRPICID). This one is another 16-bit field.

> We therefore decided to make all these optional fields, as they are 
> not used in many use cases.
> 

> not want to use them but it still wants to use other parts of PACSI.

While it is true that JVT agreed to add the IDRPICID field during the
last call comments period, it did not agree to make the
TL0PICIDX/IDRPICID combination optional in the PACSI header.  As I am
sure you recall, the PACSI header was the subject of considerable
deliberation in JVT, and its contents in the previous version of this
draft reflected a careful balance between what should be in the video
stream and what should be in the RTP payload.  This includes the
TL0PICIDX as well as the associated flags (which, for those who did not
participate in the JVT discussions, are necessary to avoid displaying
broken video in the presence of packet losses).

> As you know, this is the part we authors have not yet been able to 
> agree with each other. The "no CL-DON mode" has the following

> bitstream to handle some cases. This requires the RTP encasulator to 
> be able to create dummy NAL units, which is a burden to the 
> implementation for a streaming server that deals with pre-encoded 
> contents without such dummy NAL units.

As Thomas said, these dummy NAL units are only needed for "pure"
temporal layering, i.e. if an enhancement layer session adds additional
access units in addition to those in the base layer sessions, and does
not have additional real NAL units for the access units present in the
lower layers.

> the packet loss contains.  After finding a S0 Synchronization point, 
> the reordering can be applied without restrictions.."

There is ambiguity (I wouldn't say "non-determinism") only if every
packet of an access unit is lost from one of the RTP sessions.  As long
as you have at least one packet from each access unit, you have enough
information to reconstruct a total (decode) ordering of the access
units.  (Decode ordering of NAL units within an access unit should be
deterministic, modulo isomorphisms.)

The description of the decode order reconstruction algorithm could be
cleaned up, though.

> - The process requires the synchronizaiton of RTP timestamps between 
> all the RTP sessions conveying the same SVC bitstream.

This is normal, I think.  I believe the traditional SDP <base multicast
address>[/<ttl>]/<number of addresses> mechanism used synchronized RTP
timestamps, and certainly session-associated FEC and RTX do.

> >H.264 layer is using packetization mode 0, i.e. Roni's use case, 
> >since it requires the use of PACSIs in STAP-A packets.
> layered multicast implement at least the packetization mode 1. We 
> think this is OK because we don't see a layered multicast use case 
> where receivers implement only packetization mode 0, even they intends

> to receive only the H.264/AVC base layer. Do you see such a use case?

I believe the intention was to allow existing (legacy) H.264/AVC
receivers, which only implement packetization mode 0, to receive the
base layer, while simultaneously allowing H.264/SVC receivers to receive
the full layered stream.  I think this is an important transition
mechanism for SVC adoption.

---

From: Randell Jesup <[email-masked]>
Date: Wed, 28 Nov 2007 15:18:21 -0500

>course they will need to make changes but those are in the RTP layer

It would be quite useful to do that; otherwise you have to offer the base
stream in H264-SVC, and a separate base stream in H264 so that
non-SVC-aware devices can talk to you.  Existing SIP H264 realtime
implementations often support either only mode 0, modes 0 and 1, or
occasionally modes 0-2.  

>and they expect the parameter sets in band and not in the signaling.

Video conferencing maybe, but 1-to-1 videophones are mostly SIP, as well as
most (almost all?) PC H.264 softclients.  That said, most don't handle
sprop-parameter-sets very well if at all.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: <[email-masked]>
Date: Thu, 29 Nov 2007 02:43:12 +0200

See inline.  

>The media type registration of video/H264-SVC should probably 
>be in one section, and then a separate section can define 
>extensions to the existing registration of video/H264.
>

We didn't have intention to make extensions for the existing
registration of video/H264. The extentions to the semantics of some
existing media type parameters only apply to video/H264-SVC.

I guess the confusion is still from the paragraph "Media subtype name:
H264-SVC or H264". To make the intention clear, I think we should change
it to "Media subtype name:  H264-SVC", and change the subsequent
paragraph to an informative note. 

>"packetization-mode=1;sprop-deint-buf-req=<whatever>" is 
>invalid according to 3984:
>
>       sprop-deint-buf-req:
>If the usage we're concerned about is one where a legacy 
>endpoint can ignore/reject the H264-SVC enhancement streams 
>but successfully receive the base H264 stream, then we have to 
>make sure that the legacy endpoint won't reject the H264 
>stream because of what it believes to be an illegal 
>combination of parameters.
>
>3984 says that receivers MUST ignore unspecified parameters, 
>so if new semantics need to be defined for CL-DON mode, adding 
>new parameters is fine.  We just can't change the semantics of 
>existing ones.
>

Such an SDP line can only be used for video/H264-SVC, because the design
principle we have in mind for layered multicast is as follows. Only the
base RTP session that does not depends on any other RTP session uses
video/H264, other RTP sessions for enhancement layers all use
video/H264-SVC. This means, RFC 3984 receivers can surely receive the
base RTP session. Therefore, reusing existing parameters for layered
multicast is OK. However, I am also OK with specifying new parameters. 

>It's very hard to tell due to the formatting problems in the 
>parameter descriptions.  I take it most of this text is copied 
>verbatim from 3984.

Sure. The changes compared to RFC 3984 are mainly twofold: 1)
straightforward adaptation of the text to make the semantics applicable
to SVC, 2) added the consideration of de-packetization for layered
multicast, e.g. the change of the semantics of sprop-deint-buf-req as I
mentioned earlier. 

>For backward compatibility, I'd suggest that multiplexed 
>sessions SHOULD support packetization mode 0 for a base 
>video/H264 stream.
>
>The question is whether packetization mode 0 should be allowed 
>for a video/H264-SVC stream.  My initial thought was no, but 
>I'm reviewing this in light of Roni's most recent comment.
>

My understanding is as follows. The context of the comments made on
single-NAL unit mode by many people around the previous IETF meeting was
not for layered multicast (i.e. multiplexed sessions). That is the
reason of the current text regarding the single-NAL unit mode. Let's see
what was people's real intention. 

Technically, if a video/H264-SVC stream is independent of any other RTP
session, then the full base layer must be H.264/AVC compatible.
Therefore, a video/H264 client should still be able to get and decode a
subset of the full base layer. Note also that the video/H264-SVC stream
can be marked as video/H264. 

>itself is
>While it is true that JVT agreed to add the IDRPICID field 
>during the last call comments period, it did not agree to make 
>the TL0PICIDX/IDRPICID combination optional in the PACSI 
>header.  As I am sure you recall, the PACSI header was the 
>subject of considerable deliberation in JVT, and its contents 
>in the previous version of this draft reflected a careful 
>balance between what should be in the video stream and what 
>should be in the RTP payload.  This includes the TL0PICIDX as 
>well as the associated flags (which, for those who did not 
>participate in the JVT discussions, are necessary to avoid 
>displaying broken video in the presence of packet losses).
>

JVT has no right to make decision on this draft. The decision in the
draft was made and agreed by the authors. Now it is up to AVT to finally
decide whether to have the stuff optional or not. I have mentioned the
benefit of having them as optional. Are there any problems or
disadvantages for having them optional?

>temporal layering, i.e. if an enhancement layer session adds 
>additional access units in addition to those in the base layer 
>sessions, and does not have additional real NAL units for the 
>access units present in the lower layers.
>

First, temporal scalability is NOT UNIMPORTANT. 
Second, for the "no CL-DON mode", dummy NAL units are needed whenever
there are two access units continuous in decoding order that do not
share any RTP session.

>There is ambiguity (I wouldn't say "non-determinism") only if 
>every packet of an access unit is lost from one of the RTP 
>sessions.  As long as you have at least one packet from each 
>access unit, you have enough information to reconstruct a 
>total (decode) ordering of the access units.  (Decode ordering 
>of NAL units within an access unit should be deterministic, 
>modulo isomorphisms.)
>
>The description of the decode order reconstruction algorithm 
>could be cleaned up, though.
>

I said "non-deterministic" because the process with packet losses has
not yet been clearly specified. As such, different implementations have
to guess what to do when packet loss exists. 

You are right that reception of a single packet for each access unit is
enough for the process. However, that does not resolve the disadvantage
of the performance issue, because higher-session (>=S_N) packets are
discarded whenever a single access unit is lost in session S_N. 

>timestamps, and certainly session-associated FEC and RTX do.

I am not sure whether those are of the same essense as the context here.
However, at least layered multicast of scalable audio is the same. For
that, it seems that Colin has some concern with RTP timestamp
synchonization therein, see the attached email. 

BR, YK

Subject: Re: [AVT] RE: I-D ACTION:draft-lakaniemi-avt-rtp-evbr-00.txt
From: <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Date: Fri, 16 Nov 2007 15:24:29 +0200

Ari,

On 16 Nov 2007, at 12:38, <[email-masked]> wrote:
> Thanks for the feedback. See some comments in-line.
>
>> Section 3.5 discusses RTP header usage, stating "In case the EV-VBR
>> session consists multiple RTP sessions, the RTP sessions are further
>> separated by using different payload type (PT) values for each of the
>> RTP streams" and "Each of the RTP sessions involved in the EV-VBR
>> session SHALL use the same SSRC value". A better approach would be to
>> allow the usual choice of SSRC, and use a common RTCP CNAME to
>> associate the streams, reusing the mechanism needed for lip-sync
>> between audio and video.
>
> The intention is not to specify anything too exotic regarding the PT
> and/or SSRC usage. My thinking here was that since the audio data
> carried by the set of RTP streams originates from the same source, it
> seemed logical to use the same SSRC value for each of the RTP  
> streams -- which would also tie the streams together with the data  
> present in the RTP header.
>
> Anyway, I realize the proposed SSRC handling is not fully in line with
> the RTP spec ("This identifier [SSRC] SHOULD be chosen randomly")  
> and I
> will change this part for the next revision if it is seen problematic.
> Do you see some unfavourable consequences with the proposed approach?

Yes - one can no longer use standard RTP receiver code, using the  
RTCP CNAME to group related streams (specified in RFC 3550), to  
process this payload format. Instead, you have a special case, which  
recognises this payload, and matches based on SSRC.

> A related issue is the timestamp handling accross RTP sessions. Since
> all the layers of a certain EV-VBR frame come from a single audio  
> source and they will be combined back into a single frame in the  
> receiving end prior to decoding, using the same timestamp value for  
> the layers of the same encoded frame accross RTP sessions would  
> simplify the data handling. Is this seen problematic?

Again, it introduces a special case into the receiver code. Rather  
than use the standard mechanism to synchronise streams, one has to  
recognise this payload format, and do something different.

We have standard mechanisms to time synchronise and group flows  
across multiple RTP sessions. This payload format should use them,  
rather than trying to reinvent the wheel in an incompatible way.

Regards,
Colin

---

From: <[email-masked]>
Date: Thu, 29 Nov 2007 10:29:47 +0200

The backward compatibility issue regarding the single NAL unit mode has
been well taken and addressed for non-layered-multicast use cases.

However, for layered-multicast use cases, where an video/H264 RTP stream
and one more more video/H264-SVC RTP streamss are conveyed in different
RTP sessions, use of single NAL unit mode for the video/H264 RTP stream
is very ineffcient because of the overhead caused by encapsulating each
prefix NAL unit in its own packet. Prefix NAL unit is of small size, and
it must be present for each coded slice NAL unit in the video/H264
stream. 

Moreover, are there really commercial multicast-capable receivers that
only support the Single NAL Unit Mode? For which applications and
network environments are those? The multicast-capable systems that we
are aware of, namely DVB-H and MBMS, require support of non-interleaved
mode - and even interleaved mode in case of MBMS.

BR, YK 

>From: ext Randell Jesup [mailto:[email-masked]] 
>Sent: Wednesday, November 28, 2007 10:18 PM
>To: Even, Roni
>Cc: Jonathan Lennox; [email-masked]
>Subject: Re: [AVT] draft-ietf-avt-rtp-svc-03.txt and RFC 3984
>
>It would be quite useful to do that; otherwise you have to 
>offer the base stream in H264-SVC, and a separate base stream 
>in H264 so that non-SVC-aware devices can talk to you.  
>Existing SIP H264 realtime implementations often support 
>either only mode 0, modes 0 and 1, or occasionally modes 0-2.  
>
>Video conferencing maybe, but 1-to-1 videophones are mostly 
>SIP, as well as most (almost all?) PC H.264 softclients.  That 
>said, most don't handle sprop-parameter-sets very well if at all.
>
>--
>Randell Jesup, Worldgate (developers of the Ojo videophone), 
>ex-Amiga OS team [email-masked] "The fetters imposed on 
>liberty at home have ever been forged out of the weapons

---

From: "Even, Roni" <[email-masked]>
Date: Thu, 29 Nov 2007 15:31:38 +0200

Randell,

I understand why most H.264 video conferencing will not use
sprop-parameter-sets. I think this was not a good idea. Synchronizing
between signaling and media is not there. 
Using re-invite for changing sets or as a response for FIR is not the
best solution. We moved all media control requests (fast update) to the
media path and now you have to send the parameter sets in the signaling
path so what was gained, we lost more since INFO has less overhead on
the SIP UA then the re-Invite

Roni 

> From: Randell Jesup [mailto:[email-masked]]
> Cc: [email-masked]; Jonathan Lennox
> Video conferencing maybe, but 1-to-1 videophones are mostly SIP, as
well
> as
> Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga
OS
> team
> [email-masked]
> "The fetters imposed on liberty at home have ever been forged out of
the
> weapons
> provided for defence against real, pretended, or imaginary dangers
from
> abroad."

---

From: Randell Jesup <[email-masked]>
Date: Thu, 29 Nov 2007 12:13:04 -0500

>been well taken and addressed for non-layered-multicast use cases.

Excellent.

>it must be present for each coded slice NAL unit in the video/H264

That makes sense (given the prefix-NAL requirement).  Obviously in mode 1
they can be aggregated into a STAP.

>mode - and even interleaved mode in case of MBMS.

Multicast-capable mode-0 only?  Not that I know of at the moment though
it's possible, though unlikely, that one or two are; as I mentioned
(indirectly) I was mostly talking to point-to-point use (videophones).

---

From: Randell Jesup <[email-masked]>
Date: Thu, 29 Nov 2007 12:23:52 -0500

>between signaling and media is not there.

So how then do you guarantee the parameter set is available?  (And that
it's the 'right' one)?  In-stream use, though common, mostly works when
there's only a single set and they never, ever change.  It also wastes
bandwidth.

If you're changing/adding parameter sets in-stream, you're betting on luck
that they get through, and probably sending them on every IDR (and perhaps
either forcing extra IDRs or extra SPS/PPS's into the stream, in case they
got lost).  

Or you have to come up with a separate reliable transport over RTP and
RTCP, or re-use/mis-use some other SIP command to change them (INFO).

>best solution.

Who would use re-INVITE as a response to FIR?  Either they're in-stream (in
opposition to the RFC, though harmless if they're the same as the
sprop-parameter-set value), in which case you can repeat them in response
to an FIR, or they were in signalling to start with, in which case you
don't need to re-INVITE to send them.

>We moved all media control requests (fast update) to the

Parameter sets need a reliable transport, unlike fast-update.  Fast-update
has nothing to do with parameter sets.  And are you saying we should send
parameter updates via INFO?  We could do that, and it can be reliable and
lower-overhead than re-INVITE, but I've never seen a spec for it, and also
how often do you expect to change parameter sets?  It also could complicate
things like B2BUA's, I'd think.

---

From: "Even, Roni" <[email-masked]>
Date: Thu, 29 Nov 2007 20:34:19 +0200

Randell,
A FIR is used to get a synchronization point. For H.263 and H.261 it was
full picture. But for H.264 you need the parameter set. The assumption
is that there was for example a video switch so a party connected to the
video in the middle of the stream.
Now the source gets a FIR, what is his behavior, does it sent the
parameter sets in the signaling or in the media path. 

Roni 

> Sent: Thursday, November 29, 2007 7:24 PM
> To: Even, Roni
> Subject: Re: [AVT] draft-ietf-avt-rtp-svc-03.txt and RFC 3984 (sprop-
> parameter-sets)
> 
> Who would use re-INVITE as a response to FIR?  Either they're
in-stream
> (in
OS
the
from

---

From: Randell Jesup <[email-masked]>
Date: Thu, 29 Nov 2007 14:28:20 -0500

>video in the middle of the stream.

I wouldn't assume FIR implied a video source change (I can understand why
that might be a frequent case for true conferencing, but it's far from the
only reason - think packet loss).

Also, FIR is an upstream message, not downstream.  I assume you're thinking
the video switch sends a FIR to the (newly-chosen) source for the switch's
output, and then (when it gets an IDR from the source) actually switches
the output.

Putting the PPS/SPS in-stream as if it was the same source is asking for
problems if one of the packets is lost (since then the decoder will use the
last set of parameters it had for that index, or barf if it doesn't have
that index).  

A better solution is to put each source on a different payload value, which
means that the SPS/PPS for each possible (real) source can be pre-specified
(via INVITE/reINVITE when a party is added), allowing on-the-fly switching
with no danger of lost PPS/SPS.  You can even minimize the overhead by
sharing the same PT for all sources who use the same (or a compatible) set
of fmtp parameters.  (There is a slight danger of a mildly garbled screen
if the initial IDR is trashed until the client uses some feedback mechanism
(CCM/FIR) to request anotehr IDR, but that's minor.)

In most conferencing situations, I'd assume that the majority of
participants would be using compatible equipment and settings and so be
able to share the same PT and SPS/PPS.

>parameter sets in the signaling or in the media path. 

Neither... :-)  It communicated the SPS/PPS already via a reliable channel,
so the receiver (in the case you mention the video switch, in
point-to-point cases the other end) already are known to have them, and
the RFC says not to use them.

---

From: "Even, Roni" <[email-masked]>
Date: Thu, 29 Nov 2007 22:37:32 +0200

Randell,
We went through this discussion in H.323. If you send all parameter sets
in advance, you need the signaling for it and you also need to tell the
receiver which one to use since all streams will come through the same
port
As for losing the parameter sets on the media streams, this is similar
to losing IDR still you do not send IDR on the signaling path
Roni 

> Sent: Thursday, November 29, 2007 9:28 PM
> To: Even, Roni
> problems if one of the packets is lost (since then the decoder will
use
OS
the
from

---

From: <[email-masked]>
Date: Thu, 29 Nov 2007 23:56:12 +0200

>We went through this discussion in H.323. If you send all 
>parameter sets in advance, you need the signaling for it and 
>you also need to tell the receiver which one to use since all 
>streams will come through the same port As for losing the 
>parameter sets on the media streams, this is similar to losing 
>IDR still you do not send IDR on the signaling path Roni 

Once you have sent all parameter sets in advance, then the bitsream
itself contains which parameter set to use, as the used picture
parameter set identifier is included in the slice header, and the used
sequence parameter set identifer is included in the picture parameter
set - hence indrectly referred to by the slice header. 

Therefore, this is different from IDR picture, where there is no in-band
synchronization. Meaning that even you have transmitted all the IDR
pictures out-of-band in advance, you don't know the position of each IDR
picture. 

That's why it is suitable to transmit parameter sets out-of-band, but
IDR pictures in-band. 

BR, YK

---

From: Randell Jesup <[email-masked]>
Date: Thu, 29 Nov 2007 17:37:07 -0500

>receiver which one to use since all streams will come through the same

Sure, so?  This is what things like alternate PT's are for.  We have
signalling for it.  It works.  Per the RFC, if you want to offer multiple
profile-level-idc's or multiple sprop-parameter-sets, you need to offer
multiple PT's for the RTP stream.  From the PT you know what parameter
set(s) to use.  If the sender changes PT, the receiver needs to install the
correct parameter set(s) and re-initialize the decoder/etc to handle the
stream (level, packetization, resolution, etc).  So I don't see a problem
here.  See my description on how a video switch could make use of this
without re-INVITES.  At most you re-INVITE once whenever you add a new
participant who has a profile-level or parameter-set that doesn't already
exist in the conference description.

>to losing IDR still you do not send IDR on the signaling path

And how is this is an argument for sending the parameter sets via RTP?

There are mechanisms for recovering from a lost IDR - FIR/CCM (heck, even
INFO).  Yes, you could use the same mechanism for recovering from a lost
SPS/PPS (if you mandate they ALWAYS are sent on an IDR - a waste of bits
IMHO).  But I don't see any *gain* from doing so other than a slight
decrease in complexity for a video switch.  The SPS and PPS for mid-call
IDRs are 99.999% likely to be identical to the start-of-call ones
(sprop-parameter-sets).  Even in a video-switch case, it's probably 99%.
And in packetization mode 0 (which is common for interactive use), SPS and
PPS add two full packets to each IDR.  At low bitrates (not the sort you
probably use for videoconferences) and at moderate packet loss rates (think
wireless), even two short packets could be a significant hit to end-to-end
quality.

---

From: "Thomas Schierl" <[email-masked]>
Date: Thu, 29 Nov 2007 18:58:55 -0800

Hi Ye-Kui, all,

Just a few comments on the non-DON reordering mode below.

Thomas Schierl
--------------
Fraunhofer HHI

> Sent: Wednesday, November 28, 2007 4:43 PM
> To: [email-masked]; [email-masked]
> Subject: RE: [AVT] I-D ACTION:draft-ietf-avt-rtp-svc-03.txt
>
>
...
> additional

For sure, plain temporal scalability is important, but I have the feeling
that it will definitely play a less important role in layered multicast over
wireless broadcast channels.
Plain temporal scalability may be rather used for graceful degradation than
for format extension, which will be the main use case for layered multicast
over wireless broadcast channels.

> there are two access units continuous in decoding order that do not

But, that is what plain temporal scalability means.

> to guess what to do when packet loss exists.

I think that will not be part of a normative section. That is an
implementation issue.

> discarded whenever a single access unit is lost in session S_N.

That is also an implementation issue and can be done better than described
in the draft.
You can think of an algorithm which can work with a packet loss in a session
and can still make use of the NAL units in higher sessions.

> synchonization therein, see the attached email.

The algorithm works with RTP timestamps as well as with media time stamps
derived by using RTCP.

...

---

From: "Even, Roni" <[email-masked]>
Date: Fri, 30 Nov 2007 07:55:14 +0200

Hi,
The problem was for centralized multipoint where you can not send the
sets in advance to the receiver. Just to the bridge. 
That also causes a problem with the parameter sets numbers since if they
come from different sources there may be uniqueness problem. At one
stage Stephan even suggested that the bridge will renumber the parameter
sets.

Roni

> Sent: Thursday, November 29, 2007 11:56 PM
> To: Even, Roni; [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: RE: [AVT] draft-ietf-avt-rtp-svc-03.txt and RFC 3984(sprop-
> parameter-sets)
> 
>

---

From: [email-masked]
Date: Fri, 30 Nov 2007 06:51:38 +0100 (CET)

Hi Roni,
I see my name on an email... I can't help but hit the reply button.

A bit of history for those who did not attend JVT at the time:
-Parameter sets were originally proposed as out-of-band only
-In-band transmission was envisioned early for canned media and broadcast
-Video-switching MCUs in existence are stateless, when it comes to the
video bits.  This situation has historical roots.  It's IMHO a bad design
choice if one were to create an MCU concept today, but in the H.323 case
we had to deal with legacy, and with those who defended their
architectural choices on commercial grounds.  No crtizism here, just
observation.

This is where the whole issue comes from.

I will note that 3GPP PSM and MTSI, for example, rely on out-of-band
parameter sets.  This is a more modern, more error resilient, just simply
better design.  Personally, I wouldn't mind if the video conferencing
would adopt this for SIPish enpoints.  I personally don't see the argument
of statelessness as convincing any more--in a SIP environment, an MCU
carries so much state that it shouldn't matter to handle another couple of
KB of state for parameter sets.  A bigger (but purely implementation-)
problem may be that the parameter sets are conceptually associated with
the media path, whereas the majority of the MCU state is in the signaling.
 (And yes, this could cause hell of a headache in decomposite MCUs, but
should we really make design choices based on such niche devices?).

My two cents,
Stephan

---

From: "Even, Roni" <[email-masked]>
Date: Fri, 30 Nov 2007 11:59:48 +0200

Stephan,
My major concerns are 
1. Synchronization between the parameter sets and the payload. Mostly since they go on different paths. The parameter set numbering in not unique and payload types are not either. You can receive the same PT from different sources to the same transport address with same parameter set number

2. The issue of send end to end information on the signaling path with re-invite. In the case of FIR that will mean a Re-invite before sending the media since we explained the need for FIR not for packet loss but for synchronizing with a new source

Roni 

> From: [email-masked] [mailto:[email-masked]]
> Sent: Friday, November 30, 2007 7:52 AM
> To: Even, Roni
> Cc: [email-masked]; [email-masked]; [email-masked];
> [email-masked]

---

From: <[email-masked]>
Date: Fri, 30 Nov 2007 11:59:00 +0200

Hi Thomas, 

See inline, please. 

>For sure, plain temporal scalability is important, but I have 
>the feeling that it will definitely play a less important role 
>in layered multicast over wireless broadcast channels.
>Plain temporal scalability may be rather used for graceful 
>degradation than for format extension, which will be the main 
>use case for layered multicast over wireless broadcast channels.
>

I am not sure which type of the scalabilities will finally be loved and
used more by the market, we need to wait for quite a while to see.
However, for now, what we should do IMO is to provide good support for
all the main scalabilities (namely spatial, quality and temporal).

>But, that is what plain temporal scalability means.

If that is what "plain temporal scalabiity" means, then "plain temporal
scalabiity" surely can include also spatial and quality scalabilities.
That says, dummy NAL units may be needed anytime. 

>The algorithm works with RTP timestamps as well as with media 
>time stamps derived by using RTCP.
>

It is also possible to reply on RTCP based timestamp synchronization.
However, in this case, the receiver must wait for the reception of the
first RTCPs from all the sessions to start the de-packetization process.
This in practice may make many packets received in the beginning
uesless, in particular for low end-to-end delay applications. 

BR, YK

---

From: Randell Jesup <[email-masked]>
Date: Fri, 30 Nov 2007 11:17:24 -0500

>sets in advance to the receiver. Just to the bridge.

I don't understand why the bridge can't send the (all currently active)
parameter sets to the receiver when it connects, using the RFC standard of
separate PT.  Unless there's a requirement that it not know that for some
reason, which seems odd, or unless it need to handle so many unique param
sets that there aren't enough PTs (in which case I'd suggest a different
architecture anyways).

>stage Stephan even suggested that the bridge will renumber the parameter

Oooh, there's a painful idea.  :-)

Collision on param set numbers isn't just a chance, it's pretty much a
given (everyone starts at 0, and rarely uses more than a single set).

This is why you *must* segregate streams with conflicting paramsets by PT.
If you can't (why not?), you have to re-INVITE (and when doing so, since
the streams are incompatible you must change port numbers and/or PT values
to avoid problems, per normal SIP usage).

You cannot mix packets from streams with different param set definitions if
a packet loss could allow the receiver to try to interpret packets using
the "wrong" param set.  A single packet loss should not be able to
crash/blow up the receiver.  While in theory a receiver should handle
packets that reference a "wrong" parameter set, my bet is a lot of them
will do Bad Things if decoding data with the wrong param set, especially if
things like resolution or reference frames are wrong.  Decoders probably
handle missing param sets pretty well, but I wouldn't bet on wrong ones.

The more we discuss this and the more I think about it, the more I believe
that in-stream param sets are simply a really awful idea, especially for
multipoint sessions.  (For point-to-point, it's less bad in that if one is
lost, *and* you (virtually) never re-use a set index (i.e. increment the
set index each time you change it), at worst the receiver should ignore the
NALs that reference unknown parameter sets.  It's still not great, though.)

H.264 doesn't really seem to be designed around *changing* the definition
of a parameter set on the fly in a lossy stream - it supports it, but there
isn't a lot done to make sure that data isn't decoded using the wrong one
if an update is missed - it assumes someone is responsible for that -
i.e. reliable delivery.  Adding new ones (and abandoning the old one if
needed) is much safer.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 30 Nov 2007 11:46:06 -0500

"Even, Roni" <[email-masked]> writes:
>My major concerns are 
>1. Synchronization between the parameter sets and the payload. Mostly
>   since they go on different paths. The parameter set numbering in not
>   unique and payload types are not either. You can receive the same PT
>   from different sources to the same transport address with same
>   parameter set number

The MCU in negotiation with any new sender specifies
a) the port
b) profile-level(s)
c) the parameter set(s)
d) the PT to use for each parameter set/profile combination

In some cases, when sending data out an MCU might need to remap the
incoming PT to the correct outgoing PT for that receiver (though I think
for RFC 3984 the PTs are more restricted than for "normal" offer-answer).
Even if it's needed, that's not a big deal, and the MCU has to handle it
anyways for non-H.264 codecs and offer/answer.  (For example,
a receiver might insist on receiving H.263 on PT 99, while another might 
insist on receiving it on PT 34.)  So PT remapping has to be supported
anyways.

>2. The issue of send end to end information on the signaling path with
>   re-invite. In the case of FIR that will mean a Re-invite before sending
>   the media since we explained the need for FIR not for packet loss but
>   for synchronizing with a new source

Ummm... I still don't fully understand why a re-INVITE would be needed when
switching sources (what you referenced before).  It would be needed for one
case - where a *new* participant uses a level/paramset pair that doesn't
map to any existing level/paramset in use.

Perhaps the real objection here is to having to re-INVITE existing
participants when a new participant wants to use a new level/paramset pair.
That's needed anyways, because unless you re-invite, you don't know that
the existing participants can support that pair!  You can't just send me a
paramset that says "we were talking QCIF before, but now it's CIF, no
objections allowed, enjoy!" if I don't support CIF.  Well, you can, but I'm
not going to be able to decode it.  So re-INVITE is needed before allowing
someone to use a new unique paramset (let alone sending it to me), so that
shouldn't be the issue.

(My suspicion is that no existing MCU bothers with all this detail; it just
assumes all the participants will simply deal with random paramsets, since
most MCUs have been built assuming most if not all clients are from the
same company, and if something goes wrong, they have a human change the
params of the conference on the MCU by hand.)

---

From: Thomas Wiegand <[email-masked]>
Date: Fri, 30 Nov 2007 21:18:35 +0100

Let me make a few remarks about the CL-DON vs. no-CL-DON discussion,  
which leads to the proposal to remove the CL-DON approach.

The question that needs to be answered is, how can the decoding order  
of the SVC bitstream be established when temporal scalability is used  
over multiple RTP sessions.

ASSUMPTIONS

1) Multiple RTP sessions for SVC transport:
* We need to consider the generic case of transporting SVC in 2 (or  
more) RTP sessions, one of which is the H.264 base layer RTP sessions  
and the other(s) is (are) SVC enhancement layer RTP session(s).
* Streaming using layered multicast and with or w/o interleaving is  
one application, while low-delay communication without interleaving is  
another application that could be served by this setup. (interleaving  
= decoding order is not equal to sequence numbers)

2) H.264 base layer:
* When interleaving is used, the H.264 base layer RTP session  
conforming to RFC 3984 shall be operated using packetization mode 2.
* When no interleaving is used, the H.264 base layer RTP session  
conforming to RFC 3984 can be operated using packetization modes 0, 1  
or 2.

3) SVC enhancement layer:
* When interleaving is used, the SVC enhancement layer RTP session(s)  
shall be operated using packetization mode 2.
* When no interleaving is used, the SVC enhancement layer RTP  
session(s) can be operated using packetization modes 0, 1, or 2  
( according to the new draft)

4) PACSI:
* When packetization mode 0 is used, PACSI cannot be present.
* When packetization modes 1 or 2 are used, PACSI may be present; for  
3984-compliant devices, it would hopefully be ignored, but could be  
used by the SVC device.

5) CL-DON is part of the PACSI (according to the new draft).

CONCLUSIONS

A) When temporal scalability using multiple RTP sessions is used in  
combination with packetization mode 0 anywhere, PACSI and with that CL- 
DON is not present and the CL-DON solution is not working for  
establishing decoding order. Therefore, the no-CL-DON solution MUST be  
specified in the draft.

B) When interleaving is used in all RTP sessions, mode 2 must be used  
in all RTP sessions and the CL-DON solution is not needed since DONs  
are present and can be used the same way as CL-DONs for establishing  
decoding order.

This solution also works if the H.264 base layer RTP session can use  
mode 2 without interleaving. Then, mode 2 can be used in the SVC  
enhancement layer RTP sessions.

C) CL-DON is potentially only useful for the remaining case, i.e.
* when packetization mode 1 in the H.264 base layer RTP session (for  
legacy reasons) is used in combination with
* packetization modes 1 or 2 in the SVC enhancement layer RTP sessions  
and
* temporal scalability (i.e. having non-aligned access units).
This is a very special case.

This case can also be covered by the no-CL-DON approach, where the  
dummy NAL units can be merged into the same packet with other non- 
dummy NAL units causing very little overhead. This actually causes  
less overhead compared to the CL-DON overhead in the PACSI (occurring  
in all RTP sessions).

Since the dummy NAL units are not in separate packets, the error  
resilience issue is mitigated.

Hence, the CL-DON approach is not needed and should be removed, as the  
additional option is not justified.

Best Regards,
Thomas

---
Thomas Wiegand, Dr.-Ing.
Head, Image Communication Group
Image Processing Department
Fraunhofer Inst. for Telecommunications - HHI
Einsteinufer 37, D-10587 Berlin, Germany
Phone: +49 30 31002 617,
Mobile: +49 172 381 3814
http: iphome.hhi.de/wiegand
---

---

From: Stephan Wenger <[email-masked]>
Date: Fri, 30 Nov 2007 16:01:46 -0800

Hi Roni,
I know these two.  We have discussed them often.  Issue 1 can be  
handled, making statistical assumptions of the delays of the various  
paths, and not re-using previously used parameter set numbers.  This  
is the whole reason why we have a large numbering space for those  
numbers in the first place.  I have no good solution for problem #2.
Stephan

On Nov 30, 2007, at 1:59 AM, Even, Roni wrote:

> Stephan,
> My major concerns are
> 1. Synchronization between the parameter sets and the payload.  
> Mostly since they go on different paths. The parameter set  
> numbering in not unique and payload types are not either. You can  
> receive the same PT from different sources to the same transport  
> address with same parameter set number
>
> 2. The issue of send end to end information on the signaling path  
> with re-invite. In the case of FIR that will mean a Re-invite  
> before sending the media since we explained the need for FIR not  
> for packet loss but for synchronizing with a new source
>
> Roni
>
>>>> pictures out-of-band in advance, you don't know the position of  
>>>> each
>>> IDR
>>>> picture.
>>>>