Subject: [AVT] Layered codecs: use of PT to distinguish layers

From: Stephan Wenger <[email-masked]>
Date: Mon, 3 Apr 2006 09:54:42 +0200

Folks,
I want to feel the temperature with you about the use of the RTP  
payload type field to distinguish layers of a layered codec.  That  
is, in the same RTP session (IP/Port/SSRC), several layers would be  
sent as independent RTP streams, distinguished by the PT.  The main  
rationale is minimization of firewall pinholes.  The advantage of  
this solution, over putting the layer id into the payload header, is  
that the RTP header is not encrypted when using SRTP, which allows  
meaningful layer discarding by middleboxes.
It is understood that this verges on the border of "RTP payload  
multiplexing", which is not p.c. in AVT; however, I got encouraged by  
Colin's recent draft in DCCP, which proposes something similar for  
RTP and corresponding RTCP traffic.  So let me boldly enter this  
minefield again.
I would appreciate quick comments (over the next two days or so), as  
an endorsement of the idea by this WG could perhaps influence the  
design choices being made here in the committee which is  
standardizing the SVC layered video codec.  Thanks very much for your  
reaction.
Regards,
Stephan

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Kalleitner, Franz" <[email-masked]>
Date: Mon, 3 Apr 2006 12:27:30 +0200

hi, stefan, all. 

The mapping of SVC layers to different RTP sessions opens the door for simple scaling of SVC streams.

Please note: The term "RTP translator", is used instead of "MANE". 
In principle it should not matter, which term is used for an initial discussion. I hope so :)

The proposal might be of special interest for:

demanding less system performance for RTP translators. 
A fairly, light-weighted solution for an environment with restricted computing performance 
to run a full RTP-Translator unit.

For SVC streams that traverses slow bit-rate links, with or without varying 
throughput condition. Moreover, this would facilitate to implement a RTP-Translator 
function much easier.  In case of sensing congestion, packets could be simply
discarded or  truncated in order to adjust the packet steam to the current network  
conditions, throughput respectively  - More or less unaware to the encoded media.

However, according to my understanding following restriction would appear:

Due to the proposed layer assignment the support of interleaving might be restricted. 
That is, only a single SVC packet (NALU) can be transported with one RTP packet per RTP session. 
Except, we think about combinations of spatio-temporal or temporary-SNR or spartio-SNR or any 
other combination (access units). 
However, this would require an enhanced negotiation effort between sender and receiver.
(signaling effort: out-of-band (SDP), SEI messages, others, ... )

Reduced SVC coding efficiency because interlayer-prediction need to be switched off.
In general, the extended NALU proposal does not provide direct information if a SVC packet 
is used for interlayer prediction. According to the definition of DID, it must be assumed 
that at any temporal location, a picture of a smaller dependency_id value may be used for 
inter-layer prediction. 
For example, if one particular, RTP stream will be discarded, all higher RTP streams, that belong 
to the same GoP, need to be discarded as well, because interlayer prediction "MAY" occur; 
even no prediction is done at all. In fact, if reference frame(s) will be discarded arbitrarily,
it would cause annoying artifacts at the decoder out.  

Please note, for the latter it was assume that the hierarchical layered representation of the 
SVC stream, NALU respectively are mapped to RTP streams with increasing PT value. 

Slightly better to handle multicast and even broadcast configurations, since the receiving client 
need not to register to each multicast-group for each layer it requests to receive. 
Rather reduced signaling overhead, due to the missing IGMP (multicast only).

Another question to answer would be the maximum number of supported layers for a single SVC stream. 
Bearing in mind, the number of dynamic payload format assignment is limited from 96 to 127. 
Hence, at most 32 layers could be assigned to a SVC stream that is,
i.e., 4 spatial layers + 4 temporary layers + 2 SNR layers ( = 32 layers)
Pointing to the example above: It must be assumed that four temporary layers might be too less for 
high quality SVC video.

The proposal requires negotiating PT-values between sender and receiver.
Does it allow to map layers arbitrarily to any PT- value? According to a rule?
i.e., that force to map all present spatial layers to the PT-values first or the base layer, followed 
by a number of temporary layers. Where to place the SNR-layers?

Furthermore, FGS packets could be truncated anywhere. 
Truncateability need to be visible to RTP session that transmits FGS packets.

By the way: In case of packet truncation it would be necessary to update the parameters below RTP. 
i.e. at network layer update of length information and transport layer, update of CRC checksum. 
Furthermore SRTP will not be able to handle truncated SVC packets.

Well, FGS packets are allowed to be discarded too, thus truncation could be address just as an 
additional option at a narrowed functional scope - i.e. without supporting secure RTP.

Well, this e-mail addresses just some thoughts, while writing this text. However, a look forward
for a more detailed discussion.

cheers, franz 

-----Ursprüngliche Nachricht-----
Von: Stephan Wenger [mailto:[email-masked]] 
Gesendet: Montag, 03. April 2006 09:55
An: [email-masked]
Cc: Magnus Westerlund; Colin Perkins
Betreff: [AVT] Layered codecs: use of PT to distinguish layers

Folks,

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 3 Apr 2006 15:11:27 +0200

Hi Franz,

fairly long email here, requiring quite a bit of SVC expertise.   I  
would prefer not to discuss the details right now, as I'm busy at the  
SG16 meeting.

At this point I'm not proposing the detailed design.  I would just  
want to learn whether the group could possibly agree to the option of  
using the PT to distinguish between layers.  This would solve a whole  
bunch of problems we have now, perhaps at the expense of a limit in  
functionality.  Whether we authors take this route in the draft, is  
undecided.  It's also up to AVT to decide whether benefits (e.g. from  
an architectural viewpoint) warrant possible restrictions in codec  
functionality, based on a draft to be written.

Regards,
Stephan

On  3 Apr2006, at 12:27 , Kalleitner, Franz wrote:

> [...]

> Slightly better to handle multicast and even broadcast  
> configurations, since the receiving client

---

From: "Kalleitner, Franz" <[email-masked]>
Date: Mon, 3 Apr 2006 15:45:56 +0200

Hi Stefan,
 
Hi Stefan,

sorry for misinterpreting the initial thread.

Well, I think to map the layered media stream into RTP streams is quite a logical step since it would fairly exploit the scalability functions of SVC. I would agree that a set of problems could be solved, but at the other hand it would cause some limitations.  However, in order to prevent violating Internet rules, they might be acceptable.

cheers, franz

________________________________

Von: Stephan Wenger [mailto:[email-masked]]
Gesendet: Mo 03.04.2006 15:11
An: Kalleitner, Franz
Cc: [email-masked]; Magnus Westerlund; Colin Perkins; "Stefan Håkansson LK (LU/EAB)"; István Gódor
Betreff: Re: AW: [AVT] Layered codecs: use of PT to distinguish layers

Hi Franz,

Regards,
Stephan

---

From: Randell Jesup <[email-masked]>
Date: Mon, 03 Apr 2006 11:23:50 -0400

Stephan Wenger <[email-masked]> writes:
>At this point I'm not proposing the detailed design.  I would just  want to
>learn whether the group could possibly agree to the option of  using the PT
>to distinguish between layers.  This would solve a whole  bunch of problems
>we have now, perhaps at the expense of a limit in  functionality.  Whether
>we authors take this route in the draft, is  undecided.  It's also up to
>AVT to decide whether benefits (e.g. from  an architectural viewpoint)
>warrant possible restrictions in codec  functionality, based on a draft to
>be written.

Modulo any technical gotchas, I like using PT.  It has the advantage of
being simple and not requiring additional media streams/ports/etc.

As for the "number of PTs", two comments:
1) while not typically done, my understanding is you _can_ reuse the
   "pre-defined" PT values for other codecs via rtpmap.  Depending on the
   ones reused, you could confuse/trip up intermediate boxes that ignore
   rtpmap for things they think they understand (bad proxy!)
2) What if there's more than one scalable codec in a SIP/SDP session?
   Admittedly unusual, but should be considered.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]

---

From: "Michael F. Speer" <[email-masked]>
Date: Mon, 03 Apr 2006 10:09:04 -0700 (PDT)

All,

Actually, the current RTP specificaiton suggests more appropriate
methods for handling layered encodings.

The basic methodology suggests using different UDP port numbers and a 
differt session for each layer, but I believe the same SSRC on each
session.  

Please read the specification.

Michael

> Well, I think to map the layered media stream into RTP streams is quite a
> logical step since it would fairly exploit the scalability functions of SVC.
> I would agree that a set of problems could be solved, but at the other hand
> it would cause some limitations.  However, in order to prevent violating
> Internet rules, they might be acceptable.
> 
>  
> 
> cheers, franz
> 
> 
> Cc: [email-masked]; Magnus Westerlund; Colin Perkins; "Stefan Håkansson LK
> (LU/EAB)"; István Gódor Betreff: Re: AW: [AVT] Layered codecs: use of PT to
> distinguish layers
> 
> 
> 
> Hi Franz,
>

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 04 Apr 2006 10:12:31 +0200

Hi,

A few observations about the use of PT as layer demultiplexing point.

A) It has the potential to consume a lot of payload type code points in 
a session. I don't think this is a serious issue as long as we are 
talking about less than 20 layers. With the exception that configuring 
all the payload types will make the SDP very big. However it is likely 
that the SDP will be just as big for other solutions.

B) If PT is the only way then one losses the possibility of receiver 
driven congestion control by joining and leaving layers on different 
multicast addresses.

C) From an MANE perspective ports or PT is equivalent. The MANE will 
require the same configuration information to perform its work, if it 
detects which layer is received on ports or PTs does not matter. The 
gain is in the number of IP flows and used port numbers. Thus this gain 
will need to be traded with the other costs.

D) The handling of RTCP in MANEs may be more problematic when using PT. 
If they arrive on the same SSRC there is no possibility for the receiver 
to determine if the lost packets are thinned by the MANE or packet 
losses. With other mechanism the knowledge that lost packets belong to a 
specific layer can be taken into account when determining response to 
these losses.

Thus I am not totally against using PT. However it has some serious 
considerations that needs to be taken into account.

Cheers

Magnus

Stephan Wenger wrote:
> Folks,
> I want to feel the temperature with you about the use of the RTP payload 
> type field to distinguish layers of a layered codec.  That is, in the 
> same RTP session (IP/Port/SSRC), several layers would be sent as 
> independent RTP streams, distinguished by the PT.  The main rationale is 
> minimization of firewall pinholes.  The advantage of this solution, over 
> putting the layer id into the payload header, is that the RTP header is 
> not encrypted when using SRTP, which allows meaningful layer discarding 
> by middleboxes.
> Colin's recent draft in DCCP, which proposes something similar for RTP 
> and corresponding RTCP traffic.  So let me boldly enter this minefield 
> again.
> I would appreciate quick comments (over the next two days or so), as an 
> endorsement of the idea by this WG could perhaps influence the design 
> choices being made here in the committee which is standardizing the SVC 
> layered video codec.  Thanks very much for your reaction.
> Regards,
> Stephan
> 
> 
> 

-- 

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 4 Apr 2006 10:13:42 +0200

This is understood, but has the disadvantage that it s impractical in  
today's NATted and Firewalled world.  We are looking for a more  
pragmatic approach.
Stephan

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 4 Apr 2006 11:25:22 +0200

Hi Magnus,

thanks for this constructive comment.  You points A and C are well  
taken and part of the tradeoff discussion, on which a possible draft  
will have to elaborate.  Point B is a non-issue -- while I would want  
to consider PT-based multiplexing as an option, clearly it's not the  
sole solution.  We would never disallow classic layered transport  
utilizing one layer per RTP session (IP/Port combination with -- in  
this case -- identical SSRC, as Michael pointed out).  It's too sexy  
a concept :-)

Point D is also understood, and a serious concern.  My understanding  
is that once you mix several RTP sessions into one, you'll loose the  
session (or layer) specific feedback dimension.  It doesn't matter by  
what means you mix: PT type, god-forbid-SSRC, an RTP payload specific  
method signaled in the payload header, or by signal-based mixing  
(e.g. reconstruction and re-encoding).  Would you agree?  If we  
assume (for the sake of assumption) a hard requirement for a single  
firewall pinhole, you are screwed with respect to point D, no matter  
what you do.  Right?

For those who bothered to read the application scenario section of  
draft-wenger-avt-rtp-svc-01, let me note that the scenarios listed  
continue to apply.  We continue to expect a MANE to perform stream  
composition of layers received in their own RTP session, so to solve  
the firewall traversal problem.  The sole (?) argument for the use of  
PT over of a payload header-based information field lies simply in  
the fact that payload headers and payload may be SRTP encrypted,  
whereas the PT field in the RTP header is not.  Therefore, when using  
PT, the MANE could perform stream composition without being a trusted  
device.

Regards,
Stephan

On  4 Apr2006, at 10:12 , Magnus Westerlund wrote:

> A) It has the potential to consume a lot of payload type code  
> points in a session. I don't think this is a serious issue as long  
> as we are talking about less than 20 layers. With the exception  
> that configuring all the payload types will make the SDP very big.  
> However it is likely that the SDP will be just as big for other  
> solutions.
>
> B) If PT is the only way then one losses the possibility of  
> receiver driven congestion control by joining and leaving layers on  
> different multicast addresses.
>
> C) From an MANE perspective ports or PT is equivalent. The MANE  
> will require the same configuration information to perform its  
> work, if it detects which layer is received on ports or PTs does  
> not matter. The gain is in the number of IP flows and used port  
> numbers. Thus this gain will need to be traded with the other costs.
>
> D) The handling of RTCP in MANEs may be more problematic when using  
> PT. If they arrive on the same SSRC there is no possibility for the  
> receiver to determine if the lost packets are thinned by the MANE  
> or packet losses. With other mechanism the knowledge that lost  
> packets belong to a specific layer can be taken into account when  
> determining response to these losses.
>
>> is, in the same RTP session (IP/Port/SSRC), several layers would  
>> be sent as independent RTP streams, distinguished by the PT.  The  
>> main rationale is minimization of firewall pinholes.  The  
>> advantage of this solution, over putting the layer id into the  
>> payload header, is that the RTP header is not encrypted when using  
>> SRTP, which allows meaningful layer discarding by middleboxes.
>> multiplexing", which is not p.c. in AVT; however, I got encouraged  
>> by Colin's recent draft in DCCP, which proposes something similar  
>> for RTP and corresponding RTCP traffic.  So let me boldly enter  
>> this minefield again.
>> I would appreciate quick comments (over the next two days or so),  
>> as an endorsement of the idea by this WG could perhaps influence  
>> the design choices being made here in the committee which is  
>> standardizing the SVC layered video codec.  Thanks very much for  
>> your reaction.
>> Regards,
>> Stephan
>
>
> -- 
>
> Magnus Westerlund
>

---

From: Dave Singer <[email-masked]>
Date: Tue, 4 Apr 2006 15:00:28 +0100

Unless streams are separated on ports and have independent RTCP as 
well as independent RTP, the question of what to do about RTCP 
arises.  It also arises for any "thinnable" stream that cannot be 
separated on ports (e.g. a fine-grained scalable layer).  No matter 
what we decide here, we need to decide how thinning MANEs handle RTCP:
a) do nothing.  the reports suggest the client has lost data (bytes 
received is too small) but the percentage lost doesn't seem to match.
b) do forward notification -- tell the clients "there used to be 
bytes/packets here and I took them away" and have the client add that 
back in
c) do backward fix-up -- catch the RTCP reports on the way back and 
fix them up so as to claim the 'client' (now divided between the MANE 
and the downstream) received all the data either received at the 
client or discarded at the MANE.

there may be other approaches.
-- 
David Singer
Apple Computer/QuickTime

---

From: "Kalleitner, Franz" <[email-masked]>
Date: Tue, 4 Apr 2006 16:03:42 +0200

Hi Dave,

according to my understand the procedure is already defined by RFC3550 (RTP Translator functionality)

assuming that each RTP session (PT value) uses the same SSRC identifier. RTP translators need to change the packet count for RTCP SR. Hence, receiver should be confident if packets are lost. 

below the quoted text:

<<< The SSRC is left intact but the sender information MUST be modified if required by the translation.  If a
translator changes the data encoding, it MUST change the "sender's byte count" field.  If it also combines several data packets into one output packet, it MUST change the "sender's packet count" field. >>>

One could expect the same handling for the RTCP RR forwarded to the sender , even it is not explicitly mentioned. However, the latter may not of particular interest related to this discussion.

cheers, franz

Von: Dave Singer [mailto:[email-masked]]
Gesendet: Di 04.04.2006 16:00
An: Stephan Wenger; Magnus Westerlund
Cc: [email-masked]; Colin Perkins
Betreff: Re: [AVT] Re: Layered codecs: use of PT to distinguish layers

---

From: Colin Perkins <[email-masked]>
Date: Tue, 4 Apr 2006 19:24:56 +0100

On 4 Apr 2006, at 15:03, Kalleitner, Franz wrote:
> assuming that each RTP session (PT value) uses the same SSRC  
> identifier. RTP translators need to change the packet count for  
> RTCP SR. Hence, receiver should be confident if packets are lost.
>
> below the quoted text:
>
> <<< The SSRC is left intact but the sender information MUST be  
> modified if required by the translation.  If a
> translator changes the data encoding, it MUST change the "sender's  
> byte count" field.  If it also combines several data packets into  
> one output packet, it MUST change the "sender's packet count"  
> field. >>>
>
> One could expect the same handling for the RTCP RR forwarded to the  
> sender , even it is not explicitly mentioned. However, the latter  
> may not of particular interest related to this discussion.

Sure, but it's not particularly easy to do, especially if the streams  
are encrypted.

Colin

---

From: Colin Perkins <[email-masked]>
Date: Tue, 4 Apr 2006 19:32:22 +0100

On 4 Apr 2006, at 10:25, Stephan Wenger wrote:
> taken and part of the tradeoff discussion, on which a possible  
> draft will have to elaborate.  Point B is a non-issue -- while I  
> would want to consider PT-based multiplexing as an option, clearly  
> it's not the sole solution.  We would never disallow classic  
> layered transport utilizing one layer per RTP session (IP/Port  
> combination with -- in this case -- identical SSRC, as Michael  
> pointed out).  It's too sexy a concept :-)
>
> Point D is also understood, and a serious concern.  My  
> understanding is that once you mix several RTP sessions into one,  
> you'll loose the session (or layer) specific feedback dimension.   
> It doesn't matter by what means you mix: PT type, god-forbid-SSRC,  
> an RTP payload specific method signaled in the payload header, or  
> by signal-based mixing (e.g. reconstruction and re-encoding).   
> Would you agree?

Sure, but this is the reason why merging several streams is a  
problem, especially for a layered codec, where I'd expect reception  
quality feedback on the individual layers to be useful.

> If we assume (for the sake of assumption) a hard requirement for a  
> single firewall pinhole, you are screwed with respect to point D,  
> no matter what you do.  Right?

Why is this an assumption? Once you've gone to the trouble of  
implementing all 96 pages of ICE, running two instances of it can't  
be that much worse than running a single instance...

Or: if you have a hard requirement for a single pin-hole, then punch  
a hole for an IP-in-UDP tunnel, and run real RTP over it with as many  
ports as you need :-)

Colin

---

From: Randell Jesup <[email-masked]>
Date: Tue, 04 Apr 2006 17:09:06 -0400

Colin Perkins <[email-masked]> writes:
>> single firewall pinhole, you are screwed with respect to point D,  no
>> matter what you do.  Right?
>
>Why is this an assumption? Once you've gone to the trouble of  implementing
>all 96 pages of ICE, running two instances of it can't  be that much worse
>than running a single instance...

Well, there's a LOT of overhead per-port to set stuff up with ICE, though I
imagine once the environment is ferreted out subsequent ports _might_ be
faster.  (Gotcha is that you can have different issues on different ports
due to collisions, etc, which can cause the mapping style of the NAT to
change.)  A couple of ports - ok.  20 or 30 ports - ouch.

>Or: if you have a hard requirement for a single pin-hole, then punch  a
>hole for an IP-in-UDP tunnel, and run real RTP over it with as many  ports
>as you need :-)

So, add another layer to RTP that exists solely as a multiplex layer, but
with a lot of extra overhead.  :-) Not to say it's a bad idea - but I've
never been fond of the heavy use of ports in RTP as stream-identifiers.  It
saves a (very small) number of bits per packet, at best.  It made more
sense in pre-NAT days when it was first created, of course, but even then
there were limits on numbers of ports per machine - a server handling a lot
of SVC-type streams might have run out of ports in some (unusual) cases,
much the way http servers can have issues with ports.

Using ports as inherent stream identifiers simplifies the application
(slightly), but at a cost of complexity at the network level (STUN/ICE/etc).
Or so it seems to me (who wasn't involved in all the discussions :-)

---

From: Colin Perkins <[email-masked]>
Date: Wed, 5 Apr 2006 00:28:38 +0100

On 4 Apr 2006, at 22:09, Randell Jesup wrote:
>> implementing

I was more thinking of implementation complexity, but sure there are  
also issues with the bandwidth used and the time taken to complete  
the ICE negotiation if you're opening a lot of ports.

> Or so it seems to me (who wasn't involved in all the discussions :-)

Okay, I was perhaps a little frivolous with my initial answers, but  
there is a serious point: RTP was designed to use the lower layer  
port number as the de-multiplexing point between sessions, and to  
separate the layers of a layered coding session. If there is a need  
to change that assumption, and introduce an additional de- 
multiplexing point into RTP, it should be done in a manner that can  
be reused across applications in a general way, rather than in an  
application-specific manner.

Colin

---

From: Randell Jesup <[email-masked]>
Date: Tue, 04 Apr 2006 19:58:06 -0400

Colin Perkins <[email-masked]> writes:
>> (slightly), but at a cost of complexity at the network level (STUN/
>> ICE/etc).  Or so it seems to me (who wasn't involved in all the
>> discussions :-)
>
>Okay, I was perhaps a little frivolous with my initial answers, but there
>is a serious point: RTP was designed to use the lower layer port number as
>the de-multiplexing point between sessions, and to separate the layers of
>a layered coding session. If there is a need to change that assumption,
>and introduce an additional de- multiplexing point into RTP, it should be
>done in a manner that can be reused across applications in a general way,
>rather than in an application-specific manner.

Agreed.  A UDP tunnel (for all sorts of reasons) works but isn't a great
idea IMHO.

Would the canonical way to achieve this to be define a multiplexing
protocol for datagrams, and specify this as part of a c= line in the
SDP.  This would change the "port" value from the media line into a "stream
ID" on the multiplex.  Something like:

c=IN IP4/PLEX 23.45.67.89 1234
m=audio 2 RTP/AVP 0
m=video 4 RTP/AVP 99
a=rtpmap:99 H999/90000

That would put audio on multiplex "ports" 2 & 3, and video on 4 & 5.
All data would be sent to 23.45.67.89 port 1234.

This is more overhead than if it were designed into RTP - but it can be
used for any protocol, not just RTP.

Packet data could be something like:
[short plex_port]
[datagram]

Yes, this causes an MTU size drop, which is an issue for some.  This would
need to be handled at the sending end.  Perhaps (maybe) you could consider
optimizing this for the "common" case of "same as last packet", but you
still need at least a bit, and more likely a byte.  Probably better to
stick to two.

There would be no attempt to deal with duplication, ordering, timing or any
other issue at this layer.  KISS.  That's the job of the application these
datagrams get fed to.

If you do need to get fancier, you can add fragmentation, etc, but it gets
ugly fast, and starts approaching the complexity/overhead of UDP.