Subject: [clue] Multiple Streams - RTP muxing?

From: Peter Musgrave <[email-masked]>
Date: Sat, 29 Jan 2011 02:51:52 -0500

All, 

CLUE needs to send multiple audio and video streams between endpoints. TIP (see imtc.org) is an existing (somewhat limited) TP interop protocol which made a decision to mux the multiple video data streams into a single RTP flow (i.e. send all to the same destination port) distinguishing them by SSRC id. Audio is treated similarly.

This has some significant advantages:
- middle boxes still see one video m-line and do not need to change
- ICE needs to run only once for the media stream
- encryption can cover all the payload with one key pair

The disadvantage is that there now needs to be a mapping of data to logical RTP "channels" and this needs to be conveyed somehow. If this is done in SDP then I would think mmusic work would be required to specify it.

I think the notion of a single, multiplexed video stream for a TP endpoint makes a lot of sense in the context of CLUE.

Are there reasons not to mux?

Regards, 

Peter Musgrave

---

From: Lorenzo Miniero <[email-masked]>
Date: Sat, 29 Jan 2011 09:35:16 +0100

Well, if I have to think about a reason not to, muxing/demuxing at an application level is surely less efficient than doing it at a transport level. Besides, the RTP channel could get "overloaded", which could be an issue with packet shapers/discarders (if any).

But I guess that, if those you mentioned are the advantages, that's not a real dealbreaker.

L.

On Sat, 29 Jan 2011 02:51:52 -0500
Peter Musgrave <[email-masked]> wrote:

> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

-- 
Lorenzo Miniero <[email-masked]>

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 31 Jan 2011 13:41:33 +0100

Hi,

(AS AVTCore Chair)

I want to inject into this discussion a request to be a bit more clear
on what people are considering. Because RTP has several multiplexing
points and some may even want to go furhter than the existing ones. So
can everyone please clarify what they are talking about.

Please keep in mind that RTP has certain point of where things are
intended to separated. This also affects the possibility to use existing
RTP tools.

So a primer in RTP terminology:

RTP Session: One SSRC space, normally determined based on transport
level ports. Each RTP session normally servers one media type and one
purpose. Using different media types (audio and video) will creates
issues for some tools and SDP based signalling.

SSRC, a single media stream source. Multiple SSRC can be used in the
same RTP session.

RTP payload type, identifies the encoding of the RTP payload. Thus
possibly identifying video screen sizes etc.

So I will start with asking Peter to clarify the below statement:

> I think the notion of a single, multiplexed video stream for a TP endpoint makes a lot of sense in the context of CLUE.

I can interpret this as meaning either of:

1. One RTP session, with one SSRC that contains the encoding of multiple
different original flows through either transcoding, or a new
multiplexing mechanism?

2. One RTP session, with one SSRC per original video stream.

So please be clear on what you are discussing and suggesting here.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 31 Jan 2011 07:52:23 -0500

Hi

I was thinking (2).  (one session for audio and one for video).  I'd be open
to (1) if it did not require transcoding.  However, I am not convinced a
totally new mechanism is needed.

Though it hasn't come up yet, I was also wondering about having all video
and audio streams from the same CLUE endpoint use the same clock offset, in
order to simplify lip-sync management with multiple audio and multiple video
streams.  Not sure about that, but I do see a lot of complexity in
determining the proper delays.

Stephen Botzko

On Mon, Jan 31, 2011 at 7:41 AM, Magnus Westerlund <
[email-masked]> wrote:

> > I think the notion of a single, multiplexed video stream for a TP
> endpoint makes a lot of sense in the context of CLUE.
>

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Mon, 31 Jan 2011 07:49:59 -0800

I was thinking (2) as well; however, one concern I have is scalability. As the number of screens increases from 3 or so to say 10 or 100, does the set of pros and cons change. I would hate to have 100 m-lines, but 100 streams each with its own SSRC multiplexed in a single RTP session may be programmatic as well.
I am hoping those more familiar with the complexities and limitations of RTP stream processing can chime in. Will we have no choice but to break it into multiple RTP sessions?

Thanks,
Charles

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Stephen Botzko
> Sent: Monday, January 31, 2011 4:52 AM
> To: Magnus Westerlund
> Cc: CLUE
> Subject: Re: [clue] Multiple Streams - RTP muxing?
> 
> Hi
> >>>
> I was thinking (2).  (one session for audio and one for video).  I'd be open to (1) if it did not
> require transcoding.  However, I am not convinced a totally new mechanism is needed.
> 
> Though it hasn't come up yet, I was also wondering about having all video and audio streams from the
> same CLUE endpoint use the same clock offset, in order to simplify lip-sync management with multiple
> audio and multiple video streams.  Not sure about that, but I do see a lot of complexity in
> On Mon, Jan 31, 2011 at 7:41 AM, Magnus Westerlund <[email-masked]> wrote:
> 
> 
> 	Hi,
> 
> 	(AS AVTCore Chair)
>

---

From: Christer Holmberg <[email-masked]>
Date: Mon, 31 Jan 2011 17:15:19 +0100

Hi,

We can discuss the pros and cons of RTP muxing etc, but before making any implementation decissions we first need to decide upon the use-cases and requirements.

Regards,

Christer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Charles Eckel (eckelcu)
Sent: 31. tammikuuta 2011 17:50
To: Stephen Botzko; Magnus Westerlund
Cc: CLUE
Subject: Re: [clue] Multiple Streams - RTP muxing?

Thanks,
Charles

> 1. One RTP session, with one SSRC that contains the encoding of 
> multiple different original flows through either transcoding, or a new 
> multiplexing mechanism?
> 
> I was thinking (2).  (one session for audio and one for video).  I'd 
> be open to (1) if it did not require transcoding.  However, I am not convinced a totally new mechanism is needed.
> 
> Though it hasn't come up yet, I was also wondering about having all 
> video and audio streams from the same CLUE endpoint use the same clock 
> offset, in order to simplify lip-sync management with multiple audio 
> and multiple video streams.  Not sure about that, but I do see a lot of complexity in determining the proper delays.
> 
> Stephen Botzko
> 
> 
>

---

From: aravind sethuraman <[email-masked]>
Date: Tue, 1 Feb 2011 00:33:41 +0530

Hi all,
if we chose a single RTP session for ALL video and AUDIO, how will we signal which SSRC conforms to which screen etc? Not via RTCP like TIP does.

One problem i have with not going with multiple m lines in SDP is how do we define at session set up the following
1. Interop with systems with no CLUE
2. interop with assymetric system - for instance, i am a h264 svc 4 screen system and i invite a system capable of 3 screens h264 avc but for some reason (either bandwidth restrictions etc), this system offers back a single screen on uplink but it can still receive the 4 screens which can be rendered on the 3 system's single screen. How can we represent this via pure RTP muxing and no multiple m lines?

thanks
aravind

On Jan 31, 2011, at 6:22 PM, Stephen Botzko wrote:

> I was thinking (2).  (one session for audio and one for video).  I'd be open to (1) if it did not require transcoding.  However, I am not convinced a totally new mechanism is needed.
> 
> Though it hasn't come up yet, I was also wondering about having all video and audio streams from the same CLUE endpoint use the same clock offset, in order to simplify lip-sync management with multiple audio and multiple video streams.  Not sure about that, but I do see a lot of complexity in determining the proper delays.
> 
> Stephen Botzko
> 
> 

Aravind Sethuraman
Software Architect
Teliris, Inc. | 55 Broadway, 14th Floor, New York, NY 10006 | C +1.917.355.0119 | O +1.212.490.1065 x1408 | F +1.212.269.2869 | AIM aravindsraman

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 31 Jan 2011 14:46:02 -0500

I am not sure what you mean by "pure RTP muxing".

The semantics of multiple streams is what CLUE was formed to define. From
the charter: "The working group will define the semantics, syntax, and
transport mechanism for communicating the necessary information."

I imagine this will involve new m lines, but is not limited to that.  Many
of the use cases require quite fluid behavior, so it is possible that
setting up a multiplexed RTP stream (with an m line!) whose stream content
is managed by a different mechanism than offer/answer is a better way to
go.  We will see as the work progresses.

Certainly interoperability with existing videoconferencing systems requires
m-lines that are compatible with what those systems do today.

Stephen Botzko

On Mon, Jan 31, 2011 at 2:03 PM, aravind sethuraman <
[email-masked]> wrote:

> Hi all,
> if we chose a single RTP session for ALL video and AUDIO, how will we
> signal which SSRC conforms to which screen etc? Not via RTCP like TIP does.
>
> 2. interop with assymetric system - for instance, i am a h264 svc 4 screen
> system and i invite a system capable of 3 screens h264 avc but for some
> reason (either bandwidth restrictions etc), this system offers back a single
> screen on uplink but it can still receive the 4 screens which can be
> rendered on the 3 system's single screen. How can we represent this via pure
> RTP muxing and no multiple m lines?
>
> thanks
> aravind
>
> I was thinking (2).  (one session for audio and one for video).  I'd be
> open to (1) if it did not require transcoding.  However, I am not convinced
> a totally new mechanism is needed.
>
> Teliris, Inc. | 55 Broadway, 14th Floor, New York, NY 10006 | C
> +1.917.355.0119 | O +1.212.490.1065 x1408 | F +1.212.269.2869 | AIM
> aravindsraman
>
>
>

---

From: aravind sethuraman <[email-masked]>
Date: Tue, 1 Feb 2011 01:23:08 +0530

by "pure RTP muxing", i intended to mean a single port of receipt and transfer of the multiplexed RTP streams .

On Feb 1, 2011, at 1:16 AM, Stephen Botzko wrote:

> The semantics of multiple streams is what CLUE was formed to define. From the charter: "The working group will define the semantics, syntax, and transport mechanism for communicating the necessary information."  
> 
> I imagine this will involve new m lines, but is not limited to that.  Many of the use cases require quite fluid behavior, so it is possible that setting up a multiplexed RTP stream (with an m line!) whose stream content is managed by a different mechanism than offer/answer is a better way to go.  We will see as the work progresses.
> 
> Certainly interoperability with existing videoconferencing systems requires m-lines that are compatible with what those systems do today.
> 
> Stephen Botzko
> 
> 
> 
> On Mon, Jan 31, 2011 at 2:03 PM, aravind sethuraman <[email-masked]> wrote:
> Hi all,

Aravind Sethuraman
Software Architect

---

From: Jonathan Lennox <[email-masked]>
Date: Mon, 31 Jan 2011 16:15:10 -0500

Our existing system does SSRC multiplexing for multipoint video conferencing (allowing client-side rendering of multiple video streams without requiring a transcoding middlebox).  It works quite nicely -- we can easily have sessions with hundreds of sources, each with a unique SSRC.  (Not all being sent at once, of course, for bandwidth reasons, but potentially with dozens of them being sent at a time, switching dynamically.)

As mentioned previously, this simplifies NAT/firewall traversal substantially.  It also makes asymmetric cases clearer -- if I have three cameras and you have two, it's very unclear to me how my m= lines should map to your m= lines if we wanted to use the single-source-per-session model, whereas for SSRC multiplexing it's straightforward.

Your RTP stack has to be implemented properly, of course -- e.g., using O(1) operations like hash tables to look up SSRC values, rather than O(n) operations like linked lists.  But in general, the cost of SSRC demux is pretty trivial compared to everything else that needs to be done in order to successfully decode and render video.

I strongly encourage this behavior for telepresence as well.

--
Jonathan Lennox
[email-masked]

---

From: Peter Musgrave <[email-masked]>
Date: Tue, 1 Feb 2011 04:52:38 +0100

I find the point you make about the dynamic SSRC content and source switching very compelling. 

I also find "real code" doing this at the level of hundreds a good proof point.

Peter Musgrave