Subject: Re: [clue] Multiple Streams - RTP muxing?

From: "Aravind Sethuraman" <[email-masked]>
Date: Sat, 29 Jan 2011 07:28:42 -0800

Hi,

Is not transport level RTP muxing and demuxing expensive on low end applications?
Can we not support both RTP muxing and multiple m lines as part of the session establishment?

Regards,
Aravind

--- [email-masked] wrote:

From: Lorenzo Miniero <[email-masked]>
To: Peter Musgrave <[email-masked]>
Cc: CLUE <[email-masked]>
Subject: Re: [clue] Multiple Streams - RTP muxing?
Date: Sat, 29 Jan 2011 09:35:16 +0100

Well, if I have to think about a reason not to, muxing/demuxing at an application level is surely less efficient than doing it at a transport level. Besides, the RTP channel could get "overloaded", which could be an issue with packet shapers/discarders (if any).

But I guess that, if those you mentioned are the advantages, that's not a real dealbreaker.

L.

On Sat, 29 Jan 2011 02:51:52 -0500
Peter Musgrave <[email-masked]> wrote:

> All, 
> 
> CLUE needs to send multiple audio and video streams between endpoints. TIP (see imtc.org) is an existing (somewhat limited) TP interop protocol which made a decision to mux the multiple video data streams into a single RTP flow (i.e. send all to the same destination port) distinguishing them by SSRC id. Audio is treated similarly.
> 
> This has some significant advantages:
> - middle boxes still see one video m-line and do not need to change
> - ICE needs to run only once for the media stream
> - encryption can cover all the payload with one key pair
> 
> The disadvantage is that there now needs to be a mapping of data to logical RTP "channels" and this needs to be conveyed somehow. If this is done in SDP then I would think mmusic work would be required to specify it.
> 
> I think the notion of a single, multiplexed video stream for a TP endpoint makes a lot of sense in the context of CLUE.
> 
> Are there reasons not to mux?
> 
> Regards, 
> 
> Peter Musgrave
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

-- 
Lorenzo Miniero <[email-masked]>
_______________________________________________
clue mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/clue

---

From: Stephen Botzko <[email-masked]>
Date: Sat, 29 Jan 2011 10:59:17 -0500

Is not transport level RTP muxing and demuxing expensive on low end
applications?
If the video/audio decoding is done in the same unit as the RTP
sender/receiver, then I don't see any more computation being needed.

If some or all the audio/video decoding is done in satellite units, then
there are some extra cycles that are needed to forward the RTP packets in
both directions.  This is very small compared to the real complexity, which
is in the audio/video codecs.

Transport level multiplexing has some other advantages, which I think are
more important than the cycles. NAT traversal is simpler, and session border
controllers require less call state.  And it is easier to manage bandwidth.
In some existing videoconferencing use cases, accurate RSVP flow specs are
impossible to craft, since the individual media flows are dynamic (though
the total is not).  With transport multiplexing, this problem goes away.

   Can we not support both RTP muxing and multiple m lines as part of the
session establishment?

We can, but we are talking about a *lot* of m lines, given the number of
streams and the number of different codec options these systems already use.
This is an option certainly, though it may be simpler for everyone if we
could agree to multiplex (or not).

Stephen Botzko

On Sat, Jan 29, 2011 at 10:28 AM, Aravind Sethuraman <
[email-masked]> wrote:

> Well, if I have to think about a reason not to, muxing/demuxing at an
> application level is surely less efficient than doing it at a transport
> level. Besides, the RTP channel could get "overloaded", which could be an
> issue with packet shapers/discarders (if any).
>
> > CLUE needs to send multiple audio and video streams between endpoints.
> TIP (see imtc.org) is an existing (somewhat limited) TP interop protocol
> which made a decision to mux the multiple video data streams into a single
> RTP flow (i.e. send all to the same destination port) distinguishing them by
> SSRC id. Audio is treated similarly.
> >
> > The disadvantage is that there now needs to be a mapping of data to
> logical RTP "channels" and this needs to be conveyed somehow. If this is
> done in SDP then I would think mmusic work would be required to specify it.
> >
> > I think the notion of a single, multiplexed video stream for a TP
> endpoint makes a lot of sense in the context of CLUE.
> >

---

From: Marshall Eubanks <[email-masked]>
Date: Sat, 29 Jan 2011 11:34:17 -0500

> Is not transport level RTP muxing and demuxing expensive on low end applications?

Yes. That is my worry here. Extracting one screen from a muxed set up screens would be harder for middleware. 

There also may be  technical issues about timing. If a video bridge is going to combine and mux two streams from two different
sources, say to send to a multi-screen TP unit, then won't the two clocks have to be reconciled ? 

> Can we not support both RTP muxing and multiple m lines as part of the session establishment?

Are you saying support both as either-or options ?

Regards
Marshall

---

From: [email-masked]
Date: Sat, 29 Jan 2011 17:04:25 +0000

Either or options.
The end point can decide what is the optimal way of receiving information about multiple streams.

Also @stephen, how much is too many m lines? Are they not restricted by an end points physical # of captures and/or display units?
Or are we considering rtp relay functionalities too?

Sent via BlackBerry by AT&T

-----Original Message-----
From: Marshall Eubanks <[email-masked]>
Date: Sat, 29 Jan 2011 11:34:17 
To: <[email-masked]>
Cc: Lorenzo Miniero<[email-masked]>; <[email-masked]>
Subject: Re: [clue] Multiple Streams - RTP muxing?

Regards
Marshall

---

From: Peter Musgrave <[email-masked]>
Date: Sat, 29 Jan 2011 23:32:41 +0100

Please, no, not BOTH. I would rather HAVE to do the one I like less than to
do both.

Too many issues get left as "do both" - and what results in interop chaos
since even if both MUST be done a vendor choses to emphasize one of the
approaches in their product and the other becomes a poor non-interoperable
cousin.

Peter

---

From: Peter Musgrave <[email-masked]>
Date: Sun, 30 Jan 2011 00:17:52 +0100

Hi, 

I agree with the scenarios you mention. I agree there will be a need for multiple streams. 

I was anticipating a discussion about *how* - even though the use cases are still not final because I'd like to see CLUE start to tackle some of the technical discussions which appear (to me) to be inevitable. If we can do this in parallel with use cases then we might have a chance of hitting our milestones....

In addition to doing the RTP muxing TIP also does the negotiation in RTCP - which I think for many people is  serious layer violation. 

I think RTP muxing is pragmatic and I am concerned that m-lines might get very convoluted if we do not mux. That said there still needs to be some kind of media stream description - so perhaps that convolution is inevitable.

Peter

On 2011-01-29, at 11:51 PM, Lauwers, Chris wrote:

> My apologies, I’m new to this list and I’m trying to  better understand what the scope is of this effort. I believe what kicked off this email thread again was Allyn’s suggestion (hi Allyn!) to finalize use cases. Such a document would help greatly to guide the discussion.
>  
> I can think of at least 4 scenarios of the top of my head that would require video endpoints to send/receive multiple video streams:
>  
> -          Multi-codec system (aka immersive telepresence systems). This seems to be where most of the discussion focuses on so far.
> -          Systems using a relaying MCU (aka video router) for multi-party conferences
> -          Endpoints exchanging both live (camera) video and presentation video (similar to H.239)
> -          Systems implementing multi-party conferences using a full-mesh paradigm
>  
> Not all of these scenarios are candidates for muxing at the transport layer, so likely endpoints may need to be able to handle multiple RTP streams anyway, which diminishes the appeal of RTP muxing.
>  
> Another question: if RTP muxing is deemed a good thing to do, is there some generally-agreed on set of arguments within this group for why TIP isn’t an adequate protocol for this purpose?
>  
> Again, apologies if I’m steering the conversation off-track.
>  
>  
> Chris Lauwers
> CTO, Avistar
>  
> Voice/Video:       sip:[email-masked]
> PSTN:                    +1 (650) 525.3352
>  
>  
>  
>  
>  
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter Musgrave
> Sent: Saturday, January 29, 2011 2:33 PM
> To: [email-masked]
> Cc: [email-masked]
> Please, no, not BOTH. I would rather HAVE to do the one I like less than to do both. 
>  
> Too many issues get left as "do both" - and what results in interop chaos since even if both MUST be done a vendor choses to emphasize one of the approaches in their product and the other becomes a poor non-interoperable cousin. 
>  
> Peter
> 
> On Sat, Jan 29, 2011 at 4:28 PM, Aravind Sethuraman <[email-masked]> wrote:
> Hi,
> 
> This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. 
> If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Sat, 29 Jan 2011 18:32:17 -0800

Hi Aravind, 
Low end devices will have just a single stream in any case, and require
something like a subset of composition to be sent to them.

On other points, SDP allows multiple m-lines, but there is no agreed
upon way to do the de-muxing, so I see that as where the work will be.
It would probably require a new RTP profile or some other mechanism, so
I think the work would be done in AVT. That said, it's certainly within
scope for CLUE to discuss whether this is desirable/needed for
telepresence.

Regards,
Allyn

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Aravind Sethuraman
> Sent: Saturday, January 29, 2011 7:29 AM
> To: Lorenzo Miniero
> Cc: [email-masked]
> level. Besides, the RTP channel could get "overloaded", which could be
> an issue with packet shapers/discarders (if any).
> 
> But I guess that, if those you mentioned are the advantages, that's
not
> a real dealbreaker.
> > CLUE needs to send multiple audio and video streams between
> endpoints. TIP (see imtc.org) is an existing (somewhat limited) TP
> interop protocol which made a decision to mux the multiple video data
> streams into a single RTP flow (i.e. send all to the same destination
> port) distinguishing them by SSRC id. Audio is treated similarly.
> >
> logical RTP "channels" and this needs to be conveyed somehow. If this
> is done in SDP then I would think mmusic work would be required to
> specify it.
> >

---

From: aravind sethuraman <[email-masked]>
Date: Sun, 30 Jan 2011 04:51:13 -0500

Allyn et al,
Consider a case where a low end device, say ipad is in a "telepresence" call with a 3 screen room.....
there are 2 options to this, 
either we send a single shot from the 3 screen room or we send the 3 camera shot....
In case we chose to send 3 camera shot, If we opt for RTP Muxing, then the process of demuxing on this low end appliance may be too expensive.

either way, i think there should be an option of  supporting either multiple m lines or muxing as a session option.

-aravind

>>> Are there reasons not to mux?

Aravind Sethuraman
Software Architect
Teliris, Inc. | 55 Broadway, 14th Floor, New York, NY 10006 | C +1.917.355.0119 | O +1.212.490.1065 x1408 | F +1.212.269.2869 | AIM aravindsraman

---

From: "Roni Even" <[email-masked]>
Date: Sun, 30 Jan 2011 12:39:47 +0200

Hi,
I think that the simple case is for the low end device to receive one
stream. Receiving three multiplexed stream is not helpful without knowing
what the streams are in order to have a meaningful display on the receiver.
This will depend if the low end device will understand the multi-stream
semantics that need to be specified in CLUE.
To send the 3 camera shots the sender will need to know if the receiver can
understand CLUE.

Roni Even

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> aravind sethuraman
> Sent: Sunday, January 30, 2011 11:51 AM
> To: Allyn Romanow (allyn)
> Cc: [email-masked]
> In case we chose to send 3 camera shot, If we opt for RTP Muxing, then
> the process of demuxing on this low end appliance may be too expensive.
> 
> either way, i think there should be an option of  supporting either
> multiple m lines or muxing as a session option.
> 
> -aravind
> 
> 
> 
> 
> Teliris, Inc. | 55 Broadway, 14th Floor, New York, NY 10006 | C
> +1.917.355.0119 | O +1.212.490.1065 x1408 | F +1.212.269.2869 | AIM
> aravindsraman
> 
>

---

From: Stephen Botzko <[email-masked]>
Date: Sun, 30 Jan 2011 06:57:41 -0500

I agree.  The CLUE device needs to signal a single-stream m line in any
case, otherwise there is no compatibility with existing videoconferencing.
(Actually it will need to offer two legacy streams, since existing
videoconferencing supports presentation video along with the room video).
Accepting the offer for the traditional m-line results in getting a single
stream (perhaps full room, or perhaps center camera).

The CLUE device also needs to offer multi-stream CLUE in some fashion -
whatever mechanism we use has to result in the legacy endpoint not accepting
the CLUE offer(s).  This could possibly be done in the initial SDP, or it
could be done in a second-phase negotiation somehow.

Offering CLUE streams in 2 forms (multiplexed and not multiplexed) results
in 3 sets of stream offers for video - legacy, CLUE multiplexed, and CLUE
non-multiplexed; and probably another 3 sets for the audio.

Like Peter, I don't want to do this both ways. I don't think the
hypothetical low-end endpoint that is (a) not legacy single stream, (b)
understands CLUE, (c) can encode/ decode multiple CLUE streams, (d) cannot
do the RTP multiplexing is real.  If there's consensus that it is real, then
we need to drop the multiplexing idea.  Though personally I'd like to hear
more justification on why this actually very capable device has this
surprising limitation.

Regards
Stephen Botzko

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Sun, 30 Jan 2011 14:31:54 -0800

Yeah, I agree with you Steve and Roni-

From: Stephen Botzko [mailto:[email-masked]] 
Sent: Sunday, January 30, 2011 3:58 AM
To: Roni Even
Cc: aravind sethuraman; Allyn Romanow (allyn); [email-masked]
Subject: Re: [clue] Multiple Streams - RTP muxing?

I agree.  The CLUE device needs to signal a single-stream m line in any
case, otherwise there is no compatibility with existing
videoconferencing. (Actually it will need to offer two legacy streams,
since existing videoconferencing supports presentation video along with
the room video). Accepting the offer for the traditional m-line results
in getting a single stream (perhaps full room, or perhaps center
camera).

The CLUE device also needs to offer multi-stream CLUE in some fashion -
whatever mechanism we use has to result in the legacy endpoint not
accepting the CLUE offer(s).  This could possibly be done in the initial
SDP, or it could be done in a second-phase negotiation somehow.

Offering CLUE streams in 2 forms (multiplexed and not multiplexed)
results in 3 sets of stream offers for video - legacy, CLUE multiplexed,
and CLUE non-multiplexed; and probably another 3 sets for the audio.  

Like Peter, I don't want to do this both ways. I don't think the
hypothetical low-end endpoint that is (a) not legacy single stream, (b)
understands CLUE, (c) can encode/ decode multiple CLUE streams, (d)
cannot do the RTP multiplexing is real.  If there's consensus that it is
real, then we need to drop the multiplexing idea.  Though personally I'd
like to hear more justification on why this actually very capable device
has this surprising limitation.

Regards
Stephen Botzko

On Sun, Jan 30, 2011 at 5:39 AM, Roni Even <[email-masked]>
wrote:

Hi,
I think that the simple case is for the low end device to receive one
stream. Receiving three multiplexed stream is not helpful without
knowing
what the streams are in order to have a meaningful display on the
receiver.
This will depend if the low end device will understand the multi-stream
semantics that need to be specified in CLUE.
To send the 3 camera shots the sender will need to know if the receiver
can
understand CLUE.

Roni Even

Of

> Consider a case where a low end device, say ipad is in a
"telepresence"
> call with a 3 screen room.....