Subject: [clue] Does framework provide sufficient info for receiver?

From: Paul Kyzivat <[email-masked]>
Date: Tue, 10 Apr 2012 13:16:16 -0400

This is follow on to the discussion of mixed/composed attributes.
Restating what I said in the design team meeting today:

The fundamental problem in the scope of CLUE is, if you are an
endpoint, and you have received an advertisement, with some scenes
and captures, do you have sufficient information to select and map
the advertised scenes/captures onto the equipment you have?

If you have sufficient equipment (displays, speakers) with compatible 
geometry to directly map all of the captures from one audio and one 
video capture scene entry from each capture scene then maybe all is 
good. In this case ISTM that the mixed/composed attributes aren't needed.

If you don't have sufficient equipment to do that, then the job is 
harder, and more information is needed to figure out what to do. For 
instance:

- If you don't have suitable displays, then perhaps you can select a 
video capture scene entry and locally compose or switch some of the 
captures in order to produce a set of captures that does map to the 
available displays. The area of capture of each capture could be helpful 
to doing this, and perhaps the composed attribute as well.

- or rather than compose to fit your displays, you could switch. The 
area of capture and the composed/switched attributes can enter into the 
decision of which to switch on a single display and which to put on 
different displays.

- handling video from multiple scenes probably presents some added 
issues. By definition there is no specified spatial relationship between 
the captures of different scenes. In some sense this may make it easier.

- If you don't have sufficient speakers for all of the audio captures, 
then you have to decide to mix, switch, or drop some. The spatial 
information from the captures may help in deciding how to mix. Not 
evident that the mixed attribute helps with this - unless it is used to 
decide to switch rather than mix. Of course if you are mapping 
independent audio captures onto different speakers then they are being 
implicitly mixed.

- If you don't have speakers with similar spatial relationship to those 
of the audio captures, then you must decide whether to do sub-optimal 
assignments or mix. Again not clear if the mixed attribute helps with this.

- If you have audio from more than one scene, what should you do with 
it? Should you mix, even though you have no spatial relationships? Or 
should you switch based on active speaker? What would help you to decide?

The problems are superficially different for MCUs. But perhaps only 
superficially. It seems to me that when the MCU decides how to map its 
inputs into its advertisement, it has some virtual room layouts in mind. 
So for each virtual room layout it is addressing the same issues as a 
real endpoint with that layout. But it does have the added problem of 
deciding how to describe what it is advertising - e.g. when to apply the 
mixed/composed attributes.

Some interesting use cases for all of the above would be very helpful.

	Thanks,
	Paul

---

From: John Leslie <[email-masked]>
Date: Tue, 10 Apr 2012 17:09:21 -0400

> the advertised scenes/captures onto the equipment you have?

   Of course not!

   If we wanted that, we'd have to include for every composed scene
what it's composed from, and for every mixed audio what it's mixed
from.

> good. In this case ISTM that the mixed/composed attributes aren't

   It's unlikely that the "typical" telepresence conference will have
a separate screen for every possible video.

   Of course, the number of "capture scenes" _could_ be small enough
to give each it's own video monitor -- but I don't think that's the
general case we should design for.

   OTOH, with an appropriate "surround-sound" system, you can "place"
an arbitrarily large number of audio streams, yet make them easy to
distinguish.

> harder, and more information is needed to figure out what to do.

   Only marginally harder...

> For instance:

   One inevitable use case is filing your formal report from the airport.
The person doing so will have lousy audio, vastly inadequate video, and
typically inadequate bandwidth with excessive latency. S/he will need
some middlebox to compose a "barely-adequate video" and mono audio.

   Then there's the use case of filing you formal report from your home
office (when you're sick or somebody doesn't want to pay travel cost).
You may well have surround-sound and two or three large monitors, 20 Mbps
download with 40 msec RTT, and a decent-quality lavalier mike. You still
probably want some middlebox to compose your basic video, but you can
take individual streams as well to concentrate on particular people.

   Near the top end are the corporate telepresence rooms. They will
want all streams, and are likely to have a human being mixing them.

> - or rather than compose to fit your displays, you could switch...

   Folks _will_ do that -- I can't stop them. But I find it uninteresting.

> the captures of different scenes...

   Nonetheless, the participants will want to imagine such a relationship.

> then you have to decide to mix, switch, or drop some.

   There really isn't any such thing as "sufficient speakers", but
surround-sound should be considered "sufficient". You can always mix to
stereo or even mono: "you pays your money and you takes your choice".

> The spatial information from the captures may help in deciding how to
> mix.

   Absolutely!

> Not evident that the mixed attribute helps with this

   It help you know when mixing is less likely to work well.

> - unless it is used to decide to switch rather than mix.

   I'm not sure there is any such thing. "Mixing" frequently includes
"riding gain" to reduce background noise. Binary switching is just a
bad idea.

> Of course if you are mapping independent audio captures onto different
> speakers then they are being implicitly mixed.

   Not a useful way to think about it, IMHO. Human ears and brains can
sort out individual sources if there are enough speakers, while mixing
makes that harder.

> - If you don't have speakers with similar spatial relationship to
> those of the audio captures, then you must decide whether to do
> sub-optimal assignments or mix.

   I suppose _some_ telepresence system will hide dozens of speakers
in the telepresence room and try to do this -- to me it sounds like a
dreadful idea, but it's quite possible the people will adapt.

   "Sub-optimal" really doesn't have a clear meaning here. Even speakers
in the "wrong" left-to-right order while you can see the "correct" order
on-screen won't confuse the listener as much as speaker assignments
changing for no obvious reason.

> Again not clear if the mixed attribute helps with this.

   As an extreme example, you could always render "mixed" in mono.

> it? Should you mix, even though you have no spatial relationships?

   IMHO, you should _invent_ a spatial relationship in that case. Even
if each room invents a different spatial relationship, it will prove
less confusing.

> Or should you switch based on active speaker?

   Only if background-noise is a serious problem.

> What would help you to decide?

   It becomes quickly obvious to the listener!

> inputs into its advertisement, it has some virtual room layouts in mind. 

   I would expect so.

> deciding how to describe what it is advertising - e.g. when to apply
> the mixed/composed attributes.

   Pretty much always, unless it's feeding an exact copy of what it
receives.

> Some interesting use cases for all of the above would be very helpful.

   Did the cases I listed help?

--
John Leslie <[email-masked]>

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Tue, 10 Apr 2012 14:54:08 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> John Leslie
> Sent: Tuesday, April 10, 2012 2:09 PM
> To: Paul Kyzivat
> Cc: CLUE
> Subject: Re: [clue] Does framework provide sufficient info for
> receiver?
> 

Exactly, and that is what I was expecting to get out of framework
provided by CLUE. This was the comment I made in Paris.
If the CLUE WG has determined this is out of scope, then I will go
elsewhere for this.
Sorry I was not able to attend the design team meeting today. It seems
this subject was discussed, and I see the update on the tracker, but I
do not understand the outcome. Did the group decide it is out of scope,
or that more discussion is needed?

Cheers,
Charles
 
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 11 Apr 2012 11:05:42 -0400

Hi John,

Thanks for the comments. I've followed up inline.

On 4/10/12 5:09 PM, John Leslie wrote:
> Paul Kyzivat<[email-masked]>  wrote:
>>

I think you must have misunderstood me. Otherwise I wouldn't expect you 
to answer "Of course not". AFAIK the whole point of CLUE is to provide 
the information to make this possible.

My point in asking the question was so that we can identify what 
information we have overlooked that we need to include one way or another.

Clearly there are some tradeoffs. More information might support 
*better* mappings. So one thing to decide is how good is good enough.

> a separate screen for every possible video.

That isn't what I said. Consider what is perhaps a typical case, where 
the advertisement contains one scene with several alternative video and 
audio scene entries. Lets just consider the video. Then maybe there is 
one entry with three captures, and another with one (mixed/switched, 
whatever) capture. Then if you have three screens you can map all three 
captures in the first entry. If you have less than three screens then 
you can map the one capture from the 2nd entry onto one of your screens.

But what if there was no entry with a single capture? If you had two 
screens, would you have enough info about the three captures in the 
first entry to make a reasonable mapping onto your two screens? What 
information would we need to include in the advertisement so that it 
would be possible to do this mapping well?

> general case we should design for.

IIUC an endpoint will typically only advertise one or two scenes. One 
would correspond to the cameras and mics in its room. The other, if 
present, would correspond to a "presentation" that doesn't fit into the 
coordinate space of the room. There are cases for more, but they get 
more obscure.

So for a point-to-point clue call the recipient will only need to map 
one or two scenes. But when there is more than one it is more complex, 
requiring sharing of equipment.

The situation is more complex for an MCU. It will have as input all the 
scenes offered by all the endpoints. It then has to decide what to offer 
out to the endpoints. It could just pass through all the scenes it 
receives from all the endpoints, leaving the mapping problem entirely to 
the endpoints. (I don't think that is what most people have in mind, 
though some might.)

Or it could take on itself the job of mapping/mixing/switching all of 
those inputs and producing something more like what a typical endpoint 
would advertise - one or two scenes each with just a few captures.

Or it could do both - advertise its own simplified composite scenes and 
also all those provided by the endpoints. That would allow the endpoints 
to either take the easy way out, or else do it all themselves in a way 
that suits them. Suppose it did this. Would the endpoints that want to 
take the easy way be able to figure out which scenes and captures to use?

> some middlebox to compose a "barely-adequate video" and mono audio.

This is indeed an interesting case - one we have explicitly put in 
scope. If an MCU is already in the call then perhaps we can hope that is 
one of the entries it makes available.

If there isn't any MCU in the call, then the other endpoint might not 
advertise that option. Then what? Do we have a mechanism for that?

> want all streams, and are likely to have a human being mixing them.

I don't understand your point "are likely to have a human being mixing 
them". Can you explain further?

>     Folks _will_ do that -- I can't stop them. But I find it uninteresting.

??? We have been talking about switching a lot. Are you saying that is 
uninteresting?

If it makes sense for a middlebox to switch, doesn't it make equal sense 
for an endpoint to do so?

>     Nonetheless, the participants will want to imagine such a relationship.

ISTM that this needs further discussion.

> stereo or even mono: "you pays your money and you takes your choice".

Do we have enough information to do it "right" or "well"?
Right now the coordinate info is all optional. Does it need to be 
mandatory in order to enable this?

> "riding gain" to reduce background noise. Binary switching is just a

I have no special understanding of this. I'm just asking probing 
questions in hopes that it will result in the right decisions being made 
and the right stuff specified.

> sort out individual sources if there are enough speakers, while mixing

I have had the impression that "simple" clue telepresence rooms would be 
hoping to do direct mapping of captures to devices. If the assumption is 
that this will be an unsatisfactory approach and all endpoints should be 
prepared to do something more complex then it would be helpful to write 
that down somewhere - e.g. in the framework.

>     Pretty much always, unless it's feeding an exact copy of what it

Since I've seen people here describe cases when that isn't so, it would 
be helpful to have a more precise definition.

>     Did the cases I listed help?

I think they need to be worked out in much more detail:

- what equipment (input and output) is in each endpoint, and where.
- what is advertised and how the input equipment is mapped to
   the advertisement
- how each endpoint maps the advertised captures to its output equipment
   together with how the info in the advertisement allowed it to
   determine that mapping.

	Thanks,
	Paul

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Wed, 11 Apr 2012 18:55:16 +0200

During the CLUE design team meeting on webex yesterday we started to
discuss various use cases and if they are in scope or not for the
current work in CLUE. 

As a part of my response to Pauls questions I summarized the use cases
and grouped them into supported / not supported. I can dive into more
detail about the use cases if people are interested?

I would answer yes to the following use cases 
* Telepresence point to point, where the reproduction will happen in a
2D-plane for audio and video. Currently CLUE support announcing
capability of sending 1 - N capture (left to right) and a receiver can
request 1 - M streams for rendering (left to tight), which is the basic
for interoperability. 
* Transcoded conferencing, where the MCU act like an endpoint and
logically announces audio and video left to right and a receiver request
1 - M captures depending on how many screen and audio streams you
prefer. Typically a single video stream pr. screen and one audio pr.
screen (mono/stereo). Excluded is capabilities to select individual
speakers, the MCU is free to optimize what to compose and send to a
receiver.
* Generic multi source, sending and receiving named sources where the
user experience is controlled by an human being capable of reading
sources names, or control software written explicit to operate on the
named streams. The content attribute can be used to indicate a primary
role, RFC4796 uses 'main' and 'alt'. 

I would answer no, to the following use cases 
* Switched conferencing, where we assume that endpoints do scalable
video (either SVC or Simulcast), locally composited layout and audio
mixing. Limited work has been done on how to use Scalable video and how
to build mechanisms for complete composited layouts. 
* Advanced audio reproduction, a group of audio scenarios mentioned by
Jon Leslie and Johan Nielsen that requires more details during initial
signalling and more dynamic information for transferring dynamic
meta-information.  
* Multi view, as described in the CLUE use case document. 

(Other comments inline.)

Cheers 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Paul Kyzivat
Sent: 11. april 2012 17:06
To: John Leslie
Cc: CLUE
Subject: Re: [clue] Does framework provide sufficient info for receiver?

Hi John,

My point in asking the question was so that we can identify what
information we have overlooked that we need to include one way or
another.

Clearly there are some tradeoffs. More information might support
*better* mappings. So one thing to decide is how good is good enough.
[Espen] Agree with Paul, we clearly can support in CLUE the use cases
where a Telepresence room either offer a 1, 2 or 3 video captures, which
can be rendered on 1, 2 or 3 screens. 

[Espen] How a vendor build a room or endpoint is transparent from the
sender of media. As a sender you should not care if a room request 3
audio streams (left, center, right) and choose to mix and play out on a
single speaker. We can argue that this is not the best experience, but
still valid and typically done for PC-clients or other endpoints with a
single speaker. 

take the easy way be able to figure out which scenes and captures to
use?

[Espen] This sounds closer to a user experience discussions than a
protocol discussion. 

>> available displays. The area of capture of each capture could be 
>> helpful to doing this, and perhaps the composed attribute as well.
>
> The person doing so will have lousy audio, vastly inadequate video, 
> and typically inadequate bandwidth with excessive latency. S/he will 
> need some middlebox to compose a "barely-adequate video" and mono
audio.

If there isn't any MCU in the call, then the other endpoint might not
advertise that option. Then what? Do we have a mechanism for that?
[Espen] If an MCU detect low bandwidth or packet loss it can start using
its normal recovery mechanisms. That could lower bitrate for audio and
video (maybe by a SIP re-invite), change the CLUE offer to something
that fits inside the available bandwidth. Getting good quality audio
ande video is always a challenge, with or without clue. 

>     Then there's the use case of filing you formal report from your 
> home office (when you're sick or somebody doesn't want to pay travel
cost).
> You may well have surround-sound and two or three large monitors, 20 
> Mbps download with 40 msec RTT, and a decent-quality lavalier mike. 
> You still probably want some middlebox to compose your basic video, 
> but you can take individual streams as well to concentrate on
particular people.

I don't understand your point "are likely to have a human being mixing
them". Can you explain further?
[Espen] I see this as mostly a request for focusing in on particular
user. In a transcoded conference you can do that by looking into XCon,
if you do switched conference the endpoint typically render the layout
and can choose who to focus on.  Mixing transcoded and switched
conferencing I think makes it unnecessary complex for the initial use
cases to cover. 

>> issues. By definition there is no specified spatial relationship 
>> between the captures of different scenes...
>

>> - If you don't have sufficient speakers for all of the audio 
>> captures, then you have to decide to mix, switch, or drop some.
>
> surround-sound should be considered "sufficient". You can always mix 
> to stereo or even mono: "you pays your money and you takes your
choice".

>> Of course if you are mapping independent audio captures onto 
>> different speakers then they are being implicitly mixed.
>
>     Not a useful way to think about it, IMHO. Human ears and brains 
> can sort out individual sources if there are enough speakers, while 
> mixing makes that harder.

[Espen] Agree with Paul here, the basic scenario is well understood. A
sends two logical audio captures and a receiver play them out with the
same spatial relationship. As rule of thumb here could be that for basic
interoperability between vendors you assume pairs of audio and video
captures that can be easily rendered on matching pairs of screens and
speaker.  

>     "Sub-optimal" really doesn't have a clear meaning here. Even 
> speakers in the "wrong" left-to-right order while you can see the 
> "correct" order on-screen won't confuse the listener as much as 
> speaker assignments changing for no obvious reason.
>

>     IMHO, you should _invent_ a spatial relationship in that case. 
> Even if each room invents a different spatial relationship, it will 
> prove less confusing.
>
>> superficially. It seems to me that when the MCU decides how to map 
>> its inputs into its advertisement, it has some virtual room layouts
in mind.

[Espen] Agree, being practical about what we will offer for actual
endpoints are useful. 

	Thanks,
	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 12 Apr 2012 16:36:30 -0400

Espen - comments inline.

	Thanks,
	Paul

> request 1 - M streams for rendering (left to tight), which is the basic

In this case, suppose N > M. Who is responsible for deciding which M of 
the N streams will be used? AFAIK the recipient just picks the ones it 
wants. So then my question remains - does the recipient have sufficient 
info to select the "right" ones? (There isn't any guarantee that *any* N 
of the M streams will provide a reasonable rendering.

This would be improved if we had a mechanism for the recipient to state 
"I can only handle N streams" and the sender was obligated to meet that 
constraint. I don't think we have said that.

> speakers, the MCU is free to optimize what to compose and send to a

This is logically equivalent to the prior case.

> role, RFC4796 uses 'main' and 'alt'.

Are you talking about the text names/labels we were discussing on the 
last call?

If so, I agree that if the senders provide well conceived labels then a 
human can probably decide something reasonable.

If the human is working only with "content" attributes, then its not so 
clear. Its quite possible that the recipient will find itself trying to 
choose between multiple streams with the same content type. (E.g. all 
'main'.)

> can be rendered on 1, 2 or 3 screens.

I think its clear we can support cases where every advertisement that 
includes an entry for N (>1) captures also includes an entry with only 
one capture, and optionally entries containing other numbers of captures 
between 1 and N.

But as I mention above, its not clear we have supported case where the 
minimal entry in the advertisement is for N>1 captures.

> still valid and typically done for PC-clients or other endpoints with a

Again, I think you are assuming the receiver can specify the max number 
of streams it can handle, and that the sender will then be obligated to 
select/construct a suitable set of streams. AFAIK we have not written 
down anything that calls for that. (If I am wrong, please let me know 
where this is specified.)

We have the (largely hypothetical) recipient capabilities message, but 
its content hasn't been specified yet, and AFAIK there has been no 
statement that it is binding and the construction of the advertisement.

> [Espen] This sounds closer to a user experience discussions than a

Perhaps it is. But the point of CLUE is to develop a protocol that 
enables an especially good user experience. While we shouldn't mandate 
the user experience, I think we are obligated to do due diligence that 
the protocol is sufficient to enable that user experience.

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Fri, 13 Apr 2012 13:44:56 +0200

> request 1 - M streams for rendering (left to tight), which is the 
> basic for interoperability.

[Espen] A receiver requesting 2x streams to be rendered left, right,
will rely on the media sender to give a decent experience. E.g. the most
capable endpoint must translate its capabilities down to what lesser
capable room can do. 

[Espen] I see this as an important use case for interoperability. Its
fine for capable Telepresence room to do a advanced CLUE advertisement,
as long as there is an easy and straightforward way to request two
captures screens for left, right placed monitors. 

We should have a use cases to express that we want to support endpoints
that only support asking for 1 - N streams, rendered left to right. 

Cheers 

-Espen

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 13 Apr 2012 11:02:21 -0400

Changing the subject line. This is one specific point coming out of the 
thread on "Does framework provide sufficient info for receiver?"

Paraphrasing, I think Espen is calling for a requirement that the sender 
construct an advertisement that is consistent with receiver 
capabilities. This seems to require the (still undefined) Capabilities 
message, and to make it binding.

AFAIK we currently have no such requirement.
Do others think we should?

	Thanks,
	Paul

On 4/13/12 7:44 AM, Espen Berger (espeberg) wrote:
> In this case, suppose N>  M. Who is responsible for deciding which M of

---

From: Stephen Botzko <[email-masked]>
Date: Sat, 14 Apr 2012 13:23:03 -0400

On Fri, Apr 13, 2012 at 7:44 AM, Espen Berger (espeberg) <[email-masked]
> wrote:

> that only support asking for 1 - N streams, rendered left to right.
[sb] section 3.2 in the use case document covers the asymmetric case[/sb]

---

From: Paul Kyzivat <[email-masked]>
Date: Sat, 14 Apr 2012 17:11:14 -0400

Steven,

The use case doc does cover this case, but it doesn't say much about how 
the recipient (with fewer screens) decides what to display.

It doesn't even hint at what Espen was implying - that the recipient 
asked to sender to choose streams to meet the recipient's limitations.

	Thanks,
	Paul

On 4/14/12 1:23 PM, Stephen Botzko wrote:
>
>
> On Fri, Apr 13, 2012 at 7:44 AM, Espen Berger (espeberg)
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     [email-masked] <mailto:[email-masked]>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Sun, 15 Apr 2012 13:01:47 +0200

I was not thinking about adding the capability message back in. 

My comment was to make sure that a receiver of a CLUE advertisement
message can understand what the media consumers preferred configurations
are for a two screen endpoint. It cannot be only up to the two screen
endpoint to choose correctly between alternatives, the sender can send
some useful hints to make it easier to choose between alternatives.

-Espen 

-----Original Message-----
From: Paul Kyzivat [mailto:[email-masked]] 
Sent: 13. april 2012 17:02
To: Espen Berger (espeberg)
Cc: John Leslie; CLUE
Subject: Sender to comply with receiver limits on what it can receive
(?)

	Thanks,
	Paul

On 4/13/12 7:44 AM, Espen Berger (espeberg) wrote:
> In this case, suppose N>  M. Who is responsible for deciding which M 
> of the N streams will be used? AFAIK the recipient just picks the ones

> it wants. So then my question remains - does the recipient have 
> sufficient info to select the "right" ones? (There isn't any guarantee

> that *any* N of the M streams will provide a reasonable rendering.
>
> will rely on the media sender to give a decent experience. E.g. the 
> most capable endpoint must translate its capabilities down to what 
> lesser capable room can do.
>
> This would be improved if we had a mechanism for the recipient to 
> state "I can only handle N streams" and the sender was obligated to 
> meet that constraint. I don't think we have said that.
>
> fine for capable Telepresence room to do a advanced CLUE 
> advertisement, as long as there is an easy and straightforward way to 
> request two captures screens for left, right placed monitors.
>
> We should have a use cases to express that we want to support 
> endpoints that only support asking for 1 - N streams, rendered left to
right.

---

From: Stephen Botzko <[email-masked]>
Date: Sun, 15 Apr 2012 07:22:44 -0400

In my view, a nice sender would advertise exactly one 2-capture entry, and
send the captures it thinks are most useful.

Stephen

On Sun, Apr 15, 2012 at 7:01 AM, Espen Berger (espeberg) <[email-masked]

---

From: "Roni Even" <[email-masked]>
Date: Sun, 15 Apr 2012 18:20:05 +0300

Hi Paul,
I think that there is enough information that will allow a receiver to
select N out of M streams.
The receiver will know the semantics of the streams (people or presentation)
and the spatial information of the room and the different streams.
My view is that this is enough for the general telepresence point to point
use case.
As for multipoint use case, this is why I think it is important to know if
the advertisement is based on site switch or segment switch which will
provide the extra information needed for the selection (spatial information
provides the internal relation between the capture) and I think that the
audio level will help with knowing who are the active speakers in this case
Roni 

> Sent: Thursday, April 12, 2012 11:37 PM
> info to select the "right" ones? (There isn't any guarantee that *any*
> N of the M streams will provide a reasonable rendering.
> 
> > mixing. Limited work has been done on how to use Scalable video and
> > how to build mechanisms for complete composited layouts.
> > Of Paul Kyzivat
> > I think you must have misunderstood me. Otherwise I wouldn't expect
> > you to answer "Of course not". AFAIK the whole point of CLUE is to
> > provide the information to make this possible.
> >
> > where a Telepresence room either offer a 1, 2 or 3 video captures,
> > which can be rendered on 1, 2 or 3 screens.
> 
> one capture, and optionally entries containing other numbers of
> captures between 1 and N.
> 
> > the advertisement contains one scene with several alternative video
> > and audio scene entries. Lets just consider the video. Then maybe
> > there is one entry with three captures, and another with one
> > (mixed/switched,
> > whatever) capture. Then if you have three screens you can map all
> > three captures in the first entry. If you have less than three
> screens
> > then you can map the one capture from the 2nd entry onto one of your
> screens.
> >
> > audio streams (left, center, right) and choose to mix and play out on
> > a single speaker. We can argue that this is not the best experience,
> > but still valid and typically done for PC-clients or other endpoints
> > with a single speaker.
> 
> >>      Of course, the number of "capture scenes" _could_ be small
> >> enough to give each it's own video monitor -- but I don't think
> >> that's the general case we should design for.
> >
> > present, would correspond to a "presentation" that doesn't fit into
> > the coordinate space of the room. There are cases for more, but they
> > get more obscure.
> >
> > The situation is more complex for an MCU. It will have as input all
> > the scenes offered by all the endpoints. It then has to decide what
> to
> > offer out to the endpoints. It could just pass through all the scenes
> > it receives from all the endpoints, leaving the mapping problem
> > entirely to the endpoints. (I don't think that is what most people
> > have in mind, though some might.)
> >
> > Or it could do both - advertise its own simplified composite scenes
> > and also all those provided by the endpoints. That would allow the
> > endpoints to either take the easy way out, or else do it all
> > themselves in a way that suits them. Suppose it did this. Would the
> > endpoints that want to take the easy way be able to figure out which
> > scenes and captures to use?
> >
> > scope. If an MCU is already in the call then perhaps we can hope that
> > is one of the entries it makes available.
> >
> > [Espen] If an MCU detect low bandwidth or packet loss it can start
> > using its normal recovery mechanisms. That could lower bitrate for
> > audio and video (maybe by a SIP re-invite), change the CLUE offer to
> > something that fits inside the available bandwidth. Getting good
> > quality audio ande video is always a challenge, with or without clue.
> >
> > If it makes sense for a middlebox to switch, doesn't it make equal
> > sense for an endpoint to do so?
> >
> > questions in hopes that it will result in the right decisions being
> > made and the right stuff specified.
> >
> > I have had the impression that "simple" clue telepresence rooms would
> > be hoping to do direct mapping of captures to devices. If the
> > assumption is that this will be an unsatisfactory approach and all
> > endpoints should be prepared to do something more complex then it
> > would be helpful to write that down somewhere - e.g. in the
> framework.
> >
> > same spatial relationship. As rule of thumb here could be that for
> > basic interoperability between vendors you assume pairs of audio and
> > video captures that can be easily rendered on matching pairs of
> > screens and speaker.
> >
> >>      I suppose _some_ telepresence system will hide dozens of
> >> speakers in the telepresence room and try to do this -- to me it
> >> sounds like a dreadful idea, but it's quite possible the people will
> adapt.
> >>
> > Since I've seen people here describe cases when that isn't so, it
> > would be helpful to have a more precise definition.
> >

---

From: Stephen Botzko <[email-masked]>
Date: Sun, 15 Apr 2012 14:14:48 -0400

> It doesn't even hint at what Espen was implying - that the recipient asked
> to sender to choose streams to meet the recipient's limitations.
>
>        Thanks,
>        Paul
>
> [sb] What it says (in part) is

   1.  The 1 screen system might simply show only 1 of the 3 camera
       images, since the receiving side has only 1 screen.  Two people
       are seen at full size, but 4 people are not seen at all.  The
       choice of which 1 of the 3 streams to display could be fixed, or
       could be selected by the users.  It could also be made
       automatically based on who is speaking in the 3 screen room, such
       that the people in the 1 screen room always see the person who is
       speaking.  If the automatic selection is done at the sender, the
       transmission of streams that are not displayed could be
       suppressed, which would avoid wasting bandwidth.

This is a solution-free description of course, and perhaps more
possibilities could be outlined.   Though I think it covers the basic
ground.

Answering the original question "Does the recipient have sufficient info to
select the 'right' ones?" requires knowledge of the recipient's intent and
tacit assumptions on what is being advertised.  I think grounding this
discussion in tangible use cases (specific advertisements with specific
receiver limitations) would be more fruitful than trying to begin by
discussing it in the abstract. If those specifics are not covered in the
use cases, we should probably add them.[\sb]

>> N
>> most
>>    ______________________________**_________________
>>    clue mailing list
>>    https://www.ietf.org/mailman/**listinfo/clue<https://www.ietf.org/mailman/listinfo/clue>
>>
>>
>>
>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 16 Apr 2012 09:56:10 -0400

> I was not thinking about adding the capability message back in.

Hmm. AFAIK we didn't take it out. We just haven't worked on it much.

> some useful hints to make it easier to choose between alternatives.

Suppose the sender's preferred configuration is a 3-screen room plus a 
separate display for presentations. Is it expected to create capture 
scene entries for not only that, but for all interesting lesser 
configurations, so that a recipient need only choose a single entry?

This might mean the sender has to construct entries for all of:

- 4 screens (one for presentation)
- 3 screens
- 2 screens
- 1 screen

Alternatively, the recipient will need enough information to figure out 
what to do when none of the capture scene entries map well to the 
available equipment.

Note that this is more complex if there is more than one scene in the 
advertisement. E.g. the "natural" way to advertise this might be as one 
'main' scene and one 'presentation' scene, because the physical 
relationship of the presentation screens to the other scenes is 
irrelevant. But that means that the recipient must choose one capture 
scene entry from each scene, and those must map independently to local 
screens. If the recipient has four screens this is easy. But if the 
recipient has fewer than four screens, then it is a problem, since it 
then becomes hard to share one screen for captures from two capture 
scene entries.

I have a gut feeling that something is not quite right here - it just 
doesn't hang together smoothly. But I'm not certain precisely what is wrong.

	Thanks,
	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 16 Apr 2012 10:11:47 -0400

Please see reply inline.

On 4/15/12 2:14 PM, Stephen Botzko wrote:
>
>
> On Sat, Apr 14, 2012 at 5:11 PM, Paul Kyzivat <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Steven,
>
>     The use case doc does cover this case, but it doesn't say much about
>     how the recipient (with fewer screens) decides what to display.
>
>

Of course there are a number of ways to cope with insufficient hw to 
display everything. Of those, some (more than one) are plausible, and 
some others would probably be judged stupid by any rational person.

> Answering the original question "Does the recipient have sufficient info
> to select the 'right' ones?" requires knowledge of the recipient's
> intent and tacit assumptions on what is being advertised.  I think
> grounding this discussion in tangible use cases (specific advertisements
> with specific receiver limitations) would be more fruitful than trying
> to begin by discussing it in the abstract. If those specifics are not
> covered in the use cases, we should probably add them.[\sb]

I agree. I already suggested that this be investigated via use cases. 
The existing use cases can be a useful starting point, but they will 
need to be elaborated with futher assumptions about what rendering hw is 
available and perhaps something about preferences of the recipient.

My point in raising the issue is that so far, when just exploring some 
of these cases in my head, it seems like there probably isn't sufficient 
information. But I am not the expert here, and I may well be making 
unreasonable assumptions. I think at least some others (Espen and Roni) 
think there *is* sufficient info.

Note that I got to this question from the discussions of mixed/composed, 
because I wasn't seeing common agreement on how these would be used to 
help with this kind of decision. But my question here is more general, 
since there is a considerable amount of data defined that might affect 
this decision.

	Thanks,
	Paul

>         <[email-masked] <mailto:[email-masked]>
>         <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>
>         advertisement,
>             _________________________________________________
>             clue mailing list
>         [email-masked] <mailto:[email-masked]> <mailto:[email-masked]
>         <mailto:[email-masked]>>
>         https://www.ietf.org/mailman/__listinfo/clue
>         <https://www.ietf.org/mailman/listinfo/clue>
>
>
>
>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 16 Apr 2012 10:34:55 -0400

Hi Roni,

> My view is that this is enough for the general telepresence point to point

I think I agree for the case where there is a single scene advertised, 
and all the captures represent single cameras.

But as soon as you add in mixed or switched captures it becomes far less 
clear.

For instance, suppose the sender does something to combine some of its 
captures so that it can advertise a capture scene entry that requires 
fewer displays. E.g. suppose it has three main captures of participants 
and one presentation capture. And suppose in addition to advertising all 
of those as one entry, it also advertises another entry for a 
three-screen room, consisting of one capture for the presentation, one 
for the current speaker, and one for the prior speaker. So that is one 
fixed and two switched captures. If the recipient need to map this to 
two displays, how does it decide what to do? The obvious thing would be 
to omit the capture for the prior speaker. But how would it know which 
one that is.

> audio level will help with knowing who are the active speakers in this case

I think there are too many unknowns in the above to discuss it with 
clarity. We need some use cases.

	Thanks,
	Paul

---

From: "Roni Even" <[email-masked]>
Date: Mon, 16 Apr 2012 19:26:56 +0300

Hi Paul,
Inline
Roni

> Sent: Monday, April 16, 2012 5:35 PM
> To: Roni Even
> Cc: 'Espen Berger (espeberg)'; 'CLUE'
> > The receiver will know the semantics of the streams (people or
> > presentation) and the spatial information of the room and the
> different streams.
> > My view is that this is enough for the general telepresence point to
> > point use case.
> 
> But as soon as you add in mixed or switched captures it becomes far
> less clear.
> 
> and one presentation capture. And suppose in addition to advertising
> all of those as one entry, it also advertises another entry for a

This is an interesting case but I am not sure it will happen. My view from
current application is that for this case  there will be one switch capture
advertised and the content in the stream will be decided by the sending
entity (MCU) to be the current speaker or previous speaker depending on the
receiver.

BTW: If a consumer wants to create such application it can start by getting
the active speaker and afterwards continue to receive the active speaker and
request the stream where the previous speaker is based on the spatial
information from the three main captures. 

> 
> > As for multipoint use case, this is why I think it is important to
> > know if the advertisement is based on site switch or segment switch
> > which will provide the extra information needed for the selection
> > (spatial information provides the internal relation between the
> > capture) and I think that the audio level will help with knowing who
> > are the active speakers in this case
> 
> >>> and grouped them into supported / not supported. I can dive into
> >>> more detail about the use cases if people are interested?
> >>>
> >>> * Telepresence point to point, where the reproduction will happen
> in
> >> a
> >> of the N streams will be used? AFAIK the recipient just picks the
> >> ones it wants. So then my question remains - does the recipient have
> >> sufficient info to select the "right" ones? (There isn't any
> >> guarantee that *any* N of the M streams will provide a reasonable
> rendering.
> >>
> >>> * Generic multi source, sending and receiving named sources where
> >>> the user experience is controlled by an human being capable of
> >>> reading sources names, or control software written explicit to
> >>> operate on the named streams. The content attribute can be used to
> >>> indicate a
> >> primary
> >> If so, I agree that if the senders provide well conceived labels
> then
> >> a human can probably decide something reasonable.
> >>
> >> If the human is working only with "content" attributes, then its not
> >> so clear. Its quite possible that the recipient will find itself
> >> trying to choose between multiple streams with the same content
> type.
> >> (E.g. all
> >> 'main'.)
> >>
> >>> [Espen] Agree with Paul, we clearly can support in CLUE the use
> >>> cases where a Telepresence room either offer a 1, 2 or 3 video
> >>> captures, which can be rendered on 1, 2 or 3 screens.
> >>
> >> includes an entry for N (>1) captures also includes an entry with
> >> only one capture, and optionally entries containing other numbers of
> >> But as I mention above, its not clear we have supported case where
> >> the minimal entry in the advertisement is for N>1 captures.
> >>
> >>>>> geometry to directly map all of the captures from one audio and
> >>>>> one video capture scene entry from each capture scene then maybe
> >>>>> all is good. In this case ISTM that the mixed/composed attributes
> >>>>> aren't needed.
> >>>>
> >>>>       It's unlikely that the "typical" telepresence conference
> will
> >>>> have
> >>>
> >>> [Espen] How a vendor build a room or endpoint is transparent from
> >>> the sender of media. As a sender you should not care if a room
> >>> request 3 audio streams (left, center, right) and choose to mix and
> >>> play out on a single speaker. We can argue that this is not the
> best
> >>> experience, but still valid and typically done for PC-clients or
> >>> other endpoints with a single speaker.
> >>
> >> Again, I think you are assuming the receiver can specify the max
> >> number of streams it can handle, and that the sender will then be
> >> obligated to select/construct a suitable set of streams. AFAIK we
> >> have not written down anything that calls for that. (If I am wrong,
> >> please let me know where this is specified.)
> >>
> >> We have the (largely hypothetical) recipient capabilities message,
> >> but its content hasn't been specified yet, and AFAIK there has been
> >> no statement that it is binding and the construction of the
> advertisement.
> >>
> >>> IIUC an endpoint will typically only advertise one or two scenes.
> >>> One would correspond to the cameras and mics in its room. The
> other,
> >>> if present, would correspond to a "presentation" that doesn't fit
> >>> into the coordinate space of the room. There are cases for more,
> but
> >>> they get more obscure.
> >>>
> >>> So for a point-to-point clue call the recipient will only need to
> >>> map one or two scenes. But when there is more than one it is more
> >> complex,
> >>> offer out to the endpoints. It could just pass through all the
> >>> scenes it receives from all the endpoints, leaving the mapping
> >>> problem entirely to the endpoints. (I don't think that is what most
> >>> people have in mind, though some might.)
> >>>
> >>> Or it could take on itself the job of mapping/mixing/switching all
> >>> of those inputs and producing something more like what a typical
> >> endpoint
> >> enables an especially good user experience. While we shouldn't
> >> mandate the user experience, I think we are obligated to do due
> >> diligence that the protocol is sufficient to enable that user
> experience.
> >>
> >>>>> - If you don't have suitable displays, then perhaps you can
> select
> >> a
> >>>>> video capture scene entry and locally compose or switch some of
> >>>>> the captures in order to produce a set of captures that does map
> >>>>> to the available displays. The area of capture of each capture
> >>>>> could be helpful to doing this, and perhaps the composed
> attribute as well.
> >>>>
> >>>>       One inevitable use case is filing your formal report from
> the
> >>> airport.
> >>>> and typically inadequate bandwidth with excessive latency. S/he
> >>>> will need some middlebox to compose a "barely-adequate video" and
> >>>> mono
> >>> audio.
> >>>
> >>> scope. If an MCU is already in the call then perhaps we can hope
> >>> that is one of the entries it makes available.
> >>>
> >>> If there isn't any MCU in the call, then the other endpoint might
> >>> not advertise that option. Then what? Do we have a mechanism for
> that?
> >>>>       Then there's the use case of filing you formal report from
> >>>> your home office (when you're sick or somebody doesn't want to pay
> >>>> travel
> >>> cost).
> >>>> You may well have surround-sound and two or three large monitors,
> >>>> 20 Mbps download with 40 msec RTT, and a decent-quality lavalier
> mike.
> >>> conferencing I think makes it unnecessary complex for the initial
> >>> use cases to cover.
> >>>
> >>>>       There really isn't any such thing as "sufficient speakers",
> >>>> but surround-sound should be considered "sufficient". You can
> >>>> always mix to stereo or even mono: "you pays your money and you
> >>>> takes your
> >>> choice".
> >>>
> >>>>> The spatial information from the captures may help in deciding
> how
> >>>>> to
> >>>
> >>>>> mix.
> >>>>
> >>>>       Absolutely!
> >>>>
> >>>> "riding gain" to reduce background noise. Binary switching is just
> >>>> a bad idea.
> >>>
> >>>>       Not a useful way to think about it, IMHO. Human ears and
> >>>> brains can sort out individual sources if there are enough
> >>>> speakers, while mixing makes that harder.
> >>>
> >>> I have had the impression that "simple" clue telepresence rooms
> >>> would be hoping to do direct mapping of captures to devices. If the
> >>> [Espen] Agree with Paul here, the basic scenario is well
> understood.
> >> A
> >>>>> The problems are superficially different for MCUs. But perhaps
> >>>>> only superficially. It seems to me that when the MCU decides how
> >>>>> to map its inputs into its advertisement, it has some virtual
> room
> >>>>> layouts
> >>> in mind.
> >>>>
> >>>>       I would expect so.
> >>>>
> >>>>> real endpoint with that layout. But it does have the added
> problem
> >>>>> of
> >>>
> >>>>       Pretty much always, unless it's feeding an exact copy of
> what
> >> it
> >>>> receives.
> >>>

---

From: John Leslie <[email-masked]>
Date: Mon, 16 Apr 2012 12:55:11 -0400

Paul Kyzivat <[email-masked]> wrote:
> for the current speaker, and one for the prior speaker.

   Disclaimer: I am a sound guy -- but that doesn't mean I don't pay
attention to what video guys do...

   Current-speaker and previous-speaker are certainly useful; but when
dealing with them, you actually need to know _which_ cameras are available
as switched sources.

   (I don't especially want to re-open a question which has appeared to
be settled, but I _really_ don't understand why we settled on omitting
a list of sources among which we switch...)

   To some degree, I suppose this can be inferred; but for real
conferences, some potential speakers are in one room while others are
in a different room, quite likely under the management of only partly
compatible software.

   Thus, in the ideal, we'd be able to have a screen for current speaker
regardless of which room s/he is in.

   Previous-speaker is largely useful to allow differing decisions on
_when_ to switch. In watching professionally-staged conferences panels,
you'll notice that there is a delay in switching when someone else
starts speaking. It _looks_ as if the camera-man is slow on the uptake,
but that isn't true. The person responsible for video mixing makes the
decision on when to switch, not the cameraman. And s/he integrates into
the decision how likely the need to switch back may be...

   Thus, previous-speaker is most useful when you can use it to second-
guess the switching algorithm.

   (In any case, audio-switching is entirely different.)

> So that is one fixed and two switched captures. If the recipient needs
> to map this to two displays, how does it decide what to do?

   I'm not sure the question makes sense.

   How many displays are free will change during any real conference.
Often, the best solution is to leave a display empty rather than change
too often.

   But let me take a whack at it anyway...

   When there is a presentation display, you almost need to have a
user-activated switch to say when the presentation display is needed
versus when participant display is needed. When enough displays are
available, you want the speaker, the presentation, and something to
gauge attendee reactions. When you're one-down on displays, some
folks will want to concentrate on attendee reactions and others on
the presentation materials.

> The obvious thing would be to omit the capture for the prior speaker.
> But how would it know which one that is.

   Good question!!!

   I had naively assumed that switching policy would be included in
any advertisement of a switched source.

>> are the active speakers in this case

   (As is too often the case, Roni loses me on what difference he draws
between site-switched and segment-switched...)

   On "audio-level" I'm quite clear -- audio level should _never_
be used to second-guess a video-switching algorithm. The correlation
is often very poor. As a first approximation, audio level should be
considered orthogonal to any video characteristic.

   Automatic switching _will_ use audio-level as an input (often the
primary input); but the switching-decision will be biased by things
quite unrelated to audio-level, or related to essentially-invisible
characteristics of audio-level history.

   The switching decisions themselves need to be carried in-stream:
audio levels won't significantly help you reproduce them.

> clarity. We need some use cases.

   Agreed. (But I don't want to propose any in this email.)

---

From: "Brian Baldino (bbaldino)" <[email-masked]>
Date: Mon, 16 Apr 2012 16:19:22 -0700

Hey Paul, with regards to this bit:

"For instance, suppose the sender does something to combine some of its
one that is."

My understanding was more that the provider would advertise all valid
views with a capture entry for each, not require (or rely on) an
endpoint to be able to "find" a valid view which is a subset of one that
was actually advertised; therefore relieving the consumer of that
responsibility.  So in this case, the provider would advertise entries
that contained:
1) Just the active speaker 
2) The active speaker and previous active speaker
3) The active speaker, previous active speaker and presentation
(Maybe even a 4th that contained the active speaker and presentation)

The idea being that the provider is smart enough to know/advertise its
valid views rather than rely on the consumer to figure them out.  

Even with this there are still situations where no advertised view is
ideal for the consumer, maybe this is the situation you're referring to?
I'd hoped that the consumer could get away with choosing the "next best
fit" using information already in the advertisement (how many captures
an entry consists of, switched vs. non-switched, area of capture info,
etc.)

-Brian
 
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Paul Kyzivat
Sent: Monday, April 16, 2012 7:35 AM
To: Roni Even
Cc: 'CLUE'
Subject: Re: [clue] Does framework provide sufficient info for receiver?

Hi Roni,

different streams.

	Thanks,
	Paul

rendering.

>> last call?
>>
>> If so, I agree that if the senders provide well conceived labels then

>> trying to choose between multiple streams with the same content type.

>>>>       It's unlikely that the "typical" telepresence conference will

>>> play out on a single speaker. We can argue that this is not the best

advertisement.
>>> One would correspond to the cameras and mics in its room. The other,

>>> into the coordinate space of the room. There are cases for more, but

experience.

>>>>> could be helpful to doing this, and perhaps the composed attribute
as well.

that?

mike.

>>>>> The spatial information from the captures may help in deciding how

>>>>> to map its inputs into its advertisement, it has some virtual room

>>>>> real endpoint with that layout. But it does have the added problem

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 16 Apr 2012 19:34:23 -0400

> an entry consists of, switched vs. non-switched, area of capture info,

What you describe certainly seems possible, within limits.
As you add more "raw" captures, the number of possibilities goes up 
combinatorially. I would think there would be a temptation to just offer 
the entries you think will be popular. (E.g. the ones supported by room 
configurations you sell, and perhaps not those sold by competitors.)

If this is the solution, then perhaps we should have guidelines for the 
sorts of advertisements that should be offered.

As I mentioned somewhere, this gets harder when there is more than one 
scene in the advertisement, because the advertiser has no way to 
indicate how to balance the entries in different captures. Perhaps that 
means that a separate scene should not be used for presentations, even 
though there is no natural coordinate relationship between the 
presentation and the captures of participants.

I'm just trying to see if we have common expectations in the group, that 
those expectations are consistent with the requirements and use cases, 
and that the support for those expectations is documented.

	Thanks,
	Paul

---

From: "Roni Even" <[email-masked]>
Date: Tue, 17 Apr 2012 17:21:33 +0300

Hi,
My view is that in this case the provider can advertise the three fixed
captures and one with switched which may be the active speaker. We will need
a way to let the consumer know who is the current speaker enabling him to
choose the switched active speaker stream and one of the fixed streams which
may be the previous speaker.

Roni

> From: Brian Baldino (bbaldino) [mailto:[email-masked]]
> Sent: Tuesday, April 17, 2012 2:19 AM
> To: Paul Kyzivat; Roni Even
> Cc: CLUE
> Subject: RE: [clue] Does framework provide sufficient info for
> receiver?
> 
> endpoint to be able to "find" a valid view which is a subset of one
> that was actually advertised; therefore relieving the consumer of that
> 3) The active speaker, previous active speaker and presentation (Maybe
> even a 4th that contained the active speaker and presentation)
> 
> attribute

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 17 Apr 2012 14:44:54 -0400

Hi Roni,

On 4/17/12 10:21 AM, Roni Even wrote:
> captures and one with switched which may be the active speaker.

Do you mean one capture scene entry containing all four of these 
captures? Or do you mean one capture scene entry with the three fixed 
captures, and another one containing just the switched capture?

> We will need

So in that case the consumer would receive all four captures, and render 
the switched capture plus dynamically select one of the others to render?

If so, how is that any easier than simply receiving the three fixed 
captures and making its own decision about which two to render based on 
tracking speakers? That would have the benefit of requiring one less 
stream to be received. But it would still require receiving more streams 
than if it could select two offered by the provider that provide a good 
experience for a two screen system.

	Thanks,
	Paul

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 17 Apr 2012 15:43:01 -0400

Thinking about our discussion today, I do NOT think that we can just
push this entirely on the sender, and just say "I want N screens,
please send the appropriate set." I see two general problems :

- as screens are composited information much be carried along, and the
compositor will inevitably be making its own decisions about what to
carry, in what form, etc.

- some such decisions and serving must come from middleware (i.e., MCU
or servers in the network).
How will they know what to do without getting information from
upstream? And, if they get it, why can't
such information go directly to receivers at the edge ?

Regards
Marshall

On Mon, Apr 16, 2012 at 7:34 PM, Paul Kyzivat <[email-masked]> wrote:
> combinatorially. I would think there would be a temptation to just offer the
> entries you think will be popular. (E.g. the ones supported by room
> As I mentioned somewhere, this gets harder when there is more than one scene
> in the advertisement, because the advertiser has no way to indicate how to
> balance the entries in different captures. Perhaps that means that a
> separate scene should not be used for presentations, even though there is no
> natural coordinate relationship between the presentation and the captures of
> participants.
>
> those expectations are consistent with the requirements and use cases, and
> that the support for those expectations is documented.
>
>        Thanks,
>        Paul
>
>
>> -Brian
>>
>>>>> As a part of my response to Pauls questions I summarized the use
>>>>
>>>> cases
>>>>>
>>>>> * Telepresence point to point, where the reproduction will happen in
>>>>
>>>> a
>>>>>
>>>>> capability of sending 1 - N capture (left to right) and a receiver
>>>>
>>>> can
>>>>>
>>>>> * Advanced audio reproduction, a group of audio scenarios mentioned
>>>>
>>>> by
>>>>>
>>>>> Jon Leslie and Johan Nielsen that requires more details during
>>>>
>>>> initial
>>>>>
>>>>>>       If we wanted that, we'd have to include for every composed
>>>>
>>>> scene
>>>>>>
>>>>> *better* mappings. So one thing to decide is how good is good
>>
>> enough.
>>>>>
>>>>> That isn't what I said. Consider what is perhaps a typical case,
>>>>
>>>> where
>>>>>
>>>>>>       OTOH, with an appropriate "surround-sound" system, you can
>>>>
>>>> "place"
>>>>>>
>>>>>>> - If you don't have suitable displays, then perhaps you can select
>>>>
>>>> a
>>>>>>>
>>>>>>       One inevitable use case is filing your formal report from the
>>>>>
>>>>> airport.
>>>>>>
>>>>> quality audio ande video is always a challenge, with or without
>>
>> clue.
>>>>>
>>>>>
>>>>>>       Near the top end are the corporate telepresence rooms. They
>>>>
>>>> will
>>>>>>
>>>>> I don't understand your point "are likely to have a human being
>>>>
>>>> mixing
>>>>>
>>>>> user. In a transcoded conference you can do that by looking into
>>>>
>>>> XCon,
>>>>>
>>>>> if you do switched conference the endpoint typically render the
>>>>
>>>> layout
>>>>>
>>>>>>       Folks _will_ do that -- I can't stop them. But I find it
>>>>>
>>>>> uninteresting.
>>>>>
>>>>> ??? We have been talking about switching a lot. Are you saying that
>>>>
>>>> is
>>>>>
>>>>> uninteresting?
>>>>>
>>>>>>       Nonetheless, the participants will want to imagine such a
>>>>>
>>>>> relationship.
>>>>>
>>>>> [Espen] Agree with Paul here, the basic scenario is well understood.
>>>>
>>>> A
>>>>>
>>>>> sends two logical audio captures and a receiver play them out with
>>>>
>>>> the
>>>>>
>>>>>>       As an extreme example, you could always render "mixed" in
>>
>> mono.
>>>>>>
>>>>>>
>>>>>>       IMHO, you should _invent_ a spatial relationship in that
>>
>> case.
>>>>>>
>>>>>>> deciding how to describe what it is advertising - e.g. when to
>>>>
>>>> apply
>>>>>>>
>>>>>>       Pretty much always, unless it's feeding an exact copy of what
>>>>
>>>> it
>>>>>>
>>>>>> receives.
>>>>>
>>>>>
>>>>>>> Some interesting use cases for all of the above would be very
>>>>>
>>>>> helpful.
>>>>>>
>>>>>>
>>>>> - how each endpoint maps the advertised captures to its output
>>>>
>>>> equipment
>>>>>

---

From: John Leslie <[email-masked]>
Date: Tue, 17 Apr 2012 16:17:14 -0400

Marshall Eubanks <[email-masked]> wrote:
> please send the appropriate set."

   I quite agree. We're trying to write a standard for interoperability;
and it's simply not reasonable to expect full trust between competitors
on how screen real-estate is allocated.

> I see two general problems :
> 

   Yes, and we can't standardize all those details.

>   or servers in the network). How will they know what to do without
>   getting information from upstream? And, if they get it, why can't

   Clearly, middleware will be involved for some setups.

   Were I writing middleware, I'd want to know my best source of video
for every potential speaker (whether or not any audio matches); and to
apply my own algorithms for "current speaker" (as distinct from "loudest
audio".

   In principle, such information _could_ go to receivers at the edge,
but I expect there to be bandwidth and latency issues lasting many years
into the future.

   My point is that vendors _should_ want to give a "familiar" experience
to their customers, and it will at least subtly differ from the experience
other vendors offer.

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 17 Apr 2012 16:36:55 -0400

> to their customers, and it will at least subtly differ from the experience

How should I read the above when systems from two different vendors are 
interoperating? I can see this in two ways:

- the receiving system will endeavor to adapt the advertised information 
from the provider to produce the kind of experience the receiving system 
prefers.

- the providing system may want to do what it can to cause the receiving 
system to give the experience the providing system prefers.

Absent some counteracting force systems will probably try to do both, 
though they are in some sense conflicting goals.

Presumably the receiving system could come closest to providing its 
familiar experience to its customers if the providing system makes 
available all the raw captures and sufficient information to understand 
what they are. But that assumes the receiving system has sufficient 
bandwidth and processing capacity.

As soon as the sending system starts processing the captures, especially 
if it then fails to make all the raw captures available, it is putting 
its imprint on the receiving system's user experience.

Maybe it would make sense to distinguish here between behavior of 
endpoints and MCUs. We may expect an MCU to have a stronger influence on 
endpoint user experience than we do in a point to point session.

	Thanks,
	Paul

---

From: "Roni Even" <[email-masked]>
Date: Wed, 18 Apr 2012 00:23:21 +0300

Paul,
This is one capture scene entry since they can be provided simultaneously.
The consumer can ask for a subset of the entry and not all four captures. It
can ask first for the switched and any one of the fixed and when the active
speaker changes it can ask for a different fixed capture from the 3 fixed
captures.

I also think that there is nothing that will prevent the receiver to ask for
the three fixed and display 2 of them except we do not have a way to say
which ones are the most active speaker. This is why the switched one is
needed
Roni

> Sent: Tuesday, April 17, 2012 9:45 PM
> To: Roni Even
> Cc: 'Brian Baldino (bbaldino)'; 'CLUE'
> > My view is that in this case the provider can advertise the three
> > fixed captures and one with switched which may be the active speaker.
> 
> > a way to let the consumer know who is the current speaker enabling
> him
> > to choose the switched active speaker stream and one of the fixed
> > streams which may be the previous speaker.
> 
> So in that case the consumer would receive all four captures, and
> render the switched capture plus dynamically select one of the others
> to render?
> 
> stream to be received. But it would still require receiving more
> streams than if it could select two offered by the provider that
> provide a good experience for a two screen system.
> 
> 	Thanks,
> 	Paul
> 
> > Roni
> >
> >> "For instance, suppose the sender does something to combine some of
> >> its captures so that it can advertise a capture scene entry that
> >> requires fewer displays. E.g. suppose it has three main captures of
> >> participants and one presentation capture. And suppose in addition
> to
> >> advertising all of those as one entry, it also advertises another
> >> entry for a three-screen room, consisting of one capture for the
> >> presentation, one for the current speaker, and one for the prior
> >> speaker. So that is one fixed and two switched captures. If the
> >> recipient need to map this to two displays, how does it decide what
> >> to do? The obvious thing would be to omit the capture for the prior
> >> speaker. But how would it know which one that is."
> >>
> >> that was actually advertised; therefore relieving the consumer of
> >> that responsibility.  So in this case, the provider would advertise
> >> entries that contained:
> >> The idea being that the provider is smart enough to know/advertise
> >> its valid views rather than rely on the consumer to figure them out.
> >>
> >> I'd hoped that the consumer could get away with choosing the "next
> >> best fit" using information already in the advertisement (how many
> >> captures an entry consists of, switched vs. non-switched, area of
> >> capture info,
> >> etc.)
> >>
> >> -Brian
> >>
> >> I think I agree for the case where there is a single scene
> >> advertised, and all the captures represent single cameras.
> >>
> >> For instance, suppose the sender does something to combine some of
> >> its captures so that it can advertise a capture scene entry that
> >> requires fewer displays. E.g. suppose it has three main captures of
> >> participants and one presentation capture. And suppose in addition
> to
> >> advertising all of those as one entry, it also advertises another
> >> entry for a three-screen room, consisting of one capture for the
> >> presentation, one for the current speaker, and one for the prior
> >> speaker. So that is one fixed and two switched captures. If the
> >> recipient need to map this to two displays, how does it decide what
> >> to do? The obvious thing would be to omit the capture for the prior
> >> speaker. But how would it know which one that is.
> >>
> >>>> From: [email-masked] [mailto:[email-masked]] On
> >>>> Behalf Of Paul Kyzivat
> >>>> ones it wants. So then my question remains - does the recipient
> >>>> have sufficient info to select the "right" ones? (There isn't any
> >>>>> screen (mono/stereo). Excluded is capabilities to select
> >>>>> individual speakers, the MCU is free to optimize what to compose
> >>>>> and send to a receiver.
> >>>>
> >>>> If the human is working only with "content" attributes, then its
> >>>> not so clear. Its quite possible that the recipient will find
> >>>> itself trying to choose between multiple streams with the same
> >>>> content
> >> type.
> >>
> >>>> (E.g. all
> >>>> 'main'.)
> >>>>
> >>>>> * Switched conferencing, where we assume that endpoints do
> >>>>> scalable video (either SVC or Simulcast), locally composited
> >>>>> layout and
> >> audio
> >>
> >>>>> mixing. Limited work has been done on how to use Scalable video
> >>>>> and how to build mechanisms for complete composited layouts.
> >>>>>> what it's composed from, and for every mixed audio what it's
> >>>>>> mixed from.
> >>>>>
> >>>>> I think you must have misunderstood me. Otherwise I wouldn't
> >>>>> expect you to answer "Of course not". AFAIK the whole point of
> >>>>> CLUE is to provide the information to make this possible.
> >>>>>
> >>>> only one capture, and optionally entries containing other numbers
> >>>> of captures between 1 and N.
> >>>>
> >>>>>>> all is good. In this case ISTM that the mixed/composed
> >>>>>>> attributes aren't needed.
> >>>>>>
> >>>>> the advertisement contains one scene with several alternative
> >>>>> video and audio scene entries. Lets just consider the video. Then
> >>>>> maybe there is one entry with three captures, and another with
> one
> >>>>> (mixed/switched,
> >>>>> screens, would you have enough info about the three captures in
> >>>>> the first entry to make a reasonable mapping onto your two
> screens?
> >> What
> >>
> >>>>> request 3 audio streams (left, center, right) and choose to mix
> >>>>> and play out on a single speaker. We can argue that this is not
> >>>>> the
> >> best
> >>
> >>>>> The situation is more complex for an MCU. It will have as input
> >>>>> all the scenes offered by all the endpoints. It then has to
> decide
> >>>>> what
> >>>> to
> >>>>> problem entirely to the endpoints. (I don't think that is what
> >>>>> most people have in mind, though some might.)
> >>>>>
> >>>>> Or it could do both - advertise its own simplified composite
> >>>>> scenes and also all those provided by the endpoints. That would
> >>>>> allow the endpoints to either take the easy way out, or else do
> it
> >>>>> all themselves in a way that suits them. Suppose it did this.
> >>>>> Would the endpoints that want to take the easy way be able to
> >>>>> figure out
> >> which
> >>
> >>>>>> your home office (when you're sick or somebody doesn't want to
> >>>>>> pay travel
> >>>>> cost).
> >>>>>>        There really isn't any such thing as "sufficient
> >>>>>> speakers", but surround-sound should be considered "sufficient".
> >>>>>> You can always mix to stereo or even mono: "you pays your money
> >>>>>> and you takes your
> >>>>> choice".
> >>>>>
> >>>>>> "riding gain" to reduce background noise. Binary switching is
> >>>>>> just a bad idea.
> >>>>>
> >>>>> questions in hopes that it will result in the right decisions
> >>>>> being made and the right stuff specified.
> >>>>>
> >>>>> would be hoping to do direct mapping of captures to devices. If
> >>>>> the assumption is that this will be an unsatisfactory approach
> and
> >>>>> all endpoints should be prepared to do something more complex
> then
> >>>>> it would be helpful to write that down somewhere - e.g. in the
> >>>> framework.
> >>>>>
> >>>>>>> - If you don't have speakers with similar spatial relationship
> >>>>>>> to those of the audio captures, then you must decide whether to
> >>>>>>> do sub-optimal assignments or mix.
> >>>>>>
> >>>>>>        "Sub-optimal" really doesn't have a clear meaning here.
> >>>>>> Even speakers in the "wrong" left-to-right order while you can
> >>>>>> see the "correct" order on-screen won't confuse the listener as
> >>>>>> much as speaker assignments changing for no obvious reason.
> >>>>>>
> >>>>>>> - If you have audio from more than one scene, what should you
> do
> >>>>>>> with
> >>>>>
> >>>>>>> So for each virtual room layout it is addressing the same
> issues
> >>>>>>> as a
> >>>>>

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 17 Apr 2012 17:38:01 -0400

Hi Roni,

> This is one capture scene entry since they can be provided simultaneously.

As I understand, its possible to get things from multiple entries 
simultaneously. (But whether its possible or not depends on the 
simultaneous set entries.)

Being in a single entry should mean that all the captures in the entry 
together represent the scene. I *think* putting all the fixed captures 
*and* a switched capture that switches among those fixed ones would be 
valid, though it does involve some redundancy. (I'd be interested to 
hear if others think this is appropriate.)

> speaker changes it can ask for a different fixed capture from the 3 fixed

While I see how that would be possible, having to send a new select 
request on the clue channel every time there is a speaker switch might 
be unacceptable. Even worse if that also required an o/a exchange. (But 
maybe it won't require that.)

> which ones are the most active speaker. This is why the switched one is

We need sufficient information to determine active speaker.
Is getting a switched capture, where the switching criterion is active 
speaker, the only way to do that?

	Thanks,
	Paul

>>>> Behalf

---

From: "Roni Even" <[email-masked]>
Date: Wed, 18 Apr 2012 00:53:55 +0300

Hi Paul,
I fully agree that there is a difference between centralized conferences
with MCUs and point to point case.
My experience with MCUs is that they wish to provide the best experience to
all participants (they may have better support for the endpoints from the
same vendor). Since the MCUs cannot be sure what functionality will be
supported by the end points that they will meet in the field (I am talking
here about the solution design phase) they have some out of band mechanism
that allows the use of the endpoint to interact with the MCU conferencing
application in order to control the conference including the views that the
participants will see. Therefore I do not see much value in having a
protocol for that and I had similar feedback to XCON work of which I only
see the event package as an important block.
As for point to point I think that the current information is sufficient.
The sender will provide the capture set entries but if it provides the
simultaneous sets that consumer can ask for the capture he wants regardless
of the capture set entries which reflect the provider preference.

Roni Even

> Sent: Tuesday, April 17, 2012 11:37 PM
> To: [email-masked]
> >     My point is that vendors _should_ want to give a "familiar"
> > experience to their customers, and it will at least subtly differ
> from
> > the experience other vendors offer.
> 
> - the receiving system will endeavor to adapt the advertised
> information from the provider to produce the kind of experience the
> receiving system prefers.
> 
> - the providing system may want to do what it can to cause the
> receiving system to give the experience the providing system prefers.
> 
> As soon as the sending system starts processing the captures,
> especially if it then fails to make all the raw captures available, it
> is putting its imprint on the receiving system's user experience.
> 
> endpoints and MCUs. We may expect an MCU to have a stronger influence
> on endpoint user experience than we do in a point to point session.
> 
> 	Thanks,
> 	Paul

---

From: "Roni Even" <[email-masked]>
Date: Wed, 18 Apr 2012 00:59:49 +0300

Paul,
Inline
Roni

> Sent: Wednesday, April 18, 2012 12:38 AM
> To: Roni Even

There is such example in the framework document for a simultaneous set that
includes redundant information. Note that the consumer can create his own
preferred views based on the capture set entries and simultenous sets. The
capture set entries represent the provider preference.

> 
> > The consumer can ask for a subset of the entry and not all four
> > captures. It can ask first for the switched and any one of the fixed
> > and when the active speaker changes it can ask for a different fixed
> > capture from the 3 fixed captures.
> 
> > I also think that there is nothing that will prevent the receiver to
> > ask for the three fixed and display 2 of them except we do not have a
> > way to say which ones are the most active speaker. This is why the
> > switched one is needed

I do not think this is an issue since switching decision is a slow process
in order to prevent frequent switches when not needed. Note that switching
may require new video synchronization point.

> speaker, the only way to do that?

Currently I think this is the case as long as we get one audio stream for
the three video captures. 

> >> captures and making its own decision about which two to render based
> >> on tracking speakers? That would have the benefit of requiring one
> >> less stream to be received. But it would still require receiving
> more
> >>>>> I think that there is enough information that will allow a
> >>>>> receiver
> >>>> to
> >>>>
> >>>>> know if the advertisement is based on site switch or segment
> >>>>> switch which will provide the extra information needed for the
> >>>>> selection (spatial information provides the internal relation
> >>>>> between the
> >>>>>>> During the CLUE design team meeting on webex yesterday we
> >>>>>>> started
> >>>> to
> >>>>
> >>>>>>> discuss various use cases and if they are in scope or not for
> >>>>>>> the current work in CLUE.
> >>>>>>>
> >>>>>>> request 1 - M streams for rendering (left to tight), which is
> >>>>>>> the basic for interoperability.
> >>>>>>
> >>>>>>> * Transcoded conferencing, where the MCU act like an endpoint
> >>>>>>> and logically announces audio and video left to right and a
> >>>>>>> receiver request
> >>>>>>> 1 - M captures depending on how many screen and audio streams
> >>>>>>> you prefer. Typically a single video stream pr. screen and one
> >>>>>>> audio
> >>>> pr.
> >>>>>>> * Generic multi source, sending and receiving named sources
> >>>>>>> where the user experience is controlled by an human being
> >>>>>>> capable of reading sources names, or control software written
> >>>>>>> explicit to operate on the named streams. The content attribute
> >>>>>>> can be used
> >> to
> >>>>>>> indicate a
> >>>>>> primary
> >>>>>> Are you talking about the text names/labels we were discussing
> on
> >>>> the
> >>>>
> >>>>>> last call?
> >>>>>>
> >>>>>>>>> and captures, do you have sufficient information to select
> and
> >>>> map
> >>>>
> >>>>>> But as I mention above, its not clear we have supported case
> >>>>>> where the minimal entry in the advertisement is for N>1
> captures.
> >>>>>>
> >>>>>>>>> geometry to directly map all of the captures from one audio
> >>>>>>>>> and one video capture scene entry from each capture scene
> then
> >> maybe
> >>>>>>> video and audio scene entries. Lets just consider the video.
> >>>>>>> Then maybe there is one entry with three captures, and another
> >>>>>>> with
> >> one
> >>>>>>> (mixed/switched,
> >>>>>>> whatever) capture. Then if you have three screens you can map
> >>>>>>> all three captures in the first entry. If you have less than
> >>>>>>> three
> >>>>>> screens
> >>>>>>> But what if there was no entry with a single capture? If you
> had
> >>>> two
> >>>>
> >>>>>>> information would we need to include in the advertisement so
> >>>>>>> that
> >>>> it
> >>>>
> >>>>>>> [Espen] How a vendor build a room or endpoint is transparent
> >>>>>>> from the sender of media. As a sender you should not care if a
> >>>>>>> room request 3 audio streams (left, center, right) and choose
> to
> >>>>>>> mix and play out on a single speaker. We can argue that this is
> >>>>>>> not the
> >>>> best
> >>>>
> >>>>>> We have the (largely hypothetical) recipient capabilities
> >>>>>> message, but its content hasn't been specified yet, and AFAIK
> >>>>>> there has
> >> been
> >>>>>>> if present, would correspond to a "presentation" that doesn't
> >>>>>>> fit into the coordinate space of the room. There are cases for
> >>>>>>> more,
> >>>> but
> >>>>
> >>>>>>> they get more obscure.
> >>>>>>>
> >>>>>>> So for a point-to-point clue call the recipient will only need
> >>>>>>> to map one or two scenes. But when there is more than one it is
> >>>>>>> more
> >>>>>> complex,
> >>>>>>> [Espen] This sounds closer to a user experience discussions
> than
> >> a
> >>>>>>> protocol discussion.
> >>>>>>
> >>>>>> Perhaps it is. But the point of CLUE is to develop a protocol
> >>>>>> that enables an especially good user experience. While we
> >>>>>> shouldn't mandate the user experience, I think we are obligated
> >>>>>> to do due diligence that the protocol is sufficient to enable
> >>>>>> that user
> >>>> experience.
> >>>>>>
> >>>>>>>> an arbitrarily large number of audio streams, yet make them
> >>>>>>>> easy
> >>>> to
> >>>>
> >>>>>>>> distinguish.
> >>>>>>>>
> >>>>>>>>> If you don't have sufficient equipment to do that, then the
> >>>>>>>>> job
> >>>> is
> >>>>
> >>>>>>>>> video capture scene entry and locally compose or switch some
> >>>>>>>>> of the captures in order to produce a set of captures that
> >>>>>>>>> does
> >> map
> >>>>>>> If there isn't any MCU in the call, then the other endpoint
> >>>>>>> might not advertise that option. Then what? Do we have a
> >>>>>>> mechanism for
> >>>> that?
> >>>>>>> audio and video (maybe by a SIP re-invite), change the CLUE
> >>>>>>> offer
> >>>> to
> >>>>
> >>>>>>> conferencing I think makes it unnecessary complex for the
> >>>>>>> initial use cases to cover.
> >>>>>>>
> >>>>>>>>         Nonetheless, the participants will want to imagine
> such
> >>>>>>>> a
> >>>>>>> relationship.
> >>>>>>>
> >>>>>>> Right now the coordinate info is all optional. Does it need to
> >>>>>>> be mandatory in order to enable this?
> >>>>>>>
> >>>>>>>>         Not a useful way to think about it, IMHO. Human ears
> >>>>>>>> and brains can sort out individual sources if there are enough
> >>>>>>> basic interoperability between vendors you assume pairs of
> audio
> >>>> and
> >>>>
> >>>>>>>>> to those of the audio captures, then you must decide whether
> >>>>>>>>> to do sub-optimal assignments or mix.
> >>>>>>>>
> >>>>>>>>         I suppose _some_ telepresence system will hide dozens
> >>>>>>>> of speakers in the telepresence room and try to do this -- to
> >>>>>>>> me it sounds like a dreadful idea, but it's quite possible the
> >>>>>>>> people will
> >>>>>> adapt.
> >>>>>>>>
> >>>>>>> Since I've seen people here describe cases when that isn't so,
> >>>>>>> it would be helpful to have a more precise definition.
> >>>>>>>
> >>>>>>>        together with how the info in the advertisement allowed
> >>>>>>> it
> >> to
> >>>>>>>        determine that mapping.
> >>>>>>>

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Thu, 19 Apr 2012 18:31:27 +0200

By doing complete CLUE capability negotiation examples for the typical
use cases and typical behaviour I think we are better prepared to
discuss what is supported in the CLUE framework,  what is eventually
unclear and what we would define as CLUE extensions.

I also think it's useful to look at the last requirement, "REQMT-16:
The solution MUST include extensibility mechanisms". This indicate that
CLUE does not to support all use cases for everyone, but rather make a
good set of core feature with interoperability in mind for the important
use cases. Implementers are free to extend the core protocol and we
should provide hooks to allow extension to be made in a interoperable
way. 

Cheers 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Roni Even
Sent: 17. april 2012 23:54
To: 'Paul Kyzivat'; [email-masked]
Subject: Re: [clue] Does framework provide sufficient info for receiver?

Hi Paul,
I fully agree that there is a difference between centralized conferences
with MCUs and point to point case.
My experience with MCUs is that they wish to provide the best experience
to all participants (they may have better support for the endpoints from
the same vendor). Since the MCUs cannot be sure what functionality will
be supported by the end points that they will meet in the field (I am
talking here about the solution design phase) they have some out of band
mechanism that allows the use of the endpoint to interact with the MCU
conferencing application in order to control the conference including
the views that the participants will see. Therefore I do not see much
value in having a protocol for that and I had similar feedback to XCON
work of which I only see the event package as an important block.
As for point to point I think that the current information is
sufficient.
The sender will provide the capture set entries but if it provides the
simultaneous sets that consumer can ask for the capture he wants
regardless of the capture set entries which reflect the provider
preference.

Roni Even

> How should I read the above when systems from two different vendors 
> are interoperating? I can see this in two ways:
> 
> available all the raw captures and sufficient information to 
> understand what they are. But that assumes the receiving system has 
> sufficient bandwidth and processing capacity.
>