Subject: [clue] continuing "layout" discussion

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 5 Aug 2011 14:02:19 -0700

I'd like to continue the discussion about layout and rendering issues.  There are many separate but related things involved.  I want to break it down into separate topics, and see how the topics are related to each other.  And then we can discuss what CLUE needs to deal with and what is not in scope.

I don't know if I'm using the best terms for each topic.  If not, please suggest better terms.  My use of the term "layout" here is not consistent with draft-wenger-clue-definitions-01, because I don't limit it to the rendering side.  But my use of the terms "render" and "source selction" is consistent with that draft.

1- video layout composed arrangement within a stream - when multiple video sources are composed into one stream, they are arranged in some way.  Typical examples are 2x2 grid, 3x3 grid, 1+5 (1 large plus 5 small), 1+PiP (1 large plus one or more picture-in-picture).  These arrangements can be selected automatically or based on user input.  Arrangements can change over time.  Identifying this composed arrangement is separate from identifying or selecting which video images are used to fill in the composition.  These arrangements can be constructed by an endpoint sending video, by an MCU, or by an endpoint receiving video as it renders to a display.

2 - source selection and identification - when a device is composing a stream made up of other sources, it needs some way to choose which sources to use, and some way of choosing how to combine them or where to place video images in the composed arrangement.  Various automatic algorithms may be used, or selections can be made based on user input.  Selections can change over time.  One example is "select the two most recent talkers".  It may also be desirable to identify which sources are used and where they are placed, for example so the receiving side can use this information in the user interface.  Source selection can be done by an endpoint as it sends media, by an MUC, or by an endpoint receiving media.

3 - spatial relation among streams - how multiple streams are related to each other spatially, to be rendered such that the spatial arrangement is consistent.  The examples we've been using have multiple video streams that are related in an ordered row from left to right.  Audio is also included when it is desirable to match spatial audio to video.

4 - multi stream media format - what the streams mean with respect to each other, regardless of the actual content on the streams.  For audio, examples are stereo, 5.1 surround, binaural, linear array.  (linear array is described in the clue framework document).  Perhaps 3D video formats would also fit in this category.  This information is needed in order to properly render the media into light and sound for human observers.  I see this at the same level as identifying a codec, independent of the audio or video content carried on the streams, and independent of how any composition of sources is done.

I think there is general agreement that items 3 and 4 are in scope for CLUE, as they specifically deal with multiple streams to and from an endpoint.  And the framework draft includes these.  Items 1 and 2 are not new, those topics exist for traditional single stream videoconferencing.  I'm not sure what aspects of 1 and 2 should be in scope for CLUE.  It is hard to tell from the use cases and requirements.  The framework draft includes them only to a very limited extent.

Mark Duckworth

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 09 Aug 2011 09:03:29 -0400

> 4 - multi stream media format - what the streams mean with respect to each other, regardless of the actual content on the streams.  For audio, examples are stereo, 5.1 surround, binaural, linear ar…

I was with you all the way until 4. That one I don't understand.
The name you chose for this has connotations for me, but isn't fully in 
harmony with the definitions you give:

If we consider audio, it makes sense that multiple streams can be 
rendered as if they came from different physical locations in the 
receiving room. That can be done by the receiver if it gets those 
streams separately, and has information about their intended 
relationships. It can also be done by the sender or MCU and passed on to 
the receiver as a single stream with stereo or binaural coding.

So it seems to me you have two concepts here, not one. One has to do 
with describing the relationships between streams, and the other has to 
do with the encoding of spacial relationships *within* a single stream.

Or, are you asserting that stereo and binaural are simply ways to encode 
multiple logical streams in one RTP stream, together with their spacial 
relationships?

	Thanks,
	Paul

> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 10 Aug 2011 14:49:36 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Paul Kyzivat
> Sent: Tuesday, August 09, 2011 9:03 AM
> To: [email-masked]
> Subject: Re: [clue] continuing "layout" discussion

> > 4 - multi stream media format - what the streams mean with respect to
> each other, regardless of the actual content on the streams.  For
> audio, examples are stereo, 5.1 surround, binaural, linear array.
> (linear array is described in the clue framework document).  Perhaps 3D
> video formats would also fit in this category.  This information is
> needed in order to properly render the media into light and sound for
> human observers.  I see this at the same level as identifying a codec,
> independent of the audio or video content carried on the streams, and
> independent of how any composition of sources is done.
> 

I'm happy to change the name if you have a suggestion

> the receiver as a single stream with stereo or binaural coding.

Yes.  It could also be done by the sender using the "linear array" audio channel format.  Maybe it is true that stereo or binaural audio channels would always be sent as a single stream, but I was not assuming that yet, at least not in general when you consider other types too, such as linear array channels.

> do with the encoding of spacial relationships *within* a single stream.

Maybe that is a better way to describe it, if you assume multi-channel audio is always sent with all the channels in the same RTP stream.  Is that what you mean?

I was considering the linear array format to be another type of multi-channel audio, and I know people want to be able to send each channel in a separate RTP stream.  So it doesn't quite fit with how you separate the two concepts.  In my view, identifying the separate channels by what they mean is the same concept for linear array and stereo.  For example "this channel is left, this channel is center, this channel is right".  To me, that is the same concept for identifying channels whether or not they are carried in the same RTP stream.

Maybe we are thinking the same thing but getting confused by terminology about channels vs. streams.

> multiple logical streams in one RTP stream, together with their spacial

No, that is not what I'm trying to say.

Mark

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 11 Aug 2011 09:01:50 -0400

Inline

> I'm happy to change the name if you have a suggestion

Not yet. Maybe once the concepts are more clearly defined I will have an 
opinion.

> Maybe we are thinking the same thing but getting confused by terminology about channels vs. streams.

Maybe. Let me try to restate what I now think you are saying:

The audio may consist of several "channels".

Each channel may be sent over its own RTP stream,
or multiple channels may be multiplexed over an RTP stream.

I guess much of this can also apply to video.

When there are exactly two audio channels, they may be encoded as 
"stereo" or "binaural", which then affects how they should be rendered 
by the recipient. In these cases the primary info that is required about 
the individual channels is which is left and which is right. (And which 
perspective to use in interpretting left and right.)

For other multi-channel cases more information is required about the 
role of each channel in order to properly render them.

	Thanks,
	Paul

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Mon, 15 Aug 2011 13:21:44 -0700

Please see inline.

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of Paul Kyzivat
> Sent: Thursday, August 11, 2011 6:02 AM
> To: [email-masked]

I do not think this is necessarily true. Taking audio as an example, you
could have two audio streams that are mixed to form a single stereo
audio stream, or you could have them as two independent (not mixed)
streams that are associate with each other by some grouping mechanism.
This group would be categorized as being stereo audio with one audio
stream being the left and the other the right. The codec used for each
could be different, though I agree they would typically be the same.
Consequently, I think at attribute such as "stereo" as being more of a
grouping concept, where the group may consist of:
- multiple independent streams, each with potentially its own spatial
orientation, codec, bandwidth, etc., 
- a single mixed stream

Cheers,
Charles

> >> relationships. It can also be done by the sender or MCU and passed
on
> >> to
> > Yes.  It could also be done by the sender using the "linear array"
audio channel format.  Maybe it
> is true that stereo or binaural audio channels would always be sent as
a single stream, but I was not
> assuming that yet, at least not in general when you consider other
types too, such as linear array
> channels.
> > Maybe that is a better way to describe it, if you assume
multi-channel audio is always sent with all
> the channels in the same RTP stream.  Is that what you mean?
> >
> > I was considering the linear array format to be another type of
multi-channel audio, and I know
> people want to be able to send each channel in a separate RTP stream.
So it doesn't quite fit with
> how you separate the two concepts.  In my view, identifying the
separate channels by what they mean is
> the same concept for linear array and stereo.  For example "this
channel is left, this channel is
> center, this channel is right".  To me, that is the same concept for
identifying channels whether or
> not they are carried in the same RTP stream.
> >

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 15 Aug 2011 17:13:36 -0400

Inline

> orientation, codec, bandwidth, etc.,

[sb] I do not understand this distinction.  What do you mean when you say
"two audio streams that are mixed to form a single stereo stream", and how
is this different from the left and right grouping?

---

From: Roni Even <[email-masked]>
Date: Tue, 16 Aug 2011 00:22:28 +0300

Hi,
It looks to me like I agree with Charles but taking it to video I see that
we have two separate entities which appear now in eth framework as one.

We have three video captures devices or streams (similar to two audio
streams) and we have the grouping which is left to right. Currently the left
to right is assumed.
Roni

> Charles Eckel (eckelcu)
> Sent: Monday, August 15, 2011 11:22 PM
> To: Paul Kyzivat; [email-masked]
> > >>> 4 - multi stream media format - what the streams mean with
> respect
> to

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Mon, 15 Aug 2011 14:45:18 -0700

> From: Stephen Botzko [mailto:[email-masked]]
> Sent: Monday, August 15, 2011 2:14 PM
> To: Charles Eckel (eckelcu)
> Cc: Paul Kyzivat; [email-masked]
<[email-masked]> wrote:
> 	> >> From: [email-masked] [mailto:[email-masked]]
On
> 	Behalf Of
respect
> 	> >> needed in order to properly render the media into light and
sound
> 	for
> 	> >> human observers.  I see this at the same level as
identifying a
> 	codec,
> 	> >> independent of the audio or video content carried on the
streams,
> 	and
> [sb] I do not understand this distinction.  What do you mean when you
say "two audio streams that are
> mixed to form a single stereo stream", and how is this different from
the left and right grouping?

In one case they are mixed by the source of the stream into a single
stream, and in another they are sent as two separate streams by the
source. The end result once rendered at the receiver may be the same,
but what is sent is different. This example with audio is perhaps too
simple. If you think of it as video that is composed into a single video
stream vs. multiple via streams that are sent individually, the
difference may be more clear.

Cheers,
Charles

> 	> >> The name you chose for this has connotations for me, but
isn't
> 	fully in
> 	> Not yet. Maybe once the concepts are more clearly defined I
will have
> 	an
> 	> >> So it seems to me you have two concepts here, not one. One
has to
> 	do
> 	> >> with describing the relationships between streams, and the
other
> 	has to
> 	> >> do with the encoding of spacial relationships *within* a
single
> 	stream.
> 	> by the recipient. In these cases the primary info that is
required
> 	about
> 	> the individual channels is which is left and which is right.
(And
> 	which
> 	> >> Or, are you asserting that stereo and binaural are simply
ways to
> 	> >> encode
> 	> >> multiple logical streams in one RTP stream, together with
their
> 	spacial

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 16 Aug 2011 09:20:03 -0400

I guess by "stream" you are meaning RTP stream?  in which case by "mix" you
perhaps mean that the left and right channels are placed in a single RTP
stream???  What do you mean when you describe some audio captures as
"independent" - are you thinking they come from different rooms???.

I think in many respects audio distribution and spatial audio layout is at
least as difficult as video layout, and have some unique issues.  For one
thing, you need to sort out how you should place the audio from human
participants who are not on camera, and what should happen later on if some
of those participants are shown.

I suggest it is necessary to be very careful with terminology.  In
particular, I think it is important to distinguish composition from RTP
transmission.

Regards,
Stephen Botzko

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Tue, 16 Aug 2011 13:23:01 -0700

I am distinguishing between:

(1) a single RTP stream that consists of a single stereo audio stream
(2) two RTP streams, one that contains left speaker audio and the other
than contains right speaker audio

(2) could also be transmitted in a single RTP stream using SSRC
multiplexing. Let me call that (2b). 
(2) and (2b) are essentially the same. Just the RTP mechanism employed
is difference.
(1) is different from (2) and (2b) in that the audio signal encoded is
actually different.

Cheers,
Charles

> Sent: Tuesday, August 16, 2011 6:20 AM
> I guess by "stream" you are meaning RTP stream?  in which case by
"mix" you perhaps mean that the left
> and right channels are placed in a single RTP stream???  What do you
mean when you describe some audio
> captures as "independent" - are you thinking they come from different
rooms???.
> 
> I think in many respects audio distribution and spatial audio layout
is at least as difficult as video
> layout, and have some unique issues.  For one thing, you need to sort
out how you should place the
> audio from human participants who are not on camera, and what should
happen later on if some of those
> participants are shown.
particular, I think it is important
> to distinguish composition from RTP transmission.
> 
> Regards,
> Stephen Botzko
> 
> 
> 
<[email-masked]> wrote:
> 	>       > From: [email-masked]
[mailto:[email-masked]] On
> 	Behalf
> 	>       > >> audio, examples are stereo, 5.1 surround, binaural,
linear
> 	array.
> 	>       > >> (linear array is described in the clue framework
document).
> 	>       Perhaps 3D
> 	>       > >> video formats would also fit in this category.
This
> 	information is
> 	>       I do not think this is necessarily true. Taking audio as
an
> 	example, you
> 	>       could have two audio streams that are mixed to form a
single
> 	stereo
> 	>       audio stream, or you could have them as two independent
(not
> 	mixed)
> 	>       streams that are associate with each other by some
grouping
> 	mechanism.
> 	>       This group would be categorized as being stereo audio
with one
> 	audio
> 	>       stream being the left and the other the right. The codec
used
> 	for each
> 	>       could be different, though I agree they would typically
be the
> 	same.
> 	>       Consequently, I think at attribute such as "stereo" as
being
> 	more of a
> 	>       - multiple independent streams, each with potentially
its own
> 	spatial
> 	>       > >> I was with you all the way until 4. That one I
don't
> 	understand.
> 	>       > >> If we consider audio, it makes sense that multiple
streams
> 	can be
> 	>       > >> rendered as if they came from different physical
locations
> 	in the
> 	>       > >> receiving room. That can be done by the receiver if
it gets
> 	those
> 	>       > >> relationships. It can also be done by the sender or
MCU and
> 	passed
> 	>       on
> 	>       > >> to
> 	>       > >> the receiver as a single stream with stereo or
binaural
> 	coding.
> 	>       > >
> 	>       > > Yes.  It could also be done by the sender using the
"linear
> 	array"
> 	>       > is true that stereo or binaural audio channels would
always be
> 	sent as
> 	>       > assuming that yet, at least not in general when you
consider
> 	>       > > I was considering the linear array format to be
another type
> 	of
> 	>       > people want to be able to send each channel in a
separate RTP
> 	>       > how you separate the two concepts.  In my view,
identifying
> 	the
> 	>       > the same concept for linear array and stereo.  For
example
> 	"this
> 	>       > center, this channel is right".  To me, that is the
same
> 	concept for
> 	>       > > Maybe we are thinking the same thing but getting
confused by
> 	>       terminology about channels vs. streams.
> 	>       >
> 	>       > When there are exactly two audio channels, they may be
encoded
> 	as
> 	>       > "stereo" or "binaural", which then affects how they
should be
> 	rendered
> 	>       > For other multi-channel cases more information is
required
> 	about the

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 16 Aug 2011 17:14:01 -0400

Well, the audio in (1) and (2b) is certainly packetized differently.  But
not compressed differently (unless you are assuming that the signal in (1)
is jointly encoded stereo - which it could be I guess, but it would be
unusual for telepresence systems). Also, the audio in (1) is not mixed, no
matter how it is encoded.

In any event, I believe that the difference between (1) and (2) and (2b) is
really a transport question that has nothing to do with layout. The same
information is needed to enable proper rendering, and once the streams are
received, they are rendered in precisely the same way.

Regards,
Stephen Botzko

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Tue, 16 Aug 2011 14:40:11 -0700

Agreed. The difference I am trying to point out is that in (1), the
information you need to describe the audio stream for appropriate
rendering is already handled quite well by existing SIP/SDP/RTP and most
implementations, whereas you need CLUE for (2) and (2b).

Cheers,
Charles

> Sent: Tuesday, August 16, 2011 2:14 PM
> Well, the audio in (1) and (2b) is certainly packetized differently.
But not compressed differently
> (unless you are assuming that the signal in (1) is jointly encoded
stereo - which it could be I guess,
> but it would be unusual for telepresence systems). Also, the audio in
(1) is not mixed, no matter how
> it is encoded.
> 
> In any event, I believe that the difference between (1) and (2) and
(2b) is really a transport
> question that has nothing to do with layout. The same information is
needed to enable proper
> rendering, and once the streams are received, they are rendered in
precisely the same way.
<[email-masked]> wrote:
> 	>       >       > >>> 4 - multi stream media format - what the
streams
> 	mean with
> 	>       respect
> 	>       >       to
> 	>       >       > >> each other, regardless of the actual
content on the
> 	>       streams.  For
> 	>       >       > >> needed in order to properly render the
media into
> 	light and
> 	>       >       > >> independent of the audio or video content
carried
> 	on the
> 	>       streams,
> 	>       >       and
> 	>       >       > >> independent of how any composition of
sources is
> 	done.
> 	>       > [sb] I do not understand this distinction.  What do
you mean
> 	when you
> 	>       In one case they are mixed by the source of the stream
into a
> 	single
> 	>       stream, and in another they are sent as two separate
streams by
> 	the
> 	>       source. The end result once rendered at the receiver may
be the
> 	same,
> 	>       but what is sent is different. This example with audio
is
> 	perhaps too
> 	>       simple. If you think of it as video that is composed
into a
> 	single video
> 	>       >       > >> The name you chose for this has
connotations for
> 	me, but
> 	>       >       > Not yet. Maybe once the concepts are more
clearly
> 	defined I
> 	>       >       > >> streams separately, and has information
about their
> 	>       intended
> 	>       >       > >> So it seems to me you have two concepts
here, not
> 	one. One
> 	>       has to
> 	>       >       do
> 	>       >       > >> with describing the relationships between
streams,
> 	and the
> 	>       other
> 	>       >       has to
> 	>       >       > >> do with the encoding of spacial
relationships
> 	*within* a
> 	>       single
> 	>       >       stream.
> 	>       >       > >
> 	>       >       > > Maybe that is a better way to describe it,
if you
> 	assume
> 	>       >       > the channels in the same RTP stream.  Is that
what you
> 	mean?
> 	>       >       > Maybe. Let me try to restate what I now think
you are
> 	saying:
> 	>       >       > or multiple channels may be multiplexed over
an RTP
> 	>       >       > by the recipient. In these cases the primary
info that
> 	is
> 	>       required
> 	>       >       about
> 	>       >       > the individual channels is which is left and
which is
> 	right.
> 	>       >       > >> Or, are you asserting that stereo and
binaural are
> 	simply

---

From: Roni Even <[email-masked]>
Date: Wed, 17 Aug 2011 02:34:45 +0300

Hi guys,
In case 1 according to RFC 3551 (section 4.1) 2 channels in the rtpmap means
left and right channels described as stereo. Are you saying that for the 2
and 2b case you also assume stereo capture or can it be any other way of
creating the two audio streams  from the same room (Binaural recording (not
common), or some other arrangements of the microphones). But this talk about
the capture side.

I think that Christer talked about the rendering side and not only on the
capture side.

Roni

> Sent: Wednesday, August 17, 2011 12:40 AM
> To: Stephen Botzko
> Cc: [email-masked]

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 16 Aug 2011 21:20:26 -0400

That is not a correct conclusion.  In order to render correctly (for 1, 2,
and 2b), you need to align the apparent sound location with the rendered
video image.  This requires CLUE, it is not covered at all by existing
SIP/SDP/RTP signaling.

Regards,
Stephen

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 16 Aug 2011 21:36:39 -0400

Hi Roni

For this particular discussion, all of the two channel transmissions are
"stereo", they are just transported differently.

As far as the framework draft is concerned, the various microphone
arrangements are accounted for by the signaling of the 1-100 indices for
each channel.

Binaural is something else- either an HRTF function is applied to the two
channels prior to rendering (which was Christer's case with the central
rendering server), or you have a dummy head with microphones in the ears in
the telepresence room to make the capture.  Not sure if we need to
distinguish the capture and render cases right now.

Regards,
Stephen

---

From: Roni Even <[email-masked]>
Date: Wed, 17 Aug 2011 07:51:37 +0300

Hi Steve,

The two channel is a simple private case and using this case to define the
required information from the capture and render side is like saying that I
proved a mathematical induction for n=2 therefore it works for every n. I
see this issue since we are using a n=2 channels audio and a n=3 cameras
left to right examples to provide a solution that will scale to any n.

What I was trying to say that the current way we describe the stream by a
number is not enough if we want to go to the "being there" experience. 

We need to see what are the dimensions that the model need in order to be
able to convey the capture information and the rendering capabilities for
both audio and video. I think that there is some similarities since we have
a number of streams, how the capture is done and what is the rendering
device as the basic information. 

I think that before going to the framework model it may be beneficial to
create a list of the parameters we need to convey, provide a term for each
group of parameters and have a way to define them in the model. For example
for the capture we have the number of capture devices, the arrangement
(Spatial), the encoding process (including mixing if multiple inputs), the
capture field and others.

Regards

Roni

From: Stephen Botzko [mailto:[email-masked]] 
Sent: Wednesday, August 17, 2011 4:37 AM
To: Roni Even
Cc: Charles Eckel (eckelcu); [email-masked]
Subject: Re: [clue] continuing "layout" discussion

Hi Roni

Regards,
Stephen

Hi guys,

Roni

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 17 Aug 2011 10:05:55 -0400

For audio at least (and probably video) I agree you need the number and
placement of *captures*, but I see no value in knowing the number of capture
*devices*.  For instance, the stereo encoding we started with might be
derived from a microphone in front each of two participants, or it might be
derived from a large microphone array.  For receivers, there is no
difference, so I see no reason to signal it.   The 3D telepresence
demonstration technology in the EU used 3 cameras to derive each 3D view (I
think), but it could have also been done with a Kinect-style single camera.
Again, the number of cameras used to make the capture would make no
difference to a receiver.

I don't know what you mean by the "capture field" (or what specifically
about it you think we ought to know), so at present I have no opinion as to
whether it is needed or not.  I agree that mixing of sources from multiple
rooms needs more attention (and I think that is what the "layout"
conversation should be chiefly about).

I think the framework draft is not limited to 2 channels for audio and 3
cameras.  I haven't seen any issues for an N-image video wall and an
associated M-channel audio capture, as long as you stay within the stated
assumption in the model that the audio/video are on one wall.  Apparently
there is work to extend the model to handle multiple video walls, and of
course the audio will need to be adjusted for that.

BTW, I would challenge anyone who is either proposing an alternative
framework or extending this draft to build the needed rendering equations to
get the right sound field from a standard arrangement of speakers.  Though
rendering itself is out of scope, we do have enablement requirements for
rendering.  There are lots of things we *could* signal, but if their use in
rendering is not easily understood, then we will not achieve
interoperability.

Regards,
Stephen

On Wed, Aug 17, 2011 at 12:51 AM, Roni Even <[email-masked]> wrote:

> Hi Steve,****
> left to right examples to provide a solution that will scale to any n.****
>
> ** **
>
> number is not enough if we want to go to the "being there" experience. ***
> *
>
> device as the basic information. ****
>
> capture field and others.****
>
> ** **
>
> Regards****
>
> Roni****
>
> ** **
>
> *From:* Stephen Botzko [mailto:[email-masked]]
> *Sent:* Wednesday, August 17, 2011 4:37 AM
> *To:* Roni Even
> *Cc:* Charles Eckel (eckelcu); [email-masked]
>
> *Subject:* Re: [clue] continuing "layout" discussion****
>
> ** **
>
> Hi Roni
>
> Stephen****
> On Tue, Aug 16, 2011 at 7:34 PM, Roni Even <[email-masked]> wrote:**
> **
>
> Hi guys,
> Roni****
> > From: [email-masked] [mailto:[email-masked]] On Behalf Of*
> ***
>
> > Charles Eckel (eckelcu)
> > Cc: [email-masked]****
> > https://www.ietf.org/mailman/listinfo/clue****
>
> ** **
>

---

From: Roni Even <[email-masked]>
Date: Wed, 17 Aug 2011 18:12:01 +0300

Steve,

I also would  not agree that we need to specify how rendering is done once
the streams arrive to the receiver. I think that the receiver should be able
to provide information about his rendering capabilities which may be helpful
for the sender to create a better content.

As for my comment on 2 and 3 case. The current layout discussion is on audio
with 2 channels, I was saying that this is a simple case and if we want to
discuss layout it should be clear what it means for multi-video and audio
cases.

My comment on the current framework is that it dives immediately to the
model and the examples talk about a three camera left to right case. I was
questioning how this model scales.

I agree with your comment about the number of capture devices. By capture
field I was trying to mention the issue of the view port (is the focus on
the first row, second row, multiview,.). Maybe it can be described by the
framework but it is not explained how.

Roni

From: Stephen Botzko [mailto:[email-masked]] 
Sent: Wednesday, August 17, 2011 5:06 PM
To: Roni Even
Cc: Charles Eckel (eckelcu); [email-masked]
Subject: Re: [clue] continuing "layout" discussion

For audio at least (and probably video) I agree you need the number and
placement of captures, but I see no value in knowing the number of capture
devices.  For instance, the stereo encoding we started with might be derived
from a microphone in front each of two participants, or it might be derived
from a large microphone array.  For receivers, there is no difference, so I
see no reason to signal it.   The 3D telepresence demonstration technology
in the EU used 3 cameras to derive each 3D view (I think), but it could have
also been done with a Kinect-style single camera.  Again, the number of
cameras used to make the capture would make no difference to a receiver.

rendering.  There are lots of things we could signal, but if their use in

Regards,
Stephen

Hi Steve,

Regards

Roni

Hi Roni

Regards,
Stephen

Hi guys,

Roni

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 17 Aug 2011 11:35:13 -0400

I didn't say that we needed to *specify* how rendering is done.  However,
the information we are providing is intended to enable the rendering of an
interoperable "being there experience".

So we do need to know if the information we signal actually enables such
rendering.  The best way to ensure that is to construct a rendering method
that would work, and which would need the information we propose to provide.
We don't need to specify the rendering method, we don't even have to
describe it on the list.  But if we all can't construct a reasonable method
easily, then CLUE will have failed.

Regards,
Stephen

On Wed, Aug 17, 2011 at 11:12 AM, Roni Even <[email-masked]> wrote:

> Steve,****
> for the sender to create a better content.****
>
> ** **
>
> As for my comment on 2 and 3 case. The current layout discussion is on
> audio with 2 channels, I was saying that this is a simple case and if we
> want to discuss layout it should be clear what it means for multi-video and
> audio cases.****
>
> ** **
>
> questioning how this model scales.****
>
> ** **
>
> the first row, second row, multiview,…). Maybe it can be described by the
> framework but it is not explained how.****
>
> ** **
>
> Roni****
>
> ** **
>
> *Sent:* Wednesday, August 17, 2011 5:06 PM
>
> *To:* Roni Even
> placement of *captures*, but I see no value in knowing the number of
> capture *devices*.  For instance, the stereo encoding we started with
> might be derived from a microphone in front each of two participants, or it
> might be derived from a large microphone array.  For receivers, there is no
> rendering.  There are lots of things we *could* signal, but if their use
> in rendering is not easily understood, then we will not achieve
> interoperability.
>
> Regards,
> Stephen
>
>
> ****
>
> On Wed, Aug 17, 2011 at 12:51 AM, Roni Even <[email-masked]> wrote:*
> ***
>
> Hi Steve,****
>
>  ****
>  ****
>  ****
> *Cc:* Charles Eckel (eckelcu); [email-masked]****
>
>
>  ****
>  ****

---

From: Christian Groves <[email-masked]>
Date: Tue, 23 Aug 2011 15:58:25 +1000

Hello,

With regards to spatial relation among streams, if I start a 
telepresence session and I only use the centre screen/camera of a three 
screen/camera telepresence system. i.e. its only me in the room, the 
left and right screen/cameras are off. According to CLUE framework what 
do I send in terms of capture set?
a) VC1 - only one video capture it doesn't matter which of my screen it 
came from
b) VC2, VC1, VC3 - VC2 and VC3 would be NULL. By having VC1 it the 
middle it indicates that its the centre screen.
c) both are valid?

Now while I'm there a person comes to join me and the left screen and 
camera are turned on. I guess now there's two capture sets VC2, VC1. 
Again do I have to add VC3 as NULL, to indicate that its my left and 
centre camera/screen being used.

I guess for the far end it doesn't care it will render the VCs how it 
wants??

Regards, Christian

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 24 Aug 2011 13:47:44 -0700

Christian,

Thanks for the questions, I'll answer below.

Mark

> Christian Groves
> Sent: Tuesday, August 23, 2011 1:58 AM
> To: [email-masked]
> left and right screen/cameras are off.

Just because it is only you in the room doesn't mean you don't want to see all three screens worth of video coming from the other sites.  So I suppose it is possible you would use only one screen, I don't think it would be typical.  Using only one camera when there is nothing interesting for the other cameras to see makes more sense.

> According to CLUE framework what
 
Choice a) just VC1 by itself is the simplest and what I think it should do if it wants to send only one video capture because that is all that is interesting.  But I would say this video capture comes from a camera, not from a screen.  I don't know what a NULL media capture is, the framework doesn't include that concept.

> camera are turned on.

Again, I don't understand why your example has a correlation between how many people are in your local room and how many screens you are using to view the people from other locations.  I expect you would use all the screens in your room to view the other people, no matter how many people are with you looking at those screens. 

I guess now there's two capture sets VC2, VC1.
> centre camera/screen being used.

You would just use a capture set with (VC2, VC1) indicating two video captures, one on the left and one on the right.

> I guess for the far end it doesn't care it will render the VCs how it

The far end will know that VC2 should be rendered to the left of VC1.  It probably cares about that, otherwise it wouldn't be using CLUE. 

> > I'd like to continue the discussion about layout and rendering
> issues.  There are many separate but related things involved.  I want
> to break it down into separate topics, and see how the topics are
> related to each other.  And then we can discuss what CLUE needs to deal
> with and what is not in scope.
> >
> > I don't know if I'm using the best terms for each topic.  If not,
> please suggest better terms.  My use of the term "layout" here is not
> consistent with draft-wenger-clue-definitions-01, because I don't limit
> it to the rendering side.  But my use of the terms "render" and "source
> selction" is consistent with that draft.
> >
> > 1- video layout composed arrangement within a stream - when multiple
> video sources are composed into one stream, they are arranged in some
> way.  Typical examples are 2x2 grid, 3x3 grid, 1+5 (1 large plus 5
> small), 1+PiP (1 large plus one or more picture-in-picture).  These
> arrangements can be selected automatically or based on user input.
> Arrangements can change over time.  Identifying this composed
> arrangement is separate from identifying or selecting which video
> images are used to fill in the composition.  These arrangements can be
> constructed by an endpoint sending video, by an MCU, or by an endpoint
> receiving video as it renders to a display.
> >
> > 2 - source selection and identification - when a device is composing
> a stream made up of other sources, it needs some way to choose which
> sources to use, and some way of choosing how to combine them or where
> to place video images in the composed arrangement.  Various automatic
> algorithms may be used, or selections can be made based on user input.
> Selections can change over time.  One example is "select the two most
> recent talkers".  It may also be desirable to identify which sources
> are used and where they are placed, for example so the receiving side
> can use this information in the user interface.  Source selection can
> be done by an endpoint as it sends media, by an MUC, or by an endpoint
> receiving media.
> >
> > 3 - spatial relation among streams - how multiple streams are related
> to each other spatially, to be rendered such that the spatial
> arrangement is consistent.  The examples we've been using have multiple
> video streams that are related in an ordered row from left to right.
> Audio is also included when it is desirable to match spatial audio to
> video.
> >
> > I think there is general agreement that items 3 and 4 are in scope
> for CLUE, as they specifically deal with multiple streams to and from
> an endpoint.  And the framework draft includes these.  Items 1 and 2
> are not new, those topics exist for traditional single stream
> videoconferencing.  I'm not sure what aspects of 1 and 2 should be in
> scope for CLUE.  It is hard to tell from the use cases and
> requirements.  The framework draft includes them only to a very limited
> extent.
> >
> > Mark Duckworth

---

From: Christian Groves <[email-masked]>
Date: Thu, 25 Aug 2011 11:01:51 +1000

Hello Mark,

Sorry for the "confused" email, probably matches my reading of the 
framework. Please see my responses [CNG] below to clarify.

regards, Christian

> Just because it is only you in the room doesn't mean you don't want to see all three screens worth of video coming from the other sites.  So I suppose it is possible you would use only one screen,…
[CNG] Yes you're correct I shouldn't have mixed the camera/screen. The 
intention is that locally only one camera (centre) is used as there is 
only one participant (although the system has a 3 camera capability). 
The remote system is a 3 screen system (number of cameras not relevant).

> Choice a) just VC1 by itself is the simplest and what I think it should do if it wants to send only one video capture because that is all that is interesting.  But I would say this video capture co…
[CNG] Yes the capture comes from a camera. In this case a centre camera. 
I guess what I was trying to get at was: is a capture related to what am 
is actually being used (i.e. VC1)? or it is related to what is being 
used in terms of the of the overall system (i.e. VC2, VC1, VC3)? When I 
refer to a NULL media capture I'm referring to this case where VC2 and 
VC3 represent potential video captures. i.e. I don't want to use them 
now but they may be used in the future.

> Again, I don't understand why your example has a correlation between how many people are in your local room and how many screens you are using to view the people from other locations.  I expect you…
[CNG] Sorry, my sloppy use of screens. I'll try again. The remote end 
has three screens. It only receives one video capture VC1 (no 
information about a possible VC2 and VC3 is sent). My question is what 
screen is VC1 displayed on? I guess the answer is the centre one as the 
remote TPS figures it is talking to a single camera/screen system 
because there's only one video capture?
So whilst the VCs are described left to right, as there only one the 
remote TSP assumes that it related to a "centre" screen.

In the framework draft section 10 it discusses cases where 1, 2 and 3 
screen systems receives VC that are equal to or exceed their 
capabilities. However there's no cases describing what systems do when 
receiving VC less than their capabilities. i.e. a 3 screen systems 
receiving one or two video captures.

> You would just use a capture set with (VC2, VC1) indicating two video captures, one on the left and one on the right.
[CNG] Yes two video captures could be used. In this case the remote end 
system I'm assuming would display VC2 on the left screen, and VC1 on the 
centre screen. Now what if (VC1,VC3) was instead sent? The framework 
says they represent left to right video captures. In this case would the 
TSP display VC1 on the left screen and VC3 in the centre screen? 
Effectively moving the displayed scene from the centre to the left.

To ensure VC1 remains on the centre screen the remote TSP would need to 
compare the two video capture sets to determine the correlation between 
the old and new individual captures. It would be difficult to determine 
from the position in the VC set as the position, and number of positions 
in the set may change.

Is there the intention that the "numbering" mentioned in the framework 
document clause 6.1 be used to actually be used in the encoding of the 
video captures to help with this sort of scenario? and generally for the 
"dynamic" behaviour mentioned in section 6.1?