Subject: [clue] Call for Consensus on framework spatial relationships

From: Paul Kyzivat <[email-masked]>
Date: Fri, 10 Feb 2012 14:27:49 -0500

CLUE WG participants:

There has been a lot of discussion about coordinate systems and spatial 
relationships. The editors of the framework think they have reflected 
the group understanding of that topic in framework-03. (This is 
primarily in sections 5, 6.1.1, and 6.2.1, with an example in section 11.1)

This is a request for everyone who cares to indicate if they are 
satisfied with the treatment of this topic. If not, please specify what 
you object to, in a constructive way.

It would be helpful to have comments by the start of the interim meeting 
next Wed, Feb 15, so that they can be considered during discussions 
there. But that may be too tight for some people traveling. So lets set 
a secondary target date of Monday, Feb 20.

	Thanks,
	Paul (as co-chair)

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 10 Feb 2012 16:04:06 -0500

(As individual)

Now I get to reply to myself. I've been meaning to make these comments 
before but somehow didn't get to it:

* Identification of independent coordinate systems

Section 5 doesn't say much about this. It only says that systems without 
multiple media captures to be spatially associated don't need a 
coordinate system. This implies that captures in the same capture set 
should use the same coordinate system. But a capture set is limited to 
captures of the same media type. So audio and video captures of the same 
room are different capture sets. But to be related to one another they 
should share the same coordinate system.

ISTM that Capture Scene needs to be raised to first class status, and 
separated from Capture Set. A coordinate system should be associated 
with the Scene, and then Capture Sets should be subordinated to a 
specific Scene.

This also impacts the specification of Purpose in 6.1.1. IIUC Purpose is 
just an incomplete identification of scene. Specifically, it assumes 
that a source (room) has at most two scenes - the main one and one for 
presentations. This seems quite limiting:

- certainly there can be multiple presentations. Each is likely to be
   a separate scene in that it isn't spatially related to the other.
   But a presentation could have both video capture and an audio capture
   that should be related. (Its also possible that you have a "two
   projector" presentation that has two video captures in a fixed
   relationship to one another.)

- there could be multiple independent endpoints in the same room.
   Ideally they should share a coordinate system and an origin, so that
   their captures can be related. (But its also possible that they
   don't know about each other and don't share a coordinate system.
   In that case I see little alternative to treating them as separate
   scenes. But maybe this bears discussion.)

So, I propose that:

- Capture Scene becomes a first class entity, with its own attributes.
   The Area of Scene, and Scale attribute should belong to it,
   rather than to Capture Set.

- each capture set should be associated with a capture scene.
   (an attribute is the name, or some sort of reference, to the scene.)

- an endpoint may source capture sets from an arbitrary number of
   capture scenes

- different endpoints can source capture sets from the same
   capture scene. (Though this may be an unusual case.)

This will require some sort of naming system for scenes makes it easy 
for each endpoint to have some unique names, but that also allows 
multiple endpoints to reference some common names. The details of that 
are TBD.

* Area of Capture (Depth?)

I could swear that there was some discussion of depth of field that 
isn't reflected in the current text. What is there now is four points in 
three-space, defining a quadrilateral, but without depth. I had thought 
there was going to be the possibility of eight points, describing a 
"quadrilaterally-faced hexahedron".

Did I just imagine this?

A better way to do this might be to specify a quadrilateral and a camera 
origin, with the intent that what is included is everything in the 
tetrahedron specified by those five points and extending beyond until it 
extends beyond the area of scene.

* Area of Scene

Even more than area of capture, area of scene seems inadequate without 
being specified as a volume. 2D might work if all the captures taken 
from the same origin. Otherwise not so much. This could be especially 
obvious if you have microphones scattered throughout a large room. It 
also doesn't work for theater in the round.

Of course its complicated to specify a room with complex geometry. But 
perhaps its sufficient for now to define it as a "quadrilaterally-faced 
hexahedron", via eight points.

	Thanks,
	Paul

---

From: "Roni Even" <[email-masked]>
Date: Sat, 11 Feb 2012 00:32:24 +0200

HI,
I made some comments in my review
Roni

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Paul Kyzivat
> Sent: Friday, February 10, 2012 9:28 PM
> To: CLUE
> Subject: [clue] Call for Consensus on framework spatial relationships
> 
> CLUE WG participants:
> 
> It would be helpful to have comments by the start of the interim
> meeting next Wed, Feb 15, so that they can be considered during
> discussions there. But that may be too tight for some people traveling.
> So lets set a secondary target date of Monday, Feb 20.
> 
> 	Thanks,
> 	Paul (as co-chair)
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Wed, 15 Feb 2012 11:14:39 +0100

Hi all

I am new to this mailing list and haven't read the full history, so
please forgive me if I go where you have already been. I have however
followed the framework development, and work closely with Espen.

As I understand the framework, media captures in a capture set may have
spatial relationships, and these relationships are supposed to be fully
described using the capture area attribute.

My view is that the capture area coordinates are useful for describing
static relationships of the capture devices in a physical capture scene.
However, they may not be sufficient for describing relationships
necessary for proper spatial rendering of video and audio in some
usecases. Additional mechanisms for explicitly relating audio and video
captures and/or expressing dynamically varying relationships may be
necessary.

I suggest to expand on the examples 11.1 and 11.3 in the framework to
investigate this further. Specifically, to include more audio
capabilities and test how the capture area concept works in a switched,
pre-composed and/or dynamically changing situation.

Take the example in 11.1. The consumer is free to pick the switched
capture VC4 and the directional audio set of captures AC0-AC1-AC2. The
spatial relationships between these captures are well defined by the
capture area and can be used for rendering. But whether the rendering
will be perceived as correct or not depends on which video segment is
active at any given time, and which of the two persons on the segment is
talking.

The MCU example in 11.3 is an excellent usecase for directional audio,
and multiple audio streams should be included as options for the various
multi-screen consumers. To conclude on the capture area concept it is
necessary to see how such a capture set is described, including the
capture areas. How do they define relationships between streams. How
does a 4-screen/4-speaker receiver and a 4-screen/2-speaker receiver
know which streams to choose.

The switched stream in 11.1 is an example where spatial relationships
change during the course of the conference. The 11.3 MCU case may be
another, depending on how multiple audio streams are composed. The
static capture area concept is not well suited to convey such changes. 

The active coordinate concept mentioned in A.5 can be one additional
mechanism that complements the capture area (although it should not be
mixed up with the VAD concept).

I am in favour of consensus on the capture area coordinates, but I don't
think the topic of spatial relationships is fully resolved.

Regards
Johan

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Paul Kyzivat
Sent: 10. februar 2012 20:28
To: CLUE
Subject: [clue] Call for Consensus on framework spatial relationships

CLUE WG participants:

There has been a lot of discussion about coordinate systems and spatial
relationships. The editors of the framework think they have reflected
the group understanding of that topic in framework-03. (This is
primarily in sections 5, 6.1.1, and 6.2.1, with an example in section
11.1)

	Thanks,
	Paul (as co-chair)

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 15 Feb 2012 04:20:34 -0800

Hi Paul,

Some comments in line.

Mark

> Sent: Friday, February 10, 2012 4:04 PM
> To: [email-masked]
> Subject: Re: [clue] Call for Consensus on framework spatial relationships
> 
> (As individual)
> 
> multiple media captures to be spatially associated don't need a coordinate
> system. This implies that captures in the same capture set should use the
> same coordinate system. But a capture set is limited to captures of the same
> media type. So audio and video captures of the same room are different
> capture sets. But to be related to one another they should share the same
> coordinate system.

[Duckworth, Mark] This is not correct.  A capture set can contain entries of different media types.  From section 6.2:
"Capture sets will commonly include media captures of different types, for instance, audio captures and video captures. "

> separated from Capture Set. A coordinate system should be associated with
> the Scene, and then Capture Sets should be subordinated to a specific Scene.
> 
> This also impacts the specification of Purpose in 6.1.1. IIUC Purpose is just an
> incomplete identification of scene. Specifically, it assumes that a source
> (room) has at most two scenes - the main one and one for presentations.
> This seems quite limiting:
 
[Duckworth, Mark] It wasn't the intent to limit it to two scenes.

>    relationship to one another.)

[Duckworth, Mark] Agree.

>    scenes. But maybe this bears discussion.)

[Duckworth, Mark] I don't understand how independent endpoints can share a coordinate system.  They would be signaling things completely independent of each other.

>    rather than to Capture Set.

[Duckworth, Mark] Maybe, but the intent was that one capture set corresponds to one scene.  I'm not sure if there is a need to have multiple capture sets specifically associated with the same scene.

> This will require some sort of naming system for scenes makes it easy for
> each endpoint to have some unique names, but that also allows multiple
> endpoints to reference some common names. The details of that are TBD.
> 
> I could swear that there was some discussion of depth of field that isn't
> reflected in the current text. What is there now is four points in three-space,
> defining a quadrilateral, but without depth. I had thought there was going to
> be the possibility of eight points, describing a "quadrilaterally-faced
> hexahedron".
> 
> Did I just imagine this?

[Duckworth, Mark] I remember the topic being discussed, but I thought people generally agreed we did not need to represent depth in the area of capture.

> origin, with the intent that what is included is everything in the tetrahedron
> specified by those five points and extending beyond until it extends beyond
> the area of scene.

[Duckworth, Mark] That is what we have today with area of capture and point of capture.

> being specified as a volume. 2D might work if all the captures taken from the
> same origin. Otherwise not so much. This could be especially obvious if you
> have microphones scattered throughout a large room. It also doesn't work
> for theater in the round.

[Duckworth, Mark] I'm not sure specifying a volume of space instead of a quadrilateral will add anything useful.

---

From: "Johan Ludvig Nielsen (johaniel)" <[email-masked]>
Date: Thu, 16 Feb 2012 14:57:37 +0100

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Duckworth, Mark
Sent: 15. februar 2012 13:21
To: Paul Kyzivat; [email-masked]
Subject: Re: [clue] Call for Consensus on framework spatial
relationships

>    rather than to Capture Set.

[Duckworth, Mark] Maybe, but the intent was that one capture set
corresponds to one scene.  I'm not sure if there is a need to have
multiple capture sets specifically associated with the same scene.

[Nielsen, Johan L] 
It would be interesting to see an example of how the framework would
represent a case with multiple captures from the same scene, but where
functional relationship is more important to convey than physical
spatial relationship. 

For instance a lecture. One camera capturing the lecturer, another the
audience raising questions. Capture area can describe the physical
relationship between the captures, whether the lecturer is visible in
both streams or not. However, the consumer is not interested in that
relationship, but need to know that they are from the same scene, which
function the captures have, the priority, activity, ... 

Would these video captures be in a single or in different capture sets?
In this case there may be audio captures to be connected to one of the
video captures and not to the other. How do you convey that using the
capture area attribute? Having different capture sets from the same
scene may solve this. (Wasn't my idea, but I steal it.)

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 22 Feb 2012 11:51:54 -0800

At the interim meeting last week, we decided we should propose a change to the framework to eliminate the two different concepts of "capture scene" and "capture set".  Most people thought we don't need the two different (but closely coupled) concepts.  One suggestion was to rename a capture set to be called a capture scene, and that one entity would then encompass both of the original concepts.  The original intent was to have a one to one relationship between a capture scene and a capture set anyway.  So I'll send another message later with a proposal along those lines.

More comments inline.

Mark

> Johan Ludvig Nielsen (johaniel)
> Sent: Thursday, February 16, 2012 8:58 AM
> To: [email-masked]
> corresponds to one scene.  I'm not sure if there is a need to have multiple
> capture sets specifically associated with the same scene.
> 
> [Nielsen, Johan L]
> functional relationship is more important to convey than physical spatial
> relationship

[Duckworth, Mark] If the spatial relationship is not important, then they don't have to be in the same capture set/scene.

> relationship between the captures, whether the lecturer is visible in both
> streams or not. However, the consumer is not interested in that relationship,
> but need to know that they are from the same scene, which function the
> captures have, the priority, activity, ...

[Duckworth, Mark] Maybe you have a different idea of what is the "scene".  I say if spatial relationship is not important, then they are not in the same scene.  If I understand what you mean by function, then that is handled by the purpose attribute.  I made a proposal in another message that this should just use enumerated values from RFC4796.

> In this case there may be audio captures to be connected to one of the video
> captures and not to the other. How do you convey that using the capture
> area attribute? Having different capture sets from the same scene may solve
> this. (Wasn't my idea, but I steal it.)

[Duckworth, Mark] If the same audio capture is associated with multiple video captures, then they should all be in the same capture set/scene.  If on the other hand a video capture is not associated with the audio, and not spatially related to other video in the set/scene, then that video capture should be in a different set/scene.

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 22 Feb 2012 12:11:16 -0800

Hi Johan,

I agree the cases you point out do not yet have a clear answer as to how to spatially associate audio to video.  Yes, as you say, the active coordinate idea from A.5 is intended to address this, and it is in addition to VAD, but maybe not enough information.  I think Andy was planning on sending a more detailed proposal to the list.

Your question about choosing VC4 (switched) along with A0, A1, A2 (left, center, right) is a good one.  So far, we have not defined a way for the consumer to know which specific spatial area is represented by VC4 at any given time.  The consumer just knows that overall it represents the "entire area" (or whatever the provider advertised with the static area of capture for VC4).  But we haven't defined any way for the provider to indicate which subset of the area it represents _right now_ as it switches between different camera images.  Do you think it is necessary to provide that information?

Mark

> Sent: Wednesday, February 15, 2012 5:15 AM
> To: [email-masked]
> I am new to this mailing list and haven't read the full history, so please forgive
> me if I go where you have already been. I have however followed the
> framework development, and work closely with Espen.
> 
> My view is that the capture area coordinates are useful for describing static
> relationships of the capture devices in a physical capture scene.
> However, they may not be sufficient for describing relationships necessary
> for proper spatial rendering of video and audio in some usecases. Additional
> mechanisms for explicitly relating audio and video captures and/or
> expressing dynamically varying relationships may be necessary.
> 
> investigate this further. Specifically, to include more audio capabilities and
> test how the capture area concept works in a switched, pre-composed
> and/or dynamically changing situation.
> 
> 
> Take the example in 11.1. The consumer is free to pick the switched capture
> VC4 and the directional audio set of captures AC0-AC1-AC2. The spatial
> relationships between these captures are well defined by the capture area
> and can be used for rendering. But whether the rendering will be perceived
> as correct or not depends on which video segment is active at any given time,
> and which of the two persons on the segment is talking.
> 
> The MCU example in 11.3 is an excellent usecase for directional audio, and
> multiple audio streams should be included as options for the various multi-
> screen consumers. To conclude on the capture area concept it is necessary to
> see how such a capture set is described, including the capture areas. How do
> they define relationships between streams. How does a 4-screen/4-speaker
> receiver and a 4-screen/2-speaker receiver know which streams to choose.
> 
> The switched stream in 11.1 is an example where spatial relationships change
> during the course of the conference. The 11.3 MCU case may be another,
> depending on how multiple audio streams are composed. The static capture
> area concept is not well suited to convey such changes.
> 
> I am in favour of consensus on the capture area coordinates, but I don't think
> the topic of spatial relationships is fully resolved.
> 
> Regards
> Johan
> 
> 
> 
> relationships. The editors of the framework think they have reflected the
> group understanding of that topic in framework-03. (This is primarily in
> sections 5, 6.1.1, and 6.2.1, with an example in section
> 11.1)
> 
> This is a request for everyone who cares to indicate if they are satisfied with
> the treatment of this topic. If not, please specify what you object to, in a
> constructive way.
> 
> next Wed, Feb 15, so that they can be considered during discussions there.
> But that may be too tight for some people traveling. So lets set a secondary
> target date of Monday, Feb 20.
> 
> 
> 	Thanks,
> 	Paul (as co-chair)

---

From: "Roni Even" <[email-masked]>
Date: Thu, 23 Feb 2012 00:21:12 +0200

Hi Mark,
I think that if you chose the first signaling alternative and provide the
full SDP as explained in my individual draft for the VC1, VC2, VC3 with the
SSRC all bundled with VC4 than you will know which VC is currently
transmitted in the switched channel. 
Roni

> Sent: Wednesday, February 22, 2012 10:11 PM
> To: Johan Ludvig Nielsen (johaniel); [email-masked]
> I agree the cases you point out do not yet have a clear answer as to
> how to spatially associate audio to video.  Yes, as you say, the active
> coordinate idea from A.5 is intended to address this, and it is in
> addition to VAD, but maybe not enough information.  I think Andy was
> planning on sending a more detailed proposal to the list.
> 
> Your question about choosing VC4 (switched) along with A0, A1, A2
> (left, center, right) is a good one.  So far, we have not defined a way
> for the consumer to know which specific spatial area is represented by
> VC4 at any given time.  The consumer just knows that overall it
> represents the "entire area" (or whatever the provider advertised with
> the static area of capture for VC4).  But we haven't defined any way
> for the provider to indicate which subset of the area it represents
> _right now_ as it switches between different camera images.  Do you
> think it is necessary to provide that information?
> 
> Mark
> 
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Johan Ludvig Nielsen (johaniel)
> > As I understand the framework, media captures in a capture set may
> > have spatial relationships, and these relationships are supposed to
> be
> > fully described using the capture area attribute.
> >
> > usecases. Additional mechanisms for explicitly relating audio and
> > video captures and/or expressing dynamically varying relationships
> may be necessary.
> >
> > capabilities and test how the capture area concept works in a
> > switched, pre-composed and/or dynamically changing situation.
> >
> >
> > capture
> > relationships between these captures are well defined by the capture
> > area and can be used for rendering. But whether the rendering will be
> > perceived as correct or not depends on which video segment is active
> > at any given time, and which of the two persons on the segment is
> talking.
> >
> > and multiple audio streams should be included as options for the
> > various multi- screen consumers. To conclude on the capture area
> > concept it is necessary to see how such a capture set is described,
> > including the capture areas. How do they define relationships between
> > streams. How does a 4-screen/4-speaker receiver and a 4-screen/2-
> speaker receiver know which streams to choose.
> >
> > I am in favour of consensus on the capture area coordinates, but I
> > don't think the topic of spatial relationships is fully resolved.
> >
> > Regards
> > Johan
> >
> >
> >
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Paul Kyzivat
> > There has been a lot of discussion about coordinate systems and
> > spatial relationships. The editors of the framework think they have
> > reflected the group understanding of that topic in framework-03.
> (This
> > is primarily in sections 5, 6.1.1, and 6.2.1, with an example in
> > section
> > 11.1)
> >
> > satisfied with the treatment of this topic. If not, please specify
> > what you object to, in a constructive way.
> >
> discussions there.
> > But that may be too tight for some people traveling. So lets set a
> > secondary target date of Monday, Feb 20.
> >
> >
> > 	Thanks,
> > 	Paul (as co-chair)