Subject: Re: [clue] <capturePoint> should be optional

From: "Duckworth, Mark" <[email-masked]>
Date: Tue, 25 Mar 2014 12:13:15 -0700

Hello Roberta,

There still seems to be a mistake in the data model about <capturePoint> being mandatory, when it should be optional.  Please see below.

Maybe you didn't see the previous messages from me and Christian?  Or do you believe <capturePoint> really should be mandatory?

Regards,
Mark

> -----Original Message-----
> From: clue [mailto:[email-masked]] On Behalf Of Christian Groves
> Sent: Thursday, February 06, 2014 7:59 PM
> To: [email-masked]
> Subject: Re: [clue] <capturePoint> should be optional
> 
> Yes,
> Christian
> 
> On 7/02/2014 5:00 AM, Duckworth, Mark wrote:
> > I see <capturePoint> is still mandatory in data-model-schema-03.  Does the
> rest of the group agree we should change this to optional?
> >
> > Mark
> >
> >> -----Original Message-----
> >> From: clue [mailto:[email-masked]] On Behalf Of Christian
> >> Groves
> >> Sent: Sunday, January 19, 2014 9:38 PM
> >> To: [email-masked]
> >> Subject: Re: [clue] <capturePoint> should be optional
> >>
> >> Hello Mark,
> >>
> >> I agree.
> >>
> >> Regards,
> >> Christian
> >>
> >> On 18/01/2014 9:57 AM, Duckworth, Mark wrote:
> >>> In the data model section 10.4, in "spatialInformationType", the
> >>> "capturePoint" element is mandatory. It really should be optional.
> >>> In earlier versions of the framework it was optional. This is
> >>> described in section 10.4.1: "If the point of capture is not
> >>> specified, it means the consumer should not assume anything about
> >>> the spatial location of the capturing device."
> >>>
> >>> So I think the schema should have minOccurs="0" for "capturePoint"
> >>> element. And the text right below the schema part should say
> >>> optional instead of mandatory.
> >>>
> >>> Mark

---

From: John Leslie <[email-masked]>
Date: Tue, 25 Mar 2014 20:11:39 -0400

Duckworth, Mark <[email-masked]> wrote:
> 
> There still seems to be a mistake in the data model about <capturePoint>
> being mandatory, when it should be optional.  Please see below.
>...

   I'm not at all sure I understand this discussion.

   ISTM that spatialInformationType is included as a possible entry in
items where it is usually meaningless...

   But I am confused by the discussion of capturePoint being optional
within spatialInformationType.

   What is the meaning of an instance of spatialInformationType without
a capturePoint?

--
John Leslie <[email-masked]>

---

From: Roberta Presta <[email-masked]>
Date: Wed, 26 Mar 2014 10:16:18 +0100

Hi Mark,

I have seen the previous messages but have not forwarded my (very late) 
reply since I have tried to be clearer in the new data model text.
The current specification of the data model is compatible with the 
following statement:
"If the point of capture is not specified, it means  the consumer should 
not assume anything about the spatial location of
the capturing device."
Indeed, when describing a media capture, you have two possibilities (for 
the "choice" implementation):
- if a capture is not spatially definable, it will not have any 
<spatialInformation> within its description and it will have a 
<nonSpatiallyDefinible> tag set to true.
- otherwise (spatially definable capture), at least the capture point 
should be provided within the <spatialInformation> tag. Other 
information, like the line of capture point and the captured area, are 
optional.
I think that John shares my view.
Does it sound to you (all)?

Roberta

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 26 Mar 2014 09:09:14 -0700

Hi Roberta and John,

I disagree that <capturePoint> should be required within <spatialInformation>.  I think Christian shares this view.
I also disagree with this from section 10.4:
"Captures are spatially definible when it is possible to provide at
least the coordinates of the device position within the telepresence
room of origin."

I suggest changing it to:
"Captures are spatially definable when it is possible to provide
the coordinates of either the device position or the area of capture within the telepresence
room of origin."

The intent (in the framework, at least) was to allow a provider to specify "area of capture" but not specify "point of capture" for any particular media capture.  I think we unintentionally lost this possibility when we moved some attribute details from the framework to the data model several revisions ago.

One example scenario where this makes sense is in an advertisement from an MCU that is constructing MCCs for its own scene, and will fill it by composing and/or switching captures from other endpoints.  The MCU can specify "area of capture" using "no scale" coordinates (e.g. to maintain consistent left-to-right relationship), but in this case a "point of capture" is not useful, not meaningful, and should not be required.

Mark

> From: Roberta Presta [mailto:[email-masked]]
> Sent: Wednesday, March 26, 2014 5:16 AM
> To: Duckworth, Mark
> Cc: John Leslie; [email-masked]
> The current specification of the data model is compatible with the following
> statement:
> "If the point of capture is not specified, it means  the consumer should not
> assume anything about the spatial location of the capturing device."
> Indeed, when describing a media capture, you have two possibilities (for the
> "choice" implementation):
> - otherwise (spatially definable capture), at least the capture point should be
> provided within the <spatialInformation> tag. Other information, like the line
> of capture point and the captured area, are optional.
> >> There still seems to be a mistake in the data model about
> >> <capturePoint> being mandatory, when it should be optional.  Please see
> below.
> >> ...
> >     ISTM that spatialInformationType is included as a possible entry
> > in items where it is usually meaningless...
> >
> >     What is the meaning of an instance of spatialInformationType
> > without a capturePoint?
> >
> > --

---

From: Roberta Presta <[email-masked]>
Date: Wed, 26 Mar 2014 18:03:00 +0100

Ok Mark,

I see your point thanks to the scenario you have described.
I will change the definitions provided in the data model accordingly.

For the case you have proposed, the ADV could be the following:

capture scene A: captures coming from room A
capture scene B: captures coming from room B
capture scene C: captures coming from room C
capture scene D: MCC A (from room A's captures) - to be rendered on the 
left, MCC B (from room B's captures) - to be rendered on the centre, MCC 
C (from room C's capture) - to be rendered on the right

where the "to be rendered" information are represented by means of the 
<capturedArea> within the <spatialInformation> tag associated to the MCCs.

Could that be a correct solution?

Roberta

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 26 Mar 2014 10:35:57 -0700

Hi Roberta,
Yes, your example illustrates a case like I was talking about, where it is useful to have <captureArea> without <capturePoint>.

Mark

> Sent: Wednesday, March 26, 2014 1:03 PM
> To: Duckworth, Mark
> Cc: John Leslie; [email-masked]; Christian Groves
> ([email-masked])
> capture scene A: captures coming from room A capture scene B: captures
> coming from room B capture scene C: captures coming from room C capture
> scene D: MCC A (from room A's captures) - to be rendered on the left, MCC B
> (from room B's captures) - to be rendered on the centre, MCC C (from room
> C's capture) - to be rendered on the right
> 
> > I disagree that <capturePoint> should be required within
> <spatialInformation>.  I think Christian shares this view.
> > "Captures are spatially definable when it is possible to provide the
> > coordinates of either the device position or the area of capture
> > within the telepresence room of origin."
> >
> > The intent (in the framework, at least) was to allow a provider to specify
> "area of capture" but not specify "point of capture" for any particular media
> capture.  I think we unintentionally lost this possibility when we moved some
> attribute details from the framework to the data model several revisions ago.
> >
> > One example scenario where this makes sense is in an advertisement from
> an MCU that is constructing MCCs for its own scene, and will fill it by
> composing and/or switching captures from other endpoints.  The MCU can
> specify "area of capture" using "no scale" coordinates (e.g. to maintain
> consistent left-to-right relationship), but in this case a "point of capture" is
> not useful, not meaningful, and should not be required.
> >
> > Mark
> >
> >> I have seen the previous messages but have not forwarded my (very
> >> late) reply since I have tried to be clearer in the new data model text.
> >> following
> >> "If the point of capture is not specified, it means  the consumer
> >> should not assume anything about the spatial location of the capturing
> device."
> >> Indeed, when describing a media capture, you have two possibilities
> >> (for the "choice" implementation):
> >>>      ISTM that spatialInformationType is included as a possible
> >>> entry in items where it is usually meaningless...
> >>>
> >>>      But I am confused by the discussion of capturePoint being
> >>> optional within spatialInformationType.
> >>>

---

From: John Leslie <[email-masked]>
Date: Wed, 26 Mar 2014 14:34:24 -0400

I'm still struggling to understand what a <spatialInformationType>
instance without a <capturePoint> _means_ ...

>> capture scene C: captures coming from room C

   Three quite independent captures... I get this part...

>> capture scene D:
>>   MCC A (from room A's captures) - to be rendered on the left,
>>   MCC B (from room B's captures) - to be rendered on the centre,
>>   MCC C (from room C's capture) - to be rendered on the right

   I'm struggling here. "To be rendered..." seems to be a suggestion
to the receiver, with (AFAICT) no spatial relationship to anything in
Capture Scene A, B, C, or D.

   Why is <spatialInformation> an appropriate tool for this advice?

>> <capturedArea> within the <spatialInformation> tag associated to the MCCs.

   <captureArea> has always bothered me, because it is a two-dimensional
quantity, while we interact in three-dimensional space. I suppose I should
have objected to including it at all in <spatialInformation>; but until
now I thought we understood it to be defining space in a plane within
our three-dimensional interaction space (and therefore a three-dimensional
space from <capturePoint> to the <captureArea>).

   I find the idea of collapsing <captureArea> into dimensionless space
a bit too confusing. In particular, I am cringing in fear that someone
will try to use this anti-physical concept to describe audio...

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 26 Mar 2014 15:22:29 -0400

> instance without a <capturePoint> _means_ ...

I have a little trouble with this too, though it may be that I don't 
conceptualize this correctly.

We have area of capture, but that is a planar region. I keep wanting to 
think of that as an "area of display" that is ideal to render the 
capture. But to use that, I need to know which side of the display 
should be the front. Without a capture point, I don't know that.

I suspect that there is an implicit assumption that the "front" of the 
area of capture will always be the side that puts the smallest X 
coordinate to the left of the larger X coordinate. I don't think we 
should assume such things.

	Thanks,
	Paul

> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 26 Mar 2014 13:43:58 -0700

Hi John and Paul, comments inline.
Mark

> From: clue [mailto:[email-masked]] On Behalf Of Paul Kyzivat
> Sent: Wednesday, March 26, 2014 3:22 PM
> To: [email-masked]

[Duckworth, Mark] It means the captures within the scene have some spatial relation, and the capturePoint is either not known or not applicable.  For composed video captures I think the capturePoint would be not applicable, because there is no single capture point.

> We have area of capture, but that is a planar region. I keep wanting to think
> of that as an "area of display" that is ideal to render the capture. But to use
> that, I need to know which side of the display should be the front. Without a
> capture point, I don't know that.

[Duckworth, Mark] Yes, I think you do know that because the capture area has named coordinates bottomLeft, bottomRight, etc.  If the camera was located at the back of the room, then the left X coordinate would be greater than the right X coordinate.  Maybe this is an assumption that the left and right here are from the point of view of the camera, but I think that is easy to make explicit in the data model.  This is a little odd, given the statement "X increases from Camera-Left to Camera-Right", but if you have cameras pointing in opposite directions within the same scene then one of them has to be reversed with respect to the coordinate system.

> 
> I suspect that there is an implicit assumption that the "front" of the area of
> capture will always be the side that puts the smallest X coordinate to the left
> of the larger X coordinate. I don't think we should assume such things.

 [Duckworth, Mark] Don't have to assume it if we understand the named coordinates of capture area as I suggest.  The front is the side where the viewer would see the named left coordinates to the left of the named right coordinates, regardless of the X values.

> >>> capture scene A: captures coming from room A capture scene B:
> >>> captures coming from room B capture scene C: captures coming from
> >>> room C
> >
> >>> where the "to be rendered" information are represented by means of
> >>> the <capturedArea> within the <spatialInformation> tag associated to
> the MCCs.
> >
> >     <captureArea> has always bothered me, because it is a
> > two-dimensional quantity, while we interact in three-dimensional
> > space. I suppose I should have objected to including it at all in
> > <spatialInformation>; but until now I thought we understood it to be
> > defining space in a plane within our three-dimensional interaction
> > space (and therefore a three-dimensional space from <capturePoint> to
> the <captureArea>).
> >
> >     I find the idea of collapsing <captureArea> into dimensionless
> > space a bit too confusing. In particular, I am cringing in fear that
> > someone will try to use this anti-physical concept to describe audio...
> >
> > --

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 27 Mar 2014 11:02:02 -0400

> [Duckworth, Mark] Yes, I think you do know that because the capture area has named coordinates bottomLeft, bottomRight, etc.  If the camera was located at the back of the room, then the left X coor…

OK, I accept that.

(I didn't have in my head that the corners were named that way.)

>   [Duckworth, Mark] Don't have to assume it if we understand the named coordinates of capture area as I suggest.  The front is the side where the viewer would see the named left coordinates to the…

OK.

	Sorry,
	Paul

---

From: Roberta Presta <[email-masked]>
Date: Fri, 28 Mar 2014 16:12:01 +0100

Hi all,

about the following John's concern:

>     Why is <spatialInformation> an appropriate tool for this advice?

I agree we are using <spatialInformation> to convey information that are 
not related to any physical space coordinates.
They are instead purely "rendering suggestions" - coordinates in a 
virtual space, the typical case where to use the "no scale" scale value, 
as Mark said.
They are two slightly different concepts ("real" spatial information and 
rendering information) that we express by means of the same element.

Due to the maybe misleading name, I am wondering if  we need to make 
more explicit such a "multiple" use of the <spatialInformation> element 
or if it is better to separate the two aspects.

What is your feeling about it?

Roberta

---

From: John Leslie <[email-masked]>
Date: Fri, 28 Mar 2014 11:55:15 -0400

> rendering information) that we express by means of the same element.

   That's what I thought (after much head-scratching) :^)

> What is your feeling about it?

   It would have saved me much head-scratching if they had different
names.

---

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 28 Mar 2014 09:43:46 -0700

The group spent a lot of time discussing and finalizing the spatial information as currently defined.  It has been very stable for a long time now.  I'd rather not re-open that discussion now, unless there is strong consensus in the group to do so.

My view is that both types of usages described below are real spatial information, but only one is directly derived from physical space.  They are distinguished by the "scale" attribute of <captureScene>.

Mark

> From: John Leslie [mailto:[email-masked]]
> Sent: Friday, March 28, 2014 11:55 AM
> To: Roberta Presta
> Cc: Duckworth, Mark; [email-masked]
> >> I'm struggling here. "To be rendered..." seems to be a suggestion to
> >> the receiver, with (AFAICT) no spatial relationship to anything in
> > I agree we are using <spatialInformation> to convey information that
> > are not related to any physical space coordinates.
> > virtual space, the typical case where to use the "no scale" scale
> > value, as Mark said.
> > They are two slightly different concepts ("real" spatial information
> > and rendering information) that we express by means of the same
> element.
> 
> > more explicit such a "multiple" use of the <spatialInformation>
> > element or if it is better to separate the two aspects.
> >
>    It would have saved me much head-scratching if they had different names.
> 
> --

---

From: John Leslie <[email-masked]>
Date: Fri, 28 Mar 2014 13:42:57 -0400

Duckworth, Mark <[email-masked]> wrote:
> 
> The group spent a lot of time discussing and finalizing the spatial
> information as currently defined.

   Indeed!

> It has been very stable for a long time now.

   Umm, Seems more "dormant" to me...

> I'd rather not re-open that discussion now, unless there is strong
> consensus in the group to do so.

   Hmm...

> My view is that both types of usages described below are real spatial
> information, but only one is directly derived from physical space.

   An interesting view...

   ISTM that one describes physical space (three-dimensional) and the
other describes a mathematical concept, instantiated on a visual
display. Mixing them is inherently confusing.

   (Arguments that the mathematical concept is a "space" do not move
me -- everything in math is sometimes described as a "space".)

   I seriously fear that we've only scratched the surface of the
confusion we've uncovered.

   I do understand that bikeshedding can prevent progress until it's
too late to consider the important issues. But keeping physical space
distinct from mathematical "space" strikes me as an important issue.

> They are distinguished by the "scale" attribute of <captureScene>.

   I doubt this. I remember only "millimeters", "unknown", and "noscale".
I can imagine folks using "noscale" for three-dimensional space.

   Also, I find "scale" a rather strange way to disambiguate physical
from mathematical space...

---

From: Christian Groves <[email-masked]>
Date: Tue, 01 Apr 2014 11:52:49 +1100

Hello

In some respects I agree with John. The discussions around the use of 
spatial information for audio captures just seemed to die out without 
any real conclusion. As John has alluded to the use of the spatial 
attributes for audio is problematic. We don't seem to have any text in 
the framework discussing this or forbidding the use of the spatial 
information for audio.

With regards to the video related spatial information I didn't realize 
that there was deep seated confusion over the use of the attributes.

 From a previous email:

I thought of the <captureArea> being a quadrilateral on a plane in 3D 
space, i.e. a 2D area. The <capturePoint> and <lineOfCapturePoint> were 
points in this 3D space giving the relationship between the capture 
device and the <captureArea>. This was in order to allow the Renderer to 
perform what ever geometric corrections it needed to do in order to 
provide an optimal rendering of the image on a screen i.e. quadrilateral 
on a plane.

In the case where the image has been through a device that performs the 
geometric corrections etc. (e.g. an MCU) the resultant image still 
represents an quadrilateral on a plane (albeit with Y=0). In terms of 
the <capturePoint> and <lineOfCapturePoint> there would be "virtual" 
values for these as a result of any transformations applied to the 
image. I guess the assumption is that is would be meaningless to send 
these as no further transformation requiring these points would be required.

Regards, Christian