Subject: Re: [clue] Kick-Off Item 5: description of composed pictures

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 2 Nov 2011 13:42:08 -0700

Hi Stephan,

It seems to me the cases where this scheme might be useful are limited and not easily generalizable.  The mechanism you propose seems complicated for the limited usefulness.  Wouldn't it be better to work towards a way for the provider to be able to send the streams the consumer really wants, rather than composing and then decomposing?

If the composite image is constructed such that any of the individual images are cropped or obscured, for example with a picture-in-picture layout, I think your idea of decomposing will not work well.

If the MCU is cropping an image for composition, I'm not sure it will be able to provide accurate area of capture coordinates for the cropped image.  It won't be the same as the coordinates coming from the original provider.

Mark

From: [email-masked] [mailto:[email-masked]] On Behalf Of Stephan Wenger
Sent: Thursday, October 20, 2011 3:05 PM
To: Michael Hammer
Cc: [email-masked]
Subject: Re: [clue] Kick-Off Item 5: description of composed pictures

Hi Mike,

From: Michael Hammer <[email-masked]<mailto:[email-masked]>>
That sort of goal seems reasonable.  That would require that the mixing or aggregation process leaves the feed 1 and 2 segregated within the feed 3.  I imagine that some mixers might not do that, but it would provide flexibility.  So, you might need a de-composable flag in addition to distinguish if that can be done.  Anyway, that slides into the design.

The use case I was referring to is that the MCU performs video mixing in the traditional way MCUs perform tiling (that is: reconstruction of the video streams, aggregation on the pixel level, encoding).  In other words, feed 3 is a single RTP stream containing a single coded video stream.  To decompose that stream, you need spatial information in the renderer domain where each image is located in the tiled composed image.  That is what I want to add, in addition to the "decomposable" flag you are suggesting (which would simply be an indication that the spatial information is present for this RTP stream).

Stephan

From: Michael Hammer <[email-masked]<mailto:[email-masked]>>
Date: Thu, 20 Oct 2011 14:14:27 -0400
To: Stephan Wenger <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [clue] Kick-Off Item 5: description of composed pictures

Stephan,

What I am getting from this description is that you want the aggregated feed composed of original feeds 1 and 2 to have sufficient data about the original feeds to enable an output site to do a de-aggregation of feed 3 back into its components and to put on available displays as desired.

That sort of goal seems reasonable.  That would require that the mixing or aggregation process leaves the feed 1 and 2 segregated within the feed 3.  I imagine that some mixers might not do that, but it would provide flexibility.  So, you might need a de-composable flag in addition to distinguish if that can be done.  Anyway, that slides into the design.

Mike
On Wed, Oct 19, 2011 at 4:35 PM, Stephan Wenger <[email-masked]<mailto:[email-masked]>> wrote:
Mike,

Not sure this is going through to the list, due to the size of the attachment.  But let's try.

Attached is a little hand-drawing that hopefully helps understanding what I am shooting for.
Consider a three way conference.  The two "capturing" rooms, depicted on the left side, have two cameras each, capturing scenes one and two (for the first room), and four and five (for the second room), respectively.  The captured content is encoded by encoders marked "E" to create RTP streams 1, 2, 4, and 5.
RTP streams 1 and 2 (from the first room) terminate an an MCU, which composes them into a composed picture 3, encoded into RTP stream 3.  The composed picture contains 1 and 2 side-by-side, plus grey bars on top and bottom.
The monitors of the receiving third endpoint are shown on the right side.  Streams 4 and 5 go straight to the two decoders of the main projection wall.  Stream 3 (created by the MCU) goes through a side decoder to a third monitor (could be the monitors that many telepresence rooms have for each participant, a monitor normally dedicated for shared content, what not).  This monitor shows the composed image.

The CLUE architecture allows perfectly well to describe the capture properties of streams 4 and 5.  However, at present, the MCU hides the capture properties of streams 1 and 2.  Similarly, the MCU hides the render properties of the decoder/display of stream 3.  Accordingly, the geographic parameters of captures 1 and 2 are not known at decoder 3, and, if that decoder would want to do something a bit more sophisticated than just decoding and displaying on the screen, it couldn't because it lacks info.

One of this "more sophisticated" things could, for example be as follows.  Suppose the endpoint capturing 4 and 5 drops out of the call.  The video wall of the rendering endpoint goes blank.  It would be desirable if the info from 1 and 2 would be rendered on the video wall instead of the side screen.  One sensible way to do that would be that the stream 3 is feed into both decoders responsible for the video wall, but "cut out" and upscale the content from the composed picture created by the MCU, before rendering.  (There may be technical reasons for the presence of the MCU beyond the fact that there are only three decoders at the receiving side).  However, that's not possible as the nature of stream 3 (being composed form 1 and 2) is not visible at the rendering endpoint.

My proposal is to add information related to the captures inside a composed stream (beyond that one flag that says the stream is composed).

Clearer?

Stephan

From: Michael Hammer <[email-masked]<mailto:[email-masked]>>
Date: Sat, 15 Oct 2011 13:03:07 -0400
To: Stephan Wenger <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [clue] Kick-Off Item 5: description of composed pictures

Stephan,

You use monitor as both input and output device.  I hope that is not embedded in the scheme.
The rest of this email seems to focus only on input definitions as it probably should.

Don't understand the comment about 3-D followed by coordinates in 3-D.  It seems like 3-D is fundamental.

Do I understand correctly that the camera coordinates define the location of the lense, while the bitstream is abstraction of the pixel layout for some abstraction (via ray-tracing?) of a window?

Or is the camera coordinates defining some virtual window in space that through some changes in focal length could be a closeup and those farther from the camera and narrower field of view versus wider angel and closer to the lense with greater field of view?

When talking about spatial things, perhaps a picture would help save a thousand words.

Also, I would think that what a renderer or MCU would do could be a second discussion once the capture scheme is better understood.  (Understand that the impacts of those might drive some iterations on the basic scheme.)

Mike
On Fri, Oct 14, 2011 at 11:02 PM, Stephan Wenger <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

Here is the kickoff message as agreed in our F2F.

We are in the process of defining a mechanism that allows us to tag in room "coordinates" of captures to RTP streams.  In its most trivial incarnation, that would allow us to assign the decoded RTP streams to devices such as individual monitors.

RTP <id#1> comes from system 1, camera coordinates x, y, z, upper right x, y, z, lower left x, y, z
RTP <id#2> comes from system 1, camera coordinates (something else)

Note that we probably need a third and even a fourth point for faithfully specifying a capture in 3D space, but that's not the subject of this email, so don't beat me up on it right now.

As for the capabilities of a renderer (not defined in the framework yet), it should be clear that a monitor can trivially display more than one decoded RTP stream.  Insofar, the mapping should be per part of a virtual display wall (or however you want to call it), and not per physical screen.  I would hope that this is uncontroversial.

Things become more tricky when there is a video mixing MCU in the picture, or when an endpoint creates a single RTP stream that contains information from multiple camera devices (through tiling, picture-in-picture).  Here, we cannot meaningfully tag the a single geographic information to a single RTP stream simply because the multiple video sequences were mixed at the sample level, and the encoder didn't have a clue that it is actually encoding data stemming from more than one camera.  However, the endpoint system (or the MCU) know the geographic metadata associated with the composed picture.  So what I'm proposing is that we allow a single RTP stream to have more than one geometric information about the capture involved.  For video, one would probably have to associate regions of the reconstructed video stream (in the spatial dimension) with capture geographic information as currently proposed in the framework.

Example: a sending endpoint creating ONE RTP stream which includes TWO captures, tiled horizontally

RTP <id#2> {
    bitstream coordinates x=0 to 959, y=0, to 1080 , comes from system 1, camera coordinates x, y, z, x, y,z x, y, z
    bitstream coordinates x=960 to 1919, y=0 to 1080, comes from system 1, camera coordinates (something else)
}

Architecturally, this system would best need to be recursive, as it ought to be possible to throw an already composed image from MCU1 at the input of MCU2, which scales it down and composes it with other images to other ports.

Clear enough?

Thanks,
Stephan

_______________________________________________
clue mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/clue

_______________________________________________
clue mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/clue

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 02 Nov 2011 16:14:33 -0700

Hi Mark,
Please see inline.
Stephan

From:  "Duckworth, Mark" <[email-masked]>
Date:  Wed, 2 Nov 2011 13:42:08 -0700
To:  "[email-masked]" <[email-masked]>
Subject:  Re: [clue] Kick-Off Item 5: description of composed pictures

Hi Stephan,
 
It seems to me the cases where this scheme might be useful are limited and
not easily generalizable.  The mechanism you propose seems complicated for
the limited usefulness.  Wouldn¹t it be better to work towards a way for the
provider to be able to send the streams the consumer really wants, rather
than composing and then decomposing?

Oh, absolutely.  In fact, this is what our systems do.  However,
occasionally we get stuck with legacy equipment, connected to your system
over a gateway.  Our gateway (or endpoint) system has no standardized way to
decompose a composed picture, because this critical piece of metainfo is
missing.
 
If the composite image is constructed such that any of the individual images
are cropped or obscured, for example with a picture-in-picture layout, I
think your idea of decomposing will not work well.

True.
 
If the MCU is cropping an image for composition, I¹m not sure it will be
able to provide accurate area of capture coordinates for the cropped image.
It won¹t be the same as the coordinates coming from the original provider.

Also true, though, in this case, the MCU should be able to intercept the
CLUE messages and modify them appropriately.  Or stop cropping.

I don't think that technical issues are overly difficult to overcome.
Certainly not the two you mentioned.  The question is, should we address
this (real-world) use case now, later, or never.  My thinking is that most
of the building blocks to describe spatial relationships are already in
place; why not extend them to cover this use case now?  As the very minimum,
let's flesh out a framework (sic!) for a solution for the use case, and make
sure that the overall framework would architecturally allow for it.

Stephan
 
Mark

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Stephan Wenger
Sent: Thursday, October 20, 2011 3:05 PM
To: Michael Hammer
Cc: [email-masked]
Subject: Re: [clue] Kick-Off Item 5: description of composed pictures

Hi Mike,

> 
> From: Michael Hammer <[email-masked]>
> 
> That sort of goal seems reasonable.  That would require that the mixing or
> aggregation process leaves the feed 1 and 2 segregated within the feed 3.  I
> imagine that some mixers might not do that, but it would provide flexibility.
> So, you might need a de-composable flag in addition to distinguish if that can
> be done.  Anyway, that slides into the design.

The use case I was referring to is that the MCU performs video mixing in the
traditional way MCUs perform tiling (that is: reconstruction of the video
streams, aggregation on the pixel level, encoding).  In other words, feed 3
is a single RTP stream containing a single coded video stream.  To decompose
that stream, you need spatial information in the renderer domain where each
image is located in the tiled composed image.  That is what I want to add,
in addition to the "decomposable" flag you are suggesting (which would
simply be an indication that the spatial information is present for this RTP
stream).

Stephan

From: Michael Hammer <[email-masked]>
Date: Thu, 20 Oct 2011 14:14:27 -0400
To: Stephan Wenger <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [clue] Kick-Off Item 5: description of composed pictures

Stephan,

What I am getting from this description is that you want the aggregated feed
composed of original feeds 1 and 2 to have sufficient data about the
original feeds to enable an output site to do a de-aggregation of feed 3
back into its components and to put on available displays as desired.

That sort of goal seems reasonable.  That would require that the mixing or
aggregation process leaves the feed 1 and 2 segregated within the feed 3.  I
imagine that some mixers might not do that, but it would provide
flexibility.  So, you might need a de-composable flag in addition to
distinguish if that can be done.  Anyway, that slides into the design.

Mike

On Wed, Oct 19, 2011 at 4:35 PM, Stephan Wenger <[email-masked]> wrote:

Mike,

Not sure this is going through to the list, due to the size of the
attachment.  But let's try.

Attached is a little hand-drawing that hopefully helps understanding what I
am shooting for.

Consider a three way conference.  The two "capturing" rooms, depicted on the
left side, have two cameras each, capturing scenes one and two (for the
first room), and four and five (for the second room), respectively.  The
captured content is encoded by encoders marked "E" to create RTP streams 1,
2, 4, and 5.

RTP streams 1 and 2 (from the first room) terminate an an MCU, which
composes them into a composed picture 3, encoded into RTP stream 3.  The
composed picture contains 1 and 2 side-by-side, plus grey bars on top and
bottom.  

The monitors of the receiving third endpoint are shown on the right side.
Streams 4 and 5 go straight to the two decoders of the main projection wall.
Stream 3 (created by the MCU) goes through a side decoder to a third monitor
(could be the monitors that many telepresence rooms have for each
participant, a monitor normally dedicated for shared content, what not).
This monitor shows the composed image.

The CLUE architecture allows perfectly well to describe the capture
properties of streams 4 and 5.  However, at present, the MCU hides the
capture properties of streams 1 and 2.  Similarly, the MCU hides the render
properties of the decoder/display of stream 3.  Accordingly, the geographic
parameters of captures 1 and 2 are not known at decoder 3, and, if that
decoder would want to do something a bit more sophisticated than just
decoding and displaying on the screen, it couldn't because it lacks info.

One of this "more sophisticated" things could, for example be as follows.
Suppose the endpoint capturing 4 and 5 drops out of the call.  The video
wall of the rendering endpoint goes blank.  It would be desirable if the
info from 1 and 2 would be rendered on the video wall instead of the side
screen.  One sensible way to do that would be that the stream 3 is feed into
both decoders responsible for the video wall, but "cut out" and upscale the
content from the composed picture created by the MCU, before rendering.
(There may be technical reasons for the presence of the MCU beyond the fact
that there are only three decoders at the receiving side).  However, that's
not possible as the nature of stream 3 (being composed form 1 and 2) is not
visible at the rendering endpoint.

My proposal is to add information related to the captures inside a composed
stream (beyond that one flag that says the stream is composed).

Clearer?

Stephan    

 

From: Michael Hammer <[email-masked]>
Date: Sat, 15 Oct 2011 13:03:07 -0400
To: Stephan Wenger <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [clue] Kick-Off Item 5: description of composed pictures

Stephan,

You use monitor as both input and output device.  I hope that is not
embedded in the scheme.

The rest of this email seems to focus only on input definitions as it
probably should.

Don't understand the comment about 3-D followed by coordinates in 3-D.  It
seems like 3-D is fundamental.

Do I understand correctly that the camera coordinates define the location of
the lense, while the bitstream is abstraction of the pixel layout for some
abstraction (via ray-tracing?) of a window?

Or is the camera coordinates defining some virtual window in space that
through some changes in focal length could be a closeup and those farther
from the camera and narrower field of view versus wider angel and closer to
the lense with greater field of view?

When talking about spatial things, perhaps a picture would help save a
thousand words.

Also, I would think that what a renderer or MCU would do could be a second
discussion once the capture scheme is better understood.  (Understand that
the impacts of those might drive some iterations on the basic scheme.)

Mike

On Fri, Oct 14, 2011 at 11:02 PM, Stephan Wenger <[email-masked]> wrote:

Hi,

We are in the process of defining a mechanism that allows us to tag in room
"coordinates" of captures to RTP streams.  In its most trivial incarnation,
that would allow us to assign the decoded RTP streams to devices such as
individual monitors.

RTP <id#1> comes from system 1, camera coordinates x, y, z, upper right x,
y, z, lower left x, y, z

Note that we probably need a third and even a fourth point for faithfully
specifying a capture in 3D space, but that's not the subject of this email,
so don't beat me up on it right now.

As for the capabilities of a renderer (not defined in the framework yet), it
should be clear that a monitor can trivially display more than one decoded
RTP stream.  Insofar, the mapping should be per part of a virtual display
wall (or however you want to call it), and not per physical screen.  I would
hope that this is uncontroversial.

Things become more tricky when there is a video mixing MCU in the picture,
or when an endpoint creates a single RTP stream that contains information
from multiple camera devices (through tiling, picture-in-picture).  Here, we
cannot meaningfully tag the a single geographic information to a single RTP
stream simply because the multiple video sequences were mixed at the sample
level, and the encoder didn't have a clue that it is actually encoding data
stemming from more than one camera.  However, the endpoint system (or the
MCU) know the geographic metadata associated with the composed picture.  So
what I'm proposing is that we allow a single RTP stream to have more than
one geometric information about the capture involved.  For video, one would
probably have to associate regions of the reconstructed video stream (in the
spatial dimension) with capture geographic information as currently proposed
in the framework.

Example: a sending endpoint creating ONE RTP stream which includes TWO
captures, tiled horizontally

RTP <id#2> {

    bitstream coordinates x=0 to 959, y=0, to 1080 , comes from system 1,
camera coordinates x, y, z, x, y,z x, y, z

    bitstream coordinates x=960 to 1919, y=0 to 1080, comes from system 1,
camera coordinates (something else)

}

Architecturally, this system would best need to be recursive, as it ought to
be possible to throw an already composed image from MCU1 at the input of
MCU2, which scales it down and composes it with other images to other ports.

Clear enough?

Thanks,

Stephan

[email-masked]

[email-masked]

_______________________________________________ clue mailing list
[email-masked] https://www.ietf.org/mailman/listinfo/clue

---

From: "Andrew Pepperell (apeppere)" <[email-masked]>
Date: Thu, 3 Nov 2011 06:00:41 +0100

I'm not particularly enamored by the idea of decomposing a stream into
its constituents - as Mark points out, some layouts (whether due to
overlaid panes or cropping) may not lend themselves to it and there
might also be issues when upscaling - different sized panes' borders /
text annotations may look ugly / mismatched when scaled up.

However, there's certainly a good argument in favor of having a method
for the provider to signal to the consumer some detail of what the
composed (and also switched) stream contains in terms of mapping pixel
areas to constituent captures (it would be hard to argue against this as
given that we implemented and shipped just such a scheme as Tandberg a
couple of years ago). I think we should be looking to add a provider ->
consumer message that would allow the provider to inform the consumer of
which streams are being shown where, though this should be strictly
optional on the part of the provider and be orthogonal to any mechanisms
for layout, stream, or policy choice on the part of the consumer .
Stephan, you'd be welcome to use this information to decompose a stream
if you so wished, though I do think I would ever do so myself J

The idea of making this recursive and for one MCU to receive this
information and then incorporate / embed it in a multiply-composed
stream also seems sensible (even though I think composing streams in
this way isn't ideal MCU behavior). I'm not sure how the camera X, Y, Z
coordinates would work out in such a scheme though, as different source
systems' coordinates may not be easy to relate to each other - an
alternative would be potentially normalize them when forming the
composition description so that you knew, say, what was being shown on
the left half of the composed stream was the rightmost third of the
source system.

Regards,

Andy

 

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Stephan Wenger
Sent: Wednesday, November 02, 2011 11:15 PM
To: Duckworth, Mark; [email-masked]
Subject: Re: [clue] Kick-Off Item 5: description of composed pictures

Hi Mark,

Please see inline.

Stephan

Hi Stephan,

It seems to me the cases where this scheme might be useful are limited
and not easily generalizable.  The mechanism you propose seems
complicated for the limited usefulness.  Wouldn't it be better to work
towards a way for the provider to be able to send the streams the
consumer really wants, rather than composing and then decomposing?

Oh, absolutely.  In fact, this is what our systems do.  However,
occasionally we get stuck with legacy equipment, connected to your
system over a gateway.  Our gateway (or endpoint) system has no
standardized way to decompose a composed picture, because this critical
piece of metainfo is missing.

If the composite image is constructed such that any of the individual
images are cropped or obscured, for example with a picture-in-picture
layout, I think your idea of decomposing will not work well.

True.

If the MCU is cropping an image for composition, I'm not sure it will be
able to provide accurate area of capture coordinates for the cropped
image.  It won't be the same as the coordinates coming from the original
provider.

I don't think that technical issues are overly difficult to overcome.
Certainly not the two you mentioned.  The question is, should we address
this (real-world) use case now, later, or never.  My thinking is that
most of the building blocks to describe spatial relationships are
already in place; why not extend them to cover this use case now?  As
the very minimum, let's flesh out a framework (sic!) for a solution for
the use case, and make sure that the overall framework would
architecturally allow for it.

Stephan

Mark

Hi Mike,

	That sort of goal seems reasonable.  That would require that the
mixing or aggregation process leaves the feed 1 and 2 segregated within
the feed 3.  I imagine that some mixers might not do that, but it would
provide flexibility.  So, you might need a de-composable flag in
addition to distinguish if that can be done.  Anyway, that slides into
the design.

The use case I was referring to is that the MCU performs video mixing in
the traditional way MCUs perform tiling (that is: reconstruction of the
video streams, aggregation on the pixel level, encoding).  In other
words, feed 3 is a single RTP stream containing a single coded video
stream.  To decompose that stream, you need spatial information in the
renderer domain where each image is located in the tiled composed image.
That is what I want to add, in addition to the "decomposable" flag you
are suggesting (which would simply be an indication that the spatial
information is present for this RTP stream).

Stephan

Stephan,

What I am getting from this description is that you want the aggregated
feed composed of original feeds 1 and 2 to have sufficient data about
the original feeds to enable an output site to do a de-aggregation of
feed 3 back into its components and to put on available displays as
desired.

That sort of goal seems reasonable.  That would require that the mixing
or aggregation process leaves the feed 1 and 2 segregated within the
feed 3.  I imagine that some mixers might not do that, but it would
provide flexibility.  So, you might need a de-composable flag in
addition to distinguish if that can be done.  Anyway, that slides into
the design.

Mike

Mike,

Attached is a little hand-drawing that hopefully helps understanding
what I am shooting for.

Consider a three way conference.  The two "capturing" rooms, depicted on
the left side, have two cameras each, capturing scenes one and two (for
the first room), and four and five (for the second room), respectively.
The captured content is encoded by encoders marked "E" to create RTP
streams 1, 2, 4, and 5.

RTP streams 1 and 2 (from the first room) terminate an an MCU, which
composes them into a composed picture 3, encoded into RTP stream 3.  The
composed picture contains 1 and 2 side-by-side, plus grey bars on top
and bottom.  

The monitors of the receiving third endpoint are shown on the right
side.  Streams 4 and 5 go straight to the two decoders of the main
projection wall.  Stream 3 (created by the MCU) goes through a side
decoder to a third monitor (could be the monitors that many telepresence
rooms have for each participant, a monitor normally dedicated for shared
content, what not).  This monitor shows the composed image.

The CLUE architecture allows perfectly well to describe the capture
properties of streams 4 and 5.  However, at present, the MCU hides the
capture properties of streams 1 and 2.  Similarly, the MCU hides the
render properties of the decoder/display of stream 3.  Accordingly, the
geographic parameters of captures 1 and 2 are not known at decoder 3,
and, if that decoder would want to do something a bit more sophisticated
than just decoding and displaying on the screen, it couldn't because it
lacks info.

One of this "more sophisticated" things could, for example be as
follows.  Suppose the endpoint capturing 4 and 5 drops out of the call.
The video wall of the rendering endpoint goes blank.  It would be
desirable if the info from 1 and 2 would be rendered on the video wall
instead of the side screen.  One sensible way to do that would be that
the stream 3 is feed into both decoders responsible for the video wall,
but "cut out" and upscale the content from the composed picture created
by the MCU, before rendering.  (There may be technical reasons for the
presence of the MCU beyond the fact that there are only three decoders
at the receiving side).  However, that's not possible as the nature of
stream 3 (being composed form 1 and 2) is not visible at the rendering
endpoint.

My proposal is to add information related to the captures inside a
composed stream (beyond that one flag that says the stream is composed).

Clearer?

Stephan    

 

Stephan,

Don't understand the comment about 3-D followed by coordinates in 3-D.
It seems like 3-D is fundamental.

Do I understand correctly that the camera coordinates define the
location of the lense, while the bitstream is abstraction of the pixel
layout for some abstraction (via ray-tracing?) of a window?

Or is the camera coordinates defining some virtual window in space that
through some changes in focal length could be a closeup and those
farther from the camera and narrower field of view versus wider angel
and closer to the lense with greater field of view?

Also, I would think that what a renderer or MCU would do could be a
second discussion once the capture scheme is better understood.
(Understand that the impacts of those might drive some iterations on the
basic scheme.)

Mike

On Fri, Oct 14, 2011 at 11:02 PM, Stephan Wenger <[email-masked]>
wrote:

Hi,

We are in the process of defining a mechanism that allows us to tag in
room "coordinates" of captures to RTP streams.  In its most trivial
incarnation, that would allow us to assign the decoded RTP streams to
devices such as individual monitors.

RTP <id#1> comes from system 1, camera coordinates x, y, z, upper right
x, y, z, lower left x, y, z 

Note that we probably need a third and even a fourth point for
faithfully specifying a capture in 3D space, but that's not the subject
of this email, so don't beat me up on it right now. 

As for the capabilities of a renderer (not defined in the framework
yet), it should be clear that a monitor can trivially display more than
one decoded RTP stream.  Insofar, the mapping should be per part of a
virtual display wall (or however you want to call it), and not per
physical screen.  I would hope that this is uncontroversial.

Things become more tricky when there is a video mixing MCU in the
picture, or when an endpoint creates a single RTP stream that contains
information from multiple camera devices (through tiling,
picture-in-picture).  Here, we cannot meaningfully tag the a single
geographic information to a single RTP stream simply because the
multiple video sequences were mixed at the sample level, and the encoder
didn't have a clue that it is actually encoding data stemming from more
than one camera.  However, the endpoint system (or the MCU) know the
geographic metadata associated with the composed picture.  So what I'm
proposing is that we allow a single RTP stream to have more than one
geometric information about the capture involved.  For video, one would
probably have to associate regions of the reconstructed video stream (in
the spatial dimension) with capture geographic information as currently
proposed in the framework.

RTP <id#2> {

    bitstream coordinates x=0 to 959, y=0, to 1080 , comes from system
1, camera coordinates x, y, z, x, y,z x, y, z

    bitstream coordinates x=960 to 1919, y=0 to 1080, comes from system
1, camera coordinates (something else) 

}

Architecturally, this system would best need to be recursive, as it
ought to be possible to throw an already composed image from MCU1 at the
input of MCU2, which scales it down and composes it with other images to
other ports.

Clear enough?

Thanks,

Stephan