Subject: [clue] Use Case and Framework question

From: Christian Groves <[email-masked]>
Date: Wed, 31 Aug 2011 20:25:04 +1000

Hello,

Whilst reviewing the use case document use case 4.1 point to point 
meeting talks about the possibility for separate monophonic audio 
streams. Now this presumably this allows for a microphone to be placed 
in front of each participant.

Now consider a three position (left to right) two row telepresence 
system. Each of these has a microphone i.e. 6 audio captures, Front row 
AC0, AC1, AC2, Back row AC3, AC4, AC5.

The current framework document considers video and audio captures to be 
left to right.
So its easy to describe the first row AC0, AC1, AC2. How would I 
describe using the current framework the audio captures for the second 
row microphones?

Even if we disregard the row (depth) element, how would I then say AC0 & 
AC3, AC1 & AC4 and AC3 & AC5 relate to the same left / centre / right 
position?

Regards, Christian

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 31 Aug 2011 07:47:50 -0400

The current framework draft allows multiple captures to signal the same
linear index.  So it is fine for AC0,AC3 and the the other pairs to all
signal the same index.  This covers the case where the two rows are captured
in the same video captures (that is, the left camera VC0 captures the left
side of both rows). If the video capture set is VC0, VC1, VC2,  it is
possible to associate both AC3 and AC0 with VC0.

There is a related case, which is when there are also independent cameras
(AC3 has a corresponding VC3).  The framework also allows multiple capture
sets, so you can signal VC3, VC4, VC5, AC3, AC4, AC5 as its own independent
capture set.

There's been quite a bit of discussion on how two "separate monophonic
streams" relate to a "stereo audio stream" amongst the authors of the
framework.  At this point I believe the conclusion is that the key
distinction between them is RTP transport.  A second possible distinction is
that some audio codecs allow a stereo audio stream to be jointly encoded,
providing somewhat better compression.

BTW, though it is convenient to think about audio captures as independent
microphones, it is important to keep in mind that there are other ways to
generate them.  For instance, a microphone array can use beam-forming
techniques to construct multiple captures from the microphone array.  This
can in principle be done with video, particularly if you also have depth
information. The framework does not describe how many sensors are used to
create a capture.

Stephen Botzko

On Wed, Aug 31, 2011 at 6:25 AM, Christian Groves <
[email-masked]> wrote:

> Hello,
>
> Whilst reviewing the use case document use case 4.1 point to point meeting
> talks about the possibility for separate monophonic audio streams. Now this
> presumably this allows for a microphone to be placed in front of each
> participant.
>
> Now consider a three position (left to right) two row telepresence system.
> Each of these has a microphone i.e. 6 audio captures, Front row AC0, AC1,
> AC2, Back row AC3, AC4, AC5.
>
> So its easy to describe the first row AC0, AC1, AC2. How would I describe
> using the current framework the audio captures for the second row
> microphones?
>
> ______________________________**_________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/clue<https://www.ietf.org/mailman/listinfo/clue>
>

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 31 Aug 2011 08:07:01 -0400

> AC3, AC1 & AC4 and AC3 & AC5 relate to the same left / centre / right
My own personal opinion is that

- it is dangerous and limiting to attempt to capture equipment spatial
placement in a simple ordered list, especially when that equipment (or its
capture zone) can move and change with time and that

- we are likely to be sending kilobytes of information about codec formats
and the like , so I don't see why we can't afford a few bytes to
describe a mapping between equipment ids (numbering) and what and where they
are capturing.

If you want a specific proposal to deal with this, here is one.

There are three possible rankings of spatial equipment capture, transverse,
depth and vertical. Call them T, D and V.

Each can be declared to be relevant, with the assumption that at least one
will be. Ones not declared relevant are ignored.

Each can be declared to be fixed or variable width. In the case of fixed
width, the width is also declared.

Units are degrees (T AND V) and meters (D).

Everything is referenced to the axis of symmetry of the unit, oriented from
the point of view of an observer standing on the axis at the capture
equipment facing the participants (i.e., the point of view of an outside
observer). Order is right to left (T), front to back (D) and bottom to top
(V) . A "starting" value is given (so that we don't have to deal with
negative numbers in the list) and a "unit" value (the smallest addressable
chunk of degrees or meters).

All spatially equipment is given an ID (could just be numbered, it doesn't
really matter).

So, assuming the above AC0-AC5, this situation could be dealt with as
something like

Declare T UNIT 30 STARTING -60
Declare D UNIT 2 STARTING 2

Order (ACO, 1,2), (AC1, 2, 2), (AC2, 3,2), (AC3, 1,3), (AC4, 2,3), (AC5,
3,3),

where (ACO, 1,1) means unit ACO covers T zone 1 (-60 to -30 degrees) and D
zone 2 (2 to 4 meters into the depth of field), etc.

If there was a camera that covered all 120 degrees, and 3 others that
covered 30 degrees each, that could be

Declare T VARIABLE UNIT 30 STARTING -60

Order (VC0, 1, 4), (VC1, 1, 1), (VC1, 1, 1), (VC2, 2, 1), (VC3, 3, 1),

where (VC0, 1, 4) means unit VC0 covers from -60 to +60 degrees (4 30 degree
slices), unit VC1 covers -60 to -30 degrees (1 slice), etc.

I don't see this as really much more complicated or byte consuming and it
really would avoid a bunch of problems I see as coming from overlaying a
potentially variable 3-D spatial order onto a simple list.

Regards
Marshall

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 31 Aug 2011 10:31:38 -0700

Thanks, Marshall.
I strongly support such an approach.  In fact, we should probably add a
similar mechanism to describe the rendering side.  Doing so would provide
MCUs that can encode multiple video streams to be displayed on a multitude
of monitors so much more leverage, accommodates for monitors of different
sizes, and so forth.
Stephan

From:  Marshall Eubanks <[email-masked]>
Date:  Wed, 31 Aug 2011 08:07:01 -0400
To:  Christian Groves <[email-masked]>
Cc:  <[email-masked]>
Subject:  Re: [clue] Use Case and Framework question

On Wed, Aug 31, 2011 at 6:25 AM, Christian Groves
<[email-masked]> wrote:
> Each of these has a microphone i.e. 6 audio captures, Front row AC0, AC1, AC2,
> Back row AC3, AC4, AC5.
> 
> The current framework document considers video and audio captures to be left
> to right.
> using the current framework the audio captures for the second row microphones?
> 
> Even if we disregard the row (depth) element, how would I then say AC0 & AC3,
> AC1 & AC4 and AC3 & AC5 relate to the same left / centre / right position?
> 

Regards
Marshall

> Regards, Christian
> 
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
> <https://www.ietf.org/mailman/listinfo/clue>

_______________________________________________ clue mailing list
[email-masked] https://www.ietf.org/mailman/listinfo/clue

---

From: Christian Groves <[email-masked]>
Date: Thu, 01 Sep 2011 14:19:57 +1000

Hello Stephen,

Please see my responses below.

Regards,
Christian

On 31/08/2011 9:47 PM, Stephen Botzko wrote:
> The current framework draft allows multiple captures to signal the 
> same linear index.  So it is fine for AC0,AC3 and the the other pairs 
> to all signal the same index.  This covers the case where the two rows 
> are captured in the same video captures (that is, the left camera VC0 
> captures the left side of both rows). If the video capture set is VC0, 
> VC1, VC2,  it is possible to associate both AC3 and AC0 with VC0.
[CNG] OK, linear index is used to indicate the position of an audio 
capture rather than the left/right behaviour?

>
> There is a related case, which is when there are also independent 
> cameras (AC3 has a corresponding VC3).  The framework also allows 
> multiple capture sets, so you can signal VC3, VC4, VC5, AC3, AC4, AC5 
> as its own independent capture set.
[CNG] However in this case you would need some "depth" parameter/s?

> distinction between them is RTP transport.  A second possible 
> distinction is that some audio codecs allow a stereo audio stream to 
> be jointly encoded, providing somewhat better compression.
[CNG] I had assumed that the monophonic audio streams were separate RTP 
streams. However my question was more looking at the limitation of the 
framework model rather than opening how audio is encoded and carried.

>
> BTW, though it is convenient to think about audio captures as 
> independent microphones, it is important to keep in mind that there 
> are other ways to generate them.  For instance, a microphone array can 
> use beam-forming techniques to construct multiple captures from the 
> microphone array.  This can in principle be done with video, 
> particularly if you also have depth information. The framework does 
> not describe how many sensors are used to create a capture.
[CNG] I agree. My questions came as a result of the use cases which talk 
about monophonic stream and microphones. In the beam-forming would the 
aspect of "depth" in addition to "linear index" also be useful?

> <[email-masked] <mailto:[email-masked]>> 
> wrote:
>
>     Hello,
>
>     streams. Now this presumably this allows for a microphone to be
>     placed in front of each participant.
>
>     system. Each of these has a microphone i.e. 6 audio captures,
>     Front row AC0, AC1, AC2, Back row AC3, AC4, AC5.
>
>     The current framework document considers video and audio captures
>     to be left to right.
>     describe using the current framework the audio captures for the
>     second row microphones?
>
>     Even if we disregard the row (depth) element, how would I then say
>     AC0 & AC3, AC1 & AC4 and AC3 & AC5 relate to the same left /
>     centre / right position?
>
>     Regards, Christian
>
>     [email-masked] <mailto:[email-masked]>