Subject: [clue] Use of the media capture "composed" attribute

From: Andy Pepperell <[email-masked]>
Date: Wed, 21 Mar 2012 17:21:40 +0000

All,

The framework author team have been having a few discussions recently on
the topic of the proposed "composed" attribute, with a view to clearing up
our own and the wider group's confusion on this topic, and I thought it
would be useful to set out our current position here:

The original intent behind this attribute was to allow a media stream
consumer to distinguish between "natural" media (such as something that
could be viewed by a lens or human eye within a room) and "artificial"
media (such as a 2x2 composed layout generated by an MCU-style middle box).
The main need for this distinction was deemed to be the case where the
consumer was itself a middle box / MCU intending to form its own multi-pane
compositions, either through transcoding or by forwarding on switched
"loudest speaker" streams. In these cases, such devices would want to avoid
already-composed video captures (or, more specifically, any capture scene
entries with one or more captures so tagged).

With more thought on this, we also came to the conclusion that the use for
this attribute for video captures doesn't necessarily translate exactly
equivalently to audio captures. In a network of cascaded MCUs it *would*
often be perfectly acceptable to use a pre-mixed audio stream from a peer
MCU whereas it would be highly desirable to not use pre-composed,
multi-pane, video from those media providers. Largely because of the
potential different use of the attribute by media stream consumers, our
thinking is that we *shouldn't* define the "composed" attribute to apply to
both video and audio captures, and instead use, if required, a separate
"mixed" attribute for the audio case.

For the video aspect of this same scenario (i.e. a 3 camera room), we'd see
neither the 3 separate camera video captures nor a zoomed out central
camera needing the "composed" attribute. A case where a video capture would
be tagged as "composed" in this scenario would be where the endpoint was
sending a single video capture formed of the loudest panel plus, say, a
small version of all 3 cameras' streams superimposed on the image.

For the specific case of a 3 microphone room which can either provide 3
separate streams to the far end or a single pre-mixed audio stream, we'd
anticipate none of these audio captures necessarily being tagged as
"mixed", as this expresses no more information than is provided by the fact
that the 3 capture entry and single capture entry appear as equivalents in
the same capture scene.

Hope this clarification is useful (or at least promotes some more healthy
debate :) )

Andy

---

From: "Roni Even" <[email-masked]>
Date: Wed, 21 Mar 2012 20:32:23 +0200

Hi Andy,

I am not sure what is the issue since the MCU knows if this is an audio or
video stream and can handle them differently like all MCUs will do today.
For cascading there is a need to know if the consumer is an MCU in order to
advertise  the right information. This may still be a composed video from
the end point (like current speaker mixed with pip of others) but not
composed from an MCU.

I am not sure why you do not want to have the mixed audio from the room as
mixed. How do you know if it is mixed or just a pickup of the whole room by
one microphone from the three if both advertisements will have the whole
room co-ordinates, and like I said above the consumer will know if this is
audio or video and how to handle it.

Roni

From: [email-masked] [mailto:[email-masked]] On Behalf Of Andy
Pepperell
Sent: Wednesday, March 21, 2012 7:22 PM
To: [email-masked]
Subject: [clue] Use of the media capture "composed" attribute

All,

The framework author team have been having a few discussions recently on the
topic of the proposed "composed" attribute, with a view to clearing up our
own and the wider group's confusion on this topic, and I thought it would be
useful to set out our current position here:

The original intent behind this attribute was to allow a media stream
consumer to distinguish between "natural" media (such as something that
could be viewed by a lens or human eye within a room) and "artificial" media
(such as a 2x2 composed layout generated by an MCU-style middle box). The
main need for this distinction was deemed to be the case where the consumer
was itself a middle box / MCU intending to form its own multi-pane
compositions, either through transcoding or by forwarding on switched
"loudest speaker" streams. In these cases, such devices would want to avoid
already-composed video captures (or, more specifically, any capture scene
entries with one or more captures so tagged).

For the video aspect of this same scenario (i.e. a 3 camera room), we'd see
neither the 3 separate camera video captures nor a zoomed out central camera
needing the "composed" attribute. A case where a video capture would be
tagged as "composed" in this scenario would be where the endpoint was
sending a single video capture formed of the loudest panel plus, say, a
small version of all 3 cameras' streams superimposed on the image.

For the specific case of a 3 microphone room which can either provide 3
separate streams to the far end or a single pre-mixed audio stream, we'd
anticipate none of these audio captures necessarily being tagged as "mixed",
as this expresses no more information than is provided by the fact that the
3 capture entry and single capture entry appear as equivalents in the same
capture scene.

Andy

---

From: John Leslie <[email-masked]>
Date: Wed, 21 Mar 2012 16:23:55 -0400

Roni Even <[email-masked]> wrote:
> [Andy Pepperell wrote:]

   I'm not sure it needs to be "separate", but I would hope we have a
tag to show that two or more microphones in different positions were
"mixed" into a single audio stream.

   I view this as a warning label -- and a single bit suffices unless
we include details of the microphones being mixed (which I don't believe
is the current plan).

>> 3 capture entry and single capture entry appear as equivalents in the same

   I don't understand: why wouldn't the pre-mixed stream be labeled
"mixed"?

> video stream and can handle them differently like all MCUs will do today.

   (Use of the same bit seems OK...)

> the end point (like current speaker mixed with pip of others) but not

   I don't understand Roni here: surely we can set the "composed" bit
if we're sending split-screen regardless of who will receive that stream.

> room co-ordinates,

   Hopefully any software will know if _it_ is mixing two streams: it
would have to depend on an operator to specify that one of the streams
is from a mixer. Since I view this as a warning label, I'd like it to
be passed on.

> and like I said above the consumer will know if this is audio or video

   Yes.

> and how to handle it.

   In many cases, it would be handled the same whether mixed or not; but
I can imagine future enhancements which process the audio streams to
different apparent positions in multi-channel audio schemes.

   For our first pass, it's arguably sufficient to take a single mixed
audio stream for each room and process it to a stereo position. But if
we receive more than one audio stream from a room, processing them to
different stereo positions might help.

   (The actual use of a "mixed" bit, as I imagine it, would be to
process the audio to a less-distinct position, and probably softer
than any un-mixed audio streams from that room. If I were also adding
impulse-response, I'd probably add less for a "mixed" stream. The
point is, while the differences are subtle, subtle audio clues can
be very helpful.)

--
John Leslie <[email-masked]>

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 26 Mar 2012 14:47:31 +0200

> we include details of the microphones being mixed (which I don't believe
[sb]  Are you suggesting that every system using a steered microphone array
would be tagged as "mixed"?  I don't see that as helpful, so I think we
would need to define the meaning of "mixed" here carefully.

> "mixed",
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: John Leslie <[email-masked]>
Date: Tue, 27 Mar 2012 00:12:55 -0400

Stephen Botzko <[email-masked]> wrote:
>>>> With more thought on this, we also came to the conclusion that the use
>>>> for this attribute for video captures doesn't necessarily translate
>>>> exactly equivalently to audio captures. In a network of cascaded MCUs
>>>> it *would* often be perfectly acceptable to use a pre-mixed audio
>>>> stream from a peer MCU whereas it would be highly desirable to not
>>>> use pre-composed, multi-pane, video from those media providers.

   (I agree that pre-mixed audio may sometimes be preferred to doing
your own mixing, for the same room where pre-composed video would _not_
be preferred.)

>>>> Largely because of the potential different use of the attribute by
>>>> media stream consumers, our thinking is that we *shouldn't* define
>>>> the "composed" attribute to apply to both video and audio captures,
>>>> and instead use, if required, a separate "mixed" attribute for the
>>>> audio case.
>>
> would be tagged as "mixed"?

   Actually, yes: in the case of a steered _array_ of microphones, the
arrival-time information will be lost, just as in an audio mixer with
multiple microphones.

   This is distinct from the case of a steered "shotgun" microphone where
the mike position is fixed (though its directional pattern changes), thus
arrival-time information is preserved.

> I don't see that as helpful, so I think we would need to define the
> meaning of "mixed" here carefully.

   We probably do need a careful definition...

   To me, the essential point is that a mixer greatly obscures any
ability to process for actual physical location of the microphone --
thus I'm probably going to fail if I try to aid the listeners in
recovering the actual physical location of the speaker(s).

   Of course, regardless of our definition, such a "mixed" bit will end
up being set wrong sometimes -- but I'd still like the warning label.

   In an actual room, listeners with binaural hearing determine the
speaker's location quite unconsciously -- but they do use physical
position as one input into speaker recognition (and where to focus
their attention). When a remote speaker is presented through a
monophonic speaker, they describe it as a "dissociated voice" and
are subconsciously confused.

   Given "mixed" audio, I can still process it to a single apparent
position, but I can't relate that position to any other sound source;
nor can I usefully vary it as different speakers are included in that
audio stream.

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 27 Mar 2012 16:14:20 +0200

On Tue, Mar 27, 2012 at 6:12 AM, John Leslie <[email-masked]> wrote:

> [sb] I believe steered microphone arrays are used commonly in sonar
acquisition, so I don't believe arrival time information has to be lost.
Usually the microphones in such an array are in fixed positions.

---

From: John Leslie <[email-masked]>
Date: Tue, 27 Mar 2012 18:32:12 -0400

Stephen Botzko <[email-masked]> wrote:
>>> [sb]  Are you suggesting that every system using a steered microphone
>>> array would be tagged as "mixed"?
>>
>> acquisition,

   This is true...

>> so I don't believe arrival time information has to be lost.

   As usual, the devil is in the details...

   In sonar, there's a very clearly-positioned sound source, and the
return times of the microphone array are calibrated to that source.
(Actually, the source likely produces a "chirp" of varying wavelength
and the phase differences are processed during the chirp response.)

   In our microphone array, the position of the sound source is
variable and there's no control of its wavelengths. It is technically
possible to compare phase information of several microphones and
process the sound output to that which "would be received" by a mike
at a given position (within the range of the microphone array, of
course); but if any existing system does that, I haven't heard of it.

   Again, we're going to have to trust operators to set the "mixed"
bit: we can only offer guidance...

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 27 Mar 2012 23:34:42 +0200

[I don't know if this is as chair or individual. Maybe both.]

The discussion of the composed and mixed attributes today exposed a lot 
of vagueness about what they mean. I think this needs to be firmed up 
considerably.

Here are some thoughts of my own just to keep the discussion going:

Regarding mixed (audio):

The fact that an audio capture has mixed content doesn't preclude mixing 
it again. But there are properties of it that might make such mixing 
unsatisfactory. For instance, if AC3 is a mix of (AC1, AC2), then it is 
ok to mix AC3 and independent capture AC4. But it would be unwise to mix 
it with AC2 again. And it would also be unwise to mix it with AC5 that 
is itself a mix of (AC1, AC4).

It *might* also be the case that a mix is unwise if it includes 
(transitively) too many original inputs.

Fundamentally it seems like the decision of whether a mix makes sense 
requires some complex decision making based on way more data than 
whether the immediate inputs are themselves mixed or not.

Regarding composed (video):

This seems potentially more complex. So far the only use I have heard 
for this attribute is to drive a decision about whether it makes sense 
to compose this capture with others. But I expect that isn't always true 
either. You could have one uncomposed capture that shows two people 
(heads) side by side. You could have another *composed* capture that 
also shows two people in independently side by side each inside a frame, 
that were assembled from other captures. (Would anyone argue that this 
isn't composed?) But I can't see why it would be any more or less 
desirable to compose one of these captures than another.

Again it seems like you need more information about what has been 
composed, and at what scale (and maybe position) before you can make a 
decision about whether it makes sense to compose it. And then that may 
also depend on *how* you want to compose it.

If mixed and composed aren't intended to drive decisions about further 
mixing and composing, then what *are* they for?

	Thanks,
	Paul

---

From: "Duckworth, Mark" <[email-masked]>
Date: Wed, 28 Mar 2012 01:52:35 -0700

> mixing and composing, then what *are* they for?

I was thinking one other case where the composed attribute would be used is when a consumer (an endpoint in this case) is choosing which media captures to receive from a provider.  Suppose the provider advertises capture scene entries that are the same except one of them has video captures that use composed=true, and the other uses composed=false.  I think the consumer can reasonably assume the one with composed=true includes more information somehow, possibly with "Hollywood squares" or picture-in-picture video layout.  The consumer can use this limited information to make a choice.  This choice has nothing to do with whether or not the receiving endpoint is going to re-compose the images.

Mark

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Paul Kyzivat
> Sent: Tuesday, March 27, 2012 11:35 PM
> To: [email-masked]
> Subject: Re: [clue] Use of the media capture "composed" attribute
> 
> The discussion of the composed and mixed attributes today exposed a lot of
> vagueness about what they mean. I think this needs to be firmed up
> considerably.
> 
> The fact that an audio capture has mixed content doesn't preclude mixing it
> again. But there are properties of it that might make such mixing
> unsatisfactory. For instance, if AC3 is a mix of (AC1, AC2), then it is ok to mix
> AC3 and independent capture AC4. But it would be unwise to mix it with AC2
> again. And it would also be unwise to mix it with AC5 that is itself a mix of
> (AC1, AC4).
> 
> This seems potentially more complex. So far the only use I have heard for
> this attribute is to drive a decision about whether it makes sense to compose
> this capture with others. But I expect that isn't always true either. You could
> have one uncomposed capture that shows two people
> (heads) side by side. You could have another *composed* capture that also
> shows two people in independently side by side each inside a frame, that
> were assembled from other captures. (Would anyone argue that this isn't
> composed?) But I can't see why it would be any more or less desirable to
> compose one of these captures than another.
>

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 28 Mar 2012 23:39:34 +0200

I think this further verifies that we need a better explanation of the 
intended purpose(s) of these (mixed/composed) attributes so that we can 
assess whether we have an actionable definition sufficient to the need.

	Thanks,
	Paul

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Fri, 30 Mar 2012 06:52:08 -0700

Reiterating a comment I made a the mic, I would like to see the addition
of semantics that describe the composed/mixed media capture in terms of
other media captures, when applicable. I think this would be very useful
information for a consumer.

Cheers,
Charles

> Sent: Wednesday, March 28, 2012 11:40 PM
> To: Duckworth, Mark
> Cc: [email-masked]
> > I was thinking one other case where the composed attribute would be
> used is when a consumer (an endpoint in this case) is choosing which
> media captures to receive from a provider.  Suppose the provider
> advertises capture scene entries that are the same except one of them
> has video captures that use composed=true, and the other uses
> composed=false.  I think the consumer can reasonably assume the one
> with composed=true includes more information somehow, possibly with
> "Hollywood squares" or picture-in-picture video layout.  The consumer
> can use this limited information to make a choice.  This choice has
> nothing to do with whether or not the receiving endpoint is going to
> re-compose the images.
> >
> > Mark
> >
> >> From: [email-masked] [mailto:[email-masked]] On
Behalf
> Of
> >> unsatisfactory. For instance, if AC3 is a mix of (AC1, AC2), then
it
> is ok to mix
> >> again. And it would also be unwise to mix it with AC5 that is
itself
> a mix of
> also
> >> composed, and at what scale (and maybe position) before you can
make
> a