Subject: Re: [clue] Clue data model updates

From: Roberta Presta <[email-masked]>
Date: Tue, 16 Oct 2012 17:09:51 +0200

Hi Paul,

please find below my comments in-line.

Il 15/10/2012 18:27, Paul Kyzivat ha scritto:
>
> [snip]
>
>> - case <composed>:
>> when this element is shown in a media capture, it means that the capture
>> is made by mixing more captures together.
>> This is the case of the telepresence room audio stream composed by the
>> participants' audio taken from different camera microphones.
>> Moreover, this can be the case of a video stream realized by overlapping
>> picture-in-picture (PiP) the loudest panel stream with the other
>> participants' streams.
>> Such captures don't have a single point of capture.
>> As stated in the framework document regarding the PiP video example, the
>> biggest area which is represented by such a stream should be indicated.
>> The datamodel schema preserves that possibility, since there is a
>> <captureArea> element in the video capture type which can be used to
>> convey that information.
>> On the other hand, for mixed audio streams, no spatial information is
>> provided.
>
> Now that you have detailed the video case, ISTM that there ought to be 
> *some* spatial information about audio. (Disclaimer: I know *nothing* 
> about audio processing.) I realize that audio can't be bounded the way 
> video is. But surely there is a big difference between an 
> omni-directional mic at the capture point and a mic that is focused 
> along the capture axis. Shouldn't that distinction be captured? (I 
> realize this is primarily a question about the framework, not the data 
> model.)
>

Actually, with the current version of the data model, there is the 
possibility to discriminate between (non-composed) audio streams 
captured by directional microphones and omni-directional microphones, 
but we can not provide spatial information for composed audio captures.
When a simple audio capture is captured by a directional mic, its 
representation in the datamodel contains the reference to a 
<capturePoint> element, which is the position of the mic, with a 
<captureAxisPoint> child element that gives the mic direction.
On the other hand, if the capture is captured by an omni-directional 
mic, its representation in the datamodel contains the reference to a 
<capturePoint>, which is the position of the mic, without a 
<captureAxisPoint> child.
Just to provide you with an example, let's call the omni-directional 
capture "AC0" and the directional one "AC1".
AC0 and AC1 are captured by different mics.
The corresponding representation according to the current datamodel 
should be the following:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<clueInfo xmlns="urn:ietf:params:xml:ns:clue-info">
     <mediaCaptures>
         <mediaCapture 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:type="audioCaptureType" captureID="AC0">
             <description>room audio captured by an omni-directional 
mic</description>
             <encGroupIDREF>EG0</encGroupIDREF>
             <capturedMedia>audio</capturedMedia>
<capturePointIDREF>CP0</capturePointIDREF>
             <content>main</content>
<audioChannelFormat>mono</audioChannelFormat>
         </mediaCapture>
         <mediaCapture 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:type="audioCaptureType" captureID="AC1">
             <description>room audio captured by a directional 
mic</description>
             <encGroupIDREF>EG0</encGroupIDREF>
             <capturedMedia>audio</capturedMedia>
<capturePointIDREF>CP1</capturePointIDREF>
             <content>main</content>
<audioChannelFormat>mono</audioChannelFormat>
         </mediaCapture>
     </mediaCaptures>

     [...]

     <capturePoints scale="millimeters">
         <capturePoint pointID="CP0">
             <x>1500</x>
             <y>1500</y>
             <z>0</z>
         </capturePoint>
         <capturePoint pointID="CP1">
             <x>1500</x>
             <y>1500</y>
             <z>1500</z>
             <captureAxisPoint>
                 <x>0</x>
                 <y>0</y>
                 <z>0</z>
             </captureAxisPoint>
         </capturePoint>
     </capturePoints>
</clueInfo>

The media capture type, which we updated in the last mail, is an 
*abstract* type conceived to be a general description suitable for 
different capture types. Video capture type and audio capture type 
extends such abstract type. They are already specified in the -01 
version of the datamodel document. Video capture type extends the media 
capture type by adding optional spatial information about the captured 
area. Audio capture type extends the media capture type by adding 
information about the format of the audio channel and no further spatial 
information, since the possibility of referring to a capture point, as 
envisioned in the abstract media capture type, seems to be sufficient.

> [snip]
>
>> * a new <clueInfo> child: <capturePoints>
>>
>> In order to list the capture points in a telepresence room, a new child
>> element of the clue info type has been added: <capturePoints>.
>> The capture points therein listed can be referred in the media captures
>> through the <captureIDREF> element.
>> The current aspect of the clue info type is then the following:
>>
>>        <!-- CLUE INFO TYPE -->
>>        <xs:complexType name="clueInfoType">
>>         <xs:sequence>
>>          <xs:element name="mediaCaptures" type="mediaCapturesType"/>
>>          <xs:element name="captureScenes" type="captureScenesType"/>
>>          <xs:element name="encodings" type="encodingsType"/>
>>          <xs:element name="encodingGroups" type="encodingGroupsType"/>
>>          <xs:element name="simultaneousSets" 
>> type="simultaneousSetsType"/>
>>          <xs:element name="capturePoints" type="capturePointsType"/>
>>          <xs:any namespace="##other" processContents="lax" minOccurs="0"
>> maxOccurs="unbounded"/>
>>         </xs:sequence>
>>         <xs:attribute name="clueInfoID" type="xs:ID" use="required"/>
>>         <xs:anyAttribute namespace="##other" processContents="lax"/>
>>        </xs:complexType>
>>
>> The definition of the capture points type is the following:
>>
>>        <!-- CAPTURE POINTS TYPE -->
>>        <xs:complexType name="capturePointsType">
>>         <xs:sequence>
>>          <xs:element name="capturePoint" type="capturePointType"
>> maxOccurs="unbounded"/>
>>          <xs:element name="descritption" type="xs:string" 
>> minOccurs="0"/>
>>         </xs:sequence>
>>         <xs:attribute name="scale" type="scaleType" use="required"/>
>>        </xs:complexType>
>
> I think this mishandles some of the semantic constraints in the 
> framework.
>
> An advertisement contains multiple scenes. Each scene has a single 
> scaletype and coordinate space. While a capture can be present in 
> multiple capture sets, it can only be in the capture sets of a single 
> scene. That constraint isn't reflected in the model. If it were, then 
> there would be a separate set of capturepoints for each scene, and no 
> need to specify the scaletype within capturePointsType. As written 
> above all capture points, even from multiple scenes, must have the 
> same scale type, and so all scenes are also forced to have the same 
> scale type.
I see your point.

The current format of the data model is suitable for telepresence room 
that provides spatial information about capture scene areas and capture 
points by using a single coordinate space, having it a physical scale 
("millimeters") or a simply relative scale ("unknown").

If we want to use different coordinate spaces inside each capture scene, 
the proposed datamodel doesn't work anymore. In such a case, we should 
define in each capture scene the position of the capture points 
according to the adopted coordinate space and scale. We should then 
link, in each capture scene,  the media captures with such spatial 
information.

I am wondering if we need using different coordinate spaces for 
describing capture scenes belonging to the same telepresence room (I 
refer to an end-point site and not to a middle box forwarding strems 
from different sites). Also when using an unknown scale, if the 
coordinate space is not the same among capture scenes, we are not able 
to understand the spatial relationship among their captured area. Are we 
interested in such relationship? Are capture scenes from a telepresence 
room conceived to be spatially independent from each other?

Cheers,

Roberta

---

From: John Leslie <[email-masked]>
Date: Tue, 16 Oct 2012 11:52:06 -0400

Roberta Presta <[email-masked]> wrote:
>>...
>> *some* spatial information about audio...
> 

   True.

> <captureAxisPoint> child element that gives the mic direction.

   Actually, all mikes are directional (though some have a figure-8
pattern and should be called bi-directional, and others are multiple
microphones capable of being processed to _any_ directionality).

> <captureAxisPoint> child.

   For what is generally called "omni-directional", this is technically
wrong. "Omni-directional" mikes have a directional pattern, but it is
less pronounced than "cardioid" or "shotgun" mikes.

   I would recommend that we keep <captureAxisPoint> for all mikes,
understanding that there are cases where the sensitivity pattern is
too close to unity to be worth using. I also recommend that for all
mikes there be a field for "pattern" -- most likely an enumeration
of cases starting with "omni-directional" and "cardioid".

   In many applications, even these differences wouldn't be worth
spending any processing time; but I believe the data structure should
allow for specifying them.

--
John Leslie <[email-masked]>

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 16 Oct 2012 12:50:03 -0400

John,

Thanks for providing input from somebody who knows what they are talking 
about!

	Thanks,
	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 16 Oct 2012 19:42:00 -0400

Hi Roberta,

Thanks for helping. I've included responses inline.

[snip]

I'll defer to John Leslie and his comments on this part.
He knows *way* more about it than I do.

> link, in each capture scene,  the media captures with such spatial

That is what I think needs to be done, more or less.

> room conceived to be spatially independent from each other?

The only reason to have scenes is to support coexistence of different 
coordinate spaces. You haven't been involved for that discussion so I'll 
summarize the main points and use cases.

One reason for different coordinate spaces came about from the inclusion 
of presentations that originate on a computer. For these there is no 
camera and no area of focus within a room. Each room may have a 
different place to display presentations, and where the sender might 
want to display it has no bearing on where the receiver will display it. 
The presentation may not even be displayed in a public place in the room 
- it may be viewed on individual screens, something like people using 
webex inside a telepresence session.

Another reason has to do with MCUs. I think the expected typical 
behavior of an MCU is that it will advertise a scene with a "synthetic" 
coordinate space, and map the captures it receives onto captures within 
that synthetic space. (I've been calling it a "virtual room" or "virtual 
scene".) But another possible behavior for the MCU is to directly 
advertise some or all of the scenes it receives from other rooms, 
instead of, or in addition to, a virtual scene.

	Thanks,
	Paul

---

From: Roberta Presta <[email-masked]>
Date: Wed, 17 Oct 2012 13:33:35 +0200

Paul and John,
thank you for your clarifications.
I'll try to update the data model in light of your comments and to 
propose the results on the mailing list as soon as possible.
Cheers,

Roberta

Il 17/10/2012 01:42, Paul Kyzivat ha scritto:
> coordinate spaces. You haven't been involved for that discussion so 
> I'll summarize the main points and use cases.
>
> One reason for different coordinate spaces came about from the 
> inclusion of presentations that originate on a computer. For these 
> there is no camera and no area of focus within a room. Each room may 
> have a different place to display presentations, and where the sender 
> might want to display it has no bearing on where the receiver will 
> display it. The presentation may not even be displayed in a public 
> place in the room - it may be viewed on individual screens, something 
> like people using webex inside a telepresence session.
>
> behavior of an MCU is that it will advertise a scene with a 
> "synthetic" coordinate space, and map the captures it receives onto 
> captures within that synthetic space. (I've been calling it a "virtual 
> room" or "virtual scene".) But another possible behavior for the MCU 
> is to directly advertise some or all of the scenes it receives from 
> other rooms, instead of, or in addition to, a virtual scene.
>
>     Thanks,
>     Paul
>
>
>

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 17 Oct 2012 11:54:35 -0400

> propose the results on the mailing list as soon as possible.

Thanks Roberta! I really appreciate the help you are giving us.

	Paul

---

From: Roberta Presta <[email-masked]>
Date: Wed, 24 Oct 2012 16:26:59 +0200

Hi all,

in order to convey the information about the mic type, I propose the 
introduction of the <micPattern> element as a further feature describing 
an audio capture.
The updated schema would be the following:

      <!-- AUDIO CAPTURE TYPE -->
      <xs:complexType name="audioCaptureType">
         <xs:complexContent>
          <xs:extension base="tns:mediaCaptureType">
           <xs:sequence>
            <xs:element name="audioChannelFormat" 
type="tns:audioChannelFormatType" minOccurs="0"/>
            <xs:element name="micPattern" type="tns:micPatternType" 
minOccurs="0"/>
           </xs:sequence>
          </xs:extension>
         </xs:complexContent>
      </xs:complexType>

      <!-- MIC PATTERN TYPE -->
      <xs:simpleType name="micPatternType">
       <xs:restriction>
        <xs:enumeration value="shotgun"/>
<xs:enumeration value="uni"/>
        <xs:enumeration value="omni"/>
        <xs:enumeration value="figure8"/>
        <xs:enumeration value="cardioid"/>
        <xs:enumeration value="hyper-cardioid"/>
       </xs:restriction>
      </xs:simpleType>

Do you think this could accomplish John's recommendations?
Is there any main mic pattern  missing?

Cheers,

Roberta

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 24 Oct 2012 12:26:41 -0400

inline

>         <xs:enumeration value="hyper-cardioid"/>

Speaking from total ignorance:

Do any/all of these pattern types require numbers to parameterize them, 
in addition to the axis of capture?

These seem to describe classes of geometric shapes. If we had "square" I 
would want to know "how big". I imagine the same would be true for 
"figure8".

John?

	Thanks,
	Paul

>> Roberta Presta<[email-masked]>  wrote:
>> John Leslie<[email-masked]>
>>
>
>
>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: John Leslie <[email-masked]>
Date: Wed, 24 Oct 2012 12:41:14 -0400

> Do you think this could accomplish John's recommendations?

   In essence, this is right. However, it would be good to reference
a definition of the patterns. I suggest from Wiki:Microphone

http://en.wikipedia.org/wiki/File:Polar_pattern_directional.png
http://en.wikipedia.org/wiki/File:Polar_pattern_omnidirectional.png
http://en.wikipedia.org/wiki/File:Polar_pattern_figure_eight.png
http://en.wikipedia.org/wiki/File:Polar_pattern_cardioid.png
http://en.wikipedia.org/wiki/File:Polar_pattern_hypercardioid.png

> Is there any main mic pattern  missing?

   That Wiki page also has subcardiod and supercardiod, but I don't
consider them terribly important. What we should have is a mechanism
to add more patterns during the life of the standard.

   (I don't recommend simply linking to the Wiki:Microphone page:
it's a bit too ephemeral, while these "File:" entries aren't likely
to see major changes.)

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 24 Oct 2012 13:10:59 -0400

> it's a bit too ephemeral, while these "File:" entries aren't likely

John: Is there any stable document (preferably from some SDO) that can 
be referenced for these?

And, as I asked in an earlier message, should these patterns be 
parameterized in some way? E.g. I *guess* that some shotgun mics are 
more narrowly focused than others, and so would expect some metric that 
indicates how narrow that focus is.

	Thanks,
	Paul

---

From: John Leslie <[email-masked]>
Date: Wed, 24 Oct 2012 13:37:23 -0400

> John: Is there any stable document (preferably from some SDO) that can 

   I'm not aware of any: I have seen patterns like these from vendors,
not standards bodies.

   Furthermore, these are idealizations: no actual microphone will
exactly match any of these, and the patterns are always different at
different frequencies.

> indicates how narrow that focus is.

   I have never run into a parameterized version, though I have certainly
seen detail differences on _actual_ polar patters measured by vendors.

   If we ever get there, we could perhaps have vendors register actual
patterns for specific models.

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 24 Oct 2012 17:52:04 -0400

Maybe I missed it, but I presume that somewhere you're going to capture the
electro-acoustic sensitivity of the microphones as well? In the ITU-T
Telepresence work, this has been rolled up into the overall sending loudness
rating of the system.

...Paul

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>John Leslie
>Sent: Wednesday, October 24, 2012 1:37 PM
>To: Paul Kyzivat
>Cc: [email-masked]
>Subject: Re: [clue] Clue data model updates
>
>> more narrowly focused than others, and so would expect some metric
>> that indicates how narrow that focus is.
>
>   I have never run into a parameterized version, though I have
>certainly seen detail differences on _actual_ polar patters measured by
>vendors.
>