Subject: [clue] Add a way to link audio captures to video captures

From: John Leslie <[email-masked]>
Date: Sat, 10 May 2014 09:09:59 -0400

Whenever there are separate cameras which can be separately displayed,
the receiver faces the question of which audio source to render so as to
seem to come from that video area.

   Sometimes, of course, there will only be one audio source available;
but when there are several, it would help for the sender to link the
"most appropriate" audio to each separate video. (How "most appropriate"
is determined is way out-of-scope for this standard.)

Currently we have spatialInformation within mediaCaptureType, and
spatialInformation includes captureArea.

   This has already led to much confusion. I suggest renaming
"captureArea" to "videoCaptureFrame" and adding "audioCapturePolygon"
to spatialInformation.

   The current video captureArea defines a trapezoid in a plane
more-or-less perpendicular to the floor. Actually the video includes
infinitely many rectangles at differing distances from the point of
capture, with no way to distinguish them except that some are out of
focus. Thus video can be thought of as insensitive to distance from
the camera.

   Audio mediaCaptures, however, are mostly insensitive to height above
the floor, but necessarily concentrate on capturing the sound within
some distance of the microphone. The audio level decreases with the
square of the distance.

   While microphones pick up some sound from anywhere in the room, the
_intent_ in setting up microphones is to pick up sound in a defined
area. This deserves a way to specify it -- at least in the case where
that microphone's sound output is separately available.

   In many cases, the sound engineer will wish to only offer one mixer
output (hopefully stereo or surround) for the entire room. But when
the individual microphones are separately available, it would be very
helpful to know the area each is intended to cover. This will not be
a rectangle, so I advise defining it as a polygon.

   Given a videoCaptureFrame and audioCapturePolygons to choose from,
it's easy enough for a receiver to test overlap. (OK, maybe it is a
bit confusing to test overlap of a pyramid with a two-dimensional
plane in three-space, but it's "easy math".) This automatically
gives you a list of audio captures appropriate for a video capture
being displayed.

   OTOH, it's even easier for the sender to do this; and there could
be cases where the audio is most appropriate despite lack of overlap.
And, if the sender does it, an order of preference could be noted.

   But it's still not clear to me what XML is appropriate to record
this. I think I'd prefer a list of audio capture IDREFs within the
videoCaptureType; but that's not a strong preference if some other
way is available and appropriate.

--
John Leslie <[email-masked]>