Subject: [clue] Open Ticket 5 - composed captures

From: Stephan Wenger <[email-masked]>
Date: Tue, 24 Jul 2012 12:01:22 +0000

Hi all,

I had some private conversations with Steve and Mark about composed pictures.  They offered me time from the framework timeslot in the hopes that we can close the subject.  Thank you again.  However, I think we can close it even on the list:

To refresh everyone's memory, my proposal has been to include (optionally) in the description of a composed video capture those attributes that are needed to decompose the capture in order to forward (after re-encoding) the tiles of an MCU-generated, tiled image as individual streams.  I continue to believe that this use case is valid.  However, after thinking a bit more, I now believe that CLUE may not be the right protocol mechanism to convey the information necessary for decomposition, for example the coordinates of each tile in the composed picture.  The reason is that CLUE is not a real time protocol synchronized with the video bitstream.  In many (most?) practical MCU based sessions, there is some form of real-time image swapping, voice activated or otherwise.

Decomposition of such sessions with protocol means not synchronized with the video (at a single frame accuracy, i.e. 1/60th of a second) will lead to annoying artifacts at the receivers end.  I don't believe that the CLUE protocol will ever have such a real-time property.  Therefore, the appropriate place for de-composition information is not in a CLUE protocol transaction, but in something synchronized with video, most likely in-band as a Supplementary Enhancement Information (SEI) message.

Unless someone else points out an error in this train of thought, I'm OK with a single flag per capture indicating a composed capture for CLUE's first version.  I will also think about an appropriate SEI message format that could be suggested to the video standards bodies such as VCEG and MPEG.

Thoughts?

See you soon in YVR,
Stephan

---

From: "Roni Even" <[email-masked]>
Date: Tue, 24 Jul 2012 15:30:16 +0200

Hi Stephan,

I agree with your view about decomposition a mixed captures to individual
tiles. I am not sure about SEI message if we want this for any codec, maybe
RTP header extension is another way.

As for the use case you described, I think that people are looking at using
SVC to send a mixed image that can be decomposed easily. 

This will still leave open the issue of which source  is in the current mix
and in in which tile, which is what I think what others had in mind when
discussing composed capture description.

Roni

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Stephan Wenger
Sent: 24 July, 2012 2:01 PM
To: [email-masked]
Subject: [clue] Open Ticket 5 - composed captures

Hi all,

I had some private conversations with Steve and Mark about composed
pictures.  They offered me time from the framework timeslot in the hopes
that we can close the subject.  Thank you again.  However, I think we can
close it even on the list:

To refresh everyone's memory, my proposal has been to include (optionally)
in the description of a composed video capture those attributes that are
needed to decompose the capture in order to forward (after re-encoding) the
tiles of an MCU-generated, tiled image as individual streams.  I continue to
believe that this use case is valid.  However, after thinking a bit more, I
now believe that CLUE may not be the right protocol mechanism to convey the
information necessary for decomposition, for example the coordinates of each
tile in the composed picture.  The reason is that CLUE is not a real time
protocol synchronized with the video bitstream.  In many (most?) practical
MCU based sessions, there is some form of real-time image swapping, voice
activated or otherwise. 

Decomposition of such sessions with protocol means not synchronized with the
video (at a single frame accuracy, i.e. 1/60th of a second) will lead to
annoying artifacts at the receivers end.  I don't believe that the CLUE
protocol will ever have such a real-time property.  Therefore, the
appropriate place for de-composition information is not in a CLUE protocol
transaction, but in something synchronized with video, most likely in-band
as a Supplementary Enhancement Information (SEI) message.

Unless someone else points out an error in this train of thought, I'm OK
with a single flag per capture indicating a composed capture for CLUE's
first version.  I will also think about an appropriate SEI message format
that could be suggested to the video standards bodies such as VCEG and MPEG.

Thoughts?

See you soon in YVR,

Stephan

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Tue, 24 Jul 2012 12:55:26 +0000

For composed streams I remember we have discussed these use cases

*         UC-0; As an endpoint I want to choose captures from an MCU with or without composition enabled. If composition is enabled the MCU is free to implement any custom layout the MCU think is good, with composition disabled the receiver expect each capture to be a single participant and single segment.

*         UC-1; As an endpoint I want to know which composed layout the MCU supports, e.g. a named list of layout with a descriptive text similar to the video-layout element in XCON. [1]

*         UC-2: As an endpoint I want to request a composed capture with a preferred layout based on an available list, without a preference the MCU  is free to choose a layout that works nicely with the connected participants and captures available.

*         UC-3; As an endpoint I want to receive information about the content of a composed capture and receive updates when the composition changes.

UC-0 only requires CLUE to have a Boolean to enable/disable composed layouts, which I think is sufficient for CLUE version 1.

The remaining use cases are also valid for related IETF standards like XCON. I do think that the CLUE protocol must be extensible (as stated in the charter) and allow for integration with another protocol for layout control or as a vendor extension to CLUE.

Cheers

-Espen

[1] XCON data model, http://tools.ietf.org/html/rfc6501

From: [email-masked] [mailto:[email-masked]] On Behalf Of Roni Even
Sent: 24. juli 2012 15:30
To: 'Stephan Wenger'; [email-masked]
Subject: Re: [clue] Open Ticket 5 - composed captures

Hi Stephan,
I agree with your view about decomposition a mixed captures to individual tiles. I am not sure about SEI message if we want this for any codec, maybe RTP header extension is another way.
As for the use case you described, I think that people are looking at using SVC to send a mixed image that can be decomposed easily.

This will still leave open the issue of which source  is in the current mix and in in which tile, which is what I think what others had in mind when discussing composed capture description.

Roni

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Stephan Wenger
Sent: 24 July, 2012 2:01 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [clue] Open Ticket 5 - composed captures

Hi all,

Thoughts?

See you soon in YVR,
Stephan

---

From: "Duckworth, Mark" <[email-masked]>
Date: Tue, 24 Jul 2012 10:52:16 -0700

Espen,

You summarized the cases nicely.  I think your UC-3 is what Roni was talking about when he said there is still the open issue of which source is in the mix, etc.

I agree the current framework is sufficient for CLUE version 1.  I like the idea of using another protocol for layout control, as I think it is separable from CLUE and more broadly applicable for non-CLUE systems.

Mark

From: [email-masked] [mailto:[email-masked]] On Behalf Of Espen Berger (espeberg)
Sent: Tuesday, July 24, 2012 8:55 AM
To: Roni Even; 'Stephan Wenger'; [email-masked]
Subject: Re: [clue] Open Ticket 5 - composed captures

Cheers

-Espen

Hi Stephan,

Roni

Hi all,

Thoughts?

See you soon in YVR,
Stephan