Subject: [xrblock] Loss-Type concealment vs Buffer adjustment type concealment

From: Qin Wu <[email-masked]>
Date: Wed, 10 Oct 2012 15:41:52 +0800

Hi,
In draft-ietf-xrblock-rtcp-xr-concsec,concealment is splitted into two type of concealments.
One is Loss-Type concealment and the other is Buffer adjustment type concealment.
So the question is are these two type of concealments applied to video? If the answer is yes,
how to take video loss concealment into account? Since in the current draft, when we define
Loss-type concealment and buffer adjustment type concealment, only audio loss concealment 
is considered. See section 2.2 of draft-ietf-xrblock-rtcp-xr-concsec below for defintions of loss-type concealment
and buffer adjustment type concealment:
"    
     Loss-type concealment is reactive insertion or deletion of samples
      in the audio playout stream due to effective frame loss at the
      audio decoder.  "Effective frame loss" is the event in which a
      frame of coded audio is simply not present at the audio decoder
      when required.  In this case, substitute audio samples are
      generally formed, at the decoder or elsewhere, to reduce audible
      impairment.

      Buffer Adjustment-type concealment is proactive or controlled
      insertion or deletion of samples in the audio playout stream due
      to jitter buffer adaptation, re-sizing or re-centering decisions
      within the endpoint. Because this insertion is controlled, rather than occurring
      randomly in response to losses, it is typically less audible than
      loss-type concealment.  For example, jitter buffer adaptation
      events may be constrained to occur during periods of talker
      silence, in which case only silence duration is affected, or
      sophisticated time-stretching methods for insertion/deletion
      during favorable periods in active speech may be employed.  For
      these reasons, buffer adjustment-type concealment MAY be exempted
      from inclusion in calculations of Concealed Seconds and Severely
      Concealed Seconds.
"

Regards!
-Qin

---

From: Alan Clark <[email-masked]>
Date: Wed, 10 Oct 2012 05:02:49 -0400

Qin

The wording in draft-ietf-xrblock-xr-consec-.. is definitely related to
audio. Video does have loss concealment and there are a range of methods
used, for example:

(i) Frame freeze
In this case the impaired video frame is not displayed and the previously
displayed frame is hence ³frozen² for the duration of the loss even

(ii) Inter-frame extrapolation
If an area of the video frame is damaged by loss, the same area from the
previous frame(s) can be used to estimate what the missing pixels would have
been.  This can work well in a scene with no motion but can be very
noticeable if there is significant movement from one frame to another.
Simple decoders may simply re-use the pixels that were in the missing are,
more complex decoders may try to use several frames to do a more complex
extrapolation

(iii) Interpolation
A decoder may use the undamaged pixels in the image to estimate what the
missing block of image should have

(iv) Noise insertion
A decoder may insert random pixel values - which would generally be less
noticeable than a blank rectangle in the image

Alan

On 10/10/12 3:41 AM, "Qin Wu" <[email-masked]> wrote:

> _______________________________________________
> xrblock mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xrblock

---

From: Alan Clark <[email-masked]>
Date: Wed, 10 Oct 2012 05:20:26 -0400

Also - we should note that with video it is possible to use RTP based
retransmission and also FEC (e.g. COP3) - typically these would only be used
with IPTV as this is less delay sensitive than interactive services.

Alan

On 10/10/12 5:02 AM, "Alan Clark" <[email-masked]> wrote:

> Qin
> 
> The wording in draft-ietf-xrblock-xr-consec-.. is definitely related to audio.
> Video does have loss concealment and there are a range of methods used, for
> example:
> 
> (i) Frame freeze
> been.  This can work well in a scene with no motion but can be very noticeable
> if there is significant movement from one frame to another.  Simple decoders
> may simply re-use the pixels that were in the missing are, more complex
> decoders may try to use several frames to do a more complex extrapolation
> 
> (iii) Interpolation

---

From: Qin Wu <[email-masked]>
Date: Wed, 10 Oct 2012 18:18:54 +0800

Re: [xrblock] Loss-Type concealment vs Buffer adjustment type concealmentHi, Alan:
That's a good summary for video loss concealment method.
Please see my comments inline.
  ----- Original Message ----- 
  From: Alan Clark 
  To: Qin Wu 
  Cc: [email-masked] 
  Sent: Wednesday, October 10, 2012 5:02 PM
  Subject: Re: [xrblock] Loss-Type concealment vs Buffer adjustment type concealment

  Qin

  The wording in draft-ietf-xrblock-xr-consec-.. is definitely related to audio. Video does have loss concealment and there are a range of methods used, for example:

  (i) Frame freeze
  In this case the impaired video frame is not displayed and the previously displayed frame is hence "frozen" for the duration of the loss even

[Qin]: Agree, this method is also widely used technique in the IPTV deployment or cable TV deployment.

  (ii) Inter-frame extrapolation
  If an area of the video frame is damaged by loss, the same area from the previous frame(s) can be used to estimate what the missing pixels would have been.  This can work well in a scene with no motion but can be very noticeable if there is significant movement from one frame to another.  Simple decoders may simply re-use the pixels that were in the missing are, more complex decoders may try to use several frames to do a more complex extrapolation

[Qin]: I think one example of inter-frame extrapoltion is motion-compensated repetition. Motion-compensated repetition is widely used techniques for video loss concealment
and is used a repeat of the preceding frame to replace the part of the frame affected by the loss when loss occur.

(iii) Interpolation
A decoder may use the undamaged pixels in the image to estimate what the missing block of image should have

[Qin]: Two examples of interpolation are repair in the spatial domain and repair in the frequency domain
Repair in the spatial domain means relying on interpolation of a missing block, on the basis of the surrounding data. 
For example, the average pixel color for each of the surrounding blocks can be calculated, and the missing block can be set to the average of those colors. 
 The similar techiques used in repair in the spatial domain can be applied to repair in the frequency domain.For example, for codecs based on the DCT, 
such as MPEG, H.261, and H.263. The low-order DCT coefficients can be averaged across the surrounding blocks, to generate a fill-in for a missing block.

(iv) Noise insertion
A decoder may insert random pixel values - which would generally be less noticeable than a blank rectangle in the image

[Qin]: It looks to me Noise insertion also can be regarded as one kind of interpolation. At least, they have the similarity.
For example,when interpolation using repair in the spatial domain, the average pixel color for each of the surrounding blocks is calculated to replace the missing block.
In the Noise insertion, a random pixel color value is choosen for the missing block.

Alan

    Hi,

    Regards!
    -Qin 

----------------------------------------------------------------------------
    _______________________________________________
    xrblock mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/xrblock

------------------------------------------------------------------------------

---

From: Qin Wu <[email-masked]>
Date: Wed, 10 Oct 2012 18:21:05 +0800

Re: [xrblock] Loss-Type concealment vs Buffer adjustment type concealmentAgree.
  ----- Original Message ----- 
  From: Alan Clark 
  To: Qin Wu 
  Cc: [email-masked] 
  Sent: Wednesday, October 10, 2012 5:20 PM
  Subject: Re: [xrblock] Loss-Type concealment vs Buffer adjustment type concealment

  Also - we should note that with video it is possible to use RTP based retransmission and also FEC (e.g. COP3) - typically these would only be used with IPTV as this is less delay sensitive than interactive services. 

  Alan

    Qin

    (i) Frame freeze

    (iii) Interpolation

    (iv) Noise insertion

    Alan

      Hi,

      Regards!
      -Qin 

--------------------------------------------------------------------------
      _______________________________________________
      xrblock mailing list
      [email-masked]
      https://www.ietf.org/mailman/listinfo/xrblock

---

From: Qin Wu <[email-masked]>
Date: Fri, 19 Oct 2012 15:59:14 +0800

Re: [xrblock] Loss-Type concealment vs Buffer adjustment type concealmentHi,Alan:
one follow up question is:
Is Buffer adjustment type concealment applicable to video application?
It seems Buffer Adjustment-type concealment is usually applied to audio.

Regards!
-Qin
  ----- Original Message ----- 
  From: Alan Clark 
  To: Qin Wu 
  Cc: [email-masked] 
  Sent: Wednesday, October 10, 2012 5:20 PM
  Subject: Re: [xrblock] Loss-Type concealment vs Buffer adjustment type concealment

  Alan

    Qin

    (i) Frame freeze

    (iii) Interpolation

    (iv) Noise insertion

    Alan

      Hi,

      Regards!
      -Qin

---

From: Alan Clark <[email-masked]>
Date: Fri, 19 Oct 2012 08:21:17 -0400

Hi Qin

There is an equivalent buffer adjustment behavior in video systems.  If
there is some need to increase or decrease the playout buffer size this can
be accommodated by either temporarily adjusting the frame rate or freezing/
skipping frames.  The extreme case of this is in a video streaming system
which can pause playout for 10-30 seconds if a buffer outage occurs -
typically the client will increase the playout buffer to allow for more
variability in network or server induced delays.

Best Regards

Alan

On 10/19/12 3:59 AM, "Qin Wu" <[email-masked]> wrote:

> Hi,Alan:
>> From:  Alan Clark <mailto:[email-masked]>
>>  
>> To: Qin Wu <mailto:[email-masked]>
>>  
>> Cc: [email-masked]
>>  
>>> previous  frame(s) can be used to estimate what the missing pixels would
>>> have been.   This can work well in a scene with no motion but can be very

---

From: Qin Wu <[email-masked]>
Date: Tue, 23 Oct 2012 15:00:58 +0800

Re: [xrblock] Loss-Type concealment vs Buffer adjustment type concealmentMake sense to me. Thank for your clarification!

Regards!
-Qin
  ----- Original Message ----- 
  From: Alan Clark 
  To: Qin Wu 
  Cc: [email-masked] 
  Sent: Friday, October 19, 2012 8:21 PM
  Subject: Re: [xrblock] Loss-Type concealment vs Buffer adjustment type concealment

  Hi Qin

  There is an equivalent buffer adjustment behavior in video systems.  If there is some need to increase or decrease the playout buffer size this can be accommodated by either temporarily adjusting the frame rate or freezing/ skipping frames.  The extreme case of this is in a video streaming system which can pause playout for 10-30 seconds if a buffer outage occurs - typically the client will increase the playout buffer to allow for more variability in network or server induced delays.

  Best Regards

  Alan

    Hi,Alan:
     
    Regards!
    -Qin

      ----- Original Message ----- 

       
      Cc: [email-masked] 

      Alan

        Qin

        (i) Frame freeze

        (iii) Interpolation

        (iv) Noise insertion

        Alan

          Hi,

          Regards!
          -Qin 

----------------------------------------------------------------------
          _______________________________________________
          xrblock  mailing list
          [email-masked]
          https://www.ietf.org/mailman/listinfo/xrblock

------------------------------------------------------------------------
        _______________________________________________
        xrblock  mailing list
        [email-masked]
        https://www.ietf.org/mailman/listinfo/xrblock