Subject: [clue] Attribute to indicate whether echo cancellation is done

From: John Leslie <[email-masked]>
Date: Mon, 12 May 2014 07:12:18 -0400

Add an attribute (per audio source) to indicate whether echo cancellation
is done
====

   In any teleconferencing room with both speakers and microphones,
"echo" must be considered. Sound received from remote locations
plays through your loudspeakers, reverberates through the room
(with some delay), and is picked up by your microphones. If ignored,
this signal feeds through network delay back to its original source;
and the process repeats. This problem too often destroys communication
when someone unmutes the microphones in order to speak.

   Telepresence requires echo cancellation (where the reverberation
is modeled and subtracted from the audio signal you send) to be
performed by the sender. But there will be cases where some sender
doesn't do it; and there will probably be cases where echo cancellation
is only done for the mixed whole-room feed.

   Thus I recommend an attribute for audioEchoRemoval to the
audioCaptureType: an enumeration value, selected from "echo
cancellation done", "echo suppression done", or "none".

   Obviously, if receivers choose to use an audioCapture which does
no echo cancellation, they must find a way to keep echo under control.
Feeding it only to headphones/earbuds will always work; ensuring that
every participant is close-miked will often work; and to some degree
including it in your own echo-cancellation processing might be good
enough...

--
John Leslie <[email-masked]>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 12 May 2014 11:26:21 -0400

Comment at end...

> including it in your own echo-cancellation processing might be good

IMO the question is whether we want to include this possibility.

Putting it in obligates every consumer to have logic to deal with it.

Taking it out obligates every provider to ensure echo has been cancelled.

Since most TP endpoints will have echo suppression, they won't be 
further burdened if we leave this out, but will have extra work to do if 
we add it.

ISTM the cases that need most consideration are:

- legacy mode clients. By definition we can't expect anything special 
about their behavior. An MCU that allows them into a clue conference 
will need to do *something* about this.

- clue-aware devices that don't have the the equipment that a typcial TP 
room or endpoint has. E.g. a cell phone with the smarts to do the clue 
protocol. These might be common. Can we assume they will be capable of 
handling the echo suppression?

	Thanks,
	Paul

---

From: John Leslie <[email-masked]>
Date: Mon, 12 May 2014 12:02:44 -0400

Paul Kyzivat <[email-masked]> wrote:
>>...

   Which possibility, exactly?

> Putting it in obligates every consumer to have logic to deal with it.

   Sufficient logic to realize echo cancellation in what is sent has or
has not been performed, yes.

> Taking it out obligates every provider to ensure echo has been cancelled.

   Every sender, yes but...

   Some senders won't. :^(

   I tried to mention some of the cases in the text I elided. I suppose
I could mention them again, but won't in this message.

> further burdened if we leave this out, but will have extra work to do if 

   I can assure you that failure to cancel echo can be _very_
disruptive to communication.

   The echo cancellation one site does cannot protect that site from
echo received. Echo _suppression_ can be applied to mute a source
when echo is detected, but the detection can be a bit buggy, and a
source which drops out in the middle of a word is also disruptive.

> will need to do *something* about this.

   Which is what I said, isn't it?

> handling the echo suppression?

   IMHO: no.

   BTW, we should keep "echo suppression" distinct from "echo cancellation".
Echo cancellation removes the reverberation of sound received from the
sound you send; echo suppression merely mute the entire source when echo
is "detected".

   But we can't reasonably expect smart-phones to do either.

---

From: Paul Coverdale <[email-masked]>
Date: Mon, 12 May 2014 14:08:33 -0400

>-----Original Message-----
>From: clue [mailto:[email-masked]] On Behalf Of John Leslie
>Sent: Monday, May 12, 2014 12:03 PM
>To: Paul Kyzivat
>Cc: [email-masked]
>Subject: Re: [clue] Attribute to indicate whether echo cancellation is
>done
>
>>> Obviously, if receivers choose to use an audioCapture which does  no
>>>echo cancellation, they must find a way to keep echo under control.
>> further burdened if we leave this out, but will have extra work to do
>> if we add it.
>
>   I can assure you that failure to cancel echo can be _very_ disruptive
>to communication.
>
>echo received. Echo _suppression_ can be applied to mute a source when
>echo is detected, but the detection can be a bit buggy, and a source
>which drops out in the middle of a word is also disruptive.
>
>> - clue-aware devices that don't have the the equipment that a typcial
>> TP room or endpoint has. E.g. a cell phone with the smarts to do the
>> clue protocol. These might be common. Can we assume they will be
>> capable of handling the echo suppression?
>
>   IMHO: no.
>

[Paul Coverdale] This isn't entirely correct. Modern echo cancellers include
both echo cancellation and echo suppression. The echo canceller tries to
adaptively cancel out the linear (and sometimes non-linear) echo-path
response but generally this doesn't provide sufficient echo-path loss alone.
The remaining low-level echo is taken out by the non-linear processor (NLP)
which is effectively an echo suppressor. I doubt that anybody is using echo
suppression alone for conferencing, it would lead to very choppy audio
(might as well just use push-to-talk)

>   But we can't reasonably expect smart-phones to do either.

[Paul Coverdale] Some smart-phones, maybe. But remember that the smart-phone
only has to eliminate the echo that it generates itself (spec'd by terminal
coupling loss). If it's good enough when it's talking to another cell-phone
(or even land-line), it should be good enough when it's talking to a
telepresence end-point.

>_______________________________________________
>clue mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/clue

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 12 May 2014 16:26:21 -0400

>     Which possibility, exactly?

The possibility of captures where echo cancellation has not been done.

> has not been performed, yes.

Things are simpler for the consumer if it can assume that echo 
cancellation has been done for every capture it receives.

Once there is the possibility that it might not have been done then 
further algorithms must be available.

>> Taking it out obligates every provider to ensure echo has been cancelled.

Then that becomes like any other non-conformance.
The ones that cause trouble get identified, and those implementation 
must be fixed or get black listed.

> source which drops out in the middle of a word is also disruptive.

My assumption is that we would require that advertised captures MUST NOT 
contain echo. That will require echo cancellation or suppression if 
there is any possibility of echo in the first place. (An endpoint that 
is just playing a powerpoint with background music doesn't need to do 
anything.)

>     Which is what I said, isn't it?

Legacy mode clients are "special", because they don't send or consume 
advertisements. So there is no way for the receiver to be told whether 
the incoming audio might have echo or not.

	Thanks,
	Paul

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 12 May 2014 18:02:49 -0400

Echo Cancellation algorithms all require a mathematically linear
relationship between the electrical signal sent to the loudspeakers, and
the electrical signal of the echo picked by the microphone.

This is the case with a local system doing the cancellation, as long as the
speakers and the microphones are not overloaded so there is non-linear
distortion products.

However, in general, remote acoustic echo cancellation cannot be done.  One
reason is that the remote system simply does not have any knowledge of
exactly what signal was rendered at the source loudspeakers.  The second
reason is that most codecs that are used to compress the audio stream
introduce non-linearities which defeat the echo cancellation algorithms.

A third reason is that the sampling clock at the remote system is not
precisely matched to the source system.  Even a small mismatch in clock
rate results in a failure in the cancellation.   Commercial grade clock
sources are not that precise, and they are subject to thermal drift.  I did
a simple experiment some years ago - I started with a 60 seconds of audio.
 I then resampled to to add 1 sample (I believe this was at 48 Khz, so I
was stretching the signal from 288000 to 288001 samples).  I subtracted
those waveforms, and listened to the difference signal - and could clearly
hear the speech after about 15 seconds.  Even a very small clock mismatch
builds quickly (and since it is a form of non-linear distortion, it isn't
canceled). Commercial systems solve this problem by phase-locking their
local A/D and D/A clocks.

So in my experience, it is not possible for a remote system to remove the
echo introduced by a far end system.  All it can do is not render that
far-end's audio in the first place.  Therefore I see no point to signaling
a "hey I should have removed the echo, but I didn't" parameter.  The only
sensible strategy for a receiver is to not ask for that capture.

Steve

On Mon, May 12, 2014 at 4:26 PM, Paul Kyzivat <[email-masked]> wrote:

> The ones that cause trouble get identified, and those implementation must
> be fixed or get black listed.
>
>
> contain echo. That will require echo cancellation or suppression if there
> is any possibility of echo in the first place. (An endpoint that is just
> playing a powerpoint with background music doesn't need to do anything.)
>
>
> advertisements. So there is no way for the receiver to be told whether the
> incoming audio might have echo or not.
>
>         Thanks,
>         Paul
>
>

---

From: John Leslie <[email-masked]>
Date: Mon, 12 May 2014 20:31:50 -0400

>>>> including it in your own echo-cancellation processing might be good

   I wonder if everyone saw where I said "chooses to use"...

   If you are signaled that a source doesn't handle echo cancellation
or suppression, you can "just say no", and the reason can be obvious
to the sender. I didn't mean to imply any obligation to _use_ such a
source.

> cancellation has been done for every capture it receives.

   Such an unwarranted assumption can _really_ hurt!

> further algorithms must be available.

   Actually, the "possibility" is no greater with a signal such as I
propose -- only the probability of learning about it without disrupting
the conference changes.

> must be fixed or get black listed.

   Do you have the 800-number for the protocol police?

   (I'm sorry to come on so strongly, but I experience echo disruption
_regularly_ on IESG telechats. It _really_ is worth automating the
diagnosis of these problems!)

> is just playing a powerpoint with background music doesn't need to do 

   I have no problem adding a state for "pre-recorded audio" or whatever
you suggest.

   In addition to pre-recorded, there are "earbud-only" applications
which remove the issue. We could add a state for that, too.

> the incoming audio might have echo or not.

   I'm not sure the term "legacy-mode" implies "not clue-speaking" to
all our readers.

   I agree that how to deal with clue-less clients is outside the scope
of what we're writing here.

   But I'd prefer that we not prohibit applications that don't do
echo cancellation to our specs from speaking CLUE at all.

   (_Really_prefer_, that is...)

---

From: John Leslie <[email-masked]>
Date: Mon, 12 May 2014 20:44:47 -0400

> the electrical signal of the echo picked by the microphone.

   Yes.

> speakers and the microphones are not overloaded so there is non-linear

   Yes.

> introduce non-linearities which defeat the echo cancellation algorithms.

   Yes.

> sources are not that precise, and they are subject to thermal drift...

   Yes.

> echo introduced by a far end system.

   This is where we part ways. I quite agree that the receiving system
cannot adequately do echo-cancellation that the sender failed to do.

   But the receiving system still has echo-suppression tricks available
to it.

   And most important, if the receiving system knows beforehand that
echo-cancellation isn't being done, it can signal the problem and
refuse to use that source.

> All it can do is not render that far-end's audio in the first place.

   Which, of course, it can't do before it knows the problem exists...

> Therefore I see no point to signaling a "hey I should have removed
> the echo, but I didn't" parameter.

   (I figured we'd have a better chance of getting an honest statement
if we didn't word it _quite_ that way... ;^)

> The only sensible strategy for a receiver is to not ask for that capture.

   I mostly agree.

   But others _will_ come up with ways to deal with smart-phones that
omit echo-cancellation. I see no reason to get in their way if they
try to deal with it before it becomes disruptive to the conference.

   (Did I mention that the protocol police really don't have all that
great a record of preventing non-compliance?)

---

From: Stephen Botzko <[email-masked]>
Date: Mon, 12 May 2014 21:40:06 -0400

in-line

>    But the receiving system still has echo-suppression tricks available
Ok.  So here's a real example.
We have a mesh with Alice, Bob, and Carol.   Each has selected audio
captures from the other two.  Carol is sending audio that is not
echo-canceled.
So her audio capture includes echo from both Alice and Bob. Neither Alice
nor Bob can echo-cancel for Carol - we agree on that.

You say that there are echo-suppression tricks that Alice can use that will
work for her, without knowledge of anything Bob might (or might not) be
doing.  Please tell us what they are - especially if Alice doesn't actually
know what captures Carol has chosen to receive and render.

>    Which, of course, it can't do before it knows the problem exists...
Far more direct for Carol not to advertise her audio in the first place.
 Or not allow her system to be used without earphones.

> if we didn't word it _quite_ that way... ;^)

Though it sounds pejorative, I don't what else you can call it.

If I am using earphones, I don't need to echo-cancel, since there is no
possibility of echo in the first place.  If I have open loudspeakers,and  I
choose to echo cancel then I have taken care of any problem.  Both of those
captures can safely be used by all receivers.

The problem case is if I have open loudspeakers and deliberately didn't
echo-cancel.  I think its fair to call that "hey I should have removed it
but didn't".  It's not like this problem is unknown.  Some conferencing
products are better at echo cancellation others.  But everyone building
these devices/apps know they need to do it, and they all have something.

If for some reason they choose not to, then they should use push-to-talk,
and mute their received audio signal when they are in talk-mode.

> try to deal with it before it becomes disruptive to the conference.

Based on what people are already doing today, I don't think it is likely
that any successful devices/apps will simply omit it.

lt is far more likely that devices which are inserting echo are doing so
because their algorithms simply aren't working well at the moment.  Users
pushing up the volume so the speakers distort is one common scenario that
defeats echo cancellation.  People can certainly innovate in this area, but
they can't depend on signaling, since the devices inserting the echo
generally can't detect that they are doing so..

> great a record of preventing non-compliance?)
Not sure how that applies to this discussion.