Subject: [vmeet] What is takes to have "successful" telepresence

From: "Brian Rosen" <[email-masked]>
Date: Tue, 7 Apr 2009 16:34:26 -0400

I spent a good 5 years of my life working on what it takes to have
successful remote participation.  I'll start by saying that our goal was
getting geographically distributed small teams (8-15) people working
together, rather than 1200 people.  Most of what we learned is applicable to
larger groups.  Our project started with research.  We observed teams
interacting in person.  Then we observed them working on available tools,
like videoconference tools, then we built new tools and worked on them until
people behaved roughly the same on the tools as they did when they were in
person.  Yes, it's a product, and no, it hasn't taken the world by storm.
Lots of reasons for that. One interesting effect we notice is that use on
this tool goes up over time.  That is the opposite for most tools other than
basic audio bridges and screen sharing tools.

There are other tools that work, in somewhat difference circumstances.  The
Cisco telepresence tools work in their design space (a virtual conference
room with a small number of participants in each physical room, and a few
physical rooms) for example.

To my knowledge, no one has made tools that would work in an IETF meeting.

So, here is what I think you need:
1. One way audio delay <150 ms.  This is a cliff.  It's not true that, say,
250 ms is a lot better than 500.  You can't hold a discussion with more than
150.  It's the single largest reason most tools don't work.  We understand
why this is true; humans cannot perform adequate turn taking with more than
150 ms delay.  The single most obvious thing you notice when it's more than
150 is that you can't argue, but you notice it even if you aren't arguing.
Behavior when it's more than 150 is seriously weird.  You measure it from
mouth to ear.

2. Enough video quality to discern emotion.  As a practical matter, for say
chest and higher "head shots" it's around 400 lines.  Anyone who tells you
head shots don't work is wrong; as long as you can read emotion, it works
fine.  In fact, if you give people tradeoffs on lots of variables, they will
sacrifice human area coverage right down to a face before they give up on
other things.  You can't get really effective participation without video,
although it's crystal clear that poor video is worse than no video.

3. No detectable lip-sync issues.  This generally means the audio is within
a frame (30ish) ms of the video.  Another cliff.  When you put 1 & 3
together, you get the really hard problem trying to maintain video.  It's
relatively easy to get 150 ms delay for an audio only session.  However,
it's very hard to maintain that 150 ms on the video.  The biggest single
reason why videoconference systems don't work is 1 & 3.  They force lip
sync, and delay the audio to match the video. The single largest reason why
they can't make the video fast enough is that they use a conference bridge,
which means you need to encode and decode twice (encode at the camera,
decode at the bridge, mix, encode at the bridge, decode at the screen).  If
you have a bridge, you basically cannot maintain 150 ms on real networks,
using real quality codecs (#2).

4. Enough screen surface to maintain roughly life sized faces for the
speakers.  It's somewhat less important for non speakers, although it helps
a lot if you keep everyone lifesized.  It matters how far away you are from
the screen of course.  If you are looking at a projected image from far
away, the screen surface area is smaller than if you are looking at a close
up LCD screen.

5. Correct eye gaze.  This drives systems designers nuts.  You must be able
to maintain correct eye to eye contact.  The tolerance is a couple of
degrees.  If you care, we failed worse at this than any other spec in our
work.  You really need the camera behind the screen to make it perfect.  

6. Simultaneous work surface and human surface.  You can't share the screen
very well.  You really need to be able to see the presentation AND the other
humans.  And, BTW, it never works to have just the presenter if you actually
are having a meeting.  Turns out people look at other participants as well
as the presenter, and when you force them to only look at the presenter, it
doesn't work well.  As an amusing sidelight, this is somewhat culture
dependent. Asian cultures are more accepting of presenter only, American's
are least accepting, Europeans are in the middle.  Also, management doesn't
like, or accept this (look at other participants) finding.

7. No "Hollywood Squares".  If you present more than one person on the
screen, the displays must show the participants approximately aligned
vertically with one another (one row of people).  We did a fascinating
experiment with this one: we let people manipulate the presentations using
what is called "Velcro modeling" with static photographs and cardboard
cutouts.  We asked them what "one on one" looked like, then "two on one"
then "three on one"...  In each case, they had full length, waist up, chest
up and face, with full width and a selection of width-cropped images, in a
variety of sizes.  Everyone, and I mean no exceptions, put people in a line.
No one EVER stacked people in a matrix.  They cropped before they scaled.
You would get a lot of side by side, severely cropped faces before you got
any scaling.  3 x 4 presentations (the typical videoconference bridge
display) is optimally wrong.  When we forced the number of participants to
be really large, we finally got a single row of headshots of non speakers
below a large full chest-up view of the speakers, but our "behave the same
as you do in a room" test failed when we implemented that.

There are a couple of things that don't matter as much as you might think.
People are pretty tolerant of errors in the audio and video.  Of course,
there are more annoying and less annoying errors, so it matters more how the
errors are presented/masked then that were errors.  Frame rate is important,
but not as important as the stuff above, as long as it's above 15 or so.
20-30 works well for people, 60 doesn't help much.  

I've been through all the arguments.  You can't be successful (as we defined
it) if you violate the above. A lot of what systems designers do is dead
wrong.  A lot of what is intuitive to engineers about this stuff is wrong.
The social scientists had decent explanations for most of what we found.
For example, the "no Hollywood squares" rule is explained by "height is
status", so someone above another would have more status, but in meetings,
everyone has the same status.  I guess if the King was in the room, he could
be above the rest of the participants.  We didn't test that.

Brian

---

From: Scott Brim <[email-masked]>
Date: Tue, 7 Apr 2009 17:12:41 -0400

Excerpts from Brian Rosen on Tue, Apr 07, 2009 04:34:26PM -0400:
> 1. One way audio delay <150 ms.  This is a cliff.  It's not true
> that, say, 250 ms is a lot better than 500.  You can't hold a
> discussion with more than 150.  It's the single largest reason most
> tools don't work.  We understand why this is true; humans cannot
> perform adequate turn taking with more than 150 ms delay.  The
> single most obvious thing you notice when it's more than 150 is that
> you can't argue, but you notice it even if you aren't arguing.
> Behavior when it's more than 150 is seriously weird.  You measure it
> from mouth to ear.

Brian: this is if you want people to have the same experience they
would have if they were in the same room.  You can also mediate
discussion via tokens (there are <n> tokens and people queue up for
them; they get tokens either automatically or manually based on
moderator's discretion; they keep tokens until they are done with
their part of the discussion) or any other mechanism that removes the
majority of collisions.  I don't think there is any need to reproduce
the experience of a physical meeting.

> 2. Enough video quality to discern emotion.  As a practical matter,
> for say chest and higher "head shots" it's around 400 lines.  Anyone
> who tells you head shots don't work is wrong; as long as you can
> read emotion, it works fine.  In fact, if you give people tradeoffs
> on lots of variables, they will sacrifice human area coverage right
> down to a face before they give up on other things.  You can't get
> really effective participation without video, although it's crystal
> clear that poor video is worse than no video.

Agree about head shots, but only current speakers need "high" quality
video.  I find even poor, slowly updated video to be very useful -- I
can see when people are paying attention.  

> 3. No detectable lip-sync issues.  This generally means the audio is
> within a frame (30ish) ms of the video.  

I have an old DVR and I guess I'm used to lip sync issues.  I can pay
attention to the emotional content of video and audio separately.  We
aren't doing sales here, we're having technical discussion.

> 4. Enough screen surface to maintain roughly life sized faces for
> the speakers.  It's somewhat less important for non speakers,
> although it helps a lot if you keep everyone lifesized.  It matters
> how far away you are from the screen of course.  If you are looking
> at a projected image from far away, the screen surface area is
> smaller than if you are looking at a close up LCD screen.

eh, I find smaller than lifesize to be fine (but I'm not trying to
reproduce a face-to-face meeting).

> 5. Correct eye gaze.  This drives systems designers nuts.  You must
> be able to maintain correct eye to eye contact.  The tolerance is a
> couple of degrees.  If you care, we failed worse at this than any
> other spec in our work.  You really need the camera behind the
> screen to make it perfect.  

Amen.  When I'm listening I position the speaker's window right under
my camera, because I know what it looks like when I'm talking and my
viewers are apparently staring off into space.  

A tool could dynamically move windows around, centering the current
speaker at the ideal location (set by the user).

> 6. Simultaneous work surface and human surface.  You can't share the
> screen very well.  You really need to be able to see the
> presentation AND the other humans.  And, BTW, it never works to have
> just the presenter if you actually are having a meeting.  Turns out
> people look at other participants as well as the presenter, and when
> you force them to only look at the presenter, it doesn't work well.
> As an amusing sidelight, this is somewhat culture dependent. Asian
> cultures are more accepting of presenter only, American's are least
> accepting, Europeans are in the middle.  Also, management doesn't

But that depends on resources at the participant.  A tool itself can't
make this happen without failing at other requirements.  

> 7. No "Hollywood Squares".  

> There are a couple of things that don't matter as much as you might
> think.  People are pretty tolerant of errors in the audio and video.
> Of course, there are more annoying and less annoying errors, so it
> matters more how the errors are presented/masked then that were
> errors.  Frame rate is important, but not as important as the stuff
> above, as long as it's above 15 or so.  20-30 works well for people,
> 60 doesn't help much.  

IMHO even a frame every few seconds is fine for non-speakers.  The
point is to get IETF work done, not to have a great experience.

swb

---

From: "Brian Rosen" <[email-masked]>
Date: Tue, 7 Apr 2009 18:17:48 -0400

I titled this "successful telepresence".
This means "participation as if you were present".
It is not true that you can get meaningful participation with an artificial
mechanism for turn taking.  You can observe this because, despite our best
efforts, we don't strictly follow given turn taking rules at the microphone:
we engage in interactive discussions with regular, human turn taking.

You can try to enforce something more formal, but if you want to have
effective turn taking, you need incredibly low delay.  It is VERY
frustrating trying to discuss anything with lots of delay.  Although you
might think it isn't that bad, it is.  We tested it.

We have tested lower quality video on non-speakers.  You can indeed lower
frame rate, you can't lower video quality.  You have a large problem
switching: as soon as someone starts talking, the frame rate  has to go up.
It turns out, with the codecs we had, that changing frame rate often is very
annoying.

You may think you can handle no lip sync, but when we observe people where
we induce lag or lead of audio relative to video, people have trouble.  They
tend to not watch faces.

Again, we have tested the effect of image size.  When you get small images,
your ability to consider the participants with small video as actually
participating goes down.  You don't interact with them at all the same way.
I'd say that this is an area where compromises probably have to be made when
the number of participants is large, but the research suggests you get a big
drop off in effectiveness.

Eye gaze is much harder to fix than you imagine.  If the camera is, say,
right above your screen, then to a viewer, you seem to be looking at their
upper chest.  The most imaginative solution I've heard about is some
Microsoft research where they located the eyeballs in the image, cut them,
rotated the pupil up, and pasted it back in, in real time.

Finally, we have to decide what our goal is.  The most effective improvement
we could make is to improve the quality and delay of the audio, making it
two way, stereo, with high bandwidth and low delay.  That will get you much
more than any video will.  Once you start with video, then you have to deal
with the realities, and not guess.  You think small images refreshed once a
second will be useful.  Our research suggests it's not very helpful.
Perhaps a bit better than nothing if the main video meets the other
requirements, but observed behavior suggests that it doesn't add much.  Once
you start dealing with things like how you present the video when there are
multiple speakers selected from a large number of potential speakers, it
gets very hard with available codecs.  Not impossible, just hard.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Scott Brim
Sent: Tuesday, April 07, 2009 5:13 PM
To: [email-masked]
Subject: Re: [vmeet] What is takes to have "successful" telepresence

swb
_______________________________________________
NOTE WELL: This list operates according to
http://mipassoc.org/dkim/ietf-list-rules.html.
https://www.ietf.org/mailman/listinfo/vmeet

---

From: Eliot Lear <[email-masked]>
Date: Thu, 09 Apr 2009 08:11:15 +0200

Brian,

I don't know your background, and so forgive me if I get this wrong, but 
it occurs to me that we are having this debate absent the professionals 
(more psychologist/sociologist types).  It sounds like you've given a 
whole lot of thought to a very specific problem.  I happen to believe 
it's not the problem we're addressing.
> we engage in interactive discussions with regular, human turn taking.

In my experience it is impossible to read emotion from the person's face 
at the microphone because, particularly in the larger WGs, you can't see 
his or her face, either because one is behind the speaker, or because 
one's view is otherwise occluded by angle or by the fact that one is 
busy reading email (;-).  In addition, while I would tend to agree with 
you that a 150ms limit (a MOS comparison is probably apt) is important 
for interactive discussions, but that is generally NOT what we have in 
our meetings.  People stand up, present, ask a question, do some 
follow-up and move on.  And so the notion of a discussion is one I would 
generally challenge, in the meeting itself.

What I am saying is that I assert that interaction in a large group is 
quite different than an intimate group, and  we should be careful to not 
over-engineer, and also to assume that a modest change in behavior is 
possible.  What are the bounds of “modest”?  I don't know, quite frankly.

Eliot

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 09 Apr 2009 14:02:55 -0400

Eliot Lear wrote:
> possible. 

+10

 /d

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Scott Brim <[email-masked]>
Date: Thu, 09 Apr 2009 15:34:36 -0400

Dave CROCKER allegedly wrote on 04 09 2009 2:02 PM:
>> quite different than an intimate group, and  we should be careful to
>> not over-engineer, and also to assume that a modest change in behavior
>> is possible. 
> 
> +10
> 
> /d
> 

In a physical room you can easily have multiple queues: (1) people who
are currently making points: (2) people on the same topic; and (3)
people for new topics.  To do that with remote tools you need different
ways to request a speaking token.

---

From: Fred Baker <[email-masked]>
Date: Thu, 9 Apr 2009 13:48:35 -0700

On Apr 7, 2009, at 3:17 PM, Brian Rosen wrote:

> upper chest.

It is indeed. It can also be humorous in outcomes. A comment that came  
back on Cisco Telepresence relates to the fact that it gives the  
impression that the discussants are sitting around a conference table  
with a large hole in the center. PowerPoint display is below the  
screen, discussants are on screen, and the camera is above. That means  
that it often has the appearance that your peer is looking at your  
chest or lower. The distaff side told us they were sometimes a little  
uncomfortable, and asked for a modesty screen.

Where we go really crazy is left/right, though. With three screens in  
front, if a person is looking directly at you on the screen to his  
right, it looks to you like s/he is looking somewhere to your left.  
The right answer probably involves a separate camera per screen, but  
it's less simple than it might appear.

---

From: Doug Otis <[email-masked]>
Date: Thu, 9 Apr 2009 15:14:58 -0700

On Apr 9, 2009, at 12:34 PM, Scott Brim wrote:
> In a physical room you can easily have multiple queues: (1) people  
> who are currently making points: (2) people on the same topic; and  
> (3) people for new topics.  To do that with remote tools you need  
> different ways to request a speaking token.

Using the Cisco WebEx Event Center offers:

Capacity:
    * Up to 3000 participants, compared to 25-500 for the Meeting  
Center.

Broadcast audio
     * Attendees attend by VoIP only
     * Presenters may attend by standard phone or VoIP

Simplified attendee view–attendees can be kept anonymous

Threaded Q&A, prioritize Q&A, attention indicator, private chat,  
instant feedback icons

Flash-based interface requires no plug-ins for attendees

The event is handled by individuals assigned different roles such as  
host and panelists, in addition to the attendees.

They also offer a center to practice presentations.

http://www.webex.com/how-it-works/for-online-events.html

It was not clear whether a host can assist panelists in returning  
control.  It is also not clear how something like XMPP might be  
integrated.  There is also the issue of integrating the room's audio.   
The host can decide whether to mute a participant to control the net- 
box audio input.  This should allow a net-box attached to the digital  
projector to play the role of a panelist where the room's audio is  
feed into the net-box.

It is not clear how the net-box audio output could be used to hear  
remote participants.  Will a manual switch be needed to mute the net- 
box audio output? The audio output should only be enable when  
listening to remote participants.  Otherwise, a time lag going through  
an audio stream will make listening to a speaker within the room  
difficult.

-Doug

---

From: "Brian Rosen" <[email-masked]>
Date: Fri, 10 Apr 2009 09:57:49 -0400

I think you are incorrect, although we did not test environments like that.

One think we probably ought to differentiate is plenary sessions vs work
group sessions.  The reality in plenary is that we do strictly follow the
mic queue rules.  In work groups, we don't do that so strictly; we start
that way, but then there is give and take.

In a work group session, you definitely get emotion from across the room;
human eyes are quite remarkable.  That's not true rooms the size of a
plenary session.  Not sure where it starts to break down. 

The one thing I do want to emphasize is that the work we did showed us, very
starkly, that engineer's opinions we particularly lousy, and that would
include me.  Engineers did not do well in guessing what would work and what
wouldn't.  They were good at creatively coming up with solutions to
identified problems, but were worse than most of the team on anticipating
what the problems were, and what kinds of things work.  Once they got
educated, they got better.

Based on what I know, I would say:
1. Trying to do real video can't be done.  Don't try.
2. Working on the audio gets you a whole lot.  Making audio two way, with
less than 150 ms delay, is practical (in at least work group sessions), and
probably one of the two biggest gains we could make.  It's hard to do this,
but worth it.
3. Getting real time slides to remote participants is the other big thing.
Webex is very good, as is MS Live Meeting.  We may be able to use VNC.
There are other possible tools.
4. We need an effective way for remote participants to queue for the mic.
Perhaps we could rig up a red light on a pole in the back of the room, as if
it was a mic.  When the light is on, a remote participant is queued for the
mic.  It would be cute, but not really cost effective, to make that a large
monitor with a face (or a stack of faces), just so people in the room could
see who, (and maybe how many) people are queued remotely.  This takes some
software work.  The webex hand raise can substitute, but I suspect is not as
effective.  It works if everyone uses it, but its much less effective if
only some people use it. 

Brian 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Scott Brim
Sent: Thursday, April 09, 2009 3:35 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [vmeet] What is takes to have "successful" telepresence