Subject: [clue] 2 draft covering real-time text relevant for conferencing/ multiple participants and telepresence

From: Arnoud van Wijk <[email-masked]>
Date: Wed, 01 Jun 2011 11:46:06 +0200

Hi all,
I am getting a clue about CLUE. :-)
This is an excellent WG covering what we need for a good telePRESENCE 
without hurdles.
If I communicate remote with other persons, I'd like to "forget" that I 
participate remote. There should be no limitations in the ability to 
communicate.

The focus is at this moment on audio and video, lets make real-time text 
also a standard media to be used in all the work and scenarios as well 
with CLUE.
The use of Total Conversation, where audio, video and real-time text are 
presented and available simultaneously will actually optimize the 
communication with others.
As a deaf person, I need lipreading and real-time text together, others 
sign language, others having real-time text as support if the language 
used in the conference is not your native language and have the text in 
your own language.
Or even to talk and discuss with others during the conference with 
real-time text while listening to the main conference.

More about real-time text can be found at: http://www.realtimetext.org 
but most of you are already familiar with it.

It is not only for persons with a hearing disability, it is for all 
breathing humans who want to communicate (and for a few robots out here 
:-) )
But using Total Conversation will remove the biggest Internet 
(telephony/conferencing) communication hurdle for persons who are deaf, 
hard of hearing or have a speech impairment. And that is not a joke.

We have submitted now 2 drafts that are what I feel quite valuable for CLUE.

Text media handling in RTP based real-time conferences
https://datatracker.ietf.org/doc/draft-hellstrom-text-conference/

and
Presentation of Text Conversation in real-time and en-bloc form
https://datatracker.ietf.org/doc/draft-hellstrom-textpreview/

How would you feel about continuing these drafts under the CLUE banner?
I am looking forward to your feedback and comments and well...anything 
that you throw at me and my fellow authors :-)

Thank you.

Sincerely

Arnoud van Wijk

PS we can also help with more insight on Total Conversation regarding 
quality of the video, camera position and the angle of the view (for 
example head for lipreading, upper body and sufficient area around the 
person for sign language etc.).

---

From: "Mike Hammer (hmmr)" <[email-masked]>
Date: Wed, 1 Jun 2011 09:15:41 -0500

Arnoud,

Is that keyboard left and keyboard right?  :)

I took a quick look through the referenced ID and did not see reference
to audio and video feeds in a multi-party situation.  With TP, the
system may be designed to select one of multiple cameras to display when
there are more participants than displays.  So, is your intent that when
a 'texter' is 'speaking' that the keyboard input should direct the
camera on him/her to be the one selected for display?

How do you identify the text-speaker from someone just doing email?

Does the rate of texting translate to volume for selection of video
feed?

Is the text intended to be overwritten on the screen, or is there a
separate screen?

If there is a separate display screen, do all 'text-speakers' get equal
time, or does an algorithm select which one is displayed there?

If there is a conflict between an audio feed and a text feed for
attention, how do you decide which video is displayed?

Are there signers or text to audio speakers possible?

Lots of questions.

I think you should contribute to CLUE.

However, I would question bolting on a solution after the fact with
separate drafts.

Perhaps, all input and output devices should be integrated into the base
drafts.

Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Arnoud van Wijk
Sent: Wednesday, June 01, 2011 5:46 AM
To: [email-masked]
Subject: [clue] 2 draft covering real-time text relevant for
conferencing/ multiple participants and telepresence

Hi all,

We have submitted now 2 drafts that are what I feel quite valuable for
CLUE.

and

Thank you.

Sincerely

Arnoud van Wijk

_______________________________________________
clue mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/clue

---

From: Mary Barnes <[email-masked]>
Date: Wed, 1 Jun 2011 11:12:15 -0500

Given that this work has been submitted to DISPATCH for discussion,  the
thread of discussion needs to occur on the DISPATCH WG mailing list - in
particular the requirements for this solution.

That all said, if there is anything in the current set of CLUE requirements
that would prevent this functionality in the future, feedback of that nature
would be useful.

Regards,
Mary.
CLUE WG co-chair

On Wed, Jun 1, 2011 at 4:46 AM, Arnoud van Wijk <
[email-masked]> wrote:

> also a standard media to be used in all the work and scenarios as well with
> CLUE.
> sign language, others having real-time text as support if the language used
> in the conference is not your native language and have the text in your own
> language.
> More about real-time text can be found at: http://www.realtimetext.org but
> most of you are already familiar with it.
>
> breathing humans who want to communicate (and for a few robots out here :-)
> )
> (telephony/conferencing) communication hurdle for persons who are deaf, hard
> of hearing or have a speech impairment. And that is not a joke.
>
> I am looking forward to your feedback and comments and well...anything that
> you throw at me and my fellow authors :-)
>
> Thank you.
>
> Sincerely
>
> Arnoud van Wijk
>
> quality of the video, camera position and the angle of the view (for example
> head for lipreading, upper body and sufficient area around the person for
> sign language etc.).
>

---

From: Arnoud van Wijk <[email-masked]>
Date: Mon, 06 Jun 2011 11:57:26 +0200

Hi Mike,
Good questions actually.
We have to visualize how to have Real-Time Text (RTT) included with 
telepresence.
comments inline..

> Is that keyboard left and keyboard right?  :)

Actually, not a bad comment. If you have a conference with say 3 people. 
The camera will zoom/focus o the speaker based on microphone activity. 
The same thing is also possible when the speaker uses a keyboard. When 
typing with RTT, the camera will zoom/focus on the typing person. Use 3 
wireless keyboards here.
> camera on him/her to be the one selected for display?

Yes. :-) For example.
> How do you identify the text-speaker from someone just doing email?
Assign a hot key for RTT/TP activity for example. That key can also be 
used as a raising the hand signal to the conference floor manager as well.

> Does the rate of texting translate to volume for selection of video
We have slow typist and fast typists, I think the duration of activity 
can be used parallel as to volume yes. But I think we have to test this 
kind of scenarios on what is most convenient/realistic. Or even when 3 
people type..have 3 mosaics screens and the longest or most text typing 
person will grow full screen. I am just brainstorming a bit here.
> Is the text intended to be overwritten on the screen, or is there a

Both. Depending on what is preferred by the users. I think with 
discussions where the camera moves between several users, the RTT can be 
used as an overlay on the video or at the bottom. But at the same time 
you can use the text on a separate screen and when the camera/audio is 
locked on a certain user, the user id will be added to the RTT on the 
common screen for example.
> time, or does an algorithm select which one is displayed there?

All get equal time if a separate display is used for text. The text 
preview draft does give an example of multiple users with RTT. As in 
figure 3 for example 
https://datatracker.ietf.org/doc/draft-hellstrom-textpreview/?include_text=1

> attention, how do you decide which video is displayed?

That depends on the users participating. I encounter the issue myself 
that I have a hard time to insert myself in a discussion because my text 
get overlooked or takes more time to be noticed.
If you would allow user tags per stream identifying the speaker, you can 
perhaps add a flag that this person talks by text.
In such case the RTT may have the highest priority in the media stream 
selection. The RTT user must be perfectly aware of this then.

> Are there signers or text to audio speakers possible?

Sure, we have to incorporate them in the scenarios and work out the 
details. It is all about behavior and conference handling /protocol 
between the people and the system. But a signer can wave to the camera 
and with the rapid development of motion sensors like the kinect and 
similar, that is not really an obstacle technically. We just need to add 
it to the possible system behavior.
> Lots of questions.

Much appreciated. Questions will help. I still have to think more on 
Total Conversation with TP but the addition of RTT is and will be SO 
important!
I also like that the telepresence can be used to enable remote sign and 
speech to text interpretation services.
The biggest problem is that a remote interpreter cannot see who is 
talking and who the person is. And with video added the interpreter can 
see the body language to see if the speaker is joking or angry or 
frustrated. Such things are always very well expressible via sign. :-)

> I think you should contribute to CLUE.

Yes I agree. The more I am learning about what CLUE wants to do the more 
I see how important this work is. And to inCLUdE persons who use 
alternative modes of communication besides voice.
> Perhaps, all input and output devices should be integrated into the base

I agree. By standard all CLUE work should use Total Conversation. Then 
we have audio and video as it has been described already but also 
include Real-Time Text!
And use cases can include signing users (wave to camera or sensor to get 
focus on the signing user), type text to get the focus/"raising hand" 
and where you have a remote speech to text/ sign language interpreter 
participating in the conference session. etc etc
(you do not need to be deaf or speech impaired, since you can also have 
a remote Spanish to English interpreter participating, and the output is 
via text and audio).

cheers

Arnoud

---

From: Arnoud van Wijk <[email-masked]>
Date: Mon, 06 Jun 2011 12:05:30 +0200

Hi Mary,
I see an overlap between part what the drafts makes possible and with CLUE.
As you can see in another mail with Mike, the addition of Real-Time Text 
(RTT) as part of CLUE basis documents and scenarios will be important to 
ensure maximal inclusion and presence.

With what we will learn here and with the scenarios/use cases. We can 
also take such insights and considerations in the 2 drafts.

I encourage you all to give feedback on those 2 drafts in the DISPATCH 
list. I will appreciate that. It does not need to be as CLUE, but I have 
seen many AVT and XCON people here who may be the best WGs to adopt the 
drafts.
Or from any other WG who feels the drafts are useful.

And yes lets discuss them on the DISPATCH list.
Thanks!

Arnoud

On 01-06-11 18:12, Mary Barnes wrote:
> Given that this work has been submitted to DISPATCH for discussion, 
>  the thread of discussion needs to occur on the DISPATCH WG mailing 
> list - in particular the requirements for this solution.
>
> That all said, if there is anything in the current set of CLUE 
> requirements that would prevent this functionality in the future, 
> feedback of that nature would be useful.
>
> Regards,
> Mary.
> CLUE WG co-chair
>
> On Wed, Jun 1, 2011 at 4:46 AM, Arnoud van Wijk 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi all,
>     This is an excellent WG covering what we need for a good
>     telePRESENCE without hurdles.
>     If I communicate remote with other persons, I'd like to "forget"
>     that I participate remote. There should be no limitations in the
>     ability to communicate.
>
>     The focus is at this moment on audio and video, lets make
>     real-time text also a standard media to be used in all the work
>     and scenarios as well with CLUE.
>     The use of Total Conversation, where audio, video and real-time
>     text are presented and available simultaneously will actually
>     optimize the communication with others.
>     As a deaf person, I need lipreading and real-time text together,
>     others sign language, others having real-time text as support if
>     the language used in the conference is not your native language
>     and have the text in your own language.
>     More about real-time text can be found at:
>     http://www.realtimetext.org but most of you are already familiar
>     with it.
>
>     It is not only for persons with a hearing disability, it is for
>     all breathing humans who want to communicate (and for a few robots
>     out here :-) )
>     (telephony/conferencing) communication hurdle for persons who are
>     deaf, hard of hearing or have a speech impairment. And that is not
>     a joke.
>
>     I am looking forward to your feedback and comments and
>     well...anything that you throw at me and my fellow authors :-)
>
>     Thank you.
>
>     Sincerely
>
>     Arnoud van Wijk
>
>     PS we can also help with more insight on Total Conversation
>     regarding quality of the video, camera position and the angle of
>     the view (for example head for lipreading, upper body and
>     sufficient area around the person for sign language etc.).
>
>     [email-masked] <mailto:[email-masked]>