Subject: [Slim] Handling the complex multi-media cases

From: Brian Rosen <[email-masked]>
Date: Tue, 3 Nov 2015 11:01:58 +0900

Gunnar has described situations where it is desirable to have relatively
complex decisions on the right set of media streams, translators, relays,
etc to achieve the user's goals.  I have been consistently trying to solve
the 95% cases with simple mechanisms, and trying not to make our solutions
so complex that they don't get implemented.

However, these cases DO occur, and we need to figure out, at some point,
how we deal with them.

One suggestion (thanks Stephen) is to use clue in those cases.  It has the
kind of flexibility that Gunnar is looking for.  There is a complexity cost
(clue isn't simple), but that's the right tradeoff - simple things are
simple, complex things are harder.

I'm fairly sure clue would work unmodified, but maybe we can ask Gunnar to
take a look (not easy, sorry) at the clue document set and see if it works
for all the cases he is aware of.

Brian

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 2 Nov 2015 21:14:27 -0500

Brian said:

"One suggestion (thanks Stephen) is to use clue in those cases."

[BA] Before getting into solutions, can we first understand what use cases
are not handled by the current draft and SDP O/A functionality?

The recently adopted draft can handle asymmetric use cases, which expands
the potential list of supported disability modes considerably.

As you described, O/A cannot express preferences between combinations of
media and languages, but it is possible to negotiate multiple
media/language pairs and deal with the "preference" aspect out of band.

So it would be helpful to understand which of the complex use cases cannot
be accommodated in any shape or form.

On Mon, Nov 2, 2015 at 9:01 PM, Brian Rosen <[email-masked]> wrote:

> _______________________________________________
> SLIM mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/slim
>
>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 2 Nov 2015 21:16:25 -0500

I have proposed in the past that most of CLUE could be used complex 
multi-stream calls that are *not* telepresence. At the moment, CLUE 
advertisements are biased toward the kinds of attributes that are 
meaningful for telepresence. It would be trivial to extend that to carry 
other sorts of attributes. And it is true that what it has now might be 
sufficient to describe all of Gunnar's complex cases. (But I haven't 
analyzed that in detail.)

It could work out in a fairly smooth way. The recommended way to initial 
a clue call is by first offering non-clue media (in case the callee 
doesn't do clue), plus the clue data channel. The non-clue part of the 
offer could include all the slim stuff. If the callee doesn't support 
clue, then it will work just like we have been discussing here, and the 
clue channel will be rejected.

If the callee supports clue, then it would also accept the clue channel. 
Then the complex negotiation could happen, and other media set up.

So I concur that this is one possible way forward.

But it would only handle the complex cases if both ends do. I don't know 
whether there could be enough uptake of clue for that to be useful for 
these cases.

	Thanks,
	Paul

On 11/2/15 9:01 PM, Brian Rosen wrote:
> complex decisions on the right set of media streams, translators,
> relays, etc to achieve the user's goals.  I have been consistently
> trying to solve the 95% cases with simple mechanisms, and trying not to
> make our solutions so complex that they don't get implemented.
>
> One suggestion (thanks Stephen) is to use clue in those cases.  It has
> the kind of flexibility that Gunnar is looking for.  There is a
> complexity cost (clue isn't simple), but that's the right tradeoff -
> simple things are simple, complex things are harder.
>
> I'm fairly sure clue would work unmodified, but maybe we can ask Gunnar
> to take a look (not easy, sorry) at the clue document set and see if it
> works for all the cases he is aware of.
>
> Brian
>
>

---

From: Gunnar Hellström <[email-masked]>
Date: Tue, 3 Nov 2015 11:28:29 +0100

Yes, it is a good approach to explain the complex cases and agree on how 
important they are and check how they are met or can be met by real-time 
slim.

First, a comment on Clue. Yes, it seems to have audio, video and text. 
It is a bit vague about test, but I think it is there. It is possible to 
set a parameter for relates-to, and that is what we may be missing in 
sdp and slim so far.
So, not attractive, but could be looked at again. I agree with Paul that 
the likelihood of implementation would be low if we point to Clue for 
solving the complex slim cases.

I would rather think that we need a supplement in form of an assisting 
service parameter meaning, "if our languages and modalities are not 
compatible, here is a service that can help with translation between my 
preferences and spoken language (yes, all current relay services go 
between spoken and other modalities ).

But let us look at the use cases:

First an assumption: The user sets the settings in its terminal or 
application provider supplied user profile in a way that it is included 
in sdp of the call setup.  The vision is that this setting is not or 
only occasionally modified by the user when making a call or receiving a 
call.

1. Single two-way language  ( the simple one)
One language and modality both ways in media described in sdp as audio 
or video or text.
Straightforward. Works for spoken, written and signed languages.
Result: Possible

2.  Alternatives in the same modality (also simple )
Two or more language alternatives in the same modality.
Two or more languages both ways in media described in sdp as audio or 
video or text, but only in one modality.
Straightforward. Works for spoken, written and signed languages. The 
answering part selects.
There is a relative preference expressed by the order, and the answering 
part can try to fulfill that in the best way.
Result: Possible

3. Fairly equal alternatives in different modalities.
Two or more modality alternatives.
Two or more languages in different modalities both ways in media 
described in sdp as audio or video or text
Example a hearing person also competent in sign language declares both 
spoken and sign language competence in audio and video.

This is fairly straightforward, as long as there is no strong difference 
in preference for these alternatives.
The indication of sign language competence is needed to avoid invoking 
relay services in calls with deaf sign language users only indicating 
sign language.
Result: Possible

4. Last resort indication.
A hearing user has text capability but want to use that as last resort.
With current hunintlang specification, there is no way to describe 
preference level between modalities and no way to describe absolute 
preference.
Result: An answering service will have no guidance to which is the 
preferred modality and may select to use the modality that is the 
callers last resort even if the preferred alternative is available.
A practical case can be a sign language user with a small mobile 
terminal that has some inconvenient means for texting, but sign language 
will be strongly preferred. In order to not miss any calls, the 
indication of text as last resort would be desirable.
Possible solution: coding of an absolute preference: hi, med, lo 
together with the tag.
Result: Need for absolute preference indication.

5. Directional capabilities in different modalities
A hard-of-hearing user strongly prefers to talk and get text back. 
Getting spoken language input is appreciated.
This can be indicated by spoken language two-ways in audio, and 
reception of language in text.
But there is nothing in current coding that says that the text path is 
important. The answering part may see it as an alternative, just as case 3.
There is a way to indicate that the call shall fail if a language is not 
met, but that may be too drastic for this user. It may be important to 
be able to connect and just say something, or use residual hearing to 
get something back when the voice is familiar.
Possible solution: coding of an absolute preference: hi, med, lo 
together with the tag could solve this case if used together with the 
directional indications.
Another solution would be to indicate required grouping of media. But 
that seems more complex.
Result: Need for absolute preference indication.

6. Combination of modalities
Quite similar to case 5.
A deaf-blind person may have highest preference for signing out and 
getting text back.
Indication of sign language output in video and text reception in text, 
using the current directional attributes.
An answering part may seek suitable modalities for each direction and 
find the only possible combination.
Result: Possible

7. Person with speech disabilities who prefer speech-to-speech service.
A person who speaks in a way that is hard to understand, may be used to 
have support of a speech-to-speech relay service that adds clear speech 
when needed for the understanding.
Calls with close friends and family might be possible without the relay 
service.
Text can be used as a fall-back.
This user would indicate preference for receiving spoken language in audio.
Text output can be indicated but this user might want to use that as 
last resort.
But there is no current coding for vague or unarticulated speech or 
other needs for a speech-to-speech service.
A possibility could be to indicate no preference for spoken language 
out, a coding of proposed assisting service and an indication of text 
output on a low absolute level.
Result: Need of service indication, and absolute level of preference 
indication

8. Person with speech disabilities who prefer to type and hear.
A person who speaks in a way that is hard to understand, may be used to 
use text for output and listen to spoken language for input.
This user would indicate preference for receiving spoken language in audio.
Text output modality can be indicated.
If the answering part has text and audio capabilities, there is a match. 
If only voice, there is a need to invoke a text relay service.
Result: Possible.

9. A tele-sales center calls out and want to offer all kinds of 
possibilities so that the answering part can select.
(Hated example, but I take it anyway)
A tele-sales center has competence in multiple spoken languages and can 
invoke relay services rapidly if needed.
So, it indicates in the call setup competence in a number of spoken 
languages in audio, a number of sign languages in video and a number of 
written languages in text.
A deaf-blind person who prefers to sign out and get text back answers 
with only these capabilities.
The center can detect that and act accordingly.
Alternative1: The center calls without sdp. The deafblind user includes 
its sdp offer and the center sees what is needed to fulfill the call.
Alternative 2: The center calls out with only the spoken language 
capabilities indicated that the caller can handle.
The deaf-blind answering person, or terminal or service provider detects 
the difference compared to the capabilities of the answering party, and 
adds a suitable relay service. ( this does not use all the offerings of 
the callers competence to pull in extra services, but is maybe a more 
realistic case for what usually happens. )
Result: doable in the same way as cases 1-8.

Total result:
The currently specified directionality is needed.
An indication of absolute preference is needed.  ( maybe instead of the 
current switch between complete vs fail the call that seems too drastic 
for most cases. )
An indication of suitable service and its spoken language is needed for 
the speech-to-speech case, but can be useful for other cases as well.
No clear need for explicit grouping of modalities seem to be needed. ( 
and that was what clue could contribute, so the clue idea can probably 
be dropped)

More thoughts are needed on who really evaluates the indications and 
invokes a needed service, but the above likely covers the needs for 
language/modality indications.

Gunnar

Den 2015-11-03 kl. 03:16, skrev Paul Kyzivat:
> meaningful for telepresence. It would be trivial to extend that to 
> carry other sorts of attributes. And it is true that what it has now 
> might be sufficient to describe all of Gunnar's complex cases. (But I 
> haven't analyzed that in detail.)
>
> It could work out in a fairly smooth way. The recommended way to 
> initial a clue call is by first offering non-clue media (in case the 
> callee doesn't do clue), plus the clue data channel. The non-clue part 
> of the offer could include all the slim stuff. If the callee doesn't 
> support clue, then it will work just like we have been discussing 
> here, and the clue channel will be rejected.
>
> If the callee supports clue, then it would also accept the clue 
> channel. Then the complex negotiation could happen, and other media 
> set up.
>
> But it would only handle the complex cases if both ends do. I don't 
> know whether there could be enough uptake of clue for that to be 
> useful for these cases.
>
>     Thanks,
>     Paul
>

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 3 Nov 2015 15:43:00 -0500

On Nov 3, 2015, at 05:28, Gunnar Hellström <[email-masked]> wrote:
> With current hunintlang specification, there is no way to describe preference level between modalities and no way to describe absolute preference.
> Result: An answering service will have no guidance to which is the preferred modality and may select to use the modality that is the callers last resort even if the preferred alternative is available.
> A practical case can be a sign language user with a small mobile terminal that has some inconvenient means for texting, but sign language will be strongly preferred. In order to not miss any calls, the indication of text as last resort would be desirable.
> Possible solution: coding of an absolute preference: hi, med, lo together with the tag.

[BA] The user offers to send and receive ASL in the video m-line, as well as to receive (and send?) RTT on the text m-line. Is the concern that the Answerer will accept both due to lack of preference when ASL is enough? Since video can be congested, having RTT as a backup doesn't seem so bad, even if it is not used by agreement between the parties. Or is the concern that the Answerer could do video/ASL but will reject that while accepting RTT, for some reason?

> A hard-of-hearing user strongly prefers to talk and get text back. Getting spoken language input is appreciated.
> This can be indicated by spoken language two-ways in audio, and reception of language in text.
> But there is nothing in current coding that says that the text path is important. The answering part may see it as an alternative, just as case 3.
> There is a way to indicate that the call shall fail if a language is not met, but that may be too drastic for this user. It may be important to be able to connect and just say something, or use residual hearing to get something back when the voice is familiar.
> Possible solution: coding of an absolute preference: hi, med, lo together with the tag could solve this case if used together with the directional indications.
> Another solution would be to indicate required grouping of media. But that seems more complex.

[BA] Is the concern that an Answerer capable of sending RTT will decline to do so, since a bi-directional audio m-line has been successfully negotiated? If the Answerer accepts bi-directional audio and RTT is that an issue? It seems that after the call is set up the Offerer could indicate the preference for receiving both audio and RTT out of band.

> A person who speaks in a way that is hard to understand, may be used to have support of a speech-to-speech relay service that adds clear speech when needed for the understanding.
> Calls with close friends and family might be possible without the relay service.
> Text output can be indicated but this user might want to use that as last resort.
> But there is no current coding for vague or unarticulated speech or other needs for a speech-to-speech service.
> A possibility could be to indicate no preference for spoken language out, a coding of proposed assisting service and an indication of text output on a low absolute level.
> Result: Need of service indication, and absolute level of preference indication

[BA] How is the decision to invoke a speech-speech relay made?

If the Offerer offers to receive audio and but send (and receive?) RTT, and the Answerer accepts both m-lines, would this avoid invoking a speech-speech relay?

Bi-directional audio could be offered along with sending (and receiving?) RTT. If the Answerer accepts both, the offerer could request out of band that the Answerer use RTT if audio cannot be understood (would speech-speech relay be invoked at that point?)

---

From: Gunnar Hellström <[email-masked]>
Date: Wed, 4 Nov 2015 05:18:33 +0100

> [BA] The user offers to send and receive ASL in the video m-line, as well as to receive (and send?) RTT on the text m-line. Is the concern that the Answerer will accept both due to lack of preferen…
<GH>Yes, the last explanation. Both modaities are not needed ( even if 
good to have). So if an answerer scans the indications and find ASL and 
RTT there and has the capability to select between ASL only and RTT only 
then the preference indication should guide to answer with ASL.
> [BA] Is the concern that an Answerer capable of sending RTT will decline to do so, since a bi-directional audio m-line has been successfully negotiated? If the Answerer accepts bi-directional audio…
<GH>Yes, the user in the example will not be well served with a plain 
audio call. Receiving RTT is essential. Receiving speech is not equally 
important, but good.
So some indication of the minimum reqiured combination of modalities and 
directions seems to be needed. The preference for reception of text 
should be sufficient. It is likely best for this case that the 
indication for spoken modality is divided in one high priority for 
spoken output, but a medium preference for spoken reception. By that, 
the answering part can scan the indications and find the minimum set, 
and also answer in the best modality.
In earlier discussions, I have leaned towards a need for a specific 
service preference indication for captioned telephony for this case. But 
I think this indication would be sufficient, and enable a direct 
connection with a person who is willing to type and listen. That should 
be checked.

> Bi-directional audio could be offered along with sending (and receiving?) RTT. If the Answerer accepts both, the offerer could request out of band that the Answerer use RTT if audio cannot be under…
<GH>If this is a good choice that works for the user, yes.
For some it would work, for others not. So we need to specify 
indications suitable for both requiring a speech-to-speech service in 
that case and for accepting the spoken + RTT combination.

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 4 Nov 2015 00:26:55 -0500

On Nov 3, 2015, at 23:18, Gunnar Hellström <[email-masked]> wrote:
> <GH>Yes, the last explanation. Both modaities are not needed ( even if good to have). So if an answerer scans the indications and find ASL and RTT there and has the capability to select between ASL only and RTT only then the preference indication should guide to answer with ASL.

[BA] RFC 3264 Section 5.1 says:

"If multiple media streams of different types are present, it means that the offerer wishes to use those streams at the same time. A typical case is an audio and a video stream as part of a videoconference."

Based on the above, an Offer including both video/ASL and text/English means that the Offerer wants to use both. The Answerer may not be video or RTT capable or may decline one or both m-lines even if it is capable, but it cannot conclude that the Offerer considers one of the streams "enough" based on the media type or order of the m-lines.

> <GH>Yes, the user in the example will not be well served with a plain audio call. Receiving RTT is essential. Receiving speech is not equally important, but good.

[BA] Since Offering both is an indication that both audio and RTT are desired, based on RFC 3264, the Answerer cannot conclude that the Offerer considered audio alone sufficient.

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 4 Nov 2015 10:54:12 -0500

> [BA] Since Offering both is an indication that both audio and RTT are desired, based on RFC 3264, the Answerer cannot conclude that the Offerer considered audio alone sufficient.

Well...

Consider the analogy to an offer of audio and video. Typically there 
will be a reasonable expectation that you might get only audio. That 
will be true if the called device only supports audio, but it will also 
occur if the called user chooses not to use video.

It is likely to be different for audio + RTT?

Of course, it may be contextual. People who anticipate receiving calls 
from deaf people may behave differently from those who don't.

Also, if the initial call doesn't yield the media I want, I can always 
retry with a reinvite. Then the two ends will have had a chance to 
assess the situation, and may offer/accept modalities that didn't get 
negotiated the first time.

	Thanks,
	Paul

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 4 Nov 2015 12:15:30 -0500

On Nov 4, 2015, at 10:54 AM, Paul Kyzivat <[email-masked]> wrote:
> Consider the analogy to an offer of audio and video. Typically there will be a reasonable expectation that you might get only audio. That will be true if the called device only supports audio, but it will also occur if the called user chooses not to use video.
> 

[BA] It seems very analogous to me - an m-line of video/ASL might be declined by an Answerer supporting ASL who chooses not to use video - even though they can support it.

In such a situation though, would "priority" make a difference?

> Of course, it may be contextual. People who anticipate receiving calls from deaf people may behave differently from those who don't.

[BA] Yes - and for such a person, a question is whether accepting all m-lines that can be supported could actually cause a problem.

> Also, if the initial call doesn't yield the media I want, I can always retry with a reinvite. Then the two ends will have had a chance to assess the situation, and may offer/accept modalities that didn't get negotiated the first time.

[BA] Right. This may be particularly relevant when a non-disabled person calls someone who is disabled. We probably should look at those use cases as well.

---

From: Gunnar Hellström <[email-masked]>
Date: Thu, 5 Nov 2015 08:26:29 +0100

> [BA] Yes - and for such a person, a question is whether accepting all m-lines that can be supported could actually cause a problem.
[GH]Accepting m-lines is another thing than accepting languages.
We want to express alternatives, across media. E.g. I prefer sign 
language but can accept to use text. Then it must be acceptable that the 
answerer only accepts one of them.
It is acceptable then to deny the medium that will not be used for 
language, but that is not at all required. You can very well still 
connect and have good use of media even if it is not intended for 
language communication.

[]But we also need to be able to express required pairs or groups: E.g. 
I prefer talking and reading.
A question is if we can define an implied rule for how such grouping is 
done. Is it sufficient to say that such groupings shall be expressed 
with directional humintlang attributes, and that the answerer is 
supposed to include at least one language in each direction with the 
highest priority possible?  Or do we need to explicitly point out 
required pairs or groups?  My numbered cases did not reveal any real 
need to use any coding for explicit pairing of language indications, but 
we might need to take more examples and test.

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 5 Nov 2015 10:46:58 -0500

> we might need to take more examples and test.

There are two distinct but related issues here:

- ensuring that the necessary media are established for the desired 
modalities

- communicating to the other party what modality is desired.

For instance, if you wish to talk and read, then the other person needs 
to listen and type. Establishing the media is only half the battle. He 
still needs to use them as intended.

Putting the info in the SDP doesn't ensure that it will be rendered to 
the user in the device UI.

Is this important? Or, once the necessary media have been established, 
can we count on the two ends to figure out how to behave with each other.

(This does get more complex if the intent is use this information to 
decide whether to optionally introduce a relay to convert modalities.)

	Thanks,
	Paul

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 5 Nov 2015 11:13:47 -0500

On Nov 5, 2015, at 10:46, Paul Kyzivat <[email-masked]> wrote:
> - ensuring that the necessary media are established for the desired modalities
> 
> For instance, if you wish to talk and read, then the other person needs to listen and type. Establishing the media is only half the battle. He still needs to use them as intended.
> Putting the info in the SDP doesn't ensure that it will be rendered to the user in the device UI.
> 
> Is this important? Or, once the necessary media have been established, can we count on the two ends to figure out how to behave with each other.

[BA] The SDP Offers and answers may need to be formulated with the need to communicate desired behavior considered.  Consider the scenario where the Offerer can speak, but is hearing impaired, and therefore prefers to send audio/English (but cannot receive it) and therefore needs to receive RTT/English.  Even if the Answerer accepts both audio and text m-lines there could be some confusion about the intended communication mode. For example, if the Answerer did not understand the new language attributes and the audio m-line was send/recv he/she could start out speaking to the Offerer who cannot hear him/her. The situation might be made more clear if the audio m-line in the Offer was send-only.  However a send/recv text m-line may be desirable even if the language attribute indicates only the desire to receive English, not send it. If the text m-line was recv-only instead of send/recv then the Offerer might not have a channel to communicate their need to receive RTT, not audio.
> 
> (This does get more complex if the intent is use this information to decide whether to optionally introduce a relay to convert modalities.)
> 
>    Thanks,
>    Paul
>

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 5 Nov 2015 17:01:31 -0500

> [BA] The SDP Offers and answers may need to be formulated with the need to communicate desired behavior considered.  Consider the scenario where the Offerer can speak, but is hearing impaired, and…

Your response elaborates what I was talking about.

Even if that information is in the SDP, we have no guarantee that it 
will be communicated to the user so that he can alter his behavior. Even 
if it does communicate it to the user, he may not understand it. (Maybe 
this is the first time he has ever received a call from a deaf person.)

Still, I hopefully after a brief time of poor communication he will 
figure out what he needs to do to communicate successfully.

My question is whether we expect that the UI of the called device will 
actually be built to communicate this sort of information?

I think we can minimally assume that devices that support audio and 
video will see to it that the user knows whether audio and video are 
present in the call. And those that support RTT will make sure that the 
user knows when it is in the call, and provide a way to enter and 
display text.

Beyond that, we can probably assume that devices explicitly built for 
the deaf market will go further, to communicate modality preferences. 
Whether others will is hard to say.

I'm not aware of any UAs that have support for RTT for reasons *other* 
that supporting the deaf community. It may be that any device supporting 
RTT may be assumed to cater to the deaf community. But that may not be 
true. It is certainly conceivable that it would be used for other purposes.

IMO it would be good if all devices that support chat also supported 
RTT. They could then negotiate on a call by call basis which protocol to 
use for text exchange. (RTT, MSRP, XMPP, etc.) That would maximize the 
possibility of meeting each user's needs while establishing the session 
whenever possible. But I don't see how this is going to happen.

	Thanks,
	Paul

---

From: Gunnar Hellström <[email-masked]>
Date: Thu, 5 Nov 2015 21:23:03 +0100

> [BA] The SDP Offers and answers may need to be formulated with the need to communicate desired behavior considered.  Consider the scenario where the Offerer can speak, but is hearing impaired, and…
[GH] We should not use sendonly and recvonly as means to indicate 
modality preferences. Media flow in the other direction may be very 
desirable for other purposes, or for very occasional language transfer, 
not worth complicating the language/modality indications. So, yes, your 
main reasoning is good, we simply should not complicate our task by 
discussing sendonly and recvonly.
/Gunnar

---

From: Gunnar Hellström <[email-masked]>
Date: Thu, 5 Nov 2015 23:50:35 +0100

Den 2015-11-05 kl. 16:46, skrev Paul Kyzivat:
> For instance, if you wish to talk and read, then the other person 
> needs to listen and type. Establishing the media is only half the 
> battle. He still needs to use them as intended.
>
[GH]  Right. We have one piece left to specify. That is how offer/answer 
shall be handled. Offer is well on its way. But how is the answer composed?
Especially for the directional indications.
Should the answer only indicate the acknowledged capabilities meeting 
the offeror's indications?
Example: Offer: Japanese speech sending in audio     Answer: Japanese 
speech reception in audio.

?
/Gunnar

---

From: Gunnar Hellström <[email-masked]>
Date: Thu, 5 Nov 2015 23:54:08 +0100

> can we count on the two ends to figure out how to behave with each other.
[GH] The current draft contains a brief specification that the language 
indications should be provided to the calling and answering users.
/Gunnar

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 5 Nov 2015 22:59:41 -0500

> indications should be provided to the calling and answering users.

Sure. But that probably isn't normative (since it is about the UI, not 
the protocol), and in any case it would only be binding on those who 
support this. There is a strong possibility that this will be treated as 
something that is only applicable to a niche market.

Is there something that we can do to encourage widespread implementation 
of this draft? Making it something valuable to all would be one way. 
(Eliminating the "press 1 for Spanish" query might provide that motivation.)

	Thanks,
	Paul

---

From: Gunnar Hellström <[email-masked]>
Date: Sat, 7 Nov 2015 00:42:45 +0100

Moving on. ( I think that Paul's proposal to motivate use by that it 
reduces the number of times you need to listen through a voice menue is 
excellent. )

Another use case came to my mind.

That is the
hard-of-hearing or deaf person who experience that the lip-reading 
support achieved when you see well synced video image of the person 
talking is of value for understanding.

That requires indicating spoken language in audio both ways, and 
indication of reception of spoken language in video.
Indication of all three having high priority would be good to indicate 
that they are all needed.

So, would it also here be sufficient to indicate:
Spoken language reception in audio with high priority.
Spoken language transmission in audio with high priority
Spoken language reception in video with high priority

This is for when the user really finds it important t see the person 
talking, and does not feel it being sufficient with only audio. Then the 
three media should have the same priority.

There are of course variations in reality, when either of these 
languages can be sacrificed relatively easily, and therefore be 
indicated with lower priority.

Can we have an application rule for composing the indications saying that
If the user has different needs in different directions, it might be 
sufficient to declare the need in directional language tags.

/Gunnar

Den 2015-11-06 kl. 04:59, skrev Paul Kyzivat:
> support this. There is a strong possibility that this will be treated 
> as something that is only applicable to a niche market.
>
> Is there something that we can do to encourage widespread 
> implementation of this draft? Making it something valuable to all 
> would be one way. (Eliminating the "press 1 for Spanish" query might 
> provide that motivation.)
>
>     Thanks,
>     Paul
>

-- 
-----------------------------------------
Gunnar Hellström
Omnitor
[email-masked]
+46 708 204 288

---

From: Bernard Aboba <[email-masked]>
Date: Sat, 7 Nov 2015 10:55:36 -0500

Gunnar said:
"That is the hard-of-hearing or deaf person who experience that the
lip-reading support achieved when you see well synced video image of the
person talking is of value for understanding."

[BA] In this case, it would probably make sense to Offer audio and video
m-lines in a Lip-Sync (LS) group as defined in
https://tools.ietf.org/html/rfc5888

That requires indicating spoken language in audio both ways, and indication
of reception of spoken language in video.

[BA] Not sure what it means to negotiate spoken language in video.  For
example, if the Answerer does not indicate that it can send spoken language
in video, is this a problem?  Or is this just a way to indicate that the
video is important? The presence of a lip-sync group seems like enough of
an indication that the video is intended to sync with the audio m-line.

On Fri, Nov 6, 2015 at 6:42 PM, Gunnar Hellström <
[email-masked]> wrote:

> hard-of-hearing or deaf person who experience that the lip-reading support
> achieved when you see well synced video image of the person talking is of
> value for understanding.
>
> There are of course variations in reality, when either of these languages
> can be sacrificed relatively easily, and therefore be indicated with lower
> priority.
>

---

From: Paul Kyzivat <[email-masked]>
Date: Sat, 7 Nov 2015 11:42:30 -0500

On 11/7/15 10:55 AM, Bernard Aboba wrote:
> [BA] Not sure what it means to negotiate spoken language in video.

It makes sense to me that this would signify lip reading.

> For
> example, if the Answerer does not indicate that it can send spoken
> language in video, is this a problem?

To *send* for lip reading, you need to send video that shows the face 
clearly. Not indicating it for sending might mean that you aren't 
sending video of faces of speakers, or it could simply mean that you 
didn't indicate what you are doing.

> Or is this just a way to indicate
> that the video is important? The presence of a lip-sync group seems like
> enough of an indication that the video is intended to sync with the
> audio m-line.

I don't know about that. In principle you don't need audio to lip read. 
If there was only video, then you wouldn't need lip sync.

And if you are sending audio and video, lip sync is only saying that the 
timing is consistent between the two. It doesn't say that lips are visible.

	Thanks,
	Paul

> On Fri, Nov 6, 2015 at 6:42 PM, Gunnar Hellström
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     reduces the number of times you need to listen through a voice menue
>     is excellent. )
>
>     Indication of all three having high priority would be good to
>     indicate that they are all needed.
>
>     talking, and does not feel it being sufficient with only audio. Then
>     the three media should have the same priority.
>
>         Sure. But that probably isn't normative (since it is about the
>         UI, not the protocol), and in any case it would only be binding
>         on those who support this. There is a strong possibility that
>         this will be treated as something that is only applicable to a
>         niche market.
>
>         implementation of this draft? Making it something valuable to
>         all would be one way. (Eliminating the "press 1 for Spanish"
>         query might provide that motivation.)
>
>              Thanks,
>              Paul
>
>         [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>
>     +46 708 204 288 <tel:%2B46%20708%20204%20288>
>
>
>     [email-masked] <mailto:[email-masked]>

---

From: Paul Kyzivat <[email-masked]>
Date: Sat, 7 Nov 2015 11:49:12 -0500

> reduces the number of times you need to listen through a voice menue is

While it seems like it would be helpful, I don't know what we can do to 
encourage it. I expect the key would be to convince some important call 
centers to implement this.

Or maybe there could be some support for this added to vxml, so it is 
easy to write a script that checks for this and bypasses the query.

[snip]

> sufficient to declare the need in directional language tags.

I think this could start with a use case document, listing an assortment 
of differently skilled people, and the recommended indication for each, 
in terms of the offer they would make. That *might* eventually lead to 
something normative.

	Thanks,
	Paul

---

From: Gunnar Hellström <[email-masked]>
Date: Sat, 7 Nov 2015 21:02:52 +0100

Den 2015-11-07 kl. 16:55, skrev Bernard Aboba:
> lip-reading support achieved when you see well synced video image of 
> the person talking is of value for understanding."
>
> [BA] In this case, it would probably make sense to Offer audio and 
> video m-lines in a Lip-Sync (LS) group as defined in 
> For example, if the Answerer does not indicate that it can send spoken 
> language in video, is this a problem?  Or is this just a way to 
> indicate that the video is important? The presence of a lip-sync group 
> seems like enough of an indication that the video is intended to sync 
> with the audio m-line.
[GH]
Section 6.2 in the real-time slim draft already says:

  Note that while signed language tags are used with a video stream to
    indicate sign language, a spoken language tag for a video stream in
    parallel with an audio stream with the same spoken language tag
    indicates a request for a supplemental video stream to see the
    speaker.

[GH] The lip-sync group is a good feature, but I think it is good that we have a way to specify the need for lip-reading possiblity from a language point of view.

Bernard asks: "For example, if the Answerer does not indicate that it 
can send spoken language in video, is this a problem? "
[GH] The other way around is reassuring. Indication of sending spoken 
language in video can make the lip-reading depending caller assured that 
the call has good opportunities to be smooth.

If the indication of sending of spoken language in video is not there in 
the answer, it can be an alerting indication that the user or its 
terminal or service provider may need to make some additional measures 
to make the call flow well. E.g. add a relay service, start out with an 
explanation that text might be needed some times  etc.

This discussion leads to another interesting case.

Assume that the person with preference for lip-reading supported spoken 
language want to express a clear fall-back option meaning:
"If you cannot provide a view of you while talking, then I still want to 
talk, but I want to receive real-time text."
I think that the caller would then indicate reception of written 
language in the text stream, but with lower absolute preference than for 
the spoken language in video.

How can we code this absolute level of preference? The need for it has 
popped up as a result in many of the use cases now. A coarse coding of 
absolute level of preference is likely sufficient. High, medium and low 
or so.  We are not yet quite sure that we manage without a very specific 
grouping specification, but for some cases it is evident that a simple 
absolute preference is sufficient.

In the draft, section 3.2, there is a paragraph
"

In an offer, each language tag value MAY have an asterisk appended as
    the last character (after the registry value).  The asterisk
    indicates a request by the caller to not fail the call if there is no
    language in common."

I don't know if this syntax is really recommendable for the non-fail indication, so I think that should be discussed.
Would it be better with a clearly separate parameter in the humintlang attribute, in order to not create any collision with coding of the language tag itself?

The coding could be extended to indicate the absolute preference. Just to have a starting proposal, I suggest to handle it as a separate parameter in the Humintlang attribute, with possible values: Fail, Hi, Med, Lo
Fail= fail the call if you cannot meet this language preference.
Hi= very desirable language
Med = Good alternative
Lo = Wanted if you cannot meet my other higher prioritized preferences

Specify just modality.

I imagine that we will have cases of users who find it cumbersome to 
specify languages and would like to only specify modality for language 
use and preference.
Proposal, just omit the language tag and make sure a modality parameter 
is there
Values:
text
sign
speech

Syntax. e.g.
a=humintlang-send: [<lang>] [/<modality>][/<priority>]

/Gunnar

>     Moving on. ( I think that Paul's proposal to motivate use by that
>     it reduces the number of times you need to listen through a voice
>     menue is excellent. )
>
>     support achieved when you see well synced video image of the
>     person talking is of value for understanding.
>
>     This is for when the user really finds it important t see the
>     person talking, and does not feel it being sufficient with only
>     audio. Then the three media should have the same priority.
>
>     If the user has different needs in different directions, it might
>     be sufficient to declare the need in directional language tags.
>
>     /Gunnar
>
>
>
>
>
>
>
>         UI, not the protocol), and in any case it would only be
>         binding on those who support this. There is a strong
>         possibility that this will be treated as something that is
>         only applicable to a niche market.
>

---

From: Bernard Aboba <[email-masked]>
Date: Sat, 7 Nov 2015 16:17:51 -0500

On Nov 7, 2015, at 3:02 PM, Gunnar Hellström <[email-masked]> wrote:

> If the indication of sending of spoken language in video is not there in the answer, it can be an alerting indication that the user or its terminal or service provider may need to make some additional measures to make the call flow well. E.g. add a relay service, start out with an explanation that text might be needed some times  etc. 

[BA] yes, the Offerer might need to communicate needs out-of-band, since the Answerer probably lacks the ability.

> Assume that the person with preference for lip-reading supported spoken language want to express a clear fall-back option meaning:
> "If you cannot provide a view of you while talking, then I still want to talk, but I want to receive real-time text."
> I think that the caller would then indicate reception of written language in the text stream, but with lower absolute preference than for the spoken language in video.

[BA] Given potential issues that could arise with video in event of loss or bad lighting, might it make sense to have RTT just in case? Systems are increasingly supporting text by default (realtime or not) for this debugging/fallback reason.

> We are not yet quite sure that we manage without a very specific grouping specification, but for some cases it is evident that a simple absolute preference is sufficient. 

[BA] So far, existing grouping types (e.g. LS) seem adequate - or is there a use case that suggests the need for another grouping?

---

From: Gunnar Hellström <[email-masked]>
Date: Sun, 8 Nov 2015 00:13:13 +0100

Den 2015-11-07 kl. 22:17, skrev Bernard Aboba:
> On Nov 7, 2015, at 3:02 PM, Gunnar Hellström 
>> If the indication of sending of spoken language in video is not there 
>> in the answer, it can be an alerting indication that the user or its 
>> terminal or service provider may need to make some additional 
>> measures to make the call flow well. E.g. add a relay service, start 
>> out with an explanation that text might be needed some times  etc.
>
> [BA] yes, the Offerer might need to communicate needs out-of-band, 
> since the Answerer probably lacks the ability.
>
>> Assume that the person with preference for lip-reading supported 
>> spoken language want to express a clear fall-back option meaning:
>> "If you cannot provide a view of you while talking, then I still want 
>> to talk, but I want to receive real-time text."
>> language in the text stream, but with lower absolute preference than 
>> for the spoken language in video.
>
> [BA] Given potential issues that could arise with video in event of 
> loss or bad lighting, might it make sense to have RTT just in case? 
> Systems are increasingly supporting text by default (realtime or not) 
> for this debugging/fallback reason.
[GH]Yes, connecting all three media is always good, and we do not want 
to encourage any weakening of that ambition. We just want to indicate 
what the media are intended for, and how important it is to support them 
and  the intended use of them for language communication.  Other use of 
the media is always welcome, as well as variations in use for language 
during the call by mutual agreements between the call participants even 
if it was not indicated from the beginning.
>
>> We are not yet quite sure that we manage without a very specific 
>> grouping specification, but for some cases it is evident that a 
>> simple absolute preference is sufficient.
>
> [BA] So far, existing grouping types (e.g. LS) seem adequate - or is 
> there a use case that suggests the need for another grouping?
[GH] Yeah, possibly. I would hope that we did not need that level of 
complexity.
The LS grouping type for lip-reading seems to be a possible solution for 
indication of same level of priority for including audio and video, and 
also arranging sync on them.

Streching the logic severely, we can also apply LS grouping for 
indication of a preference for captioned telephony support. The normal 
action of that service is to add real-time text as soon as possible as a 
written form of the spoken language in the audio stream. The latency is 
probably most often between 4 and 8 seconds. So, it does not hold 
exactly as "synchronized", but at least has the ambition to be rapid.

Some other use cases only need to indicate grouping of language and 
modality in a group where the value lies in using all media in the 
group, but not requiring any sychronization.
That is the case with the deafblind user wanting to use sign language in 
video out and text back.  I imagined that we could say that 
implementations should scan sdp for a humintlang-send and a 
humintlang-recv with same priority and view them as a group that should 
be supported together.  But a new kind of RFC 5888 grouping might do the 
same job.

Also the hard-of-hearing user who want to use spoken out and text back  
can be seen as that kind of grouping just for the sake of being needed both.

And that is different from the hearing person who want to express 
preference for using spoken language both ways, but an acceptance on 
lower priority level to use text both ways. That is no grouping, just 
prioritizing.

I have not grasped yet if there will be any complication if we want to 
have two or more language usages of different priority on the same 
medium, and they need to be grouped with different media. .......  How 
about this example ( maybe we are stepping over the borderline for what 
is feasible to expect that users want to express here,....
A hard-of-hearing person in USA manages well with spoken Spanish in 
audio with a video view of the other party for complementing hearing 
with some lip-reading.
But it is harder for this person with lip-reading in English. So he 
wants to offer also spoken English out and reception of written English 
in the text stream.
That is two alternative groupings. Not totally unrealistic. There is not 
much use in setting very different absolute priorities on them. Instead 
it is the capabilities of the other party that will cause a selection.   
Can this grouping be expressed in any realistic way without conflicts?

/Gunnar

---

From: Martin J. Dürst <[email-masked]>
Date: Sun, 8 Nov 2015 11:24:09 +0900

> In the draft, section 3.2, there is a paragraph

Which draft? I thought we are speaking about
https://tools.ietf.org/html/draft-ietf-slim-negotiating-human-language-00

but there's no section 3.2 (Section 3 is extremely short).

>     indicates a request by the caller to not fail the call if there is no

What exactly is meant with this?

Is it something like:

en-US, en-AU, fr-CA, *

(i.e. essentially the same as the HTTP Accept-Language header)?

Or is it something like

en-US*, en-AU*, fr-CA*

The later would be totally out of line with previous use of language 
tags and subtags, but it's what I'd guess from a direct reading of the text.

Regards,   Martin.

---

From: Gunnar Hellström <[email-masked]>
Date: Sun, 8 Nov 2015 06:14:50 +0100

> https://tools.ietf.org/html/draft-ietf-slim-negotiating-human-language-00
[GH]Yes, sorry, I meant 6.2.  There is also an expansion of the topic in 
6.3.
> tags and subtags, but it's what I'd guess from a direct reading of the 
[GH] Yes, that was also my interpretation. And wondering what was meant. 
A complete syntax description and example is needed.
But also a discussion if this non-fail parameter should be merged with 
the other priority values that I propose.

/Gunnar

---

From: Natasha Rooney <[email-masked]>
Date: Wed, 11 Nov 2015 06:54:10 +0000

Hey guys,

It would be a good idea to capture the use case identified by Gunnar at the meeting at IETF94 and given in this thread in a document, this will allow us to be able to refer to them in one place rather than referring down a thread.

Gunnar - I can build this document, to get started can you just send a me your list of use cases with any amendments which may have been identified in this email thread? I will then add them to a document which will be in an RFC format.

After I get the initial list from Gunnar I will send the document to the list and request additions. There may be a need to push this document to be an RFC in the future, for now we will treat it as a note.

Thanks everyone!

Natasha

Natasha Rooney | Technologist, Web and Internet, W3C & IETF | GSMA | [email-masked]<mailto:[email-masked]> | +44 (0) 7730 219 765 | @thisNatasha | Skype: [email-masked]<mailto:[email-masked]>
Tokyo, Japan

On Nov 8, 2015, at 2:14 PM, Gunnar Hellström <[email-masked]<mailto:[email-masked]>> wrote:

Which draft? I thought we are speaking about
https://tools.ietf.org/html/draft-ietf-slim-negotiating-human-language-00
[GH]Yes, sorry, I meant 6.2.  There is also an expansion of the topic in 6.3.

"

Is it something like:

en-US, en-AU, fr-CA, *

Or is it something like

en-US*, en-AU*, fr-CA*

The later would be totally out of line with previous use of language tags and subtags, but it's what I'd guess from a direct reading of the text.
[GH] Yes, that was also my interpretation. And wondering what was meant. A complete syntax description and example is needed.
But also a discussion if this non-fail parameter should be merged with the other priority values that I propose.

/Gunnar

Regards,   Martin.

_______________________________________________
SLIM mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/slim

--
-----------------------------------------
Gunnar Hellström
Omnitor
[email-masked]<mailto:[email-masked]>
+46 708 204 288

This email and its attachments are intended for the above named only and may be confidential. If they have come to you in error you must take no action based on them, nor must you copy or show them to anyone; please reply to this email or call +44 207 356 0600 and highlight the error.

---

From: Gunnar Hellström <[email-masked]>
Date: Wed, 11 Nov 2015 09:03:30 +0100

Hi Natasha and all,
Thanks for the offer, I will collect the use case sources for you.
There were some in the earlier draft version also.
I don't remember why they were deleted. Maybe too much thoughts on the 
resulting routing??
Or were they far fetched?

They could also be recovered to the collection you compose.

Gunnar

Den 2015-11-11 kl. 07:54, skrev Natasha Rooney:
> Hey guys,
>
> It would be a good idea to capture the use case identified by Gunnar 
> at the meeting at IETF94 and given in this thread in a document, this 
> will allow us to be able to refer to them in one place rather than 
> referring down a thread.
>
> Gunnar - I can build this document, to get started can you just send a 
> me your list of use cases with any amendments which may have been 
> identified in this email thread? I will then add them to a document 
> which will be in an RFC format.
>
> After I get the initial list from Gunnar I will send the document to 
> the list and request additions. There may be a need to push this 
> document to be an RFC in the future, for now we will treat it as a note.
>
> Thanks everyone!
>
> Natasha
>
>
> Natasha Rooney | Technologist, Web and Internet, W3C & IETF | GSMA | 
> [email-masked] <mailto:[email-masked]> | +44 (0) 7730 219 765 | 
> @thisNatasha | Skype: [email-masked] <mailto:[email-masked]>
> Tokyo, Japan
>
>
>> On Nov 8, 2015, at 2:14 PM, Gunnar Hellström 
>> [GH] Yes, that was also my interpretation. And wondering what was 
>> meant. A complete syntax description and example is needed.
>> But also a discussion if this non-fail parameter should be merged 
>> with the other priority values that I propose.
>>
>> /Gunnar
>>>
>>> Regards,   Martin.
>>>
> This email and its attachments are intended for the above named only 
> and may be confidential. If they have come to you in error you must 
> take no action based on them, nor must you copy or show them to 
> anyone; please reply to this email or call +44 207 356 0600 and 
> highlight the error.
>

---

From: Natasha Rooney <[email-masked]>
Date: Wed, 18 Nov 2015 10:44:12 +0000

Hey Gunnar!

On Nov 11, 2015, at 5:03 PM, Gunnar Hellström <[email-masked]<mailto:[email-masked]>> wrote:

Hi Natasha and all,

Many thanks! Just send them over whenever you have them. I will setup the repo early next week to create the draft.

There were some in the earlier draft version also.
I don't remember why they were deleted. Maybe too much thoughts on the resulting routing??

I’m afraid I am not sure about this - Guys - any reason why Gunnar’s use cases were taken from the earlier draft? In any case we will feed them into this new draft, so they’ll have a proper home!

Gunnar

Thanks everyone!

Natasha

"

Is it something like:

en-US, en-AU, fr-CA, *

Or is it something like

en-US*, en-AU*, fr-CA*

/Gunnar

Regards,   Martin.

---

From: Gunnar Hellström <[email-masked]>
Date: Wed, 18 Nov 2015 13:04:38 +0100

Hi,
About use cases:

Den 2015-11-18 kl. 11:44, skrev Natasha Rooney:
> I’m afraid I am not sure about this - Guys - any reason why Gunnar’s 
> use cases were taken from the earlier draft? In any case we will feed 
> them into this new draft, so they’ll have a proper home!
They were not especially "mine". I might have contributed with some, but 
the majority came from other sources.

Gunnar

---

From: Natasha Rooney <[email-masked]>
Date: Thu, 26 Nov 2015 07:31:35 +0000

Hi all!

I have started collecting use cases within a draft format which can be found here (click on "Editor's copy" to view):
https://github.com/ietf-slim-wg/slim-use-cases/

Currently this is just a collections of Gunnar’s original use cases, I am sure I still have some to add from the rest of the mail chains. I will go through these for the rest of this week and the next week to find some more. I also realise that the "Contents" are rendering at the bottom of the page, I will ask mt as to why that might be happening, because I am 85% sure it’s not my beautiful (:/) xml!

Barry - should this be labeled as an "IETF-Note"? Obviously it’s not a draft so I assume so but thought I would check!
All - please submit any other use cases either on this list or send a PR to the repo.
mt - if you know why the table of contents is appearing at the bottom please do let me know! Despite what I said above I am actually pretty sire it’s my error!

Thanks all!

Natasha

On Nov 18, 2015, at 9:04 PM, Gunnar Hellström <[email-masked]<mailto:[email-masked]>> wrote:

Hi,
About use cases:

I’m afraid I am not sure about this - Guys - any reason why Gunnar’s use cases were taken from the earlier draft? In any case we will feed them into this new draft, so they’ll have a proper home!
They were not especially "mine". I might have contributed with some, but the majority came from other sources.

Gunnar