Subject: [Sipping] Application Interaction Requirements

From: "Fairlie-Cuninghame, Robert" <[email-masked]>
Date: Fri, 27 Sep 2002 03:05:37 -0700

Howdy all,

We are currently revisiting some of the new application interaction
requirements that have suggested since the requirements document was last
published.
(http://www.ietf.org/internet-drafts/draft-culpepper-sipping-app-interact-re
qs-01.txt).

Back in July, Eric Cheung of AT&T suggested a number of additions:
[
http://www.ietf.org/mail-archive/working-groups/sipping/current/msg02338.htm
l ]

There are two requirements that we feel need further discussion as they are
very different to the existing requirements. [I'm sorry I can't seem to
locate the original email of the first requirement.]

Both of the requirements deal with media stream synchronization.

R1:	For user activity indications, the framework must provide some form
of indication of occurrence in time relative to one or more media streams.

I don't believe that this should be a requirement - IF you need tight media
synchronization then transporting stimulus indications through "out-of-band"
mechanisms is not the correct solution for your problem. You should, for
instance, be using forked RFC2833 which has been around for a long time.
Some of the problems: for starters there may not always be RTP media or it
may be coming from different sources - this makes RTP timestamp inclusion
unworkable. Secondly, the non-real-time transport of stimulus indication
over SIP or HTTP makes real-time synchronization impractical at any rate.
Each stimulus could have an absolute wall-clock Date Time value in it,
however, SIP as a whole seems to be straying from the requirement that all
clients must implement absolute date time and thus we also feel reluctant to
impose this requirement. Relative time between stimulus indications is fine
but not absolute time.

R2:	For key-based indications, the framework must provide the ability
for the network element requesting the indications to also request that the
indications be suppressed from delivery to [all] other end points on the
media path.

There is first the conceptual question: should an application be able to
modify the media streams without being a B2BUA. Secondly you quoted a usage
case such as "* Transfer". The problems I see here is that you need to know
what numbers the user will pressed before they actually press them in order
to squelch them. If you simply want you application to stop all DTMF digits
from being passed through (such as a conference server) then is should not
have added RFC2833 to start with. 

So, I am inclined to say that these requirements are out-of-scope for this
framework but we wish to open the issue to further discussion.

Thanks,

Robert.
_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "David R. Oran" <[email-masked]>
Date: Fri, 27 Sep 2002 08:57:45 -0400

See embeddeed comments.

--On Friday, September 27, 2002 3:05 AM -0700 "Fairlie-Cuninghame, Robert" 
<[email-masked]> wrote:

> (http://www.ietf.org/internet-drafts/draft-culpepper-sipping-app-interact
> -re qs-01.txt).
>
> http://www.ietf.org/mail-archive/working-groups/sipping/current/msg02338.
> htm l ]
>
> There are two requirements that we feel need further discussion as they
> are very different to the existing requirements. [I'm sorry I can't seem
> to locate the original email of the first requirement.]
>
> I don't believe that this should be a requirement - IF you need tight
> media synchronization then transporting stimulus indications through
> "out-of-band" mechanisms is not the correct solution for your problem.

Uh, where do you leap from "indication of time relative to the media 
stream" to "tight media synchronization". There are lots of situations 
which require the former but not the latter. Examples:
- barge-in control in ASR systems
- stream markers in TTS and other playout situations
- speaker change annotations in recording systems

> You should, for instance, be using forked RFC2833 which has been around
> for a long time. Some of the problems: for starters there may not always
> be RTP media or it may be coming from different sources - this makes RTP
> timestamp inclusion unworkable. Secondly, the non-real-time transport of
> stimulus indication over SIP or HTTP makes real-time synchronization
> impractical at any rate. Each stimulus could have an absolute wall-clock
> Date Time value in it, however, SIP as a whole seems to be straying from
> the requirement that all clients must implement absolute date time and
> thus we also feel reluctant to impose this requirement. Relative time
> between stimulus indications is fine but not absolute time.
>
All this is true but IMHO completely misses the point.

> for the network element requesting the indications to also request that
> the indications be suppressed from delivery to [all] other end points on
> the media path.
>
I frankly don't understand what this means. For one thing, it seems 
inadvisable for one control element to be mandating what other endpoints 
see or don't see. If there is a requirement lurking here, it's for each 
participant to control what they see or don't see of the control stream as 
well as the media stream(s).

> modify the media streams without being a B2BUA.
I'm not sure what this means, because I could modify a stream and then 
re-originate it with a different control protocol other than SIP, in shich 
case I'm modifyng the media stream but I'm not a B2BUA. The important 
property is that an application not be permitted to modify a stream without 
being a UA. Otherwise it will run afoul of OPES sooner or later.

> Secondly you quoted a
> usage case such as "* Transfer". The problems I see here is that you need
> to know what numbers the user will pressed before they actually press
> them in order to squelch them.
Huh? There are lots of ways to eliminate control signals from the media 
path, assuming you know what's going on. I even have a patent on one 
technique that avoids introducing extra delay into the media path.

> If you simply want you application to stop
> all DTMF digits from being passed through (such as a conference server)
> then is should not have added RFC2833 to start with.
>
Since I frankly don't understand anything about the second requirement, I 
have no opinion (other than it had better get restated so we can understand 
it). My opinion on the first requirement is that we definitely need this 
(speechsc depends on it), but not with anything like the interpretation you 
have.

Cheers, Dave.

> Robert.

------------------------
David R. Oran
Cisco Systems
7 Ladyslipper Lane
Acton, MA 01720
Office: +1 978 264 2048
VoIP: +1 408 571 4576
Email: [email-masked]

---

From: Eric Cheung <[email-masked]>
Date: Fri, 27 Sep 2002 10:08:19 -0400

Hi Robert

Re your comments on the first requirement (R1 - indication of occurrence
in time relative to one or more media streams)

1) I have included the original posting below.  I think a good example
where this is needed is the multi-modal system I mentioned.
2) I believe this I-D covers non-DTMF user indications, therefore
RFC2833 is not relevant in any case.
3) The problems you raised seem to be solvable:
   - if there is no media, then this indication of reative time to media
streams is not relevant i..e can be left blank.
   - the use of SSRC (and maybe other information to uniquely identify a
stream) and the timestamp should eliminate the need for absolute
wall-clock Date Time in all entities.
   - the fact that SIP or HTTP messages are non-real-time is not a
problem for correlation subsequently.  As a matter of fact, it is
precisely because of the timing difference between these messages and
the RTP packets that make this a requirement.

Thanks
Eric 

Here is the original email from May 15 re the first requirement:

http://www1.ietf.org/mail-archive/working-groups/sipping/current/msg01874.html

Hi Bert, Robert

It seems to me that an important requirement would be that user inputs
and media stream(s) can be later correlated in time.  For example, a
voice mail system need to correlate a key press and the audio stream to
determine the end of the recording.  Another example is a multi-modal
system where user input includes clicking on a web page, pressing a key
on a phone, and speech via automatic speech recognition.  Again, the
click or key press must be correlated to the audio stream.

Therefore, I would suggest adding a requirement such as:

  For user activity indications, the mechanism must provide some form of
indication of occurrence in time relative to one or more media streams.

As an example of a possible solution, this may be the SSRC of a media
stream the reporter is transmitting and the timestamp when the input
occurred.

Regards
Eric Cheung

---

From: Eric Cheung <[email-masked]>
Date: Fri, 27 Sep 2002 13:20:10 -0400

Regarding R2 (suppression from media path):

An example is shown below:

                   /----- AS ---\             --- SIP signalling
                  /              \            === RTP (audio)
PSTN phone +++++ GW ============ SIP Phone    +++ PSTN

 User 1                            User 2

Let's say User 1 and User 2 are talking.  User 1 wishes to invoke a
"Transfer-Connect" service that resides on AS by pressing *T, then a
numeric address.

The requirement on the service is that User 2 should not hear any of
these DTMF tones.  On the other hand, any other DTMF tones should pass
through, because User 1 could well be accessing a voice mail system
controlled by DTMF.

Mapping this requirement to the App interaction requirement I-D, AS is
the network element requesting the indications *T followed by digits. 
AS also need to ask GW to remove these tones (either in the form of
RFC2833 packets or encoded as audio) from the audio stream sent to SIP
Phone, such that User 2 will not hear them.

Admittedly this may be hard to do (e.g. what if user presses *, pauses,
then 1?  Do you suppress the * when it is received?)  But it seems we
should agree whether this is a necessary requirement before thinking
about solutions.

Also note that AS is usually a B2BUA, so it does not really step on
Robert's conceptual question.

Thanks
Eric

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sat, 28 Sep 2002 12:03:58 -0400

David R. Oran wrote:

> Since I frankly don't understand anything about the second requirement, 
> I have no opinion (other than it had better get restated so we can 
> understand it). My opinion on the first requirement is that we 
> definitely need this (speechsc depends on it), but not with anything 
> like the interpretation you have.

Dave, can you elaborate on the specific need for speechsc? I am confused 
about this requirement too. My interpretation was similar to Robert's; 
if you need media synchronization, use rfc2833. Why is this wrong?

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                72 Eagle Rock Ave.
Chief Scientist                             First Floor
dynamicsoft                                 East Hanover, NJ 07936
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: "David R. Oran" <[email-masked]>
Date: Sun, 29 Sep 2002 07:40:51 -0400

--On Saturday, September 28, 2002 12:03 PM -0400 Jonathan Rosenberg 
<[email-masked]> wrote:

> about this requirement too. My interpretation was similar to Robert's; if
> you need media synchronization, use rfc2833. Why is this wrong?
>
It's not wrong, in fact it works best if you use RFC2833 for tone-based 
user interfaces. All I'm saying is that the application framework needs a 
way to obtain timing information from the media stream, and to be able to 
provide timing information to the playout control of the media stream. This 
basically boils down to being able to use RTP timestamps to corellate 
control actions with media processing.

SpeechSc has at least two specific requirements:
a) handle barge-in for ASR. The scenario is that something is being played 
out to the user (e.g. listening to internet radio, TTS output from an 
application, hearing a voicemail message), and the user says "stop playing 
this". What you want the system to do is immediately squelch playout, 
including dumping the  contents of the jitter buffer and discarding any 
packets in flight when they arrive, and then be able to play "output 
stopped, next command?" to the user. You can't get a way with just having 
the application tell the client to stop playout, because then the delays 
get to be multiple round trips to get output started again. What you really 
want is the ability for the applciation to say "don't play out anything 
lower than timestamp x"; then the app can just ensure its next media output 
has timestamp x+epsilon. For this to work the application framework needs 
the ability to discover media timestamps and compose commands which take 
these as data. Also needed are other elements present in or derivable from 
from RTP, such as SSRCs.

(Aside: my prefered machinery for barge-in control is RTCP, but this is 
under discussion in Speechsc and we're not there yet since we're still 
finishing up the requirements draft. For those interested, you can skim 
draft-ietf-speechsc-reqts-00.txt).

b) VCR control. You want the application to be able to say "advance 5 
seconds" and have that translated to the equivalent RTP timestamp domain so 
the client can move around in a potentially large jitter buffer.

This help?
Dave.

P.S. I agree with all your comments about focus. From a protocol standpoint 
however, it is not clear if the right thing to do is to do the real-time 
event arbitration at the client, or to send the stimulus to all 
applications and use some meta-control scheme to do the focus control and 
event filtering at the server end. For GUI or data-based stimulus events, 
doing everyting at the client is probably a lot cleaner, but for events in 
the media stream (e.g. tones inband or in RFC2833) the latter may work much 
better. The reason you really have to remove redundancy from the event 
stream in the case of media events is for the case where the application is 
re-constructing the event stream and passing it on as opposed to consuming 
it. Not properly removing redundancy results is horrible things like 
double-detection  of DTMF (like adding zeros on the end of the amount of 
money you deposit in my account :-) ).

------------------------
David R. Oran
Cisco Systems
7 Ladyslipper Lane
Acton, MA 01720
Office: +1 978 264 2048
VoIP: +1 408 571 4576
Email: [email-masked]