Subject: RE: [speechsc] VoiceXML maxspeech timeout not implementable withMRCPv2

From: "Shanmugham, Saravanan" <[email-masked]>
Date: Thu, 10 Nov 2005 14:50:29 -0800

Ok. 
Here is proposal for change, that should clarify this.

I will fold the description I mentioned earlier into the RECOGNIZE
definition section of the draft.

I will then add the following 2 cause-codes
   partial-match-maxtime (014)
   no-match-maxtime (015)
   Recognition-timeout - renamed to "hotword-maxtime" for the expiry of
Recognition-Timer in the Hotword mode recognition.

That also gets us away from finishing the list of cause codes list with
the unlucky 13 :-))

We then add the following clarifications for non-hotword recognition
mode.
   1. The recongizer MUST support detecting no-match condition upon
detecting end of speech.  The recognizer MAY support detecting no-match
condition  before waiting for end-ofspeech. If this is supported, this
capability is enabled by setting "Early-NoMatch" header to "true". -
upon detecting a no match condition the RECOGNIZE MUST return with
"no-match"
   2. When the Speech-Incomplete-Timer expires - SHOULD return
"partial-match" unless the recognizer cannot differentiate a
partial-match in which case it MUST return "no-match". The recognizer
MAY return results for the partially matched grammar.
   3. When the Speech-Complete-Timer expires - MUST return "success"
   4. When the Recognition-Timer expires
      4.1 And there is a partial-match  -SHOULD return
"partial-match-maxtime" unless the recognizer cannot differentiate a
partial-match in which case it MUST return "no-match-maxtime". The
recognizer MAY return results for the partially matched grammar.
      4.2 And there is full-match -  MUST return "success-maxtime"
      4.2 And there is no match - MUST return "no-match-maxtime"

For the Hotword mode recognition.
     1. The Recognition-Timer gets started at the beginning of
RECOGNIZE.
     2. When there is match at anytime, the RECOGNIZE completes with
"success".
     3. When the Recognition-Timer expires and there is not match, the
RECOGNIZE MUST complete with "hotword-maxtime"
     4. When the Recognition-Timer expires and there is a match, the
RECOGNIZE MUST return "success".

Thx,
Sarvi, Dan Burnett

     -----Original Message-----
     From: Andrew Wahbe [mailto:[email-masked]] 
     Sent: Thursday, November 10, 2005 1:26 PM
     To: Shanmugham, Saravanan
     Cc: Dave Burke; IETF SPEECHSC (E-mail)
     Subject: Re: [speechsc] VoiceXML maxspeech timeout not 
     implementable withMRCPv2
     
     I don't think we are all that far off, the devil is in the details.
     
     First, VoiceXML requires that your "no-match-maxtime" be 
     distinguished from no-match. So if we want all the 
     VoiceXML browsers out there to use MRCP then we need to 
     make the distinction.
     
     Second, we already have no-match-maxtime: its 003 
     recognition-timeout. 
     Except that it is restricted to hotword recognition in the 
     current draft. Since the client knows the recognition 
     mode, I propose that we re-use the code for both modes. So 
     no-match-maxtime is the same as 003 recognition-timeout. 
     We don't need two codes to tell the client something it 
     already knows.
     
     I think the point that I was trying to make at the start 
     of all this is captured above.
     
     I additionally pointed out in my last email on the topic 
     that reliably discerning no-match from partial-match from 
     complete-match in the return code (regardless of the timer 
     that fired) may not be feasible nor terribly useful. What 
     is important is if we have a result or not -- this is 
     effectively match vs no-match but there is a subtle 
     difference. The semantics here revolve around the presence 
     of a usable recognition result rather than which timer 
     fired. If we collapse the partial match codes in the the 
     respective no-match codes then we end up with the codes 
     looking how they did in the 06 draft -- except that we 
     have now clarified the difference between 003 and 008.
     
     I would agree though that for "completeness", if we don't 
     get rid of partial-match, then it may make sense to have 
     something like partial-match-maxtime. But again, I don't 
     think that its really worth it to have any partial-match 
     codes at all.
     
     If we don't remove the partial-match codes and base the 
     match vs no-match distinction on the presence of a result 
     (rather than timers) then I would like clarification as to 
     how confidence-threshold works into all this. If the 
     complete-timeout fires, but the resulting match is below 
     the confidence-threshold, then do you get a "match" result 
     but no recognition results? That seems broken. Section 9.4 
     (in the definition of confidence-threshold) it actually 
     says that a no-match must be returned in that case. So 
     something else would have to be changed here.
     
     Andrew

     Shanmugham, Saravanan wrote:
     
     >I just re-read this whole description and came to a different
     >conlcusion.
     >
     >When a recongizer is started and the speaker starts speaking.
     >
     >The match conditions for the spoken speech and grammar 
     matching upto
     >that point can be one of the following
     >   no-match
     >   partial-match
     >   complete-match
     >You could go one step further and say a complete-match can be of 2
     >sub-classes on further grammar possibilities(Is there 
     value to doing
     >this. My thinking is not, Any thoughts)
     >   complete-match-with-more-possibilities
     >   Complete-match-with-no-more-possibilities
     >
     >Theoretically speaking, no-match could be decided the 
     moment that the
     >speaker speaks something that does not match anything, 
     which could be
     >even the first word.
     >But, if I understand right, most commercial recognizers 
     today don't
     >match the speech as they are spoken, but rather wait for
     >end-pointing/some-silience or a timer before doing the matching.
     >
     >NOW, walking through what happens during a non-hotword RECOGNIZE
     >operation.
     >
     >The Recognition-Timer is started right at the beginning 
     of the RECOGNIZE
     >operation.
     >
     >When such a silence happens they take the speech 
     collected upto that
     >point, and match it with the grammar and do the following.
     >   1. If there was NO match - they generate   "no-match 
     001" completion
     >cause code.
     >   2. If there was partial-match - They start the
     >Speech-Incomplete-Timer
     >   3. If there was complete-match(both sub types) - They start the
     >Speech-Complete-Timer.
     >
     >Now lets look at the cases of the different timers expiring.
     >
     >Speech-Incomplete-Timer expires, we return partial-match 013
     >When Speech-Complete-Timer expires, we return "success" 000.
     >When Recognition-Timer expires we have multiple 
     possibilities based on
     >the grammar match condition.
     >    1. no-match 001
     >    2. partial-match 013
     >    3. succcess (which is differentiated as success-maxtime 008)
     >
     >So I don't see what we are missing here.
     >Now theoretically, we could create new completion cause code
     >no-match-max-time, and partial-match-maxtime to 
     differentiate them from
     >the ones that would have been generated during a 
     silence/endpointing.
     >
     >Now If I understand Andrew, he is refering to no-match-max-time as
     >Recognition-Timeout, which is fine by me, but question if 
     it is actually
     >necessary for the differeniation. Coz, I am thinking if we want to
     >different no-match-maxtime(andrew's proposed 
     recongition-timeout), and
     >success-maxtime(already defined), then we should also define
     >partial-match-maxtime, which has a higher probablity of 
     occuring than
     >success-maxtime :-))
     >
     >What do you think,
     >Sarvi 
     >
     >
     >     -----Original Message-----
     >     From: [email-masked] 
     >     [mailto:[email-masked]] On Behalf Of 
     >     Shanmugham, Saravanan
     >     Sent: Monday, November 07, 2005 10:54 AM
     >     To: Dave Burke; Andrew Wahbe; IETF SPEECHSC (E-mail)
     >     Subject: RE: [speechsc] VoiceXML maxspeech timeout not 
     >     implementable withMRCPv2
     >     
     >     I understand and see his point. I think what he is 
     >     proposing is what it used to be before it was changed 
     >     based on previous feed back. Hence I want to be carefull 
     >     before making this change. 
     >     
     >     I have this as the only Open Issue in my slides for the 
     >     SpeechSC meeting. If there is concerns raised this is a 
     >     simple enough change to get into -09.
     >     
     >     Sarvi 
     >      
     >     
     >     
     >          -----Original Message-----
     >          From: [email-masked] 
     >          [mailto:[email-masked]] On Behalf Of 
     Dave Burke
     >          Sent: Monday, November 07, 2005 9:17 AM
     >          To: Andrew Wahbe; IETF SPEECHSC (E-mail)
     >          Subject: Re: [speechsc] VoiceXML maxspeech timeout not 
     >          implementable withMRCPv2
     >          
     >          I agree with Andrew's analysis and that his 
     proposed text 
     >          clarification is warranted, i.e.
     >          
     >          "003 recognition-timeout: RECOGNIZE completed without a 
     >          match due to a recognition-timeout"
     >          
     >          Dave
     >          
     >          ----- Original Message -----
     >          From: "Andrew Wahbe" <[email-masked]>
     >          To: "IETF SPEECHSC (E-mail)" <[email-masked]>
     >          Sent: Friday, October 28, 2005 3:33 AM
     >          Subject: Re: [speechsc] VoiceXML maxspeech timeout not 
     >          implementable
     >          withMRCPv2
     >          
     >          
     >          > Man I should stop writing emails so late at night:
     >          > My new proposed text for 003 wasn't changed 
     at all. This 
     >          is what I am
     >          > proposing:
     >          >
     >          > 003 recognition-timeout: RECOGNIZE completed 
     without a 
     >          match due to a 
     >          > recognition-timeout
     >          >
     >          > Andrew
     >          >
     >          > Andrew Wahbe wrote:
     >          >
     >          >> I just noticed that my completion cause code 
     analysis 
     >          was wrong in 
     >          >> one
     >          >> case:
     >          >>
     >          >> - between drafts 06 and 07 it seems the definition 
     >     of Speech 
     >          >> Incomplete Timeout (Section 9.4.16) was 
     changed so that 
     >          >> "partial-match" (013) was returned when it fired 
     >          instead of "no-match"
     >          >> (001).
     >          >>
     >          >> So I guess what we are saying is that another way of 
     >          terminating 
     >          >> recognition is that the speech is just not 
     matching the 
     >          grammar at 
     >          >> all and that is what yeilds no-match (001). 
     (Though I 
     >          could have 
     >          >> sworn that a recognizer waits until you were done 
     >          talking even if the 
     >          >> utterance wasn't matching at all and that incomplete 
     >          timeout was used 
     >          >> in that case. Anyone care to clarify?)
     >          >>
     >          >> Anyways, I don't think this changes my real 
     point below.
     >          >>
     >          >> Andrew
     >          >>
     >          >> Andrew Wahbe wrote:
     >          >>
     >          >>> Actually, this is something I've raised before with 
     >          the 07 draft 
     >          >>> (see
     >          >>> 
     >          
     http://www1.ietf.org/mail-archive/web/speechsc/current/msg0
     >          1510.html
     >          >>> )
     >          >>> but is still not fixed in 08.
     >          >>>
     >          >>> I don't see a way to implement the VoiceXML 
     maxspeech 
     >          timeout with 
     >          >>> the current recognition completion cause codes. The 
     >          semantics of the 
     >          >>> Recognition Timer match those of VoiceXML maxspeech 
     >          timeout. There 
     >          >>> are currently 2 cause codes related to this timer:
     >          >>>
     >          >>> 003 recognition-timeout: RECOGNIZE in hotword mode 
     >          completed without 
     >          >>> a match due to a recognition-timeout
     >          >>>
     >          >>> 008 success-maxtime: RECOGNIZE request terminated 
     >          because speech was 
     >          >>> too long but whatever was spoken till that 
     point was a 
     >          full match.
     >          >>>
     >          >>> But what should be thrown in a non-hotword 
     recognition 
     >          when the 
     >          >>> timer fires and there is no match? It can't be 001 
     >          no-match; the 
     >          >>> VoiceXML interpreter will not be able to 
     distinguish a 
     >          "nomatch" 
     >          >>> from "maxspeech" in this case.
     >          >>>
     >          >>> Let's take a step back. If we really want 
     to describe 
     >          what happened, 
     >          >>> then I think we need to communicate to the client 
     >     1) why the 
     >          >>> recognition was stopped and 2) if the result a 
     >          no-match, a partial 
     >          >>> match, or a complete match when it stopped.
     >          >>>
     >          >>> 1) The recognition can stop for the 
     following reasons 
     >          (the related 
     >          >>> cause codes in parens):
     >          >>>
     >          >>> no-input timeout (002)
     >          >>> complete timeout (000)
     >          >>> incomplete timeout (001 013)
     >          >>> recognition timeout (003 and 008)
     >          >>> speech too early (007)
     >          >>> cancelled (011)
     >          >>> various errors (004, 005, 006, 009, 010, 012)
     >          >>>
     >          >>> 2) The result is irrelevant for no-input, speech 
     >     too early, 
     >          >>> cancelled, and the errors. This leaves complete 
     >          timeout, incomplete 
     >          >>> timeout, and recognition timeout. By the 
     definition of 
     >          complete 
     >          >>> timeout, this is always a complete match 
     (000). Also 
     >          by definition, 
     >          >>> incomplete timeout could result in a no 
     match (001) or 
     >          a partial 
     >          >>> match (013) -- though VoiceXML will likely throw 
     >          nomatch for either.
     >          >>>
     >          >>> So we are left with recognition timeout. It 
     could be a 
     >          complete 
     >          >>> match
     >          >>> (008 by the definition in section 9.4.11) 
     or a partial 
     >          match (008 by 
     >          >>> the definition in section 9.4.7), or a no-match...  
     >          well if it was a 
     >          >>> hotword recognition we have 003; for the 
     normal case, 
     >          we have nothing.
     >          >>>
     >          >>> So hopefully I have clarified my earlier point that 
     >          Recognition 
     >          >>> Complete Cause codes 003 and 008 need revisiting. 
     >          Sorry to be so 
     >          >>> long winded but I think it's important that 
     VoiceXML be 
     >          >>> implementable using MRCPv2; it is a very common use 
     >          case for the specification.
     >          >>>
     >          >>> Before proposing a solution, I would like 
     to again ask 
     >          why we need 
     >          >>> to have a special cause code for 
     recognition-timeout 
     >          in the hotword 
     >          >>> case. The answer to this question is here:
     >          >>> 
     >          
     http://www1.ietf.org/mail-archive/web/speechsc/current/msg0
     >          1501.html
     >          >>>
     >          >>> But I still don't see why you need a special 
     >          completion cause to say 
     >          >>> that the timer expired in hotword mode. The client 
     >          knows what mode 
     >          >>> the recognition is in. You just need to 
     tell it that 
     >          the timer fired.
     >          >>>
     >          >>> What the client does need to know is if there is a 
     >          valid result for 
     >          >>> it to process. Thus I propose that 003 be 
     >     rewritten as follows:
     >          >>>
     >          >>> 003 recognition-timeout: RECOGNIZE in hotword mode 
     >          completed without 
     >          >>> a match due to a recognition-timeout
     >          >>>
     >          >>> This will bring it back in line with the earlier 
     >          Scansoft & Nuance 
     >          >>> proposal (well now it just a Nuance 
     proposal) from a 
     >          year and a half
     >          >>> ago:
     http://www1.ietf.org/mail-archive/web/speechsc/current/msg0
     >          0560.html
     >          >>>
     >          >>> Andrew
     >          >>>
     >          >>> _______________________________________________
     >          >>> Speechsc mailing list
     >          >>> [email-masked]
     >          >>> https://www1.ietf.org/mailman/listinfo/speechsc
     >          >>>
     >          >>>
     >          >
     >          
     >          
     >          
     -----------------------------------------------------------
     >          ---------------------
     >          
     >          
     >          > _______________________________________________
     >          > Speechsc mailing list
     >          > [email-masked]
     >          > https://www1.ietf.org/mailman/listinfo/speechsc
     >          > 
     >          
     >          
     >          _______________________________________________
     >          Speechsc mailing list
     >          [email-masked]
     >          https://www1.ietf.org/mailman/listinfo/speechsc
     >          
     >     
     >     _______________________________________________
     >     Speechsc mailing list
     >     [email-masked]
     >     https://www1.ietf.org/mailman/listinfo/speechsc
     >     
     >
     >
     >  
     >

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: Claudia Daboul <[email-masked]>
Date: Mon, 14 Nov 2005 12:26:10 +0100

I agree with Andrew that partial results are problematic in particular 
with respect to semantic interpretation.
I still think it's good to leave the option to the recognizer to return 
recognition results even if there's no complete match.
As I understand the current specification, it is not ruled out to return 
results even in the nomatch (001) case, these might be partial results, 
complete results with a low confidence or other forms of incomplete results.
For example in the nomatch case from my example:
4) "Two coffee, a squishy and a beer."
It could either return a partial result:
A) Two coffee
Or it could return a complete result, but with a low confidence like:
B) "Two coffee, a pepsy and a beer." (conf = 30)
and the application could use the confidences of the individual items
coffee (conf=80), pepsy (conf=10), beer (conf=86)
Maybe it could even return something like
C)  "Two coffee, a <$unknown:<phonetic transcription>> and a beer."

I don't think it's necessary to introduce further return codes to cover 
the different kinds of results. It should rather become part of the 
standardisation of recognition results to cover also various forms of 
incomplete results.

I don't agree with Andrews interpretation in the case of the recognition 
timeout (5-5''). I think in this case it's purely accidental to get a 
complete match. Usually (in the generic case) the utterance would be cut 
off in the middle of a word. In an application I would only distinguish 
between the cases where the utterance, up to the cutoff, matched the 
grammar (partially) or where it didn't match. (I would only make this 
distinction, if the recognizer indeed would return partial results which 
the application could use.)
Also, as I said before, I would usually treat the no-match-maxtime case 
the same as the usual no-match, but as Andrew pointed out the 
distinction is made by VoiceXML.
Therefore my suggestion is to define the return codes as follows:
When "incomplete-timout" fired we may get:
-no-match (001)
-partial-match (013)
-success (000)(could come already after "complete-timeout" expired)
When "recognition-timeout" fired we may get:
-no-match-maxtime (003)
-partial-match-maxtime (008)
- no extra code for "success-maxtime", because success in this case 
would be accidental and should not be trusted. The case would be 
included in (008).
If you don't want to distinguish the "partial results"-case, just map 
013 to 001 and 008 to 003. Then you have exactly the completion events 
defined in Voice-XML.

I am content also with Sarvi's latest suggestion, since it is clearly 
specified when which case applies and I can still decide to treat (014) 
and (008) the same.

I also think the "Early-NoMatch" header is a good idea.

As I noted in my last mail some recognizers have different cases of the
"complete-timeout" corresponding to the cases:
    complete-match-with-more-possibilities
    Complete-match-with-no-more-possibilities
They are ordered such that complete-timeout-wnmp < complete-timeout-wmp 
< incomplete-timeout.
I propose to introduce this distinction. I think it's useful for tuning 
an applications reaction times.
There would be no extra return codes necessary. Both those timeouts 
should only apply, if the match was reached with a high enough 
confidence. If it would result in a confidence below 
confidence-threshold, the recognizer should wait in any case until the 
"incomplete-timeout" expires.

Claudia