Subject: [Speechsc] other timer-related issues

From: "Dan Burnett" <[email-masked]>
Date: Tue, 8 Nov 2005 19:18:41 -0500

Group,

I had several other thoughts/comments not directly related to the
recognition-timeout issue.

1) The definitions of "complete match", "full match", and "partial
   match" should be clearer, eg.,
   - Full match = Complete match = "a sequence of recognized tokens that
constitutes a
     valid complete utterance according to the grammar"
   - N-token prefix = "the first N tokens of a complete match"
   - Partial match = "any N-token prefix"

   In addition, to simplify the timer text we could add the notion of
   - Maximal match = "a complete match that is not a partial match of
any
   other complete match"

2) As I was writing out the analysis in my earlier email (while
re-reading the draft), it was not immediately
   apparent in the draft what the behavior was in all cases, hence the
   "??" in several places.  In a number of these cases one can
implicitly
   assume the behavior to result in a "000 success" or "001 no-match",
   but I couldn't find it explicitly stated.  Did I miss this, or do
   we need to add specific text?

3) It was also not immediately apparent that all the ways of
   successfully navigating past the speech-complete-timeout and
   speech-incomplete-timeout timers are addressed.  I don't believe
   there are any restrictions on the order in which these two timers
   can expire, so what about the following cases (where a certain
   situation exists when the first timer expires and another exists
   when the second expires)?

SIT           -> (then)         SCT
no partial match                (only possibility is no partial match)
-- handled
complete (no other partial)     (only possibility is complete) --
handled

SCT           -> (then)         SIT
partial match                   complete -- not handled
non-maximal complete match      nomatch -- not handled?
non-maximal complete match      complete -- not handled

Thoughts, anyone?

Interestingly, 9.4.16 says that the speech-incomplete-timeout is
usually longer than the speech-complete-timeout, which would lead to
the second group of cases here (SIT after SCT).

Again, if I've missed something, I'm sure someone will graciously
correct me :)

-- dan

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: Claudia Daboul <[email-masked]>
Date: Wed, 09 Nov 2005 22:25:32 +0100

Maybe it's useful to look at some examples to clarify the meaning of the 
various recognizer timeouts and events.

I will explain my understanding of the terms that where discussed lately 
on some examples and we can see if everybody agrees at least on these 
examples.

Consider a grammar that allows ordering drinks from a menu. It would 
allow utterances like: "A beer, please" or "Two coffee, a beer and two 
coke and an apple juice."

Consider the following sample utterances:

1) "Two coffee and a beer."
Complete, but not maximal match. (I use the term "maximal match" 
introduced by Dan.)
These utterances would result in "000 success" after
"speech-incomplete-timeout" has expired.

2) "Two coffee and a beer, please".
A maximal match, would result in "000 success" after
"speech-complete-timeout" has expired
(the reaction would usually be a bit faster than in case 1, since 
"speech-complete-timeout < speech-incomplete-timeout")

3) "Two coffee, a beer."
A partial, but not complete match,
would result in "013 partial-match" after "speech-incomplete-timeout"
has expired.

Note that some recognizers allow setting different timeouts for cases 1 
and 3. It would be something like "speech-maybecomplete-timeout", such that:
speech-complete-timeout < speech-maybecomplete-timeout < 
speech-incomplete-timeout
This intermediate timeout would then apply in case 1.

4) "Two coffee, a squishy and a beer."
A no-match, because the word "squishy" is not in the grammar. This would 
usually result in a "001 nomatch" immediately after the end of the word 
"squishy" is detected. The recognizer would not have to wait for the 
"speech-incomplete-timeout" to fire.
But this case depends on the implementation. The above behaviour could 
only be achieved, when speech is processed by the recognizer at 
realtime. Some implementations separate endpointing and speech 
recognition. Then utterances are only processed by the recognizer after 
the endpointing process has determined a complete utterance. In that 
setup the nomatch-event would come only after the word "beer" plus the 
"speech-incomplete-timeout" and the "speech-complete-timeout" and 
"speech-incomplete-timeout" could not be distinguished.

I'm still not completely sure about the behaviour, when the utterance is 
cut off by  the "recognition-timeout". For example consider the case 
where an utterance is cut off in the middle of the last word like this:

5) "Two coffee, a beer, a coke and two tea and a glass of wine and an oran"

What should be the return code in this case: "001 nomatch", "008
success-maxtime" or "013 partial-match"?

Would it make a difference if the utterance was cut off a bit earlier as in
5') "Two coffee, a beer, a coke and two tea and a glass of wine and an"
or
5'') "Two coffee, a beer, a coke and two tea and a glass of wine"
?
I think it should be "008" in all these cases.

Andrew's suggestion I interpreted as follows: The cut off utterance
6) "Two coffee, a squishy, a coke and two tea and a glass of wine and an"
which contains the unknown word "squishy" should return "003" instead of 
the usual "001". Did I get this right, Andrew?
Again this would only apply in the setup where endpointing must be 
completed before the actual recognition starts.

An application could distinguish this case from the usual no-match and 
have different prompts like:
A) "I didn't completey understand your order. Please make sure to choose 
only items from our menu." (for 001)
B) "Your order is getting too long and I didn't understand everything 
you said. Let us go through your order item by item and please make sure 
to order only items from our menu. What was the first item again?" (for 003)
I'm not sure, if it's really necessary to have this distinction, though. 
I think it will rarely be used by applications.

Anyway I hope these examples can help a bit to clarify the terminology. 
Does anybody have examples for the timeouts in hotword-mode?

Claudia

---

From: Dan Burnett <[email-masked]>
Date: Thu, 10 Nov 2005 08:27:41 -0800 (PST)

Someone pointed out to me offline that I left
something out of the definition of "partial match".  I
had originally described the prefix as being a proper
prefix (i.e. not including the entire complete match
token string) but somehow dropped it.

The definition of a partial match should be "any
N-token prefix that is not itself a complete match".

-- dan

> Again, if I've missed something, I'm sure someone

		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

---

From: Andrew Wahbe <[email-masked]>
Date: Thu, 10 Nov 2005 12:42:35 -0500

I think I can clarify when I think 003 would be returned. The most 
obvious example I can think of is actually from experience -- the caller 
didn't hang up the phone properly and put it down next to their TV. We 
just happened to be recording the utterances for tuning. What we got was 
a series of maxspeech timeouts (in VoiceXML speak) where each utterance 
was just noise (that included bits of speech). This would clearly result 
in an MRCP Recognition completion cause of 003. For other cases I don't 
think its as clear cut (according to the current spec -- even with my 
proposed modification).

However, in my opinion (after rethinking the issues a little),  if the 
recognition-timeout fires either 003 or 008 is returned. 008 is only 
returned if the utterance is a complete match and, more importantly, 
there is a valid result to be processed. For me that is the key 
question:  Is there, or is there not a valid result that the app can 
process?

If there is, the result is 008; otherwise, it's 003. Determining where 
the timer fired in the utterance and if the user was cut off is not 
really important. Additionally, I don't think its that reliable. In 
reality, one frequently gets a low confidence match rather than a 
nomatch from the recognizer. If you aren't getting a clear distinction 
between matches and no-matches, throwing partial-matches into the mix 
doesn't seem worthwhile.

Also, what are you going to do when you have a partial match? Can you 
process an attached result? I don't think so -- remember that the 
semantic interpretation rules need to run on a matched token sequence. 
It is quite likely that the logic in those rules assume full matches. If 
you run the rules on a partial match you run the risk of generating some 
weird semantic interpretation result that can't be processed. So I 
really question the usefulness of the partial-match result -- perhaps 
its worth changing this back to no-match (as it was in the 06 draft). I 
can't recall the reason for introducing partial-match either; anyone 
care to clarify?

Anyways, in my opinion the recognition timeout examples you gave should, 
strictly speaking have the following completion cause codes:

5) 003
5') 003
5'') 008
6) 003
This is because 5'' is the only complete match.

However, practically speaking you will probably get 008 for all versions 
of 5) since its quite likely that either "Two coffee, a beer, a coke and 
two tea and a glass of wine and an orange" or "Two coffee, a beer, a 
coke and two tea and a glass of wine" is matched with a reasonable 
confidence level. For 6) you might still get 003 as long as "squishy" 
wasn't mistaken for something else in the grammar.

For your examples 1-4, I agree with your analysis (though I again 
question the usefulness of partial match -- I think that strictly 
speaking 3 should be a no-match, but practically speaking you would 
probably get a match for something like "Two coffee, and a beer" with 
perhaps a lower but reasonable confidence).

Another comment: I don't think I've ever seen a recognizer return in the 
middle of an utterance because it hit a token that didn't match the 
grammar. ie. if you said "Two coffee, a squishy and a beer." and squishy 
wasn't in the grammar, the system would still wait until you finished 
speaking. This could just be because this token is matching something 
else in the grammar with a low confidence or the endpointing is taking 
place before hand as you mentioned. I'm not sure... and it probably 
depends on the recognizer you are using.

I would find that behavior a bit strange though... wouldn't you want to 
always try and wait for the user to finish speaking? Otherwise, wouldn't 
you be sentencing yourself to a speech-too-early result after each 
no-match result? Anyways, its not a big deal -- I'm not proposing we 
change the definitions of no-match or the timeouts in this regard. I 
think it does reinforce my point that focusing on complete match vs 
partial match vs no match of the token string is perhaps the wrong 
approach when generating the completion-cause code. I think its better 
to discern: I have a result for you to process vs I don't have a valid 
result.

IMO, this line of reasoning gives you
no-input (no input at all)
no-match (I got input but no valid result)
match (I got input and a valid result)
recognition-timeout (I got input, but it was too long and I didn't get a 
valid result)
success-maxtime (I got input, and it was too long but what I did get 
seemed to be a valid result).
(along with speech-too-early and all the error results).

I think these are basically the codes in the 06 draft with the 
distinction between 003 and 008 (the original concern that precipitated 
the changes) clarified. This is just what makes sense to me, at least 
for normal recognition mode. I'd need to think about hotword recognition 
some more before commenting.

Finally on your comment of the usefulness of discerning 001 from 003: I 
can understand your point of view, but the fact of the matter is that a 
VoiceXML browser requires that this distinction be made. I think it 
makes a lot of sense to cater to this need since VoiceXML browsers will 
probably one of the most (if not the most) common forms of MRCP v2 client.

Andrew

Claudia Daboul wrote:

> Maybe it's useful to look at some examples to clarify the meaning of 
> the various recognizer timeouts and events.
>
> I will explain my understanding of the terms that where discussed 
> lately on some examples and we can see if everybody agrees at least on 
> these examples.
>
> Note that some recognizers allow setting different timeouts for cases 
> 1 and 3. It would be something like "speech-maybecomplete-timeout", 
> such that:
> A no-match, because the word "squishy" is not in the grammar. This 
> would usually result in a "001 nomatch" immediately after the end of 
> the word "squishy" is detected. The recognizer would not have to wait 
> for the "speech-incomplete-timeout" to fire.
> recognition. Then utterances are only processed by the recognizer 
> after the endpointing process has determined a complete utterance. In 
> that setup the nomatch-event would come only after the word "beer" 
> plus the "speech-incomplete-timeout" and the "speech-complete-timeout" 
> and "speech-incomplete-timeout" could not be distinguished.
>
> I'm still not completely sure about the behaviour, when the utterance 
> is cut off by  the "recognition-timeout". For example consider the 
> case where an utterance is cut off in the middle of the last word like 
> this:
>
> which contains the unknown word "squishy" should return "003" instead 
> of the usual "001". Did I get this right, Andrew?
> A) "I didn't completey understand your order. Please make sure to 
> choose only items from our menu." (for 001)
> you said. Let us go through your order item by item and please make 
> sure to order only items from our menu. What was the first item 
> again?" (for 003)
> I'm not sure, if it's really necessary to have this distinction, 
> though. I think it will rarely be used by applications.
>
> Anyway I hope these examples can help a bit to clarify the 
> terminology. Does anybody have examples for the timeouts in hotword-mode?
>
> Claudia
>
>
>
>
> Dan Burnett wrote:
>
>> Group,
>>

---

From: Dan Burnett <[email-masked]>
Date: Thu, 10 Nov 2005 11:00:50 -0800 (PST)

I agree with Andrew's observations, statements and
analysis below, and I believe the conditions (and
their definitions) are what was originally intended
before the concept of partial matching entered the
document.

I would prefer to have the return conditions as you've
specified them.

I believe they would work as follows for hotword:
no-input (no input at all)
match (I got input and a valid result)
success-maxtime (I got input, and it was too long but
what I did get seemed to be a valid result).

The no-match and recognition-timeout cases don't
matter for hotword (at least for VoiceXML).

-- dan

--- Andrew Wahbe <[email-masked]> wrote:

...
> some more before commenting.
...