There is no 1 to 1 match between error types on the developers’ and users’ sides. Users are the most frustrated when they don’t understand why there was an error. Intent misclassification is the biggest cause of these frustrating errors.
What did they do?
The Japanese researchers built a fake calendar skill for Alexa that had 3 pre-planned errors. They wanted to see how users thought Alexa made errors and compare those assessments to how developers understood the error. They recorded 30 peoples’ interactions with this calendar skill using specific tasks. They then asked participants to rate their frustration for each error.
What did they find?
The most frustrating error type, and their level of frustration, was as follows:
- U1 Reason Unknown (68.3%)
- U3 Utterance Pattern Match Failure (56.7%)
- U2 Speech Misrecognition (46.2%)
They also found that “Intent Misclassification” was the largest contributor (68.8%) to the “Reason Unkown” user error type.
A little deeper
The researchers match the user and the developer’s perspectives for each of the 481 errors. We see that the users’ failure types all come from different failures from the developers’ perspective. Similarly, all but one of the developer’s failure types come from the three types of user-perceived errors. Only D6 (Partial Speech Misrecognition) is always associated with U2 (Speech Misrecognition).
D1 Intent Pattern Match Failure
D2 Slot Value Extraction Failure
D3 System Not In Listen Mode
D4 Intent Misclassification
D5 Utterance Not Directed To System
D6 Partial Speech Misrecognition
D7 Complete Speech Misrecognition.
U1 Reason Unknown
U2 Speech Misrecognition
U3 Utterance Pattern Match Failure.
Conversational designers should try to avoid design choices that can lead Intent misclassifications, as this error type tends to leave users frustrated and uncertain why the error occurred. Designers can be especially mindful of their utterance patterns to help reduce the likelihood of these errors.
Goetsu, Shiyoh, and Tetsuya Sakai. “Different Types of Voice User Interface Failures May Cause Different Degrees of Frustration.” arXiv preprint arXiv:2002.03582 (2020). Full article.