Do conversations designers understand their users’ frustrations?

TL; DR

There is no 1 to 1 match between error types on the developers’ and users’ sides. Users are the most frustrated when they don’t understand why there was an error. Intent misclassification is the biggest cause of these frustrating errors.

What did they do?

The Japanese researchers built a fake calendar skill for Alexa that had 3 pre-planned errors. They wanted to see how users thought Alexa made errors and compare those assessments to how developers understood the error. They recorded 30 peoples’ interactions with this calendar skill using specific tasks. They then asked participants to rate their frustration for each error.

What did they find?

The most frustrating error type, and their level of frustration, was as follows:

1. U1 Reason Unknown (68.3%)
2. U3 Utterance Pattern Match Failure (56.7%)
3. U2 Speech Misrecognition (46.2%)

They also found that “Intent Misclassification” was the largest contributor (68.8%) to the “Reason Unkown” user error type.

A little deeper

The researchers match the user and the developer’s perspectives for each of the 481 errors. We see that the users’ failure types all come from different failures from the developers’ perspective. Similarly, all but one of the developer’s failure types come from the three types of user-perceived errors. Only D6 (Partial Speech Misrecognition) is always associated with U2 (Speech Misrecognition).

So what?

Conversational designers should try to avoid design choices that can lead Intent misclassifications, as this error type tends to leave users frustrated and uncertain why the error occurred. Designers can be especially mindful of their utterance patterns to help reduce the likelihood of these errors.

Article citation

Goetsu, Shiyoh, and Tetsuya Sakai. “Different Types of Voice User Interface Failures May Cause Different Degrees of Frustration.” arXiv preprint arXiv:2002.03582 (2020). Full article.

---