There's a Computer That Knows How You Feel (Most of the Time)

A weekly newsletter featuring the biggest ideas from the smartest people

We struggle to communicate with the computers in our lives in a natural way. Just think about how many times you have to yell at your Xbox One in order for it to understand a simple command or how patiently you exaggerate every syllable so Siri can play the right song rather than try and call your ex-boyfriend. We dream of the day when we can speak to our machines in the same way Tony Stark talks to Jarvis. But there’s a lot more to speaking than understanding pop-culture references, idioms, and the nuances of language; there’s an emotional component to figure out.

Nathan Collins from Pacific Standard writes that a new study has found that if computers listen for the right verbal cues, they can identify emotions correctly nine times out of 10.

Researchers Reza Asadi and Harriet Fell looked at three features of human speech: mel-frequency cepstrum coefficients (MFCCs), which Collins writes “separates out the effects of the throat, tongue, and lips, which act as filters on the underlying sound of a person’s vocal cords”; Teager Energy Operators (TEOs), which measures the flow of air; and landmark features or transitional spots in speech.

Asadi and Fell wanted to see if measuring these features was enough for a program to accurately detect six emotional states (anger, fear, disgust, sadness, joy, and neutrality). They used a set of audio clips from the Linguistic Data Consortium’s Emotional Prosody and Speech Transcripts to run their test.

They wrote of their results:

“Specifically TEO features resulted in improvements in detecting anger and fear and landmark features improved the results for detecting sadness and joy. The classifier had the highest accuracy, 92 percent, in detecting anger and the lowest, 87 percent, in detecting joy.”

As we continue our advancements in artificial intelligence, one can only wonder what the future will look like — what kinds of discussions we’ll have as to the nature of what it is to be “human.” We continue to make progress in furthering the capabilities of AIs to do everything from understanding triggers for irrational anger to playing poker. Jon Iwata, senior VP of Marketing and Communications at IBM, even wonders whether Watson could be considered more than just a cognitive machine. He talks about the questions this achievement has put forth: