Speech Emotion Recognition among Couples using the Peak-End Rule and Transfer Learning

Item Type Conference or Workshop Item (Paper)
Abstract

Extensive couples’ literature shows that how couples feel after a conflict is predicted by certain emotional aspects of that conver- sation. Understanding the emotions of couples leads to a better understanding of partners’ mental well-being and consequently their relationships. Hence, automatic emotion recognition among couples could potentially guide interventions to help couples im- prove their emotional well-being and their relationships. It has been shown that people’s global emotional judgment after an experience is strongly influenced by the emotional extremes and ending of that experience, known as the peak-end rule. In this work, we leveraged this theory and used machine learning to investigate, which au- dio segments can be used to best predict the end-of-conversation emotions of couples. We used speech data collected from 101 Dutch- speaking couples in Belgium who engaged in 10-minute long con- versations in the lab. We extracted acoustic features from (1) the audio segments with the most extreme positive and negative rat- ings, and (2) the ending of the audio. We used transfer learning in which we extracted these acoustic features with a pre-trained convolutional neural network (YAMNet). We then used these fea- tures to train machine learning models — support vector machines — to predict the end-of-conversation valence ratings (positive vs negative) of each partner. The results of this work could inform how to best recognize the emotions of couples after conversation- sessions and eventually, lead to a better understanding of couples’ relationships either in therapy or in everyday life.

Authors Boateng, George; Sels, Laura; Kuppens, Peter; Hilpert, Peter & Kowatsch, Tobias
Language English
Subjects computer science
information management
social sciences
health sciences
HSG Classification contribution to scientific community
HSG Profile Area SoM - Business Innovation
Date 25 October 2020
Publisher ACM
Place of Publication New York, NY, USA
Page Range 17-21
Title of Book ICMI 2020 Late Breaking Results, Companion
Event Title 22nd ACM International Conference on Multimodal Interaction (ICMI)
Event Location Virtual
Event Dates October 25-29
Official URL https://doi.org/10.1145/3395035.3425253
Depositing User Prof. Dr. Tobias Kowatsch
Date Deposited 04 Jan 2021 08:28
Last Modified 04 Jan 2021 08:33
URI: https://www.alexandria.unisg.ch/publications/261889

Download

[img] Text
Boateng et al 2020 ICMI Couples.pdf

Download (1MB)

Citation

Boateng, George; Sels, Laura; Kuppens, Peter; Hilpert, Peter & Kowatsch, Tobias: Speech Emotion Recognition among Couples using the Peak-End Rule and Transfer Learning. 2020. - 22nd ACM International Conference on Multimodal Interaction (ICMI). - Virtual.

Statistics

https://www.alexandria.unisg.ch/id/eprint/261889
Edit item Edit item
Feedback?