Integrating emotional expressions with utterances in pragmatic inference

Wu, Y., Tessler, M. H., Asaba, M., Zhu, P., Gweon, H., & Frank, M. C. (2021). In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society. (*Co-first authors)

Abstract: Human communication involves far more than words; speakers? utterances are often accompanied by various kinds of emotional expressions. How do listeners represent and integrate these distinct sources of information to make communicative inferences? We first show that people, as listeners, integrate both verbal and emotional information when inferring true states of the world and others' communicative goals, and then present computational models that formalize these inferences by considering different ways in which these signals might be generated. Results suggest that while listeners understand that utterances and emotional expressions are generated by a balance of speakers? informational and social goals, they additionally consider the possibility that emotional expressions are noncommunicative signals that directly reflect the speaker?s internal states. These results are consistent with the predictions of a probabilistic model that integrates goal inferences with linguistic and emotional signals, moving us towards a more complete formal theory of human communicative reasoning.

• The paper can be found here.

• Links to the pre-registration of experiments, including key hypotheses about behavioral data and model predictions:

  • Prior elicitation
  • Main experiment

• Links to our online experiments, if you'd like to see what the experiments look like

  • Prior elicitation
  • Main Experiment:
    • Inferring true state
    • Inferring communicative goals

• Structure of this repository

  • data: behavioral data
  • models: all models tested
  • analysis: analysis of behavioral data and comparison between behavioral data and model predictions
  • paper: materials used in the paper