CAN GENERATIVE AI REPLACE HUMAN PARTICIPANTS IN COMMUNICATION RESEARCH? THE USE OF SYNTHETIC RESPONDENTS FOR MANIPULATION CHECKS IN EXPERIMENTS

Type
conference paper
Date Issued
2025-03-19
Author(s)
;
DOI
Abstract
This study explores whether generative AI, specifically synthetic respondents (SR) created through ChatGPT-4, can effectively support manipulation checks in communication experiments. While previous studies have validated SR's ability to emulate human responses in fields like consumer behavior and moral judgment, this study is among the first to use SR for manipulation checks in communication research. The study contrasts AI-simulated data using OpenAI's ChatGPT-4 with data collected from real humans (RH) via an online experiment about corporate social responsibility (CSR) and corporate social advocacy (CSA) communication. Key findings indicate that RH and SR yield similar response patterns across all demographic groups analyzed, but demographic variables showed varied influence on how RH and SR reacted to stimuli materials. Yet, the two samples rendered comparable results concerning manipulation effectiveness assessment, thus suggesting that SR can be a valuable tool for pretesting and refining stimuli before full-scale data collection with RH. Ethical and practical considerations surrounding AI use in communication research are discussed.
Language
English (United States)
Keywords
ChatGPT-4
generative artificial intelligence
communication research methods
experiments
Publisher
Social Science Research Network (SSRN)
Pages
29
Event Title
75th Annual Conference of the International Communication Association (ICA)
Event Location
Denver, Colorado, USA
Event Date
12-16 June 2025
Official URL
URL