The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities

Item Type Journal paper

Given the rise of the Internet, consumers increasingly engage in co-creating products and services. Whereas most co-creation research deals with various aspects of generating user-generated content, this study addresses designing ratings scales for evaluating such content. In detail, we analyze functional and perceptional aspects of two frequently used rating scales in online innovation communities. Using a multimethod approach, our experiments show that a multicriteria scale leads to higher decision quality of users than a single-criterion scale, that idea elaboration (i.e., idea length) negatively moderates this effect such that the single-criterion rating scale outperforms the multicriteria scale for long ideas, and finally that the multicriteria scale leads to more favorable user attitudes toward the Web site. To ensure robustness of our results, we applied a bootstrap-based Monte Carlo simulation based on our experimental data. We found that around 20 user ratings per idea are sufficient for creating stable idea rankings and that a combination of both rating scales leads to a 63 percent performance improvement over the single-criterion rating scale and 16 percent over the multicriteria rating scale. Our work contributes to co-creation research by offering insights as to how the interaction of the technology being used (i.e., rating scale) and the attributes of the rating object affects two central outcome measures: the effectiveness of the rating in terms of decision quality of its users and the perception of the scale by its users as a predictor of future use.

Authors Riedl, Christoph; Blohm, Ivo; Leimeister, Jan Marco & Krcmar, Helmut
Research Team Crowdsourcing, IWI6, CCC
Projects Blohm, Prof. Dr. Ivo; Leimeister, Prof. Dr. Jan Marco; Haas, Philipp; Leicht, Niklas; Knop, M.A. Nicolas; Troll, Julia & Rhyn, Marcel (2013) CC Crowdsourcing [applied research project] Official URL
Journal or Publication Title International Journal of Electronic Commerce (IJEC)
Language English
Keywords Rating Scales; Decision Quality; User Attitudes;Online Innovation Communities
Subjects information management
HSG Classification contribution to scientific community
HSG Profile Area SoM - Business Innovation
Refereed Yes
Date March 2013
Publisher Routledge, Taylor & Francis Group
Place of Publication Philadelphia, Pa.
Volume 17
Number 3
Page Range 7-36
Number of Pages 30
ISSN 1086-4415
ISSN-Digital 1557-9301
Official URL
Depositing User Michael Kunz
Date Deposited 07 Feb 2013 22:48
Last Modified 09 Jan 2017 09:07



Download (902kB) | Preview


Riedl, Christoph; Blohm, Ivo; Leimeister, Jan Marco & Krcmar, Helmut (2013) The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities. International Journal of Electronic Commerce (IJEC), 17 (3). 7-36. ISSN 1086-4415

Edit item Edit item