Combining Humans and Machine Learning: A Novel Approach for Evaluating Crowdsourcing Contributions in Idea Contests

The creative potential from innovative contributions of the crowd constitutes some critical challenges. The quantity of contributions and the resource demands to identify valuable ideas is high and remains challenging for firms that apply open innovation initiatives. To solve these problems, research on algorithmic approaches proved to be a valuable way by identifying metrics to distinguish between high and low-quality ideas. However, such filtering approaches always risk missing promising ideas by classifying good ideas as bad ones. In response, organizations have turned to the crowd to not just for generating ideas but also to evaluate them to filter high quality contributions. However, such crowd-based filtering approaches tend to perform poorly in practice as they make unrealistic demands on the crowd. We, therefore, conduct a design science research project to provide prescriptive knowledge on how to combine machine learning techniques with crowd evaluation to adaptively assign humans to ideas.