IKS Deliverable - D1.1 Report: Design of the Semantic Benchmark Experiment

Public Deliverable - The objective of this deliverable is to design a benchmark model for Content Management Systems (CMSs) in order to identify relevant requirements for the Interactive Knowledge Stack (IKS). The IKS will be a layered set of software components and specifications with the goal to improve the interaction with knowledge objects of CMSs by using Semantic Web technologies. In contrast to projects that benchmarked rather technical aspects of CMSs, we propose a model that evaluates CMSs consistently from the business perspective down to the technology layer. This approach is based on IT alignment theory that states that higher degrees of fit between business needs and IT result in increased business performance. Accordingly, IT executives of organizations that provide CMSs and IT executives of organizations that use CMSs as well as CMS developers are taken into account covering both business and technical aspects.

The deliverable is structured as follows. First, we introduce the IKS project and pose relevant research questions that the IKS benchmark model for CMSs must address. Then, related work on CMS benchmarks, semantic technologies and applications is discussed from which shortcomings of exiting benchmarks are derived and implications for the design of the IKS benchmark model are drawn. Afterwards, we present the IKS benchmark model for CMSs, which describes a study-driven and experiment-driven approach to compare CMSs. Consistent with the model, questionnaire items for the study targeted at IT executives of CMS customer and provider organizations and implementation tasks for the experiment targeted at CMS developers are provided. Finally, we conclude this deliverable by a summary and an outlook on the next steps within the IKS project.