Evaluating LLMs' Performance At Automatic Short-Answer Grading

In recent years, the use of Large Language Models (LLMs) has become more accessible and widespread. With a free-of-charge access types people have began applying the models to various tasks beyond the task of next-word prediction. In an exploratory study, we take a closer look at the use of LLMs for Automatic Short Answer Grading. We compare the grading of short-answer tasks by two human graders to this of an LLM. We discuss the results and present examples of observed shortcomings in the annotation and grading.