1. Benchmarking LLM Next-Step Recommendations with Knowledge Space Theory
When an LLM recommends which exercise a student should learn next, can it list every exercise that is ready, without slipping in ones that come too early? The authors use Knowledge Space Theory to turn this into an automatically scored task: the answer is every unlearned exercise whose prerequisites are all mastered. Across several major models and a few linear-function topics, with only exercise descriptions the recommended set matched the reference exactly in just 61.3% of responses. Asking the model to first infer the prerequisites itself and then recommend barely helped. Writing the prerequisite relations directly into the prompt raised it to 96.3%. Structure the model fills in by itself does not help, while structure given explicitly does. Which model is best also changes with the prompt and the topic. For people building recommenders, a candidate set you can compute with rules should not be left for an LLM to infer.