Evaluating Learning Outcomes and Perceptions of LLM-Generated Explanations Across Expertise Levels and Bloom’s Taxonomy

We found that expert output appears more analytical and structured, and novice outputs appear more conversational, authentic, and positive in tone. However, the model did not strongly distinguish between finer expert categories, and its explicit predictions about these differences did not fully align with its actual linguistic patterns. Given these findings, we explored whether pedagogical content that exploited the differences in the LLMs emulations of expertise might help or hinder student learning.

In an online study of 150 participants, individuals read LLM-generated explanations and completed Bloom-aligned tasks and LIWC aligned evaluation measures. The results showed that expertise level of the explanation mattered differently depending on the learning objective. The novice explanations performed similarly to expert explanations on lower-level Bloom tasks, while expert explanations led to better performance on higher-level Bloom tasks.

Additionally, increase in rating for Clout also was shown to improve scores on the learning tasks. The implications for mining LLMs for tacit statistical knowledge and for applications in the educational sector are discussed.</p>