Evaluating a Data-Driven Redesign Process for Intelligent Tutoring Systems

arXiv cs.AI / 4/1/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper evaluates the general effectiveness of a data-driven redesign process for intelligent tutoring systems by applying it to four middle-school math units selected by topic rather than prior suitability for redesign.
In a classroom study with 123 students, overall learning gains were not significantly different between the redesigned and original tutor conditions.
Despite similar learning gains, students using the redesigned tutor demonstrated improved engagement metrics, including more productive time-on-task and a higher number of skills practiced.
The redesigned tutor also produced greater total knowledge mastery, suggesting benefits that are not fully captured by standard learning-gain comparisons.
The authors conclude the approach has broader applicability, showing “promise” for data-driven redesign even when units are not preselected as likely to improve.

Abstract

Past research has defined a general process for the data-driven redesign of educational technologies and has shown that in carefully-selected instances, this process can help make systems more effective. In the current work, we test the generality of the approach by applying it to four units of a middle-school mathematics intelligent tutoring system that were selected not based on suitability for redesign, as in previous work, but on topic. We tested whether the redesigned system was more effective than the original in a classroom study with 123 students. Although the learning gains did not differ between the conditions, students who used the Redesigned Tutor had more productive time-on-task, a larger number of skills practiced, and greater total knowledge mastery. The findings highlight the promise of data-driven redesign even when applied to instructional units *not* selected as likely to yield improvement, as evidence of the generality and wide applicability of the method.