"How accurate is your gap detection?" is a question we get regularly from teachers and administrators evaluating AI tutoring tools. It is the right question to ask. The problem is that the answer depends heavily on what you mean by "gap detection" and how you are measuring accuracy, and most vendors do not distinguish between these carefully enough for the answer to be useful.
This post explains what factors actually determine gap detection accuracy, what honest accuracy claims look like, and what questions you should ask any vendor who gives you a number without context.
The accuracy problem starts with defining the gap
A knowledge gap is not a precisely defined object. When we say a student has a "fractions gap," we might mean they cannot reduce fractions to lowest terms, or they cannot perform fraction arithmetic, or they have a conceptual misunderstanding about what a fraction represents, or they can execute fraction procedures but cannot apply them in novel contexts. These are four different gaps that require four different interventions, and a diagnostic that correctly flags "fractions issue" but cannot distinguish between them has limited actionability.
Gap detection accuracy should be measured against a precisely defined gap, not against a broad topic category. The relevant question is: when the system flags a student as having a gap in a specific concept -- say, the ability to divide fractions -- is that flagging accurate? What fraction of flags are correct (precision)? What fraction of actual gaps are detected (recall)? Both numbers matter, and they involve different kinds of errors with different costs.
Precision vs. recall in gap detection
A high-precision detector rarely misidentifies a student as having a gap they do not actually have. False positive rate is low. This is important for teacher trust: if the system often flags students who turn out to be fine when the teacher checks, teachers will stop trusting the flags and stop acting on them. A detector that cries wolf quickly becomes invisible.
A high-recall detector catches most of the gaps that actually exist. False negative rate is low. This is important for the student: a gap that is not detected will not be addressed. The cost of a missed gap is a student who enters the next unit without the prerequisite and struggles in ways that seem mysterious.
In practice, there is always a tradeoff between precision and recall in any detection system. You can increase recall by lowering the threshold for what counts as a gap -- flag anyone who misses even one question on a topic. This will catch most real gaps, but it will also flag many students who are fine, reducing precision and eroding teacher trust. You can increase precision by requiring a very strong signal before flagging -- multiple consecutive failures in a structured sequence -- but then you will miss gaps that manifest in subtler ways.
The right balance depends on the context. In a high-stakes situation (a prerequisite gap that will cause serious problems in the next major unit), higher recall is worth some loss of precision -- better to check a student who turns out to be fine than to miss a student who is not. In a routine monitoring context where teacher attention is scarce, higher precision is more important -- the teacher can only act on a small number of flags, so those flags need to be reliable.
Sample size is a major constraint
The number of questions a student answers on a given concept before the system makes a gap judgment is the most direct driver of accuracy. This creates a practical tension: more questions give a more reliable signal, but students and teachers have limited time for diagnostic activity.
With three questions per concept, a student who knows a concept can still answer two incorrectly due to careless errors, and a student who has a gap can still guess correctly on two items. The signal is noisy. With eight questions per concept and a structured question design that systematically varies the form, the signal is much more reliable -- error patterns are distinguishable from genuine gaps, and consistent performance is distinguishable from lucky guessing.
The practical minimum for a reliable gap judgment -- where "reliable" means the judgment would change less than 10-15% of the time if you ran the same assessment again -- is typically five to seven well-designed items per concept, depending on the concept's complexity and how much the question variants differ from each other. Below that threshold, the accuracy numbers that vendors quote should be taken with caution.
What honest accuracy benchmarking looks like
A vendor presenting accuracy data for their gap detection should be able to answer: compared to what ground truth, measured how, on which student population, using how many questions per concept? "91% accuracy" without this context is not informative. "91% precision and 84% recall against teacher-assessed ground truth, averaged across 12 targeted concepts, using 6-question sequences, validated on a cohort of 400+ students across grades 5-8 in school pilot contexts" is a claim you can evaluate.
You should also ask what happens to the accuracy numbers when the system is applied to a different curriculum or a different grade band than it was validated on. Gap detection systems that were validated on one set of standards may perform differently when the prerequisite structure of the content differs. A system validated on Common Core grade 5-8 math may not show the same accuracy numbers on a non-Common-Core curriculum with different sequencing.
The minimum bar that makes gap detection worth using
Despite all these caveats, the relevant comparison is not gap detection against a perfect diagnostic. It is gap detection against what teachers currently have -- which, for most classrooms, is a unit test that reveals the gap after the unit is over. Even an imperfect diagnostic that operates mid-unit with 75-80% precision and recall is substantially more useful than a post-hoc unit test that identifies the gap when it is too late to address it within the current unit.
The right frame for evaluating gap detection accuracy is: does this provide the teacher with better-than-chance information about where each student's prerequisite understanding is uncertain, early enough to act on? That is a much lower bar than "perfect knowledge of every gap" -- and it is the bar that determines whether the tool is actually worth using in a classroom.