Assessment Integrity
Assessment integrity is keeping a test’s score meaning clean: the item’s stimulus states the task without supplying the answer, and the score records only what the respondent produced unassisted. The two halves fail independently — an item can leak its answer or hide its question, and a passing response can ride on a hint, a copied fragment, or an unintended leak instead of recall.
The stimulus states the task and withholds the answer
Item-writing guidelines attack both failures at once. Haladyna, Downing and Rodriguez’s taxonomy of 31 validated multiple-choice guidelines — checked against 27 testing textbooks and 27 studies and reviews since 1990 — puts the central idea in the stem instead of the choices, requires the stem’s directions to be very clear, and lists “avoid giving clues to the right answer” among the choice-writing rules: specific determiners, grammatical mismatches, and word echoes between stem and option all cue the answer without testing it. 1
- Cue leakage: the prompt carries the surface form of its own answer — a conjugation drill whose front shows the target suffix, a definition that names the thing defined. The response is read off the prompt instead of generated, and the item trains recognition while claiming to test recall. 2 flashcard-design works this failure deck-side.
- Question concealment: the stimulus never states what is being asked. A fill-in item whose target meaning (a quantity, a referent) lives only in the hint forces the respondent to open the hint to learn the question — every response is then assisted by construction, and no score from the item is clean. The repair is symmetric with cue-leak repair: the prompt carries the complete target meaning, and the hint keeps only assembly help (rules, category reminders). The fill-in blank is the pedagogical form of a variable slot (wh-words-as-variables); what the blank asks for has to be visible in the item.
Copy is not recall
Words printed in the item are read; words produced in the response are generated. Slamecka and Graf’s generation experiments found generated items remembered better than read items under cued and uncued recognition, free recall, and cued recall, across encoding rules and designs. 3 Scoring follows from that: material displayed in the prompt earns no production credit; grade only what the respondent produced. Wozniak’s rule 17 states the same distinction deck-side — active recall of a term does not guarantee passive recognition of it, and the reverse — so recognition cards and production cards are separate items. 4
Assisted answers are labeled, not averaged
A response that needed a hint, a handed-over rule, or a leak is an assisted pass: recorded as assisted (downgraded to the hesitant tier in an SRS grade scale, for instance) and never left standing as a clean recall. The scheduler is the operational reason the ledger stays honest — spaced repetition consumes each grade as a recall signal and sets the next interval from it, so a pass that rode on assistance schedules the item as if it were known (see spaced-repetition). Practice testing earns its high-utility rating through retrieval; responses that are read or cued rather than generated do not produce the same benefit. 5
Leaks come through the environment
The item is not the only channel that can name the answer. A tool-call label, a correction printed on a retry prompt, a meta-note about what the spoilers contain — each supplies the target while visibly not answering the question. When a leak escapes, the affected response is leak-assisted: downgrade it on the spot instead of letting it stand as a clean self-correction, and close the leak source as part of the item’s environment. This is the producer-side form of the measurement-channel problem; eval-awareness-and-grader-orientation tracks the consumer-side version, where subjects optimize for the scorer rather than the task. Both move the score onto channel artifacts and away from the construct it names.
The institutional version
Loosening a grading scale raises scores without raising achievement — what academic-leniency documents for a statewide grading reform. Item- and scoring-level failures drift the same way one level down: the measured number moves while the thing it stands for does not.
Related pages
- flashcard-design — card-level prompt design: cue leakage, sense markers, minimum information
- spaced-repetition — the consumer of grade signals
- wh-words-as-variables — the blank as a variable slot
- eval-awareness-and-grader-orientation — measurement-channel capture on the AI side
- academic-leniency — measurement-channel drift at policy scale
Sources
- Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). “A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment.” Applied Measurement in Education, 15(3), 309–333. Thomas M. Haladyna, Steven M. Downing, and Michael C. Rodriguez 2002 — A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
- Slamecka, N. J., & Graf, P. (1978). “The Generation Effect: Delineation of a Phenomenon.” Journal of Experimental Psychology: Human Learning and Memory, 4(6), 592–604. Norman J. Slamecka and Peter Graf 1978 — The Generation Effect: Delineation of a Phenomenon
- Wozniak, P. (1999). “Effective learning: Twenty rules of formulating knowledge.” SuperMemo. Piotr Wozniak 1999 — Effective learning: Twenty rules of formulating knowledge
- Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). “Improving Students’ Learning With Effective Learning Techniques.” Psychological Science in the Public Interest, 14(1), 4–58. 2013 — Dunlosky Effective Learning Techniques
Footnotes
-
Thomas M. Haladyna, Steven M. Downing, and Michael C. Rodriguez 2002 — A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment ↩
-
Piotr Wozniak 1999 — Effective learning: Twenty rules of formulating knowledge ↩
-
Norman J. Slamecka and Peter Graf 1978 — The Generation Effect: Delineation of a Phenomenon ↩
-
Piotr Wozniak 1999 — Effective learning: Twenty rules of formulating knowledge ↩