Item Parameter Drift in Computer Adaptive Testing Due to Lack of Content Knowledge within Sub-populations

This study was conducted to analyze potential item parameter drift (IPD) impact on person ability estimates and classification accuracy when drift affects an examinee sub-group. Using a series of simulations, three factors were manipulated: (a) percentage of IPD items in the CAT exam, (b) percentage of examinees affected by IPD, and (c) item pool targeting. IPD impact on ability estimation was evaluated using bias, root mean square error (RMSE), mean absolute difference (MAD), and correlations. The impact on classification accuracy was examined based on number and percentages of misclassifications, their significance, and rank order correlations. Person fit indices of misclassified examinees were evaluated to assess these indices’ effectiveness in real life tests to detect misfit that can occur as a result of IPD. The findings revealed that IPD exposed to a sub-group of examinees can affect classification accuracy of those examinees substantially, but IPD impact on average ability estimation was small. The study provides useful information to states and districts planning to implement or are currently implementing CAT as part of their assessments by emphasizing every examinee should be given equal opportunity to learn the content and demonstrate their true ability in the exam.