Transitioning from Manual to Automated Test Assembly: A Comparison of Equating Methods

As early as the 1960s, testing organizations began implementing Automated Test Assembly (ATA) to simplify the laborious process of manually assembling test forms and to enhance the psychometric properties of the examinations (Wightman, 1998; van der Linden, 2005). But it is unclear what impact transitioning to ATA has on equating outcomes. The purpose of this research study was to evaluate outcomes from different IRT scale linking and equating methods when a testing organization transitioned from manual test assembly to ATA. I analyzed the differences between original and transformed item difficulties to test whether the test characteristic curves, test information functions and item characteristic curves came from a common population, in addition to estimating the error of transformed item difficulties. After crossing each scale linking procedure with each equating method, I calculated error and bias indices (e.g., RMSD, MAD, and MSD) and evaluated the decision consistency of the equating outcomes. The results showed that the mean/sigma scale linking procedure outperformed other scale linking procedures. When crossed with all equating methods, however, the mean/mean scale linking procedure paired with the IRT preequating method produced the lowest bias and error, and highest level of decision consistency. The results of this study support the importance of aligning psychometric and test development procedures. The findings of this study suggest that the equating outcomes were related to the similarity in statistical test specifications. ATA resulted in more parallel test forms with better psychometric properties than forms assembled manually. Therefore the modifications to assembly practices warrant the reconsideration of a new base form for scaling and standard setting.