Looking Beyond Observations: Observed Scores versus Rasch Measures for the Analysis of Efficacy Data

Funders of educational research devote hundreds of millions of dollars each year to support theoretically sound educational interventions and research on these interventions. Unfortunately, relatively few of these interventions have been established as clearly effective; most have been shown to be weak or ineffectual compared with normal educational practice (Coalition for Evidence-based Policy, 2013). The purpose of this study was to explore one possible reason for mixed findings in educational intervention studies: the use of observed scores in parametric statistics, where interval measures should be used. The study compared the statistical conclusions derived from observed scores and Rasch measures of several outcomes, using exemplar data from a large, grant-funded, PBL intervention study. Consistent with prior research, results showed that for an objective test, attitude measures, and a rater-scored essay, linear relationships between Rasch person measures and observed scores were very strong (r > .86). For the seven outcomes tested, Rasch person measures and observed scores consistently agreed on the statistical significance of findings. Based on the dataset and analyses used in this dissertation, there is no evidence to conclude that the use of observed/ordinal scores is a likely culprit for null or mixed findings. This is in contrast with certain earlier findings that showed at least one case wherein Rasch person measures and observed scores provided divergent statistical conclusions. Present findings suggest that, when sample size is adequately large, and when measures are already determined (i.e. Rasch analysis is not being used for scale development) and meet the general standards of reliability and model fit, there is little to be gained by using Rasch modeling to convert ordinal observed scores into interval Rasch person measures. There is no basis to recommend that researchers use Rasch modeling in this manner to improve the quality of their interpretation of efficacy or conclusion of the impact of an intervention.