Three Papers on Measuring Teaching Practice

This dissertation comprises three papers that collectively address the methodological issues in using classroom observation protocols (OPs) to measure science teachers’ instructional practices. The first paper, a systematic literature review, proposes an analytical framework that address the research objectives, design, data collection strategies, and data analysis and interpretation for the studies using OPs. I used this framework to evaluate existing OPs used in science education. The results underscore the need for transparent procedures in rater training, sampling choices, and advanced statistical techniques to enhance measurement reliability and validity. The second paper introduces a comprehensive observational system based on the Rasch model, showcasing its application in 321 high school chemistry classroom observation videos. This observation system shifts focus from solely developing new OPs to a holistic approach that includes instrument validation, rater training, and advanced data analytics. The psychometric evidence supported the system’s capacity to produce highly reliable and valid measures of instructional practices. The third paper employs a Partial Credit Many-Facet Rasch Measurement (PC-MFRM) to examine three rater effects—severity, central tendency, and halo effect. These effects can introduce construct-irrelevant variance, thereby affecting the reliability and validity of the results. The study suggests that PC-MFRM was able to indicate the three rater effects. In addition, the diagnostic statistics may be used to guide rater training and calibration to improve the measurement of instructional practices with OPs.