New Insights Regarding the Reliability and Validity of the CLASS Pre-K, a Measure Being Used in High-Stakes Policy Monitoring of Early Childhood Classroom

Results from the IGPA Early Investments Initiative's Preschool-Aged Classroom Video Project were presented by UIC Honors College capstone projects by UIC undergraduate students Nicole Szydlowski, Yasmine Toledo, and D’Andre Walker. IGPA Faculty Member Rachel Gordon is the PI of the initiative and the students' capstone advisor. The students presented their findings at the UIC Undergraduate Research Forum on April 13, 2018.

The student projects center around the three core themes and innovations of the broader initiative, which collected full day streams of video for an entire week in eight preschool-aged classrooms and every Friday for over a dozen weeks in a ninth classroom. Combining a panoramic and a close-up camera to approximate what coders would see when sitting in the classroom, the design allowed the team to examine critical issues about variation in ratings across coders, time, and context. A team of a dozen faculty and students rated over 420 15-minute video clips using the Classroom Assessment Scoring System (CLASS), Pre-K allowing for examination of inter-rater reliability, within- and between-day fluctuations, and differential interpretations of the same coding instructions and classroom activities.

Aligning with these themes, the students’ projects offer preliminary evidence that:

* Coders generally maintained the inter-rater reliability level targeted by the CLASS developers, 80% "within-one" agreement (e.g., a score of 3, 4, or 5 agreeing with a 4 on a 7-point scale). Yet, reliability fluctuated more on certain dimensions (e.g., productivity) and exact agreement (e.g., a 4 agreeing with a 4) was considerably lower. (Nicole Szydlowski)

* Contributing to the challenge of exact agreement, raters differently interpreted subjective words on the CLASS Emotional Support dimension (e.g., ample, appropriate, considerable). Ratings of words by students fluent in Spanish and interviews with researchers who used the CLASS in a Spanish-speaking country revealed such varying interpretations are exacerbated cross-culturally, including with phrases like "falls through the cracks" or "good effort" (Yasmine Toledo).

* CLASS scores fluctuated throughout each day, including on the Regard for Student Perspectives dimension, calling into question standard reliance on scores that average across 15-minute cycles. These fluctuations were predictably associated with classroom activities, with teachers' regard for student perspectives being higher during one-on-one center time than during large group or transition time (D’Andre Walker).

These results importantly contribute to increasing concerns about the suitability of the CLASS Pre-K and other observational measures of classroom quality for high-stakes monitoring and accountability purposes, such as contributing to publicly-disseminated star or medal ratings that have reputational and financial consequences (e.g., Illinois' Excelerate Quality Rating and Improvement System) and encourage continued efforts to improve existing and develop new measures. One effort currently underway, for instance, is the EMOTERS project, a $1.4 million effort funded by the Institute of Education Sciences in which members of the IGPA Early Investments Project team (and collaborators from George Mason University) are using the project's video pilot data and state-of-the-art statistical models as they develop a new measure of teachers' supports for children's emotional development.

The students’ Honors College capstone projects benefited from the entire project team, including PI: Rachel A. Gordon; Co-PIs: Catherine Main, Kathleen Sheridan, and Katherine Zinsser; core coders: Beltran, Winnie Kwong, Tito Ponce, Nicole Szydlowski, Yasmine Toledo, and D’Andre Walker; and principal project coordinators: Julia Bates, Claire Christensen, and Samantha de Souza. Collection of video in the nine classrooms and its coding was funded by the Institute of Government and Public Affairs.