Development of a Comprehensive Examination in a Physical Therapist Educational Program

The Department of Physical Therapy at the University of Illinois at Chicago (UIC) administers a comprehensive examination (CE) each year to the students in the Doctor of Physical Therapy Program. In order to assess the validity of the test score interpretation, I conducted a validity study focusing on validity evidence based on test content and consequential aspect of validity evidence, specifically, the process for establishing a cut score and external source of evidence. I first gathered evidence related to the instructional content of the program curriculum. I then developed a task survey from which I developed a test blueprint for the CE and established the alignment of the CE with the test blueprint. I also convened a panel of subject matter experts to develop performance level descriptors to conceptualize a student who demonstrates minimally acceptable competent for practice. Additionally, I convened a different panel of experts to establish a cut score for the CE utilizing accepted standard setting methods. Finally, I assessed the association of student performance on the CE with their cumulative grade point average as one source of external validity evidence. Based on the findings, I made several recommendations for further development of the CE. First, item development should be targeted at the identified under-represented areas of test content. Areas of the curriculum that were identified as important were provided to facilitate targeted item development. Second, recommendations to improve the reliability of the test and associated classification consistency of the cut score were provided. Finally, recommendations for additional areas of research to support consequential aspects of validity were made.