Development, Administration, and Validity Evidence of a Novel Preparatory Test for Nephrology Licensure

ABSTRACT Purpose: Western University nephrology program trainees lack validated assessment scores from a preparatory test for their licensing examination. A 55-item constructed-response preparatory test was developed and administered to meet the needs of nephrology trainees and the program at Western University. This study examines validity evidence of assessment scores from the preparatory test. Methods: Messick’s unified validity framework was used to gather validity evidence for content, response process, internal structure, relations to other variables, and consequences. Tests were scored independently by three qualified graders using an a priori developed and modified rubric. Questionnaires completed before and after the preparatory test were used to gather data on the trainees’ perception of examination preparedness, item clarity, and curriculum adequacy. Results: There were 10 trainees and 5 faculty volunteers who took this preparatory test. The majority of trainees passed the constructed-response preparatory test. However, many scored poorly on items assessing renal pathology and physiology knowledge. The faculty members who took the preparatory test also had low scores in these two domains. Overall, constructed-response preparatory test scores reflected nephrology exposure (one year versus two years versus many years). Conclusion: The preparatory test and questionnaire data highlighted shortcomings of the Western University nephrology training program curriculum. Following the preparatory test, trainees expressed an improved sense of preparedness for their upcoming licensing examination.