Assessing Competency in Practice-Based Learning In Learning Portfolio Entries

Background: Graduate medical education is undergoing a dramatic shift toward competency-based assessment of learners. Competency assessment requires clear definitions of competency in any given area. In GME, competencies such as practice-based learning and improvement (PBL&I) are particularly difficult to assess due to the lack of validated methods of assessment. Learning portfolios, advocated and used as means for demonstrating PBL&I, may provide methods of assessing competency in PBL&I, based on portfolio entries. The purpose of this study is to identify the criteria used by expert senior surgical educators when judging competence in PBL&I. Methods: We performed a qualitative study of faculty assessments of written learning portfolio entries. Six Surgical Learning and Instructional Portfolio (SLIP) entries written by general surgery residents at the Medical College of Wisconsin during the 2009 academic year served as the documents to be assessed by three expert senior surgical clinician educators. These faculty members were asked to identify criteria for assessment of PBL&I competence, using a combination of individual (independent) and group (consensus) assessments of SLIP entries. Individual and group discussions were conducted, recorded, and transcribed to serve as the study data set. Analysis of the discussions was performed using qualitative methodology to identify themes in the individual and group discussions for the purpose of defining competence in PBL&I. The assessment themes derived from the qualitative analysis are presented in this paper, with narrative examples to describe the progression of competency in PBL&I, using behavioral anchors to ground the descriptions. Results: The collaborative coding process resulted in identification of seven themes associated with competency in PBL&I related to SLIP entries: 1) Self-awareness Regarding Impact of Actions; 2) Identification and Thorough Description of Learning Goals; 3) Cases Used as Catalyst for Reflection; 4) Reconceptualization with Appropriate Use and Critique of Cited Literature; 5) Communication Skills/Completeness of Entry Template; 6) Description of Future Behavioral Change; 7) Engagement in Process- Identifies as Personally Relevant. Conclusions: The seven themes identified, from qualitative analysis of the criteria used by expert raters to assess the SLIP entries, is consistent with and complements other literature related to reflective practice and experiential learning theory. This study provides a foundation for further development of an assessment tool for assessing learner portfolios for evidence of PBL&I.