Setting Standards for a Portfolio Component Using Qualitative Methods

This study was designed to compare inter-rater reliability for grading of patient care based reflective essays, a common portfolio component, using a conventional scoring rubric and a rubric with standards set and anchors derived from a focus group discussion. Two groups of 10 essays from a regular offering of a fourth year comprehensive care course at the University of Illinois at Chicago, College of Dentistry were graded by a group of 4 full time clinical faculty members. The first group of 10 essays was graded using the scoring guidelines provided in the course syllabus and used in the regular offering of the course. The second set of 10 similar essays was graded by the same 4 faculty using a scoring rubric that was the result of qualitative analysis of themes obtained from a focus group discussion among a similar but different group of 5 faculty members. These 5 faculty members, prior to the focus group discussion, were provided 2 borderline essays from the same course to read and grade. The focus group discussion was facilitated to describe the qualities in the essays that led the faculty to assign the grade they chose. After each set of essays was graded, exact agreement and intra-class correlations were calculated. Finally, a second focus group discussion was conducted with the faculty assessors to provide insight into their experiences with the two rubrics. The results of this study did not demonstrate the improvement in inter-rater reliability needed to support summative assessment decisions. However, these results were promising in that this technique, applied using a larger sample of essays, might show even greater improvements in inter-rater reliability. Probably most encouraging was the faculty assessor perceptions of their experiences in grading, which clearly demonstrated a greater degree of confidence and defensibility in the grades given, using the focus group derived scoring rubric and standards. The results of this study indicate that the use of qualitative methods applied to the assessment of portfolios warrants further investigation.