Behind the Scenes: EPAs Mapped to Milestones to Inform CCC on Resident Development without Cluttering Rotation Forms
Rachel H. Kon, MD
Georgetown University Hospital
Program Size: 31-100 residents
Academic Setting: University-Based
Clinical Setting: Inpatient wards
Overview
Our milestone-based evaluation system incorporates EPAs from our rotation goals and objectives into our end-of-rotation evaluation forms. We modeled much our monthly rotation forms after the University of Cincinnati(1) IM program, but mapped our EPAs directly to reporting milestones (rather than curricular milestones) to facilitate semi-annual milestone scoring. We use a “levels of entrustment” scale(2) to demonstrate progression over subsequent PGY levels within one form. The monthly forms have content and process-based EPAs listed that are graded on the levels of entrustment scale. The end of each form has an “overall clinical performance” statement for faculty to give feedback relative to level of training and an open-ended summative section. New forms were designed for each rotation (wards, MICU, continuity clinic, pulmonary, etc.), which totaled 28 separate forms. The EPAs for each form were chosen from rotation goals and objectives as well as faculty preference. Subspecialty Education Coordinators give feedback on how certain EPAs have been working for their rotation and are able to modify EPAs on their form periodically. The EPAs were mapped to the 22 ACGME IM Reporting Milestones in our New Innovations software so that level of entrustment scores for EPAs relevant to each reporting milestone are visible to the CCC members for semi-annual reports. Mapping was done by consensus of a small committee. Monthly forms do not have milestones visible to reduce confusion among raters and learners. Residents meet with their CCC advisor every 6 months to go over their Semi-Annual Milestones report scores and identify their strengths and weaknesses.
Download the Tool
Development
Determining prompts and grading scale for evaluation forms – Developed EPA statements from rotation goals and objectives then requested grading on levels of entrustment scale. Mapping – Developed milestone experts on our CCC who agreed on mapping of EPAs to reporting milestones. Each 6-month reporting period, we requested feedback from CCC members and adjusted mapping accordingly. Standardizing across programs – Our residents rotate at several DC-area hospitals with George Washington University and INOVA Fairfax residents. We collaborated with their program directors to standardize our wards evaluation forms so that all residents on the same rotation were evaluated on same scale.
Lessons Learned
Once forms were developed and mapped for core rotations, a grid was created to calculate how many EPAs were mapped to each sub-competency. While 50+ EPAs mapped to PC, MK, PROF, and ICS, we found a scarcity of SBP and PBLI relevant EPAs. We developed new “Point-of-Care” evaluation forms, containing several EPAs for journal clubs, QI projects, procedures, etc., that are available on-demand in New Innovations and mapped to less common sub-competencies. Suboptimal inter-rater consistency – Faculty development sessions on grading sale and feedback to individual raters helped to improve this, though it is still a work in progress.
Faculty Development and Training
Several grand rounds and division presentations announced the change to our evaluation system and new grading scale. Our APD for evaluation went to division faculty meetings and affiliate sites to give faculty development sessions on the reasoning behind and details of the new system. We have given periodic updates at division meetings and through short email reminders. Our APD also went to the hospitalist meetings several times a year to meet new faculty and overview system for them. An online on-boarding module is also being developed.
How Used to Inform Decisions about a Learner's Milestone
Scores on the levels of entrustment scale fit with the Dreyfus Model and demonstrate progressive development on activities related to sub-competency areas. The CCC members use averages of entrustment scores from EPAs relevant/mapped to a milestone in addition to comments, ITE scores, 360 scores, Mini-CEX, and other feedback to inform the milestone scores.
References
- Warm, Held, Mathis – Mapping Milestones and Competencies to Entrustable Professional Activitis. Workshop APDIM Spring 2013
- Anna T. Cianciolo and Jason A. Kegg (2013) Behavioral Specification of the Entrustment Process. Journal of Graduate Medical Education: March 2013, Vol. 5, No. 1, pp. 10-12.
For more information, please contact rachel.kon@gmail.com