Use of a Secure Web-Based Data Management Platform to Track Resident Operative Performance and Program Educational Quality Over Time
Adriana L. Meholick, MD1, Jonathan L. Jesneck, PhD, MS2, Ruchi M. Thanawala, MD, MS3, Neal E. Seymour, MD1
1University of Massachusetts Medical School-Baystate, Springfield, MA 2Firefly Lab, Cambridge, MA 3University of Iowa Hospitals and Clinics, Iowa City, IA
Background: In surgical residency, ACGME-mandated performance evaluation includes operating room assessment to demonstrate achievement of necessary skills and autonomy goals. Our institution uses the O-SCORE to assess resident operative skill using a web-based data management platform (Firefly). O-SCORE evaluations are delivered to faculty members for completion, facilitating timely feedback. We sought to demonstrate the platform’s utility in detecting operative performance changes in response to educational interventions.
Methods: Resident performance of laparoscopic procedures (cholecystectomy, appendectomy, inguinal and ventral hernia repair) was examined over three academic years (2016-2019). Four of 36 residents received program-assigned supplemental simulation training to improve laparoscopic skills. O-SCORE data for these residents were extracted from peer data. Data were compared pre- and post-intervention and compared to non-intervention aggregate scores (Student’s t-tests). Individual learning curves characterized improvement patterns.
Results: Upon review, 3325 cases met criteria. As expected, overall autonomy increased with number of cases performed. The four residents assigned supplemental training had pre-intervention score averages lower than that of the non-intervention group (2.25 ±0.43 vs 3.57 ±1.02; p<0.0001). All four residents’ post-intervention scores improved (increase to 3.40 ±0.61; p<0.0001) and were not significantly different compared to scores for the non-intervention group.
Conclusion: Firefly proved effective in tracking responses to supplemental training, which was associated with improved clinical laparoscopic skills and autonomy. Aggregated data analysis confirmed effectiveness of the program’s educational intervention.
Back to 2020 Posters