(P23) Determining Appropriate Clinical Skills

Objectives: There is a critical need for a capability to determine if the appropriate skills have been acquired and maintained. The purpose of this study was to evaluate the impact of the adoption and use of a novel learning management system (LMS) platform on emergency residents’ education.

Background: Clinical skill maintenance/acquisition can be supported by supplemental simulation-based training.1 Training or refresher requirements are variable among healthcare providers. The current standard is to offer training/refresher training at some predefined frequency. Our work involves the development of a novel LMS platform providing standardized multimodal measurement and assessment of knowledge/skill acquisition and performance of providers as they train and advance in their careers. In the future, these standard performance measures can be incorporated into a machine learning-based models to personalize education/training of healthcare providers to increase the potential that providers operate at peak performance in their clinical arena.

Methods: This effort involves leveraging a pre-existing learning management system platform called PREPARE to evaluate emergency medicine resident’s performance subjectively and objectively via observer-based and physiological monitoring respectively.1 Proficiency and clinical performance are generally measured at discrete time points. PREPARE offers the addition of wearable neurophysiological monitors that collect continuous performance measures in addition to observer-based and other subjective assessments that are routinely recorded during simulation-based education. One of the novel aspects of PREPARE is that all data and measures are standardized and applicable across a diverse set of disciplines. This is critical as the use of this platform will result in the creation of a comprehensive database that is necessary to train and develop system intelligence with robust machine learning algorithms. This will allow classification of expertise and predict knowledge/skill acquisition in guiding future training needs. This database includes relevant background variables (age, gender, role, years of practice/training, frequency of completing relevant procedures, etc.), self-report measures (task load, engagement, stress, etc.), cognitive and behavioral measures, and neurophysiological data (EEG, galvanic skin response [GSR], heart rate, etc.) and derived neurophysiological metrics that indicate states such as cognitive workload, stress, and fatigue.

Results: Figures 1 and 2 demonstrate the performance measures rated by expert clinical faculty using the platform for a simulation scenario involving treatment of a Hemophilia A patient. Each “event” in Figures 1 and 2 was evaluated by faculty with the platform via a color-coded red to green scale which translates to a quantitative 0-100 performance measurement. Events are organized in a standardized hierarchy as cognitive, behavioral, and psychomotor skills. Figure 3 shows the galvanic skin response (GSR) of each resident collected during the scenario. The 2nd year resident’s GSR is approximately 7.5 times greater in magnitude than that of the 3rd year resident. Average performance of the 2nd year resident was 55, 62, and 79 for behavioral, psychomotor, and cognitive skills. For the third-year resident average performance was 91, 100, and 81 for behavioral, psychomotor, and cognitive skills. A significance difference in faculty-documented performance and physiological response was observed across the residents. This indicates that the level of comfortability or stress responses potentially translates to better performance consistent with more experience and years in training. This is consistent with our prior findings when comparing performance and physiological responses of residents independent of years in training.

Conclusions: We have created and established PREPARE at the University of Toledo and have made it available to all specialties. It has been specifically and considerably used by Emergency Medicine, Anesthesiology, and Surgery. Its use as a supplement to clinical experience and training will enhance clinician performance by allowing training in a safe, simulation environment. Ultimately, this platform and the data collected and generated by it will be useful in personalizing training and education of current and future healthcare providers to ensure that they achieve peak performance in their clinical setting.

References (Optional): 1. Pappada S, Owais MH, Aouthmany S, Schneiderman J, Toy S, Schiavi A, Miller C, Guris RD, Papadimos T, Rega P. International Journal of Healthcare Simulation 2022 DOI: 10.54531/mngy8113

Funding (Optional): none