Artificial Intelligent Clinical Decision Support System Simulation Center Study for Technology Acceptance

Yale University

Status: Completed

Conditions: Gastrointestinal Hemorrhage

Treatments: Other: LLM

Study type: Interventional

Funder types: Other; NIH

Identifiers

NCT05816473

2000034521

1K23DK125718-01A1 (U.S. NIH Grant/Contract)

Details and patient eligibility

About

The purpose of this research study is to measure the effect on of a large language model interface on the usability, attitudes, and provider trust when using a machine learning algorithm-based clinical decision support system in the setting of bleeding from the upper gastrointestinal tract (upper GIB). Specifically, the investigators are looking to assess the optimal implementation of such machine learning algorithms in simulation scenarios to best engender trust and improve usability. Participants will be randomized to either machine learning algorithm alone or algorithm with a large language model interface and exposed to simulation cases of upper GIB.

Full description

The experiment will deploy a previously validated machine learning algorithm trained on existing clinical datasets within simulation scenarios in which a patient with acute gastrointestinal bleeding (at low, moderate, and high risk for poor outcome) is evaluated.

Prior to the simulation, a baseline educational module about artificial intelligence, machine learning, and clinical decision support will be provided to all participants. The investigators will establish psychological safety by detailing what is available in the room, the opportunity to call a consultant, and availability of laboratory and radiographic studies. Each clinical scenario will run for approximately 10 minutes based on real patient cases where vital signs change over time and laboratory values are made available at specific points in the assessment. The study will evaluate the effect of a large language model-based interaction with the machine learning algorithm with interpretability dashboard compared to the machine learning algorithm with interpretability dashboard alone. Each participant will receive three scenarios in randomized order of risk.

For the large language model interaction arm, participants will be provided the computer workstation a LLM chatbot interface of the algorithm and interpretability dashboard For the machine learning dashboard arm, participants will be provided the computer workstation with the algorithm and interpretability dashboard.

Enrollment: 106 patients

Sex: All

Volunteers: Accepts Healthy Volunteers

Inclusion criteria

• Internal Medicine residency trainees at study institution
• Emergency Medicine residency trainees at study institution

Exclusion criteria

• N/A

Trial design

Primary purpose: Health Services Research

Allocation: N/A

Interventional model: Parallel Assignment

Masking: None (Open label)

---

106 participants in 2 patient groups

Large Language Model-based Interaction

Experimental group

Description:

LLM-powered chatbot with the machine learning dashboard to provide the risk assessment and provide rationale based on interpretability metrics provided by the dashboard in which study participants can directly interact with using natural language. Participants will be provided the Generative Pre-trained Transformer (GPT) chatbot powered machine learning model dashboard.

Treatment:

Other: LLM

Machine Learning Dashboard

No Intervention group

Description:

Machine learning algorithm output with an interactive dashboard that can be used to explain, or interpret the input factors that contribute most towards the generated risk score. Participants will have access to the machine learning dashboard only.