Diagnostic Reasoning With Customized GPT-4 Model

Stanford University

Status: Completed

Conditions: Disease; Pathologic Processes

Treatments: Other: Immediate access to customized version of GPT-4; Other: Access to customized version of GPT-4 following use of conventional resources

Study type: Interventional

Funder types: Other

Identifiers

NCT06911645

71319c

Details and patient eligibility

About

This study will assess the impact of immediate access to a customized version of GPT-4, a large language model, on performance in case-based diagnostic reasoning tasks. Specifically, it will compare this approach to a two-step process where participants first use traditional diagnostic decision support tools to support their diagnostic reasoning before gaining access to the customized GPT-4 model.

Full description

Artificial intelligence (AI) technologies, particularly advanced large language models like OpenAI's ChatGPT, have the potential to enhance medical decision-making. While ChatGPT-4 was not specifically designed for medical applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultations. However, its impact on clinicians' diagnostic reasoning remains largely unknown.

Clinical reasoning is a complex process that involves pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could help reduce clinician workload and decrease the likelihood of missed diagnoses. However, ChatGPT-4 was neither developed nor validated for diagnostic reasoning, and it may produce misleading information, including plausible but incorrect conclusions that could misguide clinicians. If not used appropriately, it may fail to improve-and could even hinder-clinical decision-making. Therefore, it is essential to study how clinicians use large language models to support clinical reasoning before integrating them into routine patient care.

This study will examine how immediate access to a customized version of ChatGPT-4 impacts performance on case-based diagnostic reasoning tasks, compared to a stepwise approach. In the stepwise approach, participants will first use traditional diagnostic decision support tools to support their case reasoning before interacting with a customized ChatGPT-4 model, at which point they will have the opportunity to revise their initial answers.

Participants will be randomized into different study arms and will respond to diagnostic cases by providing three differential diagnoses, along with supporting and opposing findings for each. They will also identify their top diagnosis and propose next diagnostic steps. Independent reviewers, blinded to treatment assignment, will evaluate their responses.

Enrollment: 70 patients

Sex: All

Volunteers: Accepts Healthy Volunteers

Inclusion criteria

• Participants must be licensed physicians and have completed at least post-graduate year 1 (PGY1) of medical training.
• Training in Internal medicine, family medicine, or emergency medicine.

Exclusion criteria

• Not currently practicing clinically.
• Participated in one of our previous studies that used the same six diagnostic cases.

Trial design

Primary purpose: Diagnostic

Allocation: Randomized

Interventional model: Parallel Assignment

Masking: Single Blind

---

70 participants in 2 patient groups

Immediate access to customized version of GPT-4

Active Comparator group

Description:

Group will be encouraged to immediately use a customized version of GPT-4.

Treatment:

Other: Immediate access to customized version of GPT-4

Conventional resources first, then granted access to customized version of GPT-4.

Active Comparator group

Description:

Group will be encouraged to first use any resources they wish besides large language models (UpToDate, Pubmed, google, etc) and then will be granted access to a customized version of GPT-4.

Treatment:

Other: Access to customized version of GPT-4 following use of conventional resources