Evaluating the Potential of Large Language Models for Respiratory Disease Consultations (EPLLMMRDC)

North Sichuan Medical College

Status: Completed

Conditions: Pulmonary Fibrosis; Pulmonary Embolism; Bronchiectasis; Acute Bronchitis; Asthma; Tuberculosis; Lung Cancer; Acute Upper Respiratory Infection; Pneumonia; Hay Fever

Treatments: Diagnostic Test: Diagnosis by Claude instant (without search capabilities); Diagnostic Test: Diagnosis by Claude instant (with search capabilities); Diagnostic Test: Diagnosis by ChatGPT-4.0 (without search capabilities); Diagnostic Test: Diagnosis by Gemini Pro (with search capabilities); Diagnostic Test: Diagnosis by ChatGPT-4.0 (with search capabilities); Diagnostic Test: Diagnosis by ChatGPT-3.5 (with search capabilities); Diagnostic Test: Diagnosis by Claude 2 (with search capabilities); Diagnostic Test: Diagnosis by Claude 2 (without search capabilities); Diagnostic Test: Diagnosis by three human doctors; Diagnostic Test: Diagnosis by ChatGPT-3.5 (without search capabilities); Diagnostic Test: Diagnosis by Gemini Pro (without search capabilities)

Study type: Interventional

Funder types: Other

Identifiers

NCT06457269

2022LC005 (Other Grant/Funding Number)

1426887-2024-1

CBY22-QDA15 (Other Grant/Funding Number)

23JCYJPT0014 (Other Grant/Funding Number)

22XQT0309 (Other Grant/Funding Number)

Details and patient eligibility

About

The clinical trial aimes to evaluate multiple large language models in respiratory disease consultations by comparing their performance to that of human doctors across three major medical consultation scenarios.

The main question aims to answer are:

• How do large language models perform in comparison to human doctors in diagnosing and consulting on respiratory diseases across various clinical scenarios?

In three clinical scenarios including the online query section, the disease diagnosis section and the medical explanation section, research assistants or volunteers will be asked to cross-question all LLMs or real doctors using predefined online questions and their own issues. After each questioning session, a short washout period is implemented to eliminate potential biases.

Enrollment: 703 patients

Sex: All

Volunteers: No Healthy Volunteers

Inclusion criteria

1. Self-reported symptoms of common respiratory diseases, such as cough, chest tightness, fever, and wheezing
2. Ability to engage in LLM dialog operations independently or with minimal peer training
3. A health status deemed suitable for study participation by the pulmonology experts

Exclusion criteria

1. Excessively poor health status

Trial design

Primary purpose: Diagnostic

Allocation: Randomized

Interventional model: Crossover Assignment

Masking: Quadruple Blind

---

703 participants in 2 patient groups

Cross-comparison group(the disease diagnosis section)

Other group

Description:

Cross-comparison group (including human doctor controls and all LLMs)

Treatment:

Diagnostic Test: Diagnosis by Gemini Pro (without search capabilities)

Diagnostic Test: Diagnosis by ChatGPT-3.5 (without search capabilities)

Diagnostic Test: Diagnosis by Claude 2 (without search capabilities)

Diagnostic Test: Diagnosis by three human doctors

Diagnostic Test: Diagnosis by Claude 2 (with search capabilities)

Diagnostic Test: Diagnosis by ChatGPT-3.5 (with search capabilities)

Diagnostic Test: Diagnosis by ChatGPT-4.0 (with search capabilities)

Diagnostic Test: Diagnosis by Gemini Pro (with search capabilities)

Diagnostic Test: Diagnosis by ChatGPT-4.0 (without search capabilities)

Diagnostic Test: Diagnosis by Claude instant (with search capabilities)

Diagnostic Test: Diagnosis by Claude instant (without search capabilities)

Cross-comparison group(the medical explanation section)

Other group

Description:

Cross-comparison group (including human doctor controls and all LLMs)

Treatment:

Diagnostic Test: Diagnosis by Gemini Pro (without search capabilities)

Diagnostic Test: Diagnosis by ChatGPT-3.5 (without search capabilities)

Diagnostic Test: Diagnosis by Claude 2 (without search capabilities)

Diagnostic Test: Diagnosis by three human doctors

Diagnostic Test: Diagnosis by Claude 2 (with search capabilities)

Diagnostic Test: Diagnosis by ChatGPT-3.5 (with search capabilities)

Diagnostic Test: Diagnosis by ChatGPT-4.0 (with search capabilities)

Diagnostic Test: Diagnosis by Gemini Pro (with search capabilities)

Diagnostic Test: Diagnosis by ChatGPT-4.0 (without search capabilities)

Diagnostic Test: Diagnosis by Claude instant (with search capabilities)

Diagnostic Test: Diagnosis by Claude instant (without search capabilities)