Multi-Disciplinary Treatment on the Anthropomorphism of Large Language Models (MDTALLM)

North Sichuan Medical College

Status: Not yet enrolling

Conditions: Respiratory Failure; Disease; Pneumonia; Infections; Cancer; Heart Diseases

Treatments: Diagnostic Test: GPT-4o mini; Diagnostic Test: MedicalGPT; Diagnostic Test: Claude 3 Haiku; Diagnostic Test: GPT-4o; Diagnostic Test: Claude-3.5 Sonnet; Diagnostic Test: Real Doctors

Study type: Observational

Funder types: Other

Identifiers

NCT06627985

1426887-2024-3

Details and patient eligibility

About

This retrospective clinical trial aims to better explore the potential of large language models in medicine by comparing the effectiveness of MDT consultations conducted by human doctors with those conducted by large language models.

The main questions to be addressed are:

Does using large language models to conduct anthropomorphic MDT consultations yield better results than using non-anthropomorphic processes? Is there a significant performance gap between MDT consultations conducted by large language models and those conducted by humans? How much greater is the economic benefit of MDT consultations from large language models compared to those conducted by humans?

Retrospectively collect MDT consultation records from the past 20 years in northern Sichuan in China, as well as anonymized patient medical records. Group 1: Different large language models are assigned to act as doctors from different departments and as MDT secretaries to summarize consultations. Group 2: The large language model directly outputs diagnostic and treatment recommendations for patients. Compare the outputs of groups 1 and 2 with human performance retrospectively, score them, and select the best model from each department for a re-evaluation through anthropomorphic MDT consultations, once again comparing them to human results.

Enrollment: 300 estimated patients

Sex: All

Volunteers: No Healthy Volunteers

Inclusion criteria

• 1. The medical records include interdisciplinary consultation notes, with recommendations from specialists of various departments and a well-documented final summary.
• 2. The medical records contain data from at least one year prior to and one year following the consultation (including intact reports and imaging records).
• 3. The patient's discharge conditions improved due to the multidisciplinary treatment plan after the consultation.

Exclusion criteria

• 1. The medical records do not include multidisciplinary consultation notes, or the recommendations from various departmental physicians and the final summary notes are incomplete or inadequate.
• 2. The medical records lack data from 1 year before and after the consultation, or miss necessary reports and imaging data, resulting in incomplete documentation.
• 3. The patient's condition at discharge has not improved following the multidisciplinary treatment plan, or the condition has worsened.

Trial design

300 participants in 4 patient groups

Anthropomorphized Process Large Language Model Multidisciplinary Treatment Group

Description:

Using a locally deployed MedicalGPT, the commercially available online GPT-4o, Claude-3.5 Sonnet, GPT-4o mini, and Claude 3 Haiku, will each sequentially play the role of physicians from different departments involved in the Multi-Disciplinary Treatment Process. They will then sequentially take on the role of a summarizer to compile their recommendations into a final suggestion or treatment plan.

Treatment:

Diagnostic Test: Claude-3.5 Sonnet

Diagnostic Test: GPT-4o

Diagnostic Test: Claude 3 Haiku

Diagnostic Test: MedicalGPT

Diagnostic Test: GPT-4o mini

Non-anthropomorphized Process Large Language Model Multidisciplinary Treatment Group

Description:

Using a locally deployed MedicalGPT, the commercial online GPT-4o, Claude-3.5 Sonnet, GPT-4o mini, and Claude 3 Haiku to output multidisciplinary consultation results in a single instance, without separately assuming roles for each department and then compiling the results.

Treatment:

Diagnostic Test: Claude-3.5 Sonnet

Diagnostic Test: GPT-4o

Diagnostic Test: Claude 3 Haiku

Diagnostic Test: MedicalGPT

Diagnostic Test: GPT-4o mini

Real Doctors Multi-Disciplinary Treatment Group

Description:

In traditional multidisciplinary treatments, the results are documented in the consultation records of the patients involved, including the recommendations from doctors of various departments who participated in the consultation and the final summary by the secretary.

Treatment:

Diagnostic Test: Real Doctors

Best Large Language Model Multidisciplinary Treatment Group

Description:

After scoring the results of the Anthropomorphized Process Large Language Model Multidisciplinary Treatment Group against the outcomes of the Real Doctors' Multi-Disciplinary Treatment Group on a department-by-department basis, the best substitute models and the best summary models for each department were selected. These top models are set to assume roles in a Multi-Disciplinary Treatment consultation.

Treatment:

Diagnostic Test: Claude-3.5 Sonnet

Diagnostic Test: GPT-4o

Diagnostic Test: Claude 3 Haiku

Diagnostic Test: MedicalGPT

Diagnostic Test: GPT-4o mini