The Diagnostic and Triage Capacity of Laypeople-large Language Model Collaboration in China

Huazhong University of Science and Technology

Status: Completed

Conditions: LLM-based AI Dialogue Bot; Vignette Based Intervention

Treatments: Behavioral: Conventional internet search for health information; Behavioral: AI-assisted health information seeking

Study type: Interventional

Funder types: Other

Identifiers

NCT07250516

JCYJ20240813115806009

Details and patient eligibility

About

The goal of this randomized controlled trial is to evaluate the role of large language models in enhancing laypeople's ability to self-diagnose and triage common diseases. The main questions it aims to answer are:

• Does using an LLM help participants make more accurate self-diagnoses and care decisions for common illnesses, compared to their first guess without any help?
• How much better is it when people work together with an LLM, compared to using a regular search engine, using the LLM alone, or how doctors would decide? Researchers will compare participants who were randomly assigned to either the LLM group (using DeepSeek) or the search engine group to see if the LLM-assisted approach leads to better clinical judgments.

Participants will:

• Read one of 48 short, realistic health vignettes;
• Make an initial guess about what might be wrong by listing up to three possible causes, ranked from most to least likely, and choose a care level: seek immediate care, see a doctor within one day, see a doctor within one week, or manage at home without medical care.
• Use their assigned tool (either DeepSeek or a standard search engine) to look up information and update their guess and care decision;
• Submit their final diagnosis and care choice after using the tool. In addition, the study team evaluated the performance of four other AI models (GPT-4o, GPT-o1, DeepSeek-v3, and DeepSeek-r1) and 33 experienced general physicians on the same vignettes.

Enrollment: 6,360 patients

Sex: All

Ages: 18+ years old

Volunteers: No Healthy Volunteers

Inclusion criteria

• Age 18 years or older
• Current resident of mainland China
• History of high-quality participation in online surveys on Credamo platform (historical survey acceptance rate ≥ 80% and personal credit score ≥ 70)

Exclusion criteria

• Incomplete survey responses
• Failure on embedded quality-check items
• Implausibly short completion time (<180 seconds for search engine group; <360 seconds for LLM group)
• Provision of non-diagnostic or irrelevant responses (e.g., "unknown", "don't know")
• Consistent pattern of identical responses across all items

Trial design

Primary purpose: Health Services Research

Allocation: Randomized

Interventional model: Parallel Assignment

Masking: Single Blind

---

6,360 participants in 2 patient groups

layperson-LLM integrated group

Experimental group

Description:

After initially answering a clinical diagnosis and triage question without the aid of tools, the participants were asked to use a large language model (Deepseek v3 or r1) to retrieve health information and then answer the same question again

Treatment:

Behavioral: AI-assisted health information seeking

layperson-search engine group

Active Comparator group

Description:

After initially answering a clinical diagnosis and triage question without the use of tools, the participants were required to use a search engine to retrieve health information and then answer the same question again

Treatment:

Behavioral: Conventional internet search for health information