Orchestrated Multi-Agent AI Versus Consultant Ophthalmologist: Workflow Performance in Refractive Surgery and Keratoconus Diagnosis-Demonstrating Increased Diagnostic Accuracy (AEYE)

hatem ali

Status: Invitation-only

Conditions: Diagnostic Accuracy; Keratoconus; Ophthalmology; Machine Learning; Artifical Intelligence; Clinical Decision Support; Refractive Surgery

Treatments: Diagnostic Test: Multi-Agent AI Diagnostic Workflow (AEYE)

Study type: Observational

Funder types: Other

Identifiers

NCT07096232

HYC-AI-RS-2025-01

Details and patient eligibility

About

Background and Rationale:

Laser vision correction surgery, including LASIK and other refractive procedures, is highly effective for correcting vision but requires thorough preoperative screening. One of the most critical steps is identifying patients who may have keratoconus or other forms of corneal ectasia-a condition where the cornea thins and bulges outward, leading to progressive vision loss and possible complications if undiagnosed before surgery. Early and accurate detection of keratoconus is essential to prevent vision-threatening outcomes and to ensure optimal patient safety.

Despite advances in diagnostic imaging and analytics, detecting subtle or early-stage keratoconus remains challenging, even for experienced ophthalmologists. Human interpretation of complex imaging data-such as corneal topography, tomography, and biomechanical analysis-can vary between clinicians and is subject to error, especially in borderline or ambiguous cases. Artificial intelligence (AI) has the potential to enhance the accuracy, consistency, and efficiency of keratoconus screening and refractive surgery planning.

Purpose of the Study:

This study evaluates the diagnostic performance and clinical utility of a novel AI system called AEYE (Automated Evaluation for Your Eye), designed to support ophthalmologists in screening for keratoconus and planning laser vision correction procedures. AEYE is an orchestrated multi-agent AI workflow: each "agent" is specialized in a specific task, such as interpreting corneal scans, reviewing patient histories, or providing diagnostic suggestions. Working together, these digital agents aim to replicate and augment the comprehensive review process typically performed by experienced refractive surgeons.

Study Design and Methods:

The study will analyze a total of 50 real-world patient cases, comprising both retrospective and prospective records of individuals evaluated for laser vision correction or suspected keratoconus. For each case, the following workflow will be applied:

1. AI Evaluation:

   The AEYE system will independently assess all available clinical data, including patient history, slit-lamp findings, corneal imaging (e.g., Scheimpflug tomography, epithelial mapping), and relevant risk factors. The AI agents will collectively produce a diagnostic assessment-identifying the presence or absence of keratoconus, evaluating risk for post-operative ectasia, and recommending suitability for refractive surgery.

2. Human Expert Review:

   A consultant ophthalmologist, blinded to the AI output, will independently review the same data and make clinical decisions regarding diagnosis and treatment planning.

3. Outcome Comparison:

The study's main aim is to compare the performance of AEYE against the human expert, focusing on:

• Diagnostic accuracy for detecting keratoconus or suspect ectasia
• Concordance between AI and clinician recommendations for surgery eligibility
• Utility in surgical planning and case complexity
• Reliability and safety in real-world clinical scenarios

Both retrospective cases (using patient files and imaging studies from 2020 onward) and newly recruited prospective cases will be included, capturing the diversity of presentations encountered in a high-volume refractive surgery practice.

Anticipated Impact:

By analyzing the agreement between AEYE's multi-agent workflow and established clinical practice, the study will provide evidence on whether AI tools can safely assist ophthalmologists in complex diagnostic decisions. We seek to determine:

• If AEYE can improve sensitivity and specificity in detecting keratoconus
• If workflow automation reduces diagnostic errors or increases consistency
• If such systems can streamline surgical planning, especially for difficult or borderline cases

Our long-term goal is to support the adoption of advanced AI assistants in ophthalmology clinics-making eye care safer, more reliable, and less dependent on the variable experience of individual clinicians. If successful, AEYE and similar platforms could become valuable tools for standardizing refractive surgery screening and protecting patients from preventable complications.

Lay Summary:

This research tests whether a new artificial intelligence system (AEYE) can help eye doctors catch early signs of keratoconus and make safer decisions about laser eye surgery. By combining the strengths of digital technology and clinical expertise, we hope to improve eye care for all patients needing vision correction.

Full description

Study Overview

This is a prospective-retrospective diagnostic performance evaluation of AEYE, a novel, orchestrated multi-agent artificial intelligence (AI) system developed to support comprehensive clinical assessment, diagnosis, and surgical planning for patients evaluated for refractive surgery and/or keratoconus. The study benchmarks the accuracy, concordance, and efficiency of AEYE's workflow against experienced consultant ophthalmologists, utilizing real-world multimodal clinical and imaging data from both historical (retrospective) and newly enrolled (prospective) cases.

Background and Rationale

Keratoconus and related corneal ectatic disorders are major contraindications for refractive surgery. Missed or delayed diagnosis can lead to post-surgical ectasia, a devastating complication with significant visual morbidity. Despite advanced corneal imaging technologies such as Scheimpflug tomography, anterior segment optical coherence tomography (AS-OCT), and epithelial thickness mapping, accurate diagnosis remains dependent on clinician expertise and is subject to human error and variability, especially in borderline or subtle cases.

Artificial intelligence (AI) has shown promise in ophthalmology, but most current applications are narrow, device-specific algorithms or opaque deep learning models. AEYE represents a new paradigm: an orchestrated, explainable, modular, multi-agent system designed to emulate the logic and workflow of expert clinicians, but with enhanced consistency, reproducibility, and auditability. AEYE acts as a strong clinical advisor, supporting both experienced surgeons and junior doctors by structuring, validating, and documenting each step of the diagnostic and decision-making process.

AEYE System Architecture

AEYE is built as a three-agent system, each powered by large language models (LLMs) or equivalent AI engines, operating sequentially under deterministic Python-based control function and workflow steps:

History & Risk Analysis Agent:

Parses structured and unstructured clinical records to extract demographic data, chief complaint, refraction, ocular and systemic history, risk factors, and prior interventions.

Automates checklists for risk assessment (e.g., medication alerts, systemic associations).

Flags potential red flags for keratoconus and refractive surgery contraindications.

Imaging Analysis Agent:

Processes and standardizes corneal tomography (Pentacam, Sirius), AS-OCT, and epithelial mapping scans.

Extracts and validates quantitative imaging parameters per eye (e.g., Kmax, SimK, thinnest pachymetry, anterior/posterior elevation, BAD-D, PPI, ARTmax).

Integrates multimodal data, prioritizes robust and reliable metrics, and ensures per-image, per-eye isolation to avoid data misattribution.

Surgical Decision Agent:

Synthesizes outputs from the prior agents. Applies international guidelines and safety thresholds. Assigns diagnosis and stage of keratoconus (using ABCD, Amsler-Krumeich, or similar systems).

Recommends surgical eligibility (LASIK, PRK, SMILE, ICL, or other), flags for corneal cross-linking (CXL), and identifies cases needing further evaluation or contraindications to surgery.

Outputs a structured, auditable report with clinical rationale.

All agents operate within a deterministic, code controlled workflow that enforces schema validation, loop control, and standardized handoffs. Python-based modules manage data merging, validation, error flagging, and report formatting. All outputs are logged for audit, reproducibility, and quality assurance.

Study Design and Workflow Steps

Design:

Prospective-retrospective, non-randomized, diagnostic accuracy study.

Sample:

50 real world patient cases, including both: Retrospective arm: Cases evaluated for refractive surgery or keratoconus from January 2020 onward, identified from clinical records and imaging databases.

Prospective arm: Newly enrolled patients presenting for refractive surgery assessment or keratoconus workup during the study period, with data and imaging collected at the time of evaluation.

Workflow Steps:

Case Selection Identification of eligible retrospective and prospective cases based on clinical indication and completeness of data.

Confirmation of inclusion/exclusion criteria. Data Compilation Extraction of relevant clinical records: patient history, refraction, ophthalmic examination, and prior surgical data.

Collection and assignment of multimodal imaging files (corneal tomography, AS-OCT, epithelial mapping).

AI Workflow Analysis (AEYE) AEYE receives structured clinical and imaging data for each case.

The three-agent AI system processes each case sequentially:

History/Risk Agent reviews and structures risk profile. Imaging Agent extracts, validates, and standardizes imaging parameters for each eye.

Decision Agent integrates all data, assigns diagnosis and stage, provides surgical recommendations, and flags uncertainties.

AEYE's outputs are stored, timestamped, and include detailed structured reports and decision logic.

Consultant Ophthalmologist Review An experienced consultant anterior segment ophthalmologist, blinded to AEYE's output, independently reviews the same clinical and imaging data for each patient.

The consultant records their diagnosis, keratoconus staging (if applicable), and surgical recommendations using standard clinical protocols.

Outcome Comparison & Adjudication Diagnostic results, recommendations, and discrepancies between AEYE and the consultant are compared for accuracy and concordance.

Discordant or ambiguous cases may be reviewed by an adjudication panel or through multidisciplinary consensus to establish the reference standard.

Data Analysis Statistical analysis is conducted to compare diagnostic accuracy, inter-rater agreement, workflow efficiency, and subgroup outcomes (e.g., keratoconus stage, surgical eligibility).

Additional analyses include inter-model variability (where different LLMs were used), time-to-decision, and error rates.

Eligibility Criteria

Inclusion Criteria:

Patients evaluated for refractive surgery (LASIK, PRK, SMILE, ICL, lens-based procedures) or for keratoconus assessment at the study center.

Availability of complete clinical records and required imaging data (Pentacam, AS-OCT, or equivalent).

No restriction on age or prior refractive surgery, unless specified for a planned subgroup analysis.

Informed consent obtained for prospective cases; waiver of consent for retrospective records per IRB/ethics policy.

Exclusion Criteria:

Missing or poor quality imaging or clinical data that preclude assessment. Coexisting ocular or systemic conditions that significantly confound the diagnosis (e.g., corneal dystrophies other than keratoconus, advanced glaucoma).

Inability or unwillingness to provide consent for prospective enrollment.

Objectives and Endpoints

Primary Objective:

To compare the diagnostic accuracy of AEYE versus consultant ophthalmologists for detecting and staging keratoconus, using accepted clinical and imaging reference standards.

Secondary Objectives:

To evaluate concordance between AEYE and human experts in refractive surgery eligibility, surgical planning, and treatment recommendations.

To assess AEYE's ability to flag risk factors, handle ambiguous/borderline cases, and minimize diagnostic oversight.

To measure workflow efficiency, including speed (turnaround time) of AI versus human review.

To analyze inter-model variability (different LLMs within agent roles) and error rates.

To conduct subgroup analysis by keratoconus severity, surgical type, and case complexity.

Data Management, Validation, and Quality Assurance Schema Validation: All outputs are formatted in a standardized, structured schema (e.g., JSON), using clinical variable naming conventions. Missing values are flagged explicitly.

Imaging Isolation: Imaging data is processed and stored per eye and per file to prevent data leakage or misattribution (OD vs OS).

Aggregation and Memory: Modular variable storage is maintained for each case, including history, imaging, interpretation, and diagnosis. Data merging is handled by deterministic code prioritizing the highest-confidence modality or parameter.

Auditability: All outputs, decision paths, and errors are logged for reproducibility and quality assurance.

Missing Data: If fields are missing, agents report "Not available" or null. No data imputation is performed without explicit protocol logic.

Statistical Analysis

Agreement and Accuracy:

• Sensitivity, specificity, PPV, NPV, and overall diagnostic accuracy for keratoconus detection and surgical contraindication.
• Cohen's Kappa coefficient and percent agreement for AI-human concordance.
• Analysis of discordant/ambiguous cases by adjudication panel.
• Workflow Efficiency:
• Median time to decide (AI vs human).
• Inter-Model Variability:
• Standard deviation and agreement between different LLMs used in agent roles.
• Subgroup Analysis:
• Outcomes stratified by keratoconus severity, surgical recommendation, and case complexity.

Ethics, Data Security, and Oversight

• All patient data are de-identified and stored in secure, access-controlled systems.
• Retrospective analysis is conducted under IRB/ethics committee waiver of consent; all prospective patients provide informed consent prior to enrollment.
• The study complies with the Declaration of Helsinki, GDPR/local privacy regulations, and relevant institutional requirements.
• Final clinical decisions remain the responsibility of human clinicians; AEYE serves as a decision support tool and is not used for independent patient management.

Innovation and Clinical Vision

AEYE's orchestrated, agentic AI framework enables explainable, structured support across the entire refractive surgery and keratoconus evaluation workflow. Its modular design allows adaptation to new imaging modalities, disease entities, or clinical requirements. AEYE is designed not as a replacement but as a robust clinical advisor supporting both routine and complex decision making, reducing variability, and enhancing patient safety.

Future Directions

Upon successful validation, AEYE may be expanded for multicenter studies, larger-scale deployment, and integration into electronic medical record systems for real-time screening and decision support. Additional agents may be developed for pediatric keratoconus, rare disease phenotypes, or other ophthalmic domains (e.g., retinal diagnostics, glaucoma risk assessment). The ultimate goal is scalable, explainable AI support for ophthalmologists and other medical specialties worldwide.

Enrollment: 50 estimated patients

Sex: All

Volunteers: No Healthy Volunteers

Inclusion criteria

1. Diagnosis of Refractive Error or Keratoconus:

   • Patients with a confirmed clinical diagnosis of refractive errors (myopia, hyperopia, or astigmatism) or keratoconus, as determined by a qualified ophthalmologist based on current diagnostic standards.

2. Availability of Complete Data:

   • Clinical records must include comprehensive demographic data, medical and ophthalmic history, refraction measurements, and high-quality corneal imaging (e.g., corneal topography, tomography, or OCT).

3. Eligible for Both AI and Consultant Review:

   • Patients must have data sets that allow both the multi-agent AI workflow (AEYE) and an independent consultant ophthalmologist to conduct a complete diagnostic assessment.

4. Consent:

   • For prospectively enrolled patients, written informed consent must be obtained prior to participation. For retrospective cases, waiver of consent may be granted according to local IRB/ethics committee policies.

5. No Restriction on Age or Sex:

   • There are no specific age or sex limitations for inclusion in this study. Pediatric and adult cases may be included if data are available and consent is appropriately obtained.

Clinical Documentation Requirements:

• Availability of complete ophthalmic records including:
• Uncorrected and best corrected visual acuity (UCVA, BCVA)
• Manifest refraction (sphere, cylinder, axis)
• History of contact lens use and duration
• Prior ocular surgery or trauma, if applicable
• Family history of keratoconus or corneal ectasia
• Use of medications that may influence tear film stability or corneal biomechanics
• Ocular surface disease documentation if present (e.g., dry eye, blepharitis)

Imaging and Diagnostic Data Requirements:

• Availability of at least one valid and complete set of:
• Scheimpflug-based corneal tomography (e.g., Pentacam or equivalent)
• Anterior segment OCT (AS-OCT), if performed
• Epithelial thickness mapping (optional but desirable)
• Corneal biomechanical data (optional)
• Imaging files must be:
• Of adequate quality as judged by the original technician and reviewer
• Free from significant artifacts or motion blur
• Properly labeled by eye (OD/OS) and imaging modality
• Acquired with standard scan protocols

Diagnostic Spectrum Requirements:

• Patients may fall into any of the following diagnostic categories:
• Normal cornea eligible for laser refractive surgery
• Suspect keratoconus or forme fruste keratoconus
• Established keratoconus of any stage (I-IV or ABCD)
• Post-CXL (cross-linking) cornea undergoing follow-up evaluation
• Post-keratoplasty cornea (e.g., DALK, PKP) undergoing diagnostic review
• Patients disqualified from surgery due to abnormal tomography or other contraindications.

Exclusion criteria

1. Incomplete or Poor Quality Data:

   • Patients with missing, incomplete, or poor-quality clinical data or corneal imaging that precludes reliable diagnosis by either AI or consultant review.

2. Ocular Comorbidities:

   • Presence of ocular diseases that could confound the diagnosis or interpretation of refractive error or keratoconus, such as advanced glaucoma, active uveitis, significant retinal pathology, or previous corneal transplantation.

3. Severe Systemic Disease Affecting the Eye:

   • Patients with systemic diseases known to affect the cornea or refraction (e.g., connective tissue disorders with corneal involvement) will be excluded to avoid confounding effects.

4. Inability or Refusal to Consent:

   • For prospectively enrolled cases, patients (or guardians, in the case of minors) who are unwilling or unable to provide informed consent will be excluded.

5. Participation in Conflicting Studies:

   • Patients currently enrolled in other interventional studies that could interfere with the diagnostic process or outcomes measured in this study.

Clinical Documentation Exclusions:

• Missing or incomplete visual acuity data.
• Absence of manifest refraction or subjective refraction data.
• Incomplete or conflicting patient history with unresolvable discrepancies.
• Absence of necessary demographic identifiers (e.g., gender, age, laterality of findings).

Imaging and Data Quality Exclusions:

• Imaging datasets with significant artifacts that preclude AI analysis.
• Files missing key parameters (e.g., Belin/Ambrósio D index, Kmax, pachymetry) required for keratoconus classification.
• Inability to determine laterality (OD/OS) for specific scans.
• Scans acquired with outdated or non-standard imaging protocols.
• Mislabeling or duplication of imaging sets across different patients.

Case-Type Exclusions:

• Cases involving active ocular infection or inflammation at the time of evaluation.
• Cases with active herpetic eye disease, corneal ulcer, or epithelial defects interfering with imaging interpretation.
• Prior laser refractive surgery unless explicitly classified and used as part of post-refractive ectasia evaluation.
• Eyes with concurrent advanced retinal pathology that may confound diagnostic interpretation (e.g., macular edema, retinal dystrophies).

Technology-Specific Exclusions:

• Imaging performed using devices not compatible with the AEYE data ingestion framework (e.g., certain older corneal topographers).
• Non-English documentation incompatible with NLP parsing in the General Ophthalmology Agent.

Other Considerations:

• AEYE must be able to process the case within its defined schema. If key data elements are absent or the case causes schema failure, it will be excluded from quantitative analysis but may be reported qualitatively as part of system performance limitations.
• If multiple attempts at AI processing result in non-deterministic or erroneous outputs, such cases will be logged but excluded from primary outcome evaluation.
• Human reviewers (consultant ophthalmologists) must have completed their diagnostic interpretation independently prior to AEYE input; any cases where human opinion was influenced by AI output will be excluded to preserve blinding.

Trial design

50 participants in 1 patient group

Patients with refractive errors or keratoconus assessed by agentic AI workflow and consultant review

Description:

This group includes all patients with refractive errors or keratoconus evaluated at our center during the study period. Each patient case undergoes comprehensive diagnostic assessment using both an orchestrated multi-agent artificial intelligence (AI) workflow (AEYE) and independent review by a consultant ophthalmologist. The AI workflow integrates multiple specialized AI agents to analyze clinical data, imaging studies, and relevant diagnostic metrics. Results from the AI system are compared to consultant assessments to evaluate concordance, diagnostic accuracy, and workflow efficiency. Both retrospective patient records and prospectively enrolled cases are included in this group.

Treatment:

Diagnostic Test: Multi-Agent AI Diagnostic Workflow (AEYE)