Simulation Training in Emergency Department Imaging 2 (STEDI2)

NHS Trust

Status: Completed

Conditions: Stroke; Brain Tumor; Intracranial Bleed; Brain Herniation; Skull Fractures

Treatments: Other: Online training

Study type: Interventional

Funder types: Other

Identifiers

NCT05427838

310995

Details and patient eligibility

About

Background and study aims:

Computerised Tomography (CT) head scans are frequently requested by Emergency Department (ED) clinicians as one of the investigations for their patients. This often causes a delay when waiting for specialist radiologists to report the findings of the scan. The purpose of this study is to see if online training can improve the ability of ED clinicians to interpret the scans themselves, to a level sufficient to make clinical decisions based on their findings and to explore what aspects of this process they find most challenging.

Participants:

Emergency Department clinicians who are working in the Emergency Departments of participating sites between April to September 2022 (inclusive), who request CT Head scans as part of their routine clinical practice.

What does the study involve?:

180 ED clinicians will be recruited across 6 hospital sites in the United Kingdom. All will undertake a baseline online assessment to measure their accuracy in interpreting CT head scans.

One group will then undertake an online training module, with a subsequent assessment immediately afterwards, then over the following 3 months will record interpretations for 30 CT head scans.

Head images encountered in participants' routine clinical practice, and their findings, will be compared with the radiology reports for each scan. Participants will then undertake further online assessments 3 and 6 months after the start of the study. Their overall results will be compared with a control group, who will undergo the same process, but undertake the online training after they have tried to interpret 30 scans in their clinical practice.

Participants will continue to base their clinical decisions on radiologist reports, not their own interpretations, so patient care will not be affected by this study.

Full description

STUDY DESIGN

This is a randomised controlled trial (RCT) to assess the improvement in the image interpretation accuracy and confidence of reporters using online simulation training for CT head scan interpretation.

Using the RAIQC online platform (www.raiqc.com), the trial participants will be required to complete a baseline assessment where users will review and interpret a pre-existing set of CT scans, which have been curated and reported by senior radiologists. They will be asked to record their interpretation, which will be subsequently compared against that of the radiologists. The participants will be blinded to both original reports for the images, and their overall performance results. After completing the baseline assessment, participants will be given access to an online training package. After completing the training, the participants will complete a further assessment. The improvement in diagnostic accuracy pre- and post-training will be measured.

Once the participants have been trained, they will each be asked to evaluate a minimum total of 30 CT scans during clinical shifts spread over a 3-month period. For each CT scan included they will be asked to record their scan interpretation, the time the scan was performed, the time they reviewed the scan and the time the radiology report was issued. The radiology report will be considered the gold-standard diagnosis. Online assessments will be repeated at the end of the 3 month period and at 6months post training, to monitor changes in performance over time.

Subsequent comparison and analysis will aim to measure the real-world accuracy of ED staff at interpreting CT head scans. In addition, the investigators will also be able to measure potential time saving by ED staff interpreting the scans compared with the radiology report being issued.

Population:

- 180 Emergency Department clinicians (30 per site) of varying grades who routinely interpret images as part of their usual role, including: i. Junior Doctors (F1-ST2) ii. Registrars (ST3-ST6) iii. Consultants iv. Radiographers v. Nursing and Allied Health Professionals (NMAHPs) - Advanced Nurse Practitioners, Physician Associates

Methods:

Participants will be randomised to control (see section below - no online training, 30 participants, 5 from each site) versus intervention (online training - 150 participants) groups. Prior to the start of the study, each participant ID will be randomised to either grou via simple randomisation using an online number generator - the outcome of this allocation will be recorded on a linkage document held by the central study team in Oxford.

Intervention Group:

• All intervention group participants will first be required to complete a baseline online assessment during their non-clinical time, which will consist of CT head images with a clinical vignette of varying difficulty and pathology where participants will review a set of scans and provide a diagnosis and confidence score before submitting the case. Time taken to complete each individual case will be recorded by the RAIQC platform. Participants will be blinded to the results.

• Participants will be given access to an existing bespoke online training module on the RAIQC platform (www.raiqc.com) with a post-training assessment consisting of a different set of cases.

• Participants will then be required to evaluate a minimum of 30 CT Head scans each - encountered during their clinical shifts over a 3-month period. They will be asked to record results in real time on a standardised eCRF, including:

  • date/time at which CT scan was performed
  • date/time of participant review
  • participant interpretation and confidence score
  • date/time of radiology report
  • radiology report findings (reference-standard)

NB: Participants will be instructed to base their subsequent clinical decision making based on the formal radiology report as per standard clinical practice, not their own interpretation.

• Participants will then each repeat a 50-case assessment at the end of the 3-month period and again at 6 months post-training (to assess skill retention)
• Qualitative surveys will be completed by participants prior to each assessment module to obtain feedback and comment regarding their experiences of the project and monitor any changes over the course of the study. Participants will be issued with a certificate to recognise that they have undergone imaging training for Continuous Professional Development purposes

Controls:

• To provide a control for comparison as to the relative benefits of the online training phase and the clinical interpretation phase in terms of improving reporting performance, 30 participants (5 participants from each site) will be randomised to undertake the clinical interpretation phase prior to the training module
• Baseline accuracy statistics from each participant will be compared to subsequent results from each stage of the study

Analysis:

- Training:

i. Changes in reporting accuracy (sensitivity/specificity), confidence and speed will be calculated in a pooled analysis for all readers at all sites, as well as for the following subgroups:

• Clinical role
• Level of seniority
• Pathological finding
• Difficulty of image

ii. These will be repeated for each of the four sequential assessments

- Clinical Interpretation:

i. Accuracy (sensitivity/specificity), confidence and speed will be calculated for all pooled data, and the following subgroups:

• Clinical role
• Level of seniority
• Pathological finding

ii. Total and mean potential time saving will be calculated by comparing time of participant interpretation versus time of radiology report, and will be used to calculate potential cost benefit of clinician reporting versus radiology reporting

iii. Results from the qualitative surveys will be collated and used to explore the experiences and factors affecting the uptake of new image reporting roles in the ED clinical population

PARTICIPANT IDENTIFICATION

Informed Consent:

Participants will be provided with an electronic copy of the PIS via email. They will express an interest in participating in the study to the site Principal Investigator, who will discuss the study further with them and answer any questions. Consent will be sought from participants to participate in the study and for their anonymised results to be used for analysis and publication purposes. If they agree to the study, they will then be asked to complete the consent form online and return it to the site Principal Investigator.

Discontinuation/Withdrawal of Participants from Study:

Each participant has the right to withdraw from the study at any time. In addition, the Investigator may discontinue a participant from the study at any time if the Investigator considers it necessary for any reason including:

• Ineligibility (either arising during the study or retrospectively having been overlooked at screening)
• Significant protocol deviation
• Significant non-compliance with training regimen or study requirements
• Withdrawal of Consent
• Loss to follow up All data obtained to that point will be retained for analysis Withdrawn participants will not be replaced. The reason for withdrawal will be recorded in the CRF.

STATISTICS AND ANALYSIS

Sample Size:

During the pilot study, the diagnostic accuracy of participants in detecting acute intracranial abnormality improved by a mean of 8% (SD15%). The investigators calculated a sample size of 30 participants using a power of 80% and a level of significance of 5% (two sided). This was a realistically and practically achievable target for each of the six sites, and 180 participants was, therefore, chosen as the overall target for the current study.

Statistical Analyses:

• Training Assessments:

  1) Changes in reporting accuracy/sensitivity/specificity/inter-reader variability/confidence/speed will be calculated in a pooled analysis for all readers at all sites, as well as for the following subgroups:

• Clinical role

• Level of seniority

• Pathological finding

• Difficulty of image

  2) These will be repeated for each of the four sequential assessments

• Clinical Interpretation Phase:

  1. Accuracy/sensitivity/specificity/inter-reader variability/confidence will calculated for all pooled data, and the following subgroups

     Clinical role Level of seniority Pathological finding

  2. Total and mean potential time saving will by calculated by comparing time of participant interpretation versus time of radiology report, and will be used to calculate potential cost benefit of clinician reporting versus radiology reporting

  3. Results from the qualitative surveys will be collated and used explore the experiences and factors affecting the uptake of new image reporting roles in the ED clinical population

Enrollment: 173 patients

Sex: All

Volunteers: Accepts Healthy Volunteers

Inclusion criteria

1. Emergency department clinicians who review CT head scans as part of their clinical practice.
2. Working in the relevant ED during the clinical phase of the project.

Exclusion criteria

1. Previous formal postgraduate CT reporting training.
2. Previous career to registrar level in radiology/neurosurgery.
3. Staff in Emergency Department post for less than 4 months.

Trial design

Primary purpose: Diagnostic

Allocation: Randomized

Interventional model: Crossover Assignment

Masking: Single Blind

---

173 participants in 2 patient groups

Intervention (training)

Experimental group

Description:

Users will access an online training module on the RAIQC simulation platform, featuring didactic tutorials and interactive case-based simulations on CT head pathologies such as haemorrhage, infarction, mass effect, and hydrocephalus. The training, designed to take 2-4 hours, is completed during protected non-clinical time. After training, users complete a post-training assessment with 50 different CT cases to evaluate immediate learning gain, followed by a second assessment. Over the next 3 months, users interpret at least 30 CT scans each during clinical shifts, recording their interpretations, the time scans were performed and reviewed, radiology report findings, and report issue times. At the end of 3 months, they repeat the assessment to measure performance, and again after a further 3 months to assess retention of any improvements.

Treatment:

Other: Online training

Control (no Training)

No Intervention group

Description:

To serve as a control for comparing the benefits of the online training and clinical interpretation phases on reporting performance, 30 participants (6 clinicians per site) will be randomized to complete the clinical interpretation phase before the training module. These subjects will not undergo online CT head interpretation training initially.