Clinical Research Directory

What Should we Measure? A Qualitative Interview Study on Outcomes for Evaluating AI-assisted Medical Image Interpretation.

Sponsor: University of Oxford

Summary

Background: AI (artificial intelligence) tools are increasingly used to help clinicians interpret medical images such as X-rays, scans, skin photographs and eye images. In most cases the AI does not work alone: a clinician makes the final decision, using the AI as support. Before these tools are used in the NHS, they must be tested to show they are safe and helpful. They are usually tested in a "reader study", where several clinicians each interpret the same set of images, sometimes with the AI's help and sometimes without, so performance can be compared. These studies tend to focus on whether the diagnosis was correct. However, accuracy alone may not measure everything that matters. For example, what happens when the AI gets it wrong, whether clinicians rely on it too much or too little, or how the tool affects efficiency. At present there is no agreed standard for what these studies should measure, so important things are often missed and studies are hard to compare. Aim: The aim of this study is to find out, by talking to people with different perspectives, which outcomes are important when evaluating AI tools that help clinicians read medical images. Objectives: 1. Explore which outcomes people think matter when judging whether an AI tools is safe and good enough to use 2. Understand the reasons behind their views 3. Explore how priorities differ between groups, such as patients and developers. The output will be a list of important candidate outcomes, which will feed into a future project to agree a recommended set of outcomes for these studies (a "core outcome set"). Research value: Without agreement on what to measure, studies of AI tools may overlook outcomes that matter, and their results are difficult to compare and combine. Hearing directly from the people affected, rather than relying only on what previous studies happened to measure, helps ensure the recommended set reflects people's priorities. This will lead to better-designed studies and AI tools that are safer and more useful in practice. Outline of the research: The investigators will interview 24-30 people who have a stake in how these tools are tested and used, including clinicians, researchers, developers, people involved in regulation, and patients and members of the public. Interviews will last 45-60 minutes and will take place using Microsoft Teams. The investigators will ask people what they think is important to measure and why, and then explore which of those things could realistically be measured in a reader study. Risks/benefits: This is a low-risk study, with no direct benefits to participants. The topics concern methodological matters rather than people's personal health, so personal distress is not anticipated. Taking part is voluntary: interviewees can skip any question or stop at any time. The wider benefit will be better testing of AI imaging tools in future. Impact: The interviews will inform the candidate list of outcomes for the future project and the investigators will publish and present the findings.

Official title: What Should we Measure? A Qualitative Interview Study on Outcomes for Evaluating AI-assisted Medical Image Interpretation

Key Details

Conditions

Locations (1)