Clinical Research Directory

Benchmarking Large Language Models Against Tumour Boards for Oncology Treatment Recommendations

Sponsor: Assistance Publique - Hôpitaux de Paris

Summary

BEACON (Benchmarking AI for Clinical Oncology decisioNmaking) is a prospective, multicentre, comparative, blinded, non-interventional benchmark comparing the treatment recommendations of six frontier large language models (LLMs) with those of multidisciplinary tumour boards (RCP) in oncology treatment planning. One hundred standardised synthetic cases (20 per localisation, across breast, lung, urological, digestive and gynaecological cancers) are submitted as identical structured input to two independent tumour boards per localisation and to six frontier LLMs. Every recommendation, human or model, is decomposed into five predefined decision domains (intent, surgery, radiotherapy, systemic therapy with its line, work-up and biomarkers). For each domain, concordance between two recommendations is scored 2 when the same options are selected, 1 when at least one option is shared, and 0 when none is shared; the score is computed automatically from the recommendation forms, without human scoring. The primary endpoint is the mean domain score between each LLM and each tumour board, read against the same score computed between the two tumour boards of the same case, so that humans and models are measured by an identical rule and no respondent contributes to its own reference. A co-primary safety endpoint captures the proportion of recommendations carrying serious harm potential, because concordance alone can conceal dangerous errors. Where the two boards select different options, the case-domain is classified EQUIPOISE and, like every discordance, is adjudicated by an independent panel blinded to the source of the recommendation. Tumour boards record their recommendations during the meeting itself, on a dedicated platform (https://etude-beacon.fr) that reveals a case only once it is presented and time-stamps that reveal, so that no board can read a case in advance. BEACON is designed as reusable, openly licensed, pre-registered infrastructure: all synthetic cases, evaluation rubrics, scoring algorithms, verbatim prompts and raw model outputs are released for full reproducibility.

Official title: Benchmarking AI for Clinical Oncology decisioNmaking (BEACON): A Prospective, Multicentre, Blinded Evaluation of Frontier Large Language Models Against Multidisciplinary Tumour Board Recommendations in Oncology Treatment Planning

Key Details

Conditions

Interventions

Locations (5)