Overview
Large language models (LLMs) offer a scalable approach to abstracting clinical data, but generating real-world evidence requires rigorous quality assessment. To evaluate this, Flatiron Health applied its Validation of Accuracy for LLM/ML-Extracted Information and Data (VALID) framework to a dataset of 13,901 prostate cancer patients in the United Kingdom. The VALID framework evaluated data quality across three core pillars: variable-level accuracy compared to expert human abstractors, clinical logic verification checks, and replication analyses comparing survival outcomes against expert human abstraction and UK national cancer registry data.
The model's extraction accuracy met the VALID framework quality standards, with performance closely aligned to human abstractor accuracy. Clinical verification checks aligned with expected medical practices, and replication analyses showed LLM-derived overall survival in targeted treatment subcohorts matched human abstraction, while stage-specific survival aligned with national registry data.
Why this matters
Extending the VALID framework to the UK demonstrates that LLMs can accurately, securely, and scalably curate complex real-world oncology data while complying with strict local privacy regulations (UK GDPR). Establishing the reliability of LLM-extracted data in the UK against expert human abstraction and national registries builds trust among international researchers, clinicians, and health authorities, paving the way for faster real-world oncology evidence globally.