Overview
Extracting clinical information from unstructured electronic health records (EHRs) at scale is essential for generating real-world evidence, but data reliability requires rigorous evaluation. In this study, researchers applied the Validation of Accuracy for LLM/ML-Extracted Information and Data (VALID) framework to evaluate LLM-based extraction of prostate cancer variables—including diagnosis details, treatment, castration resistance, and real-world progression—using the Flatiron Health Research Database in Germany.
LLM output performance was benchmarked against human-curated datasets using variable metrics (precision, recall, F1-score) and cross-variable verification checks. After prompt engineering and iterative error analysis, the LLM achieved performance within 5 percentage points (F1-score) of expert human abstraction performance across all essential prostate-related fields. Verification checks evaluating relationships between diagnosis dates, stage, and grade confirmed alignment with clinical expectations.
Why this matters
This study demonstrates that applying LLMs within the VALID framework can produce real-world data for prostate cancer in Germany. By establishing automated extraction workflows, researchers can accelerate evidence generation, replicate established epidemiological distributions and treatment outcomes, and expand multinational oncology research while maintaining data integrity.