A Reproducible Evaluation Workflow for Quantitative Benchmarking of Medical Data Quality Tools
Main Article Content
Keywords
medical data quality, defect injection, clinical AI, benchmarking, data profiling
Abstract
Propagating data defects in Electronic Health Records (EHRs) undermine the reliability of clinical AI models. However, the absence of ground-truth defect labels in real-world datasets hinders the standardized evaluation of data quality tools. This study establishes a robust evaluation workflow for a multimodal medical data quality assurance system [A], comprising controlled defect injection, real-dataset profiling, cross-tool benchmarking, and ablation studies. Experimental results demonstrate that the system achieves high detection efficacy, with a recall of 98.00% and precision of 91.91%. The system’s discriminative power is validated by a monotonic decline in quality scores as defect ratios increase. Validation on the UCI Diabetes 130 dataset showed missing-rate statistics consistent with existing literature (maximum deviation < 0.05%). Crucially, comparative analysis reveals that general-purpose tools (Great Expectations, DeepChecks, ydata-profiling) lack native medical-semantic capabilities (e.g., unit consistency) and require clean baselines, whereas our system operates effectively without such dependencies. Ablation studies further identify completeness and accuracy as the primary drivers of detection performance. This work offers a reproducible, quantitative benchmarking framework for advancing medical data quality assessment.
References
- [1] Clemente F, Ribeiro G M, Quemy A, et al. ydata-profiling: accelerating data-centric AI with high-quality data. Neurocomputing. 2023; 554: 126585.
- [2] Weiskopf N G, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. Journal of the American Medical Informatics Association. 2013; 20(1): 144-151.
- [3] Breunig M M, Kriegel H P, Ng R T, et al. LOF: identifying density-based local outliers. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data. New York: ACM; 2000: 93-104.
- [4] Liu F T, Ting K M, Zhou Z H. Isolation forest. 2008 Eighth IEEE International Conference on Data Mining (ICDM). Piscataway: IEEE; 2008: 413-422.
- [5] Goldstein M, Uchida S. A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data. PLoS ONE. 2016; 11(4): e0152173.
- [6] Johnson A E W, Bulgarelli L, Shen L, et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data. 2023; 10: 1.
- [7] Chandola V, Banerjee A, Kumar V. Anomaly detection: a survey. ACM Computing Surveys. 2009; 41(3): 1-58.
- [8] Abhyankar S, Demner-Fushman D, Mcdonald C J. Standardizing clinical laboratory data for secondary use. Journal of Biomedical Informatics. 2012; 45(4): 642-650.
- [9] Ta C N, Weng C. Detecting systemic data quality issues in electronic health records. Studies in Health Technology and Informatics. 2019; 264: 857-861.
- [10] Batini C, Cappiello C, Francalanci C, et al. Methodologies for data quality assessment and improvement. ACM Computing Surveys. 2009; 41(3): 1-52.
- [11] Schelter S, Lange D, Schmidt P, et al. Automating large-scale data quality verification. Proceedings of the VLDB Endowment. 2018; 11(12): 1781-1794.
- [12] Yang J, Shi R, Wei D, et al. MedMNIST v2—a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data. 2023; 10: 41.
- [13] DAMA International. DAMA-DMBOK: data management body of knowledge. 2nd ed. Basking Ridge, NJ: Technics Publications; 2017.
- [14] Sweeney L. k-anonymity: a model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems. 2002; 10(5): 557-570.
Article Sidebar
This work is licensed under a Creative Commons Attribution 4.0 International License.