HMN 2026: What are the concerns about AI health-prediction models trained on unreliable datasets

Some AI models designed to predict stroke and diabetes risk may be based on datasets whose origins cannot be verified, according to new research. The study, published in BMC Medicine and led by researchers at QUT and the Australian Center for Health Services Innovation (AusHSI), examined two widely downloaded health datasets hosted on Kaggle, an online platform for sharing datasets and machine-learning resources, marketed as “the world’s AI proving ground.”

The datasets were found to have been used in 125 peer-reviewed studies, despite providing almost no information about where the data came from, how they were collected or whether they represented real patients.

Lead author Alexander Gibson, from the QUT School of Public Health and Social Work and AusHSI, said the team was shocked by what it found.

“It was an enormous surprise to come across something like this,” Gibson said.

“These datasets exhibit unusual patterns that raise serious questions about their authenticity and suitability for clinical research.”

Three prediction models based on the data had evidence of use in clinical practice; one model was cited in a medical device patent, and the models were cited in 86 review articles.

The study assessed the datasets using the internationally recognized TRIPOD+AI reporting framework and found that they scored 0 out of 9 on essential data-provenance criteria.

Gibson said this should be a red flag for journals, developers and clinicians.

“Prediction models built on data of unknown provenance have no place in clinical decision-making. Without trustworthy data, the outputs are unreliable and risk misleading clinicians and harming patients,” he said.

The authors said that journals, funders and data repositories must strengthen requirements for data-source disclosure.

They also recommended that the two Kaggle datasets be removed to prevent further misuse.

Seven articles using these datasets have been retracted from journals for being unreliable. The results of the study have also updated the Collection of Open Science Integrity Guides.

Gibson said the issue reflects a broader challenge as AI tools proliferate in health care.

“We’re seeing fast-churn research built on datasets that look scientific but lack the most basic transparency,” he said.

“Without stronger safeguards, unreliable models will continue to make their way into the literature, and potentially into practice.”

The study also included QUT researchers Professor Adrian Barnett and Associate Professor Nicole White.

This content is provided for information purposes only.