HMN 2026: How Some patient groups are far more vulnerable to near-perfect privacy attacks from medical AI

From detecting pneumonia on a chest X-ray to assessing whether a dark spot on the skin is benign or malignant, medical AI systems are playing an increasingly important role in clinical diagnosis. Unfortunately, the models used to train these AI systems are often victims of cyberattacks, specifically membership inference attacks (MIAs), which can lead to people’s personal information being stolen or revealed.

In a recent study, researchers conducted a first-ever patient-level privacy audit to see how easily individual patients could be identified from the underlying data used to train medical AI models.

At first glance, an AI model may appear to protect everyone’s privacy equally well, but a closer look reveals a different story. Researchers found that attackers can identify certain individual patients with near-perfect accuracy, exposing a hidden unfairness in privacy.

People from underrepresented groups—such as racial minorities, Medicaid recipients or patients with rare medical conditions—are significantly more vulnerable to privacy leaks than the majority population.

They also uncovered a trade-off: The better and more powerful AI models get at diagnosing disease, the bigger the threat they pose to privacy. Removing names or pseudonymizing records no longer holds up against modern AI attacks. What’s needed instead are mathematically verified patient-level safeguards.

The findings were published in Nature.

Hidden threats of averaging

Medical AI has huge potential to improve patient care, especially in places that lack specialist doctors. With each passing day, AI experts are fine-tuning models to become increasingly accurate. The more data a model is trained on, the better its prediction outcomes.

Since the data involved is sensitive patient data, it is prone to MIA, a privacy attack in which an attacker attempts to determine whether a specific individual’s data was included in an AI model’s training data set.

An attacker doesn’t need access to a medical AI system’s code to probe its privacy. Simply interacting with it like any other user can be enough.

By submitting a patient’s chest X-ray and observing a prediction—say, an 80% chance of pneumonia—they can launch an MIA to determine whether that patient’s data was used to train the model. The attack exploits the tendency of AI models to be slightly more confident when predicting cases they have seen during training.

Medical AI undergoes mandatory privacy testing to ensure data security. However, most research in this domain has measured only how often these attacks succeed on average across an entire data set. Average privacy measures can mask serious risks for individual patients, especially those with multiple, similar records.

Zooming into patient-level threats

To uncover these hidden vulnerabilities, the researchers in this study measured privacy at the patient level and asked whether some groups face greater privacy risks than others. Their study included medical information such as chest X-rays, skin images, mammograms, eye scans, heart readings (ECGs) and electronic health records from seven large, real-world data sets.

Instead of building a single AI model, the researchers trained 200 separate versions, each one fed a different, randomly selected group of patients. This gave them a way to test privacy risk for every single record in the study. For each patient, they could sort the 200 models into two piles: those that had actually trained on that patient’s data and those that hadn’t.

Comparing how these two groups of models behaved became the key to exposing just how much an AI gives away about the people behind its training data.

To determine a person’s final risk, they considered all records contributed by that patient and selected the highest risk score. The rationale is that if an attacker can identify even a single medical image, the patient’s identity and membership in the data set are effectively exposed.

The results showed that standard ways of measuring AI privacy were flawed. Aggregate metrics often hide serious individual risks, especially for racial minorities, certain insurance groups and patients with rare diseases. This imbalance raised a bigger concern: If some groups felt less protected, they might lose trust in medical AI and become less willing to share their data, making future models less accurate for them.

To address this, the researchers pointed to patient-level differential privacy, a method that adds carefully designed mathematical noise to data or model outputs to mask individual identities and protect every patient, even those with rare or highly identifiable data.

The content is provided for information purposes only.