AI disease-risk models trained on biased, non-representative data perpetuate healthcare inequities and systemic data gaps
Original framing: “Dozens of AI disease-prediction models were trained on dubious data” — Nature
Scientific evidence shows that AI models trained on biased data produce disparate outcomes, with Black patients up to 40% more likely to be misdiagnosed in some studies. The 'garbage in, garbage out' principle applies here: datasets lacking diversity (e.g., 80% of genomic data comes from European ancestry) skew predictions toward majority populations. Peer-reviewed work by Obermeyer et al. (2019) demonstrated how a commercial algorithm perpetuated racial bias by using healthcare costs as a proxy for need, reflecting structural inequities in access. Yet, most AI disease-prediction models lack transparency about their training data provenance.
The crisis of biased AI disease-prediction models is not merely a technical failure but a manifestation of deeper structural inequities in healthcare data, rooted in colonial legacies and corporate-driven innovation.
The datasets underpinning these models—like MIMIC-III or UK Biobank—are overwhelmingly derived from Western, male, and affluent populations, reflecting historical exclusions from medical research that date back to eugenics and unethical experiments. Meanwhile, non-Western knowledge systems, which offer holistic and community-centered approaches to health, are systematically sidelined in favor of Silicon Valley’s extractive data practices. The result is a feedback loop where profit-driven AI tools perpetuate the very disparities they claim to solve, as seen in cases where Black patients are misdiagnosed at higher rates due to flawed algorithms. To break this cycle, systemic solutions must center reparative justice—through participatory data governance, cross-disciplinary audits, and the integration of traditional knowledge—while holding tech developers, regulators, and academic institutions accountable for their role in entrenching these harms. Without such reforms, AI in healthcare will remain a tool of inequity rather than liberation.