AI disease-risk models trained on biased, non-representative data perpetuate healthcare inequities and systemic data gaps

Indigenous and traditional knowledge systems offer holistic frameworks for disease prediction that emphasize relationality (e.g., human-environment symbiosis) and long-term ecological balance, contrasting with the extractive, reductionist data practices underpinning AI models. These systems often prioritize community consent and collective well-being over individual risk scoring, yet their exclusion from mainstream medical AI entrenches colonial legacies in healthcare. Projects like the Māori-led 'Whakapapa AI' initiative demonstrate how indigenous data sovereignty can guide ethical algorithmic design, but such efforts remain marginalized in favor of Silicon Valley-centric solutions.

The underrepresentation of marginalized groups in medical datasets traces back to 19th-century eugenics and 20th-century clinical trials that systematically excluded women, racial minorities, and low-income populations. The Tuskegee Syphilis Study and Henrietta Lacks' case exemplify how medical research has historically exploited vulnerable groups, leaving generational gaps in data. Modern AI inherits these biases, as datasets like MIMIC-III (a critical care dataset) are 70% male and predominantly white, reinforcing historical power imbalances in healthcare. The lack of reparative data collection perpetuates these inequities.

Cross-culturally, disease prediction is framed differently: Western medicine often focuses on individual biomarkers, while systems like Traditional Chinese Medicine (TCM) or Unani use pattern recognition across body-mind-environment interactions. In South Asia, Ayurveda’s 'dosha' system predicts disease based on constitutional traits and seasonal changes, aligning with ecological rather than statistical models. African Ubuntu philosophy emphasizes communal health, suggesting that risk prediction should account for social determinants like housing and food security. These paradigms highlight the limitations of AI trained on narrow, Western-centric datasets.

Scientific evidence shows that AI models trained on biased data produce disparate outcomes, with Black patients up to 40% more likely to be misdiagnosed in some studies. The 'garbage in, garbage out' principle applies here: datasets lacking diversity (e.g., 80% of genomic data comes from European ancestry) skew predictions toward majority populations. Peer-reviewed work by Obermeyer et al. (2019) demonstrated how a commercial algorithm perpetuated racial bias by using healthcare costs as a proxy for need, reflecting structural inequities in access. Yet, most AI disease-prediction models lack transparency about their training data provenance.

Artistic and spiritual traditions often frame disease as a disruption of harmony—whether in Ayurveda’s 'dosha' imbalance, TCM’s 'qi' stagnation, or African spiritual cosmologies where illness stems from ancestral or communal disharmony. These frameworks challenge the mechanistic view of AI, which reduces risk to quantifiable variables. For example, the Māori concept of 'mauri' (life force) underpins holistic health, suggesting that prediction models should account for cultural and spiritual well-being. Creative resistance, such as data art projects visualizing algorithmic bias, can make these harms legible to broader publics.

Future models must integrate participatory design, where communities co-create datasets and define risk parameters, to avoid repeating historical harms. Scenario planning should explore 'data sovereignty' frameworks, where marginalized groups control how their health data is used, as seen in initiatives like the Global Indigenous Data Alliance. Additionally, hybrid models combining AI with traditional knowledge could improve prediction accuracy—for example, integrating TCM pulse diagnostics with Western biomarkers. Without such reforms, AI disease prediction will deepen healthcare divides, particularly in low-resource settings.

Marginalized communities—especially Black, Indigenous, low-income, and disabled populations—are disproportionately harmed by biased AI models, as their data is either excluded or misrepresented. For instance, women’s symptoms are often dismissed in medical datasets, leading to underdiagnosis of conditions like endometriosis. The lack of representation in training data means these models fail to recognize symptoms across diverse populations, exacerbating existing health disparities. Grassroots organizations like the 'Algorithmic Justice League' advocate for auditing these tools, but their concerns are sidelined in favor of tech industry narratives.