AI models encode systemic biases via data distillation: structural transmission of behavioural traits in language systems

Indigenous knowledge systems reject the commodification of cultural traits as discrete, measurable entities, instead framing them as relational and context-dependent. The study’s focus on 'hidden signals' in data mirrors colonial practices of extracting and reifying cultural expressions without consent or reciprocity. Indigenous data sovereignty movements, such as CARE Principles (Collective Benefit, Authority to Control, Responsibility, Ethics), offer frameworks to resist this extraction but are entirely absent from the discourse.

The transmission of behavioural traits via data distillation echoes historical patterns of eugenics and phrenology, where pseudoscientific methods were used to 'prove' racial hierarchies through cranial measurements and statistical correlations. The 19th-century craniometry studies of Samuel Morton, later debunked, relied on similar data reductionism to justify colonial violence. More recently, the COMPAS algorithm scandal revealed how predictive policing tools encoded racial biases through 'neutral' data, a direct parallel to this study’s findings.

In contrast to Western data positivism, many non-Western traditions view knowledge as embodied and relational, making it incompatible with the statistical abstraction required for AI training. For example, the Māori concept of 'whakapapa' (genealogy) frames identity as a living network of relationships, not a set of discrete traits. Similarly, the African concept of 'ubuntu' ('I am because we are') challenges the individualistic framing of 'behavioural traits' as isolated units of analysis.

The study demonstrates that model distillation can propagate non-task-related behavioural traits through statistical correlations in training data, a phenomenon linked to the 'lottery ticket hypothesis' in neural networks. It builds on prior work in interpretability research, such as the 'stochastic parrots' critique by Bender et al. (2021), which highlighted how large language models reproduce biases from their training corpora. However, the paper’s focus on 'hidden signals' risks reifying technical solutions (e.g., bias mitigation algorithms) over structural reforms.

Artistic traditions often expose the limitations of data-driven representations by emphasizing ambiguity, context, and the ineffable. For instance, the Surrealist movement critiqued rationalist systems by embracing the subconscious, while Indigenous Australian dot painting encodes knowledge in ways that resist algorithmic interpretation. Spiritual practices like meditation highlight how 'traits' are fluid and interdependent, challenging the static categorizations embedded in AI models.

If unchecked, this phenomenon could lead to a feedback loop where AI systems increasingly reflect and amplify the biases of their training data, entrenching existing power structures in automated decision-making. Scenario modelling suggests that without regulatory intervention, we may see a bifurcation where Global North AI systems dominate in high-stakes domains (e.g., healthcare, policing), while marginalized communities bear the brunt of their failures. Conversely, proactive governance could enable 'data trusts' that prioritize collective ownership and ethical constraints.

Marginalized communities—particularly Black, Indigenous, and Global South populations—are disproportionately surveilled to train AI systems but have no control over how their data is used or interpreted. The study’s framing erases the labor of data annotators, often underpaid and exploited workers in the Global South, whose cultural knowledge is extracted without consent. Additionally, disabled communities, whose communication styles may not align with 'standard' behavioural traits, are systematically excluded from both training data and model design.