General Scales Framework Reveals AI's Cognitive Limits and Structural Risks in Automated Evaluation Systems

Indigenous knowledge systems universally reject the reduction of intelligence to quantifiable rubrics, instead framing cognitive capacity as a dynamic, relational process tied to community and ecology. The study's reliance on decontextualized scales mirrors the extractive logic of colonial education systems, which measured 'progress' through Eurocentric lenses. Without incorporating Indigenous epistemologies—such as the Māori concept of 'mātauranga' or the Andean 'ayni' (reciprocal labor)—AI evaluation risks perpetuating epistemic violence. Even the term 'cognitive demands' reflects a Western cognitive bias that privileges abstract, individual problem-solving over embodied, communal knowledge.

The methodology echoes 20th-century psychometrics, where IQ tests and standardized assessments were used to justify racial hierarchies and eugenics policies under the guise of scientific objectivity. Like the Stanford-Binet scale or the Army Alpha tests, rubric-based AI evaluation risks reifying arbitrary metrics as 'objective' measures of capability. Historical precedents show how such systems have been co-opted by state and corporate actors to pathologize marginalized groups—e.g., the use of 'intelligence testing' to justify segregation or labor exploitation. The study's blind spot to this lineage reveals a failure to interrogate how evaluation frameworks serve power.

Cross-culturally, intelligence is not a monolithic construct but a pluralistic one, with non-Western traditions emphasizing contextual, ethical, and relational dimensions of cognition. For instance, Confucian traditions prioritize 'practical wisdom' (zhi) over abstract reasoning, while Hindu philosophy frames intelligence as a means to 'liberation' (moksha), not market productivity. The study's rubric-based approach, which rewards tasks like 'logical deduction' or 'multi-step reasoning,' implicitly privileges Western academic norms over Indigenous or Eastern epistemologies. This cultural myopia risks producing AI systems that perform well in Western contexts but fail in diverse cultural settings.

The study's methodology, while rigorous in its technical execution, suffers from a lack of epistemological reflexivity about what 'cognitive demands' actually measure. The reliance on LLMs to evaluate LLMs introduces circularity, as both the evaluator and evaluated are products of the same statistical training regimes. There is no discussion of the ecological footprint of such large-scale evaluations, nor their carbon costs—a critical oversight given the energy-intensive nature of modern AI training. Additionally, the paper does not address the reproducibility crisis in AI research, where benchmarks often fail to generalize beyond their original contexts.

Artistic and spiritual traditions across cultures have long framed intelligence as a sacred, emergent property of interconnected systems rather than a discrete, measurable trait. Indigenous storytelling, Sufi poetry, and Buddhist philosophy all describe cognition as inseparable from emotion, ethics, and environment—a stark contrast to the mechanistic rubrics of AI evaluation. The study's approach risks reducing the ineffable qualities of human (and machine) intelligence to algorithmic checklists, stripping away the wonder and mystery that define artistic and spiritual inquiry. Even the term 'explanatory power' betrays a Western scientific bias that equates understanding with prediction and control.

If rubric-based AI evaluation becomes the dominant paradigm, it risks entrenching a feedback loop where AI systems are optimized for compliance with corporate and state metrics rather than human flourishing. Scenario planning reveals that such systems could exacerbate algorithmic bias by normalizing Western cognitive norms as global standards, marginalizing non-Western knowledge systems. The long-term implication is a homogenization of intelligence, where AI—and by extension, human-AI collaboration—is constrained by the limits of proprietary evaluation frameworks. This could stifle innovation in non-Western contexts and deepen global inequalities in AI development and access.

Marginalized communities—particularly those in the Global South, Indigenous groups, and disabled individuals—are systematically excluded from the design and evaluation of AI systems, despite bearing the brunt of algorithmic harm. The study's focus on 'cognitive demands' as defined by Western academia ignores the ways marginalized groups have historically been pathologized by such metrics, from IQ tests to modern hiring algorithms. There is no engagement with disability justice perspectives, which critique the ableism embedded in standardized cognitive assessments. Additionally, the paper fails to consider how AI evaluation systems could be co-designed with affected communities to center their needs and knowledge.