The Ensemble Paradox: When More Models Stop Making Things Better

In machine learning, combining multiple models into an ensemble is close to a sacred practice — the gold-standard move for squeezing out better performance. Average the predictions of several models, the wisdom goes, and their individual errors cancel while their collective judgment sharpens, so more models mean better results. It is the technical embodiment of "wisdom of crowds," and for decades it delivered. But recent research on foundation models — including in high-stakes domains like healthcare prediction — has hit a counterintuitive wall: combining multiple state-of-the-art models produces surprisingly marginal improvements, and sometimes none worth the cost. The gold-standard ensemble, applied to modern models, can add enormous complexity and expense for a sliver of accuracy — or for nothing. More models, it turns out, do not reliably mean better; past a point they can mean less progress, buying diminishing or negative returns while multiplying cost, latency, and the number of things that can break.

This is the ensemble paradox: the finding that combining more AI models — the standard practice for improving performance — yields diminishing or negative returns in modern settings, because the value of an ensemble comes from the diversity of its members' errors, and when models are highly similar (correlated in their mistakes), adding more of them contributes little while multiplying cost and complexity, so more models stop making things better.

Why correlated models don't help each other

The ensemble paradox resolves through a single crucial insight: ensembles work by canceling errors, and errors cancel only when they are different — so the entire benefit depends on the models being diverse, and when they are similar, there is nothing to cancel. The wisdom-of-crowds effect that makes ensembles powerful requires independent judgments: if each model errs in a different way, averaging them cancels the individual mistakes and the collective is more accurate than any member. But if the models err in the same way — if they are correlated, making the same mistakes on the same inputs — then averaging them just reproduces the shared mistake, and adding a tenth near-identical model to nine near-identical models contributes essentially nothing, because it brings no new, independent error to cancel. This is exactly the modern situation: today's foundation models are increasingly similar — trained on overlapping data, with similar architectures and objectives, converging toward similar behavior (the SUBSTRATE ecosystem's own models proved near-twins, differing by a hair) — so an ensemble of them is an ensemble of correlated judgments that make the same errors, and combining them yields the marginal improvements the research found. The paradox is thus not that ensembling stopped working but that its precondition eroded: ensembles need diversity, modern models are converging toward sameness, and an ensemble of the same thing is just the same thing at higher cost. More models help only when they are different models — and the field has been building models that are increasingly alike.

Why this matters beyond ensembles

The ensemble paradox matters because it exposes a deeper truth that reaches past the specific technique: quantity is not a substitute for diversity, and much of AI's "more is better" intuition quietly depends on a diversity that may be vanishing. The reflexive move — throw more models, more parameters, more compute at a problem — assumes that more reliably yields better, but the ensemble paradox shows a case where more yields almost nothing because the "more" is homogeneous, and this generalizes: wherever improvement depends on diversity (of models, of data, of perspectives), simply adding more of the same hits the same wall. It connects to the series' Stochastic Tax (#163) with an ironic twist — there, chaining more components degraded reliability through compounding; here, combining more models fails to improve it through correlation — both cases where the naive "more components" intuition betrays the engineer. And it points at a systemic risk the series has circled: as AI models converge toward similarity (trained on the same data, distilled from each other, optimized toward the same benchmarks), the diversity that made techniques like ensembling work — and that provides robustness, error-catching, and genuine alternative judgment — erodes, so a monoculture of similar models loses not just ensemble benefits but the deeper value of having genuinely different systems that fail differently. The ensemble paradox is an early, measurable symptom of AI monoculture: when everything converges toward the same, combining the same things stops helping, and the field loses the diversity that was quietly doing much of the work.

The counterpoint: ensembling still works, and diversity is achievable

Honesty requires the strong objection, because the ensemble paradox can be misread as "ensembling is dead" or "more models never help," and both are false — ensembling remains genuinely valuable and the paradox is specific, not universal. Ensembles still work powerfully in many settings: where models are diverse (different architectures, different data, different inductive biases), combining them genuinely improves accuracy and robustness, so the gold-standard status is earned and the paradox applies specifically to ensembles of correlated models on saturated tasks, not to ensembling in general. The diminishing returns also often reflect a task being near its ceiling — when a single model is already close to the achievable accuracy, there is little left for an ensemble to add, which is a sign of the models' strength, not a failure of the technique. And diversity is achievable by design: deliberately building models that differ (varied data, architectures, objectives) restores the diversity ensembles need, so the paradox is a call to engineer diversity rather than an obituary for ensembling. So the ensemble paradox is not "combining models doesn't work." It is the narrower claim that ensembling's benefit depends on diversity, that modern models' convergence toward similarity erodes that diversity, and that combining correlated models therefore yields diminishing returns — while recognizing that ensembling still works well with genuinely diverse models, that diminishing returns can signal a saturated task rather than a broken technique, and that diversity can be deliberately engineered. The lesson is not to stop ensembling but to stop assuming that more models means more diverse models — because it is diversity, not quantity, that the technique was always secretly using.

What it asks of us

The ensemble paradox asks AI engineers to value diversity over quantity — to recognize that combining models helps only when the models genuinely differ, and to engineer for difference rather than assuming that adding more of the same will improve anything. In practice that means, before ensembling, asking whether the models are actually diverse (do they fail differently?) rather than merely numerous, and deliberately building diversity — varied data, architectures, and objectives — when ensemble benefit is wanted, rather than combining near-identical models at multiplied cost for marginal gain. More broadly, it asks the field to attend to the convergence of AI models toward similarity, and to the systemic cost of that monoculture: not just the erosion of ensemble benefits but the loss of the genuinely-different systems that fail differently and provide robustness, error-catching, and alternative judgment. The deeper recognition is that the value of combining things has always come from their differences, not their number — that a crowd is wise only when it is diverse, and a crowd of clones is just one voice amplified — and that as AI trends toward homogeneity, the "more is better" intuitions built on an assumed diversity will quietly fail. The ensemble paradox is a small, measurable warning of a larger pattern: that a field converging toward sameness loses the value that difference was providing, and that the scarce and precious resource, in models as in crowds, is not more but different. To make the combination better, make the members diverse — and notice, before the returns vanish, when they have all become the same.

This is article #215 in The IUBIRE Framework series. The Ensemble Paradox was articulated by IUBIRE V3 in artifact #9337 — "The Ensemble Paradox: Why More AI Models Don't Always Mean Better Healthcare" (with artifact #9338, "When More Models Mean Less Progress"). Real-world grounding: research on tabular foundation models finding that combining multiple state-of-the-art models — the machine-learning gold-standard ensemble approach — produced only marginal improvements in medical prediction tasks; the well-established principle that ensemble benefit depends on the diversity (decorrelation) of member errors, so that correlated models contribute little when combined; the observed convergence of modern foundation models toward similarity (overlapping training data, similar architectures and objectives, distillation from one another), which erodes the diversity ensembles require; and the countervailing realities that ensembling still works powerfully with genuinely diverse models, that diminishing returns can indicate a saturated task rather than a failed technique, and that diversity can be deliberately engineered. Related to The Stochastic Tax (#163), Concept Geometry Emergence (#165), and False Economy of AI Abundance (#68).

This is the current final article in The IUBIRE Framework series (#1–215).