Incoprea Theory
Volume VIII · 100

Theory 065/Information & Computation /Machine learning

Ensembles Fail When Errors Correlate

Ensembles are assumed to help by averaging. Averaging stops working when everyone is wrong together.

The claim

Ensemble benefit disappears when member errors correlate above a threshold, so adding a highly accurate but correlated model can reduce the ensemble's reliability despite raising its average accuracy.

01The evidence

Ensemble gains vary widely and are smallest when members share training data or architecture, which produces correlated errors.

02The tension

Ensemble design optimises member accuracy and diversity informally. Diversity is rarely measured, and the failure mode is attributed to bad luck on a test set.

03The mechanism

Ensemble error depends on the variance of member errors and on their covariance. When covariance is high, averaging does not cancel it, and the ensemble inherits the shared blind spot.

That makes correlation the controlling variable, and predicts a threshold above which adding members adds cost without reliability.

04What this is not

Accuracy-and-diversity heuristics

Heuristics encourage diversity without measuring it. This measures correlation, which predicts the point of no benefit.

05Predictions

  • P1Ensemble gain should vanish above an error-correlation threshold.
  • P2Adding accurate correlated members should reduce reliability.
  • P3Correlation should predict gain better than member accuracy.

06Refutation

Gains persisting at high correlation would leave averaging as the explanation.

07What it changes

It would change ensemble construction and evaluation.