Theory 065/Information & Computation /Machine learning
Ensembles Fail When Errors Correlate
Ensembles are assumed to help by averaging. Averaging stops working when everyone is wrong together.
Ensemble benefit disappears when member errors correlate above a threshold, so adding a highly accurate but correlated model can reduce the ensemble's reliability despite raising its average accuracy.
01The evidence
Ensemble gains vary widely and are smallest when members share training data or architecture, which produces correlated errors.
02The tension
Ensemble design optimises member accuracy and diversity informally. Diversity is rarely measured, and the failure mode is attributed to bad luck on a test set.
03The mechanism
Ensemble error depends on the variance of member errors and on their covariance. When covariance is high, averaging does not cancel it, and the ensemble inherits the shared blind spot.
That makes correlation the controlling variable, and predicts a threshold above which adding members adds cost without reliability.
04What this is not
Heuristics encourage diversity without measuring it. This measures correlation, which predicts the point of no benefit.
05Predictions
- P1Ensemble gain should vanish above an error-correlation threshold.
- P2Adding accurate correlated members should reduce reliability.
- P3Correlation should predict gain better than member accuracy.
06Refutation
Gains persisting at high correlation would leave averaging as the explanation.
07What it changes
It would change ensemble construction and evaluation.