Ensembles fail when errors correlate: a proof with a threshold surface
Supplied working report; original verdicts not validated.
Review status: Needs review. This is an archival source note, not a passed proof.
README.md
# Ensembles fail when errors correlate: a proof with a threshold surface **Verdict: partially supported.** Two of the three pre-specified checks hold and the third does not. The mathematical part of the claim is a proof, not an empirical finding: ensemble error variance is affine in the correlation between member errors, so relative gain falls monotonically as correlation rises and reaches zero, and there is nothing to discover about that. What the simulation contributes is the size of the threshold. With a new member whose variance is one quarter that of the existing members, adding it raises ensemble error once its correlation with them exceeds 0.69 for three members, 0.39 for five, 0.18 for ten and 0.059 for thirty. The third check fails: across the grid the partial rank correlation of gain with correlation is −0.919 and with member accuracy −0.945, so accuracy predicts gain at least as well, in all three replications. The claim is that ensemble benefit disappears above an error-correlation threshold; the refutation is that gains persisting at high correlation would leave averaging as the explanation. ## The derivation For member errors with variances σ² and common pairwise correlation ρ, the ensemble error is the mean error and its second moment is exactly MSE = (1/N²)(Σσᵢ² + 2ρ Σᵢ<ⱼ σᵢσⱼ). That is affine in ρ with a non-negative slope, so gain against any fixed benchmark is monotone in ρ. Adding a member of variance aσ² correlated at ρ_new with each existing member raises the ensemble MSE if and only if ρ_new exceeds (2 + 1/N − a + ρ(N−1)(2N+1)/N) / (2N√a) for equal existing variances; the unequal-variance form is used in the code. Two checks on the formula: it returns 1 at N = 1 with a = 1, the known result that a second equal member never hurts unless perfectly correlated, and 0.055 at N = 10 with a = 1 and ρ = 0, the known result that a new member must be less correlated with the ensemble than the ensemble's average pairwise correlation to be worth adding. The threshold is below one only when the new member is more accurate than the existing mean, which is the case the claim names. ## What was done Multivariate normal errors with equicorrelation ρ, member standard deviations ordered by accuracy, and an accuracy spread that is equal, moderate or wide. Correlation runs from 0 to 1 in steps of 0.02, member count takes the values 3, 5, 10 and 30, and the crossing gives 612 cells of 10,000 draws each. Errors scale mildly and member-specifically away from the centre of a five-point true-value grid, so the grid is not degenerate. The identity is checked at the centre, where the scaling is one; the Monte Carlo median relative error on the ensemble second moment is 0.010, maximum 0.044. ## What happened | members | threshold ρ* | gain at ρ = 0 | gain at ρ = 1 | | --- | --- | --- | --- | | 3 | 0.69 | 0.658 | 0.000 | | 5 | 0.39 | 0.804 | 0.000 | | 10 | 0.18 | 0.901 | 0.000 | | 30 | 0.06 | 0.966 | 0.000 | At equal member variances the gain falls linearly to zero and stops there: an ensemble of identical-quality members converges on the quality of one member, never below it. When accuracies differ it does not stop at zero. With a wide spread it reaches −3.0 at a correlation of one, which means the ensemble is four times worse than its best member, and the simulated crossings where gain reaches zero are 0.32, 0.44, 0.52 and 0.55 for three, five, ten and thirty members at a moderate spread. That is the claim's second prediction made numeric. The threshold surface is the useful output. It falls with the number of members, because a large ensemble is already close to the correlation floor, and rises with the existing correlation, because a correlated ensemble has little to lose. Of the 612 cells, 137 have a threshold strictly below one. The Monte Carlo brackets the analytic threshold in 14 of 14 cells tested: a new member placed 0.15 below it lowers MSE, and one placed 0.15 above it raises MSE. ## Reliability An ensemble that reports its uncertainty as the mean member standard deviation over the square root of the member count assumes independence, which is the assumption the claim attacks. It fails as expected: at a correlation of 0.9 or above with five members the nominal 90% interval covers the true value 0.546 of the time, against 0.892 at zero correlation. An interval built on the true ensemble standard deviation covers 0.898 at high correlation, so the shortfall is entirely the independence assumption and not the averaging. ## The third check Gain is a deterministic function of correlation, member count and accuracy spread, and the grid is noiseless apart from Monte Carlo error, so both partial correlations are near saturation. The partial rank correlation of gain with correlation, controlling for member count and mean member variance, is −0.919; with mean member variance, controlling for correlation and member count, it is −0.945. Accuracy is ahead in the primary run and in all three replication seeds, so the check genuinely fails rather than washing out. Correlation is not the better predictor of gain once member accuracy is in the model, at least when accuracy spread is one of the factors varied. ## Sensitivity A member four times more accurate than the rest and correlated with them gives a gain of 0.680 at zero correlation falling to −0.440 at one, crossing zero at 0.607, with a minimum threshold of 0.210: the threshold survives unequal variances. Heavy-tailed errors, a multivariate t with four degrees of freedom scaled to unit variance, reproduce the second-moment identity to within 1%, with ratios of 0.995, 0.990 and 0.998 at correlations of 0, 0.5 and 0.9, so the identity is a second-moment result and not a normal-theory accident. A common bias of half a standard deviation on every member is the case where reliability really does collapse: ensemble MSE becomes 0.446 at zero correlation against an unbiased 0.200, and 1.151 against 0.920 at a correlation of 0.9, because a shared bias adds its square and averaging cannot remove it. ## What this cannot see Equicorrelation is a strong simplification. Real ensembles have correlation blocks, members of different accuracy correlated more with their own kind, and errors that are not joint normal, so the threshold is exact for a model no real ensemble has. Nor does it say that a lower-correlation member is available at the same price; the claim implies it, and the test takes that as given. AI-designed and AI-executed (DeepSeek coding agent); not peer reviewed.
hypothesis.md
# Pre-specified hypothesis test — ensemble error correlation
Written before the Monte Carlo was run. The derivation below is analytic and is
part of the pre-specification: it is a proof, and the simulation is a check on
the arithmetic, not a discovery.
## Posted claim, verbatim
> Ensemble benefit disappears when member errors correlate above a threshold, so
> adding a highly accurate but correlated model can reduce the ensemble's
> reliability despite raising its average accuracy.
Posted predictions:
- **P1** Ensemble gain should vanish above an error-correlation threshold.
- **P2** Adding accurate correlated members should reduce reliability.
- **P3** Correlation should predict gain better than member accuracy.
Posted refutation:
> Gains persisting at high correlation would leave averaging as the explanation.
## Derivation, frozen before simulation
Let member errors `e_i` have variances `sigma_i^2` and common pairwise
correlation `rho` (equicorrelation). The ensemble error is the mean error, so
```
MSE_N = (1/N^2) * ( sum_i sigma_i^2 + 2*rho*sum_{i<j} sigma_i*sigma_j )
```
which is exact for any error distribution with those first two moments, not only
for the normal. Two consequences follow immediately and are stated here as
proofs rather than as findings.
**Monotonicity.** `MSE_N` is affine in `rho` with slope
`2*sum_{i<j}sigma_i sigma_j / N^2 >= 0`, so relative gain against any fixed
benchmark is monotone in `rho`. The claim's P1 is therefore true by algebra; the
interesting quantity is where the gain crosses zero.
**Threshold for adding a member.** With `N` existing members of equal variance
`sigma^2`, a new member of variance `a*sigma^2` and correlation `rho_new` to each
existing member is added. The ensemble MSE increases if and only if
```
rho_new > rho*_N(a, rho) = ( 2 + 1/N - a + rho*(N-1)*(2N+1)/N ) / (2*N*sqrt(a))
```
Derivation: `MSE_{N+1} = sigma^2*(N + a + N(N-1)rho + 2N rho_new sqrt(a))/(N+1)^2`
and `MSE_N = sigma^2*(1+(N-1)rho)/N`; set `MSE_{N+1} > MSE_N` and solve for
`rho_new`. Two checks on the formula: at `N = 1, a = 1` it returns `rho* = 1`,
which is the known result that a second member of equal variance never hurts
unless it is perfectly correlated; and at `N = 10, a = 1, rho = 0` it returns
`rho* = 0.055`, which is the known result that a new member must be less
correlated with the ensemble than the ensemble's average pairwise correlation to
be worth adding. For unequal variances the same algebra gives
```
rho* = [ (N+1)^2/N^2 * (S2 + 2*rho*Sij) - S2 - sigma_new^2 - 2*rho*Sij ]
/ ( 2*sigma_new*sum_i sigma_i )
```
with `S2 = sum_i sigma_i^2` and `Sij = sum_{i<j} sigma_i sigma_j`, and this is the
form used for the simulated ensembles. `rho* < 1` requires the new member to be
more accurate than the existing mean, which is exactly the case the claim names.
## Design, frozen
**Monte Carlo.** Errors drawn from a multivariate normal with covariance
`Sigma_ij = rho*sigma_i*sigma_j` for `i != j` and `sigma_i^2` on the diagonal;
10,000 draws per cell. Members are ordered by accuracy, `sigma_i` increasing in
`i`, with `sigma_i = 1 + 0.5*(s-1)*(2*(i-1)/(N-1) - 1)`, so `s = 1` gives equal
variances and `s = 2` gives standard deviations from 0.5 to 1.5. Benchmarks: the
best single member (member 1) and the mean member variance (accuracy).
**Grid.** Five true values `v` in {-2, -1, 0, 1, 2}, 2,000 draws each, with the
error scale of member `i` multiplied by
`h_i(v) = 1 + 0.25*(v/2)^2*(0.5 + 0.5*(i-1)/(N-1))`. The heteroscedasticity is
member-specific and mild, between a factor of 1.125 and 1.25 at the edges of the
grid, and it is there so that the grid is not degenerate: with a common factor
the absolute error would scale identically for every member, worst-case error
over the grid would be the same number as at a single point, and coverage would
not vary across the grid. The analytic identity is checked at `v = 0`, where
`h_i = 1` for every member and the model is exactly the equicorrelated one.
**Factors.** `rho` from 0 to 1 in steps of 0.02 (51 values); `N` in {3, 5, 10,
30}; accuracy spread `s` in {1.0, 1.5, 2.0}. Full crossing, 612 cells, 10,000
draws each. Seeded per cell from a fixed base seed.
**Measured quantities.** Relative gain `1 - MSE_ensemble/MSE_best_member`;
threshold `rho*` for adding a member with variance `(0.5*sigma_best)^2` and
correlation `rho_new`; 90% interval coverage using (a) the true ensemble standard
deviation and (b) the independence assumption `mean(sigma_i)/sqrt(N)`, which is
the stated uncertainty an ensemble typically reports; worst-case absolute error
over the grid; the ensemble's mean member variance, to show that average
accuracy improves while MSE rises.
**Partial correlation.** Over the simulated cells, `gain` against `rho`
controlling for `N` and mean member variance, and `gain` against mean member
variance controlling for `rho` and `N`, both on ranks, via
`code/stats.py:partial_spearman`.
## Decisive comparison, pre-specified
The claim is **SUPPORTED** if all three hold:
1. relative gain falls monotonically as `rho` rises and reaches zero (this is
analytic; it is checked numerically rather than claimed as a discovery);
2. there exists a computed `rho* < 1` at which adding a member with strictly
better individual accuracy still raises ensemble MSE, confirmed both by the
closed form and by the Monte Carlo to within Monte Carlo error;
3. the partial rank correlation of gain with `rho` exceeds in absolute value the
partial rank correlation of gain with member accuracy across the grid.
If (3) fails the claim is **partially supported**; if (2) fails the claim is
**not supported**. The report says plainly that the mathematical part is a
proof, that the simulation confirms arithmetic, and that the empirical content
is confined to the third check, which is a statement about which variable
predicts gain across a grid that the analyst chose.
## Sensitivity, pre-specified
(i) unequal variances: one member four times more accurate than the others
(variance ratio 1:4), and the threshold surface recomputed; (ii) heavy-tailed
errors, multivariate Student-t with 4 degrees of freedom scaled to unit
variance, checking whether the MSE identity and the crossing point survive;
(iii) correlated but biased members, a common bias of 0.5 standard deviations
added to every member, which is the case in which reliability genuinely
degrades and the ensemble cannot average the error away. Each variant reports
the crossing `rho` at which gain reaches zero, the `rho*` surface and the
coverage at high correlation.
## What this test cannot see
Equicorrelation is a strong simplification: real ensembles have correlation
blocks, members of different accuracy correlated more with their own kind, and
errors that are not joint normal. The threshold derived here is exact for the
model it assumes and says nothing about which correlation structure a real
ensemble has. Nor does it say that a lower-correlation member is available at
the same price; the claim implies it, and the test takes that as given.