The Dimensionality Ceiling

01The evidence

• Effective dimensionality measures now separate the number of parameters a network has from the number of directions it actually uses, and the two diverge dramatically in trained models.Two-scale complexity measure, ETH
• Complexity measures tied to function behaviour rather than parameter counts have become a productive line of work for explaining why overparameterised models generalise at all.NeurIPS, 2024

02The tension

Double descent shows that increasing capacity past the interpolation threshold can improve test performance, which destroyed the classical bias-variance picture. But it does not explain why performance eventually degrades at very large capacity or why some tasks degrade immediately. Parameter count cannot order these cases.

03The theory

A trained network does not use its parameters uniformly. Its activations and weights concentrate in a subspace, and the dimension of that subspace is what determines its capacity to fit and to generalise. Two models with the same parameter count can have effective dimensionalities that differ by a factor of ten.

Classical learning theory assumes the effective dimensionality equals the parameter count and predicts a trade-off between capacity and generalisation. If the task has an intrinsic dimensionality, the trade-off is between the model's effective dimension and the task's, not between parameters and data.

The prediction is a ceiling: past a factor of roughly two or three above the task's intrinsic dimension, additional effective dimension fits noise rather than signal, and test error rises. Below it, the model underfits. The optimum is a ratio, and ratios are testable in a way that absolute counts are not.

The reason this has not been seen is that effective dimensionality is not a hyperparameter. It emerges during training and depends on the optimiser, the architecture and the data. Measuring it requires a spectral analysis of the trained network rather than a count taken at initialisation.

04What this is not

05Predictions

• P1Across architectures and tasks, peak test performance should occur at a ratio of effective to intrinsic dimension near two to three.
• P2Pruning or projecting a model to that ratio should preserve performance, while projecting further should degrade it sharply.
• P3The ratio should hold when intrinsic dimension is estimated independently, without fitting it to the performance curve.

06What would refute it

A ratio that varies by an order of magnitude across tasks and architectures, or that cannot be estimated before training, would make the ceiling descriptive rather than predictive.

07What it would change

It would give model selection a measurable target and connect deep learning to the older idea of intrinsic dimension in statistics.