Theory 067/Information & Computation /Machine learning practice
Data Quality Beats Scale Past a Point
Scaling laws are drawn against data volume. Past a point, the curve may be set by what is in the data.
Beyond a threshold in dataset size, further improvement comes from data quality rather than quantity, so a smaller curated dataset should outperform a larger noisy one at equal compute.
01The evidence
Scaling curves flatten, and curated subsets have been shown to match larger datasets in several domains.
02The tension
Scaling laws are fitted to volume and compute, which predicts continued gains from more data and does not represent quality as a variable.
03The mechanism
A dataset contains signal and noise. Adding data adds both, and once the signal is saturated, additional data adds only noise, which the model fits.
That makes quality the binding constraint past saturation, and predicts a threshold beyond which curation outperforms collection.
04What this is not
Volume laws extrapolate from size. This predicts saturation and a quality regime, which changes the investment decision.
05Predictions
- P1Curated subsets should match larger noisy sets.
- P2Gains from volume should saturate.
- P3Quality metrics should predict performance past the threshold.
06Refutation
Continued gains from volume at controlled quality would leave the scaling law intact.
07What it changes
It would change data investment strategy.