Theory 064/Information & Computation /Computer architecture
Cache Performance Is a Ratio
Caches are compared by size. What matters is how much of the working set fits.
Cache hit rate is set by the ratio of working set to cache capacity rather than by cache size alone, so a larger cache can perform worse than a smaller one if the working set grows faster than the cache.
01The evidence
Cache behaviour depends on working set size, and applications with large working sets show poor hit rates even in large caches.
02The tension
Procurement and benchmarking compare caches by capacity, which predicts monotonic improvement and fails when working sets scale with the workload.
03The mechanism
A cache holds a working set. Hit rate depends on how much of that set fits, so the relevant variable is the ratio, and a change in workload can offset a change in hardware.
That makes the ratio the predictor, and predicts that hit rate should collapse when the working set crosses the capacity, regardless of absolute size.
04What this is not
Capacity comparison predicts bigger is better. This predicts a ratio, which explains regressions on larger hardware.
05Predictions
- P1Hit rate should collapse when working set exceeds capacity.
- P2Larger caches should fail on larger working sets.
- P3The ratio should predict across workloads.
06Refutation
Hit rate tracking capacity at controlled working set would restore size comparison.
07What it changes
It would change how caches are specified against workloads.