Incoprea Theory
Volume VIII · 100

Theory 064/Information & Computation /Computer architecture

Cache Performance Is a Ratio

Caches are compared by size. What matters is how much of the working set fits.

The claim

Cache hit rate is set by the ratio of working set to cache capacity rather than by cache size alone, so a larger cache can perform worse than a smaller one if the working set grows faster than the cache.

01The evidence

Cache behaviour depends on working set size, and applications with large working sets show poor hit rates even in large caches.

02The tension

Procurement and benchmarking compare caches by capacity, which predicts monotonic improvement and fails when working sets scale with the workload.

03The mechanism

A cache holds a working set. Hit rate depends on how much of that set fits, so the relevant variable is the ratio, and a change in workload can offset a change in hardware.

That makes the ratio the predictor, and predicts that hit rate should collapse when the working set crosses the capacity, regardless of absolute size.

04What this is not

Capacity-based cache comparison

Capacity comparison predicts bigger is better. This predicts a ratio, which explains regressions on larger hardware.

05Predictions

  • P1Hit rate should collapse when working set exceeds capacity.
  • P2Larger caches should fail on larger working sets.
  • P3The ratio should predict across workloads.

06Refutation

Hit rate tracking capacity at controlled working set would restore size comparison.

07What it changes

It would change how caches are specified against workloads.