Theory 23/Software Security /Empirical security/3 min
Vulnerabilities Follow Code Churn
Security review budgets by size. The defects may be where the code is moving.
Vulnerability density in a codebase is set by the rate of change of the code rather than by its size or its age, so projects with identical line counts can differ by an order of magnitude in vulnerability rate, and the rate should be predicted from churn before any code review.
01The evidence
- An empirical study of the security of open source web applications examined vulnerability occurrence against project characteristics, and found that the determinants are properties of the development process rather than of size alone.ESEM, 2009
- The same study reports vulnerability data across projects, which is the structure a churn-based predictor requires: a per-project rate that varies more than size alone would suggest.ACM proceedings record
02The tension
Security resources are allocated by size and by criticality: larger codebases get more review, and older code is assumed to be more mature. Both assumptions are reasonable and both fail to predict which projects have the most vulnerabilities per line.
03The theory
A vulnerability is introduced when code is written or changed. Code that never changes has already been examined, by users and by attackers, so its remaining defects are few. Code that changes constantly is new code, and new code carries new defects.
If churn is the driver, then vulnerability density should scale with change rate per line rather than with line count, and the relation should hold across languages and project ages. Mature projects should look safe not because they are old but because they have stopped moving.
The prediction is that churn measured over the preceding period should predict future vulnerability discovery better than size, age or complexity metrics, and that a project that doubles its change rate should roughly double its rate.
The practical consequence is that review effort should follow churn, which is cheap to measure from version control, rather than following size, which is the current proxy.
04What this is not
Those heuristics assume defects scale with lines and decline with age. This makes churn the driver, which predicts that a large stable project is safer than a small volatile one.
05Predictions
- P1Churn should predict future vulnerability discovery better than size or age.
- P2Vulnerability density should scale with change rate per line across projects.
- P3Doubling churn should approximately double the discovery rate.
06What would refute it
Vulnerability rate tracking size after churn is controlled would restore the size heuristic.
07What it would change
It would reallocate security review by change rate, and make version-control history a security signal.
08Evidence base
- Security of open source web applications — ESEMhttps://www.computer.org/csdl/proceedings-article/esem/2009/05314215/12OmNyGtjmg
- Security of open source web applications — ACM recordhttps://dlnext.acm.org/doi/10.1109/ESEM.2009.5314215