Test 04 — Vulnerabilities Follow Code Churn
Supplied working report; original verdicts not validated.
Review status: Held for correction. This is an archival source note, not a passed proof.
README.md
# Test 04 — Vulnerabilities Follow Code Churn **Verdict: not supported.** Churn does predict advisories, but it is not the main driver: package size explains five times more deviance than release rate, and the hypothesis's discriminating claim — churn *rather than* size or age — fails on its own terms. Source: https://incoprea.com/research-library/original-theories-vol4-20261030/contents.html ## The result in one paragraph Across 220 npm packages drawn from the dependency closure of ten popular libraries, 51 carried at least one OSV advisory. In a quasi-Poisson model with log age as the offset — so the outcome is advisories per package-year — release rate carries an incidence-rate ratio of 1.74 per standard deviation (p = 0.04). That is a real and positive association, in the direction the hypothesis predicts. But package size has an IRR of 2.03 (p = 5 × 10⁻⁶) and explains 0.101 of the deviance against churn's 0.019; popularity adds a further 0.021. Age contributes almost nothing (0.003, p = 0.42). The ordering the hypothesis needs is reversed. ## Numbers | Term | IRR per SD | p (quasi-Poisson) | Partial deviance explained | |---|---|---|---| | Releases per year (churn) | 1.74 | 0.040 | 0.019 | | Unpacked size | **2.03** | **5.2 × 10⁻⁶** | **0.101** | | Age (also the offset) | 1.33 | 0.42 | 0.003 | | Downloads last month | 1.37 | 0.034 | 0.021 | - Packages: 220; with at least one advisory: 51; mean advisories 1.34, median 0. - Median release rate: 2.33 per year; dispersion 6.18, so p-values are scaled for overdispersion. - Robustness: replacing releases per year with the median interval between releases gives an IRR of 0.64 — longer gaps, fewer advisories — the same direction as churn. ## What this means for the hypothesis The hypothesis is a claim about relative importance, not about sign, so the sign result alone does not save it. On this sample: 1. *Churn predicts advisories.* True, modestly, and it survives controls for size, age and popularity. A package publishing twice as often (in SD terms) has about 74% more advisories per year. 2. *Churn rather than size.* False. Size dominates by a factor of five in deviance. Bigger codebases have more attack surface, and the data say that is the stronger regularity. 3. *Projects with identical line counts differ by an order of magnitude.* Not testable here directly, but the model implies the spread across packages is driven mostly by size, with churn a secondary adjustment. A fair summary: churn is a real but secondary risk marker, and size is the better single predictor of how many advisories a JavaScript package accumulates. The hypothesis is wrong about which one dominates. ## Method Dependency closure (dependencies only, two levels) of `express`, `react`, `vue`, `webpack`, `eslint`, `axios`, `mongoose`, `socket.io`, `jest` and `next`, capped at 300 and sampled to 220 packages with usable metadata. Churn is versions published in the last three years divided by three, from the npm registry's publish timestamps; size is the unpacked size of the latest tarball; popularity is last month's downloads from the npm downloads API; advisories are distinct IDs returned by the OSV batch query. Quasi-Poisson regression with log age as an offset; partial deviance from dropping each term; dispersion estimated from the Pearson statistic. Everything is in `run.py`. ## Limits Advisory counts measure disclosure, not defects. OSV coverage of npm is uneven over time, and a package with no advisory is not a safe package — it may simply be unexamined. Size and churn are correlated, so the partial deviance comparison is sensitive to measurement: unpacked size counts bundled dependencies as well as source. Packages are not independent (shared maintainers, shared dependencies, shared disclosure channels) and no clustering is applied. The sample is the tooling ecosystem around ten popular libraries, not application code, and 169 of 220 packages have zero advisories, which limits power. Finally, release count is a coarse proxy for code change: a package can publish often while changing little. ## Files - `hypothesis.md` — the claim and the pre-specified criteria - `run.py` — fetch, model, figures - `data.csv` — package-level table - `results.json` — every number quoted above - `figures/fig1_churn.png`, `fig2_partial_deviance.png` - `SOURCE.md` — endpoints, counts, hashes *AI-assisted analysis (DeepSeek coding agent). Not peer reviewed.*
hypothesis.md
# Test 04 — Vulnerabilities Follow Code Churn **Source hypothesis (verbatim, Incoprea Research, *Software Security* #23).** > Vulnerability density in a codebase is set by the rate of change of the code rather > than by its size or its age, so projects with identical line counts can differ by an > order of magnitude in defect density. Published at https://incoprea.com/research-library/original-theories-vol4-20261030/contents.html ## What the hypothesis predicts - **P1.** Packages that publish more releases per year should carry more security advisories, after controlling for how long they have existed. - **P2.** The churn effect should be larger than the size effect and the age effect. This is the discriminating claim: "rather than by its size or its age". - **P3.** The relation should not be an artefact of attention. Popular packages are both more actively developed and more closely audited, so downloads are controlled for. If churn only predicts advisories through popularity, the hypothesis fails. ## Design fixed before the numbers were computed **Universe.** The dependency closure (depth 2, dependencies only, not devDependencies) of ten widely used npm packages: `express`, `react`, `vue`, `webpack`, `eslint`, `axios`, `mongoose`, `socket.io`, `jest`, `next`. Closure taken at fetch time; capped at 260 packages by seeded random sample. This is a sample of *maintained JavaScript packages*, not of all software. **Measures.** From the npm registry, per package: - churn: versions published in the last three years ÷ 3 (releases per year), and, as a robustness variant, the median interval between releases in the last three years; - age: years from first publish to the fetch date; - size: unpacked bytes of the latest version, log-transformed; - popularity: downloads in the last month, log-transformed. From the OSV database (batch query, one call per package), the number of distinct advisory IDs affecting the package. **Model.** Poisson regression of advisory count on standardised log churn, log size, log age and log downloads, with `log(age)` as an offset — the model is of the *rate* per package-year, which is the only fair comparison between a two-year-old and a ten-year-old package. Standard errors are quasi-Poisson scaled by the estimated dispersion. Partial deviance explained is reported for each term when added last. **Robustness.** (a) median release interval instead of releases per year; (b) exclude the ten root packages, which are larger and more scrutinised than their dependencies; (c) restrict to packages older than three years. ## Support / refute, fixed in advance - **Support** if churn has an incidence-rate ratio above 1 with p < 0.01, its partial deviance exceeds both size and age, and the result survives all three robustness variants. - **Partial support** if churn is significant but size or age explains more. - **Refute** if churn adds nothing once size, age and downloads are in the model, or if the sign is negative. ## What this design cannot see Advisory counts measure *disclosure*, not defects: a package can be full of bugs and never receive a CVE, and a well-audited package accumulates advisories faster than an obscure one with the same code quality. Downloads are an imperfect proxy for scrutiny, and OSV's coverage of the npm ecosystem is uneven across time. Release counts are a proxy for code change, not a measure of lines changed; a package can publish often while changing little, or change enormously in one release. Packages are not independent — they share maintainers, dependencies and disclosure channels — and no clustering or network correction is applied. Finally, the sample is the closure of ten popular packages, so it over-represents the tooling ecosystem and under-represents application code. ## Disclosure Designed and executed with AI assistance (DeepSeek coding agent). Not peer reviewed.