Machine learning · Test 08

Compression does not automatically make a model overconfident

Not supported as stated

Distillation at temperature 1 improved calibration; only high-temperature soft labels made it worse.

The posted claim

Model inference speed trades against calibration, so a distilled model that matches accuracy still produces confidence scores that are too sharp.

Source: Incoprea Theory XX, entry 6

The prediction we tested

Distilled models should be less calibrated at equal accuracy.

What we did

UCI Adult (32,561 training rows, 9,769 test); teacher 64–32 MLP; 16-unit students trained on hard labels and on soft labels at T = 1, 2, 4; expected calibration error with 15 bins.

What happened

0.010ECE, distilled T=1
0.138ECE, distilled T=4
0.029ECE, teacher

The teacher reaches 84.9% accuracy with ECE 0.029. Every student is within 1.1 points of that accuracy. Distillation at T = 1 gives ECE 0.010 — better calibrated than the teacher. T = 2 gives 0.056 and T = 4 gives 0.138, both worse. The hard-label 16-unit control is the best calibrated model in the comparison (ECE 0.004).

Calibration after compression is not uniformly worseTeacher 64–32: 0.0293639128186234040.029Teacher64–32Distilled T=1: 0.010369217689938180.010DistilledT=1Distilled T=2: 0.056111830171592050.056DistilledT=2Distilled T=4: 0.137737871912050840.138DistilledT=4Hard 16: 0.00411846666173035050.004Hard 16Hard 64–32: 0.030845288681599740.031Hard 64–32Expected calibration error (lower is better)Teacher accuracy 84.9%. Every student is within 1.1 points of it. ECE, 15 bins, n=9769.
Compression does not automatically make a model overconfident. Expected calibration error by model, with accuracy matched within about one point. Values regenerate from the code and data in this package.

Verdict

Not supported as a general claim. Compression did not force miscalibration: at T = 1 the student was better calibrated than its teacher, and the hard-label small model was better still. The prediction holds only when high-temperature soft labels are used.

Limits

What would change this answer. Vision or language-model distillation, with repeated seeds, would test whether the temperature dependence is general. The code and inputs are included so the check can be repeated.

Not peer reviewed · Working result, 30 October 2026.