How good is the model?
Before any projection appears on a lifter’s page, the model has to beat simple methods at forecasting results it never saw. This page shows that test as it came out, including where the model lost.
Can model projections be published?
Not yet. The model did not pass the publishing test for any group of lifters, so the site shows no model projections.
- All lifters: does not pass
- 2019: model CRPS 2.949 is not below the best baseline's 2.870
- 2021: model CRPS 3.353 is not below the best baseline's 3.251
- 2023: model CRPS 3.036 is not below the best baseline's 2.974
- pooled difference interval [-0.03, 0.19] does not lie below 0
- 3-4 prior results: does not pass
- 2019: model CRPS 2.944 is not below the best baseline's 2.690
- 2021: model CRPS 3.836 is not below the best baseline's 3.386
- 2023: model CRPS 2.790 is not below the best baseline's 2.647
- pooled difference interval [0.10, 0.45] does not lie below 0
- 2019: 95% intervals caught 89.3% of results, below 90%
- 5-7 prior results: does not pass
- 2019: model CRPS 2.940 is not below the best baseline's 2.838
- pooled difference interval [-0.21, 0.14] does not lie below 0
- 8-11 prior results: does not pass
- 2021: model CRPS 2.938 is not below the best baseline's 2.854
- pooled difference interval [-0.44, 0.17] does not lie below 0
- 12+ prior results: does not pass
- 2023: model CRPS 3.343 is not below the best baseline's 3.023
- pooled difference interval [-0.28, 0.26] does not lie below 0
The test was fixed before these results were computed. A group passes only if, at every cutoff, the model’s calibrated error is lower than the best simple method’s; pooled over cutoffs, the 95% interval of that difference lies wholly below zero; and the model’s 95% intervals caught at least 90% of results at every cutoff.
Trained before 1 Jan 2019
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 2.94 | 2.96 | 89% (was 88%) |
| Gradient boosting (LightGBM) | 2.69 | 2.69 | 97% (was 96%) |
| Last result + age drift | 2.74 | 2.74 | 96% (was 97%) |
| Last result carried forward | 2.94 | 2.95 | 95% (was 96%) |
| Shrunken per-lifter trend | 7.62 | 7.65 | 80% (was 84%) |
Model minus the best simple method: +0.25 (95% interval +0.01 to +0.48): higher (worse) than Gradient boosting (LightGBM).
726 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
Trained before 1 Jan 2021
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 3.84 | 3.92 | 91% (was 83%) |
| Last result + age drift | 3.39 | 3.39 | 96% (was 96%) |
| Gradient boosting (LightGBM) | 3.40 | 3.40 | 97% (was 96%) |
| Last result carried forward | 3.59 | 3.60 | 96% (was 97%) |
| Shrunken per-lifter trend | 11.73 | 11.61 | 74% (was 80%) |
Model minus the best simple method: +0.45 (95% interval +0.05 to +0.85): higher (worse) than Last result + age drift.
364 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
Trained before 1 Jan 2023
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 2.79 | 2.78 | 97% (was 92%) |
| Last result + age drift | 2.65 | 2.66 | 98% (was 98%) |
| Gradient boosting (LightGBM) | 2.67 | 2.66 | 96% (was 95%) |
| Last result carried forward | 3.04 | 3.05 | 98% (was 99%) |
| Shrunken per-lifter trend | 5.32 | 5.37 | 93% (was 95%) |
Model minus the best simple method: +0.14 (95% interval -0.24 to +0.51): not clearly different from Last result + age drift.
394 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
How this is tested
- Every method, the model’s population parameters included, is fitted only on results before the cutoff, then forecasts each lifter’s results over the next three years.
- Error is the continuous ranked probability score (CRPS) in GL points: it rewards a forecast for being close and for being honest about its uncertainty. It is computed from 39 quantiles, which overstates the exact value by a steady 2–3% for every method alike.
- Calibration widens or narrows each method’s intervals by a factor fitted on the cutoff three years earlier, whose results all happened before this cutoff. It never moves a forecast’s centre, and every method gets the same treatment.
- The interval on the difference resamples lifters rather than results, because one lifter’s results rise and fall together.
Version 3c-v1.1-d38974f-2026-09-06, data to 6 Sept 2026.