How good is the model?
Before any projection appears on a lifter’s page, the model has to beat simple methods at forecasting results it never saw. This page shows that test as it came out, including where the model lost.
Can model projections be published?
Not yet. The model did not pass the publishing test for any group of lifters, so the site shows no model projections.
- All lifters: does not pass
- 2019: model CRPS 2.949 is not below the best baseline's 2.870
- 2021: model CRPS 3.353 is not below the best baseline's 3.251
- 2023: model CRPS 3.036 is not below the best baseline's 2.974
- pooled difference interval [-0.03, 0.19] does not lie below 0
- 3-4 prior results: does not pass
- 2019: model CRPS 2.944 is not below the best baseline's 2.690
- 2021: model CRPS 3.836 is not below the best baseline's 3.386
- 2023: model CRPS 2.790 is not below the best baseline's 2.647
- pooled difference interval [0.10, 0.45] does not lie below 0
- 2019: 95% intervals caught 89.3% of results, below 90%
- 5-7 prior results: does not pass
- 2019: model CRPS 2.940 is not below the best baseline's 2.838
- pooled difference interval [-0.21, 0.14] does not lie below 0
- 8-11 prior results: does not pass
- 2021: model CRPS 2.938 is not below the best baseline's 2.854
- pooled difference interval [-0.44, 0.17] does not lie below 0
- 12+ prior results: does not pass
- 2023: model CRPS 3.343 is not below the best baseline's 3.023
- pooled difference interval [-0.28, 0.26] does not lie below 0
The test was fixed before these results were computed. A group passes only if, at every cutoff, the model’s calibrated error is lower than the best simple method’s; pooled over cutoffs, the 95% interval of that difference lies wholly below zero; and the model’s 95% intervals caught at least 90% of results at every cutoff.
Trained before 1 Jan 2019
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 3.18 | 3.25 | 91% (was 85%) |
| Gradient boosting (LightGBM) | 3.48 | 3.50 | 92% (was 91%) |
| Last result + age drift | 4.01 | 4.00 | 90% (was 90%) |
| Last result carried forward | 4.47 | 4.46 | 90% (was 90%) |
| Shrunken per-lifter trend | 5.14 | 5.29 | 89% (was 74%) |
Model minus the best simple method: -0.30 (95% interval -0.83 to +0.18): not clearly different from Gradient boosting (LightGBM).
194 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
Trained before 1 Jan 2021
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 2.94 | 3.02 | 93% (was 80%) |
| Gradient boosting (LightGBM) | 2.85 | 2.84 | 97% (was 96%) |
| Last result + age drift | 3.65 | 3.66 | 97% (was 95%) |
| Last result carried forward | 3.97 | 3.96 | 97% (was 95%) |
| Shrunken per-lifter trend | 7.98 | 8.68 | 72% (was 49%) |
Model minus the best simple method: +0.08 (95% interval -0.46 to +0.65): not clearly different from Gradient boosting (LightGBM).
172 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
Trained before 1 Jan 2023
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 3.34 | 3.38 | 93% (was 82%) |
| Gradient boosting (LightGBM) | 3.49 | 3.50 | 91% (was 91%) |
| Last result + age drift | 3.76 | 3.75 | 93% (was 91%) |
| Last result carried forward | 4.25 | 4.26 | 92% (was 92%) |
| Shrunken per-lifter trend | 5.36 | 5.60 | 88% (was 74%) |
Model minus the best simple method: -0.15 (95% interval -0.62 to +0.27): not clearly different from Gradient boosting (LightGBM).
167 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
How this is tested
- Every method, the model’s population parameters included, is fitted only on results before the cutoff, then forecasts each lifter’s results over the next three years.
- Error is the continuous ranked probability score (CRPS) in GL points: it rewards a forecast for being close and for being honest about its uncertainty. It is computed from 39 quantiles, which overstates the exact value by a steady 2–3% for every method alike.
- Calibration widens or narrows each method’s intervals by a factor fitted on the cutoff three years earlier, whose results all happened before this cutoff. It never moves a forecast’s centre, and every method gets the same treatment.
- The interval on the difference resamples lifters rather than results, because one lifter’s results rise and fall together.
Version 3c-v1.1-d38974f-2026-09-06, data to 6 Sept 2026.