How good is the model?
Before any projection appears on a lifter’s page, the model has to beat simple methods at forecasting results it never saw. This page shows that test as it came out, including where the model lost.
Can model projections be published?
Not yet. The model did not pass the publishing test for any group of lifters, so the site shows no model projections.
- All lifters: does not pass
- 2019: model CRPS 2.949 is not below the best baseline's 2.870
- 2021: model CRPS 3.353 is not below the best baseline's 3.251
- 2023: model CRPS 3.036 is not below the best baseline's 2.974
- pooled difference interval [-0.03, 0.19] does not lie below 0
- 3-4 prior results: does not pass
- 2019: model CRPS 2.944 is not below the best baseline's 2.690
- 2021: model CRPS 3.836 is not below the best baseline's 3.386
- 2023: model CRPS 2.790 is not below the best baseline's 2.647
- pooled difference interval [0.10, 0.45] does not lie below 0
- 2019: 95% intervals caught 89.3% of results, below 90%
- 5-7 prior results: does not pass
- 2019: model CRPS 2.940 is not below the best baseline's 2.838
- pooled difference interval [-0.21, 0.14] does not lie below 0
- 8-11 prior results: does not pass
- 2021: model CRPS 2.938 is not below the best baseline's 2.854
- pooled difference interval [-0.44, 0.17] does not lie below 0
- 12+ prior results: does not pass
- 2023: model CRPS 3.343 is not below the best baseline's 3.023
- pooled difference interval [-0.28, 0.26] does not lie below 0
The test was fixed before these results were computed. A group passes only if, at every cutoff, the model’s calibrated error is lower than the best simple method’s; pooled over cutoffs, the 95% interval of that difference lies wholly below zero; and the model’s 95% intervals caught at least 90% of results at every cutoff.
Trained before 1 Jan 2019
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 2.94 | 2.99 | 92% (was 86%) |
| Gradient boosting (LightGBM) | 2.84 | 2.83 | 94% (was 93%) |
| Last result + age drift | 3.12 | 3.12 | 95% (was 94%) |
| Last result carried forward | 3.28 | 3.28 | 95% (was 95%) |
| Shrunken per-lifter trend | 5.31 | 5.54 | 90% (was 80%) |
Model minus the best simple method: +0.10 (95% interval -0.12 to +0.36): not clearly different from Gradient boosting (LightGBM).
554 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
Trained before 1 Jan 2021
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 3.15 | 3.19 | 92% (was 84%) |
| Gradient boosting (LightGBM) | 3.40 | 3.39 | 96% (was 94%) |
| Last result + age drift | 4.08 | 4.08 | 93% (was 93%) |
| Last result carried forward | 4.24 | 4.24 | 93% (was 93%) |
| Shrunken per-lifter trend | 7.38 | 7.53 | 85% (was 76%) |
Model minus the best simple method: -0.25 (95% interval -0.57 to +0.08): not clearly different from Gradient boosting (LightGBM).
289 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
Trained before 1 Jan 2023
| Method | Error (CRPS) | Before calibration | 95% intervals caught |
|---|---|---|---|
| Potential model | 3.00 | 3.06 | 92% (was 89%) |
| Gradient boosting (LightGBM) | 3.07 | 3.08 | 94% (was 94%) |
| Last result + age drift | 3.20 | 3.21 | 96% (was 96%) |
| Last result carried forward | 3.46 | 3.46 | 93% (was 94%) |
| Shrunken per-lifter trend | 5.99 | 6.05 | 94% (was 84%) |
Model minus the best simple method: -0.07 (95% interval -0.44 to +0.29): not clearly different from Gradient boosting (LightGBM).
268 results in the three years after the cutoff, every method scored on the same ones. Lower error is better.
How this is tested
- Every method, the model’s population parameters included, is fitted only on results before the cutoff, then forecasts each lifter’s results over the next three years.
- Error is the continuous ranked probability score (CRPS) in GL points: it rewards a forecast for being close and for being honest about its uncertainty. It is computed from 39 quantiles, which overstates the exact value by a steady 2–3% for every method alike.
- Calibration widens or narrows each method’s intervals by a factor fitted on the cutoff three years earlier, whose results all happened before this cutoff. It never moves a forecast’s centre, and every method gets the same treatment.
- The interval on the difference resamples lifters rather than results, because one lifter’s results rise and fall together.
Version 3c-v1.1-d38974f-2026-09-06, data to 6 Sept 2026.