A calibration claim has a floor
A perfectly calibrated model does not score an Expected Calibration Error of zero. On a finite sample it scores something above zero, and on a few hundred rows that something is often larger than the differences people report. Below a certain row count, a calibration claim is not measurable at all. This page lets you check where your own numbers sit, without installing anything.
Measured ECE 0.0740
Calibrated-model floor, 95th percentile 0.1109
INCONCLUSIVE The measured figure sits inside what a perfectly calibrated model scores on 105 rows, so this dataset establishes nothing either way. See it derived from the rows.
The argument
ECE sorts predictions into bins by the probability the model gave, and in each bin compares that probability with how often the model was actually right. It averages the gaps. Zero would be perfect.
Zero is not reachable. Each bin holds a handful of cases, and a handful of outcomes does not land exactly on its own probability, any more than ten coin flips land on exactly five heads. The gaps are absolute values, so that scatter does not cancel out: it adds up. The result is a floor under ECE that has nothing to do with the model and everything to do with the row count.
So a measured ECE means nothing until you know what a perfectly calibrated model would have scored on the same number of rows. plumbline computes that floor by simulation and reads every calibration figure against it. When the figure sits inside the floor, the verdict is INCONCLUSIVE, and that is not a pass. It means the dataset cannot tell your model apart from a perfect one, so nothing was established in either direction.
Floor calculator
Enter a configuration to see the calibrated-model floor, and optionally a measured ECE to see whether it clears it.
Calibrated-model floor
- Mean
- 95th percentile
- Configuration
This is an estimate from summary numbers. Your real predictions can give a different floor at the same row count; the worked example shows why, and you can score your own predictions instead.
Worked example: the report's ECE line
The repository's example report prints this line:
Loading the example...
Below, this page derives that line from the 105 rows behind it, rather than quoting it.
The worked example's rows are assembled when the site is deployed, by
scripts/build_site.py. They are not present in this copy of the page.
-
The rows
-
The measured ECE
Sort the 105 predicted probabilities into 10 equal-width bins. In each, compare the mean probability with the share that was right. ECE is the row-weighted mean of the gaps.
Reliability bins, computed from the rows (empty bins omitted) Bin Rows Mean predicted Share right Gap ECE, row-weighted over -
The floor
Hold those 105 probabilities fixed and redraw every outcome from its own probability, 2,000 times. Each redraw is a perfectly calibrated model by construction, of exactly this size and shape. The spread of their ECEs is the floor.
-
The verdict
The same n and accuracy, from summary numbers only
Put this run's row count and accuracy into the calculator above and it gives a different floor:
- From the 105 predictions
- From n and accuracy alone
Neither is wrong. The floor depends on how the predictions are spread across the probability range, not only on how many there are and how often they were right. The calculator has to invent that spread from the accuracy. The real run put its predictions where the model put them:
| Bin | Real | Invented |
|---|
This is why plumbline needs your actual predictions. The calculator gives an estimate from summary numbers. The tool gives the real floor, from your data, and so does the next section.
Your own predictions
Paste the predictions a model made and whether each one was right, and this computes their ECE, the floor for exactly those predictions, and the verdict a plumbline report would print, word for word.
Nothing you paste leaves this page. It is read and scored by the script on this page, in your browser. It is not uploaded, not stored, and not put in the address, so reloading the page clears it.
Measured against its own floor
- Rows
- Measured ECE
- Floor mean
- 95th percentile
A binary classifier's probability of the positive class, with the true label as the outcome, can be scored the same way. That measures whether the positive class's probabilities hold, which is a different figure from the report's ECE for the same classifier: the report scores the confidence in whichever answer was chosen.
Sample-size planner
Planning estimateThe question to ask before collecting labels: to detect an ECE of a given size, how many rows do you need? This searches over the row count for the point where a measured ECE of that size clears the calibrated-model floor's 95th percentile.
Floors computed on the way
Read the answer as a planning estimate, for two reasons. It is built on the summary floor, which invents the spread of predictions from the accuracy; as the worked example shows, the floor from your real predictions can land either side of it. And it is the row count at which a measured value of that size would clear the floor, not the count at which a model miscalibrated by that much is reliably caught. For that, plan for more.
What computes these numbers
Everything on this page is computed in your browser by a JavaScript reimplementation of plumbline's floor, not by the Python itself. It reproduces numpy's seeded random stream draw for draw (the PCG64 generator and numpy's own normal, gamma, and beta samplers), so with the seeds the Python uses it redraws exactly the outcomes the Python draws. It is checked against values exported from the Python on every change, in CI, and the site does not deploy if they disagree by more than 1e-9. That tolerance is far above the rounding differences left once the random streams agree (around 1e-15) and far below what one mismatched draw would cause (around 1e-3).
- The calculator and planner call
synthetic_floor(n, n_bins, accuracy), checked against floor-golden.json. - The worked example calls
calibration_flooron the example's rows, regenerated at deploy time from the seeded mock adapter by the command the report records. The deploy fails unless the derived line matches the report character for character. No vendor's output is on this page. - Your own predictions go through
ece_figure, the call a report makes: the ECE, thencalibration_flooron the same rows. The check holds it to the Python on pasted text, including a value that sits exactly on a rounding tie. - The check is scripts/check_floor_parity.mjs; the implementation is floor.js.