plumbline

CI

Measure whether a decision model's probabilities are trustworthy on your own labeled data, and decide what to do about it.

Check where your own ECE sits against its floor, in the browser, with nothing installed: tmhsdigital.github.io/plumbline.

v0.1.0, one maintainer. The measurement behaviour is settled; the Python API and the CLI flags are not, and will change in v0.2. Pin a version if you build on it.

What it is #

A decision model is one you ask a closed question and get back a label plus a number: "which of these five categories is this ticket" together with "0.83". That number is the point. If it is honest, your code can act on 0.95 and route 0.6 to a human, and you have a system. If it is not, you have a confident guess in a trench coat.

plumbline is a command line tool that takes your labeled rows, runs one or more models over them, and tells you three things:

1. Is the number honest? A model is calibrated when the things it calls 70% likely happen about 70% of the time. plumbline measures that, and crucially measures it against what the same figure would look like if the model were perfect, because on a few hundred rows those are closer than anyone expects.
2. Where do I set my threshold? Act automatically above it, escalate below it.
3. What does that save me? A cascade runs a cheap model first and sends only the uncertain cases to an expensive one. Given what an escalation costs you and what a mistake costs you, plumbline says where to cut and what you get for it.

It is a measuring instrument. It has no opinion about which model you should pick, and it will refuse to answer a question your data cannot support.

What this is not #

plumbline is not a leaderboard. It does not rank vendors, it publishes no combined score, and it will not tell you which model is best.

If you want a cross-vendor ranking of decision models, go to JevBench and Benchmark Heaven. That is their job and they do it properly, across many models, on a shared dataset, with a published methodology. plumbline deliberately does not compete with it, and a number produced here must never be compared with a number they publish: different harness, different prompts, different scoring.

plumbline answers a different question, and it is the question a ranking structurally cannot answer:

• Does this model work on your labeled data, with your label set and your distribution?
• Where should you set your confidence threshold?
• What does a cascade at that threshold actually save you?

A leaderboard tells you how a model did on someone else's rows. Only your rows can tell you whether its probabilities mean anything where you intend to use them.

The argument #

The standard way to score calibration is Expected Calibration Error: sort the predictions into bins by confidence, and in each bin compare the claimed confidence against how often the model was actually right. Average the gaps. Zero would be perfect.

Zero is not achievable, and that is the problem. With a finite number of rows, each bin holds a handful of cases, and a handful of coin flips does not land exactly on its own probability. That scatter, binning noise, puts a floor under ECE that has nothing to do with the model. The floor rises as your row count falls. A perfectly calibrated model on a few hundred rows does not score 0, and if you do not know what it would score, you cannot read your own number.

This is not a rounding concern. On a few hundred rows, a calibration claim is frequently not measurable at all.

So plumbline computes that floor by simulation and prints every inferential figure against it. Here is a real line from the example report:

ECE 0.0740 over 105 rows (10 equal width bins), against a calibrated-model floor of 0.0707 (95th percentile 0.1109): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable.

Read "inconclusive" as an absence of a result, not a pass. It means your rows cannot tell your model apart from a perfect one, so nothing was established in either direction. A genuinely well calibrated model and a badly calibrated one both land there on too few rows, and the figure does not say which you have. Taking it as a clean bill of health inverts the conclusion, and it is the easiest mistake to make with this tool.

On that same run's 105 choice and yes/no rows, three of the four headline figures came back inconclusive and only accuracy cleared its null. A tool that printed the other three alone would be handing you numbers that look like findings and are not. That refusal is the product.

Quickstart #

Prerequisites: Python 3.12 or later, and uv. An older Python gives a resolver error rather than a clear message, so check with python --version first.

plumbline is not on PyPI, and pip install plumbline installs something else. The name on PyPI belongs to an unrelated project, so that command succeeds and gives you the wrong tool. Clone the repository, which is the intended install path for v0.1. If you use pip rather than uv, install from the repository itself:

pip install "plumbline @ git+https://github.com/TMHSDigital/plumbline"
pip install "plumbline[local] @ git+https://github.com/TMHSDigital/plumbline"   # with the local extra

Nothing in this first section needs an API key or spends anything.

git clone https://github.com/TMHSDigital/plumbline
cd plumbline
uv sync

uv run plumbline run datasets/public/jevbench-hard.jsonl `
    --adapter mock --format jevbench `
    --results results --report results/report.md

git clone https://github.com/TMHSDigital/plumbline
cd plumbline
uv sync

uv run plumbline run datasets/public/jevbench-hard.jsonl \
    --adapter mock --format jevbench \
    --results results --report results/report.md

That loads the vendored public fixture, runs a deterministic seeded mock over it, computes every metric against its null, and writes both a results artifact and a report. No network call. Both land in results/, which is gitignored, so following this leaves your clone clean.

Expected output shape:

111 rows read from datasets/public/jevbench-hard.jsonl, 111 loaded, 0 refused. ...
artifact: results/20260925T231349+0000-mock-420956a9.json
report: results/report.md

Those lines go to stderr. Without --report, the report itself is the only thing on stdout, so plumbline run ... > report.md captures just the report. The exit code is 0 when the run produced figures and 1 when it could not start (a bad option, a missing key, an unreadable file) or when every case failed; in that last case the artifact and the report are still written, and the reason is printed when all the cases share one.

uv run plumbline adapters   # what this install can run
uv run plumbline version
uv run plumbline run --help

Running a local model, still without a key #

local_logits reads option-token probabilities out of a checkpoint on your own machine, so it needs no API key. It does need the optional local extra, which a plain uv sync does not install:

uv sync --extra local

Without it every case fails with a message telling you this, so if a local run reports no figures at all, that is the first thing to check.

Name the checkpoint, pin it to a commit, and say where it runs:

uv run plumbline run datasets/public/jevbench-hard.jsonl --format jevbench \
    --adapter local_logits --model Qwen/Qwen2.5-1.5B-Instruct \
    --revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 --device cuda

--device defaults to cpu, which works anywhere and is slow. On Windows the torch that PyPI serves is CPU only; for --device cuda, install a CUDA build of torch from the PyTorch index into the same environment.

By default the arm reads each option's own token, so it scores only options that are a single token for the checkpoint and refuses the rest by name. The public fixture's choice options are mostly multi-word identifiers, so that way 66 of its 67 choice rows are refused. Add --option-style letter to ask the options as A, B, C and read the letters instead: every row is scored, and the report says the question was lettered. On the fixture that run landed at chance accuracy with an ECE of 0.349 against a floor of 0.087: a small model that is confidently wrong, which is the case a calibration figure exists to catch.

Running a hosted vendor #

This one spends money. Set a key, name an adapter, and cap the run.

Cost needs a pricing table you supply, because plumbline ships no figures for vendors whose terms treat pricing as confidential. Copy the template and fill in the rates from the vendor's own page, and the date you read them. plumbline refuses the copy until you have, so an unedited template can never price a run at nothing:

cp docs/pricing.example.json my-pricing.json

$env:TYPESAFE_API_KEY = "your-key-here"

uv run plumbline run datasets/public/jevbench-hard.jsonl `
    --adapter typesafe_wire --model jev-latest --format jevbench `
    --results results --report results/report.md `
    --limit 40 --pricing my-pricing.json

export TYPESAFE_API_KEY="your-key-here"

uv run plumbline run datasets/public/jevbench-hard.jsonl \
    --adapter typesafe_wire --model jev-latest --format jevbench \
    --results results --report results/report.md \
    --limit 40 --pricing my-pricing.json

Without a pricing table the run still works; cost reports as unpriced, and --max-cost-usd refuses rather than bounding a run it cannot cost. See Limitations.

Add --dry-run to either command first to see what the run would do without spending anything. It loads and checks the dataset, builds the adapter, applies the guard, and prints the case count, the endpoint, and the estimated cost, then exits without sending a request or writing a file. Pass --base-url to point typesafe_wire at a self-hosted endpoint, and --timeout to bound one request; both are recorded in the artifact.

Your own data goes in datasets/private/, which is gitignored, and that is the only path on which the recalibration numbers mean anything. Your own data gives the row format, what is refused and why, and the commands to run next: a local checkpoint, the cascade's two costs, and plumbline report for runs that already happened.

Example report #

docs/example-report.md is the output of one seeded mock run, and CI holds it to that: on every change and before every deploy it reruns the report's recorded command and refuses if any line differs, other than the date. The ECE line is quoted in The argument above. Three more, each showing the tool declining to do something:

Not reported. recalibration needs at least 200 held-out evaluation rows and this split has 53. Fitting a temperature on fewer rows produces a number whose uncertainty is larger than the correction it claims to make, and it arrives looking like a measurement.

Not reported. A threshold is decided by two numbers no benchmark can know: what one escalation to the expensive arm costs, and what one wrong answer costs.

38 noul rows were asked as choice questions, which is a different question from the one the dataset states. Not comparable with an arm that asked them as noul.

A Noul is a yes/no question that returns one probability directly, rather than a distribution over options. Asking a yes/no row as a two-option choice is a different question, so the report keeps the two apart instead of averaging across the difference.

The arm in that report is the seeded mock, labeled as such at the top of the page, so anyone can reproduce it with one command and no key. Its numbers are properties of plumbline's harness, not a measurement of any vendor.

Adapters and probability semantics #

Three real transports, plus a mock for smoke tests. The intent is that adding a vendor is config rather than code, and that intent has so far been verified against one endpoint; demonstrating it against a second is issue #11.

The "Run for real" column is deliberate. Two of these have only ever run against test fakes, which is issue #3 and the highest-value item in the backlog.

The Jev wire format below is one vendor's HTTP shape for typed questions: you send some state plus a question with declared options, and you get back a selected option, a probability for each, and a confidence. Several open models and self-hosted servers speak it, which is why one adapter covers all of them.

The semantics classes are the whole reason the report refuses some comparisons:

• calibrated_claim. The vendor asserts these probabilities are calibrated. plumbline records the claim and never asserts it is true. Testing it is the point.
• restricted_softmax. A softmax over the declared options only, with no calibration claim. Add an option and every number moves without anything about the case having changed, so it is a statement about your option set rather than a probability of correctness in the world. Whether it behaves like a calibrated probability on your workload is measurable, so plumbline measures it.
• none. No probability at all. Excluded from calibration entirely, never imputed, never defaulted to zero.

The report groups arms by this field and will not place figures from different groups side by side.

Limitations #

Specific, and none of them are going to surprise you later.

• Score rows get three rank-aware figures and nothing else. An ordinal level is read by mean absolute error, the ranked probability score, and a cumulative calibration error, each against its own null, in a block apart from the choice figures. Recalibration and the cascade are not applied to them.
• One request per case, no batching. Cost and latency figures are therefore conservative relative to batched use, where a single call carrying many questions against one shared state is materially cheaper and faster.
• Temperature scaling, globally or per predicted label. When one temperature is the wrong shape, the report tries one per predicted label, and refuses both rather than return a correction that does not fit. Vector and matrix scaling are not fitted.
• Recalibration needs 200 held-out rows. Below that it refuses. Most datasets people try first will not reach it.
• Cost requires a pricing table you supply. plumbline ships no figures for vendors whose terms treat pricing as confidential. The shipped entry names the page to read, and you pass your own table with --pricing. Without one, cost reports as model_not_priced and --max-cost-usd refuses the run outright, because a guard cannot bound a run it cannot cost.
• The METHODOLOGY numbers derived from the seeded mock are properties of the harness's resolution, not measurements of any vendor, and are labeled as such wherever they appear.
• Adapters reporting no distribution are excluded from multiclass Brier and recalibrate materially worse, because a single scalar carries less to correct with.
• Probabilities from a hosted API may arrive quantized. That bounds the resolution of any threshold or bin computed from them. METHODOLOGY says what the bound is and where it bites.
• Verified on Ubuntu, Windows, and macOS, Python 3.12 through 3.14, and on the oldest release of each dependency that pyproject allows.
• The generative transport has never run outside the test suite. The local arm has, against a pinned open checkpoint; see the adapters table above and issue #3.

Related work #

• JevBench and Benchmark Heaven. Cross-vendor ranking on a shared dataset, combining intelligence, calibration, speed and cost into one score. plumbline computes its own metrics against its own nulls on your data and publishes no combined score. Use theirs to shortlist, this to decide.
• SemIf (formerly OpenJev). Reads typed option probabilities straight out of an open model. Its README states the constraint plumbline's restricted_softmax class exists for: "Returned probabilities are conditional on the supplied options. Calibrate and validate them on the workload where they will make decisions." plumbline is a tool for doing exactly that. SemIf is an independent project and states it is "not affiliated with or endorsed by TypeSafe".
• fastjev, cited separately because it is a separate thing. An independently maintained fork of SemIf that preserves its history and MIT license, follows its own roadmap, and states plainly that it "does not serve Jev or reproduce Jev calibration".
• Mapika/decider. One-pass typed decisions with calibrated probabilities, fine-tuned from Qwen3.5-2B, in several sizes. Its own guidance is to check calibration on your own labels before routing on confidence. It serves the same typed question shape, so it is --base-url on typesafe_wire here rather than new code.
• Bespoke Nimble. An open recipe for typed decision models, a LoRA fine-tune on Qwen3.5-9B, trained with contrastive data curation. It serves the Jev wire format, so the unchanged typesafe_wire adapter runs it.

The pattern across the last three: they are models, and plumbline is the instrument you point at them. It has no opinion about which one you should pick.

Documentation #

• METHODOLOGY.md. How each number is computed, what it does not mean, and why each refusal is a refusal.
• docs/PLAN.md. Decisions already made, what a live run measured, and what v0.2 is for.
• docs/example-report.md. Real output.
• datasets/public/README.md. What the vendored fixture is, and why its numbers are not JevBench's numbers.
• TypeSafe documentation. The wire format the typesafe_wire adapter speaks.

Contributing #

New vendors are config entries, not new adapter modules. See CONTRIBUTING.md.

License #

Apache-2.0. See LICENSE.

The vendored JevBench fixture in datasets/public/ is MIT, Copyright (c) 2026 Florian Standhartinger and contributors, and is attributed in datasets/public/README.md with the full license text beside it in LICENSE-jevbench.