plumbline
Documentation

Rendered from datasets/public/README.md at commit fd90404, built 2026-09-25 23:24 UTC. Markdown source. Edit on GitHub.

Public datasets

A smoke-test fixture: something real to run on day one, and nothing more.

Your own data goes in datasets/private/, which is gitignored. That is the intended path for real use, and it is the only path on which plumbline's recalibration numbers mean anything.

jevbench-hard.jsonl #

The 111 public rows of the JevBench hard tier, vendored unchanged.

  • Source: https://github.com/fstandhartinger/jevbench, datasets/public/hard.jsonl
  • License: MIT, Copyright (c) 2026 Florian Standhartinger and contributors. The full license text is in LICENSE-jevbench beside the file.
  • JevBench is Benchmark Heaven's benchmark. It is not affiliated with plumbline, and plumbline is not affiliated with it.

These numbers are not JevBench's numbers #

plumbline is not reproducing this benchmark, and a figure produced here must never be compared with a figure JevBench publishes. The harness differs, the prompts differ, and the scoring differs:

  • JevBench asks each row through its own harness. plumbline composes a case text from the row's question.instructions above its state and sends that through whichever adapter is under test, with that adapter's own prompt shape.
  • JevBench's rows come in three question types: choice, noul, and score. plumbline asks each row as the type it declares, where the transport has one: a noul row is asked as a Noul by typesafe_wire, and as a two-option choice by a transport with no Noul. Those are different questions, so every record carries both what the row asks and how it was asked, and the report keeps them apart rather than averaging across the difference. The six score rows are read by rank, in a block apart from the choice figures, and their rubric is kept as the levels' descriptions; flattening ordered levels into unordered options would discard the ordering.
  • JevBench's score combines intelligence, calibration, speed, and cost into one number. plumbline computes its own metrics, against its own calibrated-null floor, and deliberately publishes no combined score.
  • Six rows write the gold label as a JSON number against string options; the loader matches 1 to "1" and reports how many rows needed it.
  • Thirty-eight rows carry criteria that describe the statement rather than each option, so their option descriptions are dropped rather than guessed at.

plumbline.datasets.load_jevbench states the same thing in its docstring, and every load report repeats it in its notes, so the caveat travels with the data rather than living only here.

Running it #

uv run python examples/smoke_public_dataset.py

Loads this file, runs the mock adapter over it, computes the metrics, and writes an artifact. It makes no network call and spends nothing. The numbers are meaningless (a seeded mock is answering), and the point is to prove the loader, the runner, the metrics, and the artifact compose on a real file before a live run turns a mistake into money.