Keel

Grounding report.

Every verification edge in simonw-llm, classified by who produces the signal and whether the actor being verified can write to that producer.

targetsimonw-llm
revision0392226e6630746ef51ffd309c2bee6a5f72b58e
generated2026-07-25T03:00:47.827Z
0.75

anchored / (anchored + self_referential + unknown) = 3 / 4

anchored 3 self_referential 1 unknown 0 not_a_check 21 excluded from the denominator

Coverage (judged)

node kindjudged
ci_step21
script2
test_target2

Nodes by kind, over the 25 node(s) carried in this report. Surfaces the gatherer cannot read are absent here rather than counted as unknown, which makes non-coverage Keel's own shoppable class — so it is stated, not implied.

ε-audit — not run

No verdict in this report carries a readable audit block, so nothing was re-decided and there is no agreement rate to report. This is not agreement, and it is not full agreement either: a rate over zero comparisons would state a measurement that was never taken. (“Readable” is load-bearing: a block can be present and state no comparison, and this state covers that too — see below if so.)

This run carries 14 probe-decided verdict(s), none of them contributing a readable comparison. (“Readable” rather than “audited”, because a block can arrive and state nothing — if any did, they are counted below.) Audit coverage is itself shoppable — audit none of the library and no disagreement can ever be found — so the absence is printed here rather than left to be inferred from a missing section.

This run judged 25 of 38 gathered edges. Every number on this page describes that sample, not the whole surface — 13 gathered edges are absent from all of it. A cap is disclosed rather than smoothed: an undisclosed sample is how a ratio gets shopped.

Scope. Keel measures the shape of verification, not its quality. A repo can be 100% anchored with terrible tests. Anchoring says the signal comes from outside; it does not say the signal is sufficient.

Node graph — 25 node(s), in gather order

anchored 3 self_referential 1 not_a_check 21

No mark is ringed: no drawn verdict is anchored below confidence 0.6. Hover a mark for its node.

anchored — 3

The producer sits outside the write boundary of the actor being verified.

Run tests
anchored
.github/workflows/publish.yml:27 · ci_step

The step invokes a language test runner directly, so the step's exit status IS the runtime's verdict on executing the committed test modules against the built artefact. A failing assertion, an uncaught exception, an import error or a crash cannot be persuaded to report green without changing the code or the tests. Documented limit, and it matters: the EXECUTION is anchored, the ORACLE is not — implementation and assertions are typically authored by the same people in the same repo, so a green run proves the code does what these tests say, not that the tests say the right thing. Whether the step blocks a merge additionally depends on the workflow's triggers, which this shape does not read.

producer the test runner process's exit code, produced by the language runtime executing the suite in the pipeline — actor can write to it: no

  • the step's run command is a direct invocation of a language test runner, with no install/provisioning command and no failure-suppressing flag

decided by probe (direct-test-runner-invocation) · confidence 0.85

Build
anchored
.github/workflows/publish.yml:47 · ci_step

`python -m build` asks the backend named in pyproject.toml to produce an sdist and a wheel from the committed source; it exits non-zero if the metadata is malformed, a declared dependency specifier is invalid, the package layout does not resolve, or the source does not import at build time. The verdict is computed by that program from the tree, so an author cannot make an unbuildable package report success without changing the tree — and this step runs in a `deploy` job whose `needs: [test]` means it only executes after the pytest matrix passed. Documented limit: a successful build proves the artefact is packageable and installable, not that it behaves correctly; it says nothing about runtime semantics.

producer the PEP 517 build backend (setuptools, invoked by `python -m build`) executing against the checked-out tree, and its process exit code — actor can write to it: no

  • .github/workflows/publish.yml:47-49
  • .github/workflows/publish.yml:35 (`needs: [test]`)
  • .github/workflows/test.yml:56-58 (the same command run as an explicit 'Check it builds' gate on every push)

decided by agent · confidence 0.82

Run tests
anchored
.github/workflows/test.yml:35 · ci_step

The step invokes a language test runner directly, so the step's exit status IS the runtime's verdict on executing the committed test modules against the built artefact. A failing assertion, an uncaught exception, an import error or a crash cannot be persuaded to report green without changing the code or the tests. Documented limit, and it matters: the EXECUTION is anchored, the ORACLE is not — implementation and assertions are typically authored by the same people in the same repo, so a green run proves the code does what these tests say, not that the tests say the right thing. Whether the step blocks a merge additionally depends on the workflow's triggers, which this shape does not read.

producer the test runner process's exit code, produced by the language runtime executing the suite in the pipeline — actor can write to it: no

  • the step's run command is a direct invocation of a language test runner, with no install/provisioning command and no failure-suppressing flag

decided by probe (direct-test-runner-invocation) · confidence 0.85

self_referential — 1

The producer sits inside the write boundary of the actor being verified.

Check if cog needs to be run
self_referential
.github/workflows/test.yml:38 · ci_step

`cog --check` recomputes the doc blocks by running the embedded generators and fails if the committed text differs — so the expected value is authored by the very code the docs describe, never by an independent specification. The write path is closed explicitly: .github/workflows/cog.yml runs `cog -r` on every `opened`/`synchronize` pull_request event and, if the regeneration changed anything, commits and pushes the result back to the PR branch (cog.yml:29-49). By the time this step reads the branch, the pipeline has already written the value it is about to compare against, so the comparison can only ever catch a regeneration that the bot did not get to run — the same shape as running `--update-snapshots` and then asserting the snapshot. The cog process's exit code is honestly produced by the interpreter; the oracle it compares to is not.

producer cog re-executing the Python snippets embedded in docs/*.md and diffing its own regenerated output against the committed markdown — actor can write to it: yes

  • .github/workflows/test.yml:38-43 (`cog --check ... docs/**/*.md docs/*.md`)
  • .github/workflows/cog.yml:29-31 (`cog -r` regenerates in-place)
  • .github/workflows/cog.yml:33-49 ('Check for changes' + 'Commit and push if changed' — the pipeline writes the expected value back to the branch)
  • .github/workflows/cog.yml:7-9 (`permissions: contents: write`)

decided by agent · confidence 0.7

not_a_check — 21

Asserts nothing about correctness, so it is excluded from the denominator. This is the one shoppable class: mis-filing a real check here shrinks the denominator and inflates the score, so each row carries the same burden of argument as any other verdict.

pytest config (pytest.ini)
not_a_check
pytest.ini · test_target

The whole file is `asyncio_default_fixture_loop_scope = function` — it selects an event-loop scope for async fixtures and declares no suite, no strictness flag, no failure threshold and no addopts. Nothing here can exit non-zero or block a merge; the edge that actually gates is `python -m pytest -vv` at .github/workflows/test.yml:35-37, gathered separately and judged on its own. Counting this file as a check as well would double-count that one edge.

producer an ini file read by pytest at startup to set one asyncio fixture default — actor can write to it: not established

  • pytest.ini:1-2
  • .github/workflows/test.yml:35-37 (the pytest invocation that actually gates)

decided by agent · confidence 0.78

pytest config (conftest.py)
not_a_check
tests/conftest.py · test_target

conftest.py contains only fixtures (`user_path`, `logs_db`, `collection`), mock model classes registered through pluggy, and canned httpx responses; the two `assert isinstance(messages, list)` lines are argument guards inside a test double's `enqueue`, not assertions about the product. It is loaded by, and cannot fail independently of, the `python -m pytest -vv` step at .github/workflows/test.yml:35-37 which is the edge that blocks the merge. Documented limit on THAT edge, recorded here because this file is what imposes it: every provider HTTP call in the suite is answered by hardcoded JSON written in this file (tests/conftest.py, `httpx_mock.add_response(... json={...})`), so the suite's oracle for third-party API behaviour is authored entirely inside the write boundary — execution is still honestly decided by the interpreter, the specification is not.

producer pytest fixture and plugin definitions loaded before any test runs — actor can write to it: not established

  • tests/conftest.py:1-502
  • tests/conftest.py:82 (assert inside MockModel.enqueue, a test double)
  • .github/workflows/test.yml:35-37

decided by agent · confidence 0.72

help
not_a_check
docs/Makefile:12 · script

The target body is `sphinx-build -M help` — it prints the list of available Sphinx builders. It reads no repository content for validity and produces no pass/fail claim; the only way it exits non-zero is if sphinx-build is not installed.

producer sphinx-build printing its own usage text to stdout — actor can write to it: not established

  • docs/Makefile:12-13

decided by agent · confidence 0.95

livehtml
not_a_check
docs/Makefile:22 · script

`sphinx-autobuild -b html` starts a watching dev server for a human to preview docs in a browser; it never terminates on its own, so it has no terminal exit code any gate could read, and no CI job invokes it (nothing under .github/workflows/ references docs/Makefile).

producer sphinx-autobuild, a long-running local preview server — actor can write to it: not established

  • docs/Makefile:22-23
  • .github/workflows/ (no reference to docs/Makefile or livehtml)

decided by agent · confidence 0.93

linter config (ruff.toml)
not_a_check
ruff.toml · ci_step

The file is a single line, `line-length = 160`. Configuration is consumed by a checker; it is not itself a producer of any signal and can never exit non-zero. The edge that produces a verdict is `ruff check .` at .github/workflows/test.yml:52-55 (outside this run's 25-node sample). Recorded limit on that edge: this file is inside the authors' write boundary, so the *scope* of what ruff enforces is author-controlled even though ruff's exit code is not.

producer a TOML table read by ruff to set one formatting parameter — actor can write to it: not established

  • ruff.toml:1
  • .github/workflows/test.yml:52-55 (`ruff check .` — the edge that emits a verdict)

decided by agent · confidence 0.75

actions/checkout@v6
not_a_check
.github/workflows/publish.yml:17 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Set up Python ${{ matrix.python-version }}
not_a_check
.github/workflows/publish.yml:18 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Install dependencies
not_a_check
.github/workflows/publish.yml:24 · ci_step

The command installs declared dependencies so that later steps can run. Without a lockfile-freezing flag the resolver will refresh a stale lock rather than error, so the step cannot report a discrepancy between the manifest and the lock; its non-zero exits mean the network or the index failed, not that the repository is wrong. The assertion in such a job lives in whatever executes against the installed environment afterwards.

producer a package manager's dependency resolver populating an environment — actor can write to it: not established

  • the run command is a bare dependency install with no --locked/--frozen equivalent

decided by probe (unfrozen-dependency-install) · confidence 0.8

actions/checkout@v6
not_a_check
.github/workflows/publish.yml:37 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Set up Python
not_a_check
.github/workflows/publish.yml:38 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Install dependencies
not_a_check
.github/workflows/publish.yml:44 · ci_step

The command installs declared dependencies so that later steps can run. Without a lockfile-freezing flag the resolver will refresh a stale lock rather than error, so the step cannot report a discrepancy between the manifest and the lock; its non-zero exits mean the network or the index failed, not that the repository is wrong. The assertion in such a job lives in whatever executes against the installed environment afterwards.

producer a package manager's dependency resolver populating an environment — actor can write to it: not established

  • the run command is a bare dependency install with no --locked/--frozen equivalent

decided by probe (unfrozen-dependency-install) · confidence 0.8

Publish
not_a_check
.github/workflows/publish.yml:50 · ci_step

The step's entire effect is promotion: it takes the dist/ files the previous step produced and uploads them to the index. It evaluates nothing about the artefact — a failure here means the upload was rejected (duplicate version, bad token, registry outage), not that the code is wrong. The verification condition on this promotion is the job's `needs: [test]` at publish.yml:35, which is a different edge; this one asserts nothing, so it must not enter the ratio in either direction. Noted separately: the action is referenced by the mutable branch ref `@release/v1` rather than a pinned SHA, which would matter if this were being counted as a check.

producer pypa/gh-action-pypi-publish uploading the built distributions to PyPI over trusted publishing — actor can write to it: not established

  • .github/workflows/publish.yml:50-51
  • .github/workflows/publish.yml:35 (`needs: [test]` — the actual promotion condition)
  • .github/workflows/publish.yml:47-49 (produces the artefact this step ships)

decided by agent · confidence 0.8

Checkout repository
not_a_check
.github/workflows/stable-docs.yml:17 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Set up Git user
not_a_check
.github/workflows/stable-docs.yml:21 · ci_step

`git config --global user.email/user.name` sets the identity later commits in this job will carry. It reads nothing from the repository and evaluates nothing about it; the only way it exits non-zero is if the runner's HOME is unwritable. The assertions in this workflow, such as they are, live in the generate and diff steps that follow.

producer git writing two keys into the runner's global config file — actor can write to it: not established

  • .github/workflows/update-model-settings.yml:44-48 (git config --global user.email / user.name)
  • .github/workflows/update-model-settings.yml:114-125 ('Create branch and push as signed commit' — the consumer of that identity)

decided by probe (runner-git-identity-config) · confidence 0.93

Create stable branch if it does not yet exist
not_a_check
.github/workflows/stable-docs.yml:25 · ci_step

The body probes whether an origin branch exists and, if not, creates it and seeds docs/ from the newest tag, then commits and pushes. Every operation mutates state; nothing compares the repository against an expectation, and the one operation that could fail on a no-op is neutralised with `|| echo "No changes"`. There is no assertion here to be grounded or ungrounded.

producer git creating a branch on the runner and pushing it to origin — actor can write to it: not established

  • .github/workflows/stable-docs.yml:25-37
  • .github/workflows/stable-docs.yml:35 (`git commit ... || echo "No changes"` — failure suppressed)

decided by agent · confidence 0.9

Handle Release
not_a_check
.github/workflows/stable-docs.yml:38 · ci_step

`git checkout stable && git reset --hard $TAG && git push --force` moves a documentation branch to point at the released commit. It reads nothing about correctness and cannot report anything about the code — it is a publish operation for the docs site, and its failure mode is a git or network error, not a defect signal.

producer git resetting the `stable` branch to a release tag and force-pushing it — actor can write to it: not established

  • .github/workflows/stable-docs.yml:38-44

decided by agent · confidence 0.92

Handle Commit to Main
not_a_check
.github/workflows/stable-docs.yml:45 · ci_step

`git diff-tree --name-only` is used here to ENUMERATE which docs/ files changed so they can be checked out onto the stable branch and committed — the diff is an input to a copy, not an assertion, and the branch where nothing changed ends in an explicit `exit 0`. The step gates nothing: it runs only when the commit message contains `!stable-docs`, and its whole output is a push to a docs branch.

producer git copying changed docs/ files onto the `stable` branch and pushing them — actor can write to it: not established

  • .github/workflows/stable-docs.yml:45-64
  • .github/workflows/stable-docs.yml:63 (`exit 0` on the no-change path)

decided by agent · confidence 0.88

actions/checkout@v6
not_a_check
.github/workflows/test.yml:21 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Set up Python ${{ matrix.python-version }}
not_a_check
.github/workflows/test.yml:22 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Install dependencies
not_a_check
.github/workflows/test.yml:28 · ci_step

The command installs declared dependencies so that later steps can run. Without a lockfile-freezing flag the resolver will refresh a stale lock rather than error, so the step cannot report a discrepancy between the manifest and the lock; its non-zero exits mean the network or the index failed, not that the repository is wrong. The assertion in such a job lives in whatever executes against the installed environment afterwards.

producer a package manager's dependency resolver populating an environment — actor can write to it: not established

  • the run command is a bare dependency install with no --locked/--frozen equivalent

decided by probe (unfrozen-dependency-install) · confidence 0.8

Install sqlite-utils pre-release
not_a_check
.github/workflows/test.yml:31 · ci_step

The command installs declared dependencies so that later steps can run. Without a lockfile-freezing flag the resolver will refresh a stale lock rather than error, so the step cannot report a discrepancy between the manifest and the lock; its non-zero exits mean the network or the index failed, not that the repository is wrong. The assertion in such a job lives in whatever executes against the installed environment afterwards.

producer a package manager's dependency resolver populating an environment — actor can write to it: not established

  • the run command is a bare dependency install with no --locked/--frozen equivalent

decided by probe (unfrozen-dependency-install) · confidence 0.8

Crystallization curve

Keel crystallization curve Crystallization curve over 15 runs from <repo>/reports. falls — total fitted change -66.5% of the mean across 15 runs (R^2 0.28, so the line explains less than half the variance — read the raw squares) CRYSTALLIZATION CURVE 15 sequential runs · 345 nodes judged of 1838 gathered · 124 anchored Keel corpus — 15 repositories, 2026-07-24 — measured corpus (declared on the command line) estimated tokens per node estimated tokens / judged node 0 300 600 run 0 · keel · estimated tokens per node = 498 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · estimated tokens per node = 369 · judged 25 of 41 gathered run 2 · openai-python · estimated tokens per node = 398 · judged 25 of 95 gathered run 3 · vercel-ai · estimated tokens per node = 410 · judged 25 of 1014 gathered run 4 · aider · estimated tokens per node = 204 · judged 25 of 62 gathered run 5 · browser-use · estimated tokens per node = 378 · judged 25 of 104 gathered run 6 · mcp-python-sdk · estimated tokens per node = 592 · judged 25 of 121 gathered run 7 · simonw-llm · estimated tokens per node = 189 · judged 25 of 38 gathered run 8 · tiktoken · estimated tokens per node = 240 run 9 · requests · estimated tokens per node = 290 · judged 25 of 106 gathered run 10 · flask · estimated tokens per node = 408 · judged 25 of 59 gathered run 11 · sinatra · estimated tokens per node = 265 · judged 25 of 59 gathered run 12 · commander-js · estimated tokens per node = 316 run 13 · anthropic-quickstarts · estimated tokens per node = 310 · judged 25 of 62 gathered run 14 · tiktoken · estimated tokens per node = 53 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 seconds per node measured s / judged node 0.0 60.0 120.0 run 0 · keel · seconds per node = 0.1 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · seconds per node = 15.8 · judged 25 of 41 gathered run 2 · openai-python · seconds per node = 15.4 · judged 25 of 95 gathered run 3 · vercel-ai · seconds per node = 105.9 · judged 25 of 1014 gathered run 4 · aider · seconds per node = 21.3 · judged 25 of 62 gathered run 5 · browser-use · seconds per node = 21.7 · judged 25 of 104 gathered run 6 · mcp-python-sdk · seconds per node = 20.1 · judged 25 of 121 gathered run 7 · simonw-llm · seconds per node = 11.8 · judged 25 of 38 gathered run 8 · tiktoken · seconds per node = 23.7 run 9 · requests · seconds per node = 16.8 · judged 25 of 106 gathered run 10 · flask · seconds per node = 20.5 · judged 25 of 59 gathered run 11 · sinatra · seconds per node = 16.9 · judged 25 of 59 gathered run 12 · commander-js · seconds per node = 18.7 run 13 · anthropic-quickstarts · seconds per node = 20.4 · judged 25 of 62 gathered run 14 · tiktoken · seconds per node = 0.1 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 probe-decided share measured share of decided nodes 0.00 0.50 1.00 run 0 · keel · probe-decided share = 0.00 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · probe-decided share = 0.00 · judged 25 of 41 gathered run 2 · openai-python · probe-decided share = 0.16 · judged 25 of 95 gathered run 3 · vercel-ai · probe-decided share = 0.08 · judged 25 of 1014 gathered run 4 · aider · probe-decided share = 0.56 · judged 25 of 62 gathered run 5 · browser-use · probe-decided share = 0.32 · judged 25 of 104 gathered run 6 · mcp-python-sdk · probe-decided share = 0.00 · judged 25 of 121 gathered run 7 · simonw-llm · probe-decided share = 0.56 · judged 25 of 38 gathered run 8 · tiktoken · probe-decided share = 0.58 run 9 · requests · probe-decided share = 0.28 · judged 25 of 106 gathered run 10 · flask · probe-decided share = 0.00 · judged 25 of 59 gathered run 11 · sinatra · probe-decided share = 0.28 · judged 25 of 59 gathered run 12 · commander-js · probe-decided share = 0.19 run 13 · anthropic-quickstarts · probe-decided share = 0.24 · judged 25 of 62 gathered run 14 · tiktoken · probe-decided share = 1.00 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 probe library size measured probes in library 0 15 30 run 0 · keel · probe library size = 0 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · probe library size = 3 · judged 25 of 41 gathered run 2 · openai-python · probe library size = 5 · judged 25 of 95 gathered run 3 · vercel-ai · probe library size = 8 · judged 25 of 1014 gathered run 4 · aider · probe library size = 10 · judged 25 of 62 gathered run 5 · browser-use · probe library size = 12 · judged 25 of 104 gathered run 6 · mcp-python-sdk · probe library size = 13 · judged 25 of 121 gathered run 7 · simonw-llm · probe library size = 15 · judged 25 of 38 gathered run 8 · tiktoken · probe library size = 17 run 9 · requests · probe library size = 19 · judged 25 of 106 gathered run 10 · flask · probe library size = 22 · judged 25 of 59 gathered run 11 · sinatra · probe library size = 23 · judged 25 of 59 gathered run 12 · commander-js · probe library size = 26 run 13 · anthropic-quickstarts · probe library size = 29 · judged 25 of 62 gathered run 14 · tiktoken · probe library size = 29 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 x axis: run index in the recorded corpus order (listed below). Squares are the raw per-run values; the dashed line is an ordinary-least-squares fit and is never shown without them. The fit is clipped to the panel, and withheld entirely (with the panel saying so) where a straight line would predict values the points cannot take — R^2 for every fit is in the trend notes below. * on run 0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13 = judged fewer nodes than gathered; per-node values are per JUDGED node. Coverage (judged) ci_step 224 · script 64 · test_target 46 · review_gate 10 · deploy_gate 1 Trend estimated tokens per node: falls — total fitted change -66.5% of the mean across 15 runs (R^2 0.28, so the line explains less than half the variance — read the raw squares). raw first 498 -> last 53. seconds per node: falls — total fitted change -72.9% of the mean across 15 runs (R^2 0.04, so the line explains less than half the variance — read the raw squares). raw first 0.1 -> last 0.1. probe-decided share: rises — total fitted change 144.9% of the mean across 15 runs (R^2 0.21, so the line explains less than half the variance — read the raw squares). raw first 0.00 -> last 1.00. probe library size: rises — total fitted change 187.7% of the mean across 15 runs (R^2 0.99). raw first 0 -> last 29. Run order run order derived from Report.generatedAt (ascending, filename as tiebreak) — no order.json manifest present 0 keel -> 1 anthropic-sdk-python -> 2 openai-python -> 3 vercel-ai -> 4 aider -> 5 browser-use -> 6 mcp-python-sdk -> 7 simonw-llm -> 8 tiktoken -> 9 requests -> 10 flask -> 11 sinatra -> 12 commander-js -> 13 anthropic-quickstarts -> 14 tiktoken Shuffle check shuffle check INCOMPLETE — the curve direction ran, the ratio direction did not Permutation (400 draws, seed 20260724): median |delta normalized slope| 0.633, 92% of draws move it by >= 0.15. A permutation reorders ALREADY-RECORDED runs; it cannot reproduce what a genuine re-run in a different order would have cost, because the probe library would have accumulated differently. It is a sanity signal on order-dependence, never a substitute for the empirical re-run. Ratio stability under shuffle was NOT checked: no re-run supplied (--shuffled) and none declared in corpus.meta.json. Permuting recorded runs cannot move a per-target ratio, so the permutation result below says nothing about it. Supply a re-run with --shuffled <dir>. Disclosures - Skipped 3 file(s) that are not usable run reports: corpus-summary.json (no string "target"); curve.json (no string "target"); keel.bindings.json (no "nodes" array). - Run 0 (keel) judged 25 of 32 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 1 (anthropic-sdk-python) judged 25 of 41 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 2 (openai-python) judged 25 of 95 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 3 (vercel-ai) judged 25 of 1014 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 4 (aider) judged 25 of 62 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 5 (browser-use) judged 25 of 104 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 6 (mcp-python-sdk) judged 25 of 121 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 7 (simonw-llm) judged 25 of 38 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 9 (requests) judged 25 of 106 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 10 (flask) judged 25 of 59 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 11 (sinatra) judged 25 of 59 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 13 (anthropic-quickstarts) judged 25 of 62 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 14 (tiktoken): decidedByProbe + decidedByAgent = 10 but nodesSampled = 12. Probe-decided share uses the decided total (10). - Provenance "measured" was declared on the COMMAND LINE (--provenance), not in <repo>/reports/corpus.meta.json. The declaration therefore lives in the invocation and is only as trustworthy as the run sheet that records it. - Token counts are ESTIMATES in 15 of 15 run(s) (RunEconomics.tokensEstimated). No API exposes session token usage to a skill, so the token axis reads "estimated tokens" and nothing here claims a measured token count. Scope. Keel measures the shape of verification, not its quality. A repo can be 100% anchored with terrible tests. Anchoring says the signal comes from outside; it does not say the signal is sufficient.

Run economics

14 decided by probe
11 decided by agent
56% probe-decided share
25 / 38 nodes judged / gathered
2,088 estimated tokens in
2,644 estimated tokens out
4m 54s wall clock
2 probes minted
15 probe library size

Token counts are estimated, and labelled so: a skill running inside an agent session has no API for its own usage, so the figure is ceil(chars/4) over the judgment payloads and responses. Wall clock and probe-decided share are measured directly. 25 of 38 gathered edges were judged — the cap is printed here and beside the ratio, never applied silently.