Keel

Grounding report.

Every verification edge in openai-python, classified by who produces the signal and whether the actor being verified can write to that producer.

targetopenai-python
revisiond4c151d92ba713e56518a4483d3b7918dd1be838
generated2026-07-25T01:42:13.604Z
0.77

anchored / (anchored + self_referential + unknown) = 10 / 13

anchored 10 self_referential 2 unknown 1 not_a_check 12 excluded from the denominator

Coverage (judged)

node kindjudged
ci_step22
test_target2
review_gate1

Nodes by kind, over the 25 node(s) carried in this report. Surfaces the gatherer cannot read are absent here rather than counted as unknown, which makes non-coverage Keel's own shoppable class — so it is stated, not implied.

ε-audit — not run

No verdict in this report carries a readable audit block, so nothing was re-decided and there is no agreement rate to report. This is not agreement, and it is not full agreement either: a rate over zero comparisons would state a measurement that was never taken. (“Readable” is load-bearing: a block can be present and state no comparison, and this state covers that too — see below if so.)

This run carries 4 probe-decided verdict(s), none of them contributing a readable comparison. (“Readable” rather than “audited”, because a block can arrive and state nothing — if any did, they are counted below.) Audit coverage is itself shoppable — audit none of the library and no disagreement can ever be found — so the absence is printed here rather than left to be inferred from a missing section.

This run judged 25 of 95 gathered edges. Every number on this page describes that sample, not the whole surface — 70 gathered edges are absent from all of it. A cap is disclosed rather than smoothed: an undisclosed sample is how a ratio gets shopped.

Scope. Keel measures the shape of verification, not its quality. A repo can be 100% anchored with terrible tests. Anchoring says the signal comes from outside; it does not say the signal is sufficient.

Node graph — 25 node(s), in gather order

anchored 10 self_referential 2 unknown 1 not_a_check 12

3 marks are ringed in the unknown hue: an anchored verdict asserted below confidence 0.6. The ring says the claim is closer to unknown than its colour suggests. Hover a mark for its node.

anchored — 10

The producer sits outside the write boundary of the actor being verified.

pytest config ([tool.pytest.ini_options])
anchored
pyproject.toml:155 · test_target

ci.yml:144/175/180 runs ./scripts/test, which ends in `rye run pytest` (scripts/test:62) under `set -e`, so a failing assertion or an unraised-warning (filterwarnings=["error"]) or a passing xfail (xfail_strict=true) makes the interpreter exit non-zero and the job red — an author cannot make a failing test report green without changing the committed source or this pinned config. Documented limit on the ORACLE axis: the assertions in tests/ are Stainless-codegenned from the same OpenAPI spec that generates src/openai, and scripts/test:33 boots a local mock server built from that spec, so the expected behaviour was authored inside the write boundary; only the execution is independent.

producer the pytest process exit code, executed as `rye run pytest` by ./scripts/test in the ci.yml `test` and `test-httpx2` jobs — actor can write to it: no

  • pyproject.toml:155
  • .github/workflows/ci.yml:143-144
  • scripts/test:62
  • scripts/test:28-34

decided by agent · confidence 0.8

linter/type-checker config ([tool.pyright])
anchored
pyproject.toml:168 · ci_step

This table is the pinned configuration pyright reads when it statically analyses the committed sources in the lint job; typeCheckingMode="strict" makes an unannotated or ill-typed symbol produce a non-zero exit that fails the job under `set -e` (scripts/lint:2). The verdict is computed by the type checker over the source tree, not asserted by anyone. Limit worth naming: the `exclude` list and the `report*=false` flags narrow what is analysed, and both are author-editable in-repo — so the scope of the check is inside the boundary even though the analysis is not.

producer the pyright process exit code, reached via ci.yml `Run lints` -> ./scripts/lint -> `rye run lint` -> `typecheck:pyright` -> scripts/run-pyright -> `pyright -p pyproject.toml` — actor can write to it: no

  • pyproject.toml:168
  • .github/workflows/ci.yml:40-41
  • scripts/lint:12
  • scripts/run-pyright:8
  • pyproject.toml:110-114

decided by agent · confidence 0.85

linter/type-checker config ([tool.mypy])
anchored
pyproject.toml:193 · ci_step

ci.yml:41 runs ./scripts/lint, whose `rye run lint` chain includes `typecheck` -> `typecheck:mypy` = `mypy .` (pyproject.toml:115); mypy's static analysis of the committed tree decides the exit code and `set -e` propagates it to the job. disallow_untyped_defs/warn_return_any etc. are enforced by the checker, not by a declaration. Same scope limit as pyright: `exclude` drops src/openai/_files.py and all of tests/ from the analysis, so what is inside the check is author-selected while the analysis itself is not.

producer the mypy process exit code from `mypy .` (rye script `typecheck:mypy`), run in the ci.yml lint job — actor can write to it: no

  • pyproject.toml:193
  • pyproject.toml:110-115
  • .github/workflows/ci.yml:40-41
  • scripts/lint:12

decided by agent · confidence 0.85

linter/type-checker config ([tool.ruff])
anchored
pyproject.toml:254 · ci_step

Of the keys here, only target-version="py310" reaches the gating signal: it determines which Python version ruff assumes when deciding rules such as FA102 (missing future annotations), which IS selected, so it changes whether `ruff check .` exits non-zero. line-length=120 does not gate anything because E501 is not in the select list (pyproject.toml:262-292) and therefore only feeds the rewrite-only formatter, and output-format="grouped" is presentation. Low confidence is deliberate: this is a thin slice of an anchored edge, not an edge of its own.

producer the ruff process exit code from `ruff check .` (rye script `check:ruff`) in the ci.yml lint job — but only via one of this table's three keys — actor can write to it: no

  • pyproject.toml:254
  • pyproject.toml:104
  • pyproject.toml:262-292
  • .github/workflows/ci.yml:40-41

decided by agent · confidence 0.5 · confidence below 0.6 on an anchored verdict

linter/type-checker config ([tool.ruff.lint])
anchored
pyproject.toml:262 · ci_step

This select list is what `ruff check .` enforces: a bare `except:` (E722), an unused import (F401), a stray `print` (T201, additionally listed as unfixable so it cannot be auto-silenced), or an unsorted import (I) makes ruff exit non-zero, `set -e` in scripts/lint aborts, and the lint job fails. The signal is produced by the linter binary reading the committed source; an author cannot make a violating file report clean without editing the file or this pinned rule set in a reviewed commit.

producer the ruff process exit code from `ruff check .` (rye script `check:ruff`), run by ./scripts/lint in the ci.yml lint job — actor can write to it: no

  • pyproject.toml:262
  • pyproject.toml:99-104
  • scripts/lint:2
  • scripts/lint:12
  • .github/workflows/ci.yml:40-41

decided by agent · confidence 0.85

linter/type-checker config ([tool.ruff.lint.flake8-tidy-imports.banned-api])
anchored
pyproject.toml:296 · ci_step

TID251 is in the select list (pyproject.toml:262-292), so this entry makes any `functools.lru_cache` import in the tree a hard ruff diagnostic and a non-zero exit that fails the lint job. The ban is decided by the linter scanning imports in committed source, not by anyone asserting the rule is respected.

producer the ruff process exit code — rule TID251 evaluated by `ruff check .` in the ci.yml lint job — actor can write to it: no

  • pyproject.toml:296
  • pyproject.toml:262-292
  • .github/workflows/ci.yml:40-41
  • scripts/lint:12

decided by agent · confidence 0.85

linter/type-checker config ([tool.ruff.lint.isort])
anchored
pyproject.toml:299 · ci_step

`I` (isort) is selected at pyproject.toml:264-265, so these ordering settings define a machine-decidable property of every source file that `ruff check .` verifies; a wrongly-ordered import block exits non-zero and reddens the lint job. Note the limit on what is being asserted: this constrains import layout only — it is a style property produced by executing a checker, not a correctness property.

producer the ruff process exit code — rule family `I` evaluated by `ruff check .` in the ci.yml lint job — actor can write to it: no

  • pyproject.toml:299
  • pyproject.toml:263-265
  • .github/workflows/ci.yml:40-41

decided by agent · confidence 0.75

linter/type-checker config ([tool.ruff.lint.per-file-ignores])
anchored
pyproject.toml:306 · ci_step

The producer is still the linter binary executing over the tree in the ci.yml lint job, and this table participates in that exit code by exempting bin/, scripts/, tests/ and examples/ from T201/T203. Confidence is deliberately low and the direction of the effect is worth stating plainly: an exemption table can only WEAKEN the gate, never make it fail, so it inherits anchoring from the check it configures while contributing no assertion of its own — a smell, surfaced rather than smoothed away.

producer the ruff process exit code from `ruff check .`; this table only narrows the file set the rules apply to — actor can write to it: no

  • pyproject.toml:306
  • pyproject.toml:270-275
  • .github/workflows/ci.yml:40-41

decided by agent · confidence 0.5 · confidence below 0.6 on an anchored verdict

pytest config (conftest.py)
anchored
tests/conftest.py · test_target

The fixtures here construct real OpenAI/AsyncOpenAI clients and hand them to tests whose pass/fail the interpreter decides during the CI pytest run, so the execution axis is outside the author's write path. Two limits belong on the record. First, base_url defaults to http://127.0.0.1:4010 — a mock Steady server that scripts/test:33 boots from the repo's own OpenAPI spec, the same spec the Stainless codegen used to write src/openai, so the responses the tests observe are a replica authored inside the boundary. Second, pytest_collection_modifyitems adds skip markers for aiohttp+respx combinations, i.e. the harness removes cases from the denominator; a skipped test produces no signal at all.

producer the pytest process exit code — conftest.py is the fixture harness for the suite ./scripts/test runs in the ci.yml `test` / `test-httpx2` jobs — actor can write to it: no

  • tests/conftest.py:1
  • tests/conftest.py:27-47
  • tests/conftest.py:53-54
  • scripts/test:28-34
  • .github/workflows/ci.yml:143-144

decided by agent · confidence 0.55 · confidence below 0.6 on an anchored verdict

Snapshot official lifecycle and public usage data
anchored
.github/workflows/python-version-review.yml:58 · ci_step

curl --fail turns any non-2xx from the two third-party endpoints into a non-zero exit, and jq --exit-status then evaluates the fetched payload's shape ('length > 5 and all(.[]; has("status") and has("end_of_life"))', and 'has("data")' for the download stats) — failing the step when the remote data is missing or malformed. Both signals originate at services outside the repository: the openai-python maintainers cannot write the Python release-cycle API or PyPI download statistics, so this step's outcome is decided by external reality. Limit: it validates the SHAPE of the fetched data, not that the numbers are right.

producer HTTP responses from peps.python.org and pypistats.org, surfaced as exit codes by `curl --fail` and then `jq --exit-status` — actor can write to it: no

  • .github/workflows/python-version-review.yml:58-92
  • .github/workflows/python-version-review.yml:72-74
  • .github/workflows/python-version-review.yml:88-90

decided by agent · confidence 0.8

self_referential — 2

The producer sits inside the write boundary of the actor being verified.

Ask Codex to assess the policy
self_referential
.github/workflows/python-version-review.yml:104 · ci_step

The step hands the repository to a language model and takes back prose, a review, or commits. Whatever verdict it renders is produced by the same class of system that produces the code being judged, so the two share correlated blind spots and agreement between them is not evidence. In the common configuration the model is also granted repository write tools, which means the reviewer can edit the very state it is reporting on.

producer a large language model invoked by a CI step — actor can write to it: yes

  • the step's `uses:` reference resolves to an LLM coding/review agent action

decided by probe (llm-agent-ci-step) · confidence 0.75

Validate and package the assessment
self_referential
.github/workflows/python-version-review.yml:125 · ci_step

The consequential output of this step — action_required, which is the sole condition gating the open_issue job at line 174 — is `head -n 1 $CODEX_OUTPUT_FILE` matched against two marker strings, i.e. the LLM decides whether its own finding is actionable by emitting the marker that says so; the envelope checks (non-empty, <60000 bytes, valid marker) constrain the shape of that artifact, not its truth. One genuinely independent sub-assertion is present and should be credited: `git diff --quiet HEAD` plus the untracked-file check is computed by git over a .git directory the codex user was explicitly denied write access to (line 101-102), so the agent cannot conceal a modification it made. That anchored fragment does not change the character of the step, whose product is an LLM's self-labelled verdict.

producer a markdown file written by the Codex LLM in the immediately preceding step (openai/codex-action, line 104); this step reads its first line and its byte count — actor can write to it: yes

  • .github/workflows/python-version-review.yml:125-160
  • .github/workflows/python-version-review.yml:144-158
  • .github/workflows/python-version-review.yml:104-116
  • .github/workflows/python-version-review.yml:171-174
  • .github/workflows/python-version-review.yml:100-102

decided by agent · confidence 0.75

unknown — 1

The fork point could not be established. Fails closed — counts against the ratio exactly like self_referential.

code owners (CODEOWNERS)
unknown
.github/CODEOWNERS · review_gate

The file's own header says it is 'used to automatically assign reviewers to PRs'. Assignment is routing, not a gate; the gate would be a 'Require review from Code Owners' branch-protection rule, and no such configuration is present anywhere in the clone (no rulesets, no settings-as-code) and it is not pinned to a revision I can read. I therefore cannot establish whether this edge blocks a merge at all, nor — since @openai/sdks-team both authors and reviews the codegenned changes — where the reviewer sits relative to the author's write boundary. Fails closed.

producer GitHub's pull-request review machinery — but whether an approving review is REQUIRED is decided by branch-protection / ruleset configuration that lives in GitHub's settings, not in this repository — actor can write to it: not established

  • .github/CODEOWNERS:1-4
  • no branch-protection or ruleset config in the clone

decided by agent · confidence 0.7

not_a_check — 12

Asserts nothing about correctness, so it is excluded from the denominator. This is the one shoppable class: mis-filing a real check here shrinks the denominator and inflates the score, so each row carries the same burden of argument as any other verdict.

linter/type-checker config ([tool.ruff.format])
not_a_check
pyproject.toml:259 · ci_step

docstring-code-format only changes how `ruff format` REWRITES docstring code blocks; it produces no pass/fail signal. Its only caller is the rye `format` chain via scripts/format, and `grep -rn 'format\|ruff' .github/workflows/` returns nothing — no CI job runs the formatter or a `--check`/`--diff` variant of it, so there is no exit code anywhere in the merge path that this table can influence.

producer `ruff format`, which rewrites files in place and is never invoked by any workflow in .github/workflows/ — actor can write to it: not established

  • pyproject.toml:259
  • pyproject.toml:89-97
  • scripts/format:8
  • .github/workflows/ (no ruff format invocation)

decided by agent · confidence 0.8

Create an unprivileged Codex user
not_a_check
.github/workflows/python-version-review.yml:32 · ci_step

The step creates a system user and group so the Codex agent runs unprivileged. It asserts nothing about openai-python's correctness; its only failure mode is the runner being unable to create an account. Real safety value (credential isolation), zero verification content — it is setup for a later step, not a check of the artifact.

producer adduser/usermod exit codes — machine provisioning on the runner — actor can write to it: not established

  • .github/workflows/python-version-review.yml:32-42

decided by agent · confidence 0.85

Check out the trusted default branch
not_a_check
.github/workflows/python-version-review.yml:44 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Prepare the isolated Codex workspace
not_a_check
.github/workflows/python-version-review.yml:50 · ci_step

Pure provisioning: creates the agent home directory, copies a config file into it, and appends a path to .git/info/exclude. No property of the repository is evaluated and no comparison is made, so there is no signal here that could come out either way on a correct-vs-incorrect codebase.

producer mkdir/cp/echo exit codes — actor can write to it: not established

  • .github/workflows/python-version-review.yml:50-56

decided by agent · confidence 0.9

Grant the Codex user controlled workspace access
not_a_check
.github/workflows/python-version-review.yml:94 · ci_step

Filesystem permission provisioning that sets up the agent sandbox (workspace group-readable, .git non-writable). It creates the conditions under which a LATER step's git-cleanliness assertion is meaningful, but on its own it evaluates no property of the code and cannot report a defect.

producer chown/chmod exit codes — actor can write to it: not established

  • .github/workflows/python-version-review.yml:94-102

decided by agent · confidence 0.85

Terminate Codex-owned processes
not_a_check
.github/workflows/python-version-review.yml:118 · ci_step

All three commands end in `|| true` and the step carries `if: always()`, so its exit status is 0 unconditionally. A signal that cannot vary carries no information — this is teardown, and structurally incapable of being a check regardless of the state of the repository.

producer pkill/chown exit codes, every one of them swallowed — actor can write to it: not established

  • .github/workflows/python-version-review.yml:118-123

decided by agent · confidence 0.95

Upload the action-required assessment
not_a_check
.github/workflows/python-version-review.yml:162 · ci_step

Artifact transport between the assess and open_issue jobs. `if-no-files-found: error` gives it a failure mode, but the only thing that failure would report is that the preceding step in the same job did not leave the file it had just declared present — an operational guard on the pipeline's own plumbing, asserting nothing about openai-python's behaviour or types. Filed here rather than in the ratio because the class of thing it could ever detect is 'the upload path is broken', not 'the code is wrong'.

producer actions/upload-artifact's own exit code — actor can write to it: not established

  • .github/workflows/python-version-review.yml:162-169
  • .github/workflows/python-version-review.yml:146-149

decided by agent · confidence 0.6

Download the Codex assessment
not_a_check
.github/workflows/python-version-review.yml:181 · ci_step

Provisioning for the publish step: it fetches the artifact the previous job uploaded into runner.temp. No property of the repository or of the assessment is evaluated — it is the inbound half of the same transport.

producer actions/download-artifact's own exit code — actor can write to it: not established

  • .github/workflows/python-version-review.yml:181-185

decided by agent · confidence 0.9

Publish the review issue
not_a_check
.github/workflows/python-version-review.yml:187 · ci_step

A side-effecting publication step: it creates a label and opens or refreshes an issue whose body is the LLM assessment verbatim. Nothing is compared and nothing can be found wanting — the step's own text calls the result 'advisory', and changing the Python floor is explicitly deferred to a separate reviewed pull request. Its failure modes are GitHub API errors, not findings.

producer the GitHub Issues API via `gh issue create/edit` — actor can write to it: not established

  • .github/workflows/python-version-review.yml:187-233
  • .github/workflows/python-version-review.yml:218

decided by agent · confidence 0.8

Calculate fetch-depth
not_a_check
.github/workflows/detect-breaking-changes.yml:14 · ci_step

Computes a clone depth from the PR's commit count so the later checkout can reach the base SHA. It sets a variable; it evaluates no property of the diff and has no failing branch that would correspond to a defect in the code.

producer `expr` arithmetic written into $GITHUB_ENV — actor can write to it: not established

  • .github/workflows/detect-breaking-changes.yml:14-21

decided by agent · confidence 0.9

actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
not_a_check
.github/workflows/detect-breaking-changes.yml:18 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Set up Rye
not_a_check
.github/workflows/detect-breaking-changes.yml:23 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Crystallization curve

Keel crystallization curve Crystallization curve over 15 runs from <repo>/reports. falls — total fitted change -66.5% of the mean across 15 runs (R^2 0.28, so the line explains less than half the variance — read the raw squares) CRYSTALLIZATION CURVE 15 sequential runs · 345 nodes judged of 1838 gathered · 124 anchored Keel corpus — 15 repositories, 2026-07-24 — measured corpus (declared on the command line) estimated tokens per node estimated tokens / judged node 0 300 600 run 0 · keel · estimated tokens per node = 498 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · estimated tokens per node = 369 · judged 25 of 41 gathered run 2 · openai-python · estimated tokens per node = 398 · judged 25 of 95 gathered run 3 · vercel-ai · estimated tokens per node = 410 · judged 25 of 1014 gathered run 4 · aider · estimated tokens per node = 204 · judged 25 of 62 gathered run 5 · browser-use · estimated tokens per node = 378 · judged 25 of 104 gathered run 6 · mcp-python-sdk · estimated tokens per node = 592 · judged 25 of 121 gathered run 7 · simonw-llm · estimated tokens per node = 189 · judged 25 of 38 gathered run 8 · tiktoken · estimated tokens per node = 240 run 9 · requests · estimated tokens per node = 290 · judged 25 of 106 gathered run 10 · flask · estimated tokens per node = 408 · judged 25 of 59 gathered run 11 · sinatra · estimated tokens per node = 265 · judged 25 of 59 gathered run 12 · commander-js · estimated tokens per node = 316 run 13 · anthropic-quickstarts · estimated tokens per node = 310 · judged 25 of 62 gathered run 14 · tiktoken · estimated tokens per node = 53 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 seconds per node measured s / judged node 0.0 60.0 120.0 run 0 · keel · seconds per node = 0.1 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · seconds per node = 15.8 · judged 25 of 41 gathered run 2 · openai-python · seconds per node = 15.4 · judged 25 of 95 gathered run 3 · vercel-ai · seconds per node = 105.9 · judged 25 of 1014 gathered run 4 · aider · seconds per node = 21.3 · judged 25 of 62 gathered run 5 · browser-use · seconds per node = 21.7 · judged 25 of 104 gathered run 6 · mcp-python-sdk · seconds per node = 20.1 · judged 25 of 121 gathered run 7 · simonw-llm · seconds per node = 11.8 · judged 25 of 38 gathered run 8 · tiktoken · seconds per node = 23.7 run 9 · requests · seconds per node = 16.8 · judged 25 of 106 gathered run 10 · flask · seconds per node = 20.5 · judged 25 of 59 gathered run 11 · sinatra · seconds per node = 16.9 · judged 25 of 59 gathered run 12 · commander-js · seconds per node = 18.7 run 13 · anthropic-quickstarts · seconds per node = 20.4 · judged 25 of 62 gathered run 14 · tiktoken · seconds per node = 0.1 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 probe-decided share measured share of decided nodes 0.00 0.50 1.00 run 0 · keel · probe-decided share = 0.00 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · probe-decided share = 0.00 · judged 25 of 41 gathered run 2 · openai-python · probe-decided share = 0.16 · judged 25 of 95 gathered run 3 · vercel-ai · probe-decided share = 0.08 · judged 25 of 1014 gathered run 4 · aider · probe-decided share = 0.56 · judged 25 of 62 gathered run 5 · browser-use · probe-decided share = 0.32 · judged 25 of 104 gathered run 6 · mcp-python-sdk · probe-decided share = 0.00 · judged 25 of 121 gathered run 7 · simonw-llm · probe-decided share = 0.56 · judged 25 of 38 gathered run 8 · tiktoken · probe-decided share = 0.58 run 9 · requests · probe-decided share = 0.28 · judged 25 of 106 gathered run 10 · flask · probe-decided share = 0.00 · judged 25 of 59 gathered run 11 · sinatra · probe-decided share = 0.28 · judged 25 of 59 gathered run 12 · commander-js · probe-decided share = 0.19 run 13 · anthropic-quickstarts · probe-decided share = 0.24 · judged 25 of 62 gathered run 14 · tiktoken · probe-decided share = 1.00 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 probe library size measured probes in library 0 15 30 run 0 · keel · probe library size = 0 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · probe library size = 3 · judged 25 of 41 gathered run 2 · openai-python · probe library size = 5 · judged 25 of 95 gathered run 3 · vercel-ai · probe library size = 8 · judged 25 of 1014 gathered run 4 · aider · probe library size = 10 · judged 25 of 62 gathered run 5 · browser-use · probe library size = 12 · judged 25 of 104 gathered run 6 · mcp-python-sdk · probe library size = 13 · judged 25 of 121 gathered run 7 · simonw-llm · probe library size = 15 · judged 25 of 38 gathered run 8 · tiktoken · probe library size = 17 run 9 · requests · probe library size = 19 · judged 25 of 106 gathered run 10 · flask · probe library size = 22 · judged 25 of 59 gathered run 11 · sinatra · probe library size = 23 · judged 25 of 59 gathered run 12 · commander-js · probe library size = 26 run 13 · anthropic-quickstarts · probe library size = 29 · judged 25 of 62 gathered run 14 · tiktoken · probe library size = 29 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 x axis: run index in the recorded corpus order (listed below). Squares are the raw per-run values; the dashed line is an ordinary-least-squares fit and is never shown without them. The fit is clipped to the panel, and withheld entirely (with the panel saying so) where a straight line would predict values the points cannot take — R^2 for every fit is in the trend notes below. * on run 0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13 = judged fewer nodes than gathered; per-node values are per JUDGED node. Coverage (judged) ci_step 224 · script 64 · test_target 46 · review_gate 10 · deploy_gate 1 Trend estimated tokens per node: falls — total fitted change -66.5% of the mean across 15 runs (R^2 0.28, so the line explains less than half the variance — read the raw squares). raw first 498 -> last 53. seconds per node: falls — total fitted change -72.9% of the mean across 15 runs (R^2 0.04, so the line explains less than half the variance — read the raw squares). raw first 0.1 -> last 0.1. probe-decided share: rises — total fitted change 144.9% of the mean across 15 runs (R^2 0.21, so the line explains less than half the variance — read the raw squares). raw first 0.00 -> last 1.00. probe library size: rises — total fitted change 187.7% of the mean across 15 runs (R^2 0.99). raw first 0 -> last 29. Run order run order derived from Report.generatedAt (ascending, filename as tiebreak) — no order.json manifest present 0 keel -> 1 anthropic-sdk-python -> 2 openai-python -> 3 vercel-ai -> 4 aider -> 5 browser-use -> 6 mcp-python-sdk -> 7 simonw-llm -> 8 tiktoken -> 9 requests -> 10 flask -> 11 sinatra -> 12 commander-js -> 13 anthropic-quickstarts -> 14 tiktoken Shuffle check shuffle check INCOMPLETE — the curve direction ran, the ratio direction did not Permutation (400 draws, seed 20260724): median |delta normalized slope| 0.633, 92% of draws move it by >= 0.15. A permutation reorders ALREADY-RECORDED runs; it cannot reproduce what a genuine re-run in a different order would have cost, because the probe library would have accumulated differently. It is a sanity signal on order-dependence, never a substitute for the empirical re-run. Ratio stability under shuffle was NOT checked: no re-run supplied (--shuffled) and none declared in corpus.meta.json. Permuting recorded runs cannot move a per-target ratio, so the permutation result below says nothing about it. Supply a re-run with --shuffled <dir>. Disclosures - Skipped 3 file(s) that are not usable run reports: corpus-summary.json (no string "target"); curve.json (no string "target"); keel.bindings.json (no "nodes" array). - Run 0 (keel) judged 25 of 32 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 1 (anthropic-sdk-python) judged 25 of 41 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 2 (openai-python) judged 25 of 95 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 3 (vercel-ai) judged 25 of 1014 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 4 (aider) judged 25 of 62 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 5 (browser-use) judged 25 of 104 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 6 (mcp-python-sdk) judged 25 of 121 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 7 (simonw-llm) judged 25 of 38 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 9 (requests) judged 25 of 106 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 10 (flask) judged 25 of 59 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 11 (sinatra) judged 25 of 59 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 13 (anthropic-quickstarts) judged 25 of 62 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 14 (tiktoken): decidedByProbe + decidedByAgent = 10 but nodesSampled = 12. Probe-decided share uses the decided total (10). - Provenance "measured" was declared on the COMMAND LINE (--provenance), not in <repo>/reports/corpus.meta.json. The declaration therefore lives in the invocation and is only as trustworthy as the run sheet that records it. - Token counts are ESTIMATES in 15 of 15 run(s) (RunEconomics.tokensEstimated). No API exposes session token usage to a skill, so the token axis reads "estimated tokens" and nothing here claims a measured token count. Scope. Keel measures the shape of verification, not its quality. A repo can be 100% anchored with terrible tests. Anchoring says the signal comes from outside; it does not say the signal is sufficient.

Run economics

4 decided by probe
21 decided by agent
16% probe-decided share
25 / 95 nodes judged / gathered
5,036 estimated tokens in
4,913 estimated tokens out
6m 25s wall clock
2 probes minted
5 probe library size

Token counts are estimated, and labelled so: a skill running inside an agent session has no API for its own usage, so the figure is ceil(chars/4) over the judgment payloads and responses. Wall clock and probe-decided share are measured directly. 25 of 95 gathered edges were judged — the cap is printed here and beside the ratio, never applied silently.