Keel

Grounding report.

Every verification edge in tiktoken, classified by who produces the signal and whether the actor being verified can write to that producer.

targettiktoken
revision08a5f3b2c987ada4fc5aa1f16c643c203fa8acaa
generated2026-07-25T03:06:06.073Z
1.00

anchored / (anchored + self_referential + unknown) = 4 / 4

anchored 4 self_referential 0 unknown 0 not_a_check 8 excluded from the denominator

Coverage (judged)

node kindjudged
ci_step12

Nodes by kind, over the 12 node(s) carried in this report. Surfaces the gatherer cannot read are absent here rather than counted as unknown, which makes non-coverage Keel's own shoppable class — so it is stated, not implied.

ε-audit — not run

No verdict in this report carries a readable audit block, so nothing was re-decided and there is no agreement rate to report. This is not agreement, and it is not full agreement either: a rate over zero comparisons would state a measurement that was never taken. (“Readable” is load-bearing: a block can be present and state no comparison, and this state covers that too — see below if so.)

This run carries 7 probe-decided verdict(s), none of them contributing a readable comparison. (“Readable” rather than “audited”, because a block can arrive and state nothing — if any did, they are counted below.) Audit coverage is itself shoppable — audit none of the library and no disagreement can ever be found — so the absence is printed here rather than left to be inferred from a missing section.

Scope. Keel measures the shape of verification, not its quality. A repo can be 100% anchored with terrible tests. Anchoring says the signal comes from outside; it does not say the signal is sufficient.

Node graph — 12 node(s), in gather order

anchored 4 not_a_check 8

No mark is ringed: no drawn verdict is anchored below confidence 0.6. Hover a mark for its node.

anchored — 4

The producer sits outside the write boundary of the actor being verified.

pypa/cibuildwheel@65b8265957fd86372d9689a0acdfd55813970d5d # v3.1.4
anchored
.github/workflows/build_wheels.yml:25 · ci_step

The step is not a bare packaging action: pyproject.toml's [tool.cibuildwheel] table sets build-frontend="build" and test-command="pytest {project}/tests --import-mode=append" with before-test="pip install pytest hypothesis", so for each of the 24 matrix cells cibuildwheel compiles src/*.rs into tiktoken._tiktoken (setup.py declares a RustExtension with Binding.PyO3) and then installs the resulting wheel and runs the repo's test suite against it. Both signals are exit codes emitted by toolchains — the Rust compiler and the CPython interpreter running pytest — executing over the committed source; an author cannot make a type-error in the Rust or a failing assertion report clean without changing the source or the pinned cibuildwheel/pyproject config. DOCUMENTED LIMIT (oracle axis): tests/test_*.py are authored by the same maintainers as src/, so execution independence holds while the specification lives inside the write boundary. Second limit: test-skip="*-macosx_arm64" means the arm64 macOS wheels are built but never exercised, so for those cells only the compile signal exists.

producer the cibuildwheel run: rustc/cargo compiling the PyO3 extension through setuptools-rust, then the pytest process exit code from `pytest {project}/tests` executed against the freshly built wheel in an isolated venv — actor can write to it: no

  • .github/workflows/build_wheels.yml:25
  • .github/workflows/build_wheels.yml:19-20 (matrix: 3 OS x 8 python versions)
  • pyproject.toml:41 test-command = "pytest {project}/tests --import-mode=append"
  • pyproject.toml:40 before-test = "pip install pytest hypothesis"
  • pyproject.toml:38 test-skip = "*-macosx_arm64"
  • setup.py:6-14 RustExtension("tiktoken._tiktoken", binding=Binding.PyO3)
  • tests/test_encoding.py, tests/test_offsets.py, tests/test_pickle.py, tests/test_simple_public.py

decided by agent · confidence 0.85

Build wheels
anchored
.github/workflows/build_wheels.yml:48 · ci_step

Identical mechanism to the x86 job, on native arm64 runners: CIBW_ARCHS=aarch64 with the shared [tool.cibuildwheel] config, so the step ends in a compile of src/*.rs plus `pytest {project}/tests` executed against the installed wheel. The pass/fail is an exit code produced by rustc and by CPython running the suite, neither of which an author can write to without changing the committed source or the pinned config. The macOS test-skip does not apply here, so every aarch64 cell that builds also runs the suite. DOCUMENTED LIMIT (oracle axis): the assertions in tests/ were authored alongside the implementation, so the runtime honestly decides pass/fail but the specification came from inside the write boundary.

producer the same cibuildwheel run on ubuntu-24.04-arm: cargo/rustc compiling the PyO3 extension for aarch64, then the pytest exit code from the configured test-command run against the built wheel — actor can write to it: no

  • .github/workflows/build_wheels.yml:48-56
  • .github/workflows/build_wheels.yml:42-43 (os: ubuntu-24.04-arm, 8 python versions)
  • pyproject.toml:21-41 [tool.cibuildwheel] test-command/before-test
  • setup.py:6-14

decided by agent · confidence 0.85

Run check-manifest
anchored
.github/workflows/build_wheels.yml:73 · ci_step

DELIBERATE RE-JUDGEMENT of a probe verdict — the `unfrozen-dependency-install` probe filed this not_a_check on the strength of the `pip install check-manifest` line, but that line is only the setup; the payload of the step is the second line, `check-manifest -v`, which builds an sdist from the checkout and diffs its contents against `git ls-files`, exiting non-zero when a tracked file is missing from the sdist or an untracked one appears. The two sides of the comparison are produced by two independent processes (setuptools packaging under MANIFEST.in, and git's index) and an author cannot make a stale MANIFEST.in report clean without changing MANIFEST.in or the tracked files themselves. LIMIT: check-manifest is installed unpinned from PyPI at run time, so the exact rule set is not frozen in the repo — the producer is still outside the write boundary (further outside, in fact), but the behaviour can drift between runs without a repo change.

producer the check-manifest process exit code, comparing the sdist that the build backend produces against the file list git reports as tracked — actor can write to it: no

  • .github/workflows/build_wheels.yml:73-76
  • MANIFEST.in:1-8
  • pyproject.toml:17-19 build-backend = setuptools.build_meta

decided by agent · confidence 0.75

Build sdist
anchored
.github/workflows/build_wheels.yml:77 · ci_step

DELIBERATE RE-JUDGEMENT of a probe verdict — `unfrozen-dependency-install` claimed this as not_a_check because the raw contains `pip install --upgrade build`, but that is the setup line; the step's payload is `python -m build --sdist`, an executed packaging build. (The `packaging-build-invocation` probe, which would have called this anchored, correctly abstained on the same 'pip install' token, so the node was decided by the weaker of two matches.) The exit code is emitted by the interpreter resolving the build-system requires (setuptools>=62.4, wheel, setuptools-rust>=1.5.2) and running the backend against the checkout; malformed metadata or an unresolvable backend requirement fails the step and no author can talk it green. LIMIT, named because it is thin: an sdist build packages sources and does not compile the Rust extension, so this edge asserts that the packaging metadata is well-formed, not that the code works — the compile-and-test signal lives in the cibuildwheel jobs.

producer the `python -m build --sdist` exit code — CPython running the setuptools build backend in an isolated environment over the committed pyproject.toml/setup.py/MANIFEST.in — actor can write to it: no

  • .github/workflows/build_wheels.yml:77-80
  • pyproject.toml:17-19 requires = ["setuptools>=62.4", "wheel", "setuptools-rust>=1.5.2"]
  • setup.py:1-19

decided by agent · confidence 0.65

not_a_check — 8

Asserts nothing about correctness, so it is excluded from the denominator. This is the one shoppable class: mis-filing a real check here shrinks the denominator and inflates the score, so each row carries the same burden of argument as any other verdict.

actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
not_a_check
.github/workflows/build_wheels.yml:23 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
not_a_check
.github/workflows/build_wheels.yml:30 · ci_step

The step's whole effect is to move a file the pipeline itself produced from one job to another. Nothing about the repository is evaluated: with `if-no-files-found: error` the worst it can report is that an earlier step in the same run did not leave the file it claimed to leave, which is a fact about the plumbing, not about whether the code is correct. Whatever assertion exists in such a job is made by the step that produced the artifact or the step that consumes it, so this edge must not enter the ratio in either direction.

producer a GitHub Actions artifact transport action (upload-artifact / download-artifact) moving files between jobs — actor can write to it: not established

  • the step body is a bare `uses:` of an artifact upload/download action with no command of its own

decided by probe (ci-artifact-transport) · confidence 0.7

actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
not_a_check
.github/workflows/build_wheels.yml:46 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

cibw-wheels-aarch64-${{ matrix.os }}-${{ strategy.job-index }}
not_a_check
.github/workflows/build_wheels.yml:58 · ci_step

The step's whole effect is to move a file the pipeline itself produced from one job to another. Nothing about the repository is evaluated: with `if-no-files-found: error` the worst it can report is that an earlier step in the same run did not leave the file it claimed to leave, which is a fact about the plumbing, not about whether the code is correct. Whatever assertion exists in such a job is made by the step that produced the artifact or the step that consumes it, so this edge must not enter the ratio in either direction.

producer a GitHub Actions artifact transport action (upload-artifact / download-artifact) moving files between jobs — actor can write to it: not established

  • the step body is a bare `uses:` of an artifact upload/download action with no command of its own

decided by probe (ci-artifact-transport) · confidence 0.7

actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
not_a_check
.github/workflows/build_wheels.yml:68 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

Install Python
not_a_check
.github/workflows/build_wheels.yml:69 · ci_step

The step's whole effect is to put files or a toolchain on the runner so later steps have something to act on. It reads nothing about the repository's correctness and emits no pass/fail verdict about the artefact — a failure here means the runner could not fetch or install, not that the code is wrong. The check in such a job is whatever executes afterwards, so this edge must not enter the ratio in either direction.

producer a GitHub Actions provisioning action (repository checkout, toolchain setup, or dependency cache restore) — actor can write to it: not established

  • the step body is a bare `uses:` of a checkout/setup/cache action with no command of its own

decided by probe (ci-provisioning-action) · confidence 0.9

cibw-wheels-sdist
not_a_check
.github/workflows/build_wheels.yml:81 · ci_step

The step's whole effect is to move a file the pipeline itself produced from one job to another. Nothing about the repository is evaluated: with `if-no-files-found: error` the worst it can report is that an earlier step in the same run did not leave the file it claimed to leave, which is a fact about the plumbing, not about whether the code is correct. Whatever assertion exists in such a job is made by the step that produced the artifact or the step that consumes it, so this edge must not enter the ratio in either direction.

producer a GitHub Actions artifact transport action (upload-artifact / download-artifact) moving files between jobs — actor can write to it: not established

  • the step body is a bare `uses:` of an artifact upload/download action with no command of its own

decided by probe (ci-artifact-transport) · confidence 0.7

Merge artifacts
not_a_check
.github/workflows/build_wheels.yml:91 · ci_step

The step's entire body is a `uses:` of the artifact merge action with a name/pattern/delete-merged config. Its only effect is to fold the cibw-wheels-* bundles that earlier jobs uploaded into a single cibw-wheels bundle; it reads none of the repository's source and evaluates no property of it, so its failure modes are 'no artifact matched the pattern' or 'the artifact API errored' — facts about run plumbing, not about whether tiktoken is correct. The real gate in this job is the `needs: [build_wheels, build_wheels_aarch64, build_sdist]` edge, whose signal is produced by those jobs and is classified where it is emitted, not here. Filing this in the ratio in either direction would credit or penalize the score for a file copy.

producer actions/upload-artifact/merge, a GitHub Actions artifact aggregation action combining artifacts the same workflow run already uploaded — actor can write to it: not established

  • .github/workflows/build_wheels.yml:91-96
  • .github/workflows/build_wheels.yml:89 needs: [build_wheels, build_wheels_aarch64, build_sdist]

decided by agent · confidence 0.85

Crystallization curve

Keel crystallization curve Crystallization curve over 15 runs from <repo>/reports. falls — total fitted change -66.5% of the mean across 15 runs (R^2 0.28, so the line explains less than half the variance — read the raw squares) CRYSTALLIZATION CURVE 15 sequential runs · 345 nodes judged of 1838 gathered · 124 anchored Keel corpus — 15 repositories, 2026-07-24 — measured corpus (declared on the command line) estimated tokens per node estimated tokens / judged node 0 300 600 run 0 · keel · estimated tokens per node = 498 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · estimated tokens per node = 369 · judged 25 of 41 gathered run 2 · openai-python · estimated tokens per node = 398 · judged 25 of 95 gathered run 3 · vercel-ai · estimated tokens per node = 410 · judged 25 of 1014 gathered run 4 · aider · estimated tokens per node = 204 · judged 25 of 62 gathered run 5 · browser-use · estimated tokens per node = 378 · judged 25 of 104 gathered run 6 · mcp-python-sdk · estimated tokens per node = 592 · judged 25 of 121 gathered run 7 · simonw-llm · estimated tokens per node = 189 · judged 25 of 38 gathered run 8 · tiktoken · estimated tokens per node = 240 run 9 · requests · estimated tokens per node = 290 · judged 25 of 106 gathered run 10 · flask · estimated tokens per node = 408 · judged 25 of 59 gathered run 11 · sinatra · estimated tokens per node = 265 · judged 25 of 59 gathered run 12 · commander-js · estimated tokens per node = 316 run 13 · anthropic-quickstarts · estimated tokens per node = 310 · judged 25 of 62 gathered run 14 · tiktoken · estimated tokens per node = 53 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 seconds per node measured s / judged node 0.0 60.0 120.0 run 0 · keel · seconds per node = 0.1 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · seconds per node = 15.8 · judged 25 of 41 gathered run 2 · openai-python · seconds per node = 15.4 · judged 25 of 95 gathered run 3 · vercel-ai · seconds per node = 105.9 · judged 25 of 1014 gathered run 4 · aider · seconds per node = 21.3 · judged 25 of 62 gathered run 5 · browser-use · seconds per node = 21.7 · judged 25 of 104 gathered run 6 · mcp-python-sdk · seconds per node = 20.1 · judged 25 of 121 gathered run 7 · simonw-llm · seconds per node = 11.8 · judged 25 of 38 gathered run 8 · tiktoken · seconds per node = 23.7 run 9 · requests · seconds per node = 16.8 · judged 25 of 106 gathered run 10 · flask · seconds per node = 20.5 · judged 25 of 59 gathered run 11 · sinatra · seconds per node = 16.9 · judged 25 of 59 gathered run 12 · commander-js · seconds per node = 18.7 run 13 · anthropic-quickstarts · seconds per node = 20.4 · judged 25 of 62 gathered run 14 · tiktoken · seconds per node = 0.1 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 probe-decided share measured share of decided nodes 0.00 0.50 1.00 run 0 · keel · probe-decided share = 0.00 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · probe-decided share = 0.00 · judged 25 of 41 gathered run 2 · openai-python · probe-decided share = 0.16 · judged 25 of 95 gathered run 3 · vercel-ai · probe-decided share = 0.08 · judged 25 of 1014 gathered run 4 · aider · probe-decided share = 0.56 · judged 25 of 62 gathered run 5 · browser-use · probe-decided share = 0.32 · judged 25 of 104 gathered run 6 · mcp-python-sdk · probe-decided share = 0.00 · judged 25 of 121 gathered run 7 · simonw-llm · probe-decided share = 0.56 · judged 25 of 38 gathered run 8 · tiktoken · probe-decided share = 0.58 run 9 · requests · probe-decided share = 0.28 · judged 25 of 106 gathered run 10 · flask · probe-decided share = 0.00 · judged 25 of 59 gathered run 11 · sinatra · probe-decided share = 0.28 · judged 25 of 59 gathered run 12 · commander-js · probe-decided share = 0.19 run 13 · anthropic-quickstarts · probe-decided share = 0.24 · judged 25 of 62 gathered run 14 · tiktoken · probe-decided share = 1.00 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 probe library size measured probes in library 0 15 30 run 0 · keel · probe library size = 0 · judged 25 of 32 gathered run 1 · anthropic-sdk-python · probe library size = 3 · judged 25 of 41 gathered run 2 · openai-python · probe library size = 5 · judged 25 of 95 gathered run 3 · vercel-ai · probe library size = 8 · judged 25 of 1014 gathered run 4 · aider · probe library size = 10 · judged 25 of 62 gathered run 5 · browser-use · probe library size = 12 · judged 25 of 104 gathered run 6 · mcp-python-sdk · probe library size = 13 · judged 25 of 121 gathered run 7 · simonw-llm · probe library size = 15 · judged 25 of 38 gathered run 8 · tiktoken · probe library size = 17 run 9 · requests · probe library size = 19 · judged 25 of 106 gathered run 10 · flask · probe library size = 22 · judged 25 of 59 gathered run 11 · sinatra · probe library size = 23 · judged 25 of 59 gathered run 12 · commander-js · probe library size = 26 run 13 · anthropic-quickstarts · probe library size = 29 · judged 25 of 62 gathered run 14 · tiktoken · probe library size = 29 0* 1* 2* 3* 4* 5* 6* 7* 8 9* 10* 11* 12 13* 14 x axis: run index in the recorded corpus order (listed below). Squares are the raw per-run values; the dashed line is an ordinary-least-squares fit and is never shown without them. The fit is clipped to the panel, and withheld entirely (with the panel saying so) where a straight line would predict values the points cannot take — R^2 for every fit is in the trend notes below. * on run 0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13 = judged fewer nodes than gathered; per-node values are per JUDGED node. Coverage (judged) ci_step 224 · script 64 · test_target 46 · review_gate 10 · deploy_gate 1 Trend estimated tokens per node: falls — total fitted change -66.5% of the mean across 15 runs (R^2 0.28, so the line explains less than half the variance — read the raw squares). raw first 498 -> last 53. seconds per node: falls — total fitted change -72.9% of the mean across 15 runs (R^2 0.04, so the line explains less than half the variance — read the raw squares). raw first 0.1 -> last 0.1. probe-decided share: rises — total fitted change 144.9% of the mean across 15 runs (R^2 0.21, so the line explains less than half the variance — read the raw squares). raw first 0.00 -> last 1.00. probe library size: rises — total fitted change 187.7% of the mean across 15 runs (R^2 0.99). raw first 0 -> last 29. Run order run order derived from Report.generatedAt (ascending, filename as tiebreak) — no order.json manifest present 0 keel -> 1 anthropic-sdk-python -> 2 openai-python -> 3 vercel-ai -> 4 aider -> 5 browser-use -> 6 mcp-python-sdk -> 7 simonw-llm -> 8 tiktoken -> 9 requests -> 10 flask -> 11 sinatra -> 12 commander-js -> 13 anthropic-quickstarts -> 14 tiktoken Shuffle check shuffle check INCOMPLETE — the curve direction ran, the ratio direction did not Permutation (400 draws, seed 20260724): median |delta normalized slope| 0.633, 92% of draws move it by >= 0.15. A permutation reorders ALREADY-RECORDED runs; it cannot reproduce what a genuine re-run in a different order would have cost, because the probe library would have accumulated differently. It is a sanity signal on order-dependence, never a substitute for the empirical re-run. Ratio stability under shuffle was NOT checked: no re-run supplied (--shuffled) and none declared in corpus.meta.json. Permuting recorded runs cannot move a per-target ratio, so the permutation result below says nothing about it. Supply a re-run with --shuffled <dir>. Disclosures - Skipped 3 file(s) that are not usable run reports: corpus-summary.json (no string "target"); curve.json (no string "target"); keel.bindings.json (no "nodes" array). - Run 0 (keel) judged 25 of 32 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 1 (anthropic-sdk-python) judged 25 of 41 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 2 (openai-python) judged 25 of 95 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 3 (vercel-ai) judged 25 of 1014 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 4 (aider) judged 25 of 62 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 5 (browser-use) judged 25 of 104 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 6 (mcp-python-sdk) judged 25 of 121 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 7 (simonw-llm) judged 25 of 38 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 9 (requests) judged 25 of 106 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 10 (flask) judged 25 of 59 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 11 (sinatra) judged 25 of 59 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 13 (anthropic-quickstarts) judged 25 of 62 gathered nodes — a cap. Per-node cost below is per JUDGED node. - Run 14 (tiktoken): decidedByProbe + decidedByAgent = 10 but nodesSampled = 12. Probe-decided share uses the decided total (10). - Provenance "measured" was declared on the COMMAND LINE (--provenance), not in <repo>/reports/corpus.meta.json. The declaration therefore lives in the invocation and is only as trustworthy as the run sheet that records it. - Token counts are ESTIMATES in 15 of 15 run(s) (RunEconomics.tokensEstimated). No API exposes session token usage to a skill, so the token axis reads "estimated tokens" and nothing here claims a measured token count. Scope. Keel measures the shape of verification, not its quality. A repo can be 100% anchored with terrible tests. Anchoring says the signal comes from outside; it does not say the signal is sufficient.

Run economics

7 decided by probe
5 decided by agent
58% probe-decided share
12 / 12 nodes judged / gathered
892 estimated tokens in
1,987 estimated tokens out
4m 45s wall clock
2 probes minted
17 probe library size

Token counts are estimated, and labelled so: a skill running inside an agent session has no API for its own usage, so the figure is ceil(chars/4) over the judgment payloads and responses. Wall clock and probe-decided share are measured directly.