How MuckLedger scores a cleanup tool

0. Which tools qualify

Only deterministic, standalone command-line tools are benchmarked. Given the same input and version, they must produce the same output. Agent-driven or LLM-backed “cleanup skills” are excluded: their edits change from run to run, so a score for them cannot be reproduced. Niche tools whose behaviour is undocumented are excluded until it is.

Go: scripted deletion, and why findings use test-rooted analysis

deadcode never edits files. The fix pass for Go is a small MuckLedger-authored program (harness/gotools/deadcodefix) that takes deadcode’s JSON output and deletes exactly those function and method declarations using the real Go AST (go/parser + go/format), then runs the official goimports to drop any import left unused by the deletion — an unused import is a hard compile error in Go, so this step is not optional. The deletion is disclosed as MuckLedger’s own script, not credited to deadcode, the same way vulture’s findings are never credited as a ruff fix.

deadcode is run with -test, which makes test files additional reachability roots. A function used only by the test suite is therefore not counted as a finding. This mirrors how knip treats test files as entry points for TypeScript: “findings” means code nothing reachable, tests included, still uses. For comparison, an informational deadcode_main_only_findings metric also records what deadcode reports with only main as the root (no -test); it is never used for scoring, exactly like vulture’s candidate count.

Consequence worth stating plainly: a Go repository with a thorough test suite can legitimately score zero findings, because everything deadcode can see is reachable from some test. That is not the tool failing to find anything to clean — it is the test suite already pinning every function in place. Such a run has no resolution rate (findings_total = 0) and sorts last, the same treatment any zero-findings run gets under the ranking rule above.

Rule set and repo config

By default every Python run uses exactly this MuckLedger rule set, with the repository’s own ruff configuration ignored (--isolated) so that all repos are judged by the same rules:

F401  unused-import               fixed automatically (safe fix)
F841  unused-variable             reported only (ruff marks the fix unsafe)
F811  redefined-while-unused      reported only (no auto-fix exists)

# ruff never auto-removes unused imports inside __init__.py files
# (they may be intentional re-exports); those are reported only.

A repository can be flagged treat_as_library = true. Then the tool respects the repo’s own config instead. This is for mature public-API packages such as psf/requests, whose per-file-ignores mark intentional re-exports: deleting those would break external callers that the live-caller gate cannot see, because they live outside the repo. The mode used is recorded on every run as the ruff_mode metric and shown on the run page.

knip has no isolated switch. In the default mode it runs with its own project auto-detection and configuration hints suppressed; a repo-level knip.json is still honoured because knip cannot ignore one. The run records this as knip_mode. The tool version is pinned by the harness and executed through npx, so a target repo cannot substitute its own copy.

1. Ranking rule

This is the exact rule implemented by the leaderboard database view.

# Gate: a run is excluded from the leaderboard entirely
# (not scored, not ranked) unless BOTH are true.
tests_passed            == true
no_live_callers_deleted == true
# NULL / unknown counts as failing the gate.

# Per run, among runs that pass both gates:
if findings_total == 0:
    is_clean_run = true          # the tool had nothing to flag on this repo
else:
    resolution_rate_pct = findings_resolved / findings_total * 100
#   findings_total     = what the tool itself flagged before its fix pass
#   findings_resolved  = flagged findings no longer reported after the fix pass
#   "safely" is enforced by the gate above, not by the formula.

# Per tool, across EVERY approved run (not just the latest):
avg_resolution_rate_pct    = avg(resolution_rate_pct) over rated runs only
avg_safe_loc_reduction_pct = avg(safe_loc_reduction_pct), same rated runs
rated_run_count            = count of runs that fed the average
clean_run_count            = count of runs with findings_total = 0

# Rank: avg_resolution_rate_pct desc, then avg_safe_loc_reduction_pct desc,
# then most recent run. A tool with only clean runs has no average (NULL)
# and sorts last — it is never scored as 0%.

• A tool that resolves more but breaks one test is not “almost first”. It is absent.
• Every approved run counts, not just the latest, so the leaderboard reflects a tool’s average performance across the whole set of benchmark repositories, not a single snapshot.
• A clean run (the tool flagged nothing on that repo) is a real, valid, approved run. It is not a failure and it is not a 0%: it is excluded from the average and counted separately, shown on each tool’s page at /tools/<slug>. Folding it in as 0% would punish a tool for being run against a repo that happened to already be clean.
• Detection-only output (such as vulture candidates) is evidence, never score.
• The score is a rate, not a volume. It answers “of what this tool claims it can clean, how much does it actually clean without breaking anything?” It does not reward flagging more.

Why resolution rate, not LOC reduction

The first version of this leaderboard ranked by lines of code removed. Two real runs showed that metric cannot score tools fairly, because a correct, safe fix often deletes no lines at all.

Both tools did exactly what they claim, safely, on genuinely AI-generated code. Under LOC reduction, knip would rank below ruff on a 0.00% vs 0.18% difference that reflects how their fixes are shaped, not how well they work. Under resolution rate both score 100%, and the LOC column breaks the tie in ruff’s favour, which is the right order for a reader who cares about lines but not a verdict on knip. LOC reduction stays on the board as a secondary column because it is still real, measured information.

Both runs above were executed by the harness in --dry-run mode on this machine; their evidence is committed with the target repos’ pinned commits (ticketctl@5301eb0, pantry@a1cd4f0) so anyone can reproduce them.

2. What one run does

1. Pin inputs. Clone the target repository at a specific commit SHA. Record the exact tool version reported by the tool itself.
2. Baseline. Run the repository’s own test suite on the untouched checkout. If the baseline fails, the run is aborted and never published: we do not benchmark against a broken repo.
3. Measure before. Count non-blank lines (loc_before) in the files matched by the repo’s harness config, excluding vendored and generated paths.
4. Run the tool. Invoke it exactly as listed in the table above. Capture stdout/stderr to tool.log and the full working-tree diff to cleanup.diff.
5. Gate: tests. Run the same test suite again. tests_passed is true only if the suite exits 0 and the tool itself did not crash. Output is saved to tests.log.
6. Gate: live callers. For every function, method or class the diff removed, search the post-cleanup tree for remaining references. Any hit is a broken live call site. no_live_callers_deleted is true only if the count is zero.
7. Measure after. Count loc_after, loc_removed, files_changed, symbols_removed.
8. Publish evidence. Upload the diff and logs to public storage, write the run and its metrics to the database with approved = false.

3. Human approval

The harness never publishes directly. Every run lands in an unapproved state and is invisible to the site. A maintainer reads the diff and logs, confirms the metrics match the evidence, and flips approved to true by hand. Only then does the run appear on the leaderboard. A run with no evidence links should never be approved.

4. What this does not measure

• Code quality of what remains. A tool that deletes safely but leaves the code ugly still scores well. Readability metrics may be added later as separate, non-ranking columns.
• Behaviour not covered by tests. The test suite is the oracle. Repos with weak suites are weaker benchmarks, which is why the repository is shown on every row.
• Cost or speed. wall_time_seconds is recorded but does not affect rank.
• Unsafe fixes. Tools are run in their safe default mode. Opt-in unsafe modes may be benchmarked later as separate tool entries.

5. Reproducing a run

cd harness
pip install -e ".[tools]"          # installs ruff + vulture at the recorded versions
python -m harness.runner \
  --tool ruff --tool-version <version> \
  --repo <repository-slug> --commit <sha> \
  --dry-run     # prints metrics + writes evidence locally, no DB write

The run page for every leaderboard entry shows the exact tool version and commit SHA to pass.