Full benchmark: four projects, Opus 5.5 and Haiku 5.5 at high effort (2026-10-18)

Code: src/eval/bench/ (run.js, calibrate.js, report.js, totals.js, the projects under projects/) with the judge and audit of src/eval/agent/ (judge.js, audit.js). The tables below come from report.js; nothing in them is edited by hand. In data/: the pre-registered selection (bench-2026-10-calibration.json), per model the report, the judge’s verdicts with the audit’s overrides and the audit verdicts (bench-2026-10-<model>.report.json, .judge.json, .audit-verdicts.jsonl), the site summaries (.summary.json) and the totals across both models (bench-2026-10-totals.json). The per-run rows, the streams with their hook events and the diffs stay on the owner’s machine: they name local paths.

What was measured

Each run is Claude Code headless (claude -p --effort high, project settings only) in a fresh clone of one of four projects, under four conditions:

The candidates cover twenty situations (S1–S20: a trap in the code path, a decision the code does not show, a review, a request that contradicts a memory, controls where no memory applies, and so on). Per run: the task’s own check, the project’s suite and lint on the changed tree (resolution); the task’s project requirements, decided by checks; tested, scoped and conventions, decided by a blind judge; peak context, session time and cost as Claude Code reports them; and for dd2, from the hook events, when the key memories arrived and what else came with them.

Validity

• Pre-registration. 72 candidate tasks ran three times each with Haiku in none (216 runs). A memory task was kept when the agent alone failed it at least once; controls were always kept; at most four per situation. 26 were selected (15 memory tasks, 11 controls) and committed before any full run (bench-2026-10-calibration.json, commit e21b16f). The rule: no task, check or memory changes after that commit except a harness fault, recorded in its commit.
• Proof per task. Every check fails on an untouched clone, passes the correct overlay, and (memory tasks with one) catches the naive overlay, which does what was asked and breaks the memory (tests/bench/). Mined and ported tasks have no naive overlay: the fix’s own test decides them.
• The guide is the store. The documentation conditions carry the same memories as DD, so they differ from dd2 only in the channel.
• Blind judge. Claude Opus 5.5, no tools, an empty directory; it sees the task, the guidance, the requirements, the diff and the developer’s report with DD’s and the guide’s names redacted, never the condition. All 624 runs were judged.
• Blind audit. Every verdict was re-read from a hashed packet with the condition redacted (audit.js), by twelve auditing agents (Claude Sonnet 5.5) and the owner’s agent: 11 overrides for Haiku, 17 for Opus, each with its reason in the audit verdicts.
• Billing. Every agent and judge ran on the owner’s subscription; the runner refuses to start with an API key in the environment. A usage limit is waited out and the run repeated, never recorded: two session limits were waited out (2 h 44 min in the Opus run, 2 h 8 min in the judge).

Harness faults found and fixed

Every row was read against its diff before any number was taken. Each fault below made a check fail work that was correct. Each fix was shown to fail the recorded diffs before and pass them after, the task’s proof was run again, and the recorded diffs were checked again (run.js --recheck, the original verdict kept on the row under rechecked). Unit tests pin the suite rules, the review matcher (with reviews that must not count), the replay client rule and the skills check; the other fixes rest on the recorded diffs and the proofs. A final review of the fixes found two of them too lenient (the review matcher credited reviews that clear the registration, the replay rule credited a partner line the change rewrites); both were tightened, with no verdict changing on the recorded runs.

Before selection (from the calibration):

• tollgate-network-retry: Haiku read the prompt as an instruction for its own session, 3/3 one-turn runs with no change; the prompt now names the bug.
• supermem-glob-star-chain: three correct fixes dropped the globToRegExp export and the hidden test crashed on import; it now goes through the matcher and the compiled index the hook uses.
• supermem-skills-npx: three correct skills wrote “Do not run npx deltadictum” and the check counted it as running npx.
• supermem-compile-overlap was dropped: a lock, a rebuild of a bodiless index and unclaimed bodiless deliveries are all defensible fixes, and its test accepted only the upstream design.

After the pre-registration (from the smoke and the full runs):

• The suite verdict failed a run when its two suite runs failed different tests (no test failed twice).
• supermem: one maintenance test failed under load in a run that touched only the UI, and passed 5/5 on that change: known flaky.
• tollgate-dedup-evict: two Opus dd2 runs gave NewDedup a logger (passing nil in their own tests); the hidden test called NewDedup() and did not build.
• patriark-short-hash: three Haiku runs wrote an instance ShortHash, like the instance StateHash beside it; the hidden test called it static.
• tollgate-replay-to: the idle-timeout requirement read one file while most runs built the client where the subcommand is wired; and the hidden test set no signing secret, failing five Haiku runs that signed every replay, as the project’s rule says, and refused to send one unsigned.
• patriark-review-command: the review matcher missed “isn’t registered”, “didn’t add”, “has no [JsonSerializable]” and other wordings; all 24 reviews name the finding.

Two selected tasks entered the selection only through these faults: with the fixed checks the calibration has no failure on tollgate-replay-to or patriark-review-command. They stay in the pre-registered set and are near the ceiling in every condition.

Results: Claude Opus 5.5, effort high

26 tasks × 4 conditions × 3 runs = 312 runs, no harness error, no timeout. Resolution = check passed, suite and lint clean. Context, time and cost: medians and mean, 0 = the lowest condition.

Project requirements are over the 15 memory tasks; the other rows over all 26.

• Against the agent alone, paired by task: DD meets 31 points more of the project’s requirements (95% interval 13 to 51), and resolves 5 points more (0 to 14). Alone, Opus fails the task or breaks a project rule in 15 of 45 memory runs: the store format not bumped (tollgate-store-attempts, 3/3), the 72-hour dedup window evicted (tollgate-dedup-package 3/3, tollgate-dedup-evict 2/3), a command that rewrites a file not counted as an edit (supermem-run-is-not-edit, 3/3), overlapping imports doubled (ledgerkit-import-dir, 2/3), and one run each of two other tasks. With DD: none of 45.
• Against the referenced guide, the same knowledge in the other channel: no difference in resolution or requirements (both at the ceiling), and DD is cheaper. Paired, $0.046 less per run (0.008 to 0.090) and 4.0k tokens less peak context (1.0k to 7.7k): the guide is read whole, DD delivers what fires.
• Delivery. The key memory reached the agent in 98% of memory runs, 39 of 44 before the first edit. Median four other memories per run; median 773 tokens injected; the hook took a median 47 ms.
• Controls (no memory applies): 100% resolved in every condition; DD added a median one memory and 254 tokens.

Results: Claude Haiku 5.5, effort high

312 runs, no timeout. One check lost its connection to the review UI server it started (ECONNRESET) and was run again on the recorded change.

• Against the agent alone, paired by task: DD meets 52 points more of the project’s requirements (30 to 72). The resolution is the same (92%), for different reasons: alone, Haiku fails memory tasks; with DD, it fails the control below.
• DD made Haiku refuse an explicit request. In ledgerkit-summary-print (S10) the user asks for the import summary on stdout with print(), which a memory forbids (“stdout carries only report output”). With DD, Haiku declined the request in 3 of 3 runs, citing the memory; with the referenced guide in 1 of 3; alone and with the unreferenced guide never. Opus did what was asked in every condition. DD’s contract is that delivered knowledge never outranks the user; with Haiku, a delivered memory did. It is the benchmark’s clearest finding against DD.
• Against the guide, DD is behind: 5 points less resolution against the referenced guide (−13 to 1) and 8 against the unreferenced one (−18 to 0). Of the six dd2 runs not resolved, three are the refusal above, two supermem-run-is-not-edit and one review that left scratch files in the repository. Requirements are level within their intervals (−2 against the referenced guide, −13 to 7). Alone, Haiku fails the task or breaks a rule in 28 of 45 memory runs; with DD in 3.
• Delivery. The key memory reached the agent in 96% of memory runs, 36 of 43 before the first edit; median three other memories and 602 tokens; the hook took a median 54 ms.
• Cost. DD is the most expensive condition with Haiku (+30% against the agent alone, $0.006 per run paired), where it is the cheapest knowledge channel with Opus.

Caveats

• Selected tasks. The memory tasks are the ones Haiku alone failed at least once; the gap against the agent alone is measured where knowledge matters, by design, and is not an estimate for an average task. Ten situations kept a memory task (S1, S2, S4, S5, S6, S8, S13, S14, S15, S18); in S3, S7, S9, S12, S16, S17, S19 and S20 Haiku alone solved every candidate. Patriark kept one memory task.
• Small samples. 45 memory runs per condition and model; intervals are wide and the guide conditions sit at the ceiling, so DD and the guide are not told apart on requirements.
• Same knowledge, same day. Every condition carries the same memories; the comparison is the channel, not the content. A real project’s guide is rarely this complete, nor its store.
• One judgment area was ruled. Where the user explicitly asks for stdout (three control tasks), the judge and the auditors split on whether printing there breaks the project’s stdout rule; the owner’s ruling, applied blind to every run of those tasks, is that it does not (the user outranks the rule). It changed 12 conventions verdicts (10 of the judge’s, 2 of an auditor’s): six on guide runs, six on runs without project knowledge, none on a dd2 run.
• One judgment area was not. In tollgate-replay-to the judge counted signing the existing replay path as out of scope; the runs that did it followed the project’s rule that replays are signed, and they are mostly dd2 and doc (Opus 3/3 each, Haiku 2/3 each, against 0/3 alone). Counting it in scope moves code quality by about one point for dd2 and doc; the verdicts stand as audited.
• Usage on a subscription. Costs are Claude Code’s list-price equivalents; the whole benchmark (calibration, smoke, both full runs and the judge) came to about $208.

What the two models say together

DD carries a project’s knowledge as well as a complete guide that CLAUDE.md points to, at a lower cost than the guide with Opus and a higher one with Haiku, and both channels lift the project’s requirements far above the agent alone (from 69% to 100% with Opus, from 43% to 96% with Haiku). The difference between DD and the guide is not in what the agent knows but in what it does with it: Opus follows a delivered memory and still does what the user asks, Haiku in one task let the memory win. That is a delivery problem DD can work on (how a memory is phrased when it contradicts the request in front of the agent), and the reason the benchmark keeps its controls.

Reproduce

node src/eval/bench/run.js --check-only
node src/eval/bench/run.js --model=claude-haiku-5-5 --effort=high --conditions=none --repeat=3 --no-suite \
  --out=bench-2026-10/calibration/results.json
node src/eval/bench/calibrate.js --in=bench-2026-10/calibration/results.json --out=docs/benchmarks/data/bench-2026-10-calibration.json
node src/eval/bench/run.js --model=claude-opus-5-5 --effort=high --selection=docs/benchmarks/data/bench-2026-10-calibration.json \
  --repeat=3 --conc=3 --out=bench-2026-10/opus/results.json      # --resume until every job has a row
node src/eval/agent/judge.js --in=bench-2026-10/opus/results.json --model=claude-opus-5-5 --conc=2   # --resume
node src/eval/agent/audit.js --in=bench-2026-10/opus/results.json      # then --apply=<verdicts>.jsonl
node src/eval/bench/report.js --in=bench-2026-10/opus/results.json
# the same for Haiku; then totals.js over both reports

Patriark’s runs need the Patriark repository beside this one; its setups and solutions are generated from it into output/eval (projects/patriark/generate.mjs), never kept here.