Benchmarks
Six coding sessions, from an 8-turn fix to a 120-turn audit, replayed through Dulus, six other agent harnesses and a naive control on byte-identical transcripts, with one tokenizer and one price table. Dulus came first under all five weightings, and cost less in all 42 head-to-head matchups.
$25.63
for all six sessions
Claude Code, the next cheapest, costs $48.00 for the same work. Dulus costs 46.6% less.
9
cache-prefix breaks
Claude Code: 74. GitHub Copilot: 220. Every break bills tokens already sent at full price again.
0
context overflows
Its largest prompt was 139.8k tokens of a 200k window, in sessions up to 120 turns long. GitHub Copilot overflowed on 222 turns.
How to read every number on this page
Dulus. The arena runs Dulus’s own context code on the transcript: the lookback window and the two-layer compaction.
The other six. Each product’s context policy is replayed as a behaviour — grow, window, or summarise at a threshold — on the byte-identical transcript. It is not their source code.
Everyone. One tokenizer and one price table, cache-read discount included. The scenarios are synthetic, calibrated to real agent work.
01 · The bill
What all six sessions cost each harness, priced from one table with the cache-read discount included. Dulus: $25.63. Claude Code, the next cheapest: $48.00.
| Harness | Session cost | Billed prompt tokens |
|---|---|---|
| DulusMeasured | 18.52M tokens | |
| Claude CodeModelled | 32.13M tokens | |
| Codex CLIModelled | 31.59M tokens | |
| CursorModelled | 30.23M tokens | |
| ClineModelled | 27.86M tokens | |
| AiderModelled | 243.97M tokens | |
| GitHub Copilot agent modeModelled | 71.71M tokens | |
| Naive loopControl | 243.97M tokens |
02 · The window
A prompt bigger than the model’s window doesn’t run slowly: it fails. Across all six scenarios, Dulus never sent one. Its largest prompt was 139.8k tokens, of a 200k window.
claude-sonnet-4-6, with its 200k window
Turn
| Harness | Largest prompt | Cuts | Turns over the window |
|---|---|---|---|
| Dulus | 134,112 | 3 | 0 |
| Claude Code | 182,719 | 17 | 0 |
| GitHub Copilot | 289,280 | 108 | 105 |
| Naive loop and Aider | 2,357,383 | 0 | 110 |
The gold dots mark Dulus’s 3 cuts: turns where the cached head of the prompt was rewritten and billed again at full price. Its smaller dips are the uncached tail changing size, while the cached head keeps growing. Claude Code cut 17 times, at every sharp drop in its line. The naive loop never cuts: it crosses the window at turn 11 and leaves the chart at turn 16, and Aider’s modelled curve is the same line.
| Harness | Largest prompt, against a 200k window | Turns over the window |
|---|---|---|
| Dulus | 139,805 | 0 over the window |
| Claude Code | 183,405 | 1 over the window |
| Codex CLI | 183,192 | 1 over the window |
| Cursor | 182,994 | 1 over the window |
| Cline | 182,499 | 1 over the window |
| GitHub Copilot | 604,731 | 222 over the window |
| Aider | 2,357,383 | 240 over the window |
| Naive loop | 2,357,383 | 240 over the window |
The one overflow each for Claude Code, Codex CLI, Cursor and Cline came in the cheap-model session, where deepseek-chat has a 64k window: each crossed it at turn 20. Dulus peaked there at 41,151 tokens.
03 · The cache
A provider bills the cached part of a prompt at a steep discount, but only while the start of the prompt stays exactly the same. Rewrite that head and every token in it is billed at full price again. Over 288 turns, Dulus rewrote it 9 times.
Claude Code’s modelled policy serves 62.2% of its prompt tokens from the cache, on average; Dulus, 61.2%. Claude Code re-sends more, and more of what it re-sends hits. Dulus keeps its cache intact while sending far less, so it pays least.
Dulus61.2%
Claude Code62.2%
Dulus18.52M
Claude Code32.13M
One account · real usage
70.4%
cache hit rate, October 12–18, 2026
155.7M input tokens that week
October 12 alone
Our own account, building with Dulus, as the model provider’s usage dashboard reports it. Real sessions on one account: a different instrument from the arena, so the two hit rates don’t compare.
04 · How it works
No setting to find, nothing to install. This is how the agent in the Windows app handles context out of the box.
The lookback window is anchored with hysteresis: it grows for a while instead of sliding every turn. It cuts the head only when the hidden part is at least twice the kept window, a cache-payoff gate; below that, the whole archive goes out and the cache absorbs it. Earlier images and reasoning stay where they were, so they never force a rewrite.
9
prefix breaks in 288 turns. The others that trim: 74 to 220.
hidden ≥ 2 × kept cut the head
otherwise send it all; the cache absorbs it
Only the last 20 user turns go to the API by default, and you can set anywhere from 2 to 250. The cut falls on user-turn boundaries, so every tool call stays paired with its result. The whole conversation stays on your machine, and the agent searches it itself with its Loopback tool: status, show, head, search.
The trade, said plainly: the model sees 28.0% of old history verbatim, against 46.1% for Claude Code. Loopback is how it gets the rest back. Dulus is the only harness in the arena whose agent can re-read dropped history: measured for Dulus, estimated as absent for the other six.
20
user turns sent, by default. From 2 to 250.
The Python tool is a persistent kernel: variables, imports and data survive between calls. So the model keeps big data in the kernel’s memory instead of the conversation, and only the slice it prints enters the context. An infinite loop or a crash kills the kernel, never Dulus, and it restarts on the next call.
20,000
characters per stream at most, capped at the source. Timeout: 30 s by default, up to 300 s.
>>> runs = load("ci/") stays in the kernel
>>> failing(runs)[:3] only this reaches the context
Memory is markdown, filed in a palace of halls that you can open and edit. The essentials, short_memory and the gold memories, load at the start of every session, on every surface. The agent saves what it learns and searches the palace with its own memory tools.
Every session
the essentials load first: short_memory and the gold memories.
| Capability | Dulus | Claude Code | Codex CLI | Cursor | Cline | Aider | GitHub Copilot |
|---|---|---|---|---|---|---|---|
| Context compaction | YesMeasured | YesVendor-public | YesVendor-public | YesEstimated | YesVendor-public | NoEstimated | No data |
| Sliding history window | YesMeasured | NoEstimated | NoEstimated | No data | YesEstimated | NoEstimated | YesEstimated |
| Cache-aware truncation gate | YesMeasured | NoEstimated | NoEstimated | No data | No data | NoEstimated | No data |
| Agent can re-read dropped history | YesMeasured | NoEstimated | NoEstimated | NoEstimated | NoEstimated | NoEstimated | NoEstimated |
Dulus’s four are measured. Most cells for the other products are estimates, and are marked that way: read them as the arena’s best guess, not a finding. The arena tracks 21 capabilities in all; these four are the ones this page is about, and the dossier lists every one.
05 · Head to head
Every scenario against every challenger. Each cell is the challenger’s cost divided by Dulus’s on the same scenario, with a 95% interval from a paired bootstrap over per-turn costs. Above 1, Dulus is cheaper. All 42 intervals sit above it.
| Scenario | Claude Code | Codex CLI | Cursor | Cline | Aider | GitHub Copilot | Naive loop |
|---|---|---|---|---|---|---|---|
| Quick fix | 1.113×1.047–1.208 | 1.113×1.047–1.208 | 1.113×1.047–1.208 | 1.113×1.047–1.208 | 1.113×1.047–1.208 | 1.113×1.047–1.208 | 3.063×2.056–4.533 |
| Feature build | 1.298×1.187–1.425 | 1.296×1.186–1.422 | 1.307×1.154–1.529 | 1.336×1.139–1.573 | 1.646×1.520–1.774 | 8.936×6.762–11.161 | 9.215×7.640–10.828 |
| Deep refactor | 1.664×1.440–1.938 | 1.732×1.482–2.023 | 1.733×1.499–1.996 | 1.937×1.641–2.243 | 3.102×2.757–3.500 | 15.019×13.171–17.020 | 22.595×19.205–26.522 |
| Marathon audit | 1.465×1.336–1.613 | 1.488×1.363–1.632 | 1.448×1.329–1.590 | 1.492×1.348–1.640 | 4.893×4.432–5.384 | 15.115×13.966–16.355 | 41.446×36.801–46.413 |
| Retrieval-heavy | 2.974×2.533–3.441 | 3.085×2.624–3.542 | 3.367×2.922–3.825 | 3.537×3.112–4.022 | 2.637×2.312–2.997 | 15.522×12.538–18.566 | 17.944×14.696–21.538 |
| Feature build on a cheap model | 2.446×2.024–2.889 | 2.491×2.077–2.913 | 2.531×2.140–2.960 | 2.601×2.217–3.016 | 3.183×2.674–3.777 | 9.166×6.811–11.782 | 10.177×8.144–12.514 |
1.11×
The closest: the 8-turn quick fix. None of the six products has trimmed yet, so they share one ratio, and Dulus still costs less.
15.5×
The widest against a product: GitHub Copilot, on retrieval-heavy.
41×
Against the naive loop, on the 120-turn marathon audit.
Retrieval-heavy is built to be hard on sliding windows, Dulus’s included: nearly every turn drags in a large retrieval blob that the model may need again later.
06 · Any weighting
What counts as best depends on what you weigh. So the arena re-scores every harness under five published profiles, from someone paying the bill to someone who needs the agent never to forget the plan. Dulus placed first under all five.
Neutral
Every metric counts the same.
Dulus 0.847
Runner-up Cursor 0.760
Cost-first
Someone paying the bill.
Dulus 0.931
Runner-up Claude Code 0.856
Fidelity-first
Someone who needs the agent not to forget the plan.
Dulus 0.764
Runner-up Cursor 0.757
Capability-first
Someone shopping for features.
Dulus 0.934
Runner-up Cline 0.690
Arena default
The catalogue’s own weights.
Dulus 0.909
Runner-up Cursor 0.824
The closest call is fidelity-first, which puts the most weight on keeping history and showing it verbatim: Dulus 0.764, Cursor 0.757.
| Harness | Neutral | Cost-first | Fidelity-first | Capability-first | Arena default |
|---|---|---|---|---|---|
| Dulus | 0.847 #1 | 0.931 #1 | 0.764 #1 | 0.934 #1 | 0.909 #1 |
| Claude Code | 0.752 #3 | 0.856 #2 | 0.751 #3 | 0.590 #4 | 0.810 #4 |
| Codex CLI | 0.748 #5 | 0.851 #3 | 0.748 #5 | 0.589 #5 | 0.806 #5 |
| Cursor | 0.760 #2 | 0.849 #4 | 0.757 #2 | 0.683 #3 | 0.824 #2 |
| Cline | 0.748 #4 | 0.836 #5 | 0.749 #4 | 0.690 #2 | 0.818 #3 |
| Aider | 0.586 #6 | 0.628 #6 | 0.687 #6 | 0.487 #6 | 0.526 #6 |
| GitHub Copilot | 0.489 #7 | 0.469 #7 | 0.595 #7 | 0.487 #7 | 0.488 #7 |
| Naive loop | 0.333 #8 | 0.158 #8 | 0.556 #8 | 0.171 #8 | 0.235 #8 |
07 · Method
The Dulus Benchmark Arena builds one transcript per scenario from a fixed seed, runs it through each harness’s context policy, and prices every turn the same way. Here is exactly what that means, and where it stops.
The arena runs Dulus’s own context code: it snips old tool results, strips stale injections and applies the lookback window, with the window’s anchor kept across turns so the cache-payoff gate and the hysteresis are really exercised. The one stand-in is the summary card, a fixed-size surrogate, because writing it would need a live model call. The naive control is the arena’s own reference loop.
Each product’s context policy is a behavioural shape: grow, window, or summarise at a threshold. It is replayed on the same transcript, and it is not that product’s source code. Where the shape is generous to a competitor, the arena says so.
Every capability carries its tier and a citation. Several rows were corrected upward for competitors against this project’s own older marketing, and an unknown is recorded as unknown, never scored as zero.
| Harness | Context policy | Evidence quality |
|---|---|---|
| DulusMeasured | Lookback window and two-layer compaction | Evidence quality 92% |
| Claude CodeModelled | Full context, late auto-compact | Evidence quality 40% |
| Codex CLIModelled | Full context, late compaction | Evidence quality 37% |
| CursorModelled | Editor-scoped context, mid-window summarisation | Evidence quality 41% |
| ClineModelled | Visible context bar, truncate on pressure | Evidence quality 40% |
| AiderModelled | Repo map and full chat history | Evidence quality 44% |
| GitHub Copilot agent modeModelled | Bounded turn window, retrieval-assembled | Evidence quality 43% |
| Naive loopMeasured | Full transcript, no cache, no trimming | Evidence quality 100% |
Evidence quality is the mean strength of the evidence behind a harness’s rows, from measured at the top to unknown at zero. It is never scored; it tells you how much to trust the rest.
The first five run on claude-sonnet-4-6 with a 200k window; the last on deepseek-chat, with 64k.
compaction.estimate_tokens, an estimate: characters ÷ 2.8, plus 4 per message, plus 10%.providers.calc_cost, with the cache-read discount.The full dossier
28 pages: the method, all twelve charts, the 42 matchups with their intervals, the full capability record and the caveats.
PDF · 28 pages · 1.75 MB
Get the Windows app. The lookback window, the cache gate, Loopback, the Python console and MemPalace come with it.