We gave one piece of accounting-firm work to an agent and ran it on three models under the same conditions: DeepSeek-V4.1-Flash-1017 on our own DGX Spark ×4 cluster, and Anthropic’s Claude Opus 5.5 and Sonnet 5.5. The models finished with the same score. What moved the score, the time and the amount of code was the tooling we handed them, not the model. This post records that relationship, and what it means for cost.
The task
Roll an audit report forward one year. The agent gets last year’s audit report (the DART filing file) and this year’s trial balance (Excel), and writes a draft of this year’s report: move the numbers, drop and add rows, fix amounts inside note paragraphs, rewrite the wording when the auditor changes, and foot every table.
This is work with an answer. The filed report exists, so every draft can be scored against it.
- One harness for every run: Claude Code headless (
claude -p), same prompt, same tools, same skill (procedure docs and CLIs). - DeepSeek runs on our cluster; Claude Code only points at a different API base URL.
- When something is ambiguous the procedure asks the engagement team. In the benchmark a scripted “team member” answers, with answers derived from the actually filed report.
- Scoring against the filed report: recall and precision of amounts in the four primary statements, recall and precision of all amounts, and cells left as “to confirm”.
Six runs, one score
Sample A is a private company we had not seen before, in the year its auditor changed, so the notes were heavily restructured.
| Run | Time (min) | Turns | Cost (USD) | Output tok/s(*1) | Code (lines) | Score: statements · all R / P | Footing checks |
|---|---|---|---|---|---|---|---|
| Sonnet medium | 5.7 | 35 | 1.31 | 113 | 85 | 0.96 / 1.00 · 0.83 / 0.80 | 227 |
| Sonnet high | 8.5 | 68 | 2.02 | 119 | 131 | same | 233 |
| Opus medium | 11.3(*2) | 59 | 3.40 | 81 | 81 | same | 240 |
| DeepSeek low | 15.0 | 80 | local | 81 | 152 | same | 202 |
| DeepSeek high | 20.7 | 136 | local | 82 | 317 | same | 208 |
| DeepSeek max | 30.3 | 150 | local | 82 | 154 | same | 219 |
(*1) Output tokens of each response divided by that response’s time, prefill estimate removed.
(*2) Opus delivered its report at 11.3 min, then sat 17 min waiting for a background sub-agent notification. That wait is a
harness artifact and is excluded.
All six runs score the same to three decimals: three models, three effort levels on DeepSeek. The remaining gap (0.83 recall) is information that is not in the trial balance at all, such as notes that exist only in the report and the new auditor’s wording. No model made those up.
The tools did the work
We built the skill over the same weeks, re-running the benchmark after each change. Most of the score now comes from the tools before any model touches the draft.
| What the tools do | Statements R / P | All amounts R / P |
|---|---|---|
| Nothing: last year’s report as is | 0.51 / 0.53 | 0.48 / 0.46 |
| Roll-forward tools alone, no model(*3) | 0.96 / 0.99 | 0.81 / 0.80 |
| Tools + any of the six runs above | 0.96 / 1.00 | 0.83 / 0.80 |
(*3) The skill’s roll command run on its own: it rolls the periods, matches each row of last year’s tables to the trial balance, and pre-fills this year’s values with references to the source cells. Measured on sample A with the skill version of that week.
The tools fill nearly every number. The model’s job shrank to the leftovers: rows that disappeared or were renamed, amounts inside note paragraphs, the new auditor’s wording, and questions for the team. That is judgment, but small judgment with an answer, and every model we tried handles it.
Each tool also changed how the models worked:
- Pre-filled values removed the table-to-sheet pairing that used to take two to four minutes of thinking in a single turn. Over these tool changes Sonnet high went from 23 to 8.5 minutes and DeepSeek low from 29 to 15, at the same score.
- A one-screen summary of what is left to check cut DeepSeek’s turns by 35% and its broken totals from 3 to 0. Without it, DeepSeek re-dumped the whole document every round.
- A channel to ask the team stopped DeepSeek from filling unknowns with plausible values. Before the channel it left 4 cells “to confirm”; with it, 44, the same as Claude.
- One line in the procedure changed DeepSeek from typing trial-balance numbers into JSON by hand to writing a generator script. It reads instructions literally, so the wording is part of the tooling.
Where the tools stop, the model shows
Across the other tasks we ran, the models differ in one place only.
| Task | Tools cover | Result |
|---|---|---|
| Trial balance roll-forward, same auditor (sample B) | most | all three equal (cells off from the filing: 65 · 66 · 68) |
| Unseen trial balance, auditor change (sample A) | most | equal across more than ten runs (statements 0.96 / 0.98–1.00) |
| Half-year REIT (sample C) | most | statements equal; DeepSeek left more “to confirm” (65 vs 48) |
| Report split into an Excel workbook → report (sample D) | little(*4) | Opus 23 · DeepSeek 42 · Sonnet 51 cells off |
(*4) An early run, one per model, at different effort levels (Opus medium, Sonnet high, DeepSeek low). At the time there was no tool for pairing the report’s tables with the workbook’s blocks; each model wrote its own, DeepSeek in 1,472 lines.
Sample D is the case where we had not yet built the tool. The models had to invent the pairing themselves, and the scores spread. That is not an argument for a bigger model so much as a map of where the next tool should go.
The price
On work like this the cheapest model is enough, so price decides. Our cluster decodes one stream at about 81 tok/s; on OpenRouter the best provider serves the same model at about 137 tok/s (P50)(*5). We re-timed each DeepSeek run at 137 tok/s(*6) and priced its tokens at OpenRouter rates.
| Run | Output tok/s | Time (min) | Cost per report (USD) |
|---|---|---|---|
| Sonnet medium | 113 | 5.7 | 1.31 |
| Sonnet high | 119 | 8.5 | 2.02 |
| Opus medium | 81 | 11.3 | 3.40 |
| DeepSeek low at 137 tok/s | 137 | 11.5 (measured 15.0) | about 0.18(*7) |
| DeepSeek high at 137 tok/s | 137 | 16.9 (measured 20.7) | about 0.26 |
| DeepSeek max at 137 tok/s | 137 | 22.6 (measured 30.3) | about 0.38 |
| DeepSeek on our cluster | 81 | 15.0 | electricity only |
(*5) OpenRouter model page, October 2026. Throughput varies by provider and by hour; we did not run the benchmark there.
(*6) Each response becomes (output tokens ÷ 137) + 1.16 s first-token latency; tool time and waits stay as measured. The
longest footing sub-agent, which the main agent waits for, is scaled the same way. At 81 tok/s and zero latency the method gives
back the measured times (15.2 · 21.0 · 30.2 min).
(*7) Tokens from these runs at the listed rate of USD 0.30/M input, USD 1.20/M output and USD 0.006/M cache read. Without cache hits the
low run would cost about USD 2.50; cache reads are 97% of its input.
The same report costs about USD 0.18 on hosted DeepSeek against USD 1.31 on Sonnet and USD 3.40 on Opus, seven to nineteen times less, in about the time Opus takes. On our own cluster the marginal cost is electricity. DeepSeek is slower mainly because it takes more turns (80 against Sonnet medium’s 35), and every turn pays latency and a re-read of the conversation; better tools have been cutting that number too.
How each model works
The score was the same; the style was not. All runs got the same instruction: ask one question at a time and apply each answer right away.
- Sonnet is practical. On medium it asked all 21 questions in two shell loops (10 seconds), applied them in five scripts, and noted in its report that it had not followed “one at a time”. The answers were mostly the defaults, so the result was the same.
- Opus asks, then checks again. It made up three questions that were not on the list (board approval date, related-party opening balance, communication table), and ran an extra footing round that found six cells where last year’s filed report itself does not add up.
- DeepSeek reads instructions literally and checks every item it is given. Clean tools help it most; noisy tool output costs it more than it costs Claude, because it verifies each item.
Takeaways
- Give the agent tools. Pre-filled values, a summary of what is left, and a channel to ask the team moved time, code and honesty more than any model change did.
- On work with an answer and tools that carry it, high intelligence is not needed. Tools alone reached 0.96 / 0.99 on the primary statements; all six model runs finished at 0.96 / 1.00.
- Where a model gap appears, it marks a missing tool.
- Then price decides: about USD 0.18 per report on hosted DeepSeek against USD 1.31–3.40 on Claude, or electricity on our own cluster.
Both the skill and the models are changing quickly; we will re-run this as they do.