We turned the reasoning down, and two of our agents got better

Every step in our analysis ran at the highest reasoning setting, because more reasoning is better. We had never checked.

Orviqa reads a repository with a chain of language-model agents. One works out what the project is, one audits the README, one finds the audience, one writes positioning, and so on. Every one of them ran at the model's highest reasoning effort. Nobody had decided that. It was the default, and it sounded right.

Then users told us their tokens ran out fast, and we looked at where the tokens went.

The thinking you pay for and never see

A reasoning model thinks before it answers, and the thinking is billed as output. We compared what each agent was billed for with the size of the answer it actually returned. For the README auditor, about three quarters of the output tokens were reasoning the user never sees. For the discoverability audit it was nearer four fifths. Across one complete analysis of a medium-sized repository, hidden reasoning came to roughly a fifth of the whole bill.

That is not automatically waste. Thinking might be exactly what makes the answer good. So the question was not "is this expensive", which it plainly was, but "does it buy anything", which we could measure.

The first experiment: five agents, four repositories

We took four public repositories of different sizes and kinds: a small machine-learning project, a Python library, a Rust research crate and an LLM testing tool with a long release history. For each, we ran five of the analysis agents at high, medium and low effort on identical inputs. Where one agent feeds another, every downstream run received the same high-effort output, so each comparison changed exactly one setting. That made 60 runs.

Then a separate model, Opus, judged them. For each agent and repository it saw the project's context and the three outputs, shuffled and labelled A, B and C, and scored each for accuracy, specificity to that project and usefulness to its maintainer. It was told outright that length is not quality.

Agent	Output tokens, high / medium / low	Score, high / medium / low
Discoverability audit	14,862 / 7,774 / 4,618	6.0 / 7.5 / 6.75
Positioning	4,789 / 2,933 / 2,584	7.0 / 8.0 / 7.75
Content strategy	6,273 / 3,109 / 1,981	7.75 / 7.25 / 6.25
Audience research	3,824 / 2,418 / 2,350	8.5 / 7.25 / 6.75
README audit	7,282 / 2,680 / 1,557	8.75 / 7.25 / 6.0

Output tokens are averages over the four repositories, and scores are averages out of 10.

The settings did not rank the same way twice. For two agents, high was the best setting by a clear margin. For two others, medium scored higher than high while costing between half and two thirds as much.

The reasons were specific, which is what made them believable. At medium and low, the README auditor missed real defects, among them a casing error in a README example that would have left a "fixed" example still failing to compile. The audience step dropped whole personas when it thought less. The discoverability audit went the other way: at high it reasoned at length and then over-produced, and the judge preferred the tighter answer.

The second experiment: one maintainer's real history

The step that decides where a project should be promoted reads the maintainer's recorded history: what they posted, where, and what happened. One production run of it at high effort had used its entire output allowance thinking and stopped before finishing its answer. The retry succeeded, but the first attempt was paid for and thrown away.

So we ran it five times on that maintainer's real history: 238 recorded actions across seven communities the output was required to grade.

Effort	Output tokens	Ran out of room	Score
High	15,011, and 22,967 for the run that ran out	1 of 2	5.0
Medium	9,705 and 10,054	0 of 2	7.5
Low	7,477	0 of 1	6.0

The run that ran out also used its retry, which is why it went past the allowance. Only the high run that finished was judged. Each score is the average of two blind orderings.

High was not just the most expensive setting here. It was the least faithful. It marked Hacker News as allowing promotion, despite a history showing two flagged Show HN submissions, and it called direct promotion appropriate almost everywhere. Low went wrong in a different way: it kept restating counts that the prompt forbids it to write, because those numbers are computed in code and a model has misquoted them before. Medium kept closest to what the maintainer had actually recorded. Every run, at every setting, graded all seven required communities.

What we changed

Three agents stay at high: the README audit, audience research and content strategy. Their quality dropped below it. Three now run at medium: the discoverability audit and positioning from the first experiment, and community research from the second.

That takes roughly 9,000 tokens off every analysis and removes the most common cause of a step running out of room. Running both experiments cost about ten dollars in model calls.

What this does not show

Every score here is one model's judgement. We shuffled the outputs and hid which setting produced them, and the judge never saw the token counts. But a model judging another model's work can share its blind spots, and nobody rated these by hand. Treat the scores as consistent and cheap, not as ground truth.

We did measure the judge's own noise, after a reader asked the obvious question: how much does its score move when nothing changes? Given the same outputs five times in the same order, it never moved any output by more than one point, and often not at all. The gaps that decided our changes held in every one of the five calls. Run again through an independent harness six times on identical input, asking the judge each time for the probability that the shipped version was better, its average stayed between 0.64 and 0.70, though single answers ranged from 0.55 to 0.75. That is a consistent preference, but a moderate one.

What that does not cover is the rest of the noise. Both measurements are of the judge alone, scoring outputs that already existed: neither generated the agents' outputs again, so the variation a fresh run sees can only be larger. The samples are small: four repositories, one maintainer's history, two runs per setting in the second experiment. Order mattered in one format and not the other: with several outputs scored 1 to 10 in one prompt, shuffling them moved one output by a full point, but when the judge compared two answers and gave a probability, showing them in the opposite order moved its answer by about 0.03. We acted only where the gap was large and the judge's stated reasons were concrete enough to check, and every agent we moved keeps a retry at low effort in case it runs out of room.

And it is one model family on one kind of task. "Medium beats high" is not a rule. It is what these steps did on these inputs.

The point

The interesting result is not that less reasoning can be better. It is that the right setting differed from one step to the next, and the only way to find out was to run it. Two of our agents were paying for reasoning that made them worse. We had assumed otherwise since the first version, because "more thinking is better" is exactly the kind of claim that sounds too obvious to test.