Our tool blamed the README on every popular project. That was noise
13 min readupdated 4 November 2026
Run Orviqa on a repo with healthy adoption. If it still names the README as the blocker there, maintainers will treat that finding as noise.
That challenge was a fair one. Orviqa reads a GitHub repository and tells its maintainer, in one line, what is most limiting the project's adoption. In almost every report we had seen, the answer was the README. That is either a real pattern in open source, where weak documentation genuinely is the common barrier, or a habit of the tool. From inside a handful of reports you cannot tell which.
So we tested it, wrote the test down before running it, and are publishing the result, which is that the challenge was right. Then we found the cause, changed it, and tested again. This is the whole story, including the part of the fix that fell short.
The test
We fixed the design before running anything and committed it, so it could not be changed after the fact, and quote its fingerprint here so the design can be checked against what was done: the pre-registration's sha256 is 9532c519557ae9331f8cf7f2dfa972eade0c4012c5546a13d790fb99f3a817e2.
- Twenty repositories. From our 2,069-repository README benchmark, ten with healthy adoption (5,000 stars or more) and ten small ones (10 to 200 stars), one per language in each group, drawn with a fixed random seed. All public, licensed, in English, pushed to in the last 90 days, and software rather than lists or courses.
- Two full analyses each, fresh, with Orviqa exactly as it was in production, so we could also see whether a result repeated: 40 analyses in all.
- One question per analysis. Does the "biggest constraint" it names point at the README or documentation, the product, awareness, something else, or nothing at all? Another model sorted each answer into those five classes without seeing the project's name, size or group, one answer at a time so none was judged against another. We then checked all 40 by hand against the unedited text.
- Three thresholds for "noise", set in advance. Any one of them would mean the finding was driven by the prompt rather than the project: five or more healthy projects named README in both runs; any healthy project with a README already above the 75th percentile still named README; or the healthy group's README share within 20 points of the small group's.
The point of writing all of this down first is that it takes the result out of our hands. We could not move the goalposts after seeing the numbers, because the goalposts were already committed to git.
The result
All three thresholds were met.
Show the numbers
| % of analyses | |
|---|---|
| Healthy adoption | 100 |
| Small | 90 |
Every healthy project got the README as its blocker, both times: Whisper, Ghidra, Puppeteer, lazygit, jq, Spree, Sway, Koel, UniGetUI and Addy Osmani's agent-skills. Six of them have READMEs that our own grader places above the 75th percentile, two at the 98th. The healthy group's README share, 100%, sat within ten points of the small group's 90%, well inside the threshold we had set.
The most revealing number is not in that chart. It is the disagreement inside a single analysis. The same pipeline that named the README as the blocker also audits the README, and that audit found the popular projects' READMEs in distinctly better shape than the small projects'.
Show the numbers
| findings per analysis | |
|---|---|
| Healthy adoption | 0.15 |
| Small | 0.85 |
The popular projects averaged 0.15 critical README findings per analysis against 0.85 for the small ones. One part of the tool had looked at these READMEs and found little wrong with them. The part that writes the headline said the README was the problem regardless.
Here is what that looks like up close. The pre-registration told us to show the healthy and small repositories whose READMEs grade most alike, so the comparison could not be picked for effect. That pair is below, both at the 95th percentile.
addyosmani/agent-skills (healthy, 113,000 stars, README at the 95th percentile). What Orviqa named as the blocker:
"The IMPORTANT README-audit finding — no examples directory showing generated SPEC.md, plan.md, or review output — means a developer has to install the pack and run it themselves to find out what '/spec, /build, /ship' actually look like in practice. Every other documentation gap is cosmetic by comparison; this one blocks the core trust claim."
torrust/torrust-index (small, 97 stars, README at the 95th percentile). What Orviqa named as the blocker:
"Arriving developers hit two walls before they can even evaluate the project: the README's self-description as 'a library' contradicts the Axum server in src/main.rs, and there is no example of calling the v1 HTTP API. [...] This is not a promotion problem; the documented path from clone to working instance isn't fully true yet."
Two READMEs our own grader scores the same, one on a project with 113,000 stars and one with 97, and the tool told each of them the same thing. A maintainer of the first would be right to read that and close the tab.
Why it happened
The cause was in two places in the prompt, and neither was subtle once we looked.
- The strategist was never told how widely a project is used. It saw the project's description, audiences, README findings, health checks and open issues, but not its stars, releases or contributors. It could not tell lazygit, with 83,000 stars, from a project with 50.
- It was told the answer before it read anything. The prompt said the blocker is "usually not reach" but developers who "cannot get to a working example", a rule said "documentation work still comes first", and the output field itself was described as "often documentation, not promotion".
We wrote those lines from experience with small projects, where they are often true. Given no evidence to weigh against them, the model applied them to everything. The failure was not a weak model. It was a prompt that handed the model a prior, true for small projects and false for large ones, and then withheld the one fact, how adopted the project already is, that would have let the model disagree.
What changed
The cause was a prompt, so the fix was one too. Version 6 of the strategist does two things.
It now gets an adoption block, counted from GitHub: stars, forks, contributors, releases and the project's age. It can finally tell lazygit from a project with fifty stars. And it is no longer handed the answer. The lines that said the blocker is "usually not reach", that "documentation work still comes first", and that the field is "often documentation, not promotion" are gone. The first step now asks what most limits this project's adoption, decided from its own evidence, and a README finding is named only when it is what most limits the project, weighed by its severity. Crucially, it is still allowed to name the README, and it is never told that popular projects have good READMEs. A fix that simply ruled the README out would have moved the bias, not removed it.
We wrote this change, and what would count as fixing it, down before running anything. The follow-up pre-registration's sha256 is 1419fd4411016d14459207a2ed5ad9800cbcfd0bef253e32e9152314cf2e874c. Then we ran the same 20 repositories twice each, fresh, classified them the same blind way, and checked all 40 by hand.
It mostly worked. The share of popular projects told the README was their blocker fell by half.
- Before (version 5)
- After (version 6)
Show the numbers
| Before (version 5) | After (version 6) | |
|---|---|---|
| Healthy adoption | 100 | 50 |
| Small | 90 | 65 |
The more stubborn form of the bias is when both runs agree on the README. That is the version a maintainer would actually experience, and it moved the furthest.
- Before (version 5)
- After (version 6)
Show the numbers
| Before (version 5) | After (version 6) | |
|---|---|---|
| Healthy adoption | 10 | 3 |
| Small | 8 | 6 |
The whole picture, project by project, shows where the answers moved. Each cell is the constraint class for the two runs.
| Group | Repository | Stars | README %ile | Before (v5) | After (v6) |
|---|---|---|---|---|---|
| healthy | addyosmani/agent-skills | 113,219 | 95 | README, README | product, product |
| healthy | openai/whisper | 111,945 | 71 | README, README | other, README |
| healthy | NationalSecurityAgency/ghidra | 97,769 | 57 | README, README | product, README |
| healthy | puppeteer/puppeteer | 95,797 | 81 | README, README | product, product |
| healthy | jesseduffield/lazygit | 83,648 | 98 | README, README | product, README |
| healthy | FuelLabs/sway | 61,354 | 71 | README, README | product, README |
| healthy | jqlang/jq | 36,032 | 95 | README, README | README, README |
| healthy | Devolutions/UniGetUI | 26,852 | 90 | README, README | product, product |
| healthy | koel/koel | 17,333 | 40 | README, README | README, README |
| healthy | spree/spree | 15,809 | 98 | README, README | README, README |
| small | wobsoriano/solid-sonner | 101 | 57 | README, README | README, README |
| small | microsoft/homebrew-mssql-release | 101 | 13 | README, product | README, product |
| small | vinicioslc/tinfoil-hat | 100 | 57 | README, README | product, product |
| small | nidhugg/nidhugg | 100 | 71 | README, README | README, README |
| small | auth0/auth0-oidc-client-net | 100 | 71 | README, README | README, README |
| small | checkmarx-ltd/cx-flow | 99 | 81 | README, README | README, README |
| small | torrust/torrust-index | 97 | 95 | README, product | product, product |
| small | jpush/jcore-react-native | 50 | 25 | README, README | product, product |
| small | kube-logging/telemetry-controller | 50 | 99 | README, README | README, README |
| small | cakephp/cache | 50 | 25 | README, README | README, README |
Read down the two right-hand columns and the change is plain. Popular projects told the README was their blocker in both runs fell from ten of ten to three: jq, koel and spree. Of the six popular projects whose READMEs are above the 75th percentile, four now get the product named instead; only jq and spree still get the README both times. And the fix did not overcorrect: small projects, whose READMEs carry most of the real faults, still get the README named 65% of the time. The tool did not simply learn to stop saying the word.
What it did not do was separate the two groups by the margin we had required. We set, in advance, that the small group's README share should sit at least 20 points above the healthy group's. It came in at 15, 65% against 50%. By the letter of our own test, that is a substantial fix and not a complete one, and we would rather say so than round it up.
There is a quieter result in the table that we are glad to see. On four popular projects, Whisper, Ghidra, lazygit and Sway, the two runs now disagree with each other, splitting between the README and the product. Our repeatability on the healthy group fell from ten of ten to six of ten because of it. That looks like a regression and is the opposite. The strategist is now weighing adoption against README severity on cases where the honest answer is genuinely close, and landing differently when the evidence is a coin toss, instead of reciting the same prior every time.
What else the test turned up
- Some projects cannot be analysed at all. The model declined PEASS-ng, a privilege-escalation toolkit used in security testing, twice, as cybersecurity-sensitive, and accepted it on a third attempt. By our pre-registered rule it was replaced by the next repository in the draw, UniGetUI. A maintainer of a project like it would currently get a generic "problem on our side" message, which is its own thing to fix.
- The small projects that changed their answer changed it for a good reason. The Homebrew tap and torrust-index both moved, between runs, from the README to an open bug that breaks installation. That is exactly the judgement we want, and it appeared where the evidence was loud enough to earn it.
Why this is really about trust
The question underneath the whole test was whether a maintainer could trust what Orviqa tells them. The honest answer is not that they cannot. It is narrower and more useful: the tool had a specific, trust-destroying habit, aimed squarely at the people most worth convincing, and that habit is now largely gone.
It mattered because of who it hit. Telling the maintainer of a top-quartile README to go fix their README is the fastest way to be dismissed, and the old version did exactly that to six of ten popular projects. After the change, two remain, jq and spree, so for those maintainers the risk is not fully removed, and we are not going to pretend otherwise.
The broader point is about how a tool like this earns trust at all. Not by claiming to be right. A product that runs on a language model will be wrong sometimes, and saying otherwise is the actual way to lose people. It earns trust by showing, in public and in full, how it checks whether it is wrong: a sample drawn to a fixed rule so we cannot cherry-pick, a failure condition written down before the data, blind classification, and a fix reported with the five points it did not close still attached. The test that produced this post is cheap and repeatable, and we would rather run it and publish the awkward half of the result than wait to be caught by someone running it for us.
Limits
Ten projects per group shows a pattern, not a precise rate. Stars are a rough proxy for adoption. Our benchmark takes the most-starred repositories in each star band, so "small" here means the upper edge of small. Some answers carry details that could hint at the project even with its name removed, which is a limit on how blind the blind classification really is. And these repositories belong to people who did not ask to be studied: the analyses read only public data, nothing was posted or sent to them, and this is about Orviqa's behaviour, not their work.
We keep the full record on file: both pre-registrations, the draw, all 40 analyses in each round, the classifications, the hand checks, and the replacement with its reason. The first run cost $31.45 in inference and the follow-up $31.12.
This was a test we would not have thought to run on our own, and it found the thing most likely to make maintainers stop trusting the tool.