Six weeks of promoting an open-source tool, and why the record cannot say what worked
7 min readupdated 26 October 2026
The question every maintainer asks is "what worked?" We had six weeks of unusually complete records to answer it with, and the honest answer is that the records cannot.
digline is an open-source regression gate for LLM applications. Its maintainer, Alessandro Prandini, launched it on PyPI on 27 August and then spent six weeks telling people about it, mostly on Reddit. When he agreed to be an Orviqa case study, he handed us the whole record: a Reddit export, his working notes, LinkedIn and Hacker News results, and every release. We turned it into 253 dated rows, joined them to PyPI's release list and daily downloads, and looked.
This post is what that data shows. It is also, deliberately, a post about what it cannot show, because that is what decided how the real experiment is now being run.
Where six weeks of effort went
- Hacker News
- Repository
- Direct contact
Show the numbers
| Hacker News | Repository | Direct contact | |||
|---|---|---|---|---|---|
| 31 Aug | 11 | 0 | 0 | 0 | 0 |
| 7 Sep | 53 | 2 | 3 | 0 | 0 |
| 14 Sep | 42 | 3 | 3 | 0 | 0 |
| 21 Sep | 33 | 2 | 1 | 0 | 0 |
| 28 Sep | 30 | 1 | 1 | 0 | 0 |
| 5 Oct | 29 | 0 | 0 | 0 | 0 |
| 12 Oct | 15 | 0 | 0 | 12 | 10 |
Reddit was 213 of the 253 actions, and 200 of those were comments. The pace peaked in his first full week, at 58 actions, and fell to about 30 a week by October. The last week looks different because it is different: on 12 October he shipped a release, published to the official MCP registry, rewrote the repository's description and topics, and started writing to people directly.
One caution on this chart. The Reddit rows come from a complete export, while most of the rest come from his notes. Reddit is counted in full; the other channels can only be undercounted.
Most comments never mentioned digline
- Named and linked
- Named, no link
- Didn't name it
Show the numbers
| Named and linked | Named, no link | Didn't name it | |
|---|---|---|---|
| 31 Aug | 2 | 1 | 8 |
| 7 Sep | 5 | 3 | 43 |
| 14 Sep | 2 | 2 | 36 |
| 21 Sep | 5 | 3 | 21 |
| 28 Sep | 4 | 1 | 22 |
| 5 Oct | 2 | 9 | 16 |
| 12 Oct | 0 | 2 | 7 |
Of the 194 Reddit comments we could classify, 153 did not name digline at all. They were answers to other people's questions about evals, judges and CI. 41 named it, and only 20 of those carried a link.
He records naming and linking separately on purpose, and he was right to. A comment that names a tool asks a reader to go and search for it. A link asks for one click. A comment that names neither may still be the most useful thing he did, if it makes his name familiar in rooms where his users are, but nothing in the data can show that, and we will not pretend it does.
Posts: the visible results were small
Alongside the comments he wrote 13 Reddit posts, 8 Hacker News submissions and 8 LinkedIn posts. Where results were recorded, they were these:
| Where | What | Best result | Typical result |
|---|---|---|---|
| Hacker News, Show HN | 3 submissions | 1 point | 2 of 3 flagged |
| Hacker News, articles | 5 submissions | 6 points, 2 comments | 1 point, no comments |
| LinkedIn, personal profile | 4 posts | 735 impressions | 1 or 2 reactions |
| LinkedIn, company page (37 followers) | 4 posts | 377 impressions | no engagement from anyone outside |
| 13 posts | not recorded | one removed by moderators for the wrong flair |
The most visible result in the whole record was not on that list. One of his essays, "Bad evals, my own", appeared in Python Weekly ten days after the same essay scored one point on Hacker News. One room ignoring a piece of writing says little about the writing.
The download chart that looks like proof
This is the chart that tempts everyone, us included.
- Release day
- No release that day
Show the numbers
| Release day | No release that day | |
|---|---|---|
| 27 Aug | 298 | 0 |
| 28 Aug | 130 | 0 |
| 29 Aug | 0 | 38 |
| 30 Aug | 0 | 6 |
| 31 Aug | 116 | 0 |
| 1 Sep | 132 | 0 |
| 2 Sep | 94 | 0 |
| 3 Sep | 0 | 42 |
| 4 Sep | 0 | 26 |
| 5 Sep | 0 | 8 |
| 6 Sep | 0 | 5 |
| 7 Sep | 0 | 25 |
| 8 Sep | 144 | 0 |
| 9 Sep | 228 | 0 |
| 10 Sep | 639 | 0 |
| 11 Sep | 407 | 0 |
| 12 Sep | 0 | 31 |
| 13 Sep | 275 | 0 |
| 14 Sep | 0 | 43 |
| 15 Sep | 327 | 0 |
| 16 Sep | 268 | 0 |
| 17 Sep | 441 | 0 |
| 18 Sep | 495 | 0 |
| 19 Sep | 201 | 0 |
| 20 Sep | 284 | 0 |
| 21 Sep | 198 | 0 |
| 22 Sep | 200 | 0 |
| 23 Sep | 296 | 0 |
| 24 Sep | 376 | 0 |
| 25 Sep | 211 | 0 |
| 26 Sep | 0 | 47 |
| 27 Sep | 343 | 0 |
| 28 Sep | 224 | 0 |
| 29 Sep | 477 | 0 |
| 30 Sep | 773 | 0 |
| 1 Oct | 506 | 0 |
| 2 Oct | 610 | 0 |
| 3 Oct | 0 | 155 |
| 4 Oct | 0 | 169 |
| 5 Oct | 689 | 0 |
| 6 Oct | 0 | 91 |
| 7 Oct | 433 | 0 |
| 8 Oct | 0 | 71 |
| 9 Oct | 0 | 27 |
| 10 Oct | 0 | 71 |
| 11 Oct | 0 | 35 |
| 12 Oct | 370 | 0 |
| 13 Oct | 0 | 159 |
Show the numbers
| actions | |
|---|---|
| 27 Aug | 0 |
| 28 Aug | 0 |
| 29 Aug | 0 |
| 30 Aug | 0 |
| 31 Aug | 0 |
| 1 Sep | 0 |
| 2 Sep | 0 |
| 3 Sep | 0 |
| 4 Sep | 8 |
| 5 Sep | 0 |
| 6 Sep | 3 |
| 7 Sep | 9 |
| 8 Sep | 7 |
| 9 Sep | 14 |
| 10 Sep | 12 |
| 11 Sep | 11 |
| 12 Sep | 0 |
| 13 Sep | 5 |
| 14 Sep | 7 |
| 15 Sep | 12 |
| 16 Sep | 11 |
| 17 Sep | 2 |
| 18 Sep | 13 |
| 19 Sep | 3 |
| 20 Sep | 0 |
| 21 Sep | 2 |
| 22 Sep | 6 |
| 23 Sep | 8 |
| 24 Sep | 11 |
| 25 Sep | 7 |
| 26 Sep | 2 |
| 27 Sep | 1 |
| 28 Sep | 0 |
| 29 Sep | 8 |
| 30 Sep | 7 |
| 1 Oct | 4 |
| 2 Oct | 13 |
| 3 Oct | 0 |
| 4 Oct | 0 |
| 5 Oct | 8 |
| 6 Oct | 0 |
| 7 Oct | 5 |
| 8 Oct | 9 |
| 9 Oct | 7 |
| 10 Oct | 0 |
| 11 Oct | 0 |
| 12 Oct | 17 |
| 13 Oct | 9 |
Put the two side by side and the story writes itself: he promotes, downloads jump. In the 40 days from his first recorded action to the end of the data, 9,415 of the 10,378 downloads, 91 percent, landed on release days. Now look at what a release does on its own.
Show the numbers
| downloads (median) | |
|---|---|
| Release day (25 days) | 343 |
| Day after (7 days) | 71 |
| Two days after (3 days) | 27 |
| Three days after (2 days) | 39.5 |
| Four or more days after (3 days) | 25 |
On a release day the median was 343 downloads. The next day it was 71, and by the second day it was back to the floor. A new version on PyPI is fetched by dependency bots, by vulnerability scanners and by every CI job that installs the latest version. Some of those downloads are people. The data has no way to say how many.
Why nothing here can be attributed
The problem is not the amount of data. It is how the days fall.
Show the numbers
| days | |
|---|---|
| Release and promotion | 23 |
| Promotion only | 8 |
| Release only | 2 |
| Neither | 7 |
He released on 25 of the 40 days and promoted on 31, and 23 days had both. These are releases of the main digline package only; the pytest plugin, the MCP server and the provider packages are versioned and released separately, and counting them would only add release days. Only 7 days had neither, and only 3 days were more than three days from a release, too few to say what an ordinary day looks like. Every effect anyone might claim is tangled with every other: a comment posted on a release day, a release announced in a comment, a post written because a release had something to show.
We learned this the hard way. Early on we fitted a regression to these downloads and reported that releases multiplied them six to eight times. He checked it and found the comparison group was three days, two of them in the quiet week before his first post. We withdrew the figure. What survives is the shape above: downloads spike on release days and are gone within a day. Their size, and anything about promotion, is not there.
Stars do not help either. GitHub no longer gives a timestamp for a star, so a star can only be dated to the interval in which it was noticed.
What we are doing instead
The case study's rules were set before any of this was analysed, and this data is why they are strict:
- Everything above is a record of shape, never evidence. It tells us where he spends his time and which rooms he knows. It does not tell us what worked.
- The signal is arrivals through tagged links, counted by his site analytics, not stars or downloads. His links were already tagged, but with a parameter his analytics treats as a source rather than a campaign, so six weeks of tags recorded nothing beyond the referrer. Links pasted into Slack, DMs and email arrive with no referrer at all, and that week 39 percent of his visits landed as "unknown".
- Two weeks of calibration, then a fixed baseline, so the normal day-to-day movement is measured before anything is changed.
- One intervention per window, with a quiet window after it, and both of us writing down our predictions, hashing them and exchanging the hashes before each window opens.
At his volume, about 30 documentation pageviews a week, no intervention will show up as a rate. What can show up is a rare event with a date and a tag on it: a visit that arrived through a link placed in one room on one day. That is a small claim. It is the first one the data will be able to support.
If you are reading your own numbers
If you promote a project and want to know what worked, this is what his record taught us:
- Separate releases from promotion, or at least record exactly when each happened. If every post goes out on release day, downloads cannot tell them apart.
- Do not read downloads as people. A release-day spike is mostly machines. Look at the days after.
- Tag every link you place, per room, with the parameter your analytics actually reads as a campaign, and check that a test click shows up before relying on it.
- Record naming and linking separately. They are different requests to a reader.
- Write down what you expect before you look. It is the only way to stop the data agreeing with whatever you already believed.
When the measured half produces a result, it gets its own post, whichever way it falls.