The usual monthly report puts a traffic line next to a list of activities and lets the client draw the arrow between them. When traffic is up, that works. When traffic is flat, the same report becomes evidence against you, and you have no way of showing that the work was sound and the market was not.
The fix is to construct a comparison before you need one. This is ordinary experimental discipline, and organic search allows more of it than most people attempt.
The method
- Choose the metric before you change anythingA title rewrite should be judged on click-through rate. An internal linking change should be judged on impressions or position. Choosing afterwards means choosing whichever number moved, which is how honest people produce dishonest reports.
- Write down what would falsify it'If these pages still show the same defect after the next crawl, it did not ship.' A claim with no falsification condition cannot be checked by anyone, including you in three months.
- Freeze a comparison group at the moment of the changePages of similar depth and similar impression volume that the change does not touch. Choose them before the outcome is known and never adjust them afterwards.
- Wait for whole weeksSearch Console data settles over days and has a strong weekday rhythm. Four weeks is the shortest window that is mostly signal, and the fix has to have landed at the start of it.
- Compare the difference of differencesNot 'the pages went up 12%', but 'the changed pages went up 12% while comparable untouched pages went up 3%'. The second is a claim about your work. The first is a claim about the season.
What this rules out, and what it does not
It rules out the big confounders: seasonality, core updates, a competitor's outage, a brand campaign, anything that moves the whole site at once. Those move the comparison group too, and the comparison survives them.
It does not rule out a change that affects the comparison group as well. A sitewide template change has no clean control by definition, and the honest response is to say the fix cannot be measured this way rather than to pick a control that was affected too and pretend otherwise.
It also does not survive a small sample. Twelve pages against eight, on a site with a few hundred impressions a month, will produce a difference that is noise. Report that as inconclusive, which is a real answer and one clients respect more than a decorated guess.
Report the failures on purpose
The first instinct is to show the wins and quietly drop the fixes that did nothing. It is exactly backwards. A report where every item worked reads as marketing, and the client discounts all of it, including the parts that were real.
A report with three wins, one no-effect and one regression reads as a measurement. It also produces a better conversation, because 'this class of change does nothing on your site, so we are going to stop doing it' is the most valuable sentence in the document.
How many pages do I need for this to work?
Enough that both groups have meaningful impressions over 28 days. There is no fixed number, but a group of pages with fewer than a hundred impressions a month will not produce a difference you can defend.
What if the client ships several changes at once?
Where the affected pages overlap, the attribution is genuinely ambiguous and the honest answer is inconclusive. Where they do not, each change has its own comparison group and can be measured independently.
Can I do this without a tool?
Yes. It needs a record of when each change landed, a frozen list of comparison URLs, and Search Console exports for both groups over the same windows. The work is the bookkeeping, which is the part a tool removes.
Try it on your own site, free
No card, and the free tier does not expire — the first verdict takes about six weeks to arrive.