Contents
Why a before-and-after increase is not proof
If organic traffic rises after a site change, the timing alone does not show that the change caused the increase. Seasonality, shifts in demand, search ranking changes, competitors, marketing activity, and tracking changes can also affect results. A before-and-after chart is useful for monitoring, but without a credible comparison it is descriptive evidence, not a causal estimate.
The relevant question is counterfactual: what would likely have happened to the same pages over the same period if the change had not been made? A test estimates that unobserved outcome by comparing treated pages or markets with a suitable control, or by modeling a baseline from unaffected data.
A control can absorb some shocks shared by both groups. It does not guarantee that treated and control pages respond identically to an algorithm update, demand change, or competitor action. The groups must be comparable, and the comparison must not be contaminated by the change being tested.
SEO page-split tests are not ordinary visitor A/B tests
An SEO page-split test usually assigns eligible pages or page clusters to treatment and control. It asks whether a change visible to search systems affected search outcomes. A visitor-level conversion-rate optimization (CRO) test asks a different question about user behavior. Neither design is automatically a substitute for the other.
Google advises against showing Googlebot a different version of a page from the one shown to users. Its guidance discusses canonical annotations for specified multiple-URL test situations and recommends a temporary 302 redirect rather than a 301 when users are temporarily redirected to a variation URL. These recommendations are not a mechanical recipe for every page-split design: preserve the intended URL and indexing signals for your implementation, and check Google’s guidance for the specific setup.
Group pages that compete for the same queries where possible. This can reduce the risk that treatment pages affect control pages through query cannibalization, but it cannot eliminate every form of spillover. Match page groups on page type and pre-test behavior, and randomize eligible groups when practical.
Write the experiment brief before implementation
From experiment brief to rollout decision
- 1Predefine the brief
Set the hypothesis, primary metric, guardrails, endpoint, and decision rule before results.
- 2Build comparable groups
Assign comparable pages or clusters and limit contamination between treatment and control.
- 3Launch and verify
Check deployed HTML, live URLs, and rendering without treating verification as proof of impact.
- 4Estimate effect
Compare with the counterfactual and report the estimate, interval type, and uncertainty.
- 5Apply the decision rule
Judge the worthwhile effect and guardrails; record uncertain evidence as inconclusive.
Agree on the plan before looking at results. A useful brief records:
- Hypothesis: the proposed change and the outcome it is expected to affect.
- Units and scope: eligible pages or page clusters, treatment assignment, and unchanged controls.
- Baseline and timing: the pre-test period, deployment date, and planned analysis endpoint.
- Primary metric: one measure aligned with the hypothesis.
- Guardrails: outcomes that must not worsen beyond an agreed limit, such as conversion rate.
- Decision rule: the minimum worthwhile effect, acceptable guardrail range, and what counts as a win, loss, or inconclusive result.
- Measurement details: reporting source, filters, attribution definition, and any reporting delay to allow for.
Set the endpoint and any interim analysis plan in advance. Repeatedly checking ordinary fixed-horizon significance tests and stopping at the first favorable result can raise false-positive risk. A fixed endpoint is a straightforward default; planned sequential designs need appropriate methods and stopping boundaries. Bayesian analysis does not make unrestricted checking automatically safe.
Statistical evidence and commercial importance are separate questions. A detectable effect may not be worth implementing, while a commercially meaningful effect may remain uncertain if the test is underpowered. Do not turn an inconclusive result into a win—or into proof that there was no effect.
Choose the strongest feasible comparison
Choose the strongest feasible comparison
| Method | Consider it when | Main limitation |
|---|---|---|
| Page-split test | There are comparable pages, stable page-level data, and only a subset can receive the change. | Page groups may respond differently to shared shocks or affect one another through query overlap. |
| Geo test | The change can be withheld in matched markets and those markets tracked similarly before launch. | Market differences and spillover can weaken the comparison. |
| Modeled baseline | A sitewide change has no holdout, but unaffected predictive data exist and historical placebo checks work. | The estimate depends on stable relationships and valid comparison data. |
| Before/after monitoring | There are few pages, no credible holdout, and no stable predictor series. | It is descriptive and cannot isolate the change from concurrent causes. |
The methods in the matrix are an editorial decision guide, not universal statistical rules. A page split is a reasonable candidate when there are enough comparable pages, stable page-level data, and the ability to change only a subset. A geo test may fit when changes can be withheld in matched markets, but check that those markets tracked similarly before launch.
For a sitewide change with no holdout, a modeled baseline may help only if suitable predictive data exist that the intervention cannot affect. Check the model against historical placebo dates. When the site has few pages, no credible holdout, and no stable predictor series, use pre/post monitoring as descriptive evidence and document other events that could explain the result.
SearchPilot has published around 1,000 organic visits or sessions per day to a tested section as its rule of thumb. That is a vendor guideline, not a universal minimum or a substitute for a power assessment. Feasibility also depends on the number and comparability of pages, outcome variability, effect worth detecting, and test duration. Ask an analyst to assess power or historical placebo performance rather than relying on a traffic cutoff alone.
Choose metrics and understand Search Console’s limits
Choose the primary metric, scope, date range, and reporting source before launch. Search Console clicks measure clicks from Google Search results; GA4 organic sessions measure visits attributed to organic search under GA4’s measurement and attribution rules. They describe different stages and may differ, so do not treat them as interchangeable.
Search Console omits anonymized queries from query tables and query-filtered data, even though chart totals can include them. Tables also have row limits, data can arrive with a delay, and Search Console’s time-zone handling may differ from analytics tools. Align dates and time zones, allow for reporting latency, and interpret query-filtered results in light of these limits.
Google’s branded-query filter is AI-assisted, available only to eligible top-level properties with sufficient volume, and may occasionally classify queries incorrectly. It is not equivalent to a regex-based query filter. Cross-check important brand segments where possible, and disclose which method you used.
Google says the Search generative AI performance report had rolled out worldwide as of August 31, 2026. It reports impressions from AI Overviews and AI Mode and provides page, country, date, and device dimensions. A property may not see the report if it has too few impressions or is excluded from those features. Treat these impressions as a supporting visibility measure—not as clicks, recommendations, conversions, revenue, or proof of business value.
Report the estimate, uncertainty, and business outcome
Report the estimated effect and its uncertainty, including the method and the type of interval. A frequentist 95% confidence interval describes a procedure with 95% long-run coverage under its assumptions. A Bayesian 95% credible interval assigns 95% posterior probability to a parameter range conditional on the model, data, and prior. These are not interchangeable interpretations.
Google CausalImpact reports posterior credible intervals and a one-sided tail-area probability for whether there is an effect. That probability alone does not tell you whether the effect is positive or negative. Report direction and magnitude from the estimate and interval as well. An interval that overlaps zero does not prove there is no effect; it means the data and model do not establish a directional effect at that interval level.
Search clicks, rankings, conversions, and revenue are different outcomes. If business impact matters, report conversion or revenue measures with their own definitions and uncertainty where the data support it. A search visibility change is not automatically a commercial gain.
A hypothetical store test—and an inconclusive decision
Consider an explicitly hypothetical WooCommerce test of revised product-page title templates. The hypothesis is that the change will increase non-branded Google Search clicks without materially harming conversion rate. Eligible product pages are grouped where they target overlapping queries; matched, unchanged clusters form the controls. The brief specifies the historical baseline, deployment date, Search Console clicks as the primary metric, and GA4 conversion rate as a guardrail. It also fixes the analysis date after allowing for crawl and reporting delays.
Suppose, for illustration only, treated pages receive 11,420 clicks and the modeled counterfactual is 10,000. The estimated uplift is (11,420 − 10,000) / 10,000 = 14.2%. Suppose the hypothetical 95% credible interval for clicks is +8% to +19%. Report that as a modeled estimate and interval, not simply as “14.2% growth.”
Now suppose the hypothetical relative conversion-rate effect is −0.5%, with an interval from −4.0% to +3.0%, and the pre-agreed unacceptable decline is −2%. The conversion guardrail does not rule out a decline beyond that limit. Under the stated decision rule, this is not a clear rollout win; record it as inconclusive for the rollout decision or follow the safety procedure set in advance. These illustrative numbers are not client results, evidence of revenue impact, or a prediction for other pages.
Verify implementation without confusing it with impact
Check deployed HTML and rendering, and inspect representative URLs. Google’s URL Inspection can show indexed-page information, test a live URL, and show Google’s rendered view. A successful deployment or crawl request does not prove that Google has processed every test page or that the change affected search performance.
Google generates title links automatically from multiple sources, so a changed title element may not appear verbatim in results. URL Inspection and rendering checks establish implementation details; the experiment’s outcome metrics estimate impact. Keep those questions separate, and remove temporary test infrastructure promptly when the test ends.