Incrementality Testing in Practice: From Investment Case to Proof
Every mature performance marketing program eventually reaches the same point: the obvious account optimizations have already been made, but the business still needs to decide where the next marketing dollar should go.
Dashboards explain what happened. They are much less useful when several opportunities are competing for the same incremental budget and the team needs to know which one will create new value rather than claim credit for demand that already exists.
A mature ecommerce brand in Japan faced exactly that decision. The account was already performing efficiently. The job was to find an opportunity that could still produce incremental growth, build a credible investment case around it, and test whether the recommendation deserved funding.
The opportunity looked strong, but incrementality was uncertain
The client’s portfolio already had high ROAS, stable conversion volume, and strong brand awareness. Brand Search looked especially attractive: performance was efficient and impression share was only 66%, leaving 34% of available impressions uncaptured.
That headroom did not settle the decision. Brand Search has a familiar cannibalization problem: would additional paid clicks create new business, or capture customers who would have converted organically?
A high reported ROAS could not answer that. The opportunity had to be weighed against other uses of the budget and against the risk that paid traffic would substitute for organic demand.
Figure 1 identifies the opportunity. The rest of the analysis determines whether that opportunity is worth funding.
Build the investment case around what could make it wrong
The MAI team first worked with the business owner to define how much to spend, what impact to expect, and which assumptions could invalidate the case.
MAI then evaluated Brand Search against other uses of the budget, considering existing efficiency, available auction headroom, expected marginal return, and the possibility that paid clicks would substitute for organic demand.
The proposed spend was sized conservatively. The economics were tested using an 80% organic-substitution assumption—an unfavorable scenario designed to show whether the investment could remain attractive even if the central uncertainty moved against it.
The conservative proposal
| Daily spend increase | Expected daily GMV increase | Expected marginal ROAS |
|---|---|---|
| +¥18.7K | +¥32K | ≈2 |
The objective was not the largest forecast. It was a proposal that remained credible under a deliberately conservative assumption.
Validate the setup before testing the investment
The simulated economics were compelling, but they were not treated as proof. A useful proposal makes its assumptions visible and testable.
Before increasing Brand Search budgets nationwide, MAI designed a geo-lift incrementality test using comparable treatment and holdout regions in Japan. An A/A test came first to check that the two groups behaved similarly before the intervention.
The experiment then asked one clear question: would the additional Brand Search investment increase total GMV beyond what would have happened anyway?
Without a credible holdout and pre-test validation, a strong post-investment result could still reflect underlying regional differences rather than the budget change.
The experiment changed the funding decision
The observed results were materially stronger than the conservative case. The experiment ultimately deployed a larger spend increase than the initial proposal, which let the team see whether the opportunity continued to hold at greater scale.
Proposal versus observed reality
| Metric | Proposal | Reality |
|---|---|---|
| Daily spend increase | ¥18.7K | ¥49.6K |
| Daily GMV increase | ¥32K | ¥301K |
| Marginal ROAS | ≈2 | ≈6 |
GMV increased much faster than spend. In this case, that moved Brand Search from a plausible opportunity to a high-confidence place for additional budget.
A dashboard, a forecast, a proposal, and an experiment do different jobs
Every marketing team has more ideas than budget. It helps to separate four jobs that often get blurred together:
- Dashboard: explain what has already happened.
- Forecast: estimate what might happen under a different level of investment.
- Proposal: specify the action, quantify the investment and expected impact, and expose the assumptions that could make the case wrong.
- Experiment: determine whether the business should act and how confident it should be.
The proposal determines what is worth testing. The experiment determines whether the recommendation deserves funding.
From recommendation to investment decision
In this case, the next dollar belonged in Brand Search. That does not mean Brand Search always deserves more budget. The useful lesson is the decision process: quantify the investment case, make the uncertain assumption explicit, and use incrementality testing to see whether the growth is actually new.
That sequence gives the business a cleaner way to fund growth: identify the opportunity, make the assumptions visible, test the uncertain part, and change the budget decision when the evidence supports it.