The Ad Report Shows 300 Sales. How Many Would Have Happened Anyway?

eBay once stopped buying paid-search ads in carefully selected U.S. markets and compared them with markets where the ads kept running. In a large-scale field experiment, researchers reported that brand-keyword ads produced no short-term benefit, while returns from other search terms were far below conventional estimates. New and infrequent customers responded more; many loyal buyers simply reached eBay another way.

The uncomfortable discovery was not that every search ad fails. It was that a dashboard can reward advertising for sales the business was already going to receive.

That distinction changes real budgets. An ad platform can correctly report that 300 buyers clicked or viewed a campaign before purchasing. It still cannot prove, from that sequence alone, that all 300 purchases were created by the campaign. Some people were already searching for the brand, returning from habit, responding to an email, or ready to buy. Incrementality asks the harder question: how many sales disappeared when comparable customers did not receive the advertising?

Attribution follows a path; incrementality tests a cause

Attribution assigns credit according to a rule: perhaps the last click, the first click, or an interaction inside a chosen window. It answers, “Which recorded touchpoint receives this conversion?” That is useful for operating campaigns, but it is not the same as measuring cause. Incrementality compares what happened with the campaign against a credible estimate of what would have happened without it.

Google describes Conversion Lift as a controlled experiment with a treatment group that can see ads and a control group held back from seeing them. The difference in downstream conversions is the lift caused by the advertising. That missing comparison is the heart of the method. Without it, high intent can masquerade as ad performance: the people most likely to click may also be the people most likely to purchase without persuasion.

Consider a clearly illustrative example. A coffee subscription campaign is credited with 300 orders. A randomized holdout shows that comparable customers who did not see the ads would have generated 240 orders anyway. The campaign did not create 300 orders; it created an estimated 60. The attributed cost per acquisition and the incremental cost per acquisition will tell very different stories.

eBay found the customer hiding inside the average

The eBay experiment matters because it separated customers by prior behavior. The researchers found that new and infrequent users were positively influenced by paid search, while existing loyal users accounted for much of the expense despite being largely unaffected. One average return had concealed two different jobs: informing people who did not know enough, and intercepting people already on their way.

The lesson is not “stop buying your brand name” or “search advertising never works.” eBay was an unusually familiar business, the experiment measured specific campaigns, and other brands face different competitors, organic visibility, and customer habits. The transferable lesson is to look for the segment where advertising changes behavior. A campaign can be weak on average and valuable for new buyers; it can also look efficient precisely because it captures existing demand.

Build the missing comparison before launch

Start with one decision. Decide whether you are testing a channel, an audience, an offer, or a creative idea. Then choose one business outcome—qualified leads, completed purchases, booked consultations, gross profit—not a bundle of convenient platform metrics. A clean question might be: “Does this prospecting campaign create first purchases from people who have not bought in twelve months?”

Next, choose the unit that can be separated fairly. Large platforms may randomize eligible users. Businesses with enough geographic spread may compare matched regions, which Google also supports in its geography-based Conversion Lift studies. A multi-location company might pair similar locations before exposing only one from each pair. The groups must be comparable before the campaign; choosing the strongest locations for treatment merely builds the desired answer into the test.

Finally, lock the major conditions long enough to learn. Keep the offer, landing experience, sales process, conversion definition, and measurement window stable. A creative A/B test can reveal which ad performs better among exposed people. Incrementality needs a no-ad comparison to reveal whether advertising created additional behavior at all. Those are different decisions and require different experiments.

Read the result like an owner

Suppose equally sized treatment and control groups produce 120 and 100 purchases. The estimated absolute lift is 20 purchases. If the campaign cost $2,000, the incremental cost per acquisition is $100. If those 20 purchases produced $4,000 in conversion value, incremental return on ad spend is 2.0. Google defines these metrics from the treatment-control difference, rather than from all conversions attributed to the campaign.

The number still needs uncertainty around it. Small effects can be difficult to distinguish from normal variation, and Google’s reporting includes intervals around lift estimates. An inconclusive test does not prove zero impact; it says the experiment did not measure the effect precisely enough. Before repeating or scaling, ask whether the sample, conversion volume, holdout size, and test duration were capable of detecting a commercially meaningful change.

Small businesses can ask the same question

Not every company can run a formal platform study; Google states that Conversion Lift is not available to every account. The discipline still applies. A smaller operator can test matched delivery areas, stagger a campaign across comparable locations, or reserve a randomized portion of an eligible customer list when the channel and consent rules allow it. The design should protect customers, avoid changing prices unfairly, and use a result large enough to influence a real budget decision.

Keep ordinary attribution for daily campaign management. Add incrementality when the decision is whether to increase, cut, or move spend. The most useful record connects the hypothesis, audience, channel, cost, holdout rule, outcome, and limitation. That is a natural OrionPilot task: coordinating the campaign and its evidence so content production and reporting answer the same business question.

A marketing report becomes more honest when it distinguishes sales that appeared near advertising from sales that were missing without it. The first number describes a path. The second earns the budget.