The marketing world has spent far too long shackled by last-click attribution, a model that fundamentally misunderstands how consumers interact with brands. True incrementality testing offers a superior path, revealing the actual lift your campaigns provide and revolutionizing how we calculate app attribution and marketing ROI. But can you truly measure what would have happened anyway?
Key Takeaways
- Implement holdout groups of at least 10% of your target audience to accurately measure campaign incrementality.
- Focus on a single, measurable primary KPI like app installs or purchases for clear incrementality testing results.
- Utilize advanced measurement tools like SKAdNetwork 4.0 or Google’s Privacy Sandbox to overcome privacy-related attribution challenges.
- Allocate a minimum of 15% of your total campaign budget to incrementality testing to gain statistically significant insights.
- Prioritize incrementality over last-click for budget allocation, shifting funds to channels that demonstrate true uplift.
I’ve seen firsthand the damage last-click attribution can inflict. It’s a comfortable lie, easy to report but utterly misleading when it comes to understanding true business impact. We’ve all been there, celebrating a “conversion” that would have happened regardless of our spend. That’s not marketing; that’s just spending money. My firm, for example, recently worked with a major e-commerce client convinced their retargeting campaigns were their golden goose. Their last-click data showed phenomenal ROAS. I knew better. We needed to prove incrementality, or rather, the lack thereof, to reallocate their substantial budget more effectively.
The core problem with last-click is its inherent bias towards channels closest to conversion. It gives undue credit to the final touchpoint, ignoring the entire journey that led a user to that point. This leads to wildly inefficient budget allocation, where perfectly good money chases phantom returns. My philosophy is simple: if you can’t prove a channel drives incremental value, cut it. Or at least, reduce its share dramatically.
Our client, a popular fashion retailer, was running a complex digital strategy focused on driving app installs and in-app purchases. Their budget for Q1 2026 was a hefty $1.5 million, spread across Meta Ads, Google UAC, and a handful of affiliate networks. Their existing attribution model, reliant on a basic mobile measurement partner (MMP), reported a blended ROAS of 2.8x. Sounds good, right? Not good enough if a significant portion of those conversions were organic or driven by other, unmeasured factors.
“With U.S. organic search traffic falling 2.5% year-over-year in January 2026 and AI referral traffic to retail sites surging 693% over the same period, a real shift in where buyers begin their research is clearly happening.”
Campaign Teardown: Proving Incremental Value for a Fashion Retailer
We designed an incrementality testing framework to challenge their assumptions. The primary goal was to identify which channels truly drove new app installs and subsequent first-time purchases, not just claimed credit for them. This wasn’t about optimizing existing campaigns; it was about fundamentally rethinking their investment strategy.
Strategy & Methodology: The Holdout Group Approach
Our strategy centered on rigorously defined holdout groups. For each major channel (Meta Ads and Google UAC, as affiliate networks presented unique challenges for this specific test), we segmented a statistically significant portion of their target audience that would NOT be exposed to specific campaign elements. This is where the rubber meets the road. You need courage to intentionally withhold ads from potential customers, but it’s the only way to get a clean read.
We established two main test cells for each platform:
- Control Group (15%): Users in this segment were explicitly excluded from seeing ANY paid ads from the specific channel being tested. They still saw organic content and ads from other channels.
- Test Group (85%): Users in this segment were exposed to the full campaign as usual.
The incrementality was then calculated by comparing the conversion rate (app installs, first-time purchases) in the test group against the control group, factoring in the baseline conversion rate of the control. We used a randomized control trial (RCT) methodology, ensuring true randomness in group assignment to minimize bias. This was critical, and something many marketers skip, leading to flawed data.
Creative Approach & Targeting
The creatives were consistent with their existing high-performing assets: engaging video ads showcasing new collections, dynamic product carousels, and clean, aspirational static images. The targeting remained broad but refined, focusing on demographics and interests aligned with their core customer base (ages 25-45, fashion enthusiasts, online shoppers). We didn’t change the creative or targeting for the test; we isolated the exposure to the ads themselves.
Key Performance Indicators (KPIs)
- Primary KPI: First-time app installs (new users only).
- Secondary KPI: First-time in-app purchases within 7 days of install.
- Tertiary KPI: ROAS (Return on Ad Spend) for incremental purchases.
Campaign Data & Analysis (Q1 2026)
Total Budget: $1,500,000
Duration: January 1, 2026, March 31, 2026 (12 weeks)
Meta Ads (Focus: App Installs)
- Budget Allocated: $750,000
- Impressions (Test Group): 150,000,000
- CTR (Test Group): 1.8%
- Total App Installs (Test Group): 45,000
- Cost Per Install (Test Group): $16.67
- Total App Installs (Control Group – Baseline): 2,500 (from organic and other channels)
- Incremental App Installs: 45,000 – (2,500 / 0.15 * 0.85) = 45,000 – 14,167 = 30,833
- Incremental Cost Per Install: $750,000 / 30,833 = $24.32
Google UAC (Focus: App Installs & In-App Purchases)
- Budget Allocated: $500,000
- Impressions (Test Group): 80,000,000
- CTR (Test Group): 1.2%
- Total App Installs (Test Group): 20,000
- Cost Per Install (Test Group): $25.00
- Total App Installs (Control Group – Baseline): 1,500
- Incremental App Installs: 20,000 – (1,500 / 0.15 * 0.85) = 20,000 – 8,500 = 11,500
- Incremental Cost Per Install: $500,000 / 11,500 = $43.48
Affiliate Networks (Budget Allocated: $250,000) showed a high volume of installs but proved incredibly difficult to isolate for true incrementality due to their nature. We suspected significant overlap with organic and other paid channels, a common pitfall. This is why you can’t just blindly trust affiliate reports; the incentives are often misaligned with true incremental growth.
What Worked: The Power of Data
The incrementality test delivered a stark, undeniable truth: Meta Ads, while still a strong performer, had a higher incremental CPI than its last-click reports suggested. Google UAC, on the other hand, was significantly less incremental than previously believed. Its reported last-click CPI was $25, but its true incremental CPI was almost double that. This means a substantial portion of the installs Google UAC was “claiming” would have happened anyway, likely through organic search or brand recognition. This was the moment of truth for the client. The data cut through all the last-click noise.
My opinion? Google UAC is great for reach and volume, but its attribution can be notoriously aggressive. You have to be incredibly careful with it, especially if you have strong organic presence. We found its incremental value was heavily skewed by users who would have installed the app after a simple brand search, regardless of seeing a UAC ad.
What Didn’t Work: Over-reliance on Last-Click & Attribution Challenges
The biggest “didn’t work” was the client’s prior over-reliance on last-click attribution. It had blinded them to inefficiencies. Furthermore, navigating the evolving privacy landscape, particularly with Apple’s SKAdNetwork 4.0 and Google’s Privacy Sandbox Attribution Reporting API, added layers of complexity to granular user-level tracking. We had to adapt our data aggregation and modeling to respect these new privacy standards while still deriving meaningful insights. This isn’t just a challenge for 2026; it’s an ongoing battle for marketers.
Another challenge: convincing stakeholders to accept the “lower” ROAS numbers that true incrementality revealed. It’s a psychological hurdle. People love big numbers, even if they’re misleading. My argument was always, “Would you rather have a 5x ROAS on phantom conversions or a 2x ROAS on truly new, incremental business?” The answer should be obvious, but it often isn’t when budgets are involved.
Optimization Steps Taken
Based on our findings, we immediately implemented several optimization steps:
- Budget Reallocation: We shifted 30% of the budget from Google UAC to Meta Ads, recognizing Meta’s stronger incremental performance for app installs. We also allocated a small portion to experimental channels with high incremental potential, like influencer marketing, which we could also test for incrementality.
- Creative Refresh: We doubled down on Meta Ads creatives that drove high-intent installs, focusing on direct-response calls to action.
- Refined Targeting: For Google UAC, we narrowed targeting to focus on audiences with lower organic propensity to install, aiming to capture users less likely to convert without an ad touchpoint. This is a nuanced approach: you want to find segments where your ads truly make a difference.
- Continuous Testing: We established a rolling incrementality testing framework, dedicating 15% of the overall budget to ongoing holdout groups and A/B tests across various channels. This ensures we’re always learning and adapting, rather than making a one-off decision.
- MMP Integration: We worked closely with their MMP to integrate custom incrementality reporting dashboards, moving beyond basic last-click views. This involved custom post-backs and data warehousing solutions to combine impression-level data with conversion data from control groups.
The results of these optimizations were compelling. In Q2 2026, with a similar budget, the client saw a 15% increase in total incremental app installs and a 10% increase in incremental first-time purchases. Their overall blended incremental ROAS, while lower than the original last-click number, was now a true reflection of their ad spend’s impact, allowing for more strategic long-term planning.
One anecdote I often share: I had a client last year who was convinced their display ads were driving massive conversions. We set up an incrementality test, and the data showed that nearly 90% of those conversions were coming from users who would have converted anyway, often after searching directly for the brand. The display ads were essentially just “reminding” people who were already on their way to buy. We cut that display budget by 70% and reallocated it to prospecting video campaigns, which showed a much higher incremental lift. Their overall marketing efficiency skyrocketed, even if their “last-click ROAS” dropped. Sometimes, you have to take a step back to move two steps forward.
True app attribution and marketing ROI are not about claiming every conversion; they’re about understanding where your investment truly moves the needle. Incrementality testing is not just a methodology; it’s a mindset shift. It’s about moving from vanity metrics to genuine business impact. And honestly, it’s the only way to sleep at night knowing you’re not just burning through a budget without real returns.
Embrace incrementality, challenge your assumptions, and be prepared for uncomfortable truths. Your budget (and your boss) will thank you.
What is incrementality testing in marketing?
Incrementality testing is a scientific approach to marketing measurement that determines the true causal impact of a marketing campaign or channel. It involves comparing the behavior of a test group exposed to an ad to a control group that is not, measuring the additional conversions that would not have occurred without the ad exposure.
Why is last-click attribution insufficient for measuring marketing ROI?
Last-click attribution gives all credit for a conversion to the very last marketing touchpoint a customer interacted with. This often overstates the effectiveness of channels closer to conversion (like retargeting or branded search) and ignores the influence of earlier touchpoints, leading to inaccurate marketing ROI calculations and inefficient budget allocation.
How do holdout groups work in incrementality testing?
Holdout groups are segments of a target audience that are intentionally excluded from seeing a specific marketing campaign or ad. By comparing the conversion rates or other KPIs of this control group to a test group that was exposed to the campaign, marketers can isolate and measure the incremental lift attributable solely to that campaign.
What are the challenges of implementing incrementality testing?
Challenges include the need for statistical rigor, careful experimental design, obtaining sufficient sample sizes for holdout groups, and convincing stakeholders to allocate budget to “dark” campaigns. Evolving privacy regulations like SKAdNetwork also add complexity to tracking and attributing incremental conversions.
What tools or platforms support advanced app attribution and incrementality?
Many mobile measurement partners (MMPs) like AppsFlyer or Adjust now offer advanced features for incrementality measurement. Additionally, platforms like Google Ads and Meta Ads provide their own experimentation tools. For more sophisticated analysis, marketers often integrate data from these sources into data warehouses and use statistical modeling software.
