Independent Agency for CTV Incrementality Testing

Finding an independent agency for CTV incrementality testing has become a harder search than it should be, because nearly every media agency now claims to "prove incrementality" while very few actually run the test that word requires. For a CMO or growth lead spending $50,000 to $200,000 a month across CTV and programmatic, the distinction matters enormously: a platform telling you it drove incremental lift, using its own attribution and its own reported baseline, is not an independent measurement of anything. It is the platform grading its own homework. Independent, in the sense that actually protects a budget decision, means a party with no financial stake in which channel gets credit is running the test, controlling the holdout, and reporting the result — whether that result flatters the media plan or not.

What "Independent" Actually Means in Incrementality Testing

The word gets used loosely enough that it is worth defining precisely before evaluating anyone against it. An independent incrementality test has three characteristics that a platform-reported lift study does not.

• No financial interest in the outcome. The party designing and reading the test does not have its own media spend or platform relationship riding on the result showing a positive lift.
• Holdout control outside the platform's own systems. The exposed and holdout groups are defined and enforced by the agency or a third-party measurement partner, not by a platform's internal experimentation tool that also decides what counts as "exposed."
• A pre-registered methodology. The test design — market selection, holdout size, duration, success metric — is set before the campaign runs and reported regardless of outcome, rather than a post-hoc analysis run only when the numbers look favorable.

A platform's own conversion lift or brand lift study can be a useful input, but it fails all three tests: the platform has a direct interest in the result, it controls both the exposed and the measurement layer, and the methodology is rarely disclosed in enough detail to audit. That is the specific gap an independent, platform-agnostic measurement partner is meant to fill.

Why Platform-Reported Lift Isn't Incrementality

Platform-reported lift studies compare users who happened to see an ad to users who did not, using the platform's own audience assignment, and then attribute the difference in conversion rate to the ad. The problem is selection bias: the users a platform's delivery algorithm chooses to show an ad to are frequently already more likely to convert, because the algorithm is optimizing delivery toward exactly that population. A true holdout removes that bias by defining the control group independently of the platform's delivery decisions — typically at the geographic market level, where an entire region is withheld from exposure rather than individual users within a shared market, which also avoids the cross-contamination that happens when "exposed" and "control" users live in the same household or see the same organic and earned-media environment.

What a Geo-Holdout Test Actually Requires

A defensible geo-holdout test has specific mechanical requirements, and this is the part of the evaluation where most agencies claiming incrementality expertise fall short in practice.

• Matched market pairs. Exposed and holdout markets need to be statistically matched on population, historical sales trend, seasonality and existing media weight before the test starts, not selected for convenience.
• A sufficient holdout size. Typically 15 to 25 percent of comparable markets, large enough to detect the minimum lift that would justify the spend, calculated in advance rather than assumed.
• A clean media boundary. Other channels — paid search, paid social, email — need to run identically in exposed and holdout markets during the test, or their variation has to be explicitly controlled for, or the test measures channel interaction rather than CTV's own effect.
• First-party outcome data, not platform-reported conversions. The read should come from CRM, point-of-sale or e-commerce data matched back to exposed and holdout geographies, not from the ad platform's own pixel.
• A pre-set test duration tied to the actual sales cycle. A high-consideration purchase with a 30-to-90-day cycle needs a test window that runs at least that long past the media flight, or the read captures only the fastest-converting fraction of influenced buyers.
• Statistical significance reporting, including a null result if that's what the data shows. A credible measurement partner reports a "no detectable lift" result exactly as readily as a positive one.

Marketing Mix Modeling vs. Incrementality Testing: When to Use Each

These two methodologies get talked about as competitors when they are better understood as complementary tools that answer different questions on different timelines.

The practical answer for most brands evaluating a new channel or a specific agency claim is to start with geo-holdout testing, because it produces a fast, causal answer to a specific question, and to layer in marketing mix modeling once there is enough historical data across a mature, multi-channel program to make a model-based approach reliable.

What to Ask Before Hiring an Agency That Claims to "Prove Incrementality"

• "Who controls the holdout — you, or the platform?" If the answer is the platform's own experimentation tool, this is platform-reported lift with different branding, not an independent test.
• "What happens if the test shows no lift?" An agency that cannot describe how it would report and act on a null result is not running a pre-registered test; it is running an analysis designed to find a positive number.
• "What outcome data does the read use — your pixel, or my CRM/POS data?" First-party outcome data matched back to geography is the standard; platform-side conversion data is not.
• "How many markets, and how were exposed and holdout markets matched?" A specific, defensible matching methodology should be describable in one or two sentences; a vague answer is a signal the test wasn't designed rigorously.
• "Is there a third-party measurement partner involved, or is the agency both running the media and grading its own result?" Not disqualifying on its own, but it changes how much independent verification the result carries.
• "Can I see a redacted example of a past holdout test design and result, including one that didn't work?" An agency with real experience here should have at least one test that produced a negative or inconclusive result and can explain why.
• "What's the minimum spend and market count you need to run a test that would actually be readable for my business?" A confident, specific answer — not "it depends" with no follow-up — indicates the agency has actually sized tests before.

What Incrementality Testing Should Cost and How Long It Should Take

Measurement spend below roughly 2 percent of media spend at any tier is a warning sign that the test is being run as a checkbox rather than as a decision-grade instrument; below that level, agencies typically cut corners on market count or test duration in ways that quietly undermine statistical power.

Certifications and Third-Party Validation: What They Actually Verify

Badges like a Measured certified partner designation or similar third-party measurement certifications verify that an agency has demonstrated proficiency using a specific measurement platform's methodology and tooling — they are a meaningful signal of technical competence, but they are not, on their own, proof that a given agency runs independent tests for its own clients rather than platform-reported studies. The certification answers "can this agency operate this measurement tool correctly." It does not answer "does this agency structure holdouts that protect the client's interest over its own media commission," which is the question that actually matters and has to be answered through the due-diligence questions above, not the badge alone.

Common Mistakes When Evaluating Incrementality Claims

• Accepting a platform-reported lift study as "incrementality" without asking who controlled the holdout.
• Hiring based on a case study with no visible methodology, market count or test duration.
• Assuming a bigger agency brand name implies a more rigorous test design. Test rigor is a function of methodology and market count, not agency size.
• Skipping the "what if it shows no lift" question. This is the single fastest way to identify an agency that only reports favorable results.
• Treating MMM output as a substitute for testing a brand-new channel. MMM needs history a new channel doesn't have yet; it's the wrong tool for a first CTV test.
• Under-funding the measurement budget below the level needed for statistical power, then blaming the channel when the test is inconclusive.

A Due-Diligence Sequence for Evaluating Independent Incrementality Testing Partners

• Define the specific decision the test needs to inform — a new-channel go/no-go, a budget reallocation, or ongoing optimization — before evaluating any agency.
• Ask each candidate agency the seven due-diligence questions above and document the specific, non-generic answers.
• Request a redacted example of a past test, including its market count, duration, matching methodology and at least one result that wasn't uniformly positive.
• Confirm whether the read will use first-party CRM or POS data matched to geography, not platform-reported conversions.
• Size the test against your actual monthly spend using the cost and duration table above, and flag any proposal that comes in meaningfully under the recommended measurement investment.
• Confirm the test is pre-registered — methodology and success metric locked before launch — rather than designed after the campaign to explain results already in hand.
• Set expectations in writing for how a null or negative result will be reported and what decision it will trigger, before the test begins.
• Run the first test at the minimum viable scale from the table above, and treat it as a decision instrument, not a marketing claim to be repeated regardless of outcome.

Where Stillwater Media Fits

Stillwater Media runs geo-holdout incrementality testing as a standard part of every engagement, not as an add-on service — holdouts are built and controlled independently of the platforms we buy media on, read against first-party CRM and sales data, and reported whether the result is favorable to our own media recommendation or not. We work with brands spending $50,000 to $500,000-plus a month across premium CTV and programmatic who need a measurement partner that has no financial interest in which channel wins the comparison. We take a limited number of new engagements each quarter. If you've been told your media is incremental and want to see the holdout that proves it, apply to work with us.

Stillwater Media is a selective performance media agency for luxury and high-consideration brands, serving clients across the US and internationally and headquartered in Charlotte, NC. We plan and buy premium CTV, programmatic, digital out-of-home, streaming audio and YouTube Select for clients including JetLinx, W Hotels, PXG, FLY Exclusive and Financial Independence Group, and we measure everything against holdouts rather than platform-reported lift. Signal. Strategy. Scale.

Frequently Asked Questions

What does "independent" mean when an agency says it runs incrementality testing?

Independent means the party designing, controlling and reading the test has no financial interest in the result, controls the holdout outside of any advertising platform's own systems, and sets the test methodology before the campaign launches rather than analyzing results after the fact to find a favorable story. A platform reporting its own conversion or brand lift does not meet this bar, because the platform has a direct interest in the outcome and controls both the exposed group and the measurement layer.

Is platform-reported ad lift the same as incrementality?

No. Platform-reported lift studies typically suffer from selection bias, because the platform's own delivery algorithm decides which users see an ad, and that algorithm is usually already optimizing toward users more likely to convert regardless of exposure. A true incrementality test uses a holdout defined independently of the platform's delivery decisions, most reliably at the geographic market level, and reads the result against first-party outcome data rather than the platform's own conversion tracking.

Should I use marketing mix modeling or incrementality testing to evaluate a new channel?

Geo-holdout incrementality testing is the right tool for evaluating a specific new channel or campaign decision, because it produces a causal read in six to twelve weeks using a controlled holdout. Marketing mix modeling is better suited to ongoing, top-down budget allocation across a mature program with eighteen months or more of historical data, and it is generally too imprecise at the individual-channel level to validate a channel with little to no spending history, such as a brand's first CTV test.

What should independent incrementality testing cost?

A defensible geo-holdout test typically requires a measurement investment of roughly 2% to 6% of monthly media spend, scaling down as a percentage as total spend increases, with a minimum of six to ten comparable markets and an eight-to-twelve-week test window tied to the business's actual sales cycle. Measurement spend under about 2% of media budget is a common signal that an agency is cutting corners on market count or test duration in ways that undermine the statistical power of the result.

Does a Measured certified partner badge mean an agency runs independent tests?

A certification like Measured's partner program verifies that an agency has demonstrated proficiency operating that platform's specific measurement tools and methodology — a meaningful technical signal, but not proof on its own that the agency structures holdouts independently of its own media recommendations for every client engagement. Buyers should treat the badge as one input and still ask directly who controls the holdout, what happens when a test shows no lift, and whether the read uses first-party outcome data rather than platform-reported conversions.