Creative AnalyticsOctober 11, 2026

Simpson’s Paradox: When Your Winning Ad Loses Both Audiences

Your winning ad may simply reach warmer buyers. Change the audience mix, watch the ranking reverse, and learn what to put in your next creative brief.

AdSights Team

AI-powered marketing tools

Marketing MeasurementCreative AnalyticsData Science

A creative can lead the account leaderboard while converting worse in every audience segment you measured. The arithmetic is consistent: the two ads received different mixes of traffic. The mistake is crediting the creative for an advantage that came from its audience.

This matters whenever you use ad-level results to write the next brief. “A produced more conversions in this delivery environment” is a valid observation. “A’s opening hook persuades people better” is a different proposition. Between them sits the allocation system: who saw each ad, who clicked, where it ran, and when its conversions matured.

Our creative feature models article explains why the winning asset is an unstable unit of learning. This article supplies a concrete diagnostic for one failure mode: a reversal caused by audience composition. The example is synthetic, deliberately small enough to audit, and not an estimate of how often reversals happen in real accounts.

Start with counts, not the leaderboard

Suppose two creatives each receive 1,000 clicks. We classify the audience as cold or warm using information that existed before the ad was shown. Warm does not mean “people who watched this ad” or “people who clicked it”; those would be outcomes of the treatment we are trying to study.

| Creative | Audience | Clicks | Purchases | Purchase / click | | --- | --- | ---: | ---: | ---: | | A | Cold | 200 | 2 | 1.0% | | A | Warm | 800 | 64 | 8.0% | | B | Cold | 800 | 12 | 1.5% | | B | Warm | 200 | 18 | 9.0% |

Creative B has the higher conversion rate in both rows that compare like audiences. Yet A produces 66 purchases from 1,000 clicks, while B produces 30. The aggregate rates are 6.6% for A and 3.0% for B. Nothing has been miscounted. A received four times as many warm clicks, and warm clicks have much higher conversion rates in this constructed example.

This is a form of Simpson’s paradox: an association within groups reverses when those groups are combined. The Stanford Encyclopedia of Philosophy’s treatment also explains why choosing between aggregate and conditional comparisons requires causal context; neither view automatically deserves priority.

Interactive: change the mix, keep the rates

Move A’s warm-audience share down or B’s up. The within-segment conversion rates stay fixed. The observed bars change because the weighting changes. Then select Equalize delivery: both observed rates become the 50/50 standardized rates.

The chart uses a fixed zero-to-10% axis. It does not rescale when a slider moves, so the visual distance remains comparable. Each slider allocates a hypothetical 1,000 clicks; fractional expected purchases at some settings are model expectations, not claims about observed people.

The key result is not that B must win. It is that the leaderboard can change without any change in segment-level performance. A creative review that records only the top-line rate discards the mechanism behind the ranking.

Write down the weighted-average equation

For each creative, aggregate conversion rate is:

Cold-click share × cold CVR + warm-click share × warm CVR.

For A, that is 0.20 × 1% + 0.80 × 8% = 6.6%. For B, it is 0.80 × 1.5% + 0.20 × 9% = 3.0%.

A standardized comparison substitutes the same weights for both creatives. With a deliberately chosen 50/50 mix, A becomes 4.5% and B becomes 5.25%. B’s difference is 0.75 percentage points, or roughly 16.7% relative to A’s standardized rate. Those are two descriptions of the same gap; do not label the percentage-point difference a percent lift.

The 50/50 mix is a teaching convention, not the correct business population by default. For an acquisition decision, you might pre-specify last quarter’s prospect mix or the eligible population for the next campaign. Record the target weights before comparing results. Selecting weights after seeing which ones favor your preferred ad is another form of cherry-picking.

Also distinguish click-weighted conversion rate from spend-weighted economics. You cannot average subgroup CPAs using click shares and expect to recover total CPA. Reconstruct the numerator and denominator: total spend divided by total purchases. The same discipline applies to MER, ROAS, and nMER.

Which question does each view answer?

The aggregate and standardized views serve different decisions.

Observed mix: What happened under the delivery and budget allocation that actually ran? This matters for accounting, operational performance, and judging the complete creative-plus-delivery policy. If the objective is the performance of that policy, allocation is part of the outcome rather than an inconvenience to remove.

Common mix: What would the arithmetic look like if both sets of observed segment rates were weighted to the same population? This is useful for detecting composition effects and making a more comparable descriptive scorecard. It does not establish what would happen if you intervened to change allocation.

Randomized assignment: What is the effect of assigning eligible people to one creative strategy rather than another? That needs a credible experiment, an agreed outcome, sufficient precision, and a denominator aligned with assignment. Purchase rate among clickers alone can be misleading even in a randomized creative experiment because the creative itself may change who clicks.

A useful readout keeps these questions separate rather than forcing one number to do all three jobs.

Why “just segment it” is not enough

Segmentation can clarify a comparison, but more slicing is not automatically more scientific.

First, verify timing. Pre-existing customer status can be a defensible adjustment variable for a particular question. “Watched 75% of the video” is downstream of creative exposure. Conditioning on that behavior can select different kinds of people in each arm and change the question entirely.

Second, check overlap. If B never reached a particular segment, its rate there is unknown. A standardized score that fills that cell with a convenient average silently adds an extrapolation. Report the missing cell or narrow the comparison to a supported population, explaining what changed.

Third, preserve uncertainty. Two purchases out of 200 clicks are not a stable estimate of a population rate. The table demonstrates an arithmetic reversal; it supplies no confidence interval or declaration of statistical significance. Real inference must respect the experiment’s assignment unit, repeated exposures, clustering, and outcome variance. A polished bar chart cannot replace those decisions.

Finally, remember that a two-segment split is coarse. Device, geography, offer eligibility, placement, acquisition history, and calendar time can still differ within “cold.” The reversal is evidence that mix matters in this example, not proof that the selected split has removed all confounding.

A practical audit for your next creative review

Export counts at the smallest useful, reliable level rather than collecting every available breakdown. For each row, preserve creative identifier, observation period, pre-exposure segment definition, spend, impressions, clicks, purchases, and attribution settings. Keep the conversion maturity cutoff consistent; our conversion-lag guide explains why that matters.

Reconcile the export to the top-line report before interpreting it. A mismatch may come from deduplication, suppressed breakdowns, timezone boundaries, or a different conversion definition. Do not manufacture an additive table when the reporting system does not guarantee additivity.

Then calculate three things: the actual segment shares, rates within each segment, and standardized rates using a documented reference mix. Show the raw counts alongside the percentages. Flag cells with insufficient evidence instead of letting a precise-looking decimal disguise the limitation.

If the aggregate ranking reverses, rewrite the learning. Instead of “A’s testimonial hook won,” use: “A delivered more purchases in the observed mix; B had higher observed click-to-purchase rates within both measured segments. Delivery was materially different. We need an assignment-based test before reusing the result as a causal hook claim.”

The next team now has a testable question instead of a rule to copy.

Bring this to your next creative review

Use this short readout before turning a winner into a creative rule:

Observed: A converted 6.6% of clicks; B converted 3.0%.
Mix check: A received 80% warm clicks; B received 20%.
Common comparison: At a 50/50 mix, B leads by 0.75 percentage points.
Next action: Test the creative difference in the intended audience before claiming the hook caused the lift.

Replace the example with your own reconciled counts. If a segment has little data or no overlap, record that gap; a standardized score cannot supply the missing evidence.

Turn the diagnosis into a better brief

A strong follow-up brief specifies the audience population, the creative difference, the primary outcome, and the treatment assignment. It also distinguishes exploration from confirmation. You can use an observational reversal to generate a hypothesis without presenting the diagnosis as a completed experiment.

For example: “Among eligible new prospects, compare product-first and problem-first openings using randomized assignment, purchase outcomes measured over the same maturity window, and a predeclared analysis plan.” If the delivery system is itself part of the strategy, define the treatment as the complete policy and analyze that policy rather than trying to subtract it afterward.

Use the creative-testing playbook for operational setup, then size the study to your own baseline and minimum useful effect. No universal conversion count makes every test reliable.

The durable learning is a conditional statement: which feature worked, for which population, under which delivery conditions, with what evidence. The audience-mix audit makes those conditions visible before a leaderboard becomes a rule for the entire creative library.

Method and source notes

All counts, rates, and slider scenarios are authored teaching examples. There is no customer dataset behind the lab. Standardization uses fixed 50/50 click weights; both observed rates are linear weighted averages. The generated hero is conceptual artwork, not a data visualization. The source below supports the statistical concept; the advertising workflow is our application of it.