# Confidence Interval **Category:** metrics **Short Description:** Range of values likely to contain the true population parameter. **Last Updated:** 2026-07-04T15:01:33Z ## Definition A confidence interval provides a range of values that likely contains the true value of a metric, given a certain confidence level. In digital advertising, it helps marketers understand the reliability of their performance measurements and make more informed decisions about campaign optimization. Wider intervals suggest more uncertainty, while narrower intervals indicate more precise estimates of true performance. ## Formula **Formula:** `CI = Point Estimate ± (Critical Value × Standard Error)` The range your true value most likely falls in — the observed estimate plus or minus a margin that widens with higher confidence and shrinks with larger samples. ## Calculation **Formula:** `CI = Point Estimate ± (Critical Value × Standard Error)` **Explanation:** For advertising metrics, the point estimate is the observed value (e.g., CTR, conversion rate), the critical value depends on desired confidence level (typically 1.96 for 95%), and standard error accounts for sample size and variance. Larger sample sizes generally lead to narrower, more reliable intervals. The interval bounds are in the same units as the metric being estimated. ### Components - **Point Estimate**: The observed metric value - **Critical Value**: The value based on the desired confidence level - **Standard Error**: The measure of sample estimate precision ## Industry Benchmarks | Segment | Typical Range | Median | Notes | | --- | --- | --- | --- | | A/B test at 95% confidence (typical) | ±2% – ±8% on conversion rate | ±4% | Width shrinks with √n — doubling sample size cuts margin roughly in half. | | Creative test (5% baseline CTR, n=5K/variant) | CTR CI ≈ 4.4% – 5.6% | ±0.6 pp | At typical DTC volumes, early reads swing ±1 pp — too narrow to call winners. | | ROAS forecast (30-day, mature account) | ±15% – ±25% of point estimate | ±20% | Wider than in-platform ROAS because incrementality and returns lag aren't captured. | **Sources:** Standard statistical practice, Evan Miller sample-size tables, Binomial proportion CI at 95%, Marketing mix modeling practitioner ranges 2025 ## Examples - 95% confidence that true campaign ROAS is between 2.1 and 2.4, informing budget decisions - Conversion rate of 3% ± 0.5% at 90% confidence for a new audience segment - CTR confidence interval narrowing from ±1% to ±0.2% as impressions increase ## Key Points - A confidence interval reports a range plus a confidence level (e.g. 95%) — not a guarantee the true value sits in the range, but the reliability of the method that produced it. - Width is the message: a wide interval means high uncertainty, a narrow one means a precise estimate you can act on. - Intervals narrow as sample size grows and as variance shrinks — more impressions or conversions tighten the range. - In A/B tests, judge significance from the interval on the difference (the lift) or your significance test — not by eyeballing whether the two variants' separate intervals overlap; overlapping intervals can still hide a real, significant difference. - Report the interval, not just the point estimate — '3% ± 0.5%' tells stakeholders how much to trust the number and how much room to plan around. ## How AdSights Helps **Tracking Confidence Interval:** Confidence intervals turn a point estimate into a decision range — but only if you're measuring the right thing. AdSights connects creative-variant performance to the interval on the lift itself, so you see not just 'Variant B leads by 0.3 pp CTR' but whether that difference's interval clears zero at your sample size. When the lift's interval spans zero, AdSights flags the test as inconclusive rather than letting teams scale a false winner; when it clears zero, it surfaces which creative attributes (hook, format, offer framing) drove the gain. ## FAQs ### What is a confidence interval? A confidence interval is a range of values, derived from your data, that is likely to contain the true underlying metric — reported with a confidence level such as 95%. In advertising it expresses how much to trust a measured rate: rather than saying a campaign's conversion rate 'is 3%,' a confidence interval says it's '3% ± 0.5% at 95% confidence,' making the uncertainty explicit so decisions account for it. ### What does a '95% confidence level' actually mean? It describes the reliability of the method, not the probability for a single interval. If you repeated the same measurement many times and built a 95% interval each time, about 95% of those intervals would contain the true value — over the long run roughly 1 in 20 would miss it. Any one realized interval either contains the true value or doesn't; the 95% is your confidence in the procedure, so treat the range as a decision-grade working estimate rather than a 95%-probability statement about that specific interval. Higher confidence (99%) widens the interval; lower confidence (90%) narrows it. ### How does sample size affect the confidence interval? Larger samples produce narrower intervals. Because the interval width depends on the standard error — which shrinks as sample size grows — more impressions, clicks, or conversions yield a more precise estimate. This is why early campaign reads have wide, unreliable intervals that can swing dramatically, and why waiting for adequate volume is essential before treating a measured difference as real. ### How do I use confidence intervals in A/B testing? Look at the confidence interval on the difference between variants (the lift), or your planned significance test — not whether the two arms' individual intervals overlap. If the lift's interval excludes zero at your confidence level, the difference is real; if it spans zero, the test is inconclusive and scaling the 'winner' risks acting on randomness. Comparing the two separate intervals is a common trap: they can overlap while the difference is still significant, so that eyeball test wrongly kills real winners. Non-overlapping individual intervals do imply significance, but that's a conservative shortcut, not the actual rule. ### What's the difference between a confidence interval and margin of error? They're two views of the same thing. The margin of error is the ± amount added to and subtracted from the point estimate; the confidence interval is the resulting range. For a 3% conversion rate with a ±0.5% margin of error, the 95% confidence interval is 2.5% to 3.5%. Margin of error is the half-width; the confidence interval is the full band it defines. ## Related Terms ### Similar Terms - **[Statistical Significance](/resources/glossary/metrics/statistical-significance)**: While statistical significance confirms differences, confidence intervals quantify uncertainty - **[Margin of Error](/resources/glossary/metrics/margin-of-error)**: Expresses uncertainty range in more accessible format for stakeholders - **[Creative Testing](/resources/glossary/creative/creative-testing)**: Confidence intervals guide minimum test durations and sample sizes ### Component Terms - **[Standard Deviation](/resources/glossary/metrics/standard-deviation)**: Determines interval width based on metric volatility ## Related Resources - [A/B Test Statistical Significance Calculator](/resources/tools/calculators/ab-test-statistical-significance-calculator) - Check whether your test result is inside or outside the confidence band. - [Sample Size](/resources/glossary/metrics/sample-size) - How many observations you need to narrow the interval to a useful width. ## Featured in topic hubs - [Creative Testing](/resources/topics/creative-testing) - [Experimentation & Statistics](/resources/topics/experimentation)