# Sample Size

**Category:** metrics  
**Short Description:** The number of observations in a sample, critical for the accuracy and reliability of statistical estimates.  
**Last Updated:** 2026-07-04T15:01:33Z

## Definition

Sample size refers to the number of observations or data points collected in a sample, and is a crucial factor in determining the precision of statistical estimates. In advertising, it directly impacts the confidence, reliability, and validity of metrics such as conversion rates, click-through rates, and return on ad spend (ROAS). The larger the sample size, the more reliable the results, as smaller samples can lead to more variability and less confidence in the conclusions drawn from the data.

## Formula

**Formula:** `n = (Z² × σ²) / E²`

How many observations you need: more variable data (σ) or a tighter precision target (E) both demand a larger sample, while higher confidence (Z) raises it too.

## Calculation

**Formula:** `n = (Z² * σ²) / E²`

**Explanation:** This formula sizes a sample for *estimating* a single rate or mean to a target margin of error: Z is the Z-score for the desired confidence level, σ² is the population variance, and E is the margin of error, and larger samples shrink E for a more precise estimate. Note it does not include statistical power or the two-group comparison an A/B test needs — power-based A/B sizing uses a two-proportion (or two-sample) test formula that factors in baseline rate, minimum detectable effect, confidence, and power, which is what the A/B-testing FAQ below refers to. Use a dedicated sample-size/power calculator for experiment planning. The result n is a count of observations (impressions, sessions, or respondents).

### Components

- **Confidence Level**: The degree of certainty you want in your results (e.g., 95% confidence)
- **Margin of Error**: The allowable deviation from the true population value, often expressed as a percentage
- **Population Variance**: A measure of how much variability exists in the population being sampled

## Industry Benchmarks

| Segment | Typical Range | Median | Notes |
| --- | --- | --- | --- |
| A/B test, 5% baseline, +10% relative MDE, 95%/80% | n ≈ 30,000 – 32,000 / variant | ~31,000 | Detecting a 5.0% → 5.5% lift needs far more traffic than teams expect. |
| A/B test, 5% baseline, +20% relative MDE | n ≈ 7,500 – 8,500 / variant | ~8,100 | Larger MDE = smaller required n — trade precision for speed consciously. |
| Creative test, 2% CTR baseline, +15% relative MDE | n ≈ 25,000 – 30,000 / variant | ~27,000 | Low baselines are expensive to test — plan fewer variants or longer flights. |
| Incrementality geo test (market-level) | 4 – 12 weeks minimum | ~8 weeks | Sample size here is time × markets, not users — underpowered geos produce false nulls. |

**Sources:** Evan Miller sample-size calculator, Binomial power analysis, Incrementality testing practitioner guides 2025

## Examples

- To estimate a 5% conversion rate with a ±1% margin of error at 95% confidence, you would need 10,000 impressions.
- Achieving statistical significance in A/B testing might require 5,000 samples per variant to detect a 1% difference in CTR.
- In a brand awareness study, a sample size of 1,200 respondents is required to achieve 95% confidence with a 3% margin of error.

## Key Points

- Sample size is the number of observations a test needs to detect a real effect with acceptable confidence and power — decide it before launching, not after.
- It's driven by three inputs: baseline rate, the minimum effect you want to detect (MDE), and your confidence/power targets.
- Smaller effects and lower baselines cost dramatically more traffic — detecting a 5.0%→5.5% lift needs far more volume than teams expect.
- Underpowered tests are worse than no test: they miss real winners (false negatives) and produce unstable results that invite bad calls.
- Fix the sample target up front and read the result at the target — 'peeking' and stopping early inflates false positives.

## How AdSights Helps

**Tracking Sample Size:** Sample-size planning fails when teams guess instead of calculating from baseline rate and MDE. AdSights tracks per-variant conversion volume in real time against the pre-set sample target, and tags which creative concepts are approaching significance vs which are still in the noise zone. That keeps 'peeking' honest — you see progress toward the plan, not a premature leaderboard.

## FAQs

### What is sample size and why does it matter?

Sample size is the number of observations — impressions, clicks, conversions, or respondents — collected in a test or measurement. It matters because it governs how much you can trust the result: too few observations and normal random variation swamps any real signal, so you can neither reliably detect a true effect nor rule one out. In advertising, adequate sample size is what separates a genuine performance difference from noise you shouldn't act on.

### How do I determine the right sample size for an A/B test?

Work from three inputs: your baseline conversion rate, the minimum detectable effect (the smallest lift worth detecting), and your confidence and power targets (commonly 95% confidence and 80% power). Feed these into a sample-size or power calculator to get the observations needed per variant. Then estimate how long that will take at your traffic level — if it's impractically long, you may need to accept a larger detectable effect, test fewer variants, or extend the flight.

### What happens if my sample size is too small?

Underpowered tests are unreliable in both directions: they frequently miss real effects (false negatives), and the estimates they do produce swing widely, so an early 'winner' can reverse with more data. Acting on small samples leads teams to scale creative or audiences that aren't actually better and to kill ones that are. A too-small sample doesn't just weaken a test — it can actively mislead.

### What is minimum detectable effect (MDE)?

MDE is the smallest true difference you want your test to be able to detect reliably. It has an inverse relationship with required sample size: the smaller the effect you want to catch, the more observations you need. Setting MDE is a deliberate trade-off — a large MDE lets you test quickly but only catches big wins, while a small MDE catches subtle improvements at the cost of much more traffic and time. Choose it based on what size of lift would actually change a decision.

### Does a bigger sample size always mean better results?

Bigger samples give more precise, reliable estimates, but there are diminishing returns and real costs. Past the point of adequate power, extra volume spent chasing ever-tinier confidence gains is traffic and time you could spend testing something else. Very large samples can also surface statistically significant differences that are too small to matter commercially. Aim for enough to detect effects that would change a decision — not the largest sample possible for its own sake.

## Related Terms

### Component Terms

- **[Margin of Error](/resources/glossary/metrics/margin-of-error)**: Sample size directly impacts the margin of error, with larger sample sizes leading to more precise estimates.
- **[Variance](/resources/glossary/metrics/variance)**: Sample size directly impacts the variance, with larger sample sizes leading to more precise estimates.
- **[Population Mean](/resources/glossary/metrics/population-mean)**: Sample size determines how closely a sample mean approximates the true population mean

## Related Resources

- [Statistical Noise: Unmasking the Illusion of Insights in Modern Marketing](/blog/topics/data-science/statistical-noise-unmasking-the-illusion-of-insights-in-modern-marketing) - Comprehensive exploration of how sample size affects statistical reliability in marketing data

## Featured in topic hubs

- [Creative Testing](/resources/topics/creative-testing)
- [Experimentation & Statistics](/resources/topics/experimentation)