
Examples of Statistical Significance, Worked Step by Step

A result is statistically significant when it would be unlikely to appear by chance alone if nothing real were going on. The usual cutoff is a p-value below 0.05. The examples below show what that looks like in an A/B test, a delivery-time study, a coin flip and a usability survey, with the arithmetic shown so you can repeat it on your own data.
TL;DR
- Statistical significance means your data would be surprising if there were no true difference. It does not mean the effect is large, important or certain.
- A p-value of 0.03 does not mean a 3% chance you are wrong. The American Statistical Association says it is not the probability that the hypothesis is true or that chance alone produced the data.
- With enough traffic, even a 1% sales lift can be significant. With little traffic, a big lift can fail the test.
- Pick the sample size and stopping rule before you start. Peeking until p drops below 0.05 inflates false positives.
Table of Contents
- What statistical significance means in plain terms
- Example 1: an A/B test on a landing page
- Example 2: the same lift that is not significant
- Example 3: a 1% sales increase at huge scale
- Example 4: a coin that looks unfair but is not proven so
- Example 5: a pizza delivery study
- Example 6: usability scores, two that pass and two that fail
- How to find statistical significance in your own data
- Statistical significance vs. practical significance
- Common mistakes that create false "wins"
- Perspective
- Run your own significance tests with Stellar
- FAQ
- Sources
What statistical significance means in plain terms
Every significance test starts by assuming there is no difference: the new page converts the same as the old one, the coin is fair, the drug does nothing. This is the null hypothesis. You then ask how often you would see a result at least as extreme as yours if that assumption were true.
That frequency is the p-value. Netflix's engineering team describes an A/B-test p-value as the probability of an outcome at least as extreme as the one observed, if treatment and control truly did not differ. If the p-value falls below your significance level (alpha, most often 0.05), you call the result statistically significant.
Three points keep this honest:
- The p-value belongs to a whole statistical model, not just the null hypothesis. The peer-reviewed Greenland et al. guide explains that it also depends on the model's assumptions, such as random assignment and independent observations.
- Smaller p-values mean the data fit the no-difference model less well, provided those assumptions hold.
- Significance is a yes-or-no label on evidence. It says nothing about how big or valuable the effect is.
Example 1: an A/B test on a landing page
This is a hypothetical test with round numbers, built to show the mechanics.
You split traffic evenly between your current landing page (control) and a new headline (variant):
| Visitors | Conversions | Conversion rate | |
|---|---|---|---|
| Control | 5,000 | 250 | 5.0% |
| Variant | 5,000 | 300 | 6.0% |
The null hypothesis is that both pages have the same true conversion rate. A two-proportion z-test works like this:
- Pool the results: 550 conversions out of 10,000 visitors gives 5.5%.
- Compute the standard error of the difference: the square root of 0.055 × 0.945 × (1/5,000 + 1/5,000), which is about 0.00456.
- Divide the observed difference by it: 0.010 ÷ 0.00456 gives z of about 2.19.
- For a two-sided test, z = 2.19 corresponds to p of about 0.028.
Because 0.028 is below 0.05, the result is statistically significant at the 5% level. If the two headlines truly performed the same, a gap this large would show up in roughly 3 of every 100 such tests.
The variant is not proven better. The test gives you reasonably strong evidence of a difference, and the confidence interval around the 1-point lift tells you how large that difference might plausibly be.
Example 2: the same lift that is not significant
Now shrink the sample. Each page gets 2,000 visitors. Control converts 100 (5.0%) and the variant converts 112 (5.6%).
- Pooled rate: 212 ÷ 4,000 = 5.3%
- Standard error: about 0.0071
- z: 0.006 ÷ 0.0071, about 0.85
- Two-sided p-value: about 0.40
A p-value of 0.40 is far above 0.05, so this is not significant. That does not mean the variant is no better. The Schober et al. article states that a non-significant p-value means insufficient evidence against the null, not proof that no effect exists. Netflix makes the same point about A/B tests: failing to reject the null is not proof of no effect.
Compare the two examples. A 1-point lift passed with 10,000 visitors in total and failed at a smaller scale. Sample size drives significance as much as the lift does. If you want to plan this ahead, our guide on the seven inputs you need to calculate statistical significance walks through baseline rate, minimum detectable effect and traffic.
Example 3: a 1% sales increase at huge scale
Nielsen Norman Group published a classic A/B-test example. A site converts at 2.00% and gets 1.5 million visitors in a test arm. At 2.00% you would expect 30,000 sales. The variant records 30,300 sales, a 2.02% rate and a 1% relative increase.
NN/g reports that if the true rate had stayed at 2.00%, the probability of seeing at least 30,300 sales would be about 4%. That clears the conventional 5% threshold, so the result is statistically significant.
This example shows how traffic changes what counts as detectable. A 1% lift is invisible on a small site and measurable on a very large one. On a high-volume store, though, a 1% lift can still be worth real money, which is the point: the question is whether the size of the effect matters to your business, not just whether the test passed.
Example 4: a coin that looks unfair but is not proven so
Netflix's coin example is a good antidote to overconfidence. You flip a coin 100 times and get 55% heads. It feels suspicious. The two-sided p-value is about 0.32. Since 0.32 is well above 0.05, you do not have statistically significant evidence that the coin is unfair.
The lesson for marketers: a variant that is "up 5 points" after a few hundred visitors can be exactly this kind of noise. Early leads in a dashboard are often coin flips.
Example 5: a pizza delivery study
Penn State's introductory statistics course uses a clean one-sample example. A pizza chain looks at 198 orders. On average, they arrived 3 minutes early, with a standard deviation of 15 minutes. Is the average really different from zero?
- Null hypothesis: the true mean difference is 0. Alternative: it is greater than 0.
- Standard error: 15 ÷ √198, about 1.07 minutes.
- Test statistic: (3 − 0) ÷ 1.07, about 2.8.
- The p-value is about 0.003, or 0.3%.
A p-value that small makes "no average difference" a poor explanation of the data. Penn State adds a caution in the same lesson: a 3-minute difference may have little practical importance even though it is statistically significant. A customer who gets pizza three minutes early will not notice.
Example 6: usability scores, two that pass and two that fail
MeasuringU reports four comparisons from usability research. They are a useful range because the p-values sit at very different distances from the line:
| Measure | Difference | p-value | At the 0.05 level |
|---|---|---|---|
| SUS (System Usability Scale) | 16.9 points | .004 | Significant |
| SEQ (Single Ease Question) | 1 point | .0499 | Significant |
| SUS | 2 points | .28 | Not significant |
| UMUX-Lite | 0.5 points | .38 | Not significant |
These come from MeasuringU's write-up on hypothesis testing, which classifies the first two as significant and the last two as not.
Notice the second row. A p-value of .0499 passes and .051 would not, yet the evidence is essentially the same. Treat 0.05 as a convention that helps teams decide, not a cliff between truth and fiction.
How to find statistical significance in your own data
The steps are the same whether you test conversions, delivery times or survey scores.
- State the hypotheses. Null: no difference. Alternative: a difference (two-sided) or an improvement (one-sided). Decide this before you look at results.
- Choose alpha. 0.05 is common. Use a stricter level when a wrong call is costly.
- Pick the right test. The University of Queensland's test-selection table points to a two-sample t-test for small independent samples, or a z-test for large ones, when comparing means or proportions. The choice depends on the research question, variable types, whether groups are independent, and assumptions such as approximate normality.
- Set sample size and duration up front. Run until you reach it.
- Calculate the test statistic and p-value. A testing platform or a statistics package does this for you.
- Read the confidence interval too. Confidence intervals estimate effect size and show which values a two-sided test at your alpha would reject.
- Decide with the effect size in mind. Ask whether the lift is worth shipping.
For a marketer-focused walkthrough of reading these numbers inside an experiment, see our guide to interpreting statistical significance in A/B test results.
Statistical significance vs. practical significance
A significant result is not automatically a useful one. The U.S. Institute of Education Sciences explains that in a large sample a trivial difference can be significant, while an important effect in a small sample may miss the threshold. The ASA statement says the same: tiny effects can produce small p-values in large, precise samples, and large effects can have unimpressive p-values in small or noisy samples.
Before you ship a winning variant, check three things:
- Size: is the lift big enough to matter in revenue, leads or retention?
- Range: does the lower end of the confidence interval still justify the change?
- Cost: what does it take to build, maintain and explain the change?
We cover that decision in more depth in when statistically significant results shouldn't make marketers ship.
Common mistakes that create false "wins"
Peeking and stopping early. Checking the dashboard daily and stopping when p dips under 0.05 is a form of p-hacking. The peer-reviewed misconceptions review warns that collecting more data until p is below .05, or changing the analysis along the way, can push the false-positive rate above 5%.
Testing many things at once. Every extra comparison adds another chance for a fluke. NIST's handbook notes that repeated pairwise comparisons inflate the overall error rate, and lists Tukey, Scheffé and Bonferroni procedures for follow-up comparisons. In practice: if you test ten metrics or ten variants, one "significant" result is not surprising.
Small samples and flexible analysis. A review of false-positive findings lists small samples, multiple testing, researcher flexibility, selective reporting, stopping once p is under .05 and non-independent observations as reasons apparent findings fail to hold up.
Misreading the confidence interval. A 95% interval does not mean a 95% chance the true value sits inside this particular range. The Pernet tutorial explains that across repeated samples, 95% of intervals built this way would contain the true parameter.
Reading "not significant" as "no effect." As Example 2 showed, it usually means you needed more data.

If you want the conversion side of this explained from scratch, what conversion means in marketing is a helpful primer before you set up a metric to test. And if your experiments involve design variations, color tests in marketing are a common place to start, and a good reminder that small visual changes usually need large samples to detect.
Perspective
Most teams that struggle with significance do not have a math problem. They have a process problem. They launch a test, glance at it every morning and call it when the dashboard turns green.
We built Stellar around the opposite habit: set up the experiment quickly, track the right goals, and let real-time analytics show you the data while you wait for enough traffic. A no-code visual editor means a marketer can launch a test without a developer queue, so you can run more experiments, each one properly sized. A script of only 5.4KB is designed to keep the page fast, so the test itself does not distort the behavior you are measuring.
Our view is simple: significance is a tool for decisions, not a trophy. Pair it with effect size, a clear goal and a stopping rule, and a small team can learn faster than a large one that guesses.
Juan
Run your own significance tests with Stellar
If you would rather watch these numbers move on your own pages than on textbook examples, Stellar is an A/B testing platform for marketers and growth teams. It includes a no-code visual editor, advanced goal tracking, real-time analytics and integrations with WordPress, Shopify, Webflow, Wix, Squarespace, Framer and Bubble.
Pricing is based on monthly tracked users: Sandbox is $0 per month, Plus is $29 per month, Pro is $99 per month and Scale is $399 per month, with Enterprise available on request. The free plan is for businesses under 25,000 monthly tracked users. See the plans on the Stellar site.

When you are ready to size your first test, our practical guide to A/B test significance for marketers is a good next step.
FAQ
What is an example of a statistically significant result?
A landing page converts at 5.0% on 5,000 visitors, and a new version converts at 6.0% on 5,000 visitors. A two-proportion z-test gives a p-value of about 0.028, below 0.05, so the difference is statistically significant. It means a gap that large would be unusual if both pages truly performed the same.
What does 5% statistical significance mean?
It means you accept a 5% risk of calling a difference real when there is none, if the null hypothesis is true and the test's assumptions hold. It is a convention, not a law of nature. A p-value below 0.05 is not a 95% chance your hypothesis is correct.
How do you find statistical significance?
State a null hypothesis, choose alpha, pick a test that fits your data, then calculate a test statistic and p-value. For two conversion rates, a two-proportion z-test works. For means, a t-test or z-test depends on sample size and assumptions. Plan the sample size before the test starts.
Does a significant result mean the change is worth making?
Not by itself. Significance tells you the data are unlikely under no difference; it does not measure effect size or importance. Check the size of the lift and its confidence interval against the cost of making the change.
Sources
- The ASA's Statement on p-Values: Context, Process, and Purpose
- Statistical tests, P values, confidence intervals, and power (Greenland et al.)
- Statistical Significance Versus Clinical Importance (Schober et al.)
- Interpreting A/B Test Results: False Positives and Statistical Significance (Netflix Technology Blog)
- Significance Testing Caveats and Ethics of Experiments (Penn State)
- Measuring 1% Increase in Sales Through A/B Testing (NN/g)
- How Does Statistical Hypothesis Testing Work? (MeasuringU)
- Choosing the Right Elementary Statistical Test (University of Queensland)
- Null hypothesis significance testing: a short tutorial
- Statistically Significant Doesn't Mean Meaningful (IES)
- 7.4.7. How can we make multiple comparisons? (NIST)
- Detecting and avoiding likely false-positive findings
- Common misconceptions about data analysis and statistics
