Hypothesis Testing: t-Tests, Chi-Square, ANOVA and A/B Tests

"Prime members spend more per order." "Evening deliveries are slower." "The new app banner improved checkout conversion." These are claims about a difference, and the data in front of you always shows some difference, because no two groups are identical. The question a hypothesis test answers is narrower and more useful: could a difference this large plausibly arise by pure chance if there were really no difference?

This lesson walks through the logic and the workflow: state a null hypothesis (no effect), pick a test that fits the data, compute a p-value, and then, just as important, measure the effect size to judge whether the difference matters to the business. We apply the two-sample t-test, its rank-based alternative (Mann-Whitney), the chi-square test for categories, one-way ANOVA for several groups, and a proper A/B test analysis. We end with the multiple-comparisons trap, the reason that testing twenty ideas always "finds" one significant result.

All tests use SciPy (scipy.stats). The calculations are short; the thinking is the real skill.

Worked examples

Example 1: Set up the comparison

Question: Do Prime and non-Prime customers place different-sized orders?

import pandas as pd
import numpy as np
from scipy import stats

orders = pd.read_csv("orders.csv", parse_dates=["order_date"])
items = pd.read_csv("order_items.csv")
customers = pd.read_csv("customers.csv")

value = (items["quantity"] * items["unit_price"]).groupby(items["order_id"]).sum().rename("value")
d = (orders.merge(value, on="order_id").merge(customers[["customer_id", "is_prime"]], on="customer_id"))
d = d[d["order_status"] == "Delivered"]

prime = d.loc[d["is_prime"], "value"]
other = d.loc[~d["is_prime"], "value"]
print("Prime n, mean  :", len(prime), round(prime.mean(), 1))
print("Other n, mean  :", len(other), round(other.mean(), 1))
print("difference     :", round(prime.mean() - other.mean(), 1))

Prime n, mean  : 1783 657.0
Other n, mean  : 3608 654.8
difference     : 2.3

How it works: We split delivered orders into two groups and compare their means. The observed difference is just a number; whether it is "real" depends on how variable the data is and how many orders we have. The test formalises that.

Reading the result: Prime orders average ₹657.0 over 1783 orders and others ₹654.8 over 3608: a difference of ₹2.3.

Example 2: The two-sample t-test

Question: Is that difference larger than chance would produce?

res = stats.ttest_ind(prime, other, equal_var=False)
print("t statistic:", round(res.statistic, 3))
print("p-value    :", round(res.pvalue, 4))
print("reject H0 at 5%?", res.pvalue < 0.05)

t statistic: 0.157
p-value    : 0.8755
reject H0 at 5%? False

How it works: The null hypothesis (H0) is that the two groups have the same true mean. The t statistic is the difference in means divided by its standard error. The p-value is the probability of seeing a difference at least this large if H0 were true. If p is below the chosen significance level (alpha, usually 0.05) we reject H0. equal_var=False gives Welch's test, which does not assume equal spread and is the safe default.

Reading the result: t = 0.16 and p = 0.875. At the 5% level this is not significant. We cannot claim Prime baskets differ from others; absence of evidence is not evidence of absence.

Example 3: Effect size and the interval

Question: How big is the difference, and how sure are we about its size?

diff = prime.mean() - other.mean()
se = np.sqrt(prime.var(ddof=1) / len(prime) + other.var(ddof=1) / len(other))
ci = (diff - 1.96 * se, diff + 1.96 * se)

pooled_sd = np.sqrt(((len(prime) - 1) * prime.var(ddof=1) + (len(other) - 1) * other.var(ddof=1)) / (len(prime) + len(other) - 2))
cohens_d = diff / pooled_sd

print("difference:", round(diff, 1), " 95% CI:", tuple(round(x, 1) for x in ci))
print("percent difference:", round(diff / other.mean() * 100, 1), "%")
print("Cohen's d:", round(cohens_d, 3))

difference: 2.3  95% CI: (np.float64(-26.0), np.float64(30.6))
percent difference: 0.3 %
Cohen's d: 0.005

How it works: A p-value says nothing about size. Report an effect size: the raw or percentage difference with a confidence interval, and Cohen's d, the difference in units of standard deviation. A rough guide for d is 0.2 small, 0.5 medium and 0.8 large. An interval that spans zero is consistent with no effect; one that excludes zero agrees with a significant test.

Reading the result: The difference is ₹2.3 (0.3%) with a 95% interval from -26 to 31 rupees (it spans zero), and Cohen's d is 0.00, a negligible effect.

Example 4: Mann-Whitney U for skewed data

Question: Do we reach the same conclusion without assuming a bell curve?

mw = stats.mannwhitneyu(prime, other, alternative="two-sided")
print("U =", mw.statistic, " p =", round(mw.pvalue, 4))
print("medians:", prime.median(), other.median())

U = 3207309.5  p = 0.8638
medians: 531.0 531.0

How it works: Mann-Whitney U compares the two groups using ranks rather than means, so it does not need normality and is less affected by outliers. It tests whether one group tends to have larger values. For skewed variables like order value it is a good companion to the t-test; when both agree, you can be more confident.

Reading the result: Mann-Whitney gives p = 0.864, agreeing with the t-test (p = 0.875). The medians are ₹531 and ₹531.

Example 5: Always look at the size of the gap

Question: Do Card and UPI orders differ in delivery time, and would anyone notice?

card = orders.loc[(orders["payment_method"] == "Card") & orders["delivery_minutes"].notna(), "delivery_minutes"]
upi = orders.loc[(orders["payment_method"] == "UPI") & orders["delivery_minutes"].notna(), "delivery_minutes"]

t5 = stats.ttest_ind(card, upi, equal_var=False)
print("means:", round(card.mean(), 2), round(upi.mean(), 2))
print("p-value:", round(t5.pvalue, 4))
print("difference in seconds:", round((card.mean() - upi.mean()) * 60, 1))

means: 23.7 23.64
p-value: 0.8018
difference in seconds: 3.6

How it works: We compare delivery times for Card and UPI orders. Whatever the p-value, look at the difference itself, here converted to seconds. Large samples can make trivially small differences "significant", and small samples can hide large ones, so always read the p-value next to the size of the gap. Significance answers "is there likely any effect?", not "is it big enough to act on?".

Reading the result: Card deliveries average 23.70 minutes and UPI 23.64, a gap of 4 seconds, with p = 0.80. The difference is not statistically significant, and even if it were, no customer would notice a few seconds.

Example 6: Chi-square test for categories

Question: Is the order outcome (delivered, cancelled, returned) related to the payment method?

table = pd.crosstab(orders["payment_method"], orders["order_status"])
print(table)

chi2, p, dof, expected = stats.chi2_contingency(table)
print("chi2 =", round(chi2, 2), " dof =", dof, " p =", round(p, 4))
print("min expected count:", round(expected.min(), 1))

order_status    Cancelled  Delivered  Returned
payment_method                                
COD                    65        871        40
Card                   74       1106        45
UPI                   202       2990       138
Wallet                 29        424        16
chi2 = 1.48  dof = 6  p = 0.9606
min expected count: 18.7

How it works: For two categorical variables, a chi-square test of independence compares the observed counts with the counts expected if the variables were unrelated. A large chi-square (small p) means the pattern differs from independence. The rule of thumb is that all expected counts should be at least 5, which we verify with expected.min().

Reading the result: chi-square = 1.48 with 6 degrees of freedom and p = 0.961. There is no evidence that the order outcome depends on the payment method. The smallest expected count is 18.7, so the test is valid.

Example 7: One-way ANOVA across several groups

Question: Does delivery time differ between delivery slots?

groups = [g["delivery_minutes"].dropna().values for _, g in orders.groupby("delivery_slot")]
names = sorted(orders["delivery_slot"].unique())
f = stats.f_oneway(*groups)

print(names)
print([round(g.mean(), 2) for g in groups])
print("F =", round(f.statistic, 3), " p =", round(f.pvalue, 4))
print("Kruskal-Wallis p =", round(stats.kruskal(*groups).pvalue, 4))

['Evening (4-8)', 'Midday (10-2)', 'Morning (6-10)', 'Night (8-11)']
[np.float64(23.73), np.float64(23.8), np.float64(23.67), np.float64(23.47)]
F = 0.521  p = 0.6676
Kruskal-Wallis p = 0.6524

How it works: With three or more groups, running many pairwise t-tests inflates the false-positive rate. ANOVA tests the single hypothesis "all group means are equal". A significant result says some group differs but not which one; follow up with pairwise comparisons if needed. Kruskal-Wallis is the rank-based equivalent when normality is doubtful.

Reading the result: The slot means range from 23.5 to 23.8 minutes. ANOVA gives p = 0.67, so the slots do not differ detectably.

Example 8: Analysing an A/B test

Question: A new checkout button was shown to half of 4,000 visitors. Did it raise conversion?

# simulated experiment: control converts at 8.0%, variant at 9.5% (invented numbers)
rng = np.random.default_rng(2026)
n_a = n_b = 2000
conv_a = rng.binomial(n_a, 0.080)
conv_b = rng.binomial(n_b, 0.095)
rate_a, rate_b = conv_a / n_a, conv_b / n_b

pooled = (conv_a + conv_b) / (n_a + n_b)
se_diff = np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
z = (rate_b - rate_a) / se_diff
p_two = 2 * (1 - stats.norm.cdf(abs(z)))

print("control", conv_a, "/", n_a, "=", round(rate_a, 4))
print("variant", conv_b, "/", n_b, "=", round(rate_b, 4))
print("lift:", round((rate_b - rate_a) / rate_a * 100, 1), "%")
print("z =", round(z, 2), " p =", round(p_two, 4))

control 152 / 2000 = 0.076
variant 199 / 2000 = 0.0995
lift: 30.9 %
z = 2.63  p = 0.0086

How it works: Conversion is a yes/no outcome per visitor, so we compare two proportions with a two-proportion z-test, using the pooled rate for the standard error under the null hypothesis. The data here is simulated with known true rates so that you can see the machinery; in a real test you would have the counts from your experiment tool. Randomly assigning visitors to the two versions is what lets us attribute a difference to the change itself rather than to who happened to see it.

Reading the result: Control converted 7.60% and the variant 9.95%, a relative lift of 30.9% (the true simulated lift was 18.8%, so chance has exaggerated it here). With z = 2.63 and p = 0.009 the result is significant at 5%.

Example 9: Statistical power

Question: How many visitors would we need to detect that lift reliably?

def power_two_props(p1, p2, n, alpha=0.05):
    se = np.sqrt(p1 * (1 - p1) / n + p2 * (1 - p2) / n)
    z_crit = stats.norm.ppf(1 - alpha / 2)
    return 1 - stats.norm.cdf(z_crit - abs(p2 - p1) / se)

for n in (500, 1000, 2000, 4000, 8000):
    print(f"n per group = {n:>5}: power = {power_two_props(0.08, 0.095, n):.2f}")

n_80 = next(n for n in range(100, 50000, 50) if power_two_props(0.08, 0.095, n) >= 0.8)
print("n per group for 80% power:", n_80)

n per group =   500: power = 0.13
n per group =  1000: power = 0.22
n per group =  2000: power = 0.39
n per group =  4000: power = 0.66
n per group =  8000: power = 0.92
n per group for 80% power: 5600

How it works: Power is the probability that a test detects a real effect of a given size. A Type II error (missing a real effect) happens with probability 1 minus power. Underpowered tests are common: with too few visitors you will often see "no significant difference" even when the change really works. The usual target is 80% power. The function here uses the normal approximation to calculate power for a given n, then we search for the n that gives 80%.

Reading the result: Detecting a lift from 8.0% to 9.5% needs about 5600 visitors per group for 80% power. With 2,000 per group the power is only 39%, so a real lift of that size would be missed about 61% of the time.

Example 10: Multiple comparisons

Question: What if we test twenty unrelated ideas?

rng = np.random.default_rng(4)
pvals = []
for _ in range(20):
    a = rng.normal(100, 15, 200)
    b = rng.normal(100, 15, 200)     # no true difference at all
    pvals.append(stats.ttest_ind(a, b).pvalue)

pvals = np.array(pvals)
print("tests with p < 0.05:", int((pvals < 0.05).sum()))
print("smallest p:", round(pvals.min(), 4))
print("chance of at least one false positive in 20 tests:", round(1 - 0.95 ** 20, 3))
print("Bonferroni threshold:", 0.05 / 20, " survivors:", int((pvals < 0.05 / 20).sum()))

tests with p < 0.05: 1
smallest p: 0.0296
chance of at least one false positive in 20 tests: 0.642
Bonferroni threshold: 0.0025  survivors: 0

How it works: Each test has a 5% chance of a false positive. Run 20 and the chance that at least one "works" is about 64%. All twenty comparisons here use data drawn with no real difference, yet some usually come out significant. The simplest guard is the Bonferroni correction: divide alpha by the number of tests. It is conservative but easy; the lesson is to count how many things you tried and to treat a lone surprising result with suspicion.

Reading the result: 1 of 20 null comparisons reached p < 0.05 (smallest p = 0.030); a different seed would give a different count, with about one expected. The chance of at least one false positive in 20 tests is 64%. After Bonferroni correction, 0 survive.

Common mistakes

1. Reading the p-value as "the probability the null is true". It is the probability of data this extreme if the null were true.

2. Treating "not significant" as "no effect". It may just mean too little data (low power).

3. Ignoring effect size, and acting on tiny but significant differences.

4. Testing many things and reporting only the winner (p-hacking). Decide the question, the metric and the sample size before looking.

5. Peeking at an A/B test every day and stopping when p < 0.05. That inflates false positives. Fix the sample size in advance.

stats.ttest_ind(prime, "other")

Traceback (most recent call last):
  File "example.py", line 1, in <module>
    stats.ttest_ind(prime, "other")
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "example.py", line 601, in axis_nan_policy_wrapper
    res = hypotest_fun_out(*samples, **kwds)
  File "example.py", line 6791, in ttest_ind
    a, b = xp_promote(a, b, force_floating=True, xp=xp)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "example.py", line 577, in xp_promote
    args = [(_asarray(arg, dtype=dtype, subok=True, xp=xp) if arg is not None else arg)
             ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "example.py", line 107, in _asarray
    array = np.asanyarray(array, order=order, dtype=dtype)
ValueError: could not convert string to float: np.str_('other')

Practice questions

Practice 1 (Easy): Test discount versus order value

Split delivered orders into those with a discount (discount_pct above 0) and those without, then run a Welch t-test on value.

disc = d.loc[d["discount_pct"] > 0, "value"]
nodisc = d.loc[d["discount_pct"] == 0, "value"]
r1 = stats.ttest_ind(disc, nodisc, equal_var=False)
print(len(disc), len(nodisc), round(disc.mean(), 1), round(nodisc.mean(), 1), round(r1.pvalue, 4))

2687 2704 656.3 654.7 0.9098

Practice 2 (Medium): Is cancellation rate different on weekends?

Create a weekend flag from order_date and test the cancellation rate with a chi-square test.

orders["weekend"] = orders["order_date"].dt.dayofweek >= 5
orders["cancelled"] = orders["order_status"] == "Cancelled"
t = pd.crosstab(orders["weekend"], orders["cancelled"])
chi2b, pb, _, _ = stats.chi2_contingency(t)
print(t)
print(orders.groupby("weekend")["cancelled"].mean().round(4).to_dict(), round(pb, 4))

cancelled  False  True 
weekend                
False       3641    238
True        1989    132
{False: 0.0614, True: 0.0622} 0.9369

Practice 3 (Hard): A permutation test on a custom statistic

Use a permutation test: shuffle the Prime label 5,000 times and see how often the shuffled difference in the 75th percentile of order value is as large as the observed one.

rng = np.random.default_rng(10)
vals = d["value"].values
labels = d["is_prime"].values
obs = np.percentile(vals[labels], 75) - np.percentile(vals[~labels], 75)

perm = np.empty(5000)
for i in range(5000):
    shuffled = rng.permutation(labels)
    perm[i] = np.percentile(vals[shuffled], 75) - np.percentile(vals[~shuffled], 75)

p_perm = (np.abs(perm) >= abs(obs)).mean()
print(round(obs, 1), round(p_perm, 4))

18.0 0.5172

Key takeaways

• State H0, choose the test, compute p, then report an effect size and interval.
• A p-value measures surprise under the null; it is not the probability the null is true.
• Use Welch t-test for means, Mann-Whitney for skewed data, chi-square for categories, ANOVA for several groups and a two-proportion test for conversion.
• Not significant does not mean no effect; check power and sample size.
• Count your comparisons; apply Bonferroni or pre-register one primary metric.
• For experiments, randomise, fix the sample size in advance and do not peek.