t-tests on different data: chick feeds & textbook prices¶

Part 2 · Lesson 15 · another dataset · data: Chickwts · Textbooks (paired)

↩ A companion to Lesson 15: t-tests: one-sample, two-sample & paired.

↩ Back: Lesson 15: t-tests: one-sample, two-sample & paired · ↑ Course map · Also: The code behind it →

The core t-tests lesson met the t-distribution and three everyday tests — one-sample, two-sample, and paired — using guinea-pig tooth growth and 12 timed swimmers. The ideas don't depend on those datasets, so here we run the very same machinery on two fresh ones and watch the logic repeat:

• Chickwts — a clean, near-balanced experiment: baby chicks were fed one of several diets and then weighed. We'll compare two feeds (casein vs horsebean) with a two-sample test, where the effect is huge and the verdict never in doubt.
• Textbooks — a different flavour of paired design: the same 73 textbooks, each priced two ways — at the campus bookstore and on Amazon. Because each book is its own control, pairing strips away the enormous book-to-book price spread.

The punchline mirrors the swim data but on prices: when two numbers come from the same unit, pairing can turn a non-result into a slam dunk. Run each grey cell with Shift+Enter, then tinker.

New to notebooks? The grey boxes below are code. Click one and press Shift + Enter to run it — the result (a number, a table, or a chart) appears right underneath. Run them top to bottom. You do not need to understand the first "Setup" cell — it just loads the tools and the data, so run it and move on. And you can't break anything: re-running a cell is always safe. (Brand new to all this? The five-minute Lesson 0 — How these notebooks work shows you the basics first.)

1) A quick refresher: t, not z, and the cost of estimating $\sigma$¶

When we compare means we build a t-statistic — a signal-to-noise ratio of "how far is the mean from what the null claims?" divided by "how much does a mean of this size typically wobble?":

$$t = \frac{\text{estimate} - \text{null value}}{\text{standard error}}.$$

It looks just like a z-score, but the standard error is built from the sample spread $s$, not the true population spread $\sigma$ (which we never have). Because $s$ is itself an estimate that wobbles, the honest reference curve is the t-distribution — the normal's more cautious cousin, with heavier tails when data are scarce. Its one knob is the degrees of freedom $df$: small $df$ means fat tails (extra caution), and as $df$ grows the t-curve slims into the normal. Here is that picture again, since it governs every test below.

The red df = 3 curve has visibly heavier tails: with little information the cutoff for "surprising" is pushed out to $t^{*} = 3.18$ instead of the normal's $1.96$. By df = 40 the two curves have all but merged. Takeaway: the t-distribution simply pays an honest tax for not knowing $\sigma$, and that tax fades as the sample grows.

2) One-sample t-test — a building block¶

Before comparing two things, recall the simplest test: is one group's mean different from a target number? The chicks were weighed after six weeks; suppose a hatchery's nominal six-week target weight is 250 grams. Take just the casein-fed chicks — is their mean weight really different from 250?

• Null hypothesis $H_0$: the true casein mean equals the target, $\mu = 250$.
• Alternative $H_a$: it does not, $\mu \neq 250$ (a two-sided test).

The recipe, simulation-first: pretend the null is true, manufacture the world of t-statistics it would produce, and see how unusual our real data looks against it.

Build the null distribution by simulation¶

If $H_0$ were true and the weights were roughly normal, samples of size $n$ drawn from a population centered at $\mu_0 = 250$ would scatter t-statistics around 0. We don't know the true shape, so we use a faithful stand-in: recenter our own sample so its mean is exactly 250 (forcing $H_0$ to hold), then bootstrap-resample it (draw a fresh sample of the same size by picking values from it at random, with repeats allowed) thousands of times and recompute $t$ each time. The pile of simulated t's is the null distribution — no formula required.

The grey histogram of simulated null t-values lands right on top of the blue textbook t-curve — that curve is not magic, it is the shape this simulation produces. Our red line sits well out in a thin tail, so the casein mean is comfortably above the 250-gram target. Confirm with the one-line library call.

Reading it three ways:

• The math: $t = (\bar{x}-\mu_0)/(s/\sqrt{n})$ on $df = 11$, giving a small p-value.
• The meaning: if the true casein mean really were 250 g, a sample this far from 250 would be rare — so we reject $H_0$.
• The real world: the casein-fed chicks weigh clearly more than 250 g on average, and the 95% CI for their mean excludes 250, telling us not just whether but roughly how much heavier.

The CI and the test are the same coin. A two-sided test rejects $H_0\!: \mu=\mu_0$ at the 5% level exactly when the 95% CI excludes $\mu_0$. They never disagree.

The one-sample test is the workhorse engine; the two-sample and paired tests below are both built from it.

3) Two-sample t-test — comparing two independent groups¶

Now the real question Chickwts was designed for: does the diet matter? Compare the casein chicks against the horsebean chicks. These are different chicks, so the two groups are independent — the classic two-sample setup.

• $H_0$: the two feeds give the same true mean weight ($\mu_{\text{casein}} = \mu_{\text{horsebean}}$).
• $H_a$: the means differ.

We'll use Welch's t-test (equal_var=False), the safe default that does not assume the two groups share the same variance. Build the null first — this time by shuffling labels.

The permutation idea — what if the feed labels meant nothing?¶

Here's the cleanest way to picture the null "the feed doesn't matter." If the casein/horsebean label truly had no effect, it's just a sticker on each chick — we could shuffle the labels at random and the gap between the two groups should be no bigger than what we see. So: pool all the weights, repeatedly deal them back into a fake "casein" pile and "horsebean" pile, and record the difference in means each time. That pile of shuffled differences is the null distribution.

A fine point. Label-permutation is the exact-test analog of the pooled (equal-variance) two-sample t-test, not strictly Welch's. The groups here are nearly balanced (12 vs 10), so the pooled and Welch tests very nearly coincide and the permutation p-value tracks Welch closely.

Reading it: casein chicks averaged about 163 grams heavier than horsebean chicks, with $t \approx 7.3$ and $p \approx 7 \times 10^{-7}$ — and the 95% CI for the difference sits far above zero. The observed gap is so large that in 20,000 random label-shuffles we essentially never saw anything like it.

This is the opposite extreme from the core lesson's OJ-vs-VC comparison, which was borderline ($p\approx0.06$). Here the effect is enormous and the verdict is never in doubt:

A clean designed experiment with a big effect. Random assignment of chicks to feeds means the only systematic difference between the groups is the diet, and the gap is so wide relative to the within-group spread that even tiny samples (10–12 birds) settle it decisively. Casein is a far better feed than horsebean.

4) Paired t-test — when the two numbers come from the same unit¶

The two-sample test treats the groups as unrelated. But sometimes each unit gives you both numbers — a before/after, a left/right, the same item measured two ways. That's a paired design, and ignoring the pairing throws away its biggest advantage.

The Textbooks data is a different flavour of pairing from the swimmers in the core lesson: instead of the same person timed twice, it's the same book priced twice — once at the UCLA campus store (ucla_new) and once on Amazon (amaz_new). Because it's the same book in both columns, the right move is to collapse each pair to its difference and ask whether the mean difference is zero.

• $H_0$: the mean within-book price difference is 0 (the two sellers charge the same on average).
• $H_a$: the mean difference is not 0.

On average the campus store charges about 12.76 dollars more per book. A paired t-test is nothing more than a one-sample t-test on the differences, testing them against 0 — the exact recipe from Section 2. Recenter the differences to have mean 0 (forcing $H_0$), bootstrap, and read off the p-value.

The campus-store premium is unmistakable: $t \approx 7.65$, $p \approx 7 \times 10^{-11}$, and a 95% CI of roughly 9.5 to 16 dollars that sits comfortably above zero. In 20,000 simulated null worlds we essentially never saw a gap this big by chance. On average, buying these books at the campus store costs about 12.76 dollars more apiece than buying them online.

5) Why pairing matters — the headline demonstration¶

Here's the punchline of the whole lesson. Watch what happens if we ignore the pairing and (wrongly) run a two-sample test, treating the 73 campus prices and 73 Amazon prices as two unrelated groups.

Same data, opposite verdict. The correct paired test screams "the campus store is pricier" ($p \approx 7\times10^{-11}$); the unpaired test shrugs ($p \approx 0.16$). How can ignoring the pairing flip a slam-dunk result into a null one? The answer is the enormous book-to-book price variation — cheap paperbacks next to expensive hardcover texts.

On the left, the raw prices fan out wildly — some books near 10 dollars, others well over 100 — and an unpaired test sees those two big overlapping clouds and concludes it can't tell them apart. But almost every connecting line tilts the same way (down toward Amazon). The right panel shows the trick: once you look at each book's own price gap, the 73 differences huddle clearly above zero. Pairing cancels the between-book variation, leaving only the clean campus-store premium — which is why the paired SE was about 5x smaller.

The real-world lesson: if your data are naturally matched (before/after, twins, left/right, the same item measured two ways), use the pairing. As with the swimmers in the core lesson, it is often the difference between detecting an effect and missing it — and "paired vs two-sample" comes down to one question: do the two numbers come from the same unit? For the chicks, no (different birds → two-sample); for the textbooks, yes (same book → paired).

6) Assumptions & robustness — when can you trust a t-test?¶

The t-test isn't unconditional. Its key assumptions:

1. Independent observations (within each group). The whole point of the paired test is to handle the one dependence we expect — the link between a unit's two measurements — by reducing to one number per unit.
2. Roughly normal data — specifically, the sampling distribution of the mean should be near normal. For the paired test this means the differences should be roughly normal, not the raw values.
3. For the two-sample test, equal variances are not required if you use Welch's version (the default we used).

The good news is robustness: thanks to the Central Limit Theorem, the t-test holds up well even for non-normal data once $n$ is moderately large, because the mean becomes normal regardless. It's most fragile with small samples of skewed data with outliers. Let's check the 73 textbook differences with a normal-quantile (Q-Q) plot — if the points hug the line, normality is a fair assumption.

Now you try¶

Predict the result first, then change a number and re-run:

1. A different one-sample target. In Section 2 the casein target is mu0 = 250.0. Change it to 320.0 (close to the casein mean) and re-run that cell and the scipy cell. Does the casein mean now look different from the target, or consistent with it? Watch the p-value and whether the 95% CI contains your new target.

2. A closer two-sample contest. Section 3 compared casein vs horsebean — a blowout. Try two feeds that are much nearer in quality. In a new cell, run

   a = ch[ch['feed']=='casein']['weight']
   b = ch[ch['feed']=='meatmeal']['weight']
   stats.ttest_ind(a, b, equal_var=False)

   Is the difference still a slam dunk, or does it get hard to call? (Predict before you run.)

3. Break the pairing on purpose. In Section 5, the magic came from the book-to-book correlation (r = 0.98). Add a cell that scrambles the Amazon prices before pairing: stats.ttest_rel(ucla, rng.permutation(amaz)). With the pairing destroyed, does the paired test's advantage survive?

4. Re-roll the simulations. Change np.random.default_rng(17) to any other number and re-run the simulated null distributions in Sections 2–4. The simulated p-values should shift only in the last decimal — confirming the result isn't an artifact of one lucky seed.

What you learned¶

• Why t, not z: when the population spread $\sigma$ is unknown we estimate it with $s$, adding uncertainty; the t-distribution absorbs that with heavier tails governed by the degrees of freedom, and it melts into the normal as $n$ grows — exactly as in the core lesson, on different data.
• A one-sample t-test compares one group's mean to a target value and is the building block for the rest; the matching 95% CI rejects the target exactly when it excludes it.
• A two-sample (Welch) t-test compares two independent groups. On Chickwts (casein vs horsebean) the effect was enormous — $t\approx7.3$, $p\approx7\times10^{-7}$ — a clean designed experiment where even a dozen chicks settle the question, the opposite of a borderline result.
• A paired t-test is just a one-sample t-test on the within-unit differences — use it whenever the two numbers come from the same unit. Here that unit was the same textbook priced two ways, giving a mean campus premium of about 12.76 dollars ($t\approx7.65$, $p\approx7\times10^{-11}$).
• Pairing matters: on the textbook prices the correct paired test ($p\approx10^{-10}$) detected a premium the wrong unpaired test missed ($p\approx0.16$), because pairing subtracts away the huge book-to-book price spread (prices correlated $r\approx0.98$) and shrank the standard error about 5-fold.
• Every test here was first built by simulation (bootstrapping a recentered sample, or shuffling group labels) and the textbook formula merely agreed — the t-curve is the shape the simulation produces.
• Assumptions: independent observations and roughly-normal means (or differences); the t-test is robust for moderate $n$ thanks to the CLT, and shakiest for small, skewed, outlier-ridden samples — check with a Q-Q plot.

---

↩ Back to the lesson: Lesson 15: t-tests: one-sample, two-sample & paired · ↑ Course map · Also: The code behind it