---
name: statistical-testing
description: >
  Selects and runs the right significance test for a research comparison, reports
  effect size and confidence intervals alongside the p-value, and controls false
  positives across a banner. Use when someone says "is this difference
  significant", "can we say this is a real difference", "which test should I
  use", "run sig testing on the cross-tabs", "is 42% versus 37% meaningful",
  "why did nothing come out significant", "how many of these differences are
  real", or asks whether a subgroup genuinely differs from another.
category: 05 Quantitative Analysis
ref: "05.02"
tier: 0
inherits: [K2, K3, K4, K5]
---

# Statistical Testing

## 1. One-line description
Establishes whether an observed difference or relationship is larger than sampling noise, how large it actually is, and whether it is large enough to matter, and refuses to answer any of the three when the data does not support the question.

## 2. What this skill is used for

**The research problem it solves.** Two failures dominate quantitative commercial research, and they are opposites. The first is claiming a difference that is not there: 44% versus 39% on bases of 120 and 140 gets written up as "younger customers are more likely to switch", and the number is noise. The second is subtler and more expensive: a difference that is statistically significant gets treated as though significance were the same thing as importance. On a sample of 8,000, a two-point gap will test significant, and it will be reported with the same weight as a twenty-point gap, because the only number in the output was a p-value. Underneath both sits the structural problem of cross-tab work, which is that a standard banner generates thousands of tests and then the analyst reads off the ones that came up. This skill supplies the discipline that prevents all three: the right test for the design, the effect size and interval alongside the p-value, and a fixed comparison set decided before the tests are run.

**Where it sits in the research lifecycle.** After the data is clean, weighted and described, and after the comparison set has been specified in the analysis plan. Before findings are turned into insight and before anything is drafted for a client.

**Typical use cases.**
- Deciding whether a subgroup difference in a cross-tab is reportable.
- Wave-on-wave testing in a tracker, where a movement has to be distinguished from drift.
- Comparing a test cell against a control in an experiment or a concept test.
- Judging whether a non-significant result means "no difference" or "not enough sample".
- Setting the testing rules for an entire tabulation job before it runs.
- Auditing a deck that claims differences, to check which of them survive.

**Who uses it.** Research executives and analysts running tabulations; research managers and directors reviewing claims before they leave the building; client-side insight managers challenging a supplier's significance flags; product and UX researchers running experiments without a statistician nearby.

## 3. When to use it

- A difference between two groups, two waves or two conditions is about to be described as real, and someone should check.
- An analysis plan needs its testing rules set: which comparisons, which test, which threshold, which correction.
- A cross-tab has come back covered in significance flags and someone has to decide which of them are findings.
- A result came back non-significant and the team is about to conclude there is no difference.
- A difference is significant and small, and someone has to decide whether it belongs in the report.
- An experiment or A/B comparison needs a pre-registered primary outcome and a decision rule.
- A claim in a draft report ("notably higher", "clearly ahead", "a real improvement") needs evidence behind it or removal.

## 4. When NOT to use it

- **The sample is a census of the population of interest.** If every branch, every employee or every account is in the data, the difference you observe is the difference. A significance test asks whether an observed gap could have arisen from sampling variation, and there was no sampling. Report the difference and its practical size. If someone insists on a test, they are implicitly treating the census as a sample from a hypothetical superpopulation of possible outcomes, which is a defensible but unusual position that must be stated explicitly rather than smuggled in.
- **The comparison was found by looking.** A subgroup difference discovered while scanning the tables is a hypothesis, not a finding, and a p-value calculated on it is not interpretable in the usual way because the comparison was selected for being extreme. Report it as an observation with its base sizes and label it for validation. See Step 6 and **01.07 Analysis Plan Development**.
- **The bases are too small to carry the question.** Below n=30 in either group, report counts or verbatim, not percentages, and do not test (**K4 §7**). Between 30 and 99, a test is arithmetically possible and practically close to useless: the minimum detectable difference is so large that only enormous effects will register, and a null result carries no information. Say what the study could have detected rather than reporting a p-value that will be misread.
- **The groups overlap.** Standard two-sample tests assume independent groups. Testing a subgroup against the total that contains it, or testing "users of product A" against "users of product B" when a respondent can be both, violates the assumption and inflates significance. Either test the subgroup against its complement, or use a test built for dependent proportions.
- **The data is weighted and no effective base has been calculated.** Running a standard test on weighted counts treats the weights as though they were extra respondents, and it will produce significance that does not exist. Get the effective base first, from **04.05 Weighting and Base Management**.
- **The question is about relationship strength, prediction or attribution rather than difference.** Correlation, regression, driver work and anything approaching a causal claim belong to **05.06 Correlation, Regression and Causal Claim Control**. A significance test on a difference does not license "X drives Y", and no amount of significance ever will.
- **The measurement is not comparable across the groups being compared.** Different question wording, different mode, different sample source, a scale that means something different in two markets. The test will still run. It will be testing an artefact. Fix the comparability question first, and if it cannot be fixed, report the two numbers separately and say why they are not compared.
- **The purpose is to find something to say.** Where the brief has produced nothing and the analyst is testing widely in the hope that something emerges, no correction and no threshold will make the output honest. That is a design problem, and the correct response is to report that the study did not answer the question.

## 5. Required inputs

**Required. Without these the skill cannot run.**
- **The specific comparison, stated as a question.** Which measure, which two (or more) groups, on which base. "Is satisfaction higher among long-tenure customers than short-tenure customers on Q7?" not "test the tables". Without this, stop and ask.
- **The data type and the design.** Whether the measure is a proportion, a mean of an interval measure, an ordinal rating or a count, and whether the groups are independent or the same respondents measured twice. This determines the test and cannot be inferred safely from a variable name (**K4 §6.2**). Without this, stop and ask.
- **Unweighted base sizes for every group in the comparison**, and weighted bases plus the weighting scheme where the data is weighted. Without these, stop.
- **The sampling approach.** Probability, quota, river, panel, convenience, or unknown. This determines whether inference to a population is licensed at all. If unknown, proceed on the assumption of non-probability and state it.

**Optional, and what each one adds.**
- **The analysis plan with pre-specified comparisons (01.07).** This is the single most valuable optional input, and it converts the whole exercise from screening into confirmation. With it, a small number of comparisons are tested at the nominal threshold and the results stand as findings. Without it, everything is exploratory and must be labelled that way.
- **A stated materiality threshold.** The difference the business would act on. Without it, practical significance becomes the analyst's opinion rather than an agreed standard, and the report will argue about it later.
- **Raw respondent-level data rather than a tabulation.** Allows variance to be calculated properly, allows non-parametric and exact tests, allows bootstrap intervals, and allows the distribution to be inspected rather than assumed.
- **The design effect or replicate weights.** Allows correct variance estimation on a complex sample instead of an approximation.
- **Previous waves or comparable studies.** Establish normal variation for the measure, which is the only way to tell a real movement from the measure's own noise floor.
- **A power calculation from the design stage.** Lets a null result be interpreted rather than merely reported.

## 6. Questions to ask before starting

1. **Was this comparison specified before the data was seen?** Determines whether the result is confirmatory or exploratory, which changes both the correction strategy and the language used in the report. Default if unanswered: treat as exploratory, and label every result as hypothesis-generating.
2. **How was the sample obtained?** Determines whether a p-value carries inferential meaning or is being used as a rough noise filter. Default: assume non-probability, test as a heuristic, disclose it, and report no margin of error (**K4 §7**).
3. **Is the data weighted, and what is the effective base?** Determines the denominator of every test. Default: if weights exist but no effective base has been supplied, calculate it and state the design effect. If neither is available, do not test.
4. **How large a difference would change a decision?** Sets the materiality threshold against which practical significance is judged. Default: state that no threshold was agreed, report the effect size, and flag the materiality judgement as a **K5** review point.
5. **Are the same respondents in both groups?** Determines paired versus independent tests, and getting this wrong is the commonest test-selection error in tracker and pre/post work. Default: assume independent only if the design makes it impossible for a respondent to appear twice, otherwise ask.
6. **How many comparisons will be run in total across this job?** Sets the false-positive arithmetic and the correction strategy. Default: count them, and if the count exceeds roughly twenty, correction or tiering is required.
7. **What threshold will be used, and is it one-sided or two-sided?** Determines the decision rule. Default: 95%, two-sided, stated in the output. A one-sided test requires a directional hypothesis fixed in advance and is rarely justified in commercial work.

## 7. Step-by-step methodology

**Step 1. Turn the request into a testable comparison.** One sentence, containing a measure, a base definition for each group, a direction of interest and nothing else: "Is the proportion selecting option 3 at Q12 higher among customers of two years or more than among customers of under one year?" A request that cannot be written this way is not yet a test, and the usual failures are questions about relationship strength (**05.06**) or composition (**05.03**). A correct result is a numbered list of one-sentence comparisons.

**Step 2. Establish whether the data licenses inference at all.** Four checks, before anything is selected. **Sampling:** on a probability sample, tests carry their usual meaning. On a non-probability sample, which covers most commercial online research, the theoretical basis of the p-value does not hold, because the test assumes random selection from a population. Practice is genuinely split between not testing at all and testing as a rough noise filter. This skill takes the second position, on condition that the result is called a noise filter, never population inference, and that no margin of error is reported. **Weighting:** compute the effective base, n_eff = (sum of weights)² / (sum of squared weights), and the design effect, n / n_eff, and use n_eff in every test. **Overlap:** confirm no respondent sits in both groups. **Base adequacy:** apply the **K4 §7** thresholds. Record the inferential basis, the effective bases, and any comparison ruled out.

**Step 3. Select the test from the design and the data type.** Nearly mechanical once Steps 1 and 2 are done.

| Measure | Two independent groups | Three or more independent groups | Same respondents, two measures | Same respondents, three or more |
|---|---|---|---|---|
| Proportion or binary | Two-proportion z-test (equivalently a 2x2 chi-square) | Chi-square test of independence, then targeted pairwise tests | McNemar's test | Cochran's Q |
| Categorical, more than two levels | Chi-square test of independence | Chi-square test of independence | Stuart-Maxwell | Generalised marginal test |
| Mean of an interval measure | Welch's t-test, the default, since it does not assume equal variances | One-way ANOVA (Welch's if variances differ), then post-hoc | Paired t-test | Repeated-measures ANOVA or mixed model |
| Ordinal or non-normal | Mann-Whitney U | Kruskal-Wallis, then Dunn's test | Wilcoxon signed-rank | Friedman |

Two notes. An **omnibus test** (chi-square across a table, ANOVA across groups) says something differs somewhere, not what: it is a gate, not a finding, and must be followed by targeted comparisons. A **net score** computed as one percentage minus another from the same question does not have the variance of a single proportion, so testing it as one overstates significance.

**Step 4. Check the assumptions, and decide what to do when they fail.** Proportion tests need roughly ten expected successes and ten expected failures per group for the normal approximation; below that, use an exact method. Chi-square wants expected cell counts of five or more; where over a fifth of cells fall below five, or any falls below one, collapse categories on a substantive basis or use Fisher's exact test. For t-tests and ANOVA what matters is normality of the sampling distribution of the mean, not of the raw data, so the central limit theorem covers most bases above thirty per group; the real threats are severe skew, heavy tails and outliers. Mann-Whitney tests stochastic dominance, and tests medians only when the distributions share a shape. A wobbling assumption is not automatically a reason to switch to a rank test, which answers a different question and surrenders the interpretable mean difference; a bootstrap interval on the mean difference usually serves better.

**Step 5. Fix the comparison set before running anything.** This is where commercial cross-tab work fails. Running m independent tests at a 5% threshold where nothing truly differs gives a probability of at least one false positive of 1 minus 0.95^m: 40% at ten tests, 64% at twenty, 92% at fifty, above 99% at a hundred. A modest banner of twelve columns produces sixty-six pairwise comparisons per row; across sixty rows that is 3,960 tests, of which roughly 198 will flag with nothing there at all. Count the tests before running them, record the count, then choose a strategy.

**Step 6. Choose the multiplicity strategy, and prefer pre-specification to correction.** Three tiers, and the first is the real answer.

- **Pre-specified comparisons.** A small set named in the analysis plan before the data was seen, with a designated primary, tested at the nominal threshold without correction because the family is small and was not chosen by looking. Set up in **01.07 Analysis Plan Development**, and it is what makes the other tiers rarely necessary.
- **Correction within a defined family.** **Bonferroni** (each tested at alpha divided by m) is simple, always valid, and loses power sharply as m grows. **Holm** controls the same family-wise error rate and is uniformly more powerful, so there is no reason to prefer Bonferroni once Holm is available: order p-values ascending, compare the i-th to alpha divided by (m minus i plus 1), stop at the first non-rejection. **Benjamini-Hochberg** controls the false discovery rate, the expected share of false positives among results declared, which is the right target for exploratory screening where a few false leads are tolerable and missing everything is not. After ANOVA use Tukey's HSD for all pairs, or Dunnett's against a single control, and define the family explicitly.
- **Exploratory, uncorrected, labelled.** Report as screening output, stating how many tests were run and how many positives chance alone would produce.

Correction limits the damage of a fishing expedition. It does not turn one into a confirmatory result.

**Step 7. Run the test and record the full result.** Test name, statistic, degrees of freedom where applicable, the exact p-value (not "p<0.05"), the threshold, one-sided or two-sided, and the bases used. Never report a p-value without the test that produced it.

**Step 8. Compute the effect size. This is not optional.** A p-value confounds effect size with sample size. For proportions, always report the **difference in percentage points**; its standardised companion is **Cohen's h**, the difference between arcsine-transformed proportions, which matters because a ten-point gap near 50% is a far smaller effect than one between 3% and 13%. For means, **Cohen's d**, the mean difference over the pooled standard deviation, expressing the gap in units of how much people vary. For a chi-square table, **Cramér's V**, scaled zero to one so association strength compares across tables of different sizes. For ANOVA, **eta-squared**, the share of total variance explained by group membership; prefer it to partial eta-squared, which is inflated by the model's other terms. For binary outcomes in a model, the **odds ratio**, which exaggerates relative difference against a risk ratio whenever the outcome is common. For rank tests, the **rank-biserial correlation** or the probability of superiority.

**Step 9. Interpret magnitude in context, not by label.** The conventional thresholds (d of 0.2, 0.5, 0.8 as small, medium, large) were offered as a last resort for fields with no established scale, and applying them mechanically is among the commonest misuses in applied work. A d of 0.2 in a mature market where nothing has moved in five years may be the largest effect anyone will find; a d of 0.8 on an unvalidated single item may be noise. Calibrate instead against the measure's own wave-on-wave variation, the effects comparable interventions produced, and the difference the business would act on.

**Step 10. Compute and lead with the confidence interval.** An interval on the difference carries everything the p-value carries and adds what it omits: plausible magnitude and precision. "Plus 8 points, 95% CI plus 2 to plus 14" and "plus 8 points, 95% CI minus 1 to plus 17" tell different stories. Interval the **difference**, not each group separately: overlapping intervals on the individual estimates do not imply a non-significant difference, and this error makes analysts discard real findings (non-overlap does imply significance; the converse does not hold). State the interpretation correctly: over repeated samples, 95% of intervals built this way contain the true value. It is not a 95% probability that this interval contains it, and not a range containing 95% of respondents.

**Step 11. Interpret a null result honestly, using power.** A non-significant result is not evidence of no difference. It means the study could not distinguish the gap from zero, which is what a real difference looks like when the sample is too small. Before writing anything about a null, calculate the minimum detectable effect: for two proportions near 50% at 80% power and a 95% two-sided threshold, roughly 20 points at n=100 per group, 14 at n=200, 9 at n=500 and 6 at n=1,000; for means, roughly 0.28 standard deviations at n=200 per group. Then say what the study could and could not have seen.

**Step 12. Assemble the claim, keeping the two questions separate.** Significance answers "could this be noise?". Practical significance answers "is this big enough to matter?". They are independent, and all four combinations occur. Significant and material is a finding. Significant and trivial is a technical result, and it is what large samples produce in quantity. Non-significant and large is an underpowered study, the most mishandled of the four, written up as "no difference" when it means "we could not tell". Non-significant and trivial is the only cell where "no meaningful difference was found" is close to fair, and even there it is bounded by what the design could detect. Write the claim to reflect its cell, in the register of **K3 §4**.

## 8. Analytical framework

    Comparison specified → Inference licensed? → Test selected → Assumptions checked
        → Result → Magnitude → Materiality → Claim

Read it as a sequence of gates rather than a checklist. A comparison that was not specified in advance passes through the second gate labelled as exploratory and can never be relabelled downstream. A comparison on a sample that does not license inference passes with its p-value described as a noise filter and no margin of error attached. A significant result that fails the materiality gate does not become a headline. A non-significant result exits the sequence carrying the study's minimum detectable effect, not the word "no".

The gate that is skipped most often is magnitude, and the gate that is skipped most damagingly is materiality, because it is the only one that requires knowledge of the business rather than knowledge of the data. That is a **K5 §2.1** judgement and it returns to a human.

## 9. Output format

**The significance test log.** One row per test run, including tests that produced nothing, because the count of tests run is what makes the false-positive arithmetic checkable.

| Ref | Comparison | Measure | Base A (n, n_eff) | Base B (n, n_eff) | Value A | Value B | Difference | Test | Statistic | p | Threshold | Correction | Effect size | 95% CI on difference | Pre-specified? | Verdict |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

**The reporting line.** Every claimed difference in a client-facing document carries these elements, in prose or in a footnote, and a difference missing any of them is not reportable:

> Long-tenure customers were more likely to select option 3 than short-tenure customers: 46% (n=412) versus 38% (n=389), a difference of 8 percentage points, 95% CI 1.2 to 14.8 points, two-proportion z-test, z=2.29, p=0.022, tested at 95%. Cohen's h = 0.16. Pre-specified primary comparison, no correction applied. The difference exceeds the 5-point threshold agreed in the analysis plan.

**Where a difference is observed but not tested**, per **K4 §3.1**: "46% versus 38% (n=412 / n=389), untested."

**Where the evidence is thin**, the format must not force a result into a slot that has none. Use these forms rather than a p-value:
- `[not tested: base below 30]` with the counts shown instead of percentages.
- `[not tested: groups overlap]` with the reason named.
- `[not tested: no effective base available for weighted data]`.
- `[non-significant, and the study could not have detected a difference smaller than X points]`.
- `[exploratory: one of N comparisons run, expected false positives at this threshold approximately M]`.

**The testing note**, appearing once in any deliverable that contains tested comparisons: the sampling basis and whether inference to a population is claimed, the weighting status and design effect, the threshold, the test family and correction approach, the total number of tests run, and the materiality threshold if one was agreed.

## 10. Quality checks

Run before any tested result is presented. **K4 §8** runs anyway; these are the checks specific to this task.

1. Does every reported difference carry both group bases, unweighted and effective?
2. Is the test named, with its threshold and whether it was one-sided or two-sided?
3. Does the design match the test: paired data on a paired test, independent groups on an independent test?
4. Have the assumptions been checked rather than assumed, and is any departure logged with what was done about it?
5. Does every tested difference carry an effect size and a confidence interval on the difference, not on each group separately?
6. Is the total number of tests run recorded, and is the correction approach or the exploratory label stated?
7. Was every comparison reported as a finding pre-specified, and if not, is it labelled exploratory in the same sentence?
8. For every significant result, has practical significance been assessed separately against a stated threshold?
9. For every null result, is the minimum detectable effect stated, and does the wording avoid claiming equivalence?
10. Has any omnibus result been reported as though it identified which groups differ?
11. Is any subgroup being compared against a total that contains it?
12. Are p-values reported exactly rather than as inequalities, and free of the false precision **K3 §4.4** prohibits?
13. Does the word "significant" appear anywhere it means "large" rather than "statistically significant"?
14. If the sample is non-probability, is that stated, and is the absence of a margin of error explicit?

## 11. Common failure modes

| Failure | How to recognise it | How to prevent it |
|---|---|---|
| **Significance read as importance** | A two-point gap on a large sample carrying the same narrative weight as a twenty-point gap | Effect size and interval mandatory in the reporting line; materiality threshold agreed in advance |
| **The banner harvest** | Findings that are a list of the cells a tabulation flagged, with no prior hypothesis behind any of them | Count the tests before running them, pre-specify the comparison set, apply Step 6 |
| **Null read as equivalence** | "There is no difference between segments" on bases of 90 and 110 | Minimum detectable effect stated with every null; equivalence testing where equivalence actually matters |
| **Paired data tested as independent** | Pre and post on the same respondents run through a two-sample test | Steps 1 to 3 fix the design before the test is chosen; assume dependence unless the design forbids it |
| **Weighted counts tested directly** | Base sizes that are non-integer or suspiciously large, significance everywhere | Effective base from **04.05** used in every test, design effect reported |
| **Overlapping groups compared** | A subgroup tested against the total, or two multi-code groups that share respondents | Explicit overlap check in Step 2; test against the complement instead |
| **Omnibus mistaken for specific** | "The chi-square was significant, so segment C is different" | Treat omnibus as a gate; require a named pairwise test for any named group claim |
| **Threshold shopping** | A result reported at 90% when 95% was used everywhere else, with no reason given | Threshold fixed in the analysis plan and stated once for the whole job |
| **Overlapping interval error** | A real difference discarded because the two group intervals overlapped | Interval the difference, never compare two separate intervals by eye |
| **AI: asserting significance without a test** | Words like "notably", "clearly ahead", "markedly" attached to untested gaps | **K4 §3.1**; every comparative adjective traced to a logged test or downgraded to plain description |
| **AI: producing a plausible p-value** | A precise p-value appearing with no statistic, no base and no test named | Nothing enters the log without the statistic and bases that generated it; an unproducible number is `[not available]` |
| **AI: mechanical effect-size labelling** | "A medium effect (d=0.52)" with no reference to the measure's own variation or the decision at stake | Step 9; the label never appears without the raw difference and a contextual benchmark |
| **Caveat shedding in transit** | An exploratory label present in the working file, absent on the summary slide | Caveats travel with the claim into every downstream document (**K3 §7**) |

## 12. AI guardrails

Universal prohibitions are inherited from **K4** and are not repeated here. In particular, **K4 §3.1** (never imply significance without testing) and **K4 §3.2** (never convert correlation into causation) govern this skill and are binding in full. The following are specific to statistical testing.

1. **Never report a test result you did not compute from supplied data.** A statistic, a p-value, a confidence interval or an effect size that cannot be traced to a calculation on real numbers is fabrication under **K4 §2.1**, regardless of how reasonable it looks. If the calculation cannot be performed, say what input is missing.
2. **Never select a test from the shape of a table alone.** Whether respondents are the same people across columns is not visible in a cross-tab, and guessing it wrong invalidates the result. Ask, or state the assumption and mark the result provisional.
3. **Never suppress the count of tests run.** Where multiple comparisons were screened, the number tested is part of the result. Reporting only the positives is cherry-picking under **K4 §4.2** even when every individual test was correctly executed.
4. **Never present an uncorrected exploratory result in the language of a finding.** The label travels with the claim. Downgrading confidence at the point of writing is not sufficient if the exploratory status is dropped.
5. **Never convert a non-significant result into a statement of no difference**, and never write "no significant difference" without the minimum detectable effect beside it.
6. **Never attach a margin of error or a population inference to a non-probability sample**, and where a test is run on one as a noise filter, say so in the same passage.
7. **Never round a p-value across the threshold**, and never describe a result as "approaching significance", "trending toward significance" or "marginally significant". A result either met the pre-stated threshold or it did not.
8. **Never re-test a comparison at a different threshold, on a different base, or with a different test after seeing a result you did not want.** If an initial test is superseded, the original stays in the log with the reason for the change.
9. **Never assert practical significance on the analyst's own authority.** Whether a difference is large enough to act on is a business judgement and returns to a human under **K5 §2.1**.

## 13. Best-practice principles

1. **Decide the comparisons before you see the data, and the rest of this becomes easy.** Almost every problem in this skill, from multiplicity to threshold shopping to the exploratory-labelling argument, dissolves when a small set of comparisons was named in advance. Correction is damage control for having failed to do this.
2. **The confidence interval is the better default output.** It contains the test result, and it answers the question the reader actually has, which is how big the difference might be. Give the interval first and the p-value second.
3. **A p-value is a statement about data under an assumption, not about the world.** It is not the probability the null hypothesis is true, not the probability the result arose by chance, and not the probability the finding will replicate. One minus p is not the probability the difference is real. These misreadings are near-universal in commercial audiences, which means the analyst has to write in a way that does not invite them.
4. **Sample size is the hidden variable in every p-value.** Before interpreting any result, know whether you are working at a base where trivial differences will flag or at a base where real ones will hide. The same 5-point gap is invisible at n=150 and inevitable at n=5,000.
5. **Know the measure's own noise floor.** Before calling a wave-on-wave movement real, look at how much the measure has moved historically when nothing happened. A tracker with a three-point ordinary fluctuation cannot report a four-point movement as news, whatever the test says.
6. **Prefer estimation to detection wherever the choice exists.** "How much do these groups differ, and how precisely do we know that?" is a more useful question than "do they differ at all?", and it is the question a decision-maker is really asking.
7. **Do not switch to a rank test reflexively when an assumption wobbles.** Rank tests answer a different question and cost you the interpretable difference in means. A bootstrap interval on the mean difference usually serves better, and outlier inspection often reveals that the assumption failure was a data problem rather than a distributional one.
8. **Report the tests that found nothing.** A null on a pre-specified primary comparison is a finding, and often a more valuable one than the positives, because it closes a question the business was going to keep asking.
9. **Distinguish the three uncertainties.** Sampling uncertainty is what a test addresses. Measurement uncertainty (did the question capture the construct) and coverage uncertainty (did the frame reach the population) are untouched by any test and are frequently larger (**K3 §6**). A highly significant result on a badly worded question is a precise statement about the wrong thing.
10. **The word "significant" should be banned from client-facing prose unless immediately followed by the test.** In ordinary English it means "important", which is exactly the confusion this skill exists to prevent. Write "8 points higher" and put the statistics in the note.
11. **Testing everything is a symptom, not a method.** When an analyst tests hundreds of comparisons, the usual cause is an analysis plan that did not specify what the study was for. The fix is upstream.
12. **A difference that survives testing is still only a finding.** It says the gap is unlikely to be noise. It does not say why the gap exists, and the step from difference to explanation is interpretation under **K2 §2.1**, not a further statistic.

## 14. Worked example

**INPUT**

A fictional consumer energy retailer, Kestrel Energy, runs an annual customer study. Wave 4 has 2,400 respondents from a non-probability online sample, weighted to regional and age targets. The team hands over the standard banner (twelve columns, sixty rows, significance flags at 95%) and asks for a claim: "customers using the new billing tool are significantly more satisfied, and satisfaction is significantly higher in the North region".

**PROCESS**

*Step 1, comparisons.* Two written out. C1: is top-2-box satisfaction at Q9 higher among tool users than non-users? C2: is top-2-box satisfaction at Q9 higher in the North than in other regions? Checking the analysis plan, C1 was pre-specified as a primary comparison. C2 was not; it came from reading the banner.

*Step 2, inference basis.* Non-probability sample, so testing proceeds as a noise filter with that stated and no margin of error reported. Weights give a sum of weights of 2,400 and a sum of squared weights that yields n_eff = 1,905, a design effect of 1.26. All bases converted: tool users n=780 (n_eff 619), non-users n=1,620 (n_eff 1,286).

*The judgement call.* Tool users are self-selected, so C1 compares people who chose the tool against people who did not. The test can say whether the two groups differ; it cannot say the tool caused it, and the requested wording ("has improved satisfaction") is a causal claim this design does not license (**K4 §3.2**). Resolved by keeping the test, reporting it as an association, and flagging the causal question as needing a pre and post or matched design. Referred to **05.06**.

*Steps 3 and 4, test and assumptions.* Both comparisons are proportions on independent groups, so a two-proportion z-test. Expected successes and failures comfortably above ten in all cells. No overlap: a respondent is a user or not, and is in one region.

*Steps 5 and 6, multiplicity.* The banner contains sixty-six pairwise column comparisons per row across sixty rows, 3,960 tests. At a 5% threshold, roughly 198 cells would flag even if nothing differed anywhere. C1 is pre-specified and is tested at 95% without correction. C2 is one of the 3,960 and is treated as exploratory. Benjamini-Hochberg across the twelve regional comparisons on this row is possible, but the honest reading is simpler: C2 was selected for being extreme, so any p-value attached to it overstates the evidence.

*Steps 7 to 10, C1.* Tool users 58% top-2-box, non-users 51%. Difference 7 points. On effective bases: z=2.87, p=0.004. Cohen's h=0.14. 95% CI on the difference, 2.2 to 11.8 points. Against the agreed materiality threshold of 5 points, the estimate clears it but the lower bound of the interval does not, which is the honest way to describe the uncertainty.

*Steps 7 to 10, C2.* North 55%, all other regions combined 52%. Difference 3 points, n_eff 210 versus 1,695, p=0.41. Not significant. Minimum detectable difference at these bases is about 10 points, so a difference of 3 to 9 points would not have been visible. The claim is dropped, and the reason recorded.

**OUTPUT**

One reportable finding, one dropped claim, and a testing note:

> Customers using the self-service billing tool report higher satisfaction than non-users: 58% versus 51% top-2-box at Q9 (n_eff 619 and 1,286; nominal n 780 and 1,620), a difference of 7 percentage points, 95% CI 2.2 to 11.8, two-proportion z-test on effective bases, p=0.004, tested at 95%. Cohen's h = 0.14. Pre-specified primary comparison. The difference exceeds the agreed 5-point materiality threshold, though the lower bound of the interval does not. Tool users are self-selected; this is an association and the design does not establish that the tool caused it.

> Regional differences in satisfaction were tested and none reached the threshold. On a North base of n_eff 210, differences smaller than about 10 points could not have been detected, so this is not evidence that regions are alike.

> Testing note: non-probability sample, weighted (design effect 1.26), effective bases used throughout, 95% two-sided threshold. One pre-specified comparison tested without correction; all other banner comparisons are exploratory, and of the 3,960 pairwise tests in the tabulation roughly 198 would flag by chance alone. No margin of error is reported.

**Researcher review recommended.** Whether a 7-point gap that may be as small as 2 points supports the investment case being built on it is a materiality judgement (**K5 §2.1**), requiring knowledge of what the tool cost and what the business would do differently at each end of the interval.

## 15. Advanced usage

**Complex samples.** Where the design has clustering, stratification or unequal selection probabilities, the effective-base adjustment in Step 2 is an approximation. Correct variance estimation uses Taylor linearisation, jackknife or replicate weights, and the difference matters most exactly where the design effect is largest. If replicate weights are supplied, use them.

**Resampling.** Bootstrap intervals free you from distributional assumptions and extend naturally to statistics with no clean closed-form variance: medians, ratios, net scores, differences of differences, index numbers. Permutation tests are the exact analogue for hypothesis testing and are particularly useful on small or awkwardly distributed data. Both require respondent-level data, and both must respect the weighting and any clustering when resampling.

**Equivalence and non-inferiority.** Where the commercially interesting answer is "these are the same", conventional testing cannot deliver it. Specify an equivalence margin in advance, then use two one-sided tests or check that the confidence interval on the difference lies wholly inside the margin. This is the correct tool for questions like whether a cheaper stimulus performs no worse than the current one.

**Interaction rather than paired comparison.** When the question is whether a difference is bigger in one group than another (whether the gap between users and non-users is wider among younger customers), do not compare two separate tests and note that one was significant and one was not. That comparison is itself a significance test and has to be run as one, as an interaction term in a model. This error is extremely common and reliably produces false claims about which segment an effect is concentrated in. See **05.06**.

**Sequential looks.** In continuously fielding studies and live experiments, checking results repeatedly and stopping when significance appears inflates the false-positive rate substantially. Either fix the sample size in advance or use a formal sequential design with adjusted boundaries.

**Replacing many tests with one model.** Where a banner is generating hundreds of comparisons, a single model with the relevant terms often answers the question better, controls for confounding between banner variables, and produces one set of estimates instead of a wall of flags. This is the natural handover to **05.06**.

## 16. Skill chain

**Recommended previous skills**
- **05.01 Descriptive Analysis.** Hands over the distributions, base sizes and data types that determine which test is admissible, and the inspection of skew and outliers that Step 4 depends on.
- **05.03 Cross-Tabulation.** Hands over the banner structure and the comparison set, and it is where the multiplicity count in Step 5 comes from.
- **01.07 Analysis Plan Development.** Hands over the pre-specified comparisons, the threshold and the materiality standard, which is the single input that most improves the quality of everything this skill produces.
- **04.05 Weighting and Base Management.** Hands over the weighting scheme, the effective base and the design effect, without which no test on weighted data is valid.

**Recommended next skills**
- **05.06 Correlation, Regression and Causal Claim Control.** Takes questions this skill deliberately refuses: relationship strength, multivariate control, interaction effects, and anything approaching attribution.
- **08.01 Finding to Insight Development.** Takes the tested differences and asks what they mean, which is the step where a significant result becomes an argument and where **K2 §2.1** applies.

**Runs well alongside**
- **13.03 AI Output Verification**, which audits finished outputs for untested significance claims, missing bases, absent effect sizes and mislabelled exploratory results.
- **K3 Confidence and Uncertainty Protocol**, which supplies the language register for each level of evidential strength a test result produces.

---
A Yazi Supplied Skill and resource.