---
name: experiment-and-ab-test-analysis
description: >
  Analyses randomised experiments and quasi-experimental designs, and checks that
  the design actually supports the causal claim before analysing anything. Covers
  sample ratio mismatch, randomisation and balance checks, unit of randomisation
  and clustering, pre-specified primary metrics, guardrails, peeking and optional
  stopping, minimum detectable effect, novelty and primacy, interference between
  arms, heterogeneous effects and subgroup mining, reading a null correctly, and
  the gap between a test result and a rollout result. Use for "analyse the A/B
  test", "did the variant win", "is this result significant", "the test looked
  good early", "why did the rollout not match the test", "can we stop the test
  now", "which segment did it work for", or "we could not randomise, can we still
  claim it worked".
category: 06 Specialist and Advanced Analysis
ref: "06.04"
tier: 2
inherits: [K2, K3, K4, K5]
---

# Experiment and A/B Test Analysis

## 1. One-line description
Analyses experiments to the standard that lets a causal claim be made at all, by first establishing that randomisation happened and worked, that the primary metric was fixed in advance, and that the test could ever have detected the effect being claimed, and then measuring the effect with the uncertainty a decision needs.

## 2. What this skill is used for

**The research problem it solves.** This is the one place in this library where causal claims are legitimate, and that licence is conditional. It comes from random assignment, which makes treatment and control groups equivalent in expectation on everything, measured and unmeasured, so that a difference in outcome can be attributed to the treatment. Every failure in experiment analysis is a failure of one of the conditions behind that sentence. Assignment did not actually randomise, or randomised and then broke. The groups are unbalanced in a way that signals a broken pipeline. The metric being reported is not the one the test was designed around, but the one that moved. Someone watched the p-value and stopped when it crossed. The test ran on a sample that could never have detected an effect of the size that matters, so a null means nothing. The effect was real for two weeks and was novelty. The arms interfered, so control was contaminated by treatment. A subgroup was mined until something appeared. And the effect that shipped was smaller than the effect that tested, because a test population and a rollout population are not the same thing. This skill runs those checks in the order that catches the most damaging failures first, and it makes the causal claim only where the checks pass.

**Where it sits in the research lifecycle.** After an experiment has run and before its result is acted on. It is also usable before a test, as the specification of what will be checked, which is where it does the most good. Where the design is quasi-experimental, it identifies the assumptions that would have to hold and tests as many of them as the data allows.

**Typical use cases.**
- Analysing a completed A/B or multivariate test and deciding whether the variant won.
- Running the pre-analysis validity checks on a test whose result looks surprising.
- Diagnosing a sample ratio mismatch and establishing whether the test is salvageable.
- Interpreting a null result correctly instead of concluding no effect.
- Assessing whether an observed subgroup effect is real or the product of mining.
- Estimating the effect of an intervention where randomisation was impossible, with the identifying assumptions stated and checked.
- Explaining a gap between a test result and a rollout result.
- Setting the analysis rules for a test before it starts.

**Who uses it.** Product and UX researchers running experiments without a statistician; experimentation and growth analysts; marketing effectiveness teams running geo or holdout tests; policy and programme evaluators working with quasi-experimental data; and research directors auditing a causal claim before it reaches a board.

## 3. When to use it

- A randomised test has completed and a decision is about to be made on it.
- A test result is being questioned, or looks too good, and the validity checks have not been run.
- The traffic split does not match what was configured and nobody has established why.
- A test came back null and the team is about to conclude the change does not work.
- Someone is quoting a result for a metric that was not the primary metric.
- A test is running and someone wants to stop it early.
- An intervention was rolled out without a control and a causal claim is being made anyway.
- A launched feature is not delivering what the test predicted.
- A causal claim from any source needs auditing against the design that produced it.

## 4. When NOT to use it

- **There was no comparison group of any kind.** A before-and-after measurement with nothing to compare against attributes to the intervention every other thing that changed in the same period, including seasonality, a competitor's action, a price change and the news. This is not a weak experiment; it is not an experiment. Where a pre-post comparison is all that exists, report the change descriptively and state that no causal claim is supported. Where a comparison group might be constructible, see Step 14 and **05.06 Correlation, Regression and Causal Claim Control**.
- **The question is about association in observational data.** Which variables relate to an outcome, which correlates are strongest, how to control statistically for a confounder: all of that is **05.06**, which owns the language discipline that keeps regression output from becoming causal. This skill takes over only where a design (randomisation, or a quasi-experimental structure with identifying assumptions) supports attribution.
- **The experiment is a designed choice exercise rather than a field experiment.** Conjoint and discrete choice studies are experiments, and they license causal claims about the effect of a shown attribute on a stated choice inside the exercise. They do not license claims about the market. That distinction belongs to **06.01 Conjoint and MaxDiff Analysis**, which handles the estimation; this skill handles experiments where the outcome is real behaviour.
- **The test cannot have detected the effect that matters.** Where the minimum detectable effect at the achieved sample is larger than the effect the business cares about, the test has no informational content in either direction: a positive result would be implausibly large and a null tells you nothing. Report the minimum detectable effect and say that the test was not capable of answering the question. Do not analyse it as though it were.
- **Randomisation is known to have failed and cannot be repaired.** Where assignment was by a mechanism correlated with the outcome (self-selection into the variant, assignment by time of day, by device where device predicts the outcome), the groups differ systematically and no analysis restores the comparison. Statistical adjustment for observed differences does not fix unobserved ones, and claiming it does is the commonest way a broken test becomes a confident wrong answer.
- **The result being reported was chosen after seeing the data.** A secondary metric that moved, a subgroup that responded, a date range that works: each of these turns a confirmatory test into an exploratory one, and the appropriate output is a hypothesis for a new test, not a result. See Steps 5 and 12.
- **The unit of analysis and the unit of randomisation differ and the clustering has not been handled.** Randomising by user and analysing by session, or randomising by store and analysing by transaction, treats correlated observations as independent and produces confidence intervals far too narrow. Fix the analysis or do not report an interval.
- **The claim is about long-run effect from a short test.** Where the mechanism plausibly decays (novelty) or builds (learning, habit), a two-week result does not estimate the steady state, and extrapolating it is not supported. Report the observed window and what would be needed to establish persistence.

## 5. Required inputs

**Required. Without these the skill cannot run. If absent, stop and ask.**
- **The unit of randomisation and the assignment mechanism.** How units were allocated, by what method, with what intended split. Without this, no causal claim can be assessed, because the licence for the claim is the mechanism.
- **The pre-specified primary metric and the decision rule.** What the test was designed to move, how it was defined, and what result would have counted as a win. Where this does not exist, the test is exploratory and everything downstream is labelled accordingly.
- **Unit counts per arm, and the intended split.** The first check in the sequence depends on these and it is the check that most often reveals a broken test.
- **The outcome data at the unit of randomisation**, or at a finer grain with the randomisation unit identifiable so clustering can be handled.
- **The test window, and any changes during it.** Deployment dates, ramp schedule, any configuration change, any external event. A test that changed mid-flight is two tests.

**Optional, and what each one adds.**
- **Pre-period outcome data for both arms.** The strongest available check on whether randomisation worked, because the arms should be indistinguishable before the treatment started, on the very metric the test is about.
- **The power calculation and its assumptions.** Turns a null result from uninterpretable into informative, and reveals whether the test was ever capable of answering the question.
- **Covariates measured before assignment.** Enable balance checks, enable variance reduction through covariate adjustment (which increases precision without biasing the estimate, provided the covariates are pre-treatment), and enable pre-specified subgroup analysis.
- **Guardrail metric definitions.** Allow the test to be judged on whether it broke something else, which is frequently the decision-relevant question.
- **A holdout or long-run arm.** The only way to observe persistence and to detect novelty or primacy directly.
- **The rollout data after launch.** Enables the test-to-rollout comparison, which is the honest feedback loop that calibrates future tests.
- **The full metric list monitored.** Necessary to state how many comparisons were made, without which any secondary result cannot be interpreted.

## 6. Questions to ask before starting

1. **What exactly was randomised, and at what unit?** Determines the clustering, the effective sample size and the interference risk. Default if unanswered: stop. This cannot be assumed, and getting it wrong invalidates every interval in the analysis.
2. **What was the primary metric, fixed when?** Determines whether this is a confirmatory test or an exploratory one. Default: if no pre-specification can be evidenced, treat the whole analysis as exploratory and label every result as hypothesis-generating.
3. **Was the result looked at before the test ended, and did anyone act on what they saw?** Determines whether the nominal error rate holds. Default: ask directly, and where looks occurred without a sequential design, state that the stated significance level understates the false positive rate.
4. **What effect size would change the decision?** Determines whether the test was adequately powered and how a null should be read. Default: compute the minimum detectable effect at the achieved sample and report it regardless.
5. **Could a unit in one arm affect an outcome in the other?** Determines whether the arms are independent, which the whole analysis assumes. Default: assess the mechanism explicitly (shared inventory, social connection, marketplace competition, shared staff) and state the assessment.
6. **What else was running at the same time?** Determines whether another test, a campaign or an outage contaminates the window. Default: request the change log and note any overlap.
7. **Where will this be applied, and to whom?** Determines the external validity gap between the tested population and the rollout population. Default: describe the tested population precisely and state that the effect applies to it.

## 7. Step-by-step methodology

**Step 1. Write down the causal claim being sought, then check the design against it.** One sentence, of the form: for this population, changing this thing from A to B changes this outcome by this much, over this period. Then ask what licenses it. Random assignment licenses it for the population randomised, for the treatment as implemented, over the period observed. Nothing else does, and the checks that follow are checks on that licence. A correct result at this step is a written claim and a written statement of the design that would support it, which is the standard the rest of the analysis is measured against.

**Step 2. Check the sample ratio, first, before anything else.** Compare the observed unit counts per arm against the intended split, using a chi-square goodness-of-fit test on the counts. A significant deviation, and the threshold here should be strict because the null is that the randomiser worked, is called a sample ratio mismatch, and it is the single strongest signal that a test is broken. It means units were lost, misassigned or double-counted in a way that is almost never random, so the arms are no longer comparable and the effect estimate is unreliable in an unknown direction. Common causes: a redirect or load failure affecting one arm, bot filtering applied unevenly, a logging bug, assignment happening after an event that itself differs by arm, or units entering the analysis dataset conditional on something the treatment affects. **A test with an unexplained sample ratio mismatch is not analysed.** It is diagnosed. A correct result is the observed split, the expected split, the test statistic and either a clean pass or a diagnosis. Where the cause is found and affects only an identifiable subset, the test may be salvageable with that subset excluded, and that decision is a **K5 §2.7** review point.

**Step 3. Verify that randomisation actually happened and that it worked.** Two different checks. *Did it happen:* inspect the assignment mechanism, not the documentation. Look for units appearing in both arms, for assignment correlated with time or with a deterministic property (user ID parity is randomisation only if IDs are assigned randomly), and for any point at which a human could choose. *Did it work:* compare the arms on pre-treatment covariates and, most powerfully, on the pre-period value of the outcome metric itself. Read the pattern rather than individual results: with twenty covariates at a 5% threshold one imbalance is expected, so a single flag is not evidence of failure, while several imbalances, or one large imbalance on a strong predictor of the outcome, is. Report every check run, per **K4 §4.2**. A correct result is a balance table with the pre-period outcome comparison at the top, and a verdict. Note the asymmetry: balance checks can reveal that randomisation failed, but passing them does not prove it worked, since the point of randomisation is to balance the unmeasured variables you cannot check.

**Step 4. Establish the unit of randomisation and handle the clustering it implies.** If units were randomised at a level above the observation (users randomised, sessions observed; stores randomised, transactions observed; classrooms randomised, pupils observed), observations within a unit are correlated and treating them as independent understates the standard error, sometimes severely. The inflation is roughly one plus the intra-cluster correlation multiplied by the average cluster size minus one, which means even a small correlation matters when clusters are large. Handle it by analysing at the randomisation unit (aggregate to the unit and analyse those means, the simplest and safest approach), or by cluster-robust standard errors, or by a mixed model. Also note the reverse error: randomising at a fine grain when the treatment is experienced at a coarse one (randomising sessions when a person has many sessions and notices the inconsistency) breaks the experience and contaminates the comparison. A correct result is the analysis performed at the correct level, with the intra-cluster correlation reported where clustering was handled statistically.

**Step 5. Fix the primary metric, and separate it from the metric that moved.** The primary metric is the one specified before the data was seen, with its definition, its window and its decision rule. Report it first, whatever it says. Everything else is secondary and labelled as such, with the count of metrics monitored stated. This is the discipline experimentation most often loses: a win declared on a secondary metric, in a report where the primary's null sits on a later page or is absent. Secondary metrics are frequently where the learning is; the rule is that a secondary result is a hypothesis whose false positive rate is proportional to how many metrics were watched, and must be reported that way. Where no metric was pre-specified, say so and label the whole analysis exploratory. A correct result is a metric hierarchy with the primary first.

**Step 6. Check the guardrails, and treat them asymmetrically.** Guardrail metrics are the things the test must not break: load time, error rate, complaint volume, unsubscribes, revenue per user, service contacts. They are analysed differently from the primary metric, because the question is not "did this improve" but "is there evidence of harm", so a directional test at a looser threshold is appropriate, and a non-significant negative movement on a guardrail is not the same as safety. Where a guardrail moves adversely, that finding outranks a positive primary result and the decision becomes a trade-off judgement rather than an analytical one. A correct result is a guardrail table with direction, magnitude, interval and an explicit harm assessment.

**Step 7. Establish whether the result was peeked at, and what that does.** Repeatedly checking a test and stopping when significance appears inflates the false positive rate substantially, because with enough looks a random walk crosses the threshold eventually, and stopping at the first crossing biases the effect estimate upward because you stopped at a high point. Three cases. Pre-committed sample size, analysed once: nominal rates hold. Monitored under a formal sequential design with adjusted boundaries: rates hold, and the design is named. Monitored informally and stopped on a crossing: the reported significance is not the true significance and the effect is over-estimated, so report it with that stated and treat it as a hypothesis. A correct result is a statement of the looking behaviour and its consequence, which is a question to ask people rather than a thing to compute.

**Step 8. Compute the minimum detectable effect at the achieved sample, always.** Before interpreting any result, and especially any null, establish what the test could have seen. For two proportions, the minimum detectable effect at 80% power and a 95% two-sided threshold is approximately what **05.02 Statistical Testing** sets out, adjusted for the clustering in Step 4 by using the effective rather than nominal sample size. Compare it against the effect the business would act on. Three cases. The minimum detectable effect is well below the material effect: the test is informative in both directions. It is close to the material effect: the test can detect only large effects, and a null is weak evidence of absence. It is above the material effect: the test could never have answered the question, and this should have been known before it ran. A correct result is a single sentence naming the smallest effect this test could reliably have detected.

**Step 9. Estimate the effect with its interval, in both absolute and relative terms.** Report the control rate, the treatment rate, the absolute difference in the metric's own units, the relative difference as a percentage of control, and the confidence interval on the difference. Both forms are needed because they mislead in opposite directions: a relative lift of 40% on a base rate of 0.5% is an absolute gain of 0.2 percentage points, and a report giving only the relative figure will be read as far larger than it is. Lead with the interval rather than the p-value, per **05.02**. Where pre-treatment covariates exist, covariate adjustment (regressing the outcome on treatment plus pre-period covariates) reduces variance and narrows the interval without biasing the estimate, provided the covariates were fixed before assignment and the adjustment was specified before the analysis. A correct result is an effect statement carrying control value, treatment value, absolute difference, relative difference, interval, and the units of the metric.

**Step 10. Assess novelty and primacy directly, not by assertion.** Novelty is a temporary positive response to change itself; primacy is a temporary negative response to disruption of a learned habit. Both decay, both mean an early estimate is not the steady state, and they point in opposite directions, so neither can be assumed. Test by plotting the treatment effect by time since first exposure at the unit level, not by calendar date, since calendar plots confound exposure duration with the composition of who has entered the test. A decaying, growing or sign-reversing effect is a real pattern and changes the recommendation. Where the test was too short to see one, say persistence was not established. A correct result is an effect-over-exposure-time plot with an explicit persistence statement.

**Step 11. Assess interference between arms.** The analysis assumes a unit's outcome depends only on its own assignment. That fails whenever the arms compete for a finite shared resource (marketplace inventory, auction budget, staff time, stock), whenever units are socially connected, and whenever a treated user influences an untreated one. Under interference the control group is contaminated, and the measured difference can be far larger than the true effect (if treatment took share from control) or far smaller (if benefits spilled over). Assess the mechanism concretely for this test, and where interference is plausible prefer a design that contains it: randomise at a cluster level holding the interaction (geography, market, time period), or use a switchback design. A correct result is a written assessment naming the specific mechanism considered, not a generic sentence.

**Step 12. Handle heterogeneous treatment effects without mining.** Whether the effect differs by segment is a legitimate and important question, and it is also the easiest place in experimentation to manufacture a false finding, because a test with ten segment variables offers dozens of splits and one will always look impressive. Three rules. *Pre-specified subgroups are analysed and reported as such*, with the interaction tested rather than the two subgroups tested separately. Testing separately and observing that one is significant and one is not is not evidence that they differ, and this error is extremely common. *Exploratory subgroups are labelled exploratory*, with the number of splits examined reported, and are output as hypotheses for a confirmatory test, never as findings. *Underpowered by construction:* subgroup analysis splits the sample, so a test adequately powered overall is usually underpowered for any subgroup, and the minimum detectable effect within a subgroup should be computed and reported alongside the subgroup result. A correct result is a subgroup table separating pre-specified from exploratory, with interaction tests and subgroup-level minimum detectable effects.

**Step 13. Interpret a null correctly.** A non-significant result means the test could not distinguish the effect from zero. It is evidence of absence only to the extent the test was powered, and the honest statement combines the point estimate, the interval and the minimum detectable effect: "the estimated effect was +0.3% with an interval from -1.1% to +1.7%; effects smaller than 1.6% could not have been detected". Where the interval excludes the material effect on both sides, that is genuinely informative: the change did not do what was hoped, within the precision available. Where the interval contains the material effect, the test is inconclusive and should be described as such rather than as a negative result. Where a claim of equivalence is actually needed, it requires an equivalence margin set in advance, per **05.02**. A correct result is a null statement containing all three quantities and no use of the phrase "no difference".

**Step 14. Where randomisation was impossible, state the identifying assumptions and test them.** Quasi-experimental designs support causal claims conditionally, and the condition is an assumption that must be named and attacked rather than assumed. *Difference-in-differences* assumes parallel trends: absent treatment the two groups would have moved together. Test it across several pre-periods; with one pre-period it is untestable and the design is weak. *Interrupted time series* assumes nothing else changed at the discontinuity and that the pre-trend would have continued; check for coincident events and for anticipation before it. *Regression discontinuity* assumes units cannot precisely manipulate their position around the threshold; check for bunching and for covariate discontinuities there. *Matching and synthetic control* assume no unobserved confounder remains after matching, which is untestable, though pre-period fit is a necessary check. *Instrumental variables* require an instrument affecting the outcome only through the treatment, which is an argument, not a test. In every case report the design, the assumption, the check and what would break it. A correct result is a causal claim whose conditions are visible in the same paragraph as the claim, per **05.06** and **K4 §3.2**.

**Step 15. State the external validity gap between test and rollout, explicitly.** A test result is an estimate for the units randomised, over the window observed, with the treatment as implemented. A rollout differs on all of these. Five named mechanisms, each to be assessed: the tested population was more engaged than average (common where tests run on active users); the effect decays after novelty; the control that any spillover benefit was measured against will not exist after rollout; capacity constraints that did not bind at test scale will bind at full scale; and the winner's curse, since a result selected as the best of several is biased upward. A correct result is a rollout expectation stated as a range below the point estimate, with the mechanisms named and a commitment to measure the rollout so the gap can be calibrated.

## 8. Analytical framework

    Claim → Ratio check → Randomisation verified → Clustering handled
        → Primary metric → Guardrails → Power → Effect with interval
            → Persistence → Interference → Heterogeneity → Rollout expectation

This is strictly ordered, and the order is the method. The first three gates are validity gates: if any fails, there is no analysis to do, only a diagnosis, and proceeding past a failed gate produces a confident number with no licence behind it. That is why the sample ratio check comes before anything interesting: it is cheap, it is decisive, and it catches broken pipelines that would otherwise be interpreted as effects.

The middle of the sequence is measurement. The last three terms are where the result leaves the experiment and enters the world, and each of them shrinks what can be claimed. A test result is a fact about a specific randomised comparison. A rollout expectation is a judgement about a different population under different conditions, and the framework keeps them separate so that the judgement is visible as judgement (**K2 §2.1**) rather than inheriting the experiment's authority.

## 9. Output format

**The validity block.** Comes first in every experiment report, before any result.

| Check | Result | Verdict |
|---|---|---|
| Sample ratio | Observed 50.4/49.6 against intended 50/50, n=184,220, chi-square p=0.31 | Pass |
| Pre-period outcome | Control 4.11%, treatment 4.08%, difference 0.03pp, interval -0.11 to +0.17 | Pass |
| Covariate balance | 14 covariates tested, 1 flagged at 95% (expected 0.7) | Pass |
| Randomisation unit | User; observations at session level, aggregated to user for analysis | Handled |
| Looks taken | 1, at pre-committed sample size | Pass |
| Overlapping activity | No other test on this surface in the window | Pass |

**The power statement.** One sentence: the smallest effect this test could reliably have detected, at the achieved effective sample, against the effect that would change the decision.

**The primary result.**

| Metric | Control (n) | Treatment (n) | Absolute difference | Relative difference | 95% CI on difference | Test | p |
|---|---|---|---|---|---|---|---|

Reported first, whatever it says.

**The guardrail table.** Metric, direction of concern, control, treatment, difference, interval, harm assessment.

**The secondary metrics.** With the count of metrics monitored, and each labelled as secondary.

**The subgroup table.** Separated into pre-specified and exploratory, with interaction tests, subgroup minimum detectable effects, and the number of splits examined.

**The persistence view.** Effect by time since first exposure, with a statement about whether persistence was established.

**The rollout expectation.** A range, below the point estimate, with the mechanisms named.

**For quasi-experimental work**, the identifying assumption block: design, assumption, check performed, result of the check, what would break it.

**Where the evidence is thin**, use these forms:
- `[not analysed: sample ratio mismatch unexplained, see diagnosis]`
- `[exploratory: metric not pre-specified, one of N monitored]`
- `[inconclusive: interval contains the material effect; MDE was X]`
- `[persistence not established: test window shorter than the decay period of interest]`
- `[interference plausible via <mechanism>; effect estimate may be biased in <direction>]`
- `[identifying assumption untestable: single pre-period]`
- `[subgroup underpowered: MDE within this segment is X against an overall MDE of Y]`

## 10. Quality checks

**K4 §8** runs anyway. These are specific to experiments.

1. Was the sample ratio checked before anything else, and is the result in the output?
2. Is the assignment mechanism described, rather than assumed from documentation?
3. Is the pre-period outcome comparison reported, where pre-period data exists?
4. Does the analysis unit match the randomisation unit, or is the clustering handled and the method named?
5. Is the primary metric identified, pre-specified, and reported first?
6. Is the number of metrics monitored stated wherever a secondary result appears?
7. Are guardrails reported with a harm assessment rather than only a significance test?
8. Is the looking behaviour stated, and if the test was stopped on a crossing, is the inflation acknowledged?
9. Does every result carry both absolute and relative effect, with the metric's units?
10. Is the minimum detectable effect stated, and stated with every null?
11. Does the word "no difference" appear anywhere a null is reported?
12. Were subgroup effects tested as interactions rather than as separate tests within each subgroup?
13. Is every exploratory subgroup labelled, with the number of splits examined?
14. Has interference been assessed with a named mechanism rather than a generic sentence?
15. For quasi-experimental designs, is the identifying assumption named, checked where possible, and stated as untestable where not?
16. Does the rollout expectation differ from the test point estimate, with the reason?

## 11. Common failure modes

| Failure | How to recognise it | How to prevent it |
|---|---|---|
| **Sample ratio mismatch ignored** | A 51.8/48.2 split on a 50/50 configuration, mentioned nowhere | First check in the sequence, before any result is computed |
| **Peeking and stopping on a crossing** | A test that ended the day it reached significance | Pre-commit the sample size, or use a formal sequential design and name it |
| **The metric that moved** | A win declared on a metric that appears nowhere in the test plan | Metric hierarchy fixed before launch; primary reported first regardless of result |
| **Clustered data analysed as independent** | Impossibly narrow intervals; n reported as sessions when users were randomised | Aggregate to the randomisation unit or use cluster-robust methods; report the intra-cluster correlation |
| **Null read as no effect** | "The change had no impact" on a test with a 4-point minimum detectable effect | MDE reported with every null; the phrase "no difference" prohibited |
| **Subgroup mining** | A striking segment effect discovered after the overall result disappointed | Pre-specified subgroups separated; interaction tests; split count reported |
| **Separate tests read as a difference** | "Significant for new users, not for returning, so it works for new users" | Test the interaction; two separate tests do not compare each other |
| **Novelty extrapolated** | A two-week lift projected annually | Effect by exposure time plotted; persistence stated as established or not |
| **Interference unassessed** | A marketplace test where treatment took supply from control | Named-mechanism assessment; cluster or switchback designs where interference is plausible |
| **Relative lift without absolute** | "40% improvement" on a base rate of half a percent | Both forms mandatory in the result table |
| **Balance adjustment presented as a fix** | A broken randomisation "corrected" by controlling for the imbalanced covariate | Adjustment handles observed differences only; a failed randomisation is not repaired |
| **Test-to-rollout gap unexplained** | A launched feature underperforming its test, treated as a mystery | Rollout expectation stated as a range with mechanisms, before launch |
| **AI: causal language on a quasi-experimental design without the assumption** | "The programme reduced churn by 3 points" with no parallel trends check | Identifying assumption in the same paragraph as the claim (**K4 §3.2**) |
| **AI: reporting a plausible p-value or interval** | A precise interval with no per-arm counts behind it | Nothing reported without the counts and the calculation |
| **AI: over-flagging every validity concern equally** | A report where the passed checks and the fatal one look identical | Verdict column; failures stop the analysis, passes are recorded and moved past |

## 12. AI guardrails

Universal prohibitions are inherited from **K4**. **K4 §3.2** (never convert correlation into causation) governs this skill in a specific way: this is the one skill in the library where causal language is licensed, and the licence is conditional on the checks in Section 7. The following are specific.

1. **Never report an experiment result before the sample ratio check.** Where the ratio is mismatched and unexplained, produce a diagnosis, not an effect estimate.
2. **Never use causal language without naming the design that licenses it in the same paragraph**, per **K4 §3.2**. "The variant increased completion by 2.1 points (randomised at user level, n=184,220)" is licensed. The same sentence without the parenthesis is not.
3. **Never present a secondary or post-hoc metric as the test result**, and never omit the primary metric's outcome because it was null.
4. **Never report an interval computed as though clustered observations were independent.**
5. **Never describe a null as showing no effect, no impact or no difference.** Report the estimate, the interval and the minimum detectable effect.
6. **Never report a subgroup effect without stating whether it was pre-specified and how many splits were examined**, and never infer that two subgroups differ from separate within-subgroup tests.
7. **Never extrapolate a short-window effect to a long-run effect** without evidence of persistence from the data.
8. **Never claim that statistical adjustment repairs a failed randomisation.** Adjustment addresses observed imbalance only.
9. **Never state a quasi-experimental causal estimate without its identifying assumption and the check performed**, and where the assumption is untestable with the data available, say so.
10. **Never present a test point estimate as the expected rollout effect.** The rollout expectation is a separate, lower, explicitly reasoned figure.
11. **Never analyse a test whose minimum detectable effect exceeds the material effect** as though the result were informative. Report the incapacity.

## 13. Best-practice principles

1. **The validity checks are the analysis.** The effect estimate is arithmetic that any tool can do. What a professional contributes is establishing whether the comparison is a comparison at all, and that work happens before the number.
2. **A sample ratio mismatch is almost never a coincidence.** Treat it as evidence of a pipeline problem until proven otherwise, and resist the temptation to explain it away. Tests with unexplained mismatches that were analysed anyway are a large share of the false results in the field.
3. **Decide everything you can before you look.** Primary metric, decision rule, sample size, subgroups, guardrails, analysis method. Almost every argument in experiment interpretation is an argument that a pre-registration would have prevented.
4. **Power is a design property, not an analysis fix.** If the test could not have detected the effect that matters, no analysis rescues it. Compute the minimum detectable effect before launch and be willing to say a test is not worth running.
5. **Guardrails are frequently the finding.** A variant that lifts the primary metric while degrading a guardrail is the most common real experimental result and the one most often reported as a clean win.
6. **Prefer the coarser randomisation unit when in doubt.** Randomising at the user level rather than the session level, or at the geography level rather than the user level, costs precision and buys validity. Precision can be recovered with sample; validity cannot be recovered at all.
7. **Look at the effect over exposure time as a matter of routine.** It is cheap, it catches novelty and primacy, and it frequently reveals that the average effect is a mixture of two different responses.
8. **A win is a distribution, not a point.** Report the interval and reason about the decision at both ends. If the decision would be the same at the lower bound, the test has settled it. If it would not, the test has not.
9. **Learn from the rollout.** The gap between tested effect and realised effect is the most valuable calibration data an experimentation programme can have, and almost nobody collects it. Where one exists, it should adjust how future test results are read.
10. **A null on a well-powered test is a real result and should be reported as one.** It closes a question and saves the cost of building the thing. Programmes that only report wins train their organisations to distrust nulls.
11. **Quasi-experimental designs are not weaker experiments; they are arguments.** Their strength is entirely the plausibility of the identifying assumption, so the analytical work is making that assumption explicit and attacking it, not producing the estimate.
12. **The population you tested on is part of the result.** An effect measured among active users is an effect among active users. Writing it as a general effect is an overgeneralisation under **K4 §3.3**, and it is the most common reason a test does not replicate at rollout.

## 14. Worked example

**INPUT**

A fictional national library service, Fenwick Libraries, tests a redesigned online renewal flow. Users arriving at the renewal page are randomised 50/50 at user level. Primary metric, pre-specified: completed renewals per user within seven days of first exposure. Guardrails: staff-assisted renewal contacts, and page error rate. Planned sample 60,000 users per arm, running four weeks. After eleven days the product lead reports treatment is up 9% and asks to ship.

**PROCESS**

*Step 1, the claim.* Written as: among users who reach the renewal page, replacing flow A with flow B changes completed renewals within seven days, over the test window. Randomisation at user level licenses this for users who reached the page during the window.

*Step 2, sample ratio.* Counts at day eleven: 31,844 control, 30,102 treatment, against an intended 50/50. Chi-square on these counts gives p below 0.001. This is a sample ratio mismatch and the analysis stops. Investigation finds the treatment flow's tracking event fires after a page transition that fails on one older browser family, so a subset of treatment users are absent from the dataset entirely. Those users are disproportionately older and, in the pre-period, renew at a lower rate. **The 9% lift is substantially an artefact of losing low-renewing users from the treatment arm.** This is the finding at day eleven, and it is the most valuable output of the whole exercise.

*Repair and rerun.* The tracking bug is fixed, the test is restarted, and the eleven days of contaminated data are discarded rather than merged, with the reason logged. The restarted test runs the full four weeks to 60,412 and 60,118 users. Sample ratio now passes (p=0.42). Pre-period renewal rate: control 22.4%, treatment 22.6%, difference 0.2 points, interval -0.5 to +0.9. Fourteen covariates tested, one flagged, which is within expectation.

*Step 4, clustering.* Randomisation is by user; the outcome is defined per user, so the units match and no clustering adjustment is needed. This is stated rather than assumed.

*Step 8, power.* At 60,000 per arm on a control rate of 22.4%, the minimum detectable effect at 80% power is approximately 0.67 percentage points, roughly a 3% relative change. The service had said a 2-point absolute gain would justify the engineering cost. The test can detect well below that, so it is informative in both directions.

*Step 9, effect.* Control 22.4%, treatment 23.9%. Absolute difference 1.5 percentage points, relative 6.7%, 95% CI on the difference 0.8 to 2.2 points, p<0.001.

*Step 6, guardrails.* Staff-assisted renewal contacts per 1,000 users: control 8.1, treatment 9.4, difference +1.3, interval +0.4 to +2.2. This is an adverse movement on a guardrail and it outranks nothing automatically but changes the decision: more people renew online and more people also contact staff, which is consistent with the new flow being clearer for some users and confusing for others. Error rate unchanged.

*Step 10, persistence.* Effect by day since first exposure is flat across the window, with no decay pattern. Persistence beyond four weeks is not established and is stated as such.

*Step 12, heterogeneity. The judgement call.* An analyst notes the effect appears larger among users on mobile. This split was not pre-specified. Eight possible splits existed in the dataset. Tested as an interaction, the device interaction has p=0.09, and the minimum detectable interaction effect at these subgroup sizes is much larger than the observed one. Resolved by reporting it as an exploratory hypothesis with the split count and the underpowering stated, and not acting on it.

*Step 15, rollout.* The tested population is users who reached the renewal page, who are more engaged than the full member base. Rollout will expose the flow to users arriving by other routes. Expected rollout effect is stated as a range from roughly 0.8 to 1.5 points on the equivalent population, lower than the point estimate.

**OUTPUT**

> The redesigned renewal flow increased completed renewals within seven days from 22.4% to 23.9%, an absolute gain of 1.5 percentage points (relative 6.7%), 95% CI 0.8 to 2.2 points, in a user-level randomised test (n=60,412 control, 60,118 treatment; sample ratio p=0.42; pre-period renewal rates balanced within 0.2 points). The design licenses a causal reading for users reaching the renewal page during the four-week window.

> Guardrail: staff-assisted renewal contacts rose from 8.1 to 9.4 per 1,000 users (+1.3, CI +0.4 to +2.2). The flow moves more renewals online and also generates more staff contact. The net operational effect depends on the cost of a staff contact against the value of an online renewal, which this test does not measure.

> The effect is flat across the four-week window with no decay. Persistence beyond four weeks is not established.

> **Exploratory, not a finding.** The effect appears larger on mobile. This split was not pre-specified, was one of eight available, and the interaction test does not reach the threshold (p=0.09) on a sample underpowered for interactions of this size. It is a hypothesis for a confirmatory test.

> **Rollout expectation.** Between 0.8 and 1.5 points, below the test point estimate. The tested population reached the renewal page and is more engaged than the full member base; the rollout will include users arriving by other routes.

> **An earlier run of this test was discarded.** At day eleven it showed a 9% lift, which was an artefact: a tracking failure on one browser family removed a subset of treatment users who, in the pre-period, renewed at a lower rate. The eleven days were discarded rather than merged. This is recorded because the discarded result had already circulated.

**Researcher decision required.** The guardrail movement makes this a trade-off, not a clean win. Whether 1.5 additional online renewals per 100 users justify 1.3 additional staff contacts per 1,000 requires the service's own cost figures, which this test does not contain (**K5 §2.1** and **§2.5**).

## 15. Advanced usage

**Variance reduction.** Adjusting the outcome using pre-experiment values of the same metric substantially narrows the interval without biasing the estimate, because the pre-period value predicts the outcome and is unaffected by treatment. This is the cheapest available increase in test sensitivity and should be standard where pre-period data exists. It must be specified before analysis, since choosing an adjustment after seeing results reintroduces the problem it solves.

**Sequential and always-valid inference.** Where a test genuinely needs to be monitored, use a design built for it: group sequential boundaries with pre-specified interim looks, or always-valid confidence sequences that remain correct under continuous monitoring at the cost of width. Both are strictly better than informal peeking, and either must be declared in advance.

**Switchback and cluster designs for interference.** In marketplaces, logistics and any setting with shared finite resources, unit-level randomisation is contaminated by design. Randomising time periods (switchback) or geographies isolates the interference inside the unit, at the cost of a much smaller effective sample and a need to handle temporal correlation.

**Multi-arm tests and the comparison count.** Testing four variants against a control multiplies the false positive rate and introduces the winner's curse: the best-performing arm's effect is biased upward, because it was selected for being highest. Correct for multiplicity within the family per **05.02**, and expect the shipped effect to be below the winning arm's estimate.

**Meta-analysis across an experimentation programme.** Where an organisation runs many tests, the distribution of true effects across them is estimable, and it is usually much tighter around zero than the published wins suggest. This distribution is the right prior for reading any single new test, and it is the most useful thing a mature programme can build. Hand to **14.04 Meta-Analysis Across Studies**.

**Experiments on brand and attitudinal outcomes.** Where the outcome is measured by survey rather than by behaviour, the experiment is still an experiment, but the measurement inherits all the survey caveats: mode effects, non-response differing by arm (which is itself a sample ratio problem), and measurement noise that inflates the required sample considerably. Randomised holdouts with survey outcomes are legitimate and need much larger samples than teams expect.

## 16. Skill chain

**Recommended previous skills**
- **01.03 Hypothesis Development** and **01.07 Analysis Plan Development.** Hand over the pre-specified primary metric, the decision rule and the subgroups, which are the inputs that determine whether this analysis is confirmatory or exploratory.
- **01.06 Sampling Strategy.** Hands over the power calculation and the required sample per arm.
- **04.01 Data Validation.** Hands over the dataset integrity checks that the sample ratio investigation builds on.
- **05.02 Statistical Testing.** Hands over the test selection, effect size and interval discipline this skill applies to the treatment comparison.

**Recommended next skills**
- **05.06 Correlation, Regression and Causal Claim Control.** Takes the causal language audit, and takes over entirely where the design turns out not to support attribution.
- **08.01 Finding to Insight Development.** Takes the validated effect and asks why it happened, which the experiment establishes only if the mechanism was designed into it.
- **08.04 Recommendation Development.** Takes the effect with its interval and the guardrail trade-off into a decision, which is where the **K5 §2.5** sign-off applies.

**Runs well alongside**
- **06.02 Pricing Research Analysis**, where a live price test is available and is stronger evidence than any stated-preference study.
- **13.03 AI Output Verification**, which audits experiment reports for unchecked ratios, post-hoc metrics and unlicensed causal language.
- **14.04 Meta-Analysis Across Studies**, which builds the programme-level distribution of effects that calibrates every individual test.

---
A Yazi Supplied Skill and resource.