---
name: data-validation
description: >
  Diagnoses everything wrong with a raw research dataset without changing any of
  it: structural integrity, base sizes against the routing, out-of-range and
  impossible values, logical contradictions, routing violations, suspicious
  distributions, speeders, straight-liners, duplicate respondents, missingness
  patterns and inconsistent category labels. Produces a severity-rated validation
  report with counts, examples and the analysis impact of each issue. Use for
  "check this dataset", "is this data any good", "validate the data file", "QC
  the survey data", "the numbers look wrong", "how many bad respondents are in
  here", "data quality check before analysis".
category: 04 Data Preparation
ref: "04.01"
tier: 1
inherits: [K2, K3, K4, K5]
---

# Data Validation

## 1. One-line description
Systematically interrogates a raw research dataset for structural, value-level, logical, distributional and behavioural defects, and reports every one of them with its severity, its count, an inspectable example and its consequence for the analysis, while changing nothing in the data: this skill produces the diagnosis, and the researcher decides on the treatment.

## 2. What this skill is used for

**The research problem it solves.** A dataset arrives looking finished. It has the right number of columns, the labels look sensible, and the first twenty rows read normally. Analysis starts, and the defects surface one at a time over the following two weeks: a filtered question with more answers than people who should have seen it, a satisfaction battery where 60 respondents chose the same point on every item, an age field containing a 3 and a 214, two records with different IDs and identical open ends, a brand list where the same brand appears under four spellings, and a headline figure that moves five points once any of it is addressed. Each discovery costs rework, and worse, each one arrives after somebody has already seen a number. The failure is not that datasets have defects. Every dataset has defects. The failure is finding them late, finding them by accident, and finding them one at a time, so that no one ever sees the full picture of how good the data actually is before deciding what to do about it.

The second failure this prevents is the opposite of carelessness. An analyst who finds a problem mid-analysis is under pressure to fix it quietly and keep moving. That is how research data gets silently altered, and it is why diagnosis is separated from treatment here: the full diagnosis is produced first, in one document, before any change is authorised.

**Where it sits.** The first step after data arrives, before cleaning, before any figure is calculated, and before anybody sees a topline.

**Typical use cases.**
- Receiving a raw dataset at the close of fieldwork and establishing whether it can support the planned analysis.
- Interim quality checking on a live dataset partway through fieldwork, so that problems can still be fixed in field.
- Auditing a dataset supplied by a third party, where the collection process is not yours and cannot be assumed.
- Diagnosing a tracker wave that looks different from the last one, before concluding the market changed.
- Establishing the evidence for a quality dispute with a supplier, with counts and examples rather than impressions.
- Checking a dataset somebody has already cleaned, where the cleaning is undocumented.

**Who uses it.** Quantitative researchers and data managers running the standard QC pass; research directors deciding whether a dataset is fit to analyse; generalists handed a data file with no idea what to look for; client-side teams receiving a supplier's file and needing an independent view of its quality.

## 3. When to use it

- A raw or unfamiliar respondent-level dataset has arrived and nothing has yet been calculated from it.
- Fieldwork is complete and a go or no-go decision on the data is needed before analysis is commissioned.
- Fieldwork is still live and there is time to correct a problem at source rather than treat it afterwards.
- A figure looks wrong, or two figures that should reconcile do not, and the cause is unknown.
- A dataset arrives from a partner, a client or a previous agency with no documentation of how it was processed.
- A tracker wave shows a movement large enough that instrument or data problems must be ruled out before it is reported as change.
- A dataset has been cleaned by someone else and you need to know what state it is actually in.
- You need a documented, defensible quality position on a dataset, for a client, a supplier or an internal reviewer.

## 4. When NOT to use it

- **You want the problems fixed. This skill fixes nothing.** It finds, counts, illustrates and rates. Every correction, recode, exclusion, deduplication and harmonisation belongs to **04.02 Data Cleaning**, and belongs there specifically so that the diagnosis exists in writing before anybody decides what to do about it. The separation is not administrative tidiness: an analyst who diagnoses and treats in one motion cannot show a reviewer what the data looked like before, and cannot demonstrate that the exclusion rule was set before its effect on the results was visible. If you find yourself editing a cell while running this skill, stop, and log the finding instead.
- **The dataset is not respondent-level.** Validation of the kind described here operates on one row per respondent (or one row per respondent per wave or per occasion, with the unit declared). A file of pre-aggregated tables cannot be validated this way, because the defects live in the rows that have already been summarised away. Ask for the respondent-level file, and if it is genuinely unavailable, say clearly that the data cannot be validated and that everything downstream inherits an unverified input.
- **The dataset is qualitative.** Transcripts, open-ended corpora and observational notes have quality problems, and they are not these. Use **07.03 Interview and Transcript Analysis** for transcript integrity and **07.02 Open-Ended Response Coding** for open-end quality. Where open ends sit inside a survey file, this skill checks them for duplication, gibberish and off-topic content as a respondent-behaviour signal, and stops there.
- **Fieldwork is still running and the problem is fixable at source.** Running a full validation pass on a partial file and treating its findings as final wastes the one advantage of being in field. Use **03.04 Fieldwork Monitoring and Response Quality** to correct routing, screener or quota problems while they can still be corrected, and run this skill on the closed dataset afterwards.
- **The intention is to identify which respondents to remove in order to change a result.** Removing cases on the basis of their answers, rather than on the basis of documented quality criteria, is not validation. It is the manufacture of a finding, and it is research misconduct whatever it is called internally. If a request arrives phrased as "check whether the detractors are real respondents", the correct response is to run the same criteria on the whole sample and report what they catch, everywhere, per K4 §4.2.
- **The base is so small that behavioural flags cannot be distinguished from ordinary variation.** Speeder and straight-liner thresholds are relative to a distribution. On a sample of 40 there is no stable distribution of completion times to be an outlier from. Report the raw completion times and the pattern counts as observations, and do not construct percentile thresholds that imply a population of responses that does not exist.
- **No questionnaire, routing map or codebook exists and none can be obtained.** Almost every check in Section 7 needs to know what should have happened. Without the instrument you can find structural and internal-consistency problems, and you cannot find routing violations, out-of-base answers, wrong scale directions or invalid codes, which are where the expensive errors live. Say which classes of check could not be run, per K4 §6.4, rather than presenting a partial validation as a complete one.
- **The data has already been analysed and reported.** This skill can still run, and its findings then concern published figures rather than a working file. That is a different and more serious conversation, and it belongs with **13.01 Research Quality Review**, which handles the question of what to do about work already in a client's hands.

## 5. Required inputs

**Required.** Without these the skill cannot run properly. If absent, ask. If no answer is available and work must proceed, run the checks that are possible, and state explicitly which classes could not be run and what that leaves unverified, per K5 §5.

- **The raw respondent-level dataset**, in the state it arrived, with nothing removed. If a partly processed file is all that exists, say so; the validation then describes that file and not the fieldwork.
- **The questionnaire as fielded**, with question wording, full response lists, scale points and their labels, and any question that changed mid-field.
- **The routing, filter and skip logic**, in a form that states which respondents should have seen each question. This is what makes routing violations and out-of-base answers detectable at all.
- **The codebook or data dictionary**: variable names, value labels, valid ranges, scale direction, and the distinct codes used for don't know, refused, not asked and system missing.
- **The sample and quota plan**: intended total, intended structure, and the base sizes each subgroup was designed to deliver, so that achieved bases can be checked against intended ones rather than merely described.

**Optional, and what each one adds.**

- **Paradata (completion timestamps, per-question timings, device type, entry mode, break-offs, attempt counts):** turns speeder detection from a crude total-duration rule into a per-section one, and is the only reliable way to distinguish a fast respondent from a respondent who abandoned and returned.
- **The fieldwork log and any in-field quality actions already taken:** tells you which problems have already been treated, so that this pass does not re-report cases the field team already removed, and so that removals already made are visible rather than invisible.
- **The previous wave's dataset and validation report:** allows the same checks with the same thresholds, and allows structural comparison, which is the fastest way to detect a question that changed silently between waves.
- **Any supplier quality documentation (fraud screening, digital fingerprinting policy, quality removals and their criteria):** tells you what has already been screened and on what basis, so that this validation adds to it rather than assuming it did not happen.
- **The analysis plan:** determines which defects matter. A problem in a variable nobody will analyse is a note. The same problem in the primary outcome measure is a stop.
- **Open-end text fields:** carry the clearest signals of inattentive, automated and duplicated responding, and are frequently the only way to catch a duplicate respondent who varied their closed answers.

## 6. Questions to ask before starting

1. **What is the intended unit of analysis, and what should be unique?** Determines what a duplicate is. One row per respondent, per household, per wave, per store visit: each makes a different set of rows legitimately repeated. *Default if unanswered:* assume one row per respondent, test the ID variable for uniqueness, and report the assumption alongside the result.
2. **What are the achieved and intended base sizes, overall and by subgroup?** Determines whether the file is complete and whether the analysis the study promised is possible. *Default:* compare achieved counts against the quota plan where one exists; where none exists, report the achieved structure and flag that it could not be checked against an intention.
3. **Which questions were asked of whom?** Without this, out-of-base answers and routing violations are undetectable. *Default:* reconstruct routing from response patterns, report it as reconstructed, and flag every question where the reconstruction is ambiguous rather than asserting a base.
4. **What quality rules were already applied in field, and by whom?** Prevents double-counting removals and reveals what the file is already missing. *Default:* assume none were applied, state that assumption, and note that any prior removals are invisible to this validation.
5. **Which variables are the primary outcome measures?** Sets severity. The same 3% error rate is trivial in a descriptive profile variable and disqualifying in the measure the decision rests on. *Default:* treat all measures named in the objectives as primary, and rate severity against the whole analysis rather than against a specific chart.
6. **Are there pre-agreed quality criteria for this study, and were they set before the data was seen?** Determines whether exclusion thresholds are being applied or invented. *Default:* report behavioural flags as counts against several thresholds rather than one, so that a threshold can be chosen afterwards without the choice looking like it was chosen to produce an outcome, and say plainly that no pre-agreed criteria existed.
7. **Is this a wave of a tracker?** Determines whether checks must match a previous specification even where a better one exists. *Default:* apply the previous wave's checks and thresholds where documented, and report any additional checks separately so the series stays comparable.

## 7. Step-by-step methodology

Run the checks in this order. The order matters: structural problems make value-level results meaningless, and value-level problems make distributional results meaningless. Nothing in this section changes the data. Every finding is written to the issue register described in Section 9.

**1. Take a fingerprint of the file as received, before anything else.** Record the file name, the date received, the source, the row count, the column count, and a checksum or equivalent identifier if one is available. Record the count of rows and the count of distinct IDs. This is the reference point that makes reconstruction possible: every later log entry in this category is an offset from this fingerprint. *Correct result:* a short block a reviewer could use to confirm they are looking at the same file you validated.

**2. Check structure against the specification.** Column count against the questionnaire, including every question, every item in a battery, every multi-response option and every derived or system variable. Row count against the intended sample and against the field report. Test the respondent ID for uniqueness and for completeness. Check the achieved base for each subgroup against the quota plan. Then check what is not there: a question in the questionnaire with no column in the data is the single most costly structural defect, because it is invisible in the file itself and is usually found when the analysis plan asks for it. *Correct result:* a reconciliation showing intended versus achieved on rows, columns, IDs and subgroup bases, with every difference itemised, not netted.

**3. Reconcile every base against the routing.** For each filtered question, derive the set of respondents who should have seen it from the routing rule, and compare that set to the set that has a valid answer. Three discrepancies matter and they are different problems. Respondents with an answer who should not have seen the question means a routing violation, a data merge error or a mis-specified filter, and it contaminates the base. Respondents who should have seen it and have nothing means item non-response, a break-off, or a filter that fired early. A base that is right in total but wrong in composition means the routing variable and the routed question disagree about who is who. *Correct result:* per-question base reconciliation with each discrepancy classified into one of those three, counted, and illustrated with at least one respondent ID.

**4. Check values against their permitted range and type.** For every variable: are all values within the defined code list or numeric range; is the type what it should be (a numeric field stored as text will sort and aggregate wrongly and looks fine on inspection); are dates valid, in range, and in a plausible order relative to each other. Look specifically for the values that are legal but impossible: an age of 3 in an adult sample, a tenure longer than the respondent's age, a spend figure three orders of magnitude above the next highest, a date of birth in the future, a purchase date before the product existed. Check text encoding: mojibake, stray control characters, and multi-byte characters rendered as question marks are signals that the file has been through a conversion that may have damaged more than the visible characters. *Correct result:* a per-variable table of invalid, out-of-range and impossible values with counts and examples, and encoding problems reported as a file-level issue rather than a per-value one.

**5. Check the logic between questions.** These are the errors that no single-variable check can catch. Numeric parts that must sum to a total and do not. A respondent who says they have never used the service and then rates their most recent experience with it. Household composition inconsistent with household size. A category selected in one question absent from the list carried forward to the next. A "none of these" option selected alongside three other options in the same multi-response question, which is a questionnaire logic defect as much as a respondent one. Frequency and recency answers that contradict each other. Build the contradiction checks from the questionnaire structure rather than from a generic list, because the informative contradictions are always specific to the instrument. *Correct result:* a list of logic checks written as testable rules, each with a pass count, a fail count, and worked examples of failures.

**6. Read the distributions for the suspicious, not merely the unusual.** This is the judgement-heaviest step and the one most often done badly. An unusual distribution is a finding. A suspicious distribution is evidence of a defect. The difference turns on whether the shape has a plausible substantive explanation. Signals that point to defect rather than finding: a spike at a single value that corresponds to a default or first-listed option, which suggests a data-entry default or a non-response coded as an answer; an impossible smoothness or perfectly even spread across categories; a bimodality that follows the order of the response list rather than anything about respondents; a variable where a large block of respondents share one value and were all collected on the same day or through the same route; a scale where the modal answer sits at the code used for missing in an adjacent variable. Cross the suspect variable by collection date, source and device, because a defect usually clusters by one of them and a genuine finding usually does not. *Correct result:* every flagged distribution carries a stated reason for suspicion and the cross-check that raised or lowered it, so that a reviewer can disagree with the judgement rather than merely accept it.

**7. Assess response behaviour, using several signals rather than one.** No single behavioural flag is sufficient evidence of a poor-quality respondent, and treating one as sufficient removes good respondents. Compute: completion time relative to the distribution of completion times, and where paradata allows it, per-section time rather than total; straight-lining within each grid or battery, measured as zero or near-zero variance across items, checked separately for batteries containing reverse-worded items where straight-lining is more diagnostic; patterned responding such as alternating or diagonal answer sequences; open ends that are empty, single characters, gibberish, off-topic, copied from the question wording, or identical to another respondent's; and near-duplicate respondents, identified by matching on a combination of demographics, open-end text and answer patterns rather than on any one of them. Report each flag separately with its count, then report the count of respondents triggering one, two, three or more flags. The multi-flag count is the useful number: a respondent who is fast is a respondent; a respondent who is fast, straight-lined the grid and wrote "good" in every open end is a data quality problem. *Correct result:* a per-flag count table plus a flag-overlap table, with thresholds stated, and with alternative thresholds shown so the threshold choice is visible as a choice.

**8. Map the missingness rather than counting it.** Distinguish the four states before any count: not asked (routed away, out of base, and not missing at all), not answered, an explicit don't know or refused, and lost through a technical failure or break-off. Then map: missingness per variable, per respondent, and by position in the questionnaire. A missingness rate that climbs steadily through the instrument is fatigue or break-off. A cliff at one question is that question. Missingness concentrated in one collection route or one device type is a technical fault. Missingness correlated with a demographic is a non-response pattern with analytical consequences. Diagnosis of the mechanism and the decision about what to do belongs to **04.03 Missing Data Handling**; this step produces the map that skill needs. *Correct result:* a missingness map by variable, by respondent and by position, with the four states counted separately and never summed.

**9. Check labels, categories and codes for consistency.** Value labels present for every code actually used, and codes present for every label. Scale direction verified against the questionnaire for every scale variable, and checked for consistency across items that share a scale, because a single reversed item in a battery is both the easiest defect to introduce and the hardest to see. Category lists checked for the same entity appearing more than once: differences of case, spacing, punctuation, abbreviation, spelling and language variant all produce separate categories for one thing, and the effect is a systematic under-count of whichever category is fragmented. Check code frames from any coded open ends for overlapping categories and for an "other" that has grown large enough to indicate a missing category. *Correct result:* a list of every variable where labels are missing, ambiguous or inconsistent, and a candidate merge list for fragmented categories, presented as candidates for **04.02** to act on rather than as merges performed here.

**10. Rate every issue for severity, against the analysis rather than against the data.** Four levels, and the rating is a judgement that must be justified in one line each. **Critical:** the analysis cannot proceed, or would be wrong in a way that changes a conclusion (a contaminated base on a primary measure, a reversed scale, a missing key variable, a duplicate rate above a few per cent). **Major:** a specific analysis is materially affected and the affected output must be qualified or held (a fragmented brand list, a logic contradiction affecting a subgroup, missingness above roughly 10% on an analysed variable). **Minor:** real, worth fixing, and does not change any conclusion (encoding artefacts in a display label, out-of-range values in a variable nobody will report). **Note:** an observation about the data worth recording, with no action implied. Severity is set by consequence, not by count: an issue affecting three respondents on the primary outcome outranks one affecting three hundred on an unreported variable. *Correct result:* every issue rated, with a one-line statement of what would go wrong if it were left untreated, expressed in terms of the analysis and not in terms of the data.

**11. Quantify the potential impact of the largest issues before anybody asks.** For the critical and major issues, state what is at stake in figures a researcher can weigh: how many cases are affected, what proportion of the total and of the relevant subgroup, and which reported measures they touch. Do not run the analysis with and without the cases at this stage and do not present a "corrected" figure: that is 04.02's sensitivity check, and producing it here inverts the sequence by letting the effect on results be seen before the treatment rule is set. Say what is affected and how much of it. *Correct result:* an impact line per major issue that lets a researcher understand the size of the decision without seeing the answer it produces.

**12. Write the report as a decision document, and stop.** The output goes to the researcher with issues ordered by severity, each with its count, its example, its impact and its options, and with no changes made to any file. Where an issue requires a judgement that only a person can make, mark it per K5 §3, name the decision, and say what turns on it. Then hand to **04.02**. *Correct result:* a document from which a researcher can authorise a cleaning specification without opening the data, and against which the resulting cleaning log can later be audited item by item.

## 8. Analytical framework

Every issue found in this skill is written in one structure, and that structure is the first four columns of the log that runs through the whole of category 04:

    Original issue → Evidence → Severity → Analysis impact → Options → (Action, deferred to 04.02)

**Original issue.** What is wrong, stated as a fact about the data: the variable, the condition, the count, the proportion.
**Evidence.** How it was detected, and at least one inspectable example by respondent ID, so a reviewer can look at it.
**Severity.** Critical, Major, Minor or Note, with the one-line justification.
**Analysis impact.** What goes wrong if nothing is done, expressed as an effect on a finding, not as an effect on the file.
**Options.** The treatments available, with what each one costs. Options, not a decision.
**Action.** Deliberately empty at the end of this skill. It is filled in by 04.02, and the fact that it is filled in later, by a named person, is the point.

Against the K2 evidence chain this skill sits below Finding: it produces evidence and analysis about the dataset itself. A validation report contains no findings about the research subject, and any statement in it about what respondents think or do has strayed outside the skill.

## 9. Output format

**1. File fingerprint.** File as received, date, source, rows, columns, distinct IDs, and the identifier by which the file can be re-identified. Plus a one-line statement that no data was altered in producing this report.

**2. Validation scope statement.** Which of the nine check classes in Section 7 were run, which could not be run, and why. Per K4 §6.4, a check that could not be run is stated, never implied to have passed.

**3. Summary judgement.** Three to six sentences: the overall state of the dataset, the number of issues at each severity, and a clear position on whether the data can support the planned analysis as it stands, with what would have to change. Confidence stated per K3 §4.

**4. Issue register**, ordered by severity.

| ID | Issue | Class | Variables affected | Cases affected (n and %) | Evidence and example ID | Severity | Analysis impact if untreated | Options |
|---|---|---|---|---|---|---|---|---|

Class is one of: structural, value, logic, distribution, behaviour, missingness, labelling.

**5. Base reconciliation table.**

| Question | Base per routing | Base in data | Difference | Discrepancy type | Cases affected |
|---|---|---|---|---|---|

**6. Response behaviour flag table.**

| Flag | Threshold used | Alternative thresholds and counts | n flagged | % of sample | Overlap with other flags |
|---|---|---|---|---|---|

Plus the flag-overlap summary: respondents triggering 1, 2, 3 or more flags.

**7. Missingness map.** Per variable: not asked, not answered, don't know, refused, lost, each counted separately. Plus the per-respondent distribution and the by-position pattern.

**8. Category consistency candidates.** Fragmented categories with their variant spellings and counts, presented as candidates for harmonisation, not as merges.

**9. Review points**, consolidated per K5 §3.3, each naming the decision, why a person is needed, and what turns on it.

**10. What could not be checked**, and what that leaves unverified.

**Where the evidence is thin**, the format does not get filled. A check that could not be run appears in section 2, not as a passed check. A distribution that looks odd with no supporting cross-check is reported as an observation at Note severity with the ambiguity stated, not promoted to an issue to give the register weight. A behavioural threshold that cannot be computed because the base is too small is reported as not computable, per K4 §1. An empty issue register on a clean dataset is a valid and useful output.

## 10. Quality checks

Run before the report is presented. These sit on top of K4 §8.

1. Is the file in exactly the state it arrived in, with a verifiable fingerprint proving it?
2. Does every issue in the register carry a count, a proportion and at least one inspectable respondent ID?
3. Is every severity rating justified by a stated consequence for the analysis rather than by the size of the count?
4. Has every base been reconciled against the routing rule rather than inferred from the count of valid responses?
5. Are the four missingness states counted separately, with not-asked kept out of the missing count entirely?
6. Is every behavioural flag reported with its threshold, and with at least one alternative threshold so the choice is visible?
7. Is the multi-flag overlap reported, rather than only per-flag counts?
8. Has any flagged distribution been called a defect without a cross-check by date, source or device?
9. Has any respondent been flagged on the basis of what they answered rather than how they answered?
10. Is every scale direction verified against the questionnaire rather than assumed from a variable name or label?
11. Does the report avoid stating any conclusion about the research subject, as opposed to about the data?
12. Has any correction, however small, been made to the file during this pass?
13. Would a reviewer with the same file and this report be able to reproduce every count in it?
14. Are the checks that could not be run listed explicitly, rather than absent?

## 11. Common failure modes

| Failure | How to recognise it | How to prevent it |
|---|---|---|
| **Fixing while checking** | The register contains an issue with the action already taken, or a cell has changed | Work on a read-only copy; the register has no action column in this skill; fingerprint the file at the start and re-check it at the end |
| **Base inferred from the data** | A base stated as "n=612" with no routing rule beside it | Derive the expected base from routing first, then compare; report reconstructed bases as reconstructed |
| **Single-flag exclusion logic** | A list of "bad respondents" built entirely from completion time | Report flags separately and in combination; the useful number is the multi-flag count, and no single flag is presented as a verdict |
| **Threshold set after seeing the effect** | A speeder threshold that lands on a round number of removals, or one chosen after the topline was run | Report several thresholds with their counts; state that no pre-agreed criteria existed where that is the case |
| **Unusual read as suspicious** | A genuine finding reported as a data defect because the shape was unfamiliar | Require a stated reason for suspicion and a cross-check by date, source or device before flagging a distribution |
| **Missingness summed across states** | A single "23% missing" figure covering routed-out, don't know and lost cases | Count the four states separately, always; not asked never enters a missing count |
| **Category fragmentation missed** | A brand or category list where the top item is under-counted, with variants further down | Sort category lists by normalised text, not by frequency, and inspect adjacent entries |
| **Reversed item inside a battery** | A battery where one item correlates negatively with the rest | Check scale direction item by item against the questionnaire; correlation within a battery is a detection aid, not proof |
| **Validation without the instrument** | A report with no routing or logic checks and no statement that they were impossible | The scope statement lists every check class as run or not run, with the reason |
| **Severity by volume** | Critical issues that affect many cases and no analysis; trivial issues on the primary measure rated Minor | Rate against the analysis plan, and state the consequence in the rating |
| **AI: plausible counts** | A count in the register that was estimated from the shape of the data rather than computed | Every count is computed and reproducible; per K4 §2.1, no cell is filled with a plausible value |
| **AI: inventing an example** | An illustrative respondent ID that is not in the file | Every example carries a real ID that a reviewer can look up; if no example can be produced, the issue is not evidenced |
| **AI: inferring meaning from names** | Valid ranges, scale direction or missing codes taken from the variable name | Per K4 §6.2, verify against the codebook, or flag the assumption as unverified in the scope statement |

## 12. AI guardrails

Skill-specific only. K4 applies in full and is not repeated here.

1. **Never alter, correct, recode, delete, reorder or re-save any part of the dataset in this skill.** Not a typo, not an obvious encoding artefact, not a single impossible value. Log it. The prohibition is absolute because the value of the diagnosis depends on it being taken before treatment.
2. **Never report a count that was not computed from the file.** No approximations, no "roughly", no figure carried from a similar project.
3. **Never present an example without a real respondent identifier from the file.** An unevidenced issue is an assertion.
4. **Never flag a respondent on the basis of the content of their answers.** Quality flags concern how a person answered: speed, invariance, pattern, duplication, internal contradiction. What they said is never a quality signal, and a request to make it one is refused and explained, per K4 §9.
5. **Never assert a base without stating whether it came from the routing specification or was reconstructed from the data.**
6. **Never merge the four missingness states into one figure**, and never include routed-out respondents in a missing count.
7. **Never present a behavioural threshold as though it were a standard.** Thresholds are choices. State the one used, state alternatives, and never imply an industry rule where none is agreed.
8. **Never state or imply that a check passed when it could not be run.** Absence of a finding on a check that was not executed is not a clean result.
9. **Never rate severity by the number of cases affected.** Severity is the consequence for the analysis.
10. **Never produce a "corrected" figure in this skill**, including as an illustration of an issue's size. Stating that 61 cases are affected is impact. Stating what the headline becomes without them is treatment, and belongs to 04.02 after the rule is set.

## 13. Best-practice principles

- **Diagnosis before treatment, always, and in writing.** The document that lists what is wrong must exist before anything is put right. It is what allows a reviewer to check the cleaning, and it is what stops a defensible fix from being indistinguishable from a convenient one.
- **Every dataset has defects.** A validation report with no issues usually means the checks were too shallow, not that the data was perfect. Expect to find something, and be suspicious of a clean sheet on a large instrument.
- **The most expensive defect is the one that looks like a finding.** A contaminated base, a reversed scale and a fragmented category list all produce numbers that are internally consistent, plausible and wrong. Checks that only look for the obviously broken will not catch them.
- **No single behavioural flag identifies a bad respondent.** Fast people exist. Consistent people exist. The signal is convergence across independent flags, and the threshold for action should require it.
- **Thresholds set after the effect is visible are not thresholds.** If a speeder rule is chosen once its effect on the headline is known, the rule is no longer a quality criterion. Set it before, or report multiple and let the decision be made and documented.
- **A count without an example is not evidence.** Every issue should be inspectable. The discipline of producing an ID for each one also catches a surprising number of checks that were themselves wrong.
- **Cross-check every suspicious distribution against collection metadata.** Defects cluster by date, source, device and interviewer. Findings usually do not. This one habit separates most real data problems from most false alarms.
- **Missingness is four things, and merging them destroys the information.** Routed-out is not missing. Don't know is an answer. Refused is a different answer. Lost is a technical event. A single percentage covering all four is worse than no figure.
- **Severity is about the analysis, not about the data.** The question is always "what conclusion would be wrong if this were left alone", and if the answer is "none", it is a Note.
- **The report is written for someone who will not open the file.** If a research director has to load the data to understand an issue, the entry is not finished.
- **What was not checked is part of the result.** A validation is only as strong as its scope statement, and a report that hides its own gaps is worse than one that finds fewer problems honestly.

## 14. Worked example

**INPUT.** A fictional national health charity commissions a study of public awareness of a screening programme. Intended sample 1,200 adults, quota-controlled on age, gender and region, with a boost of 200 in one region. The file arrives as 1,438 rows and 214 columns, with a questionnaire, a routing map and a partial codebook. Objectives concern awareness, understanding of eligibility, and barriers among the eligible but unscreened.

**PROCESS.**

*Steps 1 to 2.* Fingerprint taken: 1,438 rows, 214 columns, 1,431 distinct IDs. Seven IDs appear twice, which is the first issue. Column reconciliation against the questionnaire finds 213 expected columns present and one absent: the second half of the eligibility battery (Q17b) has no columns at all. Q17b is a primary measure. Rated Critical, because eligibility understanding is one of the three objectives and cannot be reported from this file. Achieved bases by region show the boost region at 187 against an intended 200, and one standard region at 78 against an intended 100.

*Step 3.* Q22 (barriers) was routed to those eligible and unscreened, expected base 402 from the routing variables. The data shows 431 valid answers. The 29 extra respondents all answered "already screened" at Q19. Classified as a routing violation contaminating the base of a primary measure. Rated Critical. Example IDs recorded.

*Step 4.* Age range 18 to 94, all in range. One tenure field records years since last screening greater than the respondent's age for 4 cases. Two open-end fields show mojibake on accented characters, affecting display only. The mojibake is rated Minor; the tenure contradiction is rated Major because the variable feeds a derived recency band.

*Step 6, and a judgement call.* Q14, a five-point confidence scale, shows an unusual spike at point 3, the mid-point, at 34% against 12 to 19% in the four comparable items. The instinct is to flag a defect. The cross-check by collection date shows the spike evenly spread across the field period; by device, evenly spread; by source, evenly spread. Nothing clusters. Inspection of the questionnaire shows Q14 is the only item in the battery with no "don't know" option, so respondents without a view had nowhere else to go. **Resolution:** this is a finding about the instrument, not a defect in the data. Logged at Note severity, with a recommendation to the questionnaire for the next wave, and explicitly not proposed for treatment. The 34% is real and must be interpreted with the missing option stated.

*Step 7.* Median completion 14 minutes. 96 respondents completed in under a third of the median. 71 straight-lined the 12-item barriers battery. 44 gave open ends that were empty, single-character or copied from the question. Multi-flag: 61 respondents trigger two or more flags, 19 trigger all three. Thresholds reported at three levels each so the choice is visible. No exclusion is proposed, no exclusion is made, and the register notes that no quality criteria were agreed in advance.

*Step 9.* The "which organisations have you heard of" list contains the charity's own name in six variants differing by capitalisation, an abbreviation and a trailing space. Combined they would rank second; fragmented, the largest variant ranks seventh. Rated Major, with the six variants listed as harmonisation candidates.

*Steps 10 to 12.* Two Critical, four Major, six Minor, three Notes. Summary judgement: the data supports the awareness and barriers objectives once the Q22 base contamination is resolved and the category fragmentation is harmonised; it cannot support the eligibility-understanding objective at all, because Q17b is not in the file. **Researcher decision required** on whether to request Q17b from the supplier or descope the objective, per K5 §2.7, since it turns on fieldwork cost and timing this skill does not know. **Researcher decision required** on the exclusion threshold for the multi-flagged 61 cases, per K5 §2.1, with the note that no criteria were pre-agreed and that the decision must be recorded before the effect on results is examined.

**OUTPUT.** A fingerprinted, unaltered file; a scope statement naming one check class (full logic checking on Q17b) as impossible; an issue register of 15 items with counts, examples and impacts; a base reconciliation showing the 29-case contamination; a flag table with three thresholds each and the overlap count; a missingness map; six harmonisation candidates; two decision points; and no change whatsoever to the data.

## 15. Advanced usage

**Validating in-field rather than after.** Where fieldwork is live, run steps 2, 3, 5 and 7 on the partial file at roughly 15% and 50% of quota. Routing violations, missing columns and script faults are fixable at those points and unfixable afterwards. Coordinate with **03.04 Fieldwork Monitoring and Response Quality**, which owns the in-field response, and keep the two records separate so that in-field removals are visible in the final validation rather than absorbed into it.

**Tracker validation.** The strongest check available on a tracker wave is the previous wave. Compare structure column by column, compare label sets, compare the distribution of every tracked measure, and compare missingness patterns. A tracked measure that moves more than its historical wave-on-wave variation is a candidate defect before it is a candidate finding, and should be cleared as one before anybody sees it in a chart. Variable comparability across waves is handled in **04.04**.

**Adversarial validation.** Where the sample source is uncontrolled or the incentive is high, add checks for the patterns that automated and professional respondents produce: implausibly consistent demographic profiles across many studies, straight-lining that avoids the extremes, open ends that are grammatically perfect and content-free, and near-identical answer vectors across respondents. Report matched pairs as candidates with their match evidence rather than as verdicts, since a false accusation of fraud is a serious matter and the evidence should be inspectable.

**Multi-country and multi-language files.** Validate each market separately before validating the merged file, because merged files hide market-specific defects in a large denominator, and because translation frequently changes scale labels, category lists and don't know options in ways that only appear when the markets are compared side by side.

**When the standard approach does not fit.** Behavioural data, transaction logs and passively collected data have no completion times, no straight-lining and no open ends, and the equivalent checks are session integrity, event ordering, device duplication and coverage gaps by time period. Keep the same framework (structure, value, logic, distribution, behaviour, missingness, labelling) and replace the checks inside it rather than abandoning the structure.

## 16. Skill chain

**Recommended previous skills:**
- **03.04 Fieldwork Monitoring and Response Quality.** Hands over the in-field quality record, the removals already made and the field-level issues already known, so this validation adds to that record rather than duplicating or contradicting it.
- **02.05 Survey Logic and Flow Review.** Hands over the routing map in a form that makes base reconciliation possible, and is frequently the document that reveals a routing violation was designed in rather than introduced in field.
- **01.06 Sampling Strategy** and **02.06 Screener and Quota Design.** Hand over the intended sample structure and quota plan against which achieved bases are checked.

**Recommended next skills:**
- **04.02 Data Cleaning.** Takes this issue register and turns each entry into a documented action, a reason and an impact statement. This is the direct and intended successor, and 04.02 should not run without this skill's output.
- **04.03 Missing Data Handling.** Takes the missingness map, in its four separated states, and diagnoses the mechanism and the treatment.
- **04.04 Data Transformation and Dataset Preparation.** Takes the harmonisation candidates, the label inconsistencies and the verified scale directions.
- **13.01 Research Quality Review**, where the validation finds problems serious enough to raise a question about the study rather than the file.

**Runs well alongside:**
- **K2**, for the evidence-reference format every issue example uses, and **K3**, for stating confidence in the summary judgement.
- **13.04 Bias Detection**, where distribution anomalies raise the possibility of a sampling or instrument bias rather than a data defect.
- **05.01 Descriptive Analysis**, which inherits the base register and must not run until the Critical issues here are resolved.

---
A Yazi Supplied Skill and resource.