---
name: data-cleaning
description: >
  Applies documented, authorised fixes to the problems a validation pass
  diagnosed, and produces the Data Cleaning Log that lets anyone reconstruct the
  original dataset: correction, recoding, flagging, exclusion or disclosure for
  each issue class, deduplication, category harmonisation, format and encoding
  standardisation, open-end cleaning that does not alter meaning, contradiction
  resolution, and a sensitivity check showing what changes in the headline
  figures with and without every cleaning decision. Use for "clean this dataset",
  "remove the bad respondents", "fix the data", "deduplicate the file", "tidy up
  the brand list", "prepare the data for analysis", "what did you change and why".
category: 04 Data Preparation
ref: "04.02"
tier: 1
inherits: [K2, K3, K4, K5]
---

# Data Cleaning

## 1. One-line description
Turns a diagnosed set of data problems into a set of authorised, reversible and fully documented changes, producing both a clean analysis file and a Data Cleaning Log with the structure original issue, action, reason and impact, from which the original dataset can be reconstructed in full.

## 2. What this skill is used for

**The research problem it solves.** Cleaning is where research data is most often altered without anybody being able to say afterwards what was altered or why. The alterations are almost never malicious. An analyst fixes an obvious typo, standardises a date format, merges two spellings of a brand, drops 40 respondents who look inattentive, and recodes a contradictory pair of answers to the more plausible reading. Each act is defensible on its own. Together they change the dataset in ways nobody has recorded, and six months later, when a client asks why this wave shows 58% and the supplier's own tables show 62%, there is no way to answer. The dataset has become unreconstructable, and with it, the study has become uncheckable.

There is a second and worse failure. Exclusion decisions made after their effect on the results is visible are not quality decisions. An analyst who removes cases, checks the headline, removes a few more, and stops when the number looks right has produced a finding rather than discovered one. This is not a grey area and it is not a matter of degree: **a dataset cleaned until a finding appears is research misconduct.** The protections against it are procedural, not motivational: exclusion criteria written before the effect is seen, exclusion on grounds of quality behaviour rather than answer content, per-decision authorisation by a named researcher, and a sensitivity check that reports the effect of every cleaning decision whether it is convenient or not.

**Where it sits.** Immediately after validation and before any analysis. It is the only point in the lifecycle at which the dataset legitimately changes, and everything downstream inherits both the changes and the log.

**Typical use cases.**
- Executing an authorised cleaning specification against a validation report.
- Removing duplicate and fraudulent records with documented match evidence.
- Harmonising a fragmented category list before any frequency is calculated.
- Standardising formats, encodings and date fields across a multi-source or multi-market file.
- Resolving logical contradictions between related questions with a stated rule.
- Producing the cleaning log a client, an auditor or a next wave will need.
- Re-running a previous wave's cleaning rules on a new wave so the series stays comparable.

**Who uses it.** Data managers and quantitative researchers preparing an analysis file; research directors authorising exclusions and signing off the log; client-side teams reviewing what a supplier changed; anyone who has to answer the question "what did you do to this data".

## 3. When to use it

- A validation report exists, the issues are rated, and the researcher has authorised a treatment for them.
- A dataset must be made analysis-ready and the changes must be defensible to a third party.
- Duplicate or fraudulent records have been identified with evidence and need removing.
- A category, brand or code list is fragmented and the frequencies will be wrong until it is harmonised.
- A file has been assembled from multiple sources, markets or modes with inconsistent formats and encodings.
- A tracker wave needs the same cleaning rules the previous waves received.
- Somebody else cleaned the data, undocumented, and a reconstructable record has to be built retrospectively.
- A quality dispute requires a precise account of what was removed, on what criteria, and with what effect.

## 4. When NOT to use it

- **Nothing has been diagnosed yet. This skill treats; it does not find.** Cleaning without a validation report means the treatment defines the problem, which is exactly backwards, and it means no reviewer can check whether the changes match the issues. Run **04.01 Data Validation** first and let its issue register be the input. If a validation report cannot be produced, say so, and note that everything in the log is unverified against a prior diagnosis.
- **The exclusion is being considered because of what respondents said.** Excluding cases on the basis of their answers, or excluding a subgroup whose responses are inconvenient, or applying a quality rule only to the part of the sample that produces an unwelcome result, is not cleaning. **A case is excluded on quality grounds and never on the basis of its answers.** If a request arrives in the form "check whether the negative responses are genuine", the honest execution is to run the same criteria across the entire sample and report what they catch everywhere, per K4 §4.2, and to say plainly why the narrower request could not be met, per K4 §9.
- **The effect of the cleaning decision on the results has already been seen, and the criteria were not set beforehand.** At that point an exclusion cannot be shown to be independent of its outcome, whatever the analyst's intention. The correct move is to state that the criteria were set after the fact, report the results both ways, and let the reader judge, rather than to present the cleaned figure alone.
- **The problem is missing data.** Deciding whether a non-response is excluded, retained, treated as a substantive category or imputed is a different question with a different logic, and it belongs to **04.03 Missing Data Handling**. Cleaning removes defects. It does not fill gaps, and the two must not be conflated: a value that was never collected is not a value that was collected wrongly.
- **The problem is that the data is in the wrong shape for the analysis.** Recoding into analysis categories, deriving variables, collapsing scales, reversing items, banding continuous variables, reshaping and merging all belong to **04.04 Data Transformation and Dataset Preparation**. The boundary is intent: cleaning restores the data to what it should have been; transformation builds new structures on top of correct data. A recode that fixes an invalid code is cleaning. A recode that collapses five age bands into three is transformation.
- **The imbalance is one weighting is meant to address.** Under-representation of a group is not a data defect and must never be treated by deleting cases from the over-represented group. That is **04.05 Weighting and Base Management**, and doing it by deletion discards real data and produces a smaller sample with the same problem.
- **The dataset is so compromised that no cleaning produces a usable file.** Where duplicate or flagged cases run into the tens of per cent, where a primary measure is missing, or where the base of the key subgroup collapses after defensible exclusions, the honest output is a recommendation to return to field or to descope, not a cleaned file. Cleaning cannot manufacture data that was not collected, and a heavily cleaned file presented as sound is a misrepresentation of the fieldwork.
- **The output will be presented as though it were the original data.** Any figure produced from a cleaned file travels with the cleaning disclosure. If the deliverable format will not carry it, fix the format.

## 5. Required inputs

**Required.** Without these the skill cannot run. If absent, ask. Where work must proceed without an answer, make the conservative choice, state it at the point it bites, and log it, per K5 §5.

- **The raw dataset, preserved and unmodified**, with its fingerprint from 04.01. The original file is never overwritten. Cleaning produces a new file; the original remains as the reconstruction reference.
- **The validation report and issue register** from 04.01, with severity ratings, counts and examples. This is the agenda: every action in the cleaning log should map to an issue in the register, and every unmapped action is a defect in the process.
- **The researcher's authorisation for exclusions**, per K5 §2.1, given before the effect on results is examined. Exclusion is the one class of cleaning action this skill does not take on its own initiative.
- **The questionnaire, routing and codebook**, so that every correction can be checked against what should have been collected rather than against what looks plausible.
- **The exclusion criteria, in writing, with the date they were agreed.** If none were agreed in advance, that fact goes in the log and in the disclosure, and the results are reported both ways.

**Optional, and what each one adds.**

- **The previous wave's cleaning log and rules:** allows identical treatment across waves, without which a wave-on-wave movement may be a cleaning artefact rather than a change in the market. For any tracker this is close to required.
- **Paradata and open-end text:** provide the corroborating evidence that turns a single quality flag into a defensible exclusion, and are usually what distinguishes a genuine duplicate from two similar people.
- **A category master list or house code frame:** turns harmonisation from a judgement call into a mapping exercise, and makes the result comparable with previous studies.
- **Client or sector conventions for quality removal:** determine whether a threshold is a house standard or an invention, which changes how it must be disclosed.
- **The analysis plan:** determines which contradictions matter enough to resolve and which can be left and disclosed, and prevents effort spent cleaning variables nobody will use.
- **A sample or supplier replacement agreement:** determines whether excluded cases can be replaced, which changes the exclusion calculus considerably.

## 6. Questions to ask before starting

1. **Were exclusion criteria agreed in advance, by whom, and on what date?** Determines whether exclusions can be presented as quality decisions at all. *Default if unanswered:* apply no exclusions; flag the candidate cases in the file instead; report results with and without them; and state in the disclosure that no criteria were pre-agreed.
2. **Can excluded cases be replaced from field?** Changes the whole calculus: where replacement is available, a stricter threshold costs nothing but time; where it is not, every exclusion is a permanent reduction in base. *Default:* assume no replacement, and weigh each exclusion against the base it costs.
3. **Is this a tracker wave, and what did previous waves do?** A cleaning rule changed between waves creates an artificial trend. *Default:* replicate the previous wave's rules exactly, apply any new rule to both waves where the historic data allows, and report the difference if it does not.
4. **Which contradictions must be resolved and which can be disclosed?** Resolution requires choosing which of two answers to believe, which is an assumption about the respondent. *Default:* resolve only where a variable feeds a reported measure and the rule is defensible without reference to the outcome; otherwise flag, retain and disclose.
5. **What is the minimum acceptable base after cleaning, overall and for the smallest reported subgroup?** Surfaces the collision between quality and reportability before it happens rather than after. *Default:* apply K4 §7 thresholds, and stop and flag if any planned subgroup falls below them.
6. **Does the client or the sector require a specific quality standard or disclosure format?** Determines the shape of the log and the disclosure. *Default:* use the log format in Section 9, which is designed to satisfy an external audit.
7. **Who signs off the cleaned file?** Per K5 §2.8, the exclusion set and the log carry a named person. *Default:* mark the file provisional and unsigned until a person accepts it, and say so in the disclosure.

## 7. Step-by-step methodology

**1. Freeze the original and work on a copy.** The raw file is preserved, read-only, with its fingerprint recorded. Every subsequent step writes to a working copy, and every change is written to the log as it is made rather than reconstructed afterwards. Reconstructed logs are always incomplete, because the changes that felt too obvious to record are exactly the ones that later cannot be explained. *Correct result:* two files, an untouched original and a working copy, plus a log that is being written in real time.

**2. Classify every issue in the validation register into one of five treatments, before touching anything.** This is the decision framework, and each treatment has a condition that licenses it.

- **Correct.** Available only where the true value is knowable from other data with certainty, not plausibility: a numeric field stored as text, a date in an unambiguous but non-standard format, an encoding artefact with a single possible original character, a total that fails a sum check where the component values are all present and one is transcribed wrongly. If the correction requires a guess about what the respondent meant, it is not a correction.
- **Recode.** For values that are invalid rather than wrong: an out-of-code-list value mapped to a defined missing code, a fragmented category mapped to its canonical form, a scale point mislabelled in the data but unambiguous in the questionnaire. Recoding changes the code, never the substance of the answer.
- **Flag.** For cases or values that are suspect but not demonstrably wrong: quality-flagged respondents, contradictions where neither answer can be preferred, outliers that are extreme but possible. A flag adds a variable and removes nothing, and it is the correct treatment far more often than analysts expect, because it lets the analysis run both ways.
- **Exclude.** For cases that fail pre-agreed quality criteria on evidence, and only with authorisation. Exclusion removes a whole case from analysis; it never removes a single answer from an otherwise sound case, which is what flagging and recoding are for.
- **Leave and disclose.** For issues that cannot be treated without introducing an assumption larger than the problem. Disclosure is a real treatment, not a failure to act, and it is the right answer whenever the fix requires deciding what the respondent meant.

*Correct result:* a treatment plan mapping every register entry to one of the five, with the licensing condition stated, produced before any change is made.

**3. Deduplicate on evidence, and record the match logic.** Establish first what a duplicate is for this unit of analysis. Then work outward from certainty: identical respondent ID is a system fault, not necessarily a duplicate person; identical response vector across the full instrument is near-certain duplication; matching on a combination of demographics, verbatim open-end text and answer pattern is strong evidence; matching on demographics alone is not evidence at all, since two 34-year-old women in the same region are two people. For each duplicate set, record the match criteria met, and record which record was retained and why (usually the more complete, or the earlier, and the rule must be stated and applied consistently rather than chosen per case). *Correct result:* a deduplication table showing every matched set, the evidence, the retained record and the removed ones, with the retention rule stated once and applied throughout.

**4. Harmonise categories against a canonical list, never against each other.** Build the canonical list first, from the questionnaire's intended list, a house code frame or a client master list. Then map each observed variant to a canonical value. Two rules keep this honest. Map only variants that are certainly the same entity: a differing spelling, case, spacing, punctuation or known abbreviation. Do not merge entities that are merely related, because merging a parent brand with a sub-brand is a substantive analytical decision disguised as a tidy-up, and it belongs to 04.04 with the analysis plan behind it. Second, keep the pre-harmonisation value in a retained column or in the mapping table, so the original response remains recoverable. *Correct result:* a mapping table of variant to canonical with counts, and a stated rule for what was not merged and why.

**5. Standardise formats and encodings mechanically, and check what the standardisation touched.** Dates to a single format with the source format recorded, because an ambiguous day-month order silently converted is one of the few cleaning errors that is genuinely undetectable afterwards. Numeric fields to numeric types, with the count of values that failed conversion reported rather than silently nulled. Text encoding to a single standard, with a before-and-after count of affected characters. Whitespace and case standardised for matching purposes only, in a working column, so that the displayed value is not silently changed. *Correct result:* a format log stating what was converted, from what to what, how many values were affected, and how many failed conversion.

**6. Clean open ends without altering meaning.** The permitted operations are narrow and the boundary is the same one K4 §2.3 draws for quotes. Permitted: removing leading and trailing whitespace, standardising encoding, correcting an unambiguous typographical error where the intended word is certain, and marking a response as blank, gibberish or off-topic in a separate quality variable. Not permitted: rephrasing, correcting grammar, expanding abbreviations, standardising terminology, translating, summarising, or inferring meaning from an ambiguous fragment. **The original text is retained in every case**, with cleaned text in a separate column, because an open end is verbatim evidence and a cleaned verbatim that has replaced its original is no longer evidence. *Correct result:* a cleaned text column, an original text column, and a quality classification column, with the permitted-edit convention stated once.

**7. Resolve contradictions with a rule, or do not resolve them.** For each contradiction class, ask whether one answer is demonstrably more reliable than the other on grounds independent of what it says. Grounds that qualify: a screener answer verified against a sample frame outranks a self-report later in the questionnaire; a question with an explicit reference period outranks one without; a filter answer that determined routing outranks a later answer that contradicts it, because the routing already acted on it. Grounds that do not qualify: which answer is more common, which is more plausible to the analyst, or which produces a cleaner result. Where no independent ground exists, the honest treatment is to flag both, retain both, exclude the case from analyses that depend on the contradiction, and disclose. *Correct result:* a contradiction table with the rule applied to each class, the number of cases resolved each way, and the number left unresolved and flagged.

**8. Apply exclusions last, in one authorised block, and count everything.** Exclusions come after all other treatments so that the excluded set is defined against the cleaned file rather than against defects that have since been fixed. Apply the pre-agreed criteria mechanically. Record for every excluded case: the criteria it met, the evidence, and the authorisation. Then report the exclusion accounting in full: total received, excluded by each criterion (with the overlap, since one case may meet several), total analysed, and the effect on every subgroup base that will be reported. Per K4 §7, an analysis that excluded cases states how many and why. *Correct result:* an exclusion table reconciling the received sample to the analysed sample with no unexplained difference, and a subgroup base table showing the post-exclusion position.

**9. Run the sensitivity check, and report it whatever it shows.** For each headline measure named in the objectives, calculate the figure on the cleaned file and on the file before the discretionary cleaning decisions, and report the difference. Discretionary means exclusions, contradiction resolutions and any harmonisation involving judgement; it does not include mechanical corrections, which cannot go the other way. Report the comparison as a table, for every headline measure, not only the ones where the difference is small. Two readings matter. Where cleaning moves a headline by less than about a point, the cleaning decisions are not load-bearing and this can be stated as reassurance. **Where cleaning moves a headline materially, that is not a reason to reconsider the cleaning: it is a disclosure obligation**, and both figures belong in the output with the reason for preferring the cleaned one. *Correct result:* a sensitivity table covering every headline measure, published in the deliverable rather than held in the working file.

**10. Complete the Data Cleaning Log to reconstruction standard.** The test is specific and demanding: **a researcher holding the original file and this log alone must be able to reproduce the cleaned file exactly, and a researcher holding the cleaned file and this log must be able to reconstruct the original.** That requires, per entry, the affected case IDs or a reproducible selection rule, the before value and the after value or a reversal instruction, and enough of the rule that it could be re-executed. A log entry reading "standardised brand names" fails. One reading "mapped six variants of one brand name to a canonical value in Q11, 214 cases affected, mapping table attached, original values retained in Q11_raw" passes. *Correct result:* a log that passes both reconstruction directions on a spot test of three entries chosen at random.

**11. Verify the cleaned file against the original.** Row count reconciled: original rows, minus exclusions, equals analysis rows, with no residual. Column count reconciled, including new flag and original-value columns. For a sample of at least twenty changed cases spread across every treatment class, check the before and after values against the log by hand. Confirm that no variable outside the log has changed, by comparing the cleaned file to the original on every untouched column. That last check catches the single most common cleaning accident, which is a sort, merge or type conversion that shifted values without anybody intending it. *Correct result:* a verification note recording each reconciliation, the sample checked, and any discrepancy found and resolved.

**12. Write the cleaning disclosure, and attach it to the file.** Short, plain, and travelling with every downstream deliverable: how many cases were received and analysed, how many were excluded and on what criteria, whether those criteria were pre-agreed and when, what classes of change were made, whether any headline figure moves materially with or without the cleaning, and who authorised and signed off. Per K2 §7 a caveat left behind in the analysis file is a caveat that does not exist. *Correct result:* a disclosure block a report writer can lift verbatim into a methodology section.

## 8. Analytical framework

Every entry in the Data Cleaning Log carries the same six fields, and this is the structure the whole of category 04 shares:

    Original issue → Action → Reason → Impact → Reversal → Authorisation

**Original issue.** The defect as diagnosed, referenced to its ID in the 04.01 register, with the count of cases affected.
**Action.** Which of the five treatments was applied, and exactly what was done, at a level of detail that permits re-execution.
**Reason.** The licensing condition for that treatment, and where a judgement was made, what the judgement turned on. "It was wrong" is not a reason. "The value fell outside the code list and could not be mapped to any valid code" is.
**Impact.** What this changes in the analysis: which measures are affected, how many cases, and where relevant the movement in the figure.
**Reversal.** How to undo it: the retained original column, the mapping table, the case ID list, or the selection rule.
**Authorisation.** Who approved it, and when. Mechanical corrections carry the analyst; exclusions carry a named researcher, per K5 §2.1.

Against the K2 chain, cleaning operates entirely below Finding. It changes evidence, and for that reason it is the step at which traceability is most easily and most permanently broken. The log is what keeps the chain intact across the break.

## 9. Output format

**1. Cleaning disclosure block.** Received n, analysed n, excluded n by criterion, whether criteria were pre-agreed and when, treatment classes applied, sensitivity summary in one line, authorisation and sign-off, and whether AI performed any cleaning step and whether a human verified it, per K4 §7.

**2. Data Cleaning Log.** The required output. One row per action.

| Log ID | Validation issue ID | Original issue | Variables | Cases affected | Action | Reason | Impact on analysis | Reversal instruction | Authorised by | Date |
|---|---|---|---|---|---|---|---|---|---|---|

**3. Exclusion accounting.**

| Criterion | Definition | Pre-agreed (Y/N, date) | n excluded | Overlap with other criteria | Cumulative n remaining |
|---|---|---|---|---|---|

Ending with: received, total excluded, analysed, reconciled to zero residual.

**4. Post-exclusion base table.** Every subgroup that will be reported, before and after, with any that fall below the K4 §7 thresholds flagged explicitly.

**5. Deduplication table.** Matched sets, match criteria met, record retained, records removed, retention rule.

**6. Category mapping table.** Variant, canonical value, count, and the list of variants deliberately not merged with the reason.

**7. Contradiction resolution table.** Contradiction class, rule applied, resolved one way, resolved the other, left unresolved and flagged.

**8. Sensitivity check.**

| Headline measure | Figure, cleaned | Figure, before discretionary cleaning | Difference | Base, cleaned | Base, before |
|---|---|---|---|---|---|

Every headline measure appears, not a selection.

**9. Verification note.** Reconciliations run, sample size checked by hand, untouched-column comparison result, discrepancies found and how resolved.

**10. Review points**, per K5 §3.3, and **what was left untreated and disclosed**, with the reason.

**Where the evidence is thin**, the format does not get filled. An issue that could not be treated appears in section 10 as left and disclosed, not as an action with a vague description. A duplicate set with weak match evidence stays in the file as a flag, not as a removal. A contradiction with no independent resolution rule is reported unresolved. Per K4 §1, the log having a row for something is not a reason to act on it.

## 10. Quality checks

Run before the cleaned file is released. These sit on top of K4 §8.

1. Does the original file still exist, unmodified, with a matching fingerprint?
2. Does every action in the log map to an issue in the validation register, and is every unmapped action explained?
3. Can the original be reconstructed from the cleaned file plus the log, tested on at least three randomly chosen entries?
4. Does every exclusion carry its criterion, its evidence, and a named authorisation?
5. Were the exclusion criteria set before the effect on results was examined, and if not, is that stated in the disclosure?
6. Has any case been excluded on the basis of the content of its answers rather than the manner of its responding?
7. Does the exclusion accounting reconcile received to analysed with no residual?
8. Are the original values retained for every recoded, harmonised or cleaned field, including open ends?
9. Has any open end been rephrased, expanded, translated or corrected beyond the permitted edits?
10. Has any category merge combined entities that are related rather than identical?
11. Does the sensitivity check cover every headline measure rather than a selection?
12. Has any variable outside the log changed, tested by comparing untouched columns against the original?
13. Do any reported subgroup bases fall below the K4 §7 thresholds after exclusion, and is that flagged?
14. For a tracker, were the previous wave's rules applied, and is any deviation documented?
15. Does the disclosure block travel with the file, and is it usable verbatim in a methodology section?

## 11. Common failure modes

| Failure | How to recognise it | How to prevent it |
|---|---|---|
| **Cleaning to a result** | Exclusions applied in rounds with the headline checked between them; a threshold that lands on a suspiciously convenient number | Criteria in writing with a date, applied once, mechanically; sensitivity check published whatever it shows |
| **Exclusion on answer content** | A quality rule applied only to one subgroup, or a request to verify only the unfavourable responses | Quality criteria describe how a person answered, never what they said; run every criterion across the whole sample |
| **The undocumented obvious fix** | A value that differs from the original with no log entry, usually a typo or a format | Write the log as you go, not afterwards; verify by comparing every untouched column |
| **Log too vague to reverse** | Entries like "cleaned brand names" or "removed poor quality cases" | Test reconstruction on three random entries before release; every entry names cases, before, after and how to undo |
| **Silent format conversion** | Ambiguous dates converted to one interpretation; numeric strings nulled on failed conversion | Record source format and conversion counts; report failed conversions rather than nulling them |
| **Over-merging categories** | A parent brand and its sub-brands combined during a tidy-up | Merge only variants of the same entity; related entities are an analysis decision for 04.04 |
| **Verbatim altered** | An open end that reads more fluently than the respondent wrote it | Original text always retained in its own column; permitted edits limited to whitespace, encoding and certain typographical error |
| **Contradiction resolved by plausibility** | "We kept the more sensible answer" | Resolution requires a ground independent of the content; where none exists, flag and disclose |
| **Deletion used as weighting** | Cases removed from an over-represented group to improve balance | Composition is corrected by weighting in 04.05, never by deletion |
| **Base collapse discovered late** | A key subgroup below reportable size only after cleaning is finished | Check post-exclusion subgroup bases before applying exclusions, and flag the collision as a decision |
| **Tracker rule drift** | A wave-on-wave movement that coincides with a cleaning change | Replicate previous rules exactly; where a rule changes, apply it to both waves or report the effect |
| **AI: plausible correction** | A value "corrected" to something reasonable rather than something knowable | Correction is licensed only where the true value is determinable; otherwise recode, flag or disclose |
| **AI: tidying beyond instruction** | Whitespace, case, punctuation or terminology standardised across the file without a log entry | Every change is logged, including cosmetic ones; cosmetic changes go in working columns, not over the original |
| **AI: inventing an authorisation** | An exclusion executed with no named approver, or the analyst recorded as the approver of their own exclusion | Exclusions require a named researcher per K5 §2.1; without one, flag the cases and do not remove them |

## 12. AI guardrails

Skill-specific only. K4 applies in full and is not repeated here.

1. **Never make a change that is not written to the log at the moment it is made.** A change discovered later and back-filled is a defect in the record even when the change itself was right.
2. **Never overwrite, delete or re-save the original file.** Cleaning produces a new file. The original is the reconstruction reference and it is not editable.
3. **Never exclude a case without explicit human authorisation**, per K5 §2.1. Where no authorisation is available, flag the cases, retain them, report both ways, and say the exclusion was not authorised.
4. **Never exclude a case on the basis of what it answered.** Quality criteria describe response behaviour only. A request to apply criteria selectively to part of the sample is refused and explained, per K4 §9.
5. **Never set or adjust an exclusion threshold after seeing its effect on a result.** If the effect has already been seen, say so and report both figures.
6. **Never correct a value unless the true value is determinable from other data.** Plausibility is not determinability. Where it is not determinable, recode to missing, flag, or leave and disclose.
7. **Never alter an open end beyond whitespace, encoding and a certain typographical correction**, and never without retaining the original text. Rephrasing, expanding, translating or standardising a verbatim destroys evidence, per K4 §2.3.
8. **Never merge two categories that are related rather than identical**, and never merge without a mapping table that records what was combined.
9. **Never suppress or omit the sensitivity check because the cleaning moved a figure.** A large movement is a disclosure obligation, not a reason to revisit the decision.
10. **Never present a cleaned figure without the cleaning disclosure attached**, and never let a downstream deliverable carry the figure without it.
11. **Never treat missing data in this skill.** Hand to 04.03. Recoding an invalid code to a missing code is cleaning; deciding what to do about the resulting gap is not.

## 13. Best-practice principles

- **The log is the deliverable.** The clean file is replaceable; it can be rebuilt from the original and the log. The log cannot be rebuilt from anything. Treat it as the primary output and the file as its by-product.
- **Reconstruction is the test, not tidiness.** A cleaning record is adequate when someone else can undo it. Every other standard is subjective, and every other standard has been met by records that turned out to be useless.
- **Flagging is underused and exclusion is overused.** A flag preserves the case, permits the analysis to be run both ways, and costs nothing but a column. Reach for it first, and reserve exclusion for cases that genuinely cannot contribute anything trustworthy.
- **Criteria before consequences.** The entire difference between a quality decision and a manufactured finding is whether the rule existed before its effect was visible. Write the rule down with a date, and the question never arises.
- **Cleaning cannot rescue bad fieldwork.** If a defensible clean leaves a base too small or a subgroup unreportable, the finding is about the fieldwork, and the honest output is to say so rather than to relax the criteria until the numbers survive.
- **The obvious fix is the dangerous one.** Almost every unlogged change was unlogged because it seemed too small to matter. Those are precisely the changes nobody can explain a year later.
- **Cosmetic standardisation is still alteration.** Trimming whitespace, unifying case and normalising punctuation change the data. Do them in working columns, log them, and leave the displayed values alone.
- **Two similar respondents are not a duplicate.** Demographic matches are cheap; response-pattern and verbatim matches are expensive to produce by coincidence. Require the expensive evidence before removing a person.
- **A contradiction is information.** Respondents contradict themselves for reasons: an ambiguous question, a changed reference period, a misread scale. Before resolving one, ask what it says about the instrument, because that is often worth more than the resolved value.
- **Sensitivity analysis protects the analyst.** Publishing what the cleaning changed is the single most effective defence against the accusation that it was done to a purpose, and it costs one table.
- **Consistency across waves outranks improvement within one.** A better rule introduced mid-tracker creates a movement nobody can interpret. Fix it at a declared break, apply it backwards where possible, and say what changed.

## 14. Worked example

**INPUT.** A fictional financial services provider runs an annual customer study, wave four, achieved sample 2,100. The 04.01 validation register lists two Critical, five Major and nine Minor issues. Exclusion criteria were agreed with the research director in writing before fieldwork closed: exclusion requires two or more of the four quality flags, or a confirmed duplicate. Objectives concern satisfaction, product understanding and switching intent.

**PROCESS.**

*Steps 1 to 2.* Original frozen and fingerprinted. Treatment plan built from the register: 3 issues to correct, 6 to recode, 4 to flag, 2 to exclusion, 1 to leave and disclose. The single leave-and-disclose case is a question whose reference period changed between wave three and wave four, which cleaning cannot fix and which must travel as a comparability caveat to **04.04** and the trend analysis.

*Step 3.* Eleven duplicate candidates. Four sets match on full response vector and open-end text: removed, with the earlier record retained by the stated rule. Seven match on demographics and two closed answers only: retained and flagged, because demographic coincidence in a sample of 2,100 is expected and the evidence does not support removal. The log records both decisions and the reasoning, so a reviewer can disagree with the second one on the evidence rather than in the abstract.

*Step 4.* The product-held list contains 31 observed variants of 14 canonical products. Mapping table built from the client's own product master. **Judgement call:** two variants name a product and its premium tier. Merging them would tidy the list and would also merge two products the client prices and reports separately. Not merged, recorded in the not-merged list with the reason, and referred to 04.04 in case the analysis plan wants a net of the two.

*Step 6.* 1,340 open ends. Whitespace and encoding standardised into a cleaned column, originals retained. 62 classified as blank, gibberish or single-character in a quality variable. Nothing rephrased. A draft that had "expanded common abbreviations for readability" was reverted, because expanding an abbreviation is an inference about meaning and the abbreviation may be the finding.

*Step 7.* 84 respondents said at Q6 they held no products and then rated a product at Q14. The routing filter at Q6 determined what they were shown, so the Q6 answer is the one the instrument acted on and is preferred on grounds independent of content. Those 84 are recoded out of base at Q14 and flagged. A second contradiction class, 39 cases where tenure exceeds account age, has no independent ground: neither field is verified. Left unresolved, flagged, excluded from the derived tenure band only, and disclosed.

*Step 8.* Applying the pre-agreed rule: 96 cases meet two or more quality flags, 4 are confirmed duplicates, overlap 2, so 98 excluded. 2,100 received, 98 excluded, 2,002 analysed, reconciled. Post-exclusion subgroup check: the smallest reported segment falls from 108 to 97, crossing the small-base threshold. Flagged as a **researcher review point** per K5 §2.1, since whether to report that segment directionally or hold it is a materiality judgement about the client's decision, not an analytical one.

*Step 9.* Sensitivity check on all six headline measures. Five move by less than a point. Satisfaction top-two-box moves from 61% to 63% with cleaning. The two-point movement is reported in the deliverable with both figures, the base for each, and the reason for preferring the cleaned figure (the excluded cases include 71 straight-liners on the satisfaction battery itself, whose invariance is not informative about satisfaction). It is not a reason to revisit the exclusion rule, and the log says so.

*Steps 10 to 12.* Log completed with 21 entries. Reconstruction tested on three random entries: all three reversible. Untouched-column comparison finds one unintended change, a numeric field silently converted during an early merge, which is reverted and logged. Disclosure block written and attached to the file.

**OUTPUT.** An untouched original; a cleaned file of 2,002 cases; a 21-entry Data Cleaning Log passing the reconstruction test in both directions; exclusion accounting reconciling to zero residual; a deduplication table with match evidence; a mapping table including the deliberate non-merge; a contradiction table with 84 resolved and 39 left flagged; a sensitivity table covering all six headline measures including the two-point movement; a verification note recording the one accidental change found and reverted; one review point on the small segment; and a disclosure block ready to lift into the methodology section.

## 15. Advanced usage

**Retrospective log building.** Where a dataset was cleaned undocumented, the log can be partially reconstructed by comparing the cleaned file to the original column by column and case by case. What can be recovered: which values changed, how many, and in which variables. What cannot: why, and on whose authority. Produce the difference report, mark every entry's reason and authorisation as unrecoverable, and state plainly that the cleaning cannot be audited. That is a more useful and more honest artefact than a plausible reconstruction of reasons nobody recorded.

**Multi-market and multi-mode files.** Clean each market and each mode separately against its own instrument before merging, because a rule appropriate in one is frequently wrong in another: completion-time thresholds differ by language and by device, category variants differ by market, and date conventions differ by country. Merge cleaned files, then run a second thin cleaning pass on the merged file for cross-market consistency only, with its own log section.

**Tracker cleaning specifications.** Write the cleaning rules once as a specification, version it, and apply the version by wave. When a rule needs to change, change it at a declared break, apply the new rule retrospectively to the previous wave where the archive allows, publish both series for one wave, and record the version in the log. A tracker whose cleaning rules drift silently produces trends that are artefacts, and they are indistinguishable from real change without the version record.

**Cleaning where replacement is available.** If field can replace excluded cases, the sequence changes: exclude early, replace, and re-run validation on the topped-up file. Record replacements as a separate class in the log, because a replaced case is not the same as an excluded one for the purposes of describing the sample, and the composition of the replacements needs checking against the original quota.

**When the standard approach does not fit.** For passive, behavioural or transactional data there are no open ends and no quality flags, and the equivalent decisions concern session integrity, event deduplication and coverage gaps. Keep the five treatments and the six log fields, and replace the checks. For very small qualitative-scale samples, exclusion is rarely appropriate at all: with 24 participants, removing one is a 4% change in the evidence base and the flag-and-disclose route is almost always better.

## 16. Skill chain

**Recommended previous skills:**
- **04.01 Data Validation.** Hands over the severity-rated issue register that is this skill's agenda, and the untouched fingerprinted original that makes reconstruction possible. This skill should not run without it.
- **03.04 Fieldwork Monitoring and Response Quality.** Hands over the in-field removals already made, so the exclusion accounting reconciles to what was actually fielded rather than to what arrived.
- **02.05 Survey Logic and Flow Review.** Hands over the routing that licenses contradiction resolution on independent grounds.

**Recommended next skills:**
- **04.03 Missing Data Handling.** Takes the cleaned file, in which invalid values have become defined missing codes, and decides what to do about the resulting gaps.
- **04.04 Data Transformation and Dataset Preparation.** Takes clean, harmonised data and builds the derived variables, nets, bands and structures the analysis plan needs.
- **04.05 Weighting and Base Management.** Takes the post-exclusion achieved sample and its composition, which is what the weighting is calculated against.
- **05.01 Descriptive Analysis.** Takes the cleaned file plus the disclosure block, which every figure it produces must carry.

**Runs well alongside:**
- **K5**, for the authorisation requirement on every exclusion, and **K2 §7**, for why the disclosure must travel with the figures.
- **13.01 Research Quality Review**, where the cleaning log is one of the primary documents an external reviewer inspects.
- **13.06 AI Research Governance**, where any AI-executed cleaning step must be disclosed and human verification recorded.

---
A Yazi Supplied Skill and resource.