---
name: large-scale-text-analytics
description: >
  Analyses text at volumes where reading every item is impractical, roughly 5,000
  items and up: classification against a pre-built frame, unsupervised topic
  discovery, gold-set construction, per-class precision and recall, the systematic
  under-detection of rare but important content, deduplication and template or bot
  text, language and code-switching effects, and volume as a misleading signal.
  Requires classification accuracy to be reported alongside every result derived
  from it. Use for "analyse 40,000 reviews", "code 12,000 open ends", "run topic
  modelling on this corpus", "what are people complaining about at scale", "how
  accurate is the auto-coding", "classify these support tickets", or "we have too
  much text to read".
category: 06 Specialist and Advanced Analysis
ref: "06.06"
tier: 2
inherits: [K2, K3, K4, K5]
---

# Large-Scale Text Analytics

## 1. One-line description
Turns a corpus too large to read into measured, auditable evidence, by building or importing a frame, validating a classifier against human labels per class, reporting the accuracy of every derived number, and actively hunting the rare content that volume-driven methods are structurally guaranteed to miss.

## 2. What this skill is used for

**The research problem it solves.** Above a few thousand items, reading everything stops being possible, and the failure modes change character. At small scale the risk is the analyst's interpretation. At large scale the risk is that a machine assigned labels nobody checked, a proportion was computed on those labels, and the proportion was reported as though it were a measurement. Five specific failures follow. Accuracy is reported as a single overall figure, which hides that the machine handles the large common classes well and the small important ones badly, and it is precisely the small classes that carry the risk. Topic model output is treated as though the topics were real categories, when they are statistical groupings of co-occurring words that may correspond to nothing anyone meant. Rare content is under-detected systematically, so a corpus containing forty accounts of a serious safety problem among sixty thousand reviews returns nothing, and the report says the theme did not appear. Volume is read as importance, when volume measures who writes: the articulate, the extremely angry, the incentivised and the automated. And duplicates, templates and bot text inflate everything without anyone noticing. This skill supplies the validation discipline that makes text-at-scale reportable, and the required disclosure that makes it auditable.

**Where it sits in the research lifecycle.** After a corpus has been assembled and before its content becomes findings. It sits downstream of the qualitative skills where a frame already exists, and upstream of insight development. Its most important output is frequently not the classification but the statement of what the classification cannot see.

**Typical use cases.**
- Coding tens of thousands of open-ended survey responses against an existing frame.
- Classifying support tickets, reviews or complaints into a service or product taxonomy.
- Discovering the structure of a corpus nobody has read, as a first pass before building a frame.
- Tracking the prevalence of known themes across time or across markets at scale.
- Searching a large corpus for rare, serious content that prevalence methods miss.
- Validating and reporting the accuracy of an automated coding process someone else ran.
- Assessing whether a reported text finding is a real signal or an artefact of who writes.

**Who uses it.** Research managers and analysts handling text volumes beyond manual coding; insight teams running always-on listening or review analysis; customer experience analysts working with contact records; public sector and regulatory teams handling consultation responses; and researchers auditing a supplier's automated coding claims.

## 3. When to use it

- The corpus is large enough that reading every item is impractical, roughly 5,000 items and above.
- An automated coding or classification process has been run and its accuracy has not been established.
- A frame exists from smaller-scale qualitative work and needs applying at volume.
- Nobody knows what is in a corpus and a first structural pass is needed before a frame can be built.
- Prevalence of known themes needs measuring across time, market or segment at scale.
- A serious but rare issue is suspected in a corpus and needs looking for, which is a different task from measuring what is common.
- A text-derived number is about to be reported and the disclosure requirements have not been met.

## 4. When NOT to use it

- **The corpus is small enough to read.** Below roughly 5,000 items, and often well above it where the items are rich, human or AI-assisted reading of every item produces better analysis than any classifier, because the interpretive work is the value and the classifier's error rate buys nothing. Open ends go to **07.02 Open-Ended Response Coding**, which owns the code frame construction, the coding of the full set and the reporting conventions for survey text. Depth material goes to **07.01 Thematic Analysis**. The boundary runs in both directions and is stated identically in those skills: below the threshold, this skill hands the work to them; above it, they hand the work here, and this skill still depends on them for the frame.
- **The question requires interpretation rather than measurement.** What a theme means, how participants construct an account, what is significant about what was not said, how two contradictory statements by the same person relate: none of this is a classification problem, and applying a classifier to it produces counts of things that are not the finding. **07.01** owns interpretive work at any volume, and a large corpus does not change that; it changes only how the sampling for interpretation is done.
- **The corpus is not a sample of anything.** Reviews, social posts and complaints are written by a self-selected minority, and the population they represent is "people who wrote", which is not the customer base, the public, or the market. Where the research question is about a population, a text corpus cannot answer it, however large. Report the corpus as what it is and, where prevalence in a population is needed, say what design would supply it.
- **The classifier's accuracy on the class of interest cannot be established.** Where a class is too rare to build a gold set for, or where human coders cannot agree on it, no number about that class is reportable. This is common and it usually affects the most interesting classes. Say so rather than reporting the machine's count.
- **Rare, serious content is the question and prevalence methods are the proposed answer.** Finding forty safety reports among sixty thousand items is a search problem, not a classification problem, and a classifier optimised on overall accuracy will miss them by design. Step 9 describes the targeted approach. Do not answer a search question with a prevalence method.
- **The corpus is dominated by duplicates, templates or automated text and this has not been resolved.** Where a large share of items are near-identical, every proportion is a proportion of a corpus that does not exist. Deduplication comes before analysis, not after, and where the duplicate structure cannot be resolved, the corpus is not analysable at scale.
- **The text is in languages the pipeline was not validated in.** Accuracy measured on one language does not transfer to another, and machine translation before classification introduces errors that are systematic rather than random, concentrated in exactly the idiomatic and emotional language that carries meaning. Validate per language or report per-language coverage as unassessed.
- **The output will be reported without the accuracy figures.** Where a stakeholder or a format will not accommodate the disclosure required by **K4 §7**, the analysis should not be produced, because a machine-generated proportion presented as a measurement is the specific harm this skill exists to prevent.

## 5. Required inputs

**Required. Without these the skill cannot run. If absent, stop and ask.**
- **The corpus, complete, with metadata per item.** At minimum: an item identifier, the text, the source, the date, and any available author identifier. The identifier is not optional, because every derived number must be traceable back to specific items (**K2 §4.2**) and a claim about a theme that cannot be resolved to a set of items is not checkable.
- **The research question, stated as either a measurement question or a discovery question.** These need different methods and produce different outputs. "How many mention delivery delays" is measurement and needs a frame. "What are people talking about" is discovery and produces candidate structure, not findings.
- **How the corpus was assembled.** The collection rule, the platform or channel, the time window, any filtering already applied, and any sampling. This determines what population, if any, the corpus relates to, and it is frequently unknown for inherited corpora, in which case that is the first thing to report.
- **Human capacity for labelling.** Gold-set construction requires human labels, and without any human labelling capacity there is no way to measure accuracy and therefore nothing reportable. State how much is available, because it determines how many classes can be validated.

**Optional, and what each one adds.**
- **An existing code frame from smaller-scale work.** The strongest optional input. A frame built by **07.01** or **07.02** on a read sample carries the interpretive work that a machine cannot do, and turns the exercise into measurement against a meaningful structure rather than discovery of an arbitrary one.
- **Author-level identifiers.** Allow the corpus to be analysed by author rather than by item, which is essential because a small number of prolific authors can dominate an item-level count.
- **Structured metadata (product, market, channel, date, rating).** Allows cross-cutting analysis and, critically, allows a stratified gold set so accuracy can be reported by stratum rather than only overall.
- **Previous periods of the same corpus.** Enable trend measurement, valid only where the collection rule, the classifier and the frame are all unchanged.
- **Known ground-truth cases.** Where the organisation knows of specific incidents that should appear in the corpus, they provide a direct recall test on exactly the rare-and-important content the method is worst at.
- **Language identification, where multilingual.** Enables per-language validation and prevents accuracy from one language being quoted for all.

## 6. Questions to ask before starting

1. **Is this a measurement question or a discovery question?** Determines whether the output is prevalence against a frame or candidate structure for human interpretation. Default if unanswered: run discovery first and state clearly that its output is not findings.
2. **What is the smallest class that matters, and does anyone care about it?** Determines whether the analysis can be optimised for overall accuracy or must be built for recall on rare classes, which are different pipelines. Default: ask directly, because the answer usually reveals a safety, compliance or reputational class the brief did not mention.
3. **Who wrote this text, and what selects them into the corpus?** Determines what any proportion means. Default: describe the selection mechanism explicitly and attach it to every prevalence figure.
4. **How much human labelling is available?** Determines how many classes can be validated and at what precision. Default: prioritise gold-set coverage on the classes that will be reported and on the rare-important ones, and declare the rest unvalidated.
5. **What languages and varieties are present, and in what proportion?** Determines whether one accuracy figure can stand for the corpus. Default: run language identification and report per-language volume even if per-language validation is not possible.
6. **Will the output be reported with its accuracy figures?** Determines whether the analysis should proceed at all, given **K4 §7**. Default: state the requirement at the outset and treat refusal as a stop condition.
7. **Is there a known incident or issue that should be findable in this corpus?** Provides a free recall test on the hardest part of the task. Default: ask, and where one exists, use it in Step 9.

## 7. Step-by-step methodology

**Step 1. Profile the corpus before analysing it.** Count items, unique authors, items per author, date distribution, source distribution, language distribution, and text length distribution. This profile changes the analysis more often than not. A corpus of 60,000 items from 4,000 authors is a different object from one of 60,000 items from 58,000 authors, because in the first, a handful of prolific authors can dominate any item-level count. A length distribution with a spike at two words means a large share of items carry no analysable content. A date distribution with a spike means an event, a campaign or a scraping artefact. A correct result is a profile table and a short statement of what it implies for the analysis.

**Step 2. Deduplicate, and identify template and automated text.** Three passes. *Exact duplicates:* identical strings, removed with the count logged. *Near duplicates:* items differing only in a name, a number or whitespace, found by normalised comparison or by a similarity threshold, which catches copy-paste campaigns and form responses. *Templates and automated text:* items sharing a fixed structure with variable slots, auto-responses, signature blocks, system-generated confirmations, and coordinated posting. These do not always get removed, because a coordinated campaign can be the finding, but they must be identified and analysed separately, since including them silently means every proportion measures the campaign as though it were opinion. Log the counts at each pass and report the corpus size before and after. A correct result is a cleaned corpus, a duplication log, and a separate set of identified template or automated items with a decision recorded for each group per **K4 §4.4**.

**Step 3. Establish who writes, and attach the answer to every number that follows.** A text corpus is generated by a selection process: people who bothered to write, on the platform in question, in the language of the platform, under whatever incentive existed. That process is not random with respect to the content, and its effects are large. Reviews cluster at the extremes because moderate experiences do not motivate writing. Complaint corpora are dominated by the articulate and the persistent. Incentivised reviews differ systematically from unincentivised ones. Public consultation responses over-represent organised interests. Quantify what you can: items per author, share of items from the top one percent of authors, share of items from accounts created within the window. Then state the selection mechanism in one sentence that will travel with every proportion. **Volume is a measure of who writes, not of what matters**, and that sentence is a required element of the output, not a caveat. A correct result is a written selection statement and, where author identifiers exist, an author-level view alongside the item-level one.

**Step 4. Decide between classification against a frame and unsupervised discovery, and say what each can deliver.** Classification against a pre-built frame answers "how much of this known thing is here", produces comparable numbers over time and across cuts, can be validated per class, and cannot find anything the frame does not contain. Unsupervised discovery answers "what is in here", surfaces structure nobody anticipated, and produces groupings that are statistical artefacts requiring human interpretation before they mean anything. They are complementary and the usual correct sequence is discovery first on a corpus nobody has read, human reading of samples from each discovered grouping, frame construction from that reading, then classification against the frame. Skipping the human reading step is the error that produces analyses of topics nobody understands. A correct result is a stated method choice with its consequence written down.

**Step 5. Build or import the frame, and require it to have come from reading.** A frame is a set of mutually understood categories with definitions, inclusion and exclusion rules, and examples. Where one exists from **07.01** or **07.02**, import it and check it fits: read a sample of a few hundred items and record how many fall outside every category, since a frame built on a different corpus will have a large residual. Where none exists, build one by reading a sample, which is the qualitative work, not this skill, and hand it to **07.02** for a survey corpus or **07.01** for richer material. The frame must include an explicit "other" category and an explicit "no analysable content" category, because a frame without them forces every item into a category it may not belong in. A correct result is a documented frame with definitions and boundary examples, plus the residual rate from the fit check.

**Step 6. Build the gold set properly, because everything downstream rests on it.** The gold set is a sample of items labelled by humans against the frame, and it is the only source of truth about accuracy. Five requirements. *Sample properly:* random from the deduplicated corpus, plus stratified over-sampling of rare classes, since a random 500 from a corpus where a class occurs at 0.5% contains two or three examples and measures nothing about it. *Double-code a portion* independently, measure agreement and adjudicate disagreements; where humans agree poorly on a class, no classifier can do better and the definition needs work before anything is measured. *Hold out:* the set used to measure accuracy must not be the data used to build or tune the classifier, or the figure is an optimistic in-sample fit. *Size it for the claim:* an accuracy estimate on 50 items has an interval of roughly plus or minus fourteen points. *Document the labelling instructions*, because they are the operational definition of every category. A correct result is a held-out, adjudicated gold set with its sampling rule, its size per class and its inter-coder agreement recorded.

**Step 7. Measure accuracy per class, never overall.** An overall accuracy figure on an imbalanced corpus is close to meaningless: a classifier that labels everything as the largest class can score 70% overall while detecting nothing else. Report, per class: **precision** (of the items the machine assigned to this class, what share genuinely belong), which governs whether a reported example set is trustworthy; **recall** (of the items that genuinely belong to this class, what share the machine found), which governs whether a prevalence figure is an undercount; and their combination where a single figure is needed. Produce the confusion matrix, because the pattern of errors is diagnostic: two classes that the machine confuses with each other are usually two classes humans also confuse, meaning the frame has a boundary problem, not the classifier. Note that precision and recall trade off: tuning to catch more of a class inevitably admits more false positives, and where the class matters, that trade is usually worth making and should be made deliberately rather than accepted as a default. A correct result is a per-class table of precision, recall and support, plus the confusion matrix.

**Step 8. Set a reporting threshold per class and enforce it.** Decide, before seeing the results, what accuracy a class needs to be reportable, and set it by consequence rather than by convention: a class that will be quoted in a board paper needs more than one used for internal triage. Classes below the threshold have three fates: report with the accuracy stated prominently and the direction of error named, merge with an adjacent class where the confusion matrix shows the boundary is the problem, or do not report at all. **A class the classifier cannot detect reliably is not converted into a reportable number by being counted more precisely.** A correct result is a decision per class, recorded, including the classes withheld and why.

**Step 9. Hunt the rare and important content explicitly, because the pipeline will not find it.** This is the most important step in the skill and the one most often absent. Automated classification is optimised, implicitly or explicitly, on aggregate performance, which means it performs worst on the classes with fewest examples. Those are frequently the classes that matter most: safety incidents, discrimination, regulatory breaches, fraud, severe harm. **The largest risk in text analytics is not a wrong percentage; it is silence about something that mattered.** Four techniques, used together. *Targeted keyword and pattern search* with a deliberately over-inclusive term list built with domain input, accepting a high false positive rate and reading everything it returns: recall is the objective and precision is irrelevant here. *Anomaly and outlier review:* read the items the classifier was least confident about, the items that matched no category, and the items that are unusual on length, sentiment or vocabulary. *Known-case recall test:* where the organisation knows of specific incidents that should be in the corpus, check whether the pipeline finds them, which is a direct, honest measure of the method's blind spots. *Reading a random sample regardless:* a few hundred items read by a person, purely to see what the machine's categories do not capture. A correct result is a rare-content search with the term list, the volume reviewed, what was found, and an explicit statement that absence of a theme in the classification output is not evidence of its absence in the corpus.

**Step 10. Where topic modelling is used, treat the output as a hypothesis and validate it.** Unsupervised topic output is a set of word groupings produced by co-occurrence statistics. Four disciplines. *Topics require human naming, and the name is an interpretation*, so it is marked as such and the top terms plus several representative items are shown alongside it so a reader can judge the naming. *Some topics are meaningless*, being artefacts of vocabulary, boilerplate, a single event or a specific author, and the correct handling is to discard them rather than to force an interpretation. *Stability must be checked:* run the model more than once with different initialisations and different topic counts, and report only the topics that persist, since a topic that appears at one setting and vanishes at another is not a feature of the corpus. *The number of topics is a choice, not a result*, and it should be selected on interpretability by a human reading examples, with the alternatives tried reported. A correct result is a topic table with machine terms, human name flagged as interpretation, representative items with identifiers, stability status, and discarded topics listed with the reason.

**Step 11. Handle language, dialect, code-switching and translation as first-order problems.** Four effects. *Per-language accuracy differs*, often substantially, so a single figure quoted across a multilingual corpus is misleading and per-language validation is required for any language whose volume affects a reported number. *Machine translation before classification introduces systematic error*, concentrated in idiom, sarcasm, emotional intensity and culturally specific reference, which is exactly the content that carries meaning; where translation is unavoidable, validate translated text against human labels of the original. *Code-switching within an item* defeats language identification and often classification, so mixed-language items are counted and their handling stated. *Dialect and register variation* matters: a pipeline validated on formal written text underperforms on colloquial or abbreviated writing, which is most of what people write. A correct result is a language profile with per-language volume, per-language accuracy where measured, and an explicit statement wherever accuracy was not established for a language.

**Step 12. Handle short text, where most methods degrade.** Items of a few words carry too little co-occurrence information for topic models and too little context for classifiers to disambiguate. Set a minimum length below which items are routed out of the main pipeline to exact-match rules or human review, and report the share of corpus affected. Where short items can be aggregated (all responses to one question, all reviews of one product), aggregation before modelling restores signal at the cost of losing the item as the unit. And recognise that short items are frequently uninformative rather than merely brief: "good", "n/a", "none" are content-free and belong in the no-analysable-content category rather than being classified. A correct result is a stated short-text rule with the affected volume reported.

**Step 13. Produce estimates with their accuracy carried into the number.** A machine-derived prevalence is not the true prevalence; it is the true prevalence distorted by the classifier's precision and recall. Where recall is 0.72 and precision 0.85 for a class, the machine's count understates by missing 28% of true cases and overstates by including false positives, and the net direction depends on both. State the raw machine count, the measured precision and recall, and the direction and rough magnitude of the resulting bias. Where the gold set is large enough, a corrected estimate can be computed and reported with an interval that includes the uncertainty from the gold set itself; where it is not, report the raw figure with the accuracy figures beside it and the direction of error named. **Every number derived from classification carries its classification accuracy in the same line.** A correct result is a prevalence table where no cell appears without its per-class precision and recall.

**Step 14. Make the required disclosure.** Per **K4 §7**, where an AI or automated process performed a coding, classification or extraction step, this is stated in the output, along with whether a human verified it. For this skill the disclosure is specific: what was machine-classified, against what frame, validated how, on a gold set of what size, at what measured per-class accuracy, with what human verification, and what was not validated. This is not a footnote and not an appendix item. It sits with the results, and it is what allows another researcher to judge them. A correct result is a disclosure block that a reader could use to decide how much weight to place on every figure in the output.

## 8. Analytical framework

    Corpus profile → Who wrote → Frame → Gold set → Per-class accuracy
        → Reportable classes → Rare-content search → Accuracy-carrying estimates
            → Disclosure

The framework has two independent tracks that must both run. The main track, from frame to estimates, measures what is common and is governed entirely by the gold set: nothing on this track is reportable at an accuracy better than the gold set can establish, and classes the gold set cannot cover are not on the track at all.

The second track is the rare-content search, and it exists because the main track is structurally blind to exactly the content with the highest consequence. It is not a refinement of the classification; it is a different activity with a different objective, recall rather than precision, and it must be run separately or it will not be run. The framework's final term, disclosure, is what makes both tracks auditable. An output from this skill without it is a set of numbers whose provenance a reader cannot assess, which under **K2** is not evidence.

## 9. Output format

**The corpus profile.** Items, unique authors, items per author distribution, date and source distribution, language distribution, length distribution, and the deduplication log with counts removed at each pass.

**The selection statement.** One sentence naming who wrote and what selected them, carried onto every prevalence figure.

**The frame.** Category, definition, inclusion rule, exclusion rule, boundary examples, plus "other" and "no analysable content".

**The gold set record.** Sampling rule, size overall and per class, number double-coded, inter-coder agreement, adjudication process, held-out status.

**The accuracy table.** The core of the output.

| Class | Gold set support | Precision | Recall | Confusion partner | Reportable? | Reason if not |
|---|---|---|---|---|---|---|

**The prevalence table.** No cell without its accuracy.

| Class | Machine count | Machine % of corpus | Precision | Recall | Direction of bias | Corrected estimate (interval) |
|---|---|---|---|---|---|---|

**The topic table**, where discovery was used. Machine top terms, human name (flagged as interpretation), representative item identifiers, stability across runs, retained or discarded with reason.

**The rare-content search record.** Term list used, volume reviewed, what was found, known-case recall test result, and the explicit statement that absence in the classification output is not evidence of absence.

**The disclosure block.** Per **K4 §7**: what was machine-classified, the frame, the validation method, gold set size, per-class accuracy, human verification performed, and what was not validated.

**Where the evidence is thin**, use these forms:
- `[class not validated: fewer than N gold examples, no accuracy measurable]`
- `[not reportable: recall below threshold, prevalence would be a substantial undercount]`
- `[topic discarded: unstable across initialisations]`
- `[topic uninterpretable: no coherent reading from representative items]`
- `[language accuracy not established: validated on <language> only]`
- `[short text routed out of pipeline: N items below minimum length]`
- `[absence not evidence: this class was not detected and the search was not exhaustive]`
- `[corpus provenance unknown: collection rule not supplied]`

## 10. Quality checks

**K4 §8** runs anyway. These are specific to text at scale.

1. Is the corpus profile reported, including items per author and the top-author concentration?
2. Is the deduplication log present, with counts before and after each pass?
3. Are template, automated and coordinated items identified and their handling decided and recorded?
4. Does the selection statement about who wrote appear with every prevalence figure?
5. Was the gold set sampled randomly, with rare classes over-sampled, and held out from classifier tuning?
6. Is inter-coder agreement reported, and were disagreements adjudicated?
7. Is accuracy reported per class rather than overall, with precision and recall separately?
8. Is the confusion matrix inspected, and are confused class pairs treated as a frame problem rather than a classifier problem?
9. Was the reportability threshold set before results were seen, and is every withheld class recorded?
10. Was a rare-content search run, with its term list and reviewed volume documented?
11. Was a known-case recall test performed where known cases exist?
12. Does the output state that absence of a class in the classification is not evidence of absence in the corpus?
13. For topic models: is every topic name flagged as human interpretation, with terms and representative items shown?
14. Was topic stability checked across initialisations and topic counts, with unstable topics discarded?
15. Is per-language volume reported, and is any accuracy figure quoted only for the language it was measured on?
16. Does every derived number carry its per-class precision and recall in the same line?
17. Is the **K4 §7** disclosure block present with the results, not in an appendix?

## 11. Common failure modes

| Failure | How to recognise it | How to prevent it |
|---|---|---|
| **Overall accuracy quoted** | "The classifier is 87% accurate" with no per-class breakdown | Per-class precision and recall mandatory; overall figure not reported alone |
| **Rare-important content missed** | A clean topic report from a corpus known to contain serious incidents | Step 9 run as a separate track with recall as the objective |
| **Absence read as evidence** | "No safety concerns emerged in the analysis" | The absence statement is a required element of the output |
| **Volume read as importance** | A priority list matching the order of mention counts | Selection statement on every figure; author-level view alongside item-level |
| **Duplicates inflating everything** | A theme spiking on a date, driven by identical text | Three-pass deduplication before analysis; coordinated items separated |
| **Topic names as findings** | Topic labels used as themes with no representative items shown | Names flagged as interpretation; terms and example items always displayed |
| **Unstable topics reported** | A topic that appeared in one run and is now in the report | Multiple initialisations and topic counts; only persistent topics retained |
| **Gold set used for both tuning and evaluation** | Suspiciously high accuracy figures | Held-out set, separate from anything used to build or tune |
| **Gold set too small for rare classes** | A precision figure for a class with four gold examples | Stratified over-sampling of rare classes; unvalidated classes declared |
| **Frame forced onto a different corpus** | A high "other" rate, or categories that never fire | Fit check with residual rate before applying an imported frame |
| **Translation error read as content** | Sentiment or theme patterns that differ oddly by language | Per-language validation; translated text validated against original labels |
| **Short items classified confidently** | Two-word items assigned to substantive categories | Minimum length rule with affected volume reported |
| **AI: naming a topic plausibly and moving on** | A coherent topic name that the representative items do not support | Representative items with identifiers required beside every name |
| **AI: reporting a prevalence without its accuracy** | A clean percentage from machine coding with no precision or recall | Accuracy travels in the same line as the number (**K2 §7**) |
| **AI: fabricating a representative quote** | An example that captures a class perfectly and matches no item identifier | Every quoted item resolves to an identifier in the corpus (**K4 §2.3**) |
| **Disclosure omitted for readability** | A polished report with no mention that classification was machine-performed | **K4 §7** disclosure sits with the results as a required element |

## 12. AI guardrails

Universal prohibitions are inherited from **K4**. **K4 §7** (disclosure that AI performed a classification step, and whether a human verified it) is the governing requirement for this skill and is binding in full. The following are specific.

1. **Never report a proportion derived from machine classification without the per-class precision and recall in the same line.** A machine count without its accuracy is presented as a measurement and is not one.
2. **Never report overall accuracy as though it described performance on the classes being reported.**
3. **Never state or imply that a theme is absent from a corpus** on the basis of classification output. The correct statement names what was searched and how, and says absence was not established.
4. **Never present a topic model output as a set of themes.** Topics are statistical groupings requiring human naming, and the name is an interpretation that must be marked and evidenced with representative items.
5. **Never quote an example item that does not resolve to a real item identifier in the corpus**, and never compose a representative example. This is fabrication under **K4 §2.3**.
6. **Never evaluate a classifier on data used to build or tune it**, and never report an accuracy figure without saying what it was measured on.
7. **Never apply an accuracy figure measured in one language, one source or one time period to another** without revalidating.
8. **Never analyse a corpus without deduplication**, and never report a proportion of a corpus whose duplicate structure has not been resolved.
9. **Never describe a text corpus as representing a population.** It represents people who wrote, and the selection statement is a required element.
10. **Never route short or content-free items through a classifier and report the resulting labels** as though they carried meaning.
11. **Never withhold the classes that were not reportable.** The list of classes the method could not measure is part of the result, per **K4 §4.1**.
12. **Never let the human labelling step be performed entirely by the same AI system that will do the classification.** A gold set produced by the model being evaluated measures self-consistency, not accuracy, and where AI assistance is used in labelling, human adjudication of a sample is required and the arrangement is disclosed (**K5 §6**).

## 13. Best-practice principles

1. **The gold set is the analysis.** Everything reportable is bounded by what human labels established. Time spent building a well-sampled, double-coded, adjudicated gold set buys more than any modelling improvement, and a project that skips it has produced numbers, not evidence.
2. **Optimise for the class that matters, not for the average.** Aggregate performance metrics push a pipeline toward the large classes. Decide which classes carry consequence and tune for recall on those, accepting the false positives, because a human can discard a false positive and cannot recover a missed case.
3. **Silence is the expensive failure.** A percentage that is five points wrong causes a mild misjudgement. A serious issue that appeared forty times in sixty thousand items and was never surfaced causes real harm and is invisible in every quality metric the pipeline reports.
4. **Read some of it anyway.** A few hundred items read by a person, chosen randomly, is the cheapest and most reliable check on whether the machine's categories correspond to what the corpus contains. It routinely finds something the frame has no room for.
5. **Volume tells you about the writers.** Before any ranking by mention count, ask who is selected into the corpus and what motivated them to write. Review corpora are bimodal by construction; complaint corpora over-represent persistence; consultation responses over-represent organisation.
6. **Confusion between two classes is a frame problem.** When the confusion matrix shows two categories bleeding into each other, the definitions overlap. Fix the frame; do not tune the classifier around a boundary humans cannot draw either.
7. **A frame that was not built from reading is a guess.** Categories invented from expectation produce high "other" rates and miss the things people actually say. The interpretive work belongs to the qualitative skills and cannot be substituted.
8. **Precision governs examples; recall governs counts.** When quoting instances of a class, precision is what matters. When reporting how much of it there is, recall is. These are different requirements and a pipeline good at one can be poor at the other.
9. **Trend claims require an unchanged pipeline.** A classifier retrained, a frame amended, a source added or a platform's API changed all break comparability, and a movement in a tracked theme is frequently a change in the method rather than in the world.
10. **Report what you could not measure.** The list of classes below threshold, the languages unvalidated and the share of corpus routed out as short text is a required part of the output and is often the most informative part.
11. **Author-level analysis before item-level conclusions.** Where a small number of authors generate a large share of items, item-level proportions describe those authors. Report both views and let the divergence be visible.
12. **The disclosure is not a compliance chore.** A reader who knows the classification was machine-performed at 0.68 recall on the class in question can use the number correctly. One who does not will treat it as a count. The disclosure is what makes the work usable.

## 14. Worked example

**INPUT**

A fictional national medicines regulator, the Halloway Therapeutics Agency, holds 84,000 free-text reports submitted through a public reporting portal over eighteen months. The agency wants to know what the reports are about, how the pattern has changed, and, specifically, whether anything concerning a particular device category is present. No frame exists. Prior work read roughly 300 reports.

**PROCESS**

*Step 1, profile.* 84,012 items, 71,300 unique submitter identifiers, top 1% of submitters accounting for 9% of items. Length median 61 words, with a spike of 4,100 items under five words. Two languages, 78% and 21%, with 1% mixed or unidentified. Date distribution shows a fourfold spike across three weeks in month eleven.

*Step 2, deduplication.* 1,904 exact duplicates removed. Near-duplicate detection finds a cluster of 2,830 items with near-identical wording, all within the month-eleven spike, all submitted through the same referral path. These are not deleted: they are separated and analysed as a coordinated submission, because their existence is itself a finding. Corpus for prevalence analysis: 79,278.

*Step 3, who wrote.* The selection statement: these are people who found and used a public portal, self-selected, disproportionately those with a serious or unresolved concern, and for three weeks disproportionately those reached by one referral campaign. Attached to every subsequent figure.

*Steps 4 and 5, method and frame.* Discovery first, since no frame exists. Topic modelling at several topic counts, followed by human reading of 40 items per candidate topic. Of 25 candidate topics at the selected setting, 6 are discarded: three are unstable across initialisations, two are boilerplate artefacts (portal instructions pasted into the field), and one is uninterpretable from its items. The remaining 19 inform a frame, which is built by human reading and handed back from **07.02**, at 14 categories plus "other" and "no analysable content".

*Step 6, gold set.* 1,200 items sampled randomly, plus stratified over-sampling of four rare categories to 60 gold examples each. 300 double-coded; agreement is high on eleven categories and poor on two that concern degree of harm, where coders disagree on the boundary. Those two are merged, following the confusion evidence, and the frame is revised to 13 categories.

*Step 7, accuracy.* Per class, precision ranges 0.71 to 0.94 and recall 0.58 to 0.91. The largest category performs well. Two rare categories have recall of 0.58 and 0.61, meaning roughly four in ten true instances are missed.

*Step 8, thresholds.* Set before results at recall 0.70 for reporting a prevalence and precision 0.80 for quoting examples. Two categories fall below on recall and are reported with the undercount stated rather than withheld, because the agency needs the direction; one falls below on precision for examples and its example set is human-verified before use.

*Step 9, the rare-content search, and the judgement call.* The device category question is a search problem, not a prevalence problem. An over-inclusive term list is built with agency clinicians, including brand terms, generic terms, misspellings, colloquial names and descriptions of the mechanism, returning 2,140 items at an expected precision well under 20%. All 2,140 are read by humans. This surfaces 31 items describing the device concern, of which the classifier had assigned only 12 to any relevant category. *The judgement call:* the classifier's output alone would have reported the concern at roughly a third of its true frequency, and a report based on it would have understated a safety signal. Resolved by reporting the search result rather than the classification for this question, documenting both numbers, and stating explicitly that the classification pipeline is not a safety-detection method.

**OUTPUT**

> The 84,012 submitted reports reduce to 79,278 after removing 1,904 exact duplicates and separating a coordinated cluster of 2,830 near-identical submissions received through a single referral path across three weeks in month eleven. That cluster is analysed separately and accounts for the volume spike in that period; it should not be read as a change in underlying reporting.

> These reports were written by people who found and chose to use a public portal. They are not a sample of patients, users or the public, and the proportions below describe the corpus, not a population.

> Classification was machine-performed against a 13-category frame built from human reading, validated on a held-out gold set of 1,440 items with 300 double-coded (agreement adjudicated). Per-class precision ranges 0.71 to 0.94 and recall 0.58 to 0.91. Every prevalence figure below carries its class accuracy. Two categories fall below the reporting threshold on recall (0.58 and 0.61) and their figures are undercounts of unknown but substantial size.

> **On the device category question, the classification output should not be used.** A targeted search using an over-inclusive clinical term list returned 2,140 items, all read by humans, and identified 31 reports describing the concern. The classifier had assigned 12 of these to any relevant category. The classification pipeline detects roughly a third of the relevant reports and is not a safety-detection method. The search was not exhaustive, and 31 is a floor, not a count.

> **What could not be measured.** Accuracy in the second language was validated on 180 gold items and is lower than in the primary language (recall 0.51 to 0.79); figures for that language segment carry a larger undercount. 4,100 items under five words were routed out of the pipeline as content-free and are excluded from all denominators. The two categories concerning degree of harm were merged after human coders failed to agree on the boundary, so harm severity is not distinguished in this analysis.

**Researcher decision required.** The two categories reported below the recall threshold give the agency a directional signal at the cost of a figure that will be quoted as a count. Whether to publish them with the undercount stated, or withhold them, is a judgement about how the numbers will be used downstream (**K5 §2.1** and **§2.4**).

## 15. Advanced usage

**Human and machine in a loop.** The strongest configuration is not full automation but a pipeline where the machine handles the large, well-defined classes and routes low-confidence items, unmatched items and rare-class candidates to human review. Measure and report the share routed to humans, because it is a direct measure of how much of the corpus the automation genuinely handled.

**Active learning for rare classes.** Where a rare class matters, iteratively label the items the model is most uncertain about rather than a random sample, which builds usable training and gold data for that class far faster. It biases the gold set, so a separate random held-out set is still required for the accuracy figures.

**Corpus comparison rather than corpus description.** Comparing two corpora (before and after, market against market, complainers against praisers) on relative term or class frequency is frequently more informative and more robust than describing either alone, because the selection effects partly cancel where the collection rule is unchanged.

**Linking text to structured outcomes.** Where items carry identifiers that join to behaviour (a ticket that resolves, a customer who churns, a report that becomes an investigation), the class labels can be related to outcomes, which converts a description of what people wrote into evidence about what predicts something. Hand the estimation to **05.06** and keep the classification uncertainty visible, because measurement error in the class label attenuates any relationship estimated from it.

**Multilingual pipelines done properly.** Rather than translating everything into one language and classifying once, validate per language with per-language gold sets and report per-language accuracy. Where volume in a language is too small to validate, say so and exclude it from derived numbers rather than absorbing it silently.

**Auditing someone else's automated coding.** A frequent and valuable use. Request the frame, the gold set, the sampling rule, the per-class accuracy and the held-out status. Where any of these is unavailable, the coding is unvalidated regardless of how the output looks, and the correct finding is that the accuracy of the reported numbers is unknown. Build an independent gold set on a sample and measure it directly, which is usually cheap relative to the decision at stake.

## 16. Skill chain

**Recommended previous skills**
- **07.02 Open-Ended Response Coding.** Owns code frame construction and the coding of survey text below the volume threshold, and hands the frame here for application at scale. The boundary runs both ways: below roughly 5,000 items the work belongs there, above it here, and the frame still comes from there.
- **07.01 Thematic Analysis.** Owns interpretive theme development at any volume, and hands over the frame and the interpretive reading that a classifier cannot produce. Where the question is what a theme means rather than how much of it there is, the work returns there.
- **04.02 Data Cleaning.** Hands over the assembled corpus with item identifiers and metadata intact.
- **04.04 Data Transformation and Dataset Preparation.** Hands over the structure that makes author-level and metadata-stratified analysis possible.

**Recommended next skills**
- **08.01 Finding to Insight Development.** Takes validated prevalence and the rare-content findings and asks what they mean, which is where a classified proportion becomes an argument.
- **06.05 Customer Experience and Journey Analysis.** Takes stage-classified text volumes as evidence within a journey structure, with the accuracy figures attached.
- **05.05 Trend and Tracker Analysis.** Takes tracked theme prevalence, and enforces the pipeline-comparability discipline that any text trend claim requires.

**Runs well alongside**
- **07.04 Quote and Evidence Extraction**, which supplies the verification discipline for any item quoted from the corpus.
- **13.03 AI Output Verification**, which audits text analytics output for missing accuracy figures, unevidenced topic names and absence claims.
- **13.06 AI Research Governance**, which owns the organisational form of the **K4 §7** disclosure this skill produces.

---
A Yazi Supplied Skill and resource.