# Data pack: Can AI write original content? We tested it Publisher: Elementera (https://www.elementera.com) Report: https://www.elementera.com/resources/can-ai-write-original-content Findings recorded: 2026-09-23 to 2026-09-26. Each finding is given in the exact wording we approved for publication, with the sample it counts, the date it was recorded in our research register, the AI models involved and what it does not show. ## Starter prompt Copy it into your AI assistant, fill in the two lines, then paste this whole file where it says so. ``` The text below is a data pack from Elementera's study "Can AI write original content? We tested it" (https://www.elementera.com/resources/can-ai-write-original-content). It holds the study's findings, recorded in September 2026. Each finding gives the result in the wording approved for publication, the sample it counts, the date it was recorded, the AI models involved and what the finding does not show. About me: [your role and the kind of company you work for] My question: [what you want to know] [paste the data pack here] ``` ## Findings ### 1. AI writer models on facts they could not know - Finding: "We asked two OpenAI writer models to write short, publishable expert copy answering 38 questions whose answers exist only in our own unpublished measurements. Across 71 answers, none got the fact right (0 of 71; 95% interval 0% to 5%)." - Finding: "On ordinary facts the same models do know, the same checker scored 140 of 140 correct, so the zero is not a broken ruler." - Sample: 71 passages: 38 questions, each given to the same two models; 5 of the 76 passages could not be read by the checker. The check on the checker: 140 passages on facts the models know, such as chemical symbols. - Recorded: 2026-09-23 - Models: gpt-5.4 and gpt-5.5 (OpenAI), requested by name through OpenAI's API, no web search - Does not show: It is about facts a model cannot know, not about AI writing in general. Both models come from one vendor. The models wrote without searching the web. ### 2. Stating a value anyway, and how often without a hedge - Finding: "In 52 of the 71 answers the model stated a specific value anyway (73%; 95% interval 62% to 82%); 44 of the 71 passages carried no hedge at all (62%; 50% to 72%); and in 31 of the 71 it stated a wrong value with no hedge (44%; 33% to 55%)." - Finding: "In 52 of the 71 answers (73%; 95% interval 62% to 82%) the model committed to a specific value, and every one of those 52 values was wrong." - Sample: The same 71 passages. - Recorded: 2026-09-23 - Models: gpt-5.4 and gpt-5.5 (OpenAI) - Does not show: "No hedge" means the passage contained none of a fixed list of hedging words; it measures how the passage reads, not what the model believed. ### 3. Asked directly instead of told to write - Finding: "Asked the same 38 questions directly, the same models were also never right (0 of 76), but they said they were unsure: the median confidence they reported was about 15%. Told to write, they stopped saying so." - Finding: "Only 1 of those 76 direct answers was both wrong and given with 70% confidence or more (95% interval 0.2% to 7%)." - Sample: 76 direct answers: the same 38 questions put to the same two models as plain questions. - Recorded: 2026-09-23 - Models: gpt-5.4 and gpt-5.5 (OpenAI) - Does not show: Confidence here is the score the model reported for itself, a different measure from the hedge-word count, so the two arms are compared in words, not as one statistic. ### 4. Hedging words do not reveal the invented facts - Finding: "You cannot find the invented facts by reading for hedging words. The passages on questions the models could not know were no easier to pick out by their hedging than the passages on facts they did know: most of both had no hedge at all (44 of 71 and 108 of 140), and a separation score where 0.5 means no separation came out at 0.55 (95% interval 0.49 to 0.62), below the 0.65 we set in advance as the least a useful warning sign would need." - Sample: 71 passages on facts the models could not know and 140 on facts they did know. - Recorded: 2026-09-23 - Models: gpt-5.4 and gpt-5.5 (OpenAI) - Does not show: It tests one fixed list of hedging words; the floor was written down before the test. ### 5. A second AI model asked whether a statement is true - Finding: "Asking a second AI model whether a statement is true does not catch the invention either. Three OpenAI models rated 38 of our true, unpublished findings and 64 invented versions of them as equally believable (on average 0.38 and 0.38 on a 0 to 1 scale), while rating well-known facts at 0.99. Their ratings separated true from invented no better than chance (0.51, where 0.5 is chance; 95% interval 0.39 to 0.63, below the 0.65 we set in advance)." - Sample: 38 true, unpublished findings and 64 invented versions of them. - Recorded: 2026-09-23 - Models: gpt-4.1, gpt-4o and gpt-5.1 (OpenAI), as raters - Does not show: Each invented version is a true finding with one value swapped, so the test is about believability, not about a model's general accuracy. ### 6. A rewrite of the top search results - Finding: "When we had an AI model write a short passage for each of 60 searches from the top five Google results, only 58% of the separate statements it made could be traced to a quoted line in those results (240 of 414; 95% interval 53% to 63%), and only 7 of the 60 passages were supported from start to finish (12%; 6% to 22%). It reassembled what the results said and joined the pieces with statements of its own." - Sample: 414 separate statements in 60 passages, one passage per search. - Recorded: 2026-09-26 - Models: gpt-4.1-mini-2025-04-14 (OpenAI), as the writer - Does not show: The model saw only each result's title and short description, and a statement we could not trace is not necessarily false. The two rates count different things and are never one number. ### 7. A simple automated novelty score can reward invention - Finding: "A simple automated novelty score can reward invention. We asked two AI models how widely known each of 143 real, unpublished findings of ours was, and each of 143 invented ones. They rated the invented findings as even less known, so that score, built on "how many people already know this?", ranked inventions as more original than real findings (0.33, where 0.5 is chance; 95% interval 0.28 to 0.38). The same score did separate our real findings from generic text (0.87; 0.86 to 0.89)." - Sample: 143 real, unpublished findings and 143 invented ones. - Recorded: 2026-09-25 - Models: gpt-4.1-2025-04-14 (OpenAI) and Llama 4 Scout (Meta), as judges - Does not show: It is about one simple kind of automated score at one setting. The two sentences travel together: the score fails on invented findings and works on generic text. ### 8. Precise-looking numbers move such a score - Finding: "Precise-looking numbers move such a score too. When we added an invented figure with two decimals and a 95% interval to 60 real ChatGPT answers, changing nothing else they said, a simple automated novelty score rated 95% of them as new, up from 60% (a rise of 35 points; 95% interval 23 to 47). In a side-by-side comparison against a generic statement, the same edit moved an AI judge by 21 points (95% interval 15 to 27)." - Sample: 60 real ChatGPT answers, each read before and after one invented figure was added. - Recorded: 2026-09-26 - Models: gpt-4.1-2025-04-14 (OpenAI) and Llama 4 Scout (Meta), as judges; the answers came from the ChatGPT web interface, and the model that served them is not known - Does not show: It does not separate the effect of the number's look from the effect of the invented value itself. ### 9. Side by side, an AI judge can still tell generic text from new information - Finding: "Compared side by side, an AI judge can still tell generic text from new information. Shown a passage from a real ChatGPT answer and one of our new findings, written to look equally precise, two AI judges (AI models, not people) picked our finding as the newer one in 93% of 60 pairs (95% interval 89% to 96%). In a larger set of 143 pairs our findings won even though they carried fewer precise-looking details than the generic text (about 5 against 14 per 100 words)." - Sample: 60 pairs, and a larger set of 143 pairs. - Recorded: 2026-09-26 - Models: gpt-4.1-2025-04-14 (OpenAI) and Llama 4 Scout (Meta), as judges - Does not show: The judges are AI models, not people. It does not tell a true claim from an invented one, and it does not show that the judge resists added decimals.