Research / Study 02
The Novelty Test
Asking LLMs for unique ideas? Bad idea.
We gave ChatGPT and Claude 119 ideas for research that already exists, and asked if each one was unique.
54%of the time LLMs couldn’t see if a company’s idea was original
47%of the time LLMs gave a different answer when asked again
31%of the time LLMs falsely called a company’s idea new when someone had already done it
Why we tested this
Before anyone pays for research, someone asks a chatbot: “has this been done?”
It’s the question we hear most from marketing teams: can’t I just ask ChatGPT? It’s fast and free, and the answer sounds sure.
But if the answer is wrong, you pay for research someone already published, and launch a finding that isn’t new.
So we tested it. We gave ChatGPT and Claude 119 research ideas that had already been done, asked each one “has anyone already done this?” with web search on and off, and checked every answer against the pages it pointed to.
01Right answers
On companies’ ideas, ChatGPT and Claude got it right less than half the time.
We counted an answer as right only when it said “already done” and pointed to the study that did it.
Even ChatGPT, the one most people use, got fewer than half right.
The rest of the time it said “not sure”, or that nobody had done the research. “Not sure” isn’t wrong. But it doesn’t help you decide either.
If it’s right less than half the time, you can’t trust it with your research budget.
02Ask again
Ask the same question twice, and you can get a different answer.
We asked the same ideas three times each, with exactly the same words.
Each square is one idea, asked three times
The answer changed between asksSame answer all three times
Nothing about the idea changed between asks. The answer did. On one idea, Claude said “already done”, then “nobody has done this”, then “already done” again.
And the two tools often changed their minds on different ideas, so asking a second chatbot doesn’t settle it either.
If asking again can change the answer, you can’t rely on the first one.
03Called new
ChatGPT and Claude called ideas new that someone had already done.
That’s the costly mistake.
31%Nearly 1 in 3 answers on companies’ ideas called them new when the research already existed.
Example: Claude found the study, then said nobody had done it.
- 2016
Princeton researchers measure tracking on 1 million websites, with and without privacy tools.
- We ask Claude
“How do the most-visited websites recognize returning visitors, and do browser privacy protections make a difference?”
- Claude answersNEW
It names the Princeton study as “the closest match in spirit”, then still says the idea is new.
This wasn’t a one-off. In 43% of Claude’s wrong “new” answers, it had found the study itself, then still called the idea new.
ChatGPT made this mistake less often than Claude: 23% of the time, against 38%.
Finding the study isn’t enough. Someone has to read it and decide.
04Without search
When it doesn’t search the web, it gets worse.
ChatGPT searched the web on just 34.5% of queries in February 2026, down from 46% in late 2024, according to Semrush. Longer questions were less likely to get a search, and a research idea is a long question.
The same 119 ideas, web search off
Without search, ChatGPT mostly couldn’t say either way. Claude did say, and was often wrong: it called 43% of companies’ ideas new when they had already been done.
Without search, it only knows what it learned in training. Newer research isn’t in there.
What other research says
Other studies found the same.
- <1 in 100
times, AI tools gave the same list of brand recommendations twice.
SparkToro and Gumshoe, 600 volunteers, 2,961 runs, January 2026 - 34.5%
of ChatGPT queries used web search in February 2026, down from 46% in late 2024.
Semrush, U.S. clickstream data, October 2024 to February 2026 - 76%
of coding tasks on one benchmark got no two identical answers when ChatGPT was asked several times.
Ouyang et al., 829 coding problems, ACM TOSEM
Every answer
The full breakdown.
All 119 ideas, web search on. Every idea had already been done.
| What the answer did | ChatGPT | Claude |
|---|---|---|
| Said “already done” and pointed to the study | 68% | 61% |
| Said “not sure” | 18% | 11% |
| Said “nobody has done this” | 12% | 24% |
| Said “already done” but pointed to no study of it | 2% | 3% |
| No answer | 0% | 1% |
We tested ideas from many different industries, such as cyber security, cloud infrastructure, fitness and SaaS. Every industry had wrong answers.
What this means
A chatbot gives you an answer. ClaimTheStory gives you proof.
Before you spend a research budget, you need to know what already exists, and to see it for yourself.
| Question | Asking a chatbot | ClaimTheStory |
|---|---|---|
| How it searches | One question, one answer | Searches for the idea in several ways |
| What it reads | You can’t tell | Opens the pages it finds and reads them |
| What you get | “New” or “already done” | The study, and the sentence that shows it’s been done |
| Ask again | The answer can change | The proof stays on the page |
One AI answer isn’t proof.
ClaimTheStory finds the content idea only your company can publish, and checks nobody has published it first.
Book a demoUse these numbers
Quote any figure from this study in articles, decks or posts. Please link to this page.
Cite asClaimTheStory Research, “The Novelty Test”, October 2026. claimthestory.com/research/asking-llms-for-unique-ideas/
How we did this. In October 2026 we gave ChatGPT and Claude 119 research ideas that someone had already done, and asked whether each was new, with and without web search. We checked every answer against the pages it pointed to. Our checker reads pages with AI (Claude Sonnet 5), as our product does. The headline figures are for ideas written for B2B and B2C companies; we also tested ideas written from published studies, shown in the full breakdown. We also asked a set of ideas three times each.