Ask Twice, Get a Different Answer: How Consistent Are AI Brand Recommendations?
Run the same prompt twice and you will probably get a different list of brands. Here is what that means for how you measure AI visibility.
- In a study of nearly 3,000 runs, the odds of an AI tool returning the same list of brand recommendations twice were under 1 in 100. The same list in the same order: about 1 in 1,000.
- Lists also varied in length, from two or three options to ten or more, for the same prompt.
- Under the noise, there is a signal: the strongest brands appeared in 60% to 90% of answers for a given intent.
- That makes “how often do we appear?” a far better measure than “what rank are we?” or a single screenshot.
The experiment: 600 people, the same 12 prompts
If you have ever asked ChatGPT for a recommendation, then asked again and gotten a different answer, you have seen this firsthand. In January 2026, SparkToro and Gumshoe.ai measured it at scale. They had 600 volunteers run the same 12 prompts through ChatGPT, Claude, and Google’s AI nearly 3,000 times, and compared the brand lists that came back.1
The headline finding: across tools and prompts, the odds of getting the same list twice were under 1 in 100. The odds of the same list in the same order were closer to 1 in 1,000.
The answers did not just reshuffle. Their length varied too. The same request could return two or three recommendations one time and ten or more the next.
Try it: ask the same question again and again
The simulator below recreates the effect with a fictional category and fictional brands. Each brand has a different underlying chance of being recommended, and every answer draws a list of random length in random order, just as the study observed. Press Ask again a few times, then run 100 at once and watch the tallies settle.
Two things happen as the runs pile up. First, almost every answer is unique, and exact repeats stay rare. Second, the bars on the right stop jumping around. Ridgeline appears in most answers, while brands further down the list show up only occasionally. The individual answers are noisy, but each brand’s rate of appearance is remarkably stable.
The signal under the noise
That is exactly what the study found in the real world. Even though the lists rarely matched, some brands appeared “again and again across dozens of runs, even though their position jumped around.” For a given intent, the strongest names showed up in 60% to 90% of responses, across categories as different as hospitals, agencies, and consumer brands.1
The study also found that AI tools returned similar brand sets for the same underlying intent, even when people phrased their prompts in very different ways. In headphones, for example, the same leaders surfaced most often regardless of wording.
of answers named the strongest brands for a given intent, even though the full list almost never repeated.SparkToro and Gumshoe.ai, January 2026.
Why AI answers vary
Language models generate text by sampling from probabilities, so some variation is built in by design. On top of that, engines with live search pull in different pages from run to run. As our look at citation drift shows, ChatGPT search swaps about 74% of its cited sources week to week. Different sources lead to different lists.
None of that is a flaw to wait out. It is the nature of the channel, and it changes what good measurement looks like.
How to measure AI visibility when answers keep changing
Stop tracking rank
With the same order turning up roughly once in 1,000 runs, “we are number two in ChatGPT” describes one moment, not a position. Rank in a single answer is close to meaningless.
Track appearance rate
The stable number is the share of answers that include your brand. Run each important prompt many times, and across several phrasings of the same intent, then count how often you appear. That is your visibility score, and it is the number worth moving.
Compare against competitors on the same basis
Appearance rates are most useful side by side. If a competitor appears in 80% of answers and you appear in 35%, you have a clear, measurable gap, regardless of how any single answer happens to read.
A worked example: calculating appearance rate
Here is what this looks like in practice. Say you pick 20 questions that matter to your business and write three phrasings of each, for 60 prompts. You run each prompt five times in each engine you care about, for 300 answers per engine.
If your brand shows up in 105 of the 300 answers, your appearance rate in that engine is 35%. If the category leader shows up in 240, theirs is 80%. Run the same set again next month and the change in that percentage is your trend line, a number that holds up even though no two individual answers match.
The exact counts are up to you. What matters is running enough answers that one lucky or unlucky response cannot move the result, and keeping the prompt set consistent from month to month so the trend is comparable.
- Retire the screenshot. A single answer, good or bad, is roughly a one-in-a-hundred snapshot. Do not report it as a result.
- Measure share of answers. Run priority prompts repeatedly and report the percentage of answers that include your brand.
- Group prompts by intent. Test several phrasings of the same question, since AI returns similar brand sets for the same underlying need.
- Benchmark competitors the same way. The gap in appearance rate between you and category leaders is the clearest target for your strategy.
Frequently asked questions
Why does ChatGPT give a different answer every time?
Language models sample from probabilities when generating text, and engines with live search may pull different sources each time. Together, that makes repeat answers rare.
How often do AI tools repeat the same brand recommendations?
Rarely. SparkToro and Gumshoe.ai found the odds of the same list twice were under 1 in 100, and the same list in the same order about 1 in 1,000.
Does ranking position matter in AI answers?
Not much on its own, because the order changes almost every time. How often a brand appears across many answers is a more reliable measure.
How many times should I run a prompt to measure visibility?
Enough for the percentages to settle. In our simulator the rates stabilize after dozens of runs, so treat a handful of checks as anecdotal rather than as a measurement.
Sources
- “AI recommendation lists repeat less than 1% of the time: Study,” Search Engine Land, January 28, 2026, reporting research by Rand Fishkin (SparkToro) and Patrick O’Donnell (Gumshoe.ai). searchengineland.com
Convert ad spend to mentions in AI answers.
