Ask AI the same question four times, get four different shortlists
We asked an AI assistant the same 39 buyer questions four times each. Only about 60% of the businesses in a typical answer showed up every time, and the first name changed in more than half the questions.

A business owner asks ChatGPT "Who's the best HVAC company in Phoenix?", sees their company in the answer, and decides their AI visibility is fine. A competitor asks the same question an hour later and sees a different list.
Both of them are right about what they saw. Neither of them learned much.
AI assistants don't look answers up in a fixed table. They write each answer fresh, often after a fresh web search, and small differences in what they find and how they phrase things can change which businesses make the list. That has big consequences for anyone trying to understand, or improve, how often AI recommends them.
We wanted to measure how much answers move. So we asked an AI assistant the same 39 buyer questions four times each, on the same day, and compared the businesses it named each time.
The short version
- The same question produced a noticeably different shortlist each time. A typical answer named about 13 businesses. Across four runs of the same question, about 20 different businesses appeared.
- About 60% of the names in a typical answer showed up every time. The other 40% came and went between runs.
- Almost a third of all businesses named appeared in only one of the four runs. If you'd checked once and seen one of them, you'd have had no way to know it was a one-off.
- The first name changed in more than half the questions. The same business was named first in all four runs for 18 of the 39 questions (46%).
- Software was the most stable industry. Local healthcare and wellness were the least. For a software question, 81% of a typical answer's brands appeared every time. For a local healthcare or wellness question, 37% did.
Same question, four runs
Here's how the shortlists moved, averaged across all 39 questions.

Put another way: if you checked once and saw your business in the answer, there's a real chance you were one of the names that comes and goes. And if you checked once and didn't see yourself, you may simply have caught a run where you weren't included.
What it looks like in practice
Two examples show the range.
A local question: a dermatologist in Vancouver
When we asked "Who's the best dermatologist in Vancouver for acne?", two dermatologists appeared in every run. Two more appeared in three of the four. In the third run, the assistant swapped in three businesses that never appeared in any other run.

A dermatology practice checking its AI visibility on the day of run 3 would have seen a very different competitive picture from one checking on any other run.
A software question: help desk software
When we asked "Which help desk software do growing SaaS companies use?", the answers were far steadier. Nine brands appeared in all four runs, and the differences came down to a few smaller names at the end of the list.

Even here, though, the first name wasn't fixed. Two different brands took the top spot across the four runs.
Stability by industry
Some industries were much steadier than others. This chart shows the share of a typical answer's businesses that appeared in all four runs.

The pattern makes sense once you think about how each market looks to an AI assistant.
In software and online retail, a handful of well-known brands dominate. In most software categories, the same few names appear in nearly every comparison article and buying guide. The assistant finds them every time it searches, so they show up every time it answers.
In local, trust-heavy categories, there are hundreds of comparable providers. Every city has dozens of dermatologists, yoga studios, real estate agents and contractors, and few of them are well known beyond their neighborhood. Which ones make the list depends heavily on which pages the assistant's search turns up on a given run.
Industrial was a special case. Industrial questions produced the longest answers in the study, about 22 names on average, often lists of distributors or manufacturers. The biggest suppliers appeared every time, but the long tail of smaller names changed a lot.
Local services were steadier than you might expect. For questions about salons, auto repair and dog groomers, about 70% of names appeared every time. In our study of where AI gets its answers, these questions leaned almost entirely on Yelp. A single dominant source seems to produce steadier answers than a mix of many.
That last point has a practical upside. In the least stable categories, no one has locked in the answer yet. A local clinic, wellness studio or real estate agent that becomes a consistent presence in the sources AI reads has a real chance to become one of the names that shows up every time.
Who stays on the list
We can't see inside the assistant, so we can't say exactly why some names stick and others don't. But the businesses that appeared in all four runs had one thing in common: they showed up in more than one of the sources the assistant read. A plumber on Yelp and on two local "best of" lists has three chances to be found on any given search. A plumber on one list has one.
That's consistent with how these assistants work. Each answer is assembled from whatever pages the search returns. The more of those pages mention you, and the more consistently they describe you, the more likely you are to make the list on any particular run.
The first name matters most, and it moves
Being named is good. Being named first is better. Many buyers start with the first name that sounds right, especially when they need something quickly.
In our study, the first name stayed the same across all four runs for 18 of the 39 questions. For the other 21, the top spot changed hands at least once. Even in steady categories like software, the top spot sometimes swapped between two or three leading brands.
For a business tracking its AI visibility, that means the first-mention rate, how often you're named first across many questions and runs, is a more honest measure than a single "are we first?" check.
What this means for checking your own visibility
One answer is not a measurement. Seeing yourself in a single AI answer, or not seeing yourself, tells you very little on its own. Ask your most important questions several times before drawing a conclusion.
Measure rates, not results. The useful question isn't "Did AI mention us?" It's "Out of all the times buyers ask, how often does AI mention us, and how often first?" That takes many questions, asked more than once.
Compare like with like. If you ask a question once before a change and once after, the difference may just be normal variation. Compare rates across many questions and runs instead.
Aim to be one of the names that always appears. The businesses named in every run tend to be the ones with a strong, consistent presence across the sources AI reads: lists, review sites, directories and their own websites. See where AI gets its answers for which sources matter in each industry, and how to get onto the sources AI cites.
Track it over time. Because answers vary from run to run, repeat the same questions on a regular schedule and look at the trend. Our guide on how to check your AI visibility yourself explains how to set that up.
How we did this
We took 39 buyer questions from our study of where AI gets its answers, three from each of 13 industries. Questions about local businesses named one of 10 cities in the US and Canada. We asked each question four separate times, each in a fresh conversation, with a leading AI assistant with live web search turned on and its location set to the question's city. All answers were collected on November 2, 2026.
For each answer, we recorded every business named as an option, in order. Because the same business can be written different ways (a lawyer's name in one run, their firm's name in the next), we then had an AI model merge name variants for each question into a single list of businesses, and recorded which runs each one appeared in. We counted how many businesses appeared in all four runs, how many appeared in at least one, and whether the first-named business stayed the same.
Answers were collected within the same day, so this study measures variation from run to run, not change over weeks or months. Over longer periods, we'd expect answers to vary at least as much.
An earlier version of this article matched business names by spelling only, which counted some name variants as different businesses. The figures above use the improved matching and replace the earlier ones.
See how often AI recommends you, not just once
Enter your website and get a free AI visibility report. It takes a few minutes, and you'll see:
- Your AI visibility score, built from many questions, not one
- How often AI names you first, and who it names instead
- Which topics you win and lose, and how often each one is searched
- What AI gets wrong about you, and what to fix first
- A downloadable PDF you can share with your team