Why AI answers change (and how to measure honestly)

Because a language model generates text by sampling, and because what it reads before answering keeps changing. Ask "best accounting software for freelancers" twice and you may get the same list, a reordered list, or a different list. Five separate forces cause this, and only one of them has anything to do with your website.

Language models pick each next word from a probability distribution. A setting called temperature controls how adventurous that choice is. OpenAI's API documentation describes temperature as running from 0 to 2, with higher values making output more random and lower values more focused.^[1] Consumer apps like ChatGPT don't expose this setting at all.

Lowering the temperature doesn't make answers fully repeatable either. OpenAI says its seed option is best effort and that determinism is not guaranteed.^[1] In a September 2025 experiment, Thinking Machines sent the same prompt 1,000 times at temperature 0 and got 80 unique completions, which it traced to how inference servers batch requests under changing load.^[2] An academic study of settings meant to be deterministic found accuracy varying by up to 15% across runs.^[3]

For brand visibility, that means a product near the edge of a "top 5" list will appear in some runs and not others, with nothing having changed at all.

Logged-in assistants adapt to the person asking. ChatGPT's memory can use saved memories and reference past chats to personalize responses.^[5] OpenAI also says that, with memory on, ChatGPT may use relevant memories when rewriting a search query, giving the example of a vegan user in San Francisco getting vegan restaurant searches.^[4]

So when a colleague says "ChatGPT recommended us", that tells you about one account in one moment. For measurement, you want what a member of the public sees: logged out or memory off, with no custom instructions. Glossy Data collects that logged-out public view wherever it reads consumer apps, and labels answers that come from developer APIs instead.

Engines localize. OpenAI says ChatGPT estimates a general location from your IP address and may pass it to search providers, so "good restaurants near me" can become a search for top restaurants in a specific city.^[4] The same thing happens less visibly for "best bank for small businesses" asked in Toronto versus Johannesburg, or "mejor CRM" asked in Madrid versus Bogotá.

Language can change the sources too: a question asked in Japanese can draw on different pages from the same question in English. A visibility number is only meaningful with its market and language attached. Glossy Data measures in 49 countries, with native-language question packs in 7 languages (English, German, French, Spanish, Portuguese, Dutch and Italian). Other markets are asked in English and labeled that way.

When an engine searches before answering, its answer depends on what the search returns that day. New reviews, a fresh comparison article or a competitor's updated pricing page can change the result. Changes on your side take time to register: Google says recrawling can take from several days to several months.^[7]

Different surfaces from the same company also behave differently. Google says AI Overviews and AI Mode may use different models and techniques, so their responses and links vary, and that AI Overviews only appear when its systems judge them useful, so they often don't trigger.^[7] Whether an AI Overview appears at all is itself a rate you need to measure.

Engines ship new models regularly. OpenAI's release notes record that on May 5, 2026, GPT-5.5 Instant replaced GPT-5.3 Instant as ChatGPT's default model, with changes that included when ChatGPT decides to search the web.^[6] A change like that can move many brands at once, in every category.

The tell-tale sign of an engine change is that unrelated brands move together on the same day. A good tracker watches for that pattern, marks it on your charts, and holds back alerts, so you don't spend a week investigating a drop you didn't cause.

A single check answers yes or no, but your real visibility is a rate. Suppose your brand truly appears in 40% of ChatGPT answers to "best project management tool for a small agency". The numbers below are an illustration we calculated with standard binomial math, not measured data.

• One check: you are absent 60% of the time. You would conclude "invisible" more often than not.
• Five checks: there is about a 34% chance you appear 0 or 1 times, and about a 32% chance you appear 3 or more times. Two people running the same five checks can easily reach opposite conclusions.

Ranges make this visible. The table shows 90% Wilson intervals, a standard method for proportions that behaves well at small sample sizes, unlike the simple plus-or-minus formula.^[8] Each row observes the same 40% rate.

Precision improves with the square root of the sample, so halving the range takes about four times as many answers. Even a "yes" from one check only tells you, at 90% confidence, that your rate is somewhere between 27% and 100%.

Is a move from 30% to 40% real?

With 40 answers in each week, the 90% range for the change runs from about -7 to +27 points. It includes zero, so it is within noise. With 120 answers each week, the range is about 0 to +20: borderline. With 240 each week, it is about +3 to +17, a change you can act on. Same headline, three very different conclusions.

Ask many related questions, many times, under fixed conditions, and report the uncertainty along with the number. In practice:

1. Use paraphrases. Ask three to five natural wordings of each question. Real people don't all type the same words, and a result that holds across wordings is more trustworthy.
2. Spread across the topic. Twenty to forty questions per market beats a hundred runs of one prompt.
3. Fix the conditions. Same market, language and logged-out state each time, and keep consumer-app results separate from API results.
4. Pool over time. Combine weeks so each figure rests on enough answers, and weight recent weeks more.
5. Show n and a range. Every percentage should state how many answers it is based on.
6. Label small moves. If the range of a change includes zero, call it within noise and don't act on it yet.
7. Separate branded questions. Being named when someone asks about you by name is expected. Score the questions that don't name you.

This is how the Honest Score works: weekly sampling across paraphrased questions, a 90% range on every number from a hierarchical beta-binomial model, and changes inside the noise flagged as "within noise". The methodology has the full detail.

Whatever tool you use, these questions show quickly whether its numbers can carry a decision. The comparison of AI visibility tools applies them across the market.

• How many answers sit behind each percentage, and is that number shown?
• Does it show a range, and does it say when a change is within noise?
• Does it ask paraphrases, or one fixed prompt?
• Does it measure from my market and in my language, and label the ones it can't?
• Does it separate consumer-app answers from API answers, and branded from unbranded questions?
• How does it detect and annotate engine-wide changes?