Skip to content
Glossy Data

Methodology

How we measure AI visibility

AI engines give different answers each time you ask. So we ask many questions, many times, and show how many answers sit behind every number. This page explains the method in plain English.

Honest Score v1 · updated

Illustration

The problem with asking once

Ask an AI engine “what's the best project management tool for agencies?” and you'll get a list. Ask again a minute later and you'll often get a different one. So any tool that asks once and reports “you're ranked #2” is reporting a single roll of the dice.

We built Glossy Data around that fact. Here's how.

1. We collect answers the way the public sees them

We ask each engine from the market you choose (a country, and a city when you're a local business), using its normal settings. For ChatGPT, Google AI and Gemini we collect what a logged-out member of the public sees, using specialist data providers. Perplexity, Claude, Grok and DeepSeek come from their developer APIs; search availability is explained on each engine page. We never log in to anyone's account and never get around security checks.

Every answer is labeled with where it came from: the consumer app, or the developer API. The two often disagree, so we never mix them without telling you.

2. We ask in your market and your language

AI answers differ by country and language, so a number measured from one country doesn't describe another. We measure in 49 countries. For each one we use that country's search location, its default language and, for local businesses, its own directories and review sites.

Questions are written in the market's language where we have a native question pack (English, German, French, Spanish, Portuguese, Dutch, Italian). In other languages we use English questions for now and say so on the result. The same question asked in different markets keeps the same template, so markets can be compared like for like.

Engines roll out country by country. We keep a list of where each engine is available (last reviewed 2026-10). We never spend on an engine that isn't available in a market, and where availability isn't confirmed yet, an empty answer is recorded as “not checked”, never as a 0%.

3. We ask many questions, many times, over weeks

One question asked five times gives a margin of error of about ±38 percentage points. That's too wide to be useful. So instead of hammering one question, we ask 3–5 natural wordings of each question, spread across your topic, once per scheduled run, and pool the results over a rolling four weeks.

On the Starter plan that's about 120 answers per engine a month, a range of about ±9 points. Across five engines, about ±4.

Why one answer means little

Say you're really named in half of all answers. Pick how many answers we've seen:

At n = 5, the 90% range is 13–87%, a margin of ±37 points.

Normal approximation at a 50% rate, the widest case.

4. We account for small samples

When a question has few answers, its estimate is adjusted towards the average for its topic. As more answers arrive, the question's own results have more influence. Older answers receive less weight, so recent results contribute more to the score.

For statisticians: a hierarchical beta-binomial model fitted by empirical Bayes, with 90% credible intervals.

5. The headline score uses questions that don't name you

Your AI Visibility Score is a weighted average of how often you're recommended in answers to questions that don't mention your name. Questions closer to a buying decision (“best project management tool for agencies, with pricing”) count more than early research questions. Engine weights are shown and you can change them. Branded questions are tracked separately, so direct questions about your brand do not inflate its visibility in general buying questions.

6. We check our own reading of the answers

Software identifies the brands, recommendations and citations in each answer. A sample is checked by a second model. Automated analysis can still make mistakes, so the original answer remains available for review. Independent accuracy reports are planned; none has been published yet.

7. We distinguish larger changes from normal variation

If your score moves by 3 points in a week, that's usually normal variation. We show it in grey as “within noise”. We only alert you when a change is very likely to be bigger than 10 points (you can change that threshold), and before we do, we run an extra batch of 10–20 answers to confirm it.

within noise significant significant

8. We watch the engines, too

Every day we ask each engine a fixed panel of neutral questions (10 per engine at launch, as capacity allows). Similar changes across unrelated brands can indicate an engine update. We mark these periods on charts and pause alerts while the results are less reliable.

9. We reuse matching answers

If several customers track exactly the same question, engine and market in the same week, we ask once and use that one real answer for everyone. The original answer keeps its collection date and is counted once in each customer's analysis.

10. What we'll never show you

A bare rank or percentage without the number of answers behind it. Every figure carries one of three labels: Low sample, Within noise or Significant (▲ or ▼).

11. How to audit any vendor's numbers, including ours

Every AI visibility tool, ours included, is watching answers from the outside. These are the questions we'd ask of any of them. They fit straight into an RFP or a security review, and you can check our answers on this page.

  1. How many answers sit behind each number, and is that sample size shown next to it?
  2. Is there a range (a confidence or credible interval) on every percentage, and at what level?
  3. How is a real change told apart from normal variation between runs?
  4. Are branded questions (that name you) reported separately from unbranded ones?
  5. Is each answer labeled with how it was collected: the consumer app, logged out, or a developer API?
  6. From which country, and in which language, was each question asked?
  7. When an engine isn't available in a market, is that reported as “not checked” or as 0%?
  8. How often does the software misread who an answer names, and is that error rate published?
  9. Can you see the full answer text and the sources behind every number?
  10. Can you export the raw answers and your full history?

12. Open questions we can't answer yet

Being honest about the method means saying where it stops. These are the questions we're still working on:

  • How closely logged-out answers match what signed-in, personalized users see. We measure the public baseline and say so.
  • How much of an answer's variation comes from the engine changing versus the question wording. Our engine-change panel separates part of it, not all.
  • Our extraction error rate on a completed, independently reviewed test set. No accuracy report has been published yet.
  • How AI visibility translates into visits and sales. We don't publish ROI estimates we haven't measured.

13. Accuracy reports and replication studies

We'll publish the results of our extraction checks, and replication studies of well-known claims about AI search, on the reports page, each with its sample and data, whichever way the result comes out. None has been published yet. The vendor index applies the same method to AI visibility vendors, us included.

What this means for you

Use Glossy Data to understand the evidence behind a change before deciding what to do. Sample sizes, ranges and original answers help you judge how much weight to give each finding.

Found a flaw in the method? Email the details for review: hello@glossydata.com.

See it on your own website: run a free check, or read how pricing works.

See how AI talks about your business.

Free website check · No card needed · Full AI tracking on paid plans

  • Free
  • Verify your email
  • Free limited report