Your AI visibility score is counting the wrong thing
An AI answer can mention a brand or cite it, and the two are won in different places. What we learned measuring both across four model families, why three of them could never cite anyone, and the word-matching bug that inflated every score.
We built a scanner that asks AI models the questions a buyer would ask, then reads the answers for a brand. It reported one number, a visibility score, and the number was wrong in three separate ways. Three of the four models it asked could not cite a source at all. The mention counter counted an ordinary English word as the brand. And the single score pooled two things that are won in different places, so it pointed at no work anyone could do.
The fixes are not exotic. They are mostly about being honest about what a probe can see.
Two slots, won in different places
An AI answer has two places a brand can appear, and they behave differently.
| Citation | Mention | |
|---|---|---|
| The brand is | the evidence for a claim | the thing to use, or a name in a list |
| Answer shape | prose with a sources rail | a shortlist or table of named vendors |
| You earn it by | being the origin of a fact | being in the pages the model reads |
| The work happens on | your own pages | other people's pages |
A citation is won on your own site: original data, a dated statistic, a page a crawler can actually read. A mention is won elsewhere: the review sites, listicles and articles the model pulls from when it builds a shortlist. Average the two into one percentage and the number moves for reasons nobody can act on. So the rule for the rebuild is two scores, never one, kept separate all the way down to the stored report.
Three of four probes could never cite
The scanner named four engines in its interface. Underneath, each was a plain text-generation call through a model gateway, with no web search tool attached. Only Perplexity's Sonar has retrieval built in, so it was the only probe that ever returned a source.
That is not a parsing bug. The three ungrounded models answered from training memory, which is a snapshot of an older web, not AI search. One of them gave three mutually exclusive answers to what a company used to be called, which is what recall without retrieval looks like. Any claim of the form "this share of the sources ChatGPT cited" was unsupportable, internally or in a press release, and we stopped making it.
Mentions were a different story. A mention is a match in the answer text, and every model produces answer text, so mention rate is comparable across all four.
The honest framing until retrieval is attached everywhere: mention rate across model families, citations from the grounded probe only. An engine that cannot cite is excluded from the citation metric and named as excluded. It is never reported as a zero, because a zero says "you were not cited", and the truth is "we did not look".
Even grounded, the API is not the product
Attaching a search tool narrows the gap. It does not close it. A consumer chat product differs from its own API in ways no caller can reach: its system prompt, its decision about whether and how deeply to search, its routing between models under load, its personalisation, and its own citation extraction for the sources rail.
| Tier | What you call | What it tells you |
|---|---|---|
| 1 | the consumer product itself | what a buyer sees; no API exists, and automating the browser breaks the terms |
| 2 | the provider's API with its own native search | close; native retrieval and native citations |
| 3 | the provider's API with a third-party search tool | misleading; the model cites your search provider's results as its own |
| 4 | the provider's API with no search | what the model remembers, not what it would find |
We were on tier 4 for three probes. Tier 3 is the trap: it is the cheapest way to make the citation column non-zero, and the citations it produces are a worse overclaim than none, because they come from a retriever you chose. So the defensible claim is "what AI models say about you", never "what a named chat product says about you", and engine labels name model families rather than products.
The brand was an ordinary word
The mention counter was a word-boundary regular expression on the brand name. For a brand whose name is also a common word, every ordinary use scored. Picture a brand called Pitch, and an answer that says "when you pitch a journalist, keep it short". That counted. The bug fails silently: the score just reads higher.
The fix had to be grammatical, not about capital letters, because not every brand is capitalised. Across the captured answers there were 41 matches: 36 were the brand and 5 were the word, and all five sat behind a linking verb or "get". So the counter now does three things:
- Drops a match that follows one of those leads.
- Counts a match that clearly reads as the brand.
- Asks a small judgement model about anything in between, one batched call per question, with a 0.6 probability bar to count. On the captured answers, nothing was ambiguous enough to ask.
Casing is judged against the brand's own form. A brand that is genuinely lowercase accepts any casing; a capitalised one treats a lowercase hit mid-sentence as suspect. The test for that was the one that caught a lowercase brand being sent to the judge for no reason.
Asking a model a yes-or-no question about one sentence is exactly the shape a typed decision model is cheap at, and it only runs on the cases the rule cannot settle.
The questions have to be the buyer's
The second silent failure was upstream of every probe. When the scanner could not work out what a brand did, it fell back to generic questions written for software: "what are the best tools in this category?" A candle brand was then scanned with software questions, was never mentioned, and scored close to zero. Every brand outside software did.
The fix was to resolve the brand properly before asking anything: read the site (with an archive fallback when the live page will not load), extract a name, an industry and a one-line description, and confirm them before the scan runs. A wrong company is undetectable downstream, so the confirmation is a gate, not a courtesy. The questions themselves should come from the buyer's words, from sales calls, support tickets, search queries and forum titles, and be typed by what a good answer contains: a cited fact, a shortlist, or steps.
One reading is noise
Each question was asked once per model. A model's answer varies run to run, so a single reading cannot tell a real change from sampling noise, and a monthly report built on single readings reports noise with confidence. Three repeats is the floor, stored per repeat so the variance can be recovered later. That multiplies the cost, which is the point: measure what a real run costs before you price it, because 20 questions across four grounded models at three repeats is 240 generations, a different order of magnitude from a quick scan.
What to do with your own numbers
- Split the score. Report mentions and citations separately. If a vendor gives you one blended number, ask which slot moved.
- Check what each probe can see. If a probe has no retrieval, its citation count is not a zero, it is missing. Ask which models are grounded and with whose search.
- Read your matches. Pull a sample of counted mentions and check them by eye, then keep them as a labelled set for the next change to the counter. If your brand is also a word, expect inflation.
- Check the questions. Generic fallback questions score every non-software brand near zero. The question set is the asset; keep it versioned so this month compares to last month.
- Start with crawlability. Before spending anything on models, check whether AI crawlers can reach your pages at all: robots rules, and how much body text a crawler actually gets back from a client-rendered page. It costs nothing and is often the whole answer.
If you track this for your own brand, I would like to see your split between mentions and citations, and which probes you trust for each.