Writing

Your AI visibility score is counting the wrong thing

An AI answer can mention a brand or cite it, and the two are won in different places. What we learned measuring both across four model families, why three of them could never cite anyone, and the word-matching bug that inflated every score.

We built a scanner that asks AI models the questions a buyer would ask, then reads the answers for a brand. It reported one number, a visibility score, and the number was wrong in three separate ways. Three of the four models it asked could not cite a source at all. The mention counter counted an ordinary English word as the brand. And the single score pooled two things that are won in different places, so it pointed at no work anyone could do.

The fixes are not exotic. They are mostly about being honest about what a probe can see.

~2,000
Probe runs per model family
one run is one buyer question answered by one model
~23,000
Cited sources, the one grounded probe
the only probe with web search attached
0
Cited sources, the other three
no search tool, so nothing to cite
5 of 41
Matches that were the English word
all five behind 'get' or 'is'

Two slots, won in different places

An AI answer has two places a brand can appear, and they behave differently.

CitationMention
The brand isthe evidence for a claimthe thing to use, or a name in a list
Answer shapeprose with a sources raila shortlist or table of named vendors
You earn it bybeing the origin of a factbeing in the pages the model reads
The work happens onyour own pagesother people's pages

A citation is won on your own site: original data, a dated statistic, a page a crawler can actually read. A mention is won elsewhere: the review sites, listicles and articles the model pulls from when it builds a shortlist. Average the two into one percentage and the number moves for reasons nobody can act on. So the rule for the rebuild is two scores, never one, kept separate all the way down to the stored report.

Three of four probes could never cite

The scanner named four engines in its interface. Underneath, each was a plain text-generation call through a model gateway, with no web search tool attached. Only Perplexity's Sonar has retrieval built in, so it was the only probe that ever returned a source.

Cited sources captured, about 2,000 runs each
  1. Perplexity Sonar22,881
  2. OpenAI model0
  3. Anthropic model0
  4. Google model0

That is not a parsing bug. The three ungrounded models answered from training memory, which is a snapshot of an older web, not AI search. One of them gave three mutually exclusive answers to what a company used to be called, which is what recall without retrieval looks like. Any claim of the form "this share of the sources ChatGPT cited" was unsupportable, internally or in a press release, and we stopped making it.

Mentions were a different story. A mention is a match in the answer text, and every model produces answer text, so mention rate is comparable across all four.

Mention rate across the same runs, percent
  1. Perplexity Sonar5.7%
  2. OpenAI model4.5%
  3. Anthropic model4.4%
  4. Google model2.8%

The honest framing until retrieval is attached everywhere: mention rate across model families, citations from the grounded probe only. An engine that cannot cite is excluded from the citation metric and named as excluded. It is never reported as a zero, because a zero says "you were not cited", and the truth is "we did not look".

Even grounded, the API is not the product

Attaching a search tool narrows the gap. It does not close it. A consumer chat product differs from its own API in ways no caller can reach: its system prompt, its decision about whether and how deeply to search, its routing between models under load, its personalisation, and its own citation extraction for the sources rail.

TierWhat you callWhat it tells you
1the consumer product itselfwhat a buyer sees; no API exists, and automating the browser breaks the terms
2the provider's API with its own native searchclose; native retrieval and native citations
3the provider's API with a third-party search toolmisleading; the model cites your search provider's results as its own
4the provider's API with no searchwhat the model remembers, not what it would find

We were on tier 4 for three probes. Tier 3 is the trap: it is the cheapest way to make the citation column non-zero, and the citations it produces are a worse overclaim than none, because they come from a retriever you chose. So the defensible claim is "what AI models say about you", never "what a named chat product says about you", and engine labels name model families rather than products.

The brand was an ordinary word

The mention counter was a word-boundary regular expression on the brand name. For a brand whose name is also a common word, every ordinary use scored. Picture a brand called Pitch, and an answer that says "when you pitch a journalist, keep it short". That counted. The bug fails silently: the score just reads higher.

The fix had to be grammatical, not about capital letters, because not every brand is capitalised. Across the captured answers there were 41 matches: 36 were the brand and 5 were the word, and all five sat behind a linking verb or "get". So the counter now does three things:

Casing is judged against the brand's own form. A brand that is genuinely lowercase accepts any casing; a capitalised one treats a lowercase hit mid-sentence as suspect. The test for that was the one that caught a lowercase brand being sent to the judge for no reason.

Asking a model a yes-or-no question about one sentence is exactly the shape a typed decision model is cheap at, and it only runs on the cases the rule cannot settle.

The questions have to be the buyer's

The second silent failure was upstream of every probe. When the scanner could not work out what a brand did, it fell back to generic questions written for software: "what are the best tools in this category?" A candle brand was then scanned with software questions, was never mentioned, and scored close to zero. Every brand outside software did.

The fix was to resolve the brand properly before asking anything: read the site (with an archive fallback when the live page will not load), extract a name, an industry and a one-line description, and confirm them before the scan runs. A wrong company is undetectable downstream, so the confirmation is a gate, not a courtesy. The questions themselves should come from the buyer's words, from sales calls, support tickets, search queries and forum titles, and be typed by what a good answer contains: a cited fact, a shortlist, or steps.

One reading is noise

Each question was asked once per model. A model's answer varies run to run, so a single reading cannot tell a real change from sampling noise, and a monthly report built on single readings reports noise with confidence. Three repeats is the floor, stored per repeat so the variance can be recovered later. That multiplies the cost, which is the point: measure what a real run costs before you price it, because 20 questions across four grounded models at three repeats is 240 generations, a different order of magnitude from a quick scan.

What to do with your own numbers

If you track this for your own brand, I would like to see your split between mentions and citations, and which probes you trust for each.