Preprint · 2026–09–06

How Much Measurement Does an AI Visibility Claim Need?

Muhammad Arshad

DOI 10.5281/zenodo.22480733 · CC BY 4.0

Abstract

A commercial category now sells businesses a measurement of how often generative search engines cite them, typically from a few dozen prompts sampled once. We ask what such a measurement can actually support. Using 4,320 observations in which identical queries were issued three times to each of four generative answer surfaces, we quantify three things that this literature does not report: the rate at which repeated identical queries disagree with themselves; how the width of a confidence interval on a citation rate falls with the number of prompts sampled; and the minimum difference two samples of a given size can distinguish from noise.

Findings

1. Instability is substantial and differs sharply by platform: from 1.4% to 40.3% of repeated prompt-platform cells return non-unanimous outcomes, meaning that on one widely used assistant a repeated identical question contradicts its own earlier answer two times in five.
2. At twelve prompts — a plausible commercial configuration — the 95% interval on a platform citation rate spans 53 percentage points on the least precise platform, and the smallest difference resolvable between two samples ranges from 25 to 36 points.
3. It follows that a monitoring product reporting month-over-month movement at that sample size is, for much of its range, reporting variation below its own detection threshold.
4. We give a sample-size table practitioners can use directly, and release the harness and observations so the thresholds can be recomputed as the platforms change. We do not claim the underlying products are without value; we claim that the precision at which their numbers are presented is not supported by the sampling behind them, and that this is straightforwardly fixable by reporting intervals and replicate counts.

Data and code

The full observation set, both prompt corpora and the measurement harness are released under CC BY 4.0 with the record at 10.5281/zenodo.22480733, so every figure above can be recomputed rather than taken on trust.

← All papers