A Single AI Recommendation Proves Almost Nothing

A September 2026 preprint audited 1,536 product-recommendation responses and found that the products named often changed when the same request was repeated. It also found that the consumer interfaces for ChatGPT and Gemini showed radically different source domains from their corresponding APIs.

For businesses, the decision is no longer whether a brand appears in one impressive answer; it is whether the brand remains visible across repeated, real consumer conditions—and what to change when it does not.

The study, “If I Had to Buy Just ONE,” was published on arXiv by Lucas Marin and seven coauthors and has not been peer reviewed. Its warning is still operationally useful: a screenshot is an anecdote, not a market signal.

Define the Buying Decision Before You Test

Start with one commercial decision, not a vanity question. “Does AI know our brand?” is too vague. “When a U.S. buyer wants a quiet countertop blender under $250 for a small apartment, are we considered—and what evidence supports or excludes us?” creates a testable job.

Assemble five inputs: the buyer’s constraint, market and language, current product facts, approved proof, and the page that should substantiate the claim. Freeze those inputs for the test window. If price, availability or specifications change midway, mark the change rather than pretending every run occurred under the same conditions.

The desired output is not a rank. It is a recommendation map showing which brands appeared, why they were included, which sources were displayed, and which claims remained stable.

Run the Same Question Through Five Controlled Passes

Use a five-step sequence. First, write three natural versions of the same buying question: direct recommendation, comparison and problem-led phrasing. Second, run each version several times in the consumer-facing interface under comparable conditions, recording date, market, account state and whether personalization was active. Third, capture the products named, order, stated reason, displayed sources and any uncertainty language.

Fourth, let AI normalize the observations into a consistent table, but require it to preserve the original wording beside every classification. Fifth, separate organic recommendations, cited sources and paid placements. OpenAI said in September that ChatGPT Ads had expanded to more than 60 countries; treating sponsored presence as equivalent to an organic recommendation would corrupt the audit before analysis begins.

Do not use an API result as a convenient substitute for what customers see. The preprint found mean source-domain overlap of only 12.0% between ChatGPT’s interface and API, and 14.8% for Gemini. Different surfaces are different test conditions.

Keep the Meaning at a Human Promise Gate

AI can cluster reasons, flag contradictory claims and count appearances. A person must decide whether the apparent gap is commercially meaningful and whether the proposed response is truthful. A missing mention may reflect weak evidence, an unavailable product, an ambiguous query or ordinary model variation; it is not automatic proof of an optimization problem.

OrionPilot’s public How It Works approach is designed to connect business context, strategy, execution, connected data, analytics and continuous learning. Its role in this workflow is continuity: keep the buyer question, approved facts, observed recommendation patterns and chosen marketing response attached to one another. That does not assume a direct integration with any AI answer interface; the evidence collection can remain manual while OrionPilot helps preserve the decision chain.

The human gate should approve one response only when the company can support it: clarify a specification, strengthen a comparison page, add missing proof, or leave the marketing unchanged because the variation is too weak to justify action.

Illustrative Example: A Blender Brand in an Unstable Category

An illustrative blender company tests three buyer prompts, five times each, once a week for four weeks. Its product appears frequently when the prompt emphasizes quiet operation, rarely when the prompt asks for the easiest cleanup, and inconsistently when price leads. Sources repeatedly cite an outdated review that lists an old noise specification.

The AI-assisted table proposes three interventions. A person rejects “quietest blender” because the company lacks comparative proof. The approved output is narrower: update the specifications page with the current tested operating range and publication date, add a plain-language cleaning explanation, and ask the review publisher to correct the obsolete figure. The next audit uses the same prompt set; it does not invent easier questions to manufacture improvement.

Measure Stability, Then Test the Explanation

Use recommendation stability as the primary measure: the percentage of comparable runs in which the product appears for the intended buyer job with a factually supportable reason. Report the rate by prompt type and interface, with the run count beside it. Fifteen observations are a useful minimum for learning, not a universal benchmark or proof of market share.

Run the audit weekly for four to six weeks, long enough to see whether an evidence change precedes a durable pattern rather than one lucky answer. Also track claim accuracy and source recurrence. If visibility rises while unsupported statements rise too, the result is not success.

The serious failure mode is optimizing for the model instead of the customer: publishing repetitive pages, exaggerated comparisons or artificial language solely to trigger a mention. That can damage trust and obscure whether the product actually solves the buyer’s problem. The audit should improve the evidence a customer can verify, not teach the brand to chase a moving sentence.