What an AI Shopping Test Can—and Cannot—Establish

An AI shopping test is most useful when its conclusion names the exact point the buyer reached. Finding the right product, putting it in a cart and completing a purchase are different observations. Treating them as one result can hide both a useful success and an important gap.

For a merchant, the practical question is: what evidence would show that this buyer objective worked under these test conditions? Define that before running the journey.

Start with a buyer objective

“Test our store with AI” leaves too much open. A more useful objective specifies the product need, important constraints and permitted stopping point.

For example, a fictional buyer needs a desk lamp with a replaceable bulb and an adjustable arm, within a stated budget. The test is authorized to inspect public pages and build a cart, then stop before submitting personal information or payment. This is an illustrative scenario, not a finding about a real merchant.

A record of that test should identify the date, relevant URLs, agent or browser environment, configuration and any human assistance. If a person intervened to choose a variant or dismiss a dialog, record that intervention. It affects how someone should interpret the result.

Keep the evidence stages separate

Use the following distinctions to describe what happened:

1. Discovery: the agent located a relevant page. This does not establish that it correctly understood the offer.
2. Product understanding: the agent connected the buyer’s requirements to information visible in the tested sources. Unanswered specifications remain unanswered.
3. Configuration and cart: the selected variant, quantity and displayed price persisted into the cart. A button click alone does not establish that the cart changed.
4. Checkout access: the intended items reached the checkout interface. That does not establish payment acceptance, fulfillment or delivery.
5. Completed buyer outcome: a separately authorized purchase and delivery check established the specific result claimed. For a digital resource, receiving a link and retaining a usable file are themselves different observations.

These are BSH’s explanatory distinctions for reading a test, not a universal score or certification scheme. The existing independent-test library presents bounded, point-in-time observations and discloses that the featured companies did not commission those assessments.

Write a conclusion that matches the observation

Return to the fictional lamp example. Suppose the correct model and color remain visible in the cart, but the test stops before checkout.

A precise result would say:

In this illustrative test, the selected lamp and color remained in the cart. Checkout, payment and delivery were not tested.

That statement gives a developer something concrete to preserve. “The AI can buy from this store” would claim more than the example established.

If the displayed cart remained empty after an attempted addition, record that visible result, the attempted action and whether an error appeared. The observation alone does not identify the cause. A site defect, session condition, interaction limitation or other factor may need separate investigation.

Separate discovery from transaction performance

Being discoverable in AI-assisted search does not demonstrate that an agent can finish a buying journey. Google’s documentation addresses eligibility for supporting links in AI Overviews and AI Mode: it describes indexing and snippet eligibility, and explicitly does not guarantee indexing or serving. That is a different question from whether a particular cart or checkout interaction works. Google Search Central: AI features and your website

Similarly, a successful journey in one environment does not establish compatibility with every agent, device, account state or future site version. Preserve the conditions alongside the result so a later comparison has a useful starting point.

Turn a finding into a focused next check

For each unresolved point, identify the smallest check that could resolve it. If a variant did not persist, check that specific transition. If a subscription condition was unclear, verify the applicable offer language. If payment was outside scope, leave it marked untested rather than treating the boundary as a failed payment.

A useful retest repeats the relevant buyer objective, records what changed and compares the same observable outcome. It should explain what the new evidence resolves and what remains outside scope.

Review BSH’s Agent Readiness assessment scope if you need a bounded assessment of an actual buyer journey.

Sources and context

• Google Search Central — AI features and your website: search eligibility and its limits.
• BSH Independent AI Shopping Tests: existing public examples of scoped findings and relationship disclosures.

The lamp scenario and explanatory sequence above are original illustrations. They do not reproduce a merchant test or claim a completed purchase.