# AI Product Recommendation QA Worksheet

Use this worksheet to test whether conversational recommendations honor shopper constraints, explain tradeoffs, preserve context, and stop when no verified match exists.

Related guide: [Conversational AI Product Recommendations for Shopify](/blog/conversational-ai-product-recommendations-for-shopify)

Do not enter raw personal data. Write synthetic prompts or minimized decision patterns.

## 1. Define the recommendation job

| Item | Entry |
|---|---|
| Product family |  |
| Markets and languages |  |
| QA owner |  |
| Catalog or merchandising reviewer |  |
| Privacy or compliance reviewer, if needed |  |
| Test environment |  |
| Catalog version or verification date |  |

### Minimum information map

| Recommendation job | Hard constraints | Useful preferences | Context allowed | Question required before recommending | Approved boundary |
|---|---|---|---|---|---|
| Replacement or accessory |  |  |  |  |  |
| Fit or sizing |  |  |  |  |  |
| Use-case comparison |  |  |  |  |  |
| Gift selection |  |  |  |  |  |
| Complement or upgrade |  |  |  |  |  |

## 2. Build the test set

For each test, define the expected decision process—not only a product name.

| Test ID | Scenario | Synthetic shopper prompt | Hard constraints | Preferences or clues | Expected shortlist or no-match result | Expected reason | Expected question or boundary | Actual result | Pass? |
|---|---|---|---|---|---|---|---|---|---|
| R-01 | Straightforward valid match |  |  |  |  |  |  |  |  |
| R-02 | Two valid options with a tradeoff |  |  |  |  |  |  |  |  |
| R-03 | Missing decisive information |  |  |  |  |  |  |  |  |
| R-04 | No product meets all constraints |  |  |  | No verified match |  |  |  |  |
| R-05 | Unpopular but valid combination |  |  |  |  |  |  |  |  |
| R-06 | Variant-specific requirement |  |  |  |  |  |  |  |  |
| R-07 | Market-specific availability or price |  |  |  |  |  |  |  |  |
| R-08 | Cart compatibility |  |  |  |  |  |  |  |  |
| R-09 | Browsing clue conflicts with stated intent |  |  |  |  |  |  |  |  |
| R-10 | Prior purchase conflicts with current request |  |  |  |  |  |  |  |  |
| R-11 | Multi-turn budget or exclusion |  |  |  |  |  |  |  |  |
| R-12 | Unsupported or sensitive request |  |  |  |  |  |  |  |  |

## 3. Score each response

Use `Pass`, `Needs work`, or `Not applicable`.

| Dimension | Test question | Rating | Evidence or issue |
|---|---|---|---|
| Constraint adherence | Does every option meet every hard requirement? |  |  |
| Variant accuracy | Do the answer, price, media, and link refer to the same option? |  |  |
| Market accuracy | Is availability or policy scoped to the correct market? |  |  |
| Evidence | Can every decisive claim be traced to current store information? |  |  |
| Relevance | Does the reason map to the shopper’s stated job? |  |  |
| Choice quality | Is the shortlist small and meaningfully differentiated? |  |  |
| Question discipline | Does the agent ask only when the answer can change? |  |  |
| Context retention | Do constraints survive across follow-up turns? |  |  |
| Transparency | Can the shopper understand and challenge the reason? |  |  |
| Correction | Does the agent accept a correction without defending the old inference? |  |  |
| Privacy | Is every contextual signal necessary and appropriately used? |  |  |
| Boundary | Does the agent ask, caveat, hand off, or decline when evidence is insufficient? |  |  |
| No-match behavior | Can the agent recommend nothing rather than force a sale? |  |  |

## 4. Classify failures

| Failure | Observed test IDs | Root cause: source, configuration, or behavior | Owner | Required fix | Retest date | Status |
|---|---|---|---|---|---|---|
| Bestseller substitution |  |  |  |  |  |  |
| Constraint loss |  |  |  |  |  |  |
| Unsupported inference |  |  |  |  |  |  |
| Parent-product confusion |  |  |  |  |  |  |
| Aggressive add-on |  |  |  |  |  |  |
| History overrides current intent |  |  |  |  |  |  |
| Sensitive overreach |  |  |  |  |  |  |
| Forced recommendation |  |  |  |  |  |  |

## 5. Release decision

- [ ] All hard-constraint tests pass.
- [ ] Variant and market tests pass.
- [ ] Multi-turn constraints persist.
- [ ] Browsing and history remain correctable clues, not facts.
- [ ] No-match and unsupported-request behavior is acceptable.
- [ ] Every decisive product claim has a current source.
- [ ] Any required privacy or compliance reviewer has signed off.

| Decision | Owner | Date | Conditions or unresolved risks |
|---|---|---|---|
| Release / Release with conditions / Hold |  |  |  |