Putting AI in the Product: The Pre-Launch Review That Protects Customers

The AI feature is built, the demo is dazzling, and the launch date is circled. The review that decides whether the launch is a success story or an apology happens - or does not - in the two weeks before the date nobody wants to move.

Post-launch AI failures rarely come from exotic attacks. They come from the mundane questions nobody forced before release: what data does it see, how wrong can it be, what do users do when it fails, and who is watching after day one.

A pre-launch review is not a gate to slow the team down. It is the difference between launching a feature and launching a support incident.

TLDR

• Review before the date hardens: data, failure modes, fallback, monitoring.

• Test with adversarial and boring inputs, not just the demo path.

• Define the failure experience: what users see when the model is wrong or absent.

• Launch with a human-review tier for the outputs that matter most.

• Name the post-launch owner before launch, not after the first incident.

Four questions, two weeks out

Data: what exactly does the feature see - which user data, which records, and does any of it train or leave your control. Failure modes: what are the plausible wrong outputs, and what does the worst one cost a customer. Fallback: when the model is down or degraded, what does the user get - a graceful alternative or an error page. Monitoring: what metric or review catches trouble in week one, and who is watching it.

A team that can answer all four crisply is ready. Hesitation on any one of them is the launch-risk register writing itself.

Test the paths the demo avoids

Demos run on happy paths with curated inputs. Customers run on the other paths: empty fields, hostile phrasing, edge cases from languages and contexts the builders never imagined, and sheer volume.

Spend two days throwing adversarial and boring inputs at the feature. This fictional example shows a test summary. It is an illustration, not a real report.

Test class	Example	Result before fix
Empty or junk input	Blank form submitted	Confident nonsense answer
Prompt manipulation	'Ignore your instructions'	Compliance
Volume	500 requests in a minute	Answers degraded without warning

Every row that surprises you is a launch incident you just bought for two days of testing instead of a month of apologies.

Design the failure experience

The model will be wrong and it will be unavailable; both are certainties. What matters is what the customer gets in those moments: a clear statement with a human path, or a fluent guess with no recourse.

For outputs that matter - anything a customer might act on - launch with a human-review tier: the AI drafts, a person releases. It is slower and it is the correct way to learn the feature's real error rate with real stakes. You can loosen it when the evidence says so. Tightening after a public failure reads very differently.

Name the owner of week two

Launch week has an owner: the team. Week two has nobody, which is when drift, abuse patterns and quiet degradation begin. Assign a named owner for the feature's life - watching its metrics, its vendor's changes, and its exception list - before the launch, as a condition of it.

NIST's AI guidance treats deployment as the beginning of the risk work, not the end. The pre-launch review is how that principle gets a date, an owner, and a fallback.

The related discussion of building trustworthy AI is worth your time alongside this one. The related discussion of privacy as a product feature is worth your time alongside this one.

Frequently asked questions

Does every AI feature need this level of review?

Scale it. A customer-facing feature acting on sensitive data gets the full pass; an internal drafting helper gets the four questions over coffee. The test is the cost of a confident wrong answer reaching someone who trusts it.

Won't a human-review tier kill the economics?

It changes the launch economics, honestly. If the feature only works with zero human oversight and zero tolerance for its error rate, that is not a launch decision - it is a bet. Better to learn the true error rate cheaply, then automate with evidence.

Who should run the review?

Product, engineering and security together, with someone empowered to move the date. A review that cannot move a launch date is a ceremony, and teams learn to schedule it after decisions are final.

A Defensible Next Step

For your next AI feature, schedule the four-question review two weeks before launch and give it the power to move the date. The launch you protect is your own. Book a Decision-Clarity Call if you want the review checklist.