The AI Incident Nobody Planned For: Putting Model Failures in the Response Plan

The response plan knows what to do when systems go down and when attackers get in. It has nothing to say about the week the AI feature quietly gave wrong answers to customers - no outage, no intrusion, just confident wrongness at scale.

AI incidents are ordinary incidents wearing new clothes: the failure modes are wrong outputs, silent drift, leaked prompts, and a vendor's model update that changed behavior overnight. They still need detection, containment, communication and learning.

The response plan needs one new chapter, not a rewrite. Write it before the first incident writes it for you.

TLDR

• AI failures are incidents: detect, contain, communicate, learn.

• The hard part is detection - wrong answers do not set off alarms.

• Define containment for each AI feature: throttle, fallback, or suspend.

• Know what your vendor's model updates change, and when.

• Add AI scenarios to the next tabletop; they expose unfamiliar gaps.

Failure modes worth naming

Four patterns cover most AI incidents. Confident wrongness: outputs that are fluent, plausible and false, reaching users at scale. Silent drift: the model or its inputs change, and quality degrades for weeks before anyone measures it. Exposure: sensitive data appears in prompts or outputs where it should not. Vendor surprise: a provider updates the model and your feature's behavior changes without a release note you read.

Each one fails differently from a server outage: no pager fires, no graph turns red. That is what makes them incidents nobody declares.

Detection is the design work

Servers announce their failures; models do not. Detection has to be built: sampled human review of outputs, quality metrics watched over time, user feedback channels someone actually reads, and a definition of 'bad enough to act' agreed in advance.

This fictional example shows trigger definitions. It is an illustration, not a real threshold set.

Signal	Threshold to act	First response
Quality review score	Drops two weeks running	Suspend feature for affected use case
User complaints citing wrong answers	Cluster, not one-off	Review outputs; consider rollback
Sensitive data seen in outputs	Any confirmed case	Suspend, assess exposure, notify owners

Containment: know the off positions

For every AI feature, decide now what containment looks like: throttle it to a smaller audience, fall back to the pre-AI process, or suspend it entirely. The fallback path is the key design requirement - a feature with no fallback has no containment option, which means its incidents end when the vendor feels like it.

Assign the authority to suspend to a named role, with the same clarity as any other incident decision. The middle of an AI incident is the wrong time to discover that nobody is sure who may turn the feature off.

Add it to the tabletop rotation

One scenario for the next exercise: the AI feature has been confidently wrong for a week and a customer just noticed publicly. Watch the room work out who detects it, who decides to suspend, what customers are told, and what the vendor is asked.

NIST's AI risk framework expects monitoring and incident handling to extend to AI systems specifically. Folding model failures into the existing response plan - same roles, same log, same learning loop - is the fastest way to get there.

The related discussion of building trustworthy AI is worth your time alongside this one. The related discussion of a resilient incident response program is worth your time alongside this one.

Frequently asked questions

Is a wrong AI answer really an 'incident'?

When it reaches users at scale, yes - the business impact is real whether or not a server failed. Treating it as an incident gets you the machinery: declared ownership, a log, communication discipline, and a post-incident review.

Who should own AI incident response?

The same team that owns any incident, with the AI feature's owner and vendor contacts added. A separate AI-only process duplicates the machinery and guarantees confusion on the day.

What about vendor-hosted models we do not control?

You control the feature and the fallback, which is what containment needs. Add the vendor's update cadence and support path to the plan, and require notice of model changes in the contract where you can get it.

A Defensible Next Step

Take your response plan and add one page: your AI features, their failure modes, their off positions, and who may use them. Run it in the next tabletop. Book a Decision-Clarity Call if you want the chapter drafted.