The best AI incident response practices start from an uncomfortable fact: an AI failure is not a security incident, and a plan written for one handles the other badly. There is no adversary to expel, no perimeter to close, and no moment when the bad thing stops happening on its own. What you have instead is a system that produced plausible wrong output, possibly for weeks, that people downstream have already acted on. Response means pinning the version, finding every affected output, and deciding who needs to be told, in that order.
The Five Things That Make an AI Incident Different
- There is no attacker. Nobody is expelled, and the system was working exactly as designed while producing the wrong answer.
- Detection is statistical, not signature-based. You notice a drift in a rate, or a customer complains, rather than an alert firing.
- The harm is already distributed. Wrong outputs left the building as advice, quotes, or decisions before anyone knew.
- Containment means pinning a version. Rolling back is the closest analogue to isolation, and most teams have never tested it.
- Evidence lives in traces, not logs. If request-level model version and inputs were not retained, the investigation cannot be done.
Written for IT managers and operations leads at 50 to 500 employee firms running AI in a customer-facing or decision-support path.
Why Your Security Incident Plan Does Not Fit
A security incident plan does not fit an AI failure because nearly every step assumes an adversary and a containable blast radius. Isolate the host, revoke the credential, close the vector. None of those verbs apply when the system behaved exactly as built and the output was simply wrong. We keep the security plan and write a short separate procedure beside it, because forcing one document to cover both produces a plan that handles neither cleanly.
The Detection Problem Nobody Solves With Alerts
Security detection looks for known-bad patterns. An AI failure has no signature: a model that starts producing subtly worse answers looks identical to one producing good ones until somebody reads the output carefully. The practical detections are a shifting rate, a rise in human corrections, a change in escalation volume, or a complaint. We instrument the rates rather than the outputs, because rates move before anyone notices individual answers, and we sample real outputs weekly regardless. The structured approach to that sampling is the subject of continuous AI evaluation and risk control, and without it the first detector in the chain is a customer.
Severity Levels That Mean Something for Output Quality
Security severity scales measure exposure. An AI severity scale should measure consequence and reach: how wrong, to whom, and how many. We use three levels. An internal-only wrong answer someone caught is low. A wrong answer that reached a customer without correction is medium. A wrong answer that reached a customer and was acted on, or that touches a regulated decision, is high and pages someone. The distinction that matters is not how badly the model performed but whether anybody relied on it, and teams consistently over-weight the first and under-weight the second.
Who Gets Called, and Why It Is Not the Security Team
The responder for an AI failure is whoever understands the workflow and can judge the output, which is usually a domain expert rather than an engineer. A security incident calls the security function; an AI failure calls the person who knows what the answer should have been. Name them in the procedure with a deputy, because the bus factor here is genuinely one person in most mid-sized firms. Security still gets told, since some AI failures turn out to have a security cause, and our existing data breach incident response path takes over the moment that becomes likely.
The Response Sequence That Actually Works
The sequence is stop the bleeding, establish scope, remediate the outputs, then fix the cause. Teams reliably do these in the wrong order, starting with the cause because it is the intellectually interesting part, while wrong answers keep going out.
Stop the Bleeding: Pin, Disable, or Gate
Three options exist and most teams have tested none of them. Pinning holds the system on a known-good model version, which requires that your platform supports version pinning and that you recorded which version was good. Disabling switches the feature off and falls back to the manual process, which is the safest option and the one people resist because it is visible. Gating inserts human review between the output and the recipient, which preserves capability at the cost of throughput and is usually the right middle choice. Decide in advance which is available for each deployment, and rehearse the switch, because discovering during an incident that nobody knows how to pin a version costs hours that matter.
Establish Scope: Which Outputs, Over What Window
Scope means every output the affected behavior touched, which is a data question rather than a judgment call. It needs request-level records: what went in, which model version answered, what came out, and who received it. Teams that kept only aggregate metrics cannot answer this, and the investigation collapses into guesswork about a start date. Set retention for these traces deliberately, balancing the investigation need against the sensitivity of what the traces contain, since a full record of every prompt is itself a data-protection consideration. Where several agents coordinate, scope spreads across handoffs, which is one of the design consequences in multi-agent systems and coordinating AI agents.
Remediate the Outputs, Not Just the System
This is the step with no security analogue and the one that matters most to the people affected. For each wrong output that reached someone, decide whether it needs correcting, and correct it. A quote that was wrong gets reissued. Advice that was wrong gets a follow-up. A record updated on a bad inference gets fixed, including anything derived from it downstream. This work is unglamorous, takes longer than the technical fix, and is the entire difference between an incident handled and an incident concealed. Where the output informed a regulated decision, involve counsel before deciding what constitutes adequate correction.
Fix the Cause Without Guessing at It
Only after the outputs are handled does the cause become the priority, and the honest starting position is that you may not be able to establish it. An AI failure has several candidate causes that present identically: the vendor changed the model underneath you, your prompt was edited, a retrieval source went stale or was partially reindexed, an upstream data feed changed format, or the input distribution shifted because the business started doing something new. Work through them in the order you can rule them out cheaply. Version history and vendor release notes are fast. Prompt and configuration change logs are fast if you kept them and impossible if you did not. Retrieval freshness takes an afternoon. Input distribution shift is the one most often missed and is worth checking early, because it is not a defect at all: the system is being asked questions it was never evaluated on, and the fix is scope rather than repair. Where you genuinely cannot determine the cause, say so in the record and describe the compensating control you left in place, which is a far better outcome than a confident wrong diagnosis that closes the incident and lets it recur.
Preparing Before You Need It
Preparation for AI incidents is mostly about three artifacts that take an afternoon each and are worthless if created during the incident instead.
The Rollback Runbook, Tested Once a Quarter
Write down, per deployment, how to pin a version, how to disable the feature, how to enable human gating, who can authorize each, and how long each takes. Then execute it once a quarter in production during a quiet hour. The rehearsal always surfaces something: a permission nobody has, a setting that requires a vendor ticket, a fallback process that no longer works because the manual steps were removed when the tool went in. A runbook that has never been run is a document, not a control, which is the same lesson as a cyber incident response plan that lives in a drawer.
The Known-Good Baseline
Record what normal looks like for each deployment: the correction rate, the escalation rate, the typical output length, the refusal rate. Without a baseline, a degraded system is invisible until it is badly degraded, because nobody can say what changed. This costs almost nothing when set up alongside the deployment and is close to impossible to reconstruct afterward, since the historical data you would use to establish the baseline is the data you now suspect.
The Vendor Escalation Path You Have Actually Used
Find out before the incident who at your vendor answers a quality question rather than a billing one, what evidence they need, and what their response commitment is. Many AI failures originate upstream in a model change you did not control, and your leverage in that conversation depends entirely on having request-level evidence and a named contact. The governance wrapper around all of this sits in the AI agent governance guide, and the exposure model in are AI agents safe for enterprise use.
What to Tell People, and When
Communication during an AI failure is harder than during a breach because there is no established template and the instinct is to say nothing until the cause is known. That instinct is wrong for the same reason it is wrong in a security incident: the people who acted on a wrong output need to know before they act again, and the cause can follow later. Tell affected recipients what was wrong, over what period, and what you are doing about their specific case, in plain language that does not use the word model. Internally, tell the staff who relied on the system first, because they are answering customers right now and will otherwise repeat the error with confidence. Where a regulated decision is involved, counsel decides the wording and the timing, and that call should be made early rather than after a draft has circulated. What consistently damages trust is not the failure but a correction that arrives weeks later carrying the implication that somebody knew sooner. A short, early, accurate note ages far better than a thorough one that was held.
Frequently Asked Questions
Is a wrong AI output a reportable incident?
Not usually under breach rules, since no data was exposed, but it can be reportable under sector regulation where an automated decision affected someone. Define internally what counts as an incident regardless of external reportability, because an internal definition is what triggers the response at all.
How do we detect an AI failure before customers do?
Instrument rates rather than individual outputs: human correction rate, escalation rate, refusal rate, and output length distribution. Those move before anyone notices a specific bad answer. Pair the rates with a weekly sample of real outputs reviewed by someone who knows the domain.
What is the AI equivalent of containment?
Pinning the model version, disabling the feature in favor of the manual process, or inserting human review between output and recipient. Decide which is available per deployment before you need it, and rehearse the switch, because these are configuration paths most teams have never exercised.
How far back do we need request-level records?
Far enough to establish when a degradation began, which in practice means months rather than days because subtle drift goes unnoticed. Balance that against the sensitivity of the records themselves, since retained prompts and outputs are a data-protection consideration in their own right.
Who should lead an AI incident?
Whoever can judge whether the output was right, usually a domain expert rather than an engineer or the security team. Name that person and a deputy in the procedure. Security is informed and takes over if a security cause becomes likely, which does happen and should not be the default assumption.
Who Is Behind This Guidance
Our team runs both security incident response and AI deployments for firms between 50 and 500 employees, which is an unusual combination and the reason this article insists the two are different. We have watched capable security teams apply a containment mindset to a model quality problem and make no progress for a day, because the verbs did not fit the situation. We have also seen AI failures that turned out to have a security root cause, which is why the handoff matters. Matt Rosenthal, Mindcore’s CEO, keeps the practice focused on making technology risk legible to the people accountable for it, which here means a procedure an operations lead can execute at eight in the evening without an engineer. That is the standard this is written to.
Write the Short Procedure Now
Take an afternoon per deployment. Write the rollback runbook covering pin, disable, and gate, with the authorizer and the elapsed time for each, then rehearse it once this quarter. Record the known-good baseline rates while the system is behaving. Find your vendor’s quality escalation contact and what evidence they need. Define three severity levels by consequence and reach, name the responder and a deputy, and set trace retention deliberately. None of that requires new tooling and all of it is close to impossible to produce during an actual incident. Most teams we work with have an excellent security incident plan and nothing at all for the failure mode they are far more likely to meet this year. If you want the response procedure built alongside the deployment, our enterprise AI integration practice covers both. Bring your current incident plan to a free strategy call and we will show you which steps stop working when there is no attacker.
