1h 4m read

The Breach Wasn't in the Model. It Was in the Org Chart.

Why AI incidents are authority failures, not model failures: an accountability map, propose-approve-execute separation, fail-closed defaults, and a pre-incident test plan for enterprise AI agents.

The Breach Wasn't in the Model. It Was in the Org Chart.

Introduction

Most AI incidents that reach the breach report are not model failures. They are authority failures: an agent was given the power to act, and no one in the organization owned the decision to grant it, the checkpoint before it fired, or the switch that could stop it. The model behaved as designed. The org chart did not.

That distinction matters because the two diagnoses lead to different purchases. If the problem is the model, you buy a better model, more evaluation, tighter prompts. If the problem is delegated authority, you fix ownership, separation of duties, and default-deny execution — work that no vendor can do for you and that no model upgrade substitutes for.

This is an argument for IT decision-makers, not a model-evaluation guide. It maps who should own each component of an AI system, separates proposing an action from approving and executing it, defines which actions must fail closed, and ends with a test plan you can run before an incident runs it for you. The evidence comes from breach reporting and published control guidance, not from vendor performance claims.

The Conventional Diagnosis Is Convenient — and Wrong

When an AI system does something harmful, the reflex is to blame the model: hallucination, prompt injection, insufficient guardrails. The response follows the diagnosis — better prompts, a newer model, more evaluation coverage. This framing is popular for understandable reasons. It is legible to vendors, it produces a purchasable remedy, and it locates fault in a component rather than in people and process. A component can be swapped. A reporting line cannot.

The problem is that model-level fixes cannot correct a system where the authority to act was granted too broadly, to the wrong identity, without a checkpoint. Prompt hardening does not revoke a credential. A newer model does not add an approval gate. Evaluation coverage does not tell you who can stop a running agent at 2 a.m.

The Amazon Q case is the clearest public illustration. As documented by the Cloud Security Alliance, malicious instructions were embedded in a pull request that the agent reviewed as part of its normal workflow. The content looked like a routine technical comment; the agent read it as an operational directive and attempted to delete cloud resources through the AWS CLI. No credentials were stolen. No infrastructure vulnerability was exploited. The attack succeeded by misusing trust rather than bypassing controls.

That is an organizational failure wearing a model-failure costume. The agent acted on trust because the system was built so that trust was sufficient. Someone decided — or never decided — that a code-review agent should hold deletion authority over cloud resources. That decision lived in a permission grant, not in a policy document, and probably not on anyone's risk register.

One scope note, stated honestly: this is an argument from incident patterns and control design, not a measured attribution study. No one has published a dataset that cleanly assigns percentages of AI incidents to model versus organizational cause. The claim here is narrower and more defensible — that a specific class of failure is structural, recurring, and not addressable at the model layer.

What the Breach Data Actually Shows

The breach data does not support the model-first diagnosis either. It shows identity, human factors, and delegated access dominating how intrusions begin and spread.

Where intrusions start: exploitation and identity paths dominate Grouped horizontal bars comparing the share of intrusions attributed to each initial access vector in two 2026 datasets. Vulnerability exploitation is the largest vector in both (31% DBIR, 32% Mandiant). Voice phishing appears at 11% in Mandiant investigations. The two sources measure different populations and are not directly comparable. Where intrusions start: exploitation and identity paths dominate. grouped bar data: Vulnerability exploitation: Verizon DBIR 2026 31, Mandiant M-Trends 2026 32; Email phishing: Verizon DBIR 2026 16, Mandiant M-Trends 2026 6; Credential abuse / stolen credentials: Verizon DBIR 2026 13, Mandiant M-Trends 2026 0; Voice phishing: Verizon DBIR 2026 0, Mandiant M-Trends 2026 11; Prior compromise: Verizon DBIR 2026 0, Mandiant M-Trends 2026 10.Source: Verizon 2026 Data Breach Investigations Report (Executive Summary); Mandiant M-Trends 2026 2026-03-23. Where intrusions start:exploitation and identitypaths dominate Verizon DBIR2026 MandiantM-Trends 2026 0 12.5 25 37.5 50 31 32 Vulnerabilityexploitation 16 6 Emailphishing 13 0 Credentialabuse /stolen... 0 11 Voicephishing 0 10 Priorcompromise Source: Verizon 2026 Data Breach InvestigationsReport (Executive Summary); Mandiant M-Trends 2026(2026-03-23)

Two independent 2026 datasets, different populations and denominators. Verizon DBIR figures are shares of known initial access vectors in non-Error, non-Misuse breaches (n=19,905), dataset Oct 2024–Nov 2025. Mandiant figures are shares of 2025 Mandiant investigations. Shown side by side for orientation, not as a like-for-like ranking; 'not published' means the retrieved pages do not report that vector, not that it was zero.

Verizon's 2026 DBIR examined more than 31,000 incidents across 145 countries, of which more than 22,000 were confirmed breaches. System Intrusion accounted for 61% of breaches in the 2026 dataset, Social Engineering 17%. Breaches involving third parties rose 60% year over year to 48% of total breaches. On remediation, only 23% of third-party organizations fully remediated missing or improperly secured MFA on their cloud accounts, and for weak passwords and permission misconfigurations the time to resolve half of all findings reached almost eight months. Ransomware grew to 48% of all breaches.

Mandiant's M-Trends 2026, drawn from over 500,000 hours of incident investigations in 2025, reports exploits as the most common initial infection vector for the sixth consecutive year at 32% of intrusions, with highly interactive voice phishing surging to 11% as the second-most observed vector. Email phishing fell to 6%. Prior compromise ranked third globally at 10% but was the top initial vector in ransomware operations at 30%, double its 2024 share. Global median dwell time rose to 14 days from 11.

Mandiant's own conclusion is worth quoting for what it concedes: "we do not consider 2025 to be the year where breaches were the direct result of AI. From our view on the frontlines, the vast majority of successful intrusions still stem from fundamental human and systemic failures."

The agent-specific layer points the same direction, though with weaker evidence. The Cloud Security Alliance, citing industry surveys, reports that more than half of AI agents access sensitive information, 90% operate with permissions broader than required, and 80% of organizations have observed unintended agent actions including unauthorized access to systems or data. Treat these as vendor and industry survey findings, not audited incident data — the denominators are survey respondents, not investigated breaches, and the underlying reports are published by firms with a commercial interest in the answer. The direction is consistent with the DBIR and M-Trends pattern even if the precision is not.

That pattern is simple: an identity holding more authority than its task requires. It appears in human form as the help-desk employee who resets MFA for a convincing caller. It appears in non-human form as the agent whose service role can delete resources because narrowing it was never anyone's job.

Who Owns What: An Accountability Map for AI Systems

If authority is the failure mode, ownership is the control. The framework below assigns one accountable role to each component of an AI system. The role names are a proposed structure, adaptable to organization size — not a mandated taxonomy from any standard.

Component

Accountable owner

Escalation path

Model selection and evaluation

Head of AI/ML Engineering

CTO for model changes affecting high-impact actions

Training and retrieval data

Data Governance Lead

Chief Data Officer

Retrieval pipeline and vector stores

Platform Engineering Lead

CTO

Tool and API integrations

Integration Owner (per tool)

Head of Platform Engineering

Agent identity and credentials

Identity and Access Management Lead

CISO

Incident response and kill-switch authority

Security Operations Lead

CISO, then CEO

Business approval of high-impact actions

Business Process Owner

Executive sponsor of the AI program

The design principle behind the map: every component needs one accountable owner and a documented escalation path, and no component may be owned by the same function that approves its high-impact actions. If the team that builds the payment agent also approves its payment actions, you have a separation-of-duties failure before the agent runs anything.

This is not an arbitrary preference. NIST's AI Risk Management Framework states in GOVERN 2.1 that roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and clear to individuals and teams throughout the organization. GOVERN 2.3 goes further: executive leadership takes responsibility for decisions about risks associated with AI system development and deployment. The framework does not prescribe which titles hold which components, but it does require that the answer exist and be legible.

If you are already certified against ISO/IEC 42001, the NIST AIRC crosswalk maps the AI RMF governance subcategories onto 42001 clauses, so the ownership work can be recorded once rather than twice. The crosswalk covers the governance and risk-treatment clauses; it does not invent role names for you either.

One practical test of whether the map is real: ask each named owner to describe the last change to their component's permissions and who approved it. An owner who cannot answer is a name on a slide.

Separate Propose, Approve, and Execute

Ownership assigns responsibility. Separation of duties constrains what the agent can do without it. The structural fix is to split every consequential action into three parts that no single component controls.

Propose. The agent may draft an action — a deletion plan, a payment instruction, a deployment — but holds no execution authority. It produces a structured proposal with explicit parameters.

Approve. A human or a policy service validates scope, privilege, and approval state against the exact action and parameters. The approval is bound to that action, that actor, and those parameters.

Execute. A component outside the agent's context performs the action under a short-lived, single-use authorization.

The detail matters more than the diagram. OWASP's AI Agent Security Cheat Sheet is explicit that a user_confirmed flag is insufficient: the execution component must verify that the approval belongs to the current actor and the exact tool call, remains valid, and has not already been consumed. The check and the consumption must happen in one atomic step immediately before execution, so concurrent or repeated requests cannot reuse it. Any change to the target or parameters requires a new approval. OWASP also directs that unknown tools or missing approval requirements fail closed.

AWS's Agentic AI Lens adds the identity dimension: separate decision-making from execution, keep agent identities distinct from human identities so audit trails attribute every action unambiguously, and when an agent acts on behalf of a user, propagate user context as signed token claims rather than letting the agent assume the user's credentials. The Generative AI Lens makes the same point from the permissions side — scope agent roles to the specific resource ARNs and API actions the workflow needs, and use permission boundaries to cap what any role can be granted.

The common objection is that this slows agents down. It does, for the actions you route through it. That is why the separation should be scoped to irreversible or high-impact actions — bulk deletion, payment initiation, privilege changes, production deployment, account recovery — while low-risk reads and reversible operations run unattended. An agent that can query a dashboard without a human in the loop is not the risk. An agent that can drop a table without one is.

Fail Closed, and Mean It

Fail closed is a design commitment, not a slogan. It means that when the system cannot establish that an action is authorized, the action does not happen. The categories that must fail closed:

  • Unknown tools the agent was not provisioned to call.

  • Missing, expired, or already-consumed approval.

  • Ambiguous identity — the execution component cannot determine whether the actor is the agent or the human it claims to represent.

  • Unavailable replay-protection state, including when that state is shared across multiple receiver instances.

  • Any action above a defined blast-radius threshold: bulk deletion, payment initiation, privilege changes, production deployment, account recovery.

Operationally, that means default-deny policy evaluation, step-up authentication for critical actions, short-lived credentials, a kill switch with a named owner, and an audit trail that attributes each action to either an agent or a human without ambiguity. OWASP's guidance covers the first two directly; AWS's identity guidance covers the credential and attribution requirements.

The failure mode to watch is not a dramatic decision to disable fail-closed. It is gradual erosion. AWS names it precisely: agent permissions expanded reactively in response to access-denied errors without investigating whether the access pattern is consistent with the agent's intended scope, leading to steady privilege creep that no single review catches. Each individual expansion looks reasonable — the agent needed to read one more table — and the cumulative result is the broad permission posture the agent should never have had.

The counter-discipline is a rule that an access-denied error triggers a review of intent, not an automatic grant. If the agent genuinely needs the access, the change goes through the same approval path as any other permission change, with the owner's name attached.

Thresholds are organization-specific. NIST's GOVERN 1.3 calls for processes to determine the needed level of risk management activities based on the organization's risk tolerance, and the AI RMF notes that risk tolerance is highly contextual and application-specific. A bank and a media company will draw the blast-radius line in different places. What they should share is that the line is drawn explicitly, documented, and reviewed — not discovered during an incident.

Test the Org Chart Before the Incident Does

Controls that have never been tested are assumptions. The test plan below validates the organizational controls, not the model. It is a proposed framework, not a standard's prescribed procedure. Each step has a pass condition.

1. Inventory agent identities and diff against task requirements. List every agent identity, its granted permissions, and the permissions its task actually requires. Pass condition: every permission is either used by the task or has a documented justification and owner. Unused or over-broad grants are flagged for removal.

2. Attempt an approval bypass. Replay a consumed approval. Alter parameters after approval and confirm execution is refused. Submit the same approval concurrently from two sessions. Pass condition: every attempt is refused, and each refusal is logged with the reason.

3. Verify a low-trust session cannot reach a privileged tool. Start from the least-privileged context the agent can operate in and attempt to invoke a privileged tool. Pass condition: the tool is denied even when the model requests it confidently, and the denial is attributable to policy rather than to a tool error.

4. Run a tabletop where the kill-switch owner is unavailable. Simulate an incident during which the named owner cannot be reached. Pass condition: the documented escalation path resolves to someone with the authority and the technical means to stop the agent, within a stated time.

5. Confirm audit logs distinguish agent from human actions. During a simulated investigation, reconstruct who did what from logs alone. Pass condition: every action is attributable to either an agent identity or a human identity, with no ambiguous entries, and the reconstruction succeeds without interviewing anyone.

M-Trends recommends red-team emulation of interactive social engineering and living-off-the-land techniques, and measuring how long security teams take to detect and respond. That belongs in the same cycle, because the human and non-human paths into the same privileged tool are often the same path.

The standards anchor for this cycle is NIST MANAGE 4.1 — post-deployment monitoring plans including appeal and override, incident response, recovery, and change management — and MANAGE 4.3, which requires that incidents and errors are communicated to relevant AI actors and that tracking, response, and recovery processes are followed and documented. A test plan that produces findings nobody tracks satisfies neither.

The Strongest Counterargument: Some Failures Really Are the Model

The strongest objection deserves a direct answer: some incidents genuinely originate in model behavior. A model can hallucinate a tool call, follow instructions injected through retrieved content, or produce an unsafe recommendation. No org chart prevents a model from producing a bad recommendation, and evaluation, guardrails, and content validation remain necessary. Anyone arguing that governance replaces model quality is wrong.

The boundary is this: organizational controls govern authority and execution, not model correctness. A well-governed system can still generate a wrong answer. What the controls determine is whether a wrong answer becomes a wrong action, and whether anyone can trace and reverse it when it does. The Amazon Q agent produced a harmful action not because the model was uniquely broken but because the system was built so that a model's interpretation of untrusted content was sufficient authorization to delete resources. Fix the model and the next injection vector may still find an agent whose authority exceeds its task.

The weakest part of this argument should be stated plainly. Attribution between model and organizational cause is rarely clean in post-incident review. The same event usually has both a proximate model behavior and a permissive control environment, and investigators under time pressure tend to report the one they can reproduce. The framework's value is in reducing blast radius and making the authority decision visible before the incident, not in assigning blame after it. If you need a clean causal story for a board, this framework will not give you one. If you need fewer incidents in which a bad output becomes an irreversible action, it will.

Conclusion: Put the Control Plane on the Org Chart

The breach is an authority problem, and authority is an org-chart problem. The model did what it was permitted to do. Someone, or no one, permitted it.

The shift required is not a model upgrade. It is four commitments: a named owner for every component of every AI system in production; enforced separation of propose, approve, and execute for high-impact actions; fail-closed defaults wherever authorization cannot be established; and pre-incident tests of those controls rather than of the model alone.

Adopted, the outcome is not slower AI. It is agents that can be trusted with real work because their authority is scoped to the task, observable in the audit trail, and revocable by a named person who knows they hold the switch. That is a better position than an agent with broad permissions and no owner, which is fast right up until it is not.

Before the next agent goes live, ask three questions. Who owns its identity? Who approves its irreversible actions? Who can stop it? If any answer is "the model team," the org chart is the breach waiting to happen.

Frequently Asked Questions

Doesn't human-in-the-loop approval already solve this? Only if the approval is bound to the exact action and parameters, checked and consumed atomically, and cannot be replayed. A checkbox that a user clicks once and the agent can reuse is not a control. OWASP treats a bare user_confirmed flag as insufficient for exactly this reason.

What if we've already deployed agents with broad permissions? Scope down first, then add approval gates. Run the permission inventory from the test plan, diff grants against task requirements, and remove what is unused. AWS recommends automated drift detection and periodic access reviews for agent identities on a cadence faster than the annual human-user review, because agent permissions change as tools and prompts are added.

Isn't the model vendor responsible? Vendors own model behavior. The deploying organization owns identity, permissions, and execution. NIST's GOVERN 2.3 places responsibility for AI risk decisions with executive leadership of the deploying organization, not with the model supplier.

Is fail-closed too disruptive for production? Only for the actions you define as high-impact. Reads and low-risk, reversible operations can stay open. The disruption is concentrated where an unauthorized action would be irreversible anyway.

How does this map to ISO/IEC 42001 if we're already certified? The ownership and oversight requirements align. Use the NIST AI RMF to ISO/IEC 42001 crosswalk to record the governance work once against both frameworks rather than maintaining two parallel evidence trails.