Key Insights
Unlike conventional software flaws, AI security vulnerabilities span probabilistic models, the data that shapes them, and the agents that act on their outputs. They also give attackers entry points that traditional application security was never designed to cover.
These types of security issues once meant little more than a chatbot saying something embarrassing, but AI now operates within business workflows where errors and manipulation can cause operational harm. Organizations increasingly rely on these systems, making broken security assumptions more consequential. The Stanford AI Index documented 362 AI incidents in 2025, up from 233 in 2024, a jump that reflects how much wider the exposure surface has become.
This guide maps the vulnerabilities that appear along the AI attack path, from training data and model supply chains through prompt and retrieval layers to autonomous agents and their outputs. Each layer introduces its own failure modes, and a weakness at one boundary can quietly propagate into unauthorized actions, leaked data, or corrupted decisions further down the chain.
Key Takeaways
Every weakness enters at an identifiable trust boundary in training data, a retrieved document, a tool description, or an agent's memory, and boundaries are easier to defend than model judgment.
Unrelated-looking flaws connect: a poisoned dependency upstream becomes a leaked secret or an unauthorized business action downstream.
Autonomy raises the stakes because persistent permissions and delegated access let a manipulated response act on real systems.
Performance failures and adversarial compromise require different responses, so teams must distinguish evaluation problems from security incidents.
What Are AI Security Vulnerabilities?
AI security vulnerabilities are weaknesses in the data, models, applications, integrations, and AI system identities around an AI system that an attacker can exploit to compromise confidentiality, integrity, availability, or authorized behavior. That definition excludes people using a working model to write malicious phishing emails or malware, which is abuse rather than a system flaw.
Teams collect and clean data before developers train or fine-tune a large language model (LLM). At deployment, an application assembles prompts from user input and retrieved documents. Increasingly, the application also drives agents that call tools and write to memory. Gartner projects that 40% of enterprise applications will feature task-specific AI agents by 2026, up from less than 5% in 2025, meaning the number of these agent handoffs is about to expand sharply. Each handoff is a trust boundary where one party accepts content from another as trustworthy.
Conventional software flaws do not disappear here. Model-behavior weaknesses include prompt injection, poisoning, model extraction, and unsafe autonomy. A broken access control flaw on a tool endpoint may be moderate on its own; when an agent that reads untrusted email can reach it, the flaw becomes a zero-click exploit.
Major AI Security Vulnerabilities Across the Attack Path
AI security vulnerabilities form a connected attack path. Upstream data and models feed prompt and retrieval systems, whose responses then reach agents and outputs.
Trust Boundary | Entry Point | How Compromise Propagates | Potential Impact |
|---|---|---|---|
Data and model supply chain | Public datasets, downloaded model files, dependencies, fine-tuning data | Backdoors and poisoned behavior survive fine-tuning and ship inside the deployed model | Triggers can remain hidden while attackers steal model functionality or expose training records |
Prompt and retrieval | User input, retrieved documents, email, web pages, embeddings | Injected instructions execute as if they came from the developer; context leaks through outputs | Attackers can cause sensitive data exfiltration, including leaked system prompts; answers can also be poisoned |
Agents and outputs | Tool descriptions, memory files, peer agents, generated code | A manipulated response becomes a tool call, a stored instruction, or a production artifact | Attackers can take unauthorized actions or plant persistent backdoors, and generated code may remain vulnerable |
An attacker who controls the top row can often reach the bottom one without touching anything in between.
Data and Model Supply Chains: Poisoning, Drift, and Model Theft
Poisoned documents can backdoor models without changing their normal behavior. A backdoor may remain dormant until a trigger appears and survive later fine-tuning or safety training. It can therefore be difficult to spot before deployment.
A 2025 Anthropic-led poisoning study, run with the UK AI Security Institute and the Alan Turing Institute, found that as few as 250 malicious training documents can create a persistent model backdoor regardless of model size or training data volume, meaning even a very small corruption of an otherwise trusted dataset can compromise a production model.
Poisoned data is one of several ways a dependency arrives compromised. Model files in pickle-based formats execute arbitrary code when software loads them, and models carrying reverse shells have repeatedly appeared on public model hubs. Drift belongs here as a category rather than an attack: production data stops resembling training data, accuracy decays with no attacker involved, and the symptoms can resemble the effects of data poisoning.
At this layer, confidentiality attacks trade queries for training data or model internals. Model extraction attacks rebuild functionality through a model's prediction application programming interface (API), while membership inference reveals whether the training set contained a specific record. More invasive techniques recover the records themselves or reconstruct attributes and images from confidence scores.
Prompt and Retrieval Layers: Injection, Leakage, and Vector Weaknesses
Language models have no formal boundary between instructions and data, so any text the model reads can steer it. The main attack paths exploit how applications combine those instructions, retrieved content, and stored representations:
Direct and Indirect Injection: Direct injection is a user-typed override; indirect injection hides instructions in an email, web page, or fetched document. Retrieved content shares context with trusted directions, so the model may follow hidden instructions the user never saw or typed.
Broad Retrieval Permissions: Retrieval permissions determine how much an indirect injection can reach. Broad repositories or documents with unclear provenance expose more sensitive context than a narrowly scoped store.
Context Leakage: The same mechanism can leak that context outward: a request to repeat prior instructions may return the system prompt and any credentials developers placed there, while benign-looking retrieval-augmented generation (RAG) queries can pull database rows and document text verbatim.
Vector Inversion: Inversion attacks can also reconstruct source text from stored vectors.
Spotlighting, input classifiers, and instruction-hierarchy training reduce injection risk, but filters cannot reliably identify instructions that attackers planted. The Open Worldwide Application Security Project (OWASP) states that "it is unclear if there are fool-proof methods of prevention for prompt injection" because of the stochastic way generative models work, which is why retrieval scope matters as much as filtering: it affects both whether hidden instructions reach the model and how much information a successful injection can expose.
Agent and Output Layers: Excessive Agency, Tool Misuse, and Unsafe Results
An agent acts with real credentials through approved interfaces, so a manipulated request can look identical to a legitimate one. Excessive agency describes permissions broader than the task and high-impact functions such as shell execution or production writes. The danger increases when consequential actions require no confirmation.
Compromise can then propagate through tools, memory, peer agents, and generated outputs:
Tool Descriptions: Tools are the conversion point. Developers designed function calling and Model Context Protocol (MCP) servers for interoperability first, so a malicious server can hide instructions in tool descriptions that the model treats as developer intent.
Persistent Memory: Memory adds persistence because agents can retain malicious instructions in files across sessions.
Peer Agent Requests: Multi-agent setups compound the risk when an agent accepts from a peer a request it would refuse from a human, and organizations commonly run agents under generic service accounts with no dedicated identity.
Unsafe Outputs: Output is the last boundary. When applications pass responses into browsers, shells, or Structured Query Language (SQL) without validation, they recreate classic injection bugs, while AI-generated code carries ordinary weaknesses into production at volume. Fabricated citations, synthetic media, and poisoned answers can likewise enter downstream systems as authoritative artifacts.
Unsanctioned shadow AI tools put many of these paths outside anyone's view.
How AI Security Vulnerabilities Affect Enterprise Organizations
AI security breaches can cause sensitive information exposure, corrupt automated decisions, enable unauthorized system access, interrupt operations, and create legal and reputational consequences. One attack often causes several effects at once because the chain crosses multiple boundaries.
A procurement agent reads supplier emails, checks invoices against a contract database, and can update vendor payment details in the finance system. An attacker sends an invoice PDF with white-text instructions: confirm the attached bank details, then include last quarter's supplier payments in the reply. The agent, holding a service account with write access rather than least privilege access, updates the bank record, replies with internal payment data, and saves a memory note that this supplier prefers the new account. Confidentiality, integrity, and the next payment run fail together, and the note makes the change persist.
Some bad outcomes occur without an attack. A model that confabulates a policy detail or drifts as the business changes produces wrong decisions with no adversary involved. Performance failures call for evaluation and retraining; adversarial compromise calls for containment, credential revocation, and forensics. A confabulated approval and a manipulated one can both move money.
How to Reduce AI Security Vulnerabilities
Effective AI security combines controls at each trust boundary with continuous testing, constrained autonomy, clear ownership, and incident readiness. This approach limits how far a successful compromise can propagate.
Preventive Controls at Trust Boundaries
Each trust boundary requires controls tailored to its layer. The following practices improve prevention across your tech stack:
Data and Model Provenance: Training and retrieval data should come from known origins, with dataset history preserved and integrity verified. Models should use formats that store only numerical tensors rather than executable code, and an AI bill of materials should cover models and datasets alongside ordinary dependencies.
Trusted and Untrusted Content Separation: Systems should mark retrieved documents, email bodies, and tool results as data rather than instructions, a technique called spotlighting, and the system prompt should hold no secrets.
Least Privilege and Scoped Identities: Each agent needs a dedicated identity with least privilege access bounded to its task and an allowlist of tools it may call. Tool servers should use policy-based access control and reviewed tool definitions so a silent change cannot widen what the agent may do.
Output Validation and Approval Gates: Applications should treat model output as untrusted input to whatever consumes it, and a human should approve payments, deletions, and credential changes before execution.
Privilege separation applies these ideas architecturally. In a Dual LLM pattern, a privileged model creates a fixed plan from the user's request, while a quarantined model without tool access handles untrusted content. Injection may compromise the quarantined model, but it cannot alter control flow or reach a tool.
Detective Controls for Models, Agents, and Data
Detection has to cover what the model saw, said, remembered, and did, because compromise may surface only at the last stage. The following controls give teams visibility into each of those layers:
Prompt and Retrieval Logs: Trace an unexpected action back to its source document, while tool-call logs capture the agent identity and parameters used. Together, these records show whether suspicious content influenced a response or propagated into an action.
Memory and Reasoning Baselines: Poisoned memory may produce gradual deviation across sessions without a single suspicious event, so teams can compare an agent's decisions against its own history. Data-drift monitoring on retrieval and training inputs, plus anomaly detection on tool-call volume, provide related signals.
Retrieval Auditing: Logging who queried what, and how much came back verbatim, exposes extraction attempts against RAG stores.
Red Teaming: Testing prompt injection, data poisoning, membership inference, and model extraction across the whole path, rather than the model alone, closes the loop. A useful exercise asks whether a document in the retrieval store can cause a tool call, and whether anyone would notice.
Together, these techniques connect model testing to the downstream permissions and logs that determine an incident's actual impact.
Governance Controls for Ownership, Compliance, and Response
Major frameworks provide a division of labor across the AI governance landscape:
OWASP LLM and Agentic Lists: Identify vulnerability classes specific to language models and agent systems.
NIST AI Risk Management Framework: Organizes governance, mapping, measurement, and management activities across the National Institute of Standards and Technology's four core functions.
MITRE ATLAS: Traces attack paths from initial access to impact, giving teams a threat-informed view of adversary behavior.
ISO AI Management System Standard: Gives auditors an evidence structure aligned with International Organization for Standardization requirements.
Governance starts with a discovered inventory rather than a declared one because agents can deploy without security approval and shadow AI can spread employee by employee. That inventory should record each system's owner, data sources, tools, permissions, and fallback process so reviews can follow the same boundaries used in technical testing. Organizations preparing for high-risk systems under the European Union (EU) AI Act may need to document resilience against poisoning, adversarial inputs, and confidentiality attacks.
Governance most often fails on paper at the response layer. Management needs to pre-assign a named role the authority to contain an agent, revoke its credentials, switch to a fallback process, and preserve logs and memory as evidence. Teams also need to exercise that incident response plan before an incident tests whether it works.
Common Misconceptions About AI Security Vulnerabilities
AI security programs often fail when they rely on assumptions that overlook how models, data, and agents expose risk. The misconceptions below recur across security reviews and shape how teams scope controls, so naming them directly helps close gaps before they become incidents:
Prompt Injection Filters: Filters can reduce model prompt injection, but they cannot replace scoped permissions and approval gates.
Private Deployments: Internal hosting narrows the attacker pool but does not eliminate sensitive information exposure.
Explainability as Security: Interpretable decisions support audits but stop neither poisoned dependencies nor overprivileged tools.
Agents as User Accounts: Agents act continuously with standing credentials, so they require tighter identity and permission controls.
Correcting these assumptions helps teams apply model defenses and conventional application security where each works best.
Building AI Systems That Fail Safely
The strongest 2026 posture treats AI security as an attack-path problem. A poisoned data input can cross retrieval, memory, tools, and authorized actions unless least privilege access and other controls contain it at each boundary. Programs that hold up test those connections end to end and rehearse shutting agents down before an incident forces the question.
Frequently Asked Questions
What Evidence Should an AI Vulnerability Assessment Produce?
An assessment should produce a scoped inventory, reproducible test cases, prompt and retrieval records, tool-call logs, documented impacts, and prioritized remediation steps. Teams also need enough evidence to retest each issue after model or configuration changes.
How Often Should Teams Conduct AI Vulnerability Assessments?
Assessment frequency should reflect how often the model, data, tools, permissions, and surrounding application change. Teams can test after material updates and schedule recurring reviews for systems that handle sensitive data or perform consequential actions.
How Can Teams Test AI Systems with Nondeterministic Outputs?
Teams can repeat the same tests across multiple runs and record both expected behavior and unacceptable outcomes. Testing should examine response ranges, tool calls, memory changes, and downstream effects rather than relying on one exact output.
How Do Model Version Changes Affect Remediation Validation?
A fix that teams validate against one model version may behave differently after an update or fine-tuning cycle. Teams should retest the original exploit, nearby variations, tool permissions, and downstream controls whenever a model or its surrounding configuration changes.