Key Insights
According to IBM's 2025 Cost of a Data Breach Report, 13% of organizations reported breaches of AI models or applications, and 97% of those compromised lacked proper AI access controls. Sensitive information disclosure in a large language model (LLM) is one risk that can lead to these breaches, where the model or its surrounding application reveals data a user should never see, whether the content originated in training, retrieval, user input, or a connected tool. The Open Worldwide Application Security Project (OWASP) ranks it second in its 2025 Top 10 for LLM Applications, reflecting how quickly the risk has moved from theory to a production concern.
The paths that lead there are not interchangeable. Memorized training text, permissive retrieval pipelines, cross-session caches, and prompt-injected agents each carry data out through a different mechanism, and each needs its own preventive, detective, and governance controls. This guide maps out how sensitive information disclosure happens, explores real-world incidents, and lays out the testing and ownership practices that keep disclosure risk measurable as an LLM system changes.
Key Takeaways
Sensitive data can escape during training, retrieval, inference, shared serving, and agent actions.
Several high-impact paths run through the application layer, which controls what the model can reach.
No single filter or policy covers every path, so each mechanism needs its own preventive, detective, and governance controls.
Extraction testing, scenario-based red teaming, and named ownership keep disclosure risk measurable after launch.
What Is Sensitive Information Disclosure in LLMs?
Sensitive information disclosure occurs when an LLM or its surrounding application exposes confidential data to an unauthorized user, whether that data originates in training or comes from a user or connected system. OWASP's 2025 definition spans a wider range of data than most privacy reviews assume, including:
Personally Identifiable Information (PII): Names, identifiers, and contact details tied to an individual.
Financial and Health Records: Payment details, account information, and protected health information.
Confidential Business Data: Internal documents, pricing formulas, and strategic plans.
Security Credentials: API keys, database passwords, and other secrets.
Legal Documents: Contracts, filings, and privileged communications.
Proprietary Model Assets: Training methods and source code in closed or foundation models.
A review that only hunts for names and Social Security numbers will miss a leaked database password or pricing formula. Exposure also comes from two directions at once: the model memorizes training data and infers facts it was never told, while the application controls what enters the context window, so an assistant that loads every customer's records into a prompt will eventually answer the wrong customer.
Root causes typically include broken access control that opens a path to information the user lacks permission to see, weak session isolation, missing server-side permission checks, and manipulated prompts that coax the model into revealing more than it should. OWASP's LLM entry names the exposure itself, regardless of what caused it. Controls therefore have to address each of these upstream failures rather than the symptom alone.
How Sensitive Information Disclosure Happens Across the LLM Lifecycle
Sensitive data can escape at multiple points in an LLM system, from training through agent actions.
Training-Data Memorization and Extraction
Models store fragments of their training data and can repeat them verbatim. Memorization increases as models grow and text repeats. More context in prompts also increases memorization. Alignment training does not erase it either. According to a divergence attack study on arXiv, asking a production chatbot to repeat a word such as "poem" made it emit training data at 150 times the normal rate.
System Prompt and Configuration Leakage
Attackers can extract configuration and role information held in system prompts. They can also extract the prompts' rules, often with simple, manually crafted queries. OWASP's example is a banking chatbot whose prompt revealed a transaction limit and a loan total. Defenses that match known attack phrasings leave new techniques open.
Retrieval and Vector-Store Leakage
Retrieval-augmented generation (RAG) can leak sensitive data when it pulls document chunks into the model's context without enforcing the original files' permissions. A user can then receive text from a document they could never open directly.
Vector stores add a second path. A few leaked document-embedding pairs let attackers train models that reconstruct text from vectors, so teams have to protect an embedding store holding a medical-dialogue or email corpus like the corpus itself.
Cross-User and Cross-Session Leakage
Shared deployments can leak data between users when an application mixes up sessions or serving infrastructure shares information. Inference servers often keep a key-value cache, which stores intermediate results for a prompt prefix, and key it on tokens alone. Cache sharing can allow an attacker to reconstruct other users' prompts. Disabling cross-user cache sharing closes the channel but gives up the speed of prefix caching.
Agent and Tool-Mediated Exfiltration
Agents that act on external content can be tricked into leaking data they were never asked to share: an attacker hides instructions in a web page, file, or email that the model processes through indirect prompt injection, and the agent carries them out as if a user had typed them. Data then leaves in the URL of a link to an attacker's server, or through a direct tool call. In one scenario, a prompt-injected agent wrote sensitive data to a public code repository.
Inference-Time Over-Sharing and Attribute Inference
A model can disclose information that no document ever stated. A 2023 privacy study on arXiv found that current LLMs inferred personal attributes from real online forum profiles with up to 85% top-1 accuracy. Those attributes included location and income. They also included sex. Top-1 accuracy means the model's first guess was right. Over-sharing is the everyday version: a calendar assistant asked about availability also lists the attendees.
Types of Sensitive Information at Risk
Each category of data does different damage when an LLM exposes it.
Personal, Financial, Health, and Legal Data
Leaked health or payment data can trigger obligations under the Health Insurance Portability and Accountability Act (HIPAA) and the California Consumer Privacy Act (CCPA). The General Data Protection Regulation (GDPR) may also apply. Permission-aware retrieval and PII detection in outputs can protect these records.
Credentials, Configuration, and System Instructions
Credentials and verbose error messages give attackers access or a roadmap. Leaked system prompts help attackers plan follow-on attacks when they hold API keys or role definitions. Permission structures provide another source of information.
Proprietary Business and Training Data
For providers, training data and methods are intellectual property, and OWASP notes that disclosing training data can enable inversion attacks, which reconstruct private inputs from outputs. Employee pastes into public chatbots are a common route for business data.
Real-World Disclosure Patterns and Their Impact
Public incidents since 2023 follow recurring patterns, and each traces back to a failed safeguard.
Employee Input Exposes Confidential Business Data
This pattern occurs when organizations lack a usage policy and an approved alternative. In March 2023, employees at a large electronics manufacturer pasted semiconductor equipment data and software code into a public chatbot. They also pasted meeting details, and the information may have entered the provider's training data.
As a result, the company banned generative AI tools. Bans are hard to enforce when employees can use personal accounts and browser extensions, so a sanctioned tool with no-training terms plus prompt-level data loss prevention (DLP) fills the gap. Abnormal's AI Governance is designed to help with the visibility side of that gap: it discovers approved and shadow AI tools across an organization and attributes usage to an owner, so a policy ban isn't the only control standing between employees and a public chatbot.
Session Failures Reveal Another User's Information
The March 2023 chatbot bug exposed conversation histories and payment information across accounts because the application failed to separate user contexts. Italy's national data protection authority reported the breach before imposing temporary limits on the provider on broader grounds. Scope caches and session stores per user, and use automated tests that try to read across accounts.
Crafted Prompts Extract Instructions or Training Data
In February 2023, a user extracted the hidden initial instructions of a major search engine's chat assistant. Relying on the model to keep its own instructions confidential failed, and exposed rules give an attacker a map for targeted bypasses. Keep secrets out of prompts and use a separate system to inspect outputs.
Indirect Injection Turns Connected Tools into Exfiltration Paths
In the June 2025 zero-click flaw, hidden instructions in one crafted email got past the vendor's injection classifier and other defenses, and an assistant that could read the victim's files sent internal documents to an attacker's server. The vendor issued emergency patches and reported no exploitation in the wild, though the flaw pointed to design weaknesses shared by retrieval-based assistants.
Least-privilege tool access and user confirmation before data leaves the environment could have limited what the injected instructions could reach; link redaction alone did not stop this attack.
How to Prevent Sensitive Information Disclosure
Prevention works when every exposure path has a control that blocks it, one that detects it, and an owner who governs it. The matrix below pairs each exposure path with controls at all three layers:
Exposure Path | Preventive Control | Detective Control | Governance Control |
|---|---|---|---|
Training memorization | Sanitization, deduplication, differential privacy | Extraction and canary testing | Training-data documentation |
System prompt leakage | Secrets kept out of prompts | Output scanning for prompt fragments | Configuration review |
Retrieval leakage | Permission-aware retrieval, tenant isolation | Retrieval access logs | Approved data-source list |
Cross-user leakage | Session and cache isolation | Cross-account tests | Infrastructure change review |
Agent exfiltration | Least-privilege tools, user confirmation | Tool-call and outbound-link logging | Agent permission audits |
User input and over-sharing | Prompt DLP, data minimization | PII detection in outputs | Acceptable-use and retention policy |
A row with only a preventive control fails silently, since nothing reveals a bypass and no one is accountable for the fix. For example, permission-aware retrieval without access logs or an approved data-source list means a misconfigured connector can expose documents for weeks before anyone notices, and even then it's unclear who should remediate it.
Data Minimization, Sanitization, and De-Identification
OWASP recommends scrubbing or masking sensitive content before training, and deduplication cuts how often a model repeats memorized text. Even masked training data can leak PII when prompts ask a model to recall and rank candidate values.
Access Control, Tenant Isolation, and Least Privilege
Access control has to hold at both training time and runtime. OWASP's guidance on fine-tuning says not to train on information the highest-privileged user can access if lower-privileged users will query the model, and at runtime the server-side session should provide identity rather than a browser-editable field, with retrieval checking permissions on every chunk.
Input and Output Inspection, Redaction, and Data Loss Prevention
Pattern matching and tokenization can catch card numbers, keys, and names in prompts and responses. Named entity recognition can do the same, preferably in a system separate from the model. OWASP's own attack scenario, though, has an attacker slipping past input filters.
Privacy-Preserving Training and Its Trade-Offs
Privacy-preserving training can limit memorization, but each method carries trade-offs. Differential privacy adds calibrated noise during training and can reduce unintended memorization, at the cost of lower model utility and slower training. Federated learning keeps data decentralized. Gradient inversion can still recover training examples, so federated learning is insufficient alone.
Secure Retrieval, Tool Use, and Agent Permissions
A secure setup scopes each tool to the minimum data it needs and asks the user to confirm before anything leaves the organization. It also renders agent Markdown safely so links cannot carry data out. Tool calls and outbound links need logging. Abnormal's AI Agent Security is designed to help at this layer specifically: it maps each agent's privileges and data reach, watches prompts for off-task behavior, and can help trigger automated containment when an agent's activity looks rogue.
User Policies, Transparency, and Retention Controls
Clear terms of use and training opt-outs govern how people and services handle sensitive data as part of broader data governance practices alongside technical controls. Published retention and deletion policies provide additional governance. Free public AI services offer no contractual privacy or security assurances, which makes the consumer-versus-enterprise line a policy decision.
Testing, Monitoring, and Governing Sensitive Information Disclosure Risk
Testing and monitoring keep disclosure risk measurable after launch.
Extraction Testing and Privacy-Focused Red Teams
The NIST Generative AI Profile recommends red team penetration testing for training-data output and model extraction. It also recommends testing for membership inference and warns that memorization creates risk even for data present only in a small number of training samples. Canary testing plants unique marker strings in training data and checks whether prompts can recover them. Membership inference results need care because performance can vary on large pretrained models.
Scenario Tests for Retrieval, Sessions, and Connected Tools
A team can seed two test tenants with marker documents and confirm neither can retrieve the other's content. The team can also run concurrent sessions to look for crossover, then plant an instruction-laden email to see whether an agent attempts an outbound call.
Output Monitoring and Periodic Privacy Review
The same profile calls for periodic monitoring for privacy risks and PII exposure, plus detection of sensitive data in outputs. Teams can implement this guidance by running PII and secret detectors on logged output and feeding confirmed exposures into the same incident response process used for other security events, reviewed on a set schedule.
Standards Mapping and Shared Responsibility
The frameworks below turn disclosure risk into something testable, and each divides responsibility differently across model providers, application developers, and deploying organizations:
OWASP Top 10 for LLMs: Its sensitive information disclosure entry describes what gets exposed. The Prompt Injection entry describes how an attacker changes model behavior, and System Prompt Leakage is the narrower case in which an attacker exposes the prompt itself.
MITRE ATLAS: The framework catalogs LLM data leakage as an attack technique.
ISO AI Management Standard: The standard offers a certifiable AI management system.
Shared Responsibility Models: The Coalition for Secure AI and Cloud Security Alliance frameworks make application developers accountable for integration security and data connections. They give model providers a supporting role for plugins and safety systems and place prompt filtering and application security with the customer in software-as-a-service (SaaS) deployments.
Privacy Regulators:European Data Protection Board Opinion treats a model as anonymous only when extracting personal data through queries is very unlikely, so extraction testing doubles as GDPR evidence.
Mapping controls to these frameworks turns audit questions into concrete tests and clarifies which team owns each gap when disclosure risk shows up in review.
Keep Sensitive Information Disclosure Measurable as Systems Change
A useful evaluation traces each sensitive dataset an assistant touches and identifies the mechanism that could expose it. It then confirms that the path has a preventive control, a detective control, and a named owner. Repeating that trace whenever an assistant gains a connector, data source, or agent permission keeps disclosure controls aligned with the system as it changes.
As LLMs extend into agents and connected tools, that same tracing exercise increasingly has to cover AI activity itself. Abnormal's AI Security is designed to apply the same behavioral approach used in email security to AI tool adoption, employee AI requests, and agent activity, helping security teams see where sensitive data could leave through an AI system. For instance, AI Employee Guardrails can connect requests to identity and apply data policies at the moment of the prompt, before sensitive content leaves the organization.
Book a demo to see how Abnormal fits alongside an existing disclosure-risk review.
Frequently Asked Questions
Does Deleting a Source Document Stop Future Disclosure?
Deleting a document from a retrieval index can prevent the application from retrieving it in future sessions. It does not erase text that a model already memorized during training, so the response depends on where the information entered the system.
Should Monitoring Logs Store Full Prompts and Responses?
Only when the monitoring purpose requires them. Full prompts and responses can contain the same sensitive data the controls are meant to protect, so access limits, data minimization, and retention rules should also apply to logs.
Does Disabling Shared Prompt Caching Affect Performance?
Yes. Isolating or disabling cross-user prefix caches removes a potential path between users, but it also gives up some of the speed gained by reusing stored prompt calculations.
What Should User Confirmation Cover Before an Agent Sends Data?
A useful confirmation should make the destination and the information the agent will send clear before the action occurs. This gives the user a chance to stop an unexpected outbound transfer without blocking ordinary internal tool use.
When Should Disclosure Tests Run Again?
Teams should repeat testing whenever the assistant gains a new connector, data source, tool, cache configuration, or agent permission. Each change can create an exposure path that earlier tests did not cover.