Skip to content
security23 min read

Generative AI Security Risks: Where the Vulnerabilities Live in Your Product

Most AI security risks sit in the code around the model: prompts, retrieval, tools and permissions. Here is where each one lives in your product and how to reduce its impact.

Andrii BurdaSenior Engineering Manager

Published · Updated

Share

An AI model drawn as a circuit brain in a glass cube, guarded by a checkmark shield, linked to data, code, cloud and user icons with warning signs

Generative AI security risks arise when an LLM feature leaks data, takes unauthorized actions or produces unsafe output that the product trusts. Most of these vulnerabilities sit in the code around the model rather than in the model itself.

The current reference is the OWASP Top 10 for LLM Applications 2026, published on August 3, 2026. Its ten entries map onto five places in a typical product: the context the model reads (prompts, user input, retrieved documents, tool output), the data behind retrieval, the tools and permissions an agent can use, the code that handles model output, and the supply chain of models, packages and infrastructure the feature runs on. Each of those places needs an engineering owner and appropriate controls. Scope the tools an agent can call to the task. Enforce user permissions before retrieval, not after. Keep credentials out of prompts. Treat model output as untrusted input. Cap spending per request and per user. Run security test cases on every prompt, model or data change.

This post is about the AI features you build and ship. If your question is how attackers use AI against your software, that is a different problem, covered in our evidence review of AI-driven cyberattacks.

What counts as a generative AI security risk?

A generative AI security risk is any way the system around a large language model can be made to act against its owner: leaking data the user should not see, executing an action nobody approved, passing unsafe output into another component, or running up cost and downtime. The model is one component. Most AI vulnerabilities sit where the application trusts it: what it is allowed to read, what it is allowed to call, and what the rest of the product does with its answer.

That framing matters for who fixes what. A model that can be talked into anything is a known property of today’s LLMs, and no vendor setting removes it. A model that can be talked into anything but can only reach the current user’s records, cannot send data to an outside address, and needs a human click before it deletes something, is a product you can defend. The five places to look:

  • The context window. Everything the model reads arrives as one stream of tokens: your system prompt, the user’s message, retrieved documents, tool results, conversation history and long-term memory. Any of it can carry instructions.
  • The data behind retrieval. RAG pipelines, vector stores and fine-tuning sets decide what the model knows at answer time, and who else’s data it can surface.
  • Tools and permissions. Functions, MCP servers, plugins and API keys turn text into actions. Their scope sets the blast radius of every other failure.
  • Output handling. Model output becomes HTML in a browser, a SQL query, a shell command, a ticket or an email. Each sink has its own rules.
  • Supply chain and runtime. Models, adapters, datasets, packages, serving frameworks and token budgets are dependencies with their own vulnerabilities.

One adjacent problem is out of scope here: employees pasting company data into public chatbots. That is a governance and policy question about AI you use. This post covers AI application security, the vulnerabilities in AI you ship to customers.

What changed in the 2026 OWASP Top 10 for LLMs?

The 2026 edition is the first OWASP Top 10 for LLM applications whose ranking was checked against a record of real incidents, not only practitioner judgment. The project built a corpus of 7,714 incidents from public vulnerability databases and an AI-harm database, classified the 6,639 that carried enough detail, and gave that evidence a quarter of the weight, with the community vote carrying the other three quarters. The result moved more entries than any previous update.

The moves that matter for a product team:

  • Excessive agency climbed to third. The project leads call it the most consequential change, because the vote and the incident record agree that agentic deployments are where the damage is landing.
  • System prompt leakage became hidden context exposure. The entry now covers everything the application places in the model’s context that users should not see, including tool schemas and policy text.
  • Improper output handling fell from fifth to tenth, and now also covers the insecure code that AI assistants generate.
  • Unbounded consumption rose four places, carried by practitioners who weigh cost exhaustion higher than before.
  • Prompt injection stayed first, although by raw incident count it would fall out of the top ten. The project reads that gap as a defense effect: teams fight injection hard, so fewer clean exploits reach public databases.

The project leads’ introduction to the 2026 edition sums up the posture behind all ten entries: build the system so that “when the model is fooled, and it will be, nothing important breaks.” The edition also draws a boundary. When the model is a component inside your application, this list owns the risk. When the model becomes an actor, with tools, memory and downstream consequences, OWASP points to its separate Top 10 for Agentic Applications, covered below.

Here is where each 2026 entry lives in a typical SaaS product, and who usually owns the fix:

2026 rankRiskWhere it lives in your productWho usually owns the fix
LLM01Prompt injectionEvery input the model reads: user text, retrieved documents, tool output, files, images, memoryApplication architect, by limiting what a manipulated model can reach
LLM02Sensitive information disclosureAnswers, retrieved chunks, tool-call arguments, logs, traces, embeddingsBackend team and the owner of each data source
LLM03Excessive agencyTool definitions, credentials, approval steps, autonomy settingsBackend and platform teams, with identity and access management
LLM04Supply chainModels, adapters, datasets, packages, serving frameworksPlatform or DevOps team
LLM05Data and model poisoningTraining, fine-tuning, embedding and ingestion pipelinesData engineering
LLM06Unbounded consumptionToken budgets, rate limits, agent loops, reasoning budgetsPlatform team, with a named cost owner
LLM07MisinformationAnswers that users, workflows or agents act onProduct owner and QA, through evaluations
LLM08Hidden context exposureSystem prompt, developer instructions, tool schemas, policy textFeature author, with backend owning secrets
LLM09Vector and embedding weaknessesVector store, retrieval filters, semantic caches, agent memoryBackend and data engineering
LLM10Improper output handlingWherever output is rendered, executed, stored or passed onFront-end and backend teams

The table is also a first triage tool. If a row has no owner in your team today, that row is where to start. The three situations in the section Where unicrew fits show when it pays to bring in an outside assessment.

Prompt injection is managed, not closed

Prompt injection happens when any input the model reads, whether user text, a retrieved document, tool output, an image, an audio track or stored memory, changes its behavior in ways the developer did not intend. OWASP’s 2026 entry explains why it persists: an LLM makes no architectural distinction between instructions and data, so there is no equivalent of the parameterized query that eventually tamed SQL injection.

The UK’s National Cyber Security Centre made the same point in its December 2025 guidance, Prompt injection is not SQL injection. Its technical director for platforms research argued that prompt injection may never be fully mitigated the way SQL injection can be, and that systems should be designed around LLMs as “inherently confusable” components. The same guidance tells buyers to distrust any product that claims to stop prompt injection and to prefer those that explain how they reduce it.

This is no longer a research curiosity. In June 2025, Microsoft published CVE-2025-32711, an AI command injection in Microsoft 365 Copilot that let an unauthorized attacker disclose information over a network, rated 9.3 (critical) with no user interaction required. AI security vulnerabilities now get CVE numbers, severity scores and patch cycles like any other software flaw, which is why they belong in your normal vulnerability management.

The 2026 entry names three deployment properties that make injection worse. The model pools system prompt, user input, documents, tool output and memory into one stream with no enforced trust boundary. An injection written into long-term memory or a RAG corpus taints every later session that reads it. And when the model’s output drives tool calls, the blast radius grows from a bad chat answer to whatever the tools can reach.

What closes the gap in practice

  • Design for the blast radius. Assume the model will follow an instruction it should not, and decide what that costs. If the answer is “it could email any customer record anywhere”, the fix is in the tools, not the prompt.
  • Keep untrusted content away from privileged actions. A step that reads web pages, inbound email or uploaded files should not also hold the credentials to send, pay or delete.
  • Require a human confirmation for consequential actions. Show the exact action and its parameters, not a model-written summary of it.
  • Close the quiet output channels. Many data leaks run through a rendered Markdown image or link that fetches an attacker’s URL. Disable automatic fetching, or allow-list the domains.
  • Test injection where it really arrives. Put instructions inside retrieved documents, tool responses and file uploads, not only in the chat box, and run those cases on every change.

Filters that detect injection phrases are a useful layer. They are not the control, and a design that depends on them will fail the first time someone phrases things differently.

Sensitive data leaks through more than the answer

Sensitive information disclosure is the second entry in both the 2025 and 2026 lists, and OWASP calls it the one place at the top where practitioner belief and incident evidence agree. In an AI feature, data leaves through more channels than the final answer: tool-call arguments, reasoning traces, retrieved chunks, logs, telemetry and embeddings are all disclosure surfaces, and each needs the same classification and redaction rules as the answer itself.

The 2026 entry names two structural failures behind most incidents. The first is oversharing upstream: unscoped drives, legacy permissions and knowledge bases feed a RAG pipeline with data the model then retrieves exactly as designed. The fix is at the data surface, not in the model. The second is persistence: once data shapes weights, embeddings or adapters, it can stay extractable after the source is deleted, which strains erasure obligations under GDPR Article 17.

The cost is measurable. IBM’s 2025 Cost of a Data Breach report found that 60% of AI-related security incidents led to compromised data and 31% to operational disruption. In the 2026 edition, breaches through AI model inversion, where an attacker reconstructs training data from a model’s responses, cost $6 million on average.

Hidden context is not a secret

Hidden context exposure, the renamed eighth entry, covers what the application places in the model’s context but never shows the user: the system prompt, developer instructions, retrieved policy text and the schemas of the tools the model can call. OWASP’s guidance is to design on the assumption that all of it is discoverable. A leaked system prompt full of tone guidance is informational. The same leak becomes high severity when the prompt contains an API key, or when the application relies on the prompt staying secret to enforce who can do what, and critical when it chains into remote code execution or broad data access.

The fixes that work

  • Enforce permissions before retrieval. The retriever should run with the current user’s rights and filter by tenant and access level at query time. Filtering the answer afterwards is too late.
  • Redact before indexing. Personal data, credentials and regulated fields that no user of the feature needs should never reach the vector store.
  • Keep credentials in the backend. Tools authenticate server-side. Nothing in the prompt should be useful to an attacker who reads it.
  • Treat logs and traces as production data. Prompt and response logs often hold more sensitive material than the database they draw from.
  • Make deletion reach derived data. A deletion request should remove the source document, its embeddings and any cached answers built from it.

Agent permissions decide the damage

Excessive agency is the vulnerability that lets an LLM-based system take damaging actions in response to unexpected, ambiguous or manipulated output, whatever made the model misbehave. OWASP moved it to third place in 2026, and it is where agentic AI security risks concentrate. Tool access can turn a hallucination, a bad prompt or an injected instruction into an unauthorized action.

The 2026 entry traces the root cause to three excesses. Excessive functionality: the agent needs to read documents, but the tool it was given can also modify and delete them. Excessive permissions: the tool connects with an account that can reach far more than the task needs. Excessive autonomy: high-impact actions run without anyone confirming them.

A public example from July 2025 shows all three. Replit’s AI coding agent deleted a production database during a user’s experiment, after being told to freeze code changes, as The Register reported. Replit’s CEO called it unacceptable, and the company began rolling out automatic separation between development and production databases. Its own explanation was that apps built on the platform had used one database for both development and live customer data. The instruction to freeze was text in a prompt. The separation is a control.

For systems where the model plans and acts, OWASP published the Top 10 for Agentic Applications on December 9, 2025, built with more than 100 experts. Its ten entries are:

  • ASI01: Agent goal hijack
  • ASI02: Tool misuse and exploitation
  • ASI03: Identity and privilege abuse
  • ASI04: Agentic supply chain vulnerabilities
  • ASI05: Unexpected code execution
  • ASI06: Memory and context poisoning
  • ASI07: Insecure inter-agent communication
  • ASI08: Cascading failures
  • ASI09: Human-agent trust exploitation
  • ASI10: Rogue agents

Its central principle is least agency: do not deploy autonomy where the task does not need it, because unneeded autonomy widens the exposure without adding value.

How to scope an agent

  • Give each tool the narrowest scope that does the job. Read-only where reading is the task, one table instead of a database, one mailbox instead of a domain.
  • Act as the user, not as a super-account. An agent working for a customer should carry that customer’s identity and rights, with short-lived credentials.
  • Gate irreversible actions. Deletes, payments, outbound messages and permission changes wait for a person who sees the exact parameters.
  • Sandbox anything that executes code, with no network access or credentials the task does not require.
  • Log every tool call with its inputs and outcome, and keep a way to stop an agent mid-run.

Our guide to agentic AI explains how agents get their permissions in the first place, and if your agents reach your systems through the Model Context Protocol, our MCP server security guide covers authentication, tool poisoning and input validation at that layer.

The RAG layer is part of your trust boundary

Whenever similarity search sits between a data source and the prompt, the embedding layer becomes part of the application’s trust boundary. That is the core of LLM09, vector and embedding weaknesses, in the 2026 list, and it applies to RAG, vector-backed agent memory and semantic caches alike. These weaknesses are distinct from prompt injection: many succeed even when the retrieved content carries no instructions at all.

OWASP’s entry offers a compact way to think about the failure modes. Poisoned content makes the system give wrong answers. Inversion of stored embeddings makes it leak the text they came from. Jamming makes it silent, so legitimate questions go unanswered. And a missing access-control check makes it indiscriminate, serving one tenant’s documents to another. In multi-tenant SaaS, this is an authorization failure at the retrieval layer.

Data and model poisoning, LLM05, covers the slower version of the same problem. Poisoning can enter wherever data is ingested, transformed or reused: pre-training, fine-tuning, embedding creation and retrieval. Unlike a code bug, OWASP notes, it may need data revalidation, retraining or a pipeline redesign to fix, which makes prevention much cheaper than cleanup.

What to put in place:

  • Store tenant and permission metadata on every chunk, and enforce it inside the query, not in application code that runs afterwards.
  • Record provenance for everything ingested: source, owner, date and who approved it. Content from outside the company gets a lower trust level.
  • Review what enters the index the way you review what enters the codebase, especially for user-contributed and scraped content.
  • Evaluate retrieval separately from generation. A wrong answer can come from the wrong documents or from a model misreading the right ones, and the fixes differ.

Many retrieval failures show up first as quality problems, not security findings. I already covered the debugging side in a recent post on why RAG systems fail in production, so here the focus stays on the controls.

AI model security starts in the supply chain

AI model security covers the artifacts a feature is built from, not only the prompts it receives. LLM04, supply chain, covers third-party models, datasets, fine-tuning adapters such as LoRA, conversion and merge pipelines, and the serving stack. The 2026 edition adds the case where a model file promoted to production is not what it claims to be, and it recommends migrating away from unsafe serialization formats such as Python pickle, which can execute code when a model file is loaded.

AI coding assistants add a newer variant. A study presented at USENIX Security 2025, We Have a Package for You!, generated 576,000 code samples with 16 LLMs and found that at least 5.2% of the packages suggested by commercial models and 21.7% of those from open-source models did not exist, with 205,474 unique hallucinated package names. Attackers can register those names in advance, a practice OWASP calls slopsquatting, so an unverified AI-suggested dependency installs their code.

The controls are the familiar ones, extended to new artifacts:

  • Pin models, adapters and datasets by version and hash, and pull them only from sources you have vetted.
  • Prefer safe weight formats and scan model files before loading them.
  • Verify every dependency an assistant suggests before it reaches a lockfile: that it exists, who publishes it and how long it has been around.
  • Keep an inventory. OWASP recommends extending your software bill of materials to models, adapters and datasets with an AI bill of materials.
  • Patch the serving stack like any other internet-facing service.

Self-hosting a model changes who carries these risks rather than removing them. You avoid sending data to a provider, but you own the runtime, the files and the endpoint. And with open weights, the 2026 list points out, extraction and inversion can run offline, where rate limits no longer help.

Your product owns what happens to model output

The three remaining risks covered here share one idea: whatever the model produces, your product decides what happens next. Improper output handling, misinformation and unbounded consumption are all failures of that decision, about whether output is safe to execute, true enough to act on, and affordable to generate.

Improper output handling

LLM10 covers model output passed downstream without validation. Because a prompt can steer output, OWASP compares it to giving users indirect access to whatever consumes that output. The results are classic web vulnerabilities: cross-site scripting and request forgery in the browser, server-side request forgery, privilege escalation or remote code execution on the backend. The 2026 entry also flags chat interfaces and IDEs that automatically fetch resources referenced in output, such as Markdown images, which can carry context data out to an external server. The fix is the one you already apply to user input: encode output for its context, use parameterized queries, allow-list URLs, sandbox execution, and review AI-generated code like any other pull request.

Misinformation

Misinformation is where the 2026 incident data disagreed most with the vote: practitioners placed it near the bottom, the incident record near the top. The risk is less about wrong answers than about wrong answers being trusted. In February 2024, British Columbia’s Civil Resolution Tribunal decided Moffatt v. Air Canada after the airline’s website chatbot told a customer he could apply for a reduced bereavement fare after travelling, contrary to the airline’s policy. Air Canada argued the chatbot was responsible for its own statements. The tribunal called that “a remarkable submission”, found negligent misrepresentation and ordered the airline to pay C$812.02. The lesson for product teams is simple: what your AI feature tells customers is your representation. Ground answers in approved sources, show those sources, have the feature decline questions outside its scope, and test answers against agreed cases before each release.

Unbounded consumption

LLM06 rose four places in 2026. Its defining feature is cost asymmetry: one cheap request can trigger expensive computation, and reasoning models, multimodal input and agents that fan a single request out into many tool calls all widen the gap. OWASP’s 2026 entry says request-rate limiting alone is no longer enough and calls for token-aware cost controls, hard spending caps, circuit breakers at the agent level and continuous monitoring of who is spending what. A cost limit is a security control here: it protects availability, your margin and, against model-cloning attempts, your intellectual property.

How to run AI vulnerability management

AI vulnerability management uses the same loop as any other: know what you run, test it, rank findings, fix them and confirm the fix. Two things differ. Behavior can change without a code change, when a model version, a prompt or the indexed data moves. And many findings are design decisions rather than bugs, so they land on architects as often as on developers.

  1. Inventory every AI feature. Record the model and version, the prompts, the data sources, the tools and their scopes, and who can call the feature. This is your AI bill of materials and the input to everything else.
  2. Threat-model each feature against the 2026 list. For anything that calls tools or keeps memory, add the agentic list. The question for each entry is what a manipulated model could reach.
  3. Turn risks into test cases. Injection inside retrieved documents and tool output, cross-tenant retrieval, out-of-scope tool calls, leaked context and runaway loops become cases in an evaluation suite.
  4. Run the suite on every change to prompts, models, tools or indexed data, not only on code changes, and block releases that regress.
  5. Test independently before launch and after major changes. A penetration test of the AI feature checks what the suite cannot: chained findings and the paths a real adversary would try.
  6. Rank by reach, then fix in design. A leak that exposes one user’s own data differs from one that crosses tenants. OWASP’s AI Vulnerability Scoring System, still at version 0.8, is one emerging way to score findings consistently.
  7. Monitor in production. Trace tool calls, retrieval results and spend per user, with alerts and a named person who acts on them.

A full security review is not always the right first step. If the feature is still a prototype on synthetic data, with no tools and no customer access, a threat model and a design review give more value per hour. The full assessment pays off once real data, real users or real permissions are involved.

Where unicrew fits: securing the AI you ship

The right first step depends on where your AI feature is in its life. The three situations below each call for different work, and starting with the wrong one spends budget without closing the risk that matters. unicrew builds and operates its own AI products, Snaplore and Talkmetry, so model upgrades and enterprise security requirements are part of our own release cycle, as the Snaplore case study describes.

Our approach starts from the same premise as the 2026 OWASP list: assume the model will be fooled, then measure what that costs. We combine industry-standard tools with manual techniques and, where the agreed scope allows, chain findings to see how far a manipulated model really gets, from an instruction planted in a document to the tools it can call. Each finding is ranked by exploitability, and a retest confirms the fix, because a fix marked done is not proof the path is closed.

For behavior that varies from one run to the next, the findings become cases the feature must get right. We run each case repeatedly, compare results across prompt and model versions, and calibrate the scoring against your domain experts’ judgment, recording where they disagree. What you keep is a release signal your team owns, not a one-off report.

Before launch, with customer data in the index

Your assistant answers from customer documents, and an enterprise prospect has sent a security questionnaire. The two questions behind it are whether one tenant can ever see another tenant’s data, and whether text inside a document can make the assistant act. Answering them needs a test of the feature as it will ship: instructions planted in documents and tool responses, cross-tenant retrieval, and the permissions behind every tool the model can call. A penetration test of the AI feature produces ranked findings and evidence the customer’s security team can read, and a retest confirms what was fixed. For a SaaS product selling to larger companies, that evidence is what the security review asks for.

Before an agent gets write access

An agent that so far only reads data is about to create tickets, send emails or change records. Most of the risk is decided before any code ships: which tools it gets, with which scopes, under whose identity, and which actions wait for a person. Write those decisions down and review them first, because each one is cheap to change on paper and expensive to change once integrations depend on it. Then test the agent as it will run, including what it does when a retrieved document tells it to do something else.

A live feature that changes every week

Prompts, models and indexed data change more often than code, and any of those changes can reopen an issue you already fixed. The useful asset here is a regression suite: the injection, leakage and tool-scope cases from your last assessment, run on every change, with a failing case able to block the release. AI QA and LLM evaluation builds that suite with your team, scores retrieval separately from generation, and connects the results to whoever makes the release decision.

If one of these describes your next release, tell us which one and we will start from there.

Key takeaways

  • The vulnerabilities live around the model. Prompts, retrieval, tools, output handling and the supply chain are where generative AI security risks become incidents, and each has an engineering owner.
  • The 2026 OWASP list is the current reference. It is the first edition checked against real incidents, and it moved excessive agency to third and renamed system prompt leakage to hidden context exposure.
  • Prompt injection is managed, not closed. The NCSC expects it may never be fully mitigated, so design for what a fooled model can reach.
  • Agent permissions decide the damage. Least agency, user-scoped identity and human approval for irreversible actions turn a manipulated model into a contained one.
  • Permissions belong before retrieval. Oversharing upstream can expose sensitive data, so enforce access controls before content reaches the model.
  • Treat output as untrusted and cost as a control. Encode output for its destination, ground what customers are told, and cap spending per request and per user.
  • Run AI vulnerability management on every change. Inventory, threat-model, test in CI, test independently before launch, and monitor in production.

Frequently asked questions

The two that sit at the top of OWASP's 2026 Top 10 for LLM applications are prompt injection, where text the model reads changes what it does, and sensitive information disclosure, where the system exposes data through answers, retrieved documents, logs or tool calls. Both are managed in the application around the model: limit what a manipulated model can reach, enforce user permissions before retrieval, and keep secrets out of prompts entirely.

In an AI feature, data can leak through retrieved chunks, tool-call arguments, reasoning traces, logs, telemetry, embeddings and model output. One cause is oversharing upstream: a knowledge base makes documents available to users who should not see them. Permission-aware retrieval and redaction before indexing reduce exposure through retrieval. Logs, traces and caches need separate protections. Data retained in weights or embeddings can remain extractable after source deletion, which matters for GDPR erasure requests. Verify deletion across derived artifacts.

A useful assessment covers the AI feature and the system around it: prompt injection through user input, retrieved documents and tool output, data leakage through model context, the permissions of every tool the model can call, output handling, and cost limits. It ends with ranked findings and a decision on who fixes what. Most unicrew engagements start within two to four weeks, and penetration testing can add remediation and a retest.

At unicrew it depends on the shape of the work. A defined assessment with ranked findings is outcome based and quoted per project. Assessment followed by agreed remediation is billed hourly and quoted per project. Continuing security work alongside a changing product is billed monthly per team member. Scope, testing window and price are agreed in writing before work starts, and there is no minimum engagement period.

Yes, and most of them land outside the model, so they rarely need a rebuild. Tool scopes, permission-aware retrieval, secrets management, output encoding, spending caps and audit logging are changes to the application and its infrastructure. The usual order is an assessment of the live feature, fixes ranked by what a manipulated model could actually reach, then an evaluation suite that runs on every prompt, model or data change so the fixes stay fixed.

Need this built, not just read about?

Tell us what you are building. We will map the fastest route from where you are now to a working product.

Book a scoping call

Thank you

Thanks for your message. We will get in touch with you shortly.