Organisations are adding large language models (LLMs) to customer support, internal search, coding tools and business workflows at remarkable speed. Each of these features introduces risks that traditional web application testing does not fully cover. This guide explains what AI and LLM penetration testing is, the main risks it looks for, how a test is run and how to prepare.
What is AI and LLM penetration testing?
AI and LLM penetration testing is the authorised, adversarial testing of applications built on large language models. Testers try to make the system leak data, ignore its rules, misuse the tools it has access to, or harm other users, and then recommend controls to stop it. The target is not only the model, but the whole application around it: system prompts, retrieval pipelines, plugins and tools, permissions, and the code that handles the model’s output.
Why AI applications need their own testing
Traditional applications follow rules written in code. LLM applications take instructions in natural language, and that changes the threat model:
- Instructions and data are mixed. The model cannot reliably tell the difference between your instructions and text it reads from a document, web page or email. Anyone who can put text in front of the model can try to give it instructions.
- Behaviour is probabilistic. An attack that fails nine times may succeed on the tenth. Testing must be repeated and success rates measured.
- AI agents act. When a model can send emails, query databases or call APIs, a successful manipulation becomes a real action, not just a strange answer.
The OWASP Top 10 for LLM Applications
The OWASP Top 10 for LLM Applications is the most widely used reference for these risks. In summary:
| Risk | What it means in practice |
|---|---|
| Prompt injection | Crafted input, direct or hidden in content the model reads, overrides the system’s instructions. |
| Sensitive information disclosure | The model reveals personal data, secrets or other users’ information. |
| Supply chain | Risks from third-party models, datasets, plugins and libraries. |
| Data and model poisoning | Manipulated training, fine-tuning or retrieval data changes behaviour. |
| Improper output handling | Model output is trusted and passed to browsers or systems, leading to XSS or injection. |
| Excessive agency | The AI has more tools, permissions or autonomy than its task requires. |
| System prompt leakage | Hidden instructions, sometimes containing secrets, are revealed. |
| Vector and embedding weaknesses | Weak access control in RAG lets users retrieve documents they should not see. |
| Misinformation | Confident but false output that users or systems rely on. |
| Unbounded consumption | Abuse that drives up cost or degrades service. |
How an AI penetration test works
- Scoping and architecture review: mapping the model, data sources, tools, permissions and user roles.
- Threat modelling: deciding what an attacker must never achieve, such as reading another customer’s data or triggering a payment.
- Adversarial testing: direct and indirect prompt injection, jailbreaks, data extraction and tool abuse, manually and with automated tooling.
- Application testing: authentication, authorisation, rate limiting and output handling around the AI endpoints.
- Reporting: each finding with reproducible prompts, success rates, impact and layered mitigations.
What good defences look like
Because prompt injection cannot be fully prevented by filters, the strongest controls limit what a successful attack can do:
- Give the AI the minimum data access and tool permissions it needs, and enforce them outside the model.
- Apply the user’s own permissions to every retrieval and tool call, rather than a shared service identity.
- Treat all model output as untrusted: encode it before display and validate it before passing it to other systems.
- Require human confirmation for sensitive actions such as sending messages, changing records or spending money.
- Log prompts, tool calls and outcomes so misuse can be detected and investigated.
How to prepare
- Provide a test environment and accounts for each user role.
- Document the data sources, tools and permissions the AI can use.
- Share system prompts and guardrail configuration if you want a white-box review.
- Check your model provider’s usage and testing policies.
Cenvox provides AI and LLM penetration testing for chatbots, copilots, RAG systems and AI agents. If you are planning an AI launch, talk to us about scope.