AI & Emerging Technology Security
AI and LLM penetration testing
Testing of chatbots, copilots, RAG systems and AI agents for prompt injection, data leakage and unsafe tool use, plus the application around the model.
Key areas covered
- Coverage
- Chatbots, copilots, RAG systems, AI agents with tool access, and AI features inside existing apps
- Approach
- Manual adversarial testing of prompts, data sources and tools, supported by automated prompt-attack tooling
- Reference standards
- OWASP Top 10 for LLM Applications, MITRE ATLAS, NIST AI RMF
- You receive
- Executive summary, findings with reproducible prompts and success rates, debrief and optional retest
Overview
About AI and LLM penetration testing
Chatbots, copilots, retrieval-augmented generation (RAG) systems and AI agents can read company documents, call internal APIs and take actions on a user’s behalf. That makes them a new kind of attack surface. A carefully worded message, or a poisoned document the model reads, can persuade it to ignore its instructions, reveal data it should not, or misuse the tools it has been given.
Our AI and LLM penetration testing attacks your AI system the way a real adversary would. We test the model’s behaviour and the application built around it: prompts, retrieval pipeline, plugins and tools, permissions, and the way outputs are used. It extends our web application and API testing to AI-powered features, and is mapped to the OWASP Top 10 for LLM Applications.
A typical starting point
You are launching a chatbot, AI assistant or agent that can see company data or take actions, and need to know it cannot be manipulated.
The security problem
How AI and LLM applications are attacked
Direct and indirect prompt injection
Instructions hidden in user input, web pages or documents that override the system prompt.
Data leakage through the model
The assistant reveals system prompts, other users’ data or documents the user should not see.
Excessive agency
Tools and plugins that let the model take actions without the right checks or user confirmation.
Weak controls around the model
Retrieval systems without access control, and unvalidated model output passed to other systems.
What’s covered
What we test in an AI or LLM application
Area 01
Prompt injection and jailbreaks
- Direct prompt injection and attempts to override system instructions
- Indirect injection through documents, web pages, emails, tickets or any data the model reads
- Jailbreak techniques against guardrails, content filters and policy layers
- Multi-turn and role-play manipulation
Area 02
Sensitive information disclosure
- Leakage of system prompts, secrets, API keys and configuration
- Retrieval of documents or records the user is not entitled to through RAG
- Cross-user and cross-tenant data leakage in conversation history and memory
- Personal data exposure relevant to India’s DPDP Act
Area 03
Agents, tools and excessive agency
- Tools and permissions beyond what the task needs
- Abuse of plugins and function calls to reach internal systems (SSRF, injection)
- Actions taken without appropriate user confirmation
- Chained attacks across multiple agents or tools
Area 04
Output handling and the surrounding app
- Model output rendered unsafely in the browser or passed to other systems (XSS, SQL and command injection)
- Authentication, authorisation and rate limiting of AI endpoints
- Unbounded consumption: cost and denial-of-wallet abuse
- Vector store and embedding access controls
Good to know
There is no single fix for prompt injection
Prompt injection cannot currently be fully prevented by filtering alone. The most effective defence is limiting what the AI can access and do, isolating untrusted content, and requiring confirmation for sensitive actions. Our recommendations focus on those layered controls, not only on better prompts.
Good to know
We test your AI application, not the provider’s model
We test the AI system you have built and how you use the model. Testing of a provider’s underlying model or platform is governed by that provider’s own policies, which we respect. Read more in our guide to AI and LLM penetration testing.
Our approach
How AI penetration testing is carried out
Step 1: Scoping and architecture review
We map the model, system prompts, data sources, tools, permissions and user roles, and agree what an attacker should never be able to achieve.
Step 2: Threat modelling
Using MITRE ATLAS and the OWASP LLM Top 10, we identify what an attacker could gain: data, actions, money or influence over other users.
Step 3: Adversarial testing
We attempt direct and indirect prompt injection, jailbreaks, data extraction and tool abuse, manually and with automated prompt-attack tooling.
Step 4: Application and API testing
We test the surrounding application and APIs: authentication, authorisation, output handling, rate limits and the vector store.
Step 5: Reporting and debrief
Findings include reproducible prompts, success rates, business impact and layered mitigations, walked through with your AI and engineering teams.
Step 6: Retesting (optional)
We re-run the attacks after changes to prompts, guardrails, permissions or the model itself.
Standards and references we work to
- OWASP Top 10 for LLM Applications
- Prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.
- MITRE ATLAS
- A knowledge base of adversary tactics and techniques against AI systems.
- NIST AI RMF
- NIST’s AI Risk Management Framework, used to frame findings for risk owners.
- OWASP API Security Top 10
- Used for findings in the APIs that serve and connect the AI system.
What you receive
What you receive from an AI penetration test
Executive summary
What an attacker could achieve through your AI system, in plain language.
Technical findings
Each issue with its OWASP LLM category, reproducible prompts, success rate, impact and mitigations.
Architecture recommendations
Permission, isolation and confirmation controls that reduce risk even when prompts are bypassed.
Test case library
The attack prompts used, so you can add them to your own regression tests.
Debrief
A working session with your AI and engineering teams.
Retest report
If in scope, results of re-running the attacks after your changes.
Sample finding
Hidden instructions in an uploaded document control the assistant
Customer support assistant: document upload and email tool
OWASP Top 10 for LLM Applications (2025): LLM01 Prompt Injection, LLM06 Excessive Agency
- What we found
- Instructions hidden in an uploaded document are followed by the assistant, including sending email through its connected tool.
- Evidence
- We uploaded a test PDF with hidden text telling the assistant to email a conversation summary to an external test address. It was sent without user confirmation. Prompts, responses and logs attached.
- Business impact
- A single document could be used to leak conversation or customer data, or to trigger actions in connected systems.
- Fix
- Treat retrieved content as untrusted data, require user confirmation for email and other actions, limit tool permissions and filter sensitive data from outputs.
- Status
- OpenFixed, verified at retest
Before we start
What we need to test your AI application
- Access to a test environment of the AI application, with test accounts for each user role.
- A description of the data sources, tools and permissions the AI can use.
- System prompts and guardrail configuration, if you want a white-box review.
- Test data that is safe to expose during testing.
- Confirmation of your model provider’s usage and testing policies.
Not sure you have everything?
We confirm exactly what AI and LLM penetration testing needs from your side during the scoping call, before any work is agreed.
Book a scoping callFAQ
Questions about AI and LLM penetration testing
Still have a question?
Ask about scope, timing or cost for AI and LLM penetration testing, even if you are not ready to commission work yet.
Contact our teamWhat is AI or LLM penetration testing?
It is security testing of applications built on large language models. Testers try to make the AI system leak data, ignore its rules, misuse its tools or harm other users, using techniques such as prompt injection and jailbreaks, and then recommend controls to stop it.
Is this the same as AI red teaming?
The terms overlap. AI red teaming often refers to testing a model for harmful or unsafe outputs. Our testing focuses on security: what an attacker can achieve through your AI application, including data access and actions, while also reporting unsafe behaviour we observe.
Can you test an application built on OpenAI, Anthropic, Google or open-source models?
Yes. Our testing works with any model provider or self-hosted model, within the provider’s usage policies.
Will testing generate harmful content?
Testing involves trying to make the system misbehave, which can produce content it should not. This happens only in your test environment, and evidence is handled confidentially.
How often should AI applications be tested?
Before launch, and whenever you change the model, the system prompt, the data sources or the tools the AI can use. Small changes to any of these can change how the system behaves.
Related services
Often combined with AI and LLM penetration testing
Talk to us about AI and LLM penetration testing
Tell us what has prompted the conversation. We will reply to arrange a short, no-obligation scoping call for AI and LLM penetration testing.