An AI provider security assessment should test the AI layer directly. Map inputs, retrieval, tools and outputs to the OWASP Top 10 for LLM Applications, run sandbox tests for indirect prompt injection, permission leaks and tool abuse, check red-team scope and model version, then apply agreed severity rules.
Your AI provider passed the security questionnaire. Then a supplier invoice carrying hidden instructions makes the assistant forward your pricing file elsewhere.
Checking an AI product with a standard questionnaire is like inspecting an office for good locks while missing a letterbox that opens into the safe. Everything checked is real, but the risk sits elsewhere. This guide shows how to map the AI attack surface, run six sandbox tests, read an AI penetration test report and set acceptance rules your team can defend.
Standard reviews miss the AI layer
Most security reviews check encryption, access control, incident response and an independent report such as SOC 2 or an ISO/IEC 27001 certificate. Those controls still matter. A large language model, however, takes instructions from the same channel as data. Anything it reads, whether a prompt, a document or a web page, can try to change what it does.
This weakness is called prompt injection: hidden text that manipulates the model. IBM's 2025 research found that 13% of surveyed organisations had suffered an attack affecting their AI models or applications. Of breached organisations with an AI-related incident, 97% said they lacked proper AI access controls.
Independent reports rarely settle the question. Their scope usually covers the hosting platform and corporate controls, not the model, the retrieval pipeline or the tools an assistant can call. The assessment therefore has to examine the AI layer directly.
Map the attack surface in six layers
The OWASP Top 10 for LLM Applications 2025 gives security teams a shared vocabulary. The table maps each layer to its main failure and the evidence worth requesting.
| Layer | What can go wrong | OWASP 2025 entry | Evidence to request |
|---|---|---|---|
| Inputs and context | Hidden instructions in files, emails or web pages | LLM01 Prompt Injection | Indirect injection test results |
| System instructions | Secrets or business logic exposed to users | LLM07 System Prompt Leakage | Confirmation that no secrets sit in prompts |
| Retrieval and memory | Another user's content surfacing in answers | LLM08 Vector and Embedding Weaknesses | Where permissions are checked per user |
| Tools and actions | The model sends, deletes or changes things | LLM06 Excessive Agency | Tool list, permissions and approval points |
| Outputs | Unsafe rendering, or data sent to the wrong person | LLM05 Improper Output Handling | Output encoding and link rendering rules |
| Supply chain and usage | Compromised components or runaway consumption | LLM03 Supply Chain; LLM10 Unbounded Consumption | Component inventory and rate limits |
Run six hands-on tests in a sandbox
Questionnaire answers show intent; a short test shows behaviour. Ask for a trial tenant configured as you would use it, and agree in writing that you may run benign adversarial tests. A refusal is itself a finding.
Use synthetic data only and create two test users with different permissions. Plant canary values, which are unique invented strings, in documents only one user can open. Any leak is then unambiguous.
- Indirect injection: upload a document with a hidden instruction. It passes if the instruction is ignored or flagged.
- Permission boundary: as the junior user, ask about the other user's documents. It passes if no canary appears.
- System prompt extraction: ask for the assistant's instructions. Exposure matters only if secrets or access logic appear.
- Output handling: request scripts, images and external links. It passes if nothing executes or loads automatically.
- Tool abuse: try to change records outside the user's role. It passes if the action is blocked or held for approval.
- Consumption limits: send very long inputs and rapid requests. It passes if limits trigger cleanly.
Together these take a capable engineer a day or two, with no specialist tooling.
A worked example (hypothetical)
A legal team assesses two contract-review assistants, both with current SOC 2 Type II reports. In the permission test, Provider A returned a canary clause the junior user could not open. Its search index was shared, and permissions were checked when files opened, not when passages were retrieved. Provider B returned nothing.
Provider A's report was accurate; the leak simply sat outside its scope. One day of testing reversed a decision the paperwork alone would have made.
Read the red-team report critically
Most providers will share a penetration test summary. For an AI product, ask whether it included AI red teaming, meaning adversarial testing of the model layer. Then check four points.
- Scope: testers attacked the assistant, retrieval and tools, not only the web application.
- Method: the report cites a recognised taxonomy, such as OWASP or NIST AI 100-2 E2025.
- Model version: the model tested is the one now in production.
- Retests: high and critical findings show fix status and retest evidence.
A summary with only generic web findings rarely means the model is safe. More often it means nobody tested it.
Turn findings into a decision
Agree acceptance rules with the risk owner before testing starts. The outcome is then not negotiated after the results arrive.
| Severity | Example finding | Rule |
|---|---|---|
| Critical | Cross-user data exposure; actions outside the user's role | Do not proceed until fixed and retested |
| High | Indirect injection changes outputs | Fix before production, with a contract date |
| Medium | Prompt reveals internal logic but no secrets | Accept with compensating controls and a plan |
| Low | Verbose error messages | Track and review at renewal |
Then write the outcome into the contract. Consider a right to repeat agreed tests annually or after material model changes, notice of new tools or connectors, and updated test summaries. The Guidelines for Secure AI System Development, published by the UK National Cyber Security Centre and partners in November 2023, offer a useful structure for those clauses.
The bottom line
Conclusion
An inspector who checks the locks but misses the letterbox has done honest work and still missed the risk. Independent reports usually cover the platform, while the new exposure sits in what the model reads, retrieves and is allowed to do.
The prudent course is a short, structured test of those layers, a critical reading of the provider's AI testing and rules agreed before results arrive. This week, ask your shortlisted provider for a sandbox tenant and written permission to run the six tests on synthetic data.
Frequently asked questions
What is indirect prompt injection?
An attack in which instructions are hidden in content an AI assistant reads, such as a document, email or web page. OWASP lists prompt injection as LLM01 in its 2025 Top 10 for LLM Applications.
Does a SOC 2 report cover AI model security?
Only if the AI feature, its retrieval pipeline and its tools are within the report's scope, which is often not the case. Confirm the scope and ask separately for adversarial testing of the model layer.
Can I security test an AI provider's product myself?
Many providers allow benign testing in a trial tenant with written permission. Use synthetic data, separate test users and canary values, and agree the scope before starting.
What is AI red teaming?
Adversarial testing of an AI system to find ways it can be manipulated, misused or made to leak data. Ask which model version was tested and whether findings were retested.
How often should an AI provider be retested?
At least annually, and after material changes to the underlying model, system instructions, connectors or tools, because each can reopen previously closed findings.
References
- OWASP, Top 10 for LLM Applications 2025
- IBM, Cost of a Data Breach Report 2025: AI findings
- Secureframe, SOC 2 Trust Services Criteria explained
- ISO, ISO/IEC 27001 Information security management
- NIST, AI 100-2 E2025: Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations (March 2025)
- UK National Cyber Security Centre, Guidelines for secure AI system development (27 November 2023)
Disclaimer: General information only, not legal, security, compliance or financial advice. Check the cited sources and seek professional advice. Names belong to their owners; no endorsement implied.