To evaluate an AI provider before signing, pilot it on your own data with agreed pass marks, verify independent evidence such as an in-scope SOC 2 Type II report, obtain written answers on training use, retention and prompt injection, check service levels, then score with must-pass gates.
A polished demo proves a model can perform once. Your contract needs proof it will perform every day, on your data, under your risk rules.
Buying AI on the strength of a demo is like hiring a chef after one dish cooked in their own kitchen. The dish may be excellent, but it says little about a busy Friday in yours. This guide sets out a five-part framework for evaluating AI providers, and a scoring method you can defend to your board, auditors and security team.
Why AI evaluations go wrong
Most failed AI purchases stem from an evaluation that measured the wrong things, not from a poor model. Demos are curated and pilots run on clean sample data. Questions about data use, model changes and accountability are left until after signature.
Gartner predicted in 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. In 2025 it predicted that over 40% of agentic AI projects will be cancelled by the end of 2027. It also warned of "agent washing": products rebranded as agentic without substantial agentic capability.
Security risk compounds the commercial risk. Verizon's 2025 Data Breach Investigations Report found third parties involved in 30% of breaches, double the previous year. An AI tool is a third party that reads, stores and sometimes acts on your most sensitive data, so it merits more scrutiny, not less.
The five-part evidence framework
Use the framework as the spine of your evaluation. Treat parts two and three as gates: a provider that fails either should not reach scoring, however strong the demo.
| Part | Question it answers | Evidence to request | Red flag |
|---|---|---|---|
| 1. Product proof | Does it work on our real tasks? | A scored pilot on your data, with agreed thresholds | Pilot only on the provider's samples |
| 2. Independent evidence | Can someone other than the seller verify the claims? | SOC 2 Type II report, ISO/IEC 27001 certificate | Only a policy document or trust page |
| 3. AI-specific security | Are model and data risks controlled? | Written answers on training, retention, sub-processors, prompt injection | Vague or verbal answers on data use |
| 4. Operational maturity | Will it run reliably at our scale? | Service levels, rate limits, model change policy | No notice period for model changes |
| 5. Business fit | Is it worth it over three years? | Comparable references, three-year cost, exit plan | References from unrelated use cases only |
Parts 1 and 2: prove the product, then verify the evidence
Design the pilot before it starts. Build a fixed "golden set" of real tasks, including awkward edge cases, with expected outcomes recorded by the people who do the work. Score accuracy, review time, behaviour when the tool does not know, and cost per task at realistic volume.
A SOC 2 report is an attestation by an independent CPA firm against the AICPA Trust Services Criteria. A Type II report covers how controls operated over a period; a Type I covers design at one point. Check that the report is recent and the AI product is in scope, and read the exceptions.
Ask the same scope question of an ISO/IEC 27001 certificate. For AI governance, ISO/IEC 42001, published in December 2023, sets requirements for an AI management system.
Part 3: AI-specific security
Standard questionnaires predate generative AI, so the OWASP Top 10 for LLM Applications 2025 is a useful map. It includes prompt injection, where hidden instructions in content hijack the tool, and excessive agency, where a tool acts beyond its remit. Request written answers on these points:
- Training use: whether inputs, outputs or files train any model, including suppliers' models.
- Retention: how long prompts, outputs and logs are kept, where, and whether you control it.
- Model supply chain: which third-party models and sub-processors handle your data, and how changes are notified.
- Prompt injection: how the product is tested against instructions hidden in documents, emails or web pages.
- Agency limits: what the tool may do without human approval, and whether that can be restricted by role.
The NIST AI Risk Management Framework, with its Govern, Map, Measure and Manage functions, offers a structured way to record the answers and the residual risk you accept.
Parts 4 and 5: operations and business fit
Availability commitments deserve arithmetic. 99.9% allows about 43 minutes of downtime in a 30-day month; 99.99% allows about 4.3 minutes. Also ask about rate limits at peak, notice before model changes and the fallback if the model provider fails.
Ask for references that resemble you in use case, data sensitivity and scale. Then model three-year cost, including usage at realistic volume, review time and exit. Confirm in writing that data, prompts and logs can be exported and deletion confirmed.
Contract clauses that protect you
Findings protect you only once they reach the contract. Ask your counsel to consider these positions, adapted to your risk appetite:
- Data use: an explicit prohibition on training with your data, including by sub-processors.
- Model changes: advance written notice, with the right to test and to exit if quality drops.
- Incident notification: a commitment measured in hours, not "promptly".
- Evidence refresh: renewed assurance reports and certificates delivered annually.
- Exit: data export in usable formats and written confirmation of deletion.
Scoring providers: gates first, then weights
Apply the two gates first. Then score the remaining providers from 1 to 5 on each part, weighted to your priorities, and record the evidence behind each score. In this hypothetical example, a team buying a contract summarisation tool weights the parts as shown.
| Part (weight) | Provider A | Provider B |
|---|---|---|
| Product proof (35%) | 5 | 3 |
| Independent evidence (10%, gate) | 4 (pass) | 4 (pass) |
| AI-specific security (10%, gate) | 2 (fail: inputs may train models) | 4 (pass) |
| Operational maturity (20%) | 4 | 4 |
| Business fit (25%) | 4 | 4 |
| Decision | Excluded at the gate | Proceed; write pilot thresholds into the contract |
Provider A had the better demo and pilot, which is precisely why gates matter. Without them, the highest scorer would have won despite an unacceptable data-use position. The written record also gives security and procurement a shared basis for sign-off.
The bottom line
Conclusion
The chef who cooks one perfect dish in their own kitchen has proved talent, not reliability. A defensible AI decision rests on five kinds of proof: results on your tasks, independent evidence, written answers on AI risks, operations that hold at scale and a sensible three-year cost.
Skipping that work simply moves the cost to after signature, when your leverage has gone. This week, write down five or six real tasks your team would use as a golden set, with the expected outcome for each.
Frequently asked questions
What questions should I ask an AI provider about data?
Ask whether your inputs, outputs or files will train any model, how long prompts and logs are kept and where, which sub-processors handle your data, and how changes will be notified.
Is a SOC 2 report enough to trust an AI tool?
No. A SOC 2 Type II report shows how security controls operated over a period. Check that the AI product is in scope, read the exceptions, and cover AI risks such as training use separately.
How long should an AI proof of concept run?
Long enough to score a fixed set of your real tasks, including edge cases, at realistic volume. Agree the tasks and pass marks in writing before it starts.
What contract clauses matter most when buying AI software?
Commonly negotiated positions include a ban on training with your data, notice of model changes, incident notification in hours, annual evidence refresh and clear exit terms. Ask your counsel to adapt them.
What is agent washing?
Gartner uses the term for products rebranded as agentic AI without substantial agentic capability. Test what an agent can actually do without human approval before relying on the label.
References
- Gartner, 30% of generative AI projects will be abandoned after proof of concept by end of 2025 (29 July 2024)
- Gartner, over 40% of agentic AI projects will be canceled by end of 2027 (25 June 2025)
- Verizon, 2025 Data Breach Investigations Report (press release)
- Secureframe, SOC 2 Trust Services Criteria explained
- ISO, ISO/IEC 27001 Information security management
- IEC Webstore, ISO/IEC 42001:2023 Artificial intelligence management system
- OWASP, Top 10 for LLM Applications 2025
- NIST, AI Risk Management Framework (AI RMF 1.0)
Disclaimer: General information only, not legal, security, compliance or financial advice. Check the cited sources and seek professional advice. Names belong to their owners; no endorsement implied.