To evaluate AI software for production, roll it out in stages (shadow, assisted, limited, general) with written exit criteria, enforce guardrails outside the model, sample real outputs for human grading every week, and rerun a frozen test set before any model, prompt or data change.
An AI tool can pass every pilot test, then change quietly after a model update. Few teams would notice if quality fell.
A new driver who passes the test on quiet roads is not handed the delivery van for the motorway at rush hour. They drive with an instructor first, and a fleet manager watches the telematics for months. This guide applies that logic to AI software: staged rollout, guardrails, quality monitoring and change control.
Why AI that passed evaluation fails in production
Production differs from a pilot in ways that break quiet assumptions. Real inputs drift from the test set as new users and products arrive. Volume exposes cost and latency the pilot never touched. Reviewers who saw a hundred correct drafts begin approving the next one unread.
Some inputs are also hostile. Prompt injection, meaning instructions hidden in emails or documents the tool reads, can redirect its behaviour. The OWASP Top 10 for LLM Applications 2025 lists it alongside excessive agency and unbounded consumption, which covers runaway usage and cost.
None of these failures is announced, so evaluation cannot end at go-live. NIST's AI Risk Management Framework treats measuring and managing risk as continuing work for this reason.
Roll out in four stages with exit criteria
Each stage answers one question with real traffic before more people depend on the answer. Write the exit criteria before the stage begins, and name who signs them off.
| Stage | What users experience | Exit criteria (examples) | Roll back if |
|---|---|---|---|
| 1. Shadow | Nothing; AI runs on live inputs unseen | Matches human outcomes for two weeks | Agreement falls well below the pilot |
| 2. Assisted | Staff see drafts and decide every outcome | Time saved, no critical errors | A critical error passes review |
| 3. Limited release | One team uses it, with sampled review | Sampled error rate within the agreed band | Error rate breaches the band |
| 4. General availability | Everyone, with alerts and a kill switch | Weekly sampling, monthly drift and cost review | Alerts breached or a severe incident |
Good criteria name a metric, a threshold, a duration and an owner. For example, the critical-field error rate stays below the manual rate for three weeks, signed off by the finance operations lead. Include one criterion about serious errors, not only averages.
Put guardrails outside the model
A guardrail is a control outside the model that limits harm whatever the model does. Prompt instructions alone do not qualify, because injected content can override them. Layer the controls so each one catches what the last one misses.
- Input: limit size and file types, and mark content from untrusted sources.
- Context: treat retrieved documents as untrusted, and restrict retrieval to sources the user may see.
- Output: validate structured answers, scan for personal data and require citations.
- Action: grant the narrowest permissions and require approval above set thresholds.
- Consumption: set per-user quotas, timeouts and budget alerts.
- Kill switch: a tested way to return to the manual process within minutes.
Human review is the key guardrail in stages two and three, and it decays unnoticed. Seed the queue with a few known-bad outputs each week and record how many reviewers catch. A falling catch rate, or a sudden drop in time per review, means people have stopped reading.
Monitor quality, not only uptime
Infrastructure dashboards show whether a service is up, not whether its answers are right. Track a sampled quality score alongside escalation rates, input drift, user overrides and cost per task.
Sample size decides whether monitoring can see a problem at all. In a hypothetical service producing 5,000 outputs a day, the baseline error rate is 3%. A weekly sample of 50 has a margin of error near five points, so a doubling to 6% would pass unseen. Draw the sample at random across the week, and add a targeted sample of high-impact outputs.
| Weekly sample | Margin of error | Can it detect 3% to 6%? | Reviewer time |
|---|---|---|---|
| 50 outputs | About 5 points | No | About 1.7 hours |
| 300 outputs | About 2 points | Yes | About 10 hours |
Treat every model, prompt or data change as a release
Behaviour depends on the model version, the system prompt, the document index and guardrail settings. Each can change without passing through your usual pipeline. Version all four, and record which combination produced every output.
Before any change reaches users, rerun the frozen test set, plus failures gathered from production. Then shadow the new version beside the old one, and keep the previous configuration ready to restore. DORA's change fail rate and recovery time work well for tracking AI changes too.
Regulation adds weight. EU AI Act transparency obligations under Article 50 have applied since 2 August 2026. Obligations for high-risk systems listed in Annex III now apply from 2 December 2027.
The bottom line
Conclusion
The new driver earns the motorway through supervised hours and a record someone keeps watching. AI software earns production the same way: shadow traffic first, exit criteria for each stage, guardrails outside the model and a test-set gate before every change.
Most of this is ordinary release discipline applied to a component that changes more often than code. This week, choose the AI feature closest to full rollout and write down the error rate you would act on and the weekly sample needed to detect it.
Frequently asked questions
What is shadow mode for AI?
Shadow mode runs the AI on live inputs while people keep working as before, so its outputs can be compared with real outcomes before anyone relies on them.
How do I monitor AI output quality in production?
Grade a random weekly sample of real outputs with the pilot rubric, sized to detect the change that matters, and track escalations, overrides, input drift and cost alongside it.
Are prompt instructions enough to keep AI safe?
No. Injected content can override prompt instructions, so limits on inputs, outputs, actions and spending should be enforced by controls outside the model.
What should trigger an AI rollback?
Common triggers include a sampled error rate outside the agreed band, a critical error reaching a customer, a spike in blocked actions or sustained cost above budget.
Do model updates need retesting?
Yes. A new model version, prompt edit or index refresh can change behaviour, so each should be tested on the frozen test set and shadowed before release.
References
- OWASP, Top 10 for LLM Applications 2025
- OWASP GenAI Security Project, LLM10:2025 Unbounded Consumption
- NIST, AI Risk Management Framework (AI RMF 1.0)
- DORA, Software delivery metrics: the four keys
- European Commission, Transparency obligations under Article 50 of the AI Act
- Gibson Dunn, EU AI Act Omnibus: postponed high-risk deadlines
Disclaimer: General information only, not legal, security, compliance or financial advice. Check the cited sources and seek professional advice. Names belong to their owners; no endorsement implied.