LLM features often reach demo quality quickly. Production quality is different. The system must handle edge cases, cost spikes, policy failures, model changes, user confusion and operational support.
A production readiness review gives teams a structured gate before launch. It should be lightweight enough to use, but strict enough to catch predictable failure modes.
Review the use case and risk level
Start by defining the supported intents, user groups, data classification and automation level. A read-only summariser needs a different gate from an agent that can update records or send messages.
Link the review to risk appetite and controls. See AI risk appetite and AI risk registers.
Check evaluation evidence
Every launch should have evidence that the system works for the target domain. That includes representative test cases, evaluation rubrics, failure analysis and regression tests for known risky behaviours.
The review should ask whether the evaluation set reflects real user language, edge cases and policy constraints. If not, the launch is relying on optimism rather than evidence.
Confirm observability and support
Production systems need enough telemetry to diagnose incidents. Minimum signals include model route, prompt version, retrieval sources, tool calls, latency, token use, refusal rate and user feedback.
Support teams also need runbooks. If a user reports a bad answer, the team should know how to trace it, classify it and decide whether to patch prompts, data, retrieval or policy.
Assess cost and latency controls
LLM production readiness includes unit economics. Teams should define cost budgets, quota controls, anomaly alerts, caching posture and fallback behaviour for provider or model degradation.
See LLM cost guardrails, cost anomaly detection and latency engineering.
Require a reversible rollout
Do not launch LLM features as a one-way door. Use feature flags, tenant rings, canaries and rollback triggers. Have a known fallback experience if the model route is disabled.
A compact readiness checklist should answer:
- What risks are accepted, mitigated or blocked?
- What evidence proves the system is good enough to launch?
- What telemetry proves it is behaving in production?
- Who owns incidents, model changes and user feedback?
- How can the launch be paused or reversed?
Readiness reviews do not slow strong teams down. They reduce avoidable rework after launch.