Most teams discover the refusal problem in two ways: the assistant answers something it should not, or it refuses something it should help with. Both outcomes reduce trust. Calibration is the work of tuning your system so refusals happen for the right reasons and at the right frequency.
Refusal is not a single mechanism. In practice you have multiple "exit paths": answer, ask a clarifying question, give a safe partial answer, hand off to a human, or refuse. Your goal is to make these paths consistent and measurable.
Define refusal vs abstention vs clarification
Start with clear definitions:
- Refusal. The user intent is disallowed by policy.
- Abstention. The intent may be allowed, but evidence is missing or confidence is too low.
- Clarification. The intent is allowed, but the query is underspecified.
This framing prevents "refuse by default" behaviour that frustrates legitimate users. It also connects naturally to RAG answerability logic (see answerability gates).
Use policy tiers and risk levels
Not all intents have the same risk. A useful pattern is to define tiers:
- Always allowed. Low-risk information and drafting help.
- Allowed with constraints. Needs citations, disclaimers, or limited scope.
- Allowed with human approval. Sensitive actions or decisions.
- Disallowed. Unsafe, illegal, privacy-invasive or policy-prohibited requests.
Tiering aligns with layered policy design (see policy layering and guardrail taxonomy).
Calibrate thresholds with evaluation datasets
Calibration needs data. Build an evaluation set that includes:
- Clearly allowed requests that should not be refused.
- Clearly disallowed requests that should be refused reliably.
- Ambiguous requests that should trigger clarification.
- Allowed but low-evidence requests that should abstain safely.
Score with a rubric so different reviewers make consistent decisions (see evaluation rubrics and evaluation datasets).
Design refusal messages that preserve user trust
A refusal is part of the product experience. A strong refusal message:
- States the boundary clearly and briefly.
- Explains the category of why (policy, privacy, safety).
- Offers an alternative path when possible (general guidance, safe summary, human contact).
This is also a transparency control (see user transparency).
Monitor refusal rates and false refusals in production
Calibration is not finished after launch. Track:
- Refusal rate. Overall and by intent.
- Clarification rate. High clarification often means your UX is missing context fields.
- Override signals. User retries, user complaints, and human review overrides.
When refusal rate spikes after a prompt or policy change, treat it as a regression (see prompt registries and configuration drift).
Calibration is what turns a policy document into a usable product. Safer answers and fewer false refusals can be achieved at the same time, but only if you measure and tune deliberately.