AI support quality means the customer received correct help and could take the necessary next action. A closed chat is a conversation state rather than proof of a resolved issue. To distinguish them, define the outcome, review a sample, and check repeat contact; silence after an answer remains uncertain unless additional evidence supports a stronger conclusion.
Define the outcome from the customer’s perspective
For each reviewed intent, describe what the customer should know or do after the reply, including limitations and situations requiring an individual decision. The definition distinguishes information sufficient for action from a polite message that leaves the original need unresolved.
Example: for a deadline question, the result is understanding the correct deadline and exception, not merely receiving an offer link. Do not credit AI with completing an order or booking just because it explained the procedure.
Intercom distinguishes confirmed resolutions from those assumed from customer behavior, illustrating why metric definitions need checking before comparison. Intercom: reporting metric definitions.
This assesses the full issue; a single draft for an agent is covered in evaluating AI Copilot suggestions.
Separate confirmed, assumed, and missing outcomes
Use manual categories for confirmed outcome, unconfirmed outcome, further help needed, and incorrect answer, with a clear assignment rule. Every rating cites a message or available evidence; no further customer message is not the same signal as an explicit confirmation.
Example: “I managed to save it” confirms an action, while leaving chat after instructions keeps the outcome unknown. Do not import another tool’s metric definition without checking exactly what it calls a resolution.
Scroll the table sideways to see all columns.
| Check | Action | Evidence of completion |
|---|---|---|
| Define the outcome from the customer’s perspective | For each reviewed intent, describe what the customer should know or do after the reply, including limitations and situations requiring an individual decision. | The definition distinguishes information sufficient for action from a polite message that leaves the original need unresolved. |
| Separate confirmed, assumed, and missing outcomes | Use manual categories for confirmed outcome, unconfirmed outcome, further help needed, and incorrect answer, with a clear assignment rule. | Every rating cites a message or available evidence; no further customer message is not the same signal as an explicit confirmation. |
| Include difficult conversations in the sample | Combine randomly selected dialogs with a separate group of complaints, unanswered questions, handoffs, and new topics, stating each group’s share. | Show the selection method and reviewed case count so a problem-focused sample does not pretend to represent all conversations. |
| Check returns to the same issue | Set an observation window and a way to recognize the same case from available context, keeping uncertain matches separate. | A return concerns the same unresolved need rather than every later question from that person about a different topic. |
| Assign a cause and a specific correction | For failed outcomes, distinguish an incorrect fact, missing condition, unclear next action, authority restriction, and a handoff problem. | Each correction has an owner and a testable effect, such as an added exception, rather than a vague instruction to “improve AI”. |
| Review improvement in the next period | Compare the same intent and a similar sample after the change, retaining the shares of unconfirmed outcomes and human-transferred cases. | Limit the conclusion to the observed group and period rather than guaranteeing quality across future conversations. |
A worksheet can contain intent, customer goal, fact correctness, necessary conditions, outcome evidence, repeat contact, and correction. If the customer only says “thanks”, interpret it in the context of their question; politeness does not always confirm task completion. Comparisons need the same observation window and rating rules. The operating scope is covered in AI in customer support.
Include difficult conversations in the sample
Combine randomly selected dialogs with a separate group of complaints, unanswered questions, handoffs, and new topics, stating each group’s share. Show the selection method and reviewed case count so a problem-focused sample does not pretend to represent all conversations.
Example: alongside ordinary opening-hours questions, inspect new exceptions and conversations where the visitor requested a person. Do not review only conversations with positive feedback or cases previously selected as successful.
Three customer replies need different ratings: “I managed to save it” confirms an action, “thank you” may only acknowledge information, and “I will try this evening” leaves the outcome unknown. Do not compress them into one success category. Preserve the original goal: saving a file does not resolve a question about whether its contents are correct.
Have two reviewers assign categories independently before discussing differences against the exact message. Document a revised rule and use it consistently in the next comparison instead of quietly changing what counts as success.
Check returns to the same issue
Set an observation window and a way to recognize the same case from available context, keeping uncertain matches separate. A return concerns the same unresolved need rather than every later question from that person about a different topic.
Example: after login instructions, the customer returns with the same error; the need remained open despite the earlier closed chat. Do not promise complete matching of anonymous visits or a built-in FCR report when the evidence cannot support it.
- Define the outcome before reading the sample.
- Separate explicit confirmation from no further message.
- Include difficult cases and document their selection.
- Check repeat contact under one agreed matching rule.
- Assign a source or process correction and review it again.
A suitable chatbot-to-human handoff can be a correct AI-stage outcome while the overall issue still needs team work.
Assign a cause and a specific correction
For failed outcomes, distinguish an incorrect fact, missing condition, unclear next action, authority restriction, and a handoff problem. Each correction has an owner and a testable effect, such as an added exception, rather than a vague instruction to “improve AI”.
Example: the answer links to the right page but omits the required role, preventing the customer from completing the action. Not every failure requires more sources; removing a contradiction or clarifying a boundary may be enough.
Meet AI provides a private answer test with cited sources, while the team can review conversations in the inbox. After finding a wrong fact, correct the source, publish knowledge, and test again. Access requires an account, active application, eligible plan, and permissions. Keep the suggested outcome worksheet manually; do not assume a built-in report verifying every customer resolution.
Review improvement in the next period
Compare the same intent and a similar sample after the change, retaining the shares of unconfirmed outcomes and human-transferred cases. Limit the conclusion to the observed group and period rather than guaranteeing quality across future conversations.
Example: after adding a restriction, fewer people return with the same obstacle, but you still review cases lacking customer confirmation. Fewer contacts are not better quality if the path to the support team has become harder to find. A separate guide explains how to calculate your ticket deflection rate without hiding the contact button.
For a content correction, return to preparing chatbot knowledge and write for the customer’s actual next action.
Questions that come up in practice
Is a closed chat a resolved issue? Not always. A chat can close after the visitor leaves or after information that did not enable action. Look for outcome evidence and retain an uncertain category.
Does handoff mean poor AI quality? No, if the issue needs a person and the transfer is clear and available. Assess whether AI recognized its boundary and preserved context rather than expecting it to solve every case alone.
Can a small sample support a decision? It can reveal a concrete defect worth fixing, but it does not justify a claim about all customers. State its size, selection method, and observation scope.
