AI chatbot regression testing checks whether a knowledge update broke answers that previously worked. Keep the questions and acceptance rules, change the source, then compare both versions. This is an ongoing review after an initial launch: testing only the new policy misses the older answers that might have lost their source or picked up a conflicting detail.
Freeze your chatbot regression test set
Keep customer questions in their original wording alongside the source reference used by the previous correct answer. Record the required facts and acceptable refusal for each question rather than demanding one perfect sentence.
Example: the delivery question stays unchanged even when you update only the returns instructions. Do not delete an old test because the new answer no longer passes it.
Quickchat documentation recommends rerunning the same dataset after knowledge changes and comparing runs; the acceptance criteria here are our suggested process. Quickchat testing documentation.
The publication workflow belongs in updating chatbot knowledge; this review evaluates the effect of a prepared change.
A sample dataset after a delivery-deadline change covers six distinct behaviors. Replace offer names and facts with your approved information, retaining control questions about content unaffected by the update.
- “When will the service be delivered?” — the answer gives the revised deadline from the current policy, not the previous value.
- “Does the exception still apply?” — the previously correct reply preserves the unchanged exception condition.
- “Where can I check my case?” — the contact route and correct address remain intact after an unrelated update.
- “And after that deadline?” — the second turn preserves the delivery context and the correct boundary.
- “Do you offer a variant absent from the website?” — the assistant does not create a new offer.
- “Which source confirms this deadline?” — the citation opens the approved version supporting that exact fact.
Keep the earlier result and any intentionally changed expected fact for each case. Do not relax the checks for unaffected questions merely because the main update question now passes.
Separate intended changes from regressions
Label cases whose correct outcome must change with the company policy and retain a control group of untouched topics. An approved new fact in the source proves an intended change; different wording alone does not.
Example: a revised processing deadline changes one test, while the contact-address question still needs the previously correct address. Avoid exact sentence matching when meaning and all required conditions remain correct.
Scroll the table sideways to see all columns.
| Check | Action | Evidence of completion |
|---|---|---|
| Freeze your chatbot regression test set | Keep customer questions in their original wording alongside the source reference used by the previous correct answer. | Record the required facts and acceptable refusal for each question rather than demanding one perfect sentence. |
| Separate intended changes from regressions | Label cases whose correct outcome must change with the company policy and retain a control group of untouched topics. | An approved new fact in the source proves an intended change; different wording alone does not. |
| Compare answers in the same context | Repeat questions in the same language and with the same earlier conversation, keeping single-turn and multi-turn cases separate. | Record the date, active knowledge version, and result for every case so the comparison can be repeated. |
| Review citations and safe answer boundaries | Open cited sources and check that they support the actual answer, including its limitations and exceptions. | A passing result needs a correct fact, an appropriate source, and no unauthorized commitment. |
| Make an explicit decision on the new version | Assign each regression to source coverage, content editing, or an unclear test rule, then specify a correction before retesting. | The acceptance record lists approved changes, remaining defects, and the person responsible for the decision. |
| Add a case after a real failure | Add a question when it exposes a distinct condition or repeatable failure while preserving the existing control cases. | Each added case needs a reason and an expected outcome approved by the information owner. |
A document is enough for the comparison: one row per question, separate old and new results, and a reason for the decision. The initial acceptance review has a different purpose, covered in testing a chatbot before launch.
Compare answers in the same context
Repeat questions in the same language and with the same earlier conversation, keeping single-turn and multi-turn cases separate. Record the date, active knowledge version, and result for every case so the comparison can be repeated.
Example: after asking about an offer, the visitor writes “and for children?”; both versions receive the same history. Repeat an uncertain case and keep both results instead of selecting the more favorable answer.
A multi-turn example: the customer asks “Can I change the variant after purchase?”, receives the conditions, then adds “What if the deadline has passed?”. Store both messages and the fact required in the second answer: recognizing the exception without granting unauthorized permission.
Restart the full sequence for each version, because the second message alone does not have the same meaning without its history. If the new reply merely repeats general conditions, flag lost context even when the first turn still passes.
Make acceptance decisions by consequence rather than the number of green rows. In a worked example, the updated deadline is correct, an exception answer cites a removed page, and an out-of-scope question receives an invented commitment. Accept the first as an intended change, investigate the second source, and block the third because of its false fact.
Retest corrections together with unchanged control topics so fixing one defect does not conceal another. Retain earlier reasons after a successful rerun; they explain why those conditions belong in the dataset.
Review citations and safe answer boundaries
Open cited sources and check that they support the actual answer, including its limitations and exceptions. A passing result needs a correct fact, an appropriate source, and no unauthorized commitment.
Example: a question about an unavailable service should still produce an honest information gap rather than an invented price list. A false fact blocks acceptance even if other cases have improved the average score.
- Keep the previous results and the active knowledge version.
- Label facts intentionally changed by the revised policy.
- Include at least one question without an available answer.
- Open the source for every answer marked as a regression.
- Record the decision and correction order before another run.
An unanswered question and a fabricated answer need different remedies; source-related causes of fabrication are covered in why chatbots invent answers.
Make an explicit decision on the new version
Assign each regression to source coverage, content editing, or an unclear test rule, then specify a correction before retesting. The acceptance record lists approved changes, remaining defects, and the person responsible for the decision.
Example: removing a document removed the only explanation of an exception; restore the approved content in the appropriate source. Do not treat a technically successful publication as proof that every answer is correct.
Meet AI provides a private test where you can review answers and their cited sources. You need an account, the appropriate permissions, and an active application in an eligible plan. Tests share the generation allowance with replies and Copilot; maintain your own comparison worksheet and publish knowledge separately.
Add a case after a real failure
Add a question when it exposes a distinct condition or repeatable failure while preserving the existing control cases. Each added case needs a reason and an expected outcome approved by the information owner.
Example: a question combines two products with different conditions and reveals an answer borrowed from the wrong offer. Avoid filling the dataset with paraphrases that add no meaningful coverage.
Budget both review time and test usage before another update with the guide to AI customer support costs.
Questions that come up in practice
Does every edit require the full dataset? A small edit can begin with affected cases, but retain a stable control group outside that topic. Before accepting a substantial publication, run the complete comparison agreed by your team.
Is different wording a regression? Not by itself. Review facts, conditions, sources, and the next action. A stylistic difference becomes a defect when it hides an exception or changes the meaning of an instruction.
What if the old answer was wrong? Correct the expected result using an approved source and preserve the reason for that correction. An inaccurate previous answer must not become the acceptance standard for the next version.
