FullSession | Best AI customer service software: a workflow-based evaluation guide | Roman Mohren | October 2026
https://www.fullsession.io/blog/best-ai-customer-service-software

AI CUSTOMER SERVICE SOFTWARE EVALUATION RUBRIC
A reusable scoring tool for comparing two or three AI customer service products on your own content. Every score and observation field is blank on purpose. Nothing in this file reports a completed test.

HOW TO USE THIS FILE
Part A records your shortlist and the conditions every product gets.
Part B sets your task weights (100 points in total) and the three must-pass gates.
Part C holds the eight test task cards. Run each task at least twice per product and score it from 0 to 3.
Part D calculates the weighted score.
Part E records each finalist's billable unit and the costs outside it.
Part F is the decision record.
"A pass looks like" lines describe an expectation, not an observation. Record what you actually see, with the date, product version and channel.

SCORING SCALE (use for every task)
0 = Fail or unsafe: wrong answer stated as fact, data shown to the wrong identity, change made without confirmation, or no path to a person.
1 = Partial: right direction, but a person had to correct or finish it, a condition was missing, or the customer repeated the story after handoff.
2 = Pass with friction: correct and safe, but slower or clumsier than it should be, or you changed a setting during the test.
3 = Pass: correct, sourced and complete; conditions kept, final state confirmed, or a clean handoff with nothing repeated.
Rule: no saved transcript, no score.


PART A: SHORTLIST AND TEST CONDITIONS

Evaluation owner:
Date started:
Deployment path (circle one): keep our helpdesk / adopt a desk that comes with the agent / both considered
Helpdesk we keep, if any:
Systems the agent must read or change (store, billing, our application, other):
Channel tested first (website chat / in-app / other):

Product A:
Vendor documentation checked (URLs and date):
Capabilities not documented, to ask in the demo:

Product B:
Vendor documentation checked (URLs and date):
Capabilities not documented, to ask in the demo:

Product C (optional):
Vendor documentation checked (URLs and date):
Capabilities not documented, to ask in the demo:

Same conditions for every product (tick when done):
[ ] Same approved knowledge sources loaded, stale versions removed
[ ] Same test account and test identities (verified customer and a second identity)
[ ] Same questions, word for word, taken from our real queue
[ ] Same office hours and away message configured
[ ] Same handoff destination and assignable teammate
[ ] Privacy and masking settings we would launch with


PART B: WEIGHTS AND GATES

Give the eight tasks 100 points in total. Start from the share of last quarter's conversations each task represents, then move points toward risk. Keep at least 5 points on every task.

Illustrative starting weights from the article (replace with your own):
Task                          Ecommerce   SaaS   Portals and internal apps
1 Known answer                20          15     15
2 Missing content             10          10     10
3 Conflicting sources         10          10     15
4 Account-specific question   20          15     20
5 Supported action            15          20     10
6 Handoff                     10          15     15
7 After hours                 10          5      5
8 Returning customer          5           10     10
Total                         100         100    100

Our weights:
Task 1 Known answer:
Task 2 Missing content:
Task 3 Conflicting sources:
Task 4 Account-specific question:
Task 5 Supported action:
Task 6 Handoff:
Task 7 After hours:
Task 8 Returning customer:
Total (must equal 100):
Reason for the largest weight:

Must-pass gates: tasks 4, 5 and 6. A score of 0 on any gate removes the product, whatever its total.
Additional gates we add, if any:


PART C: THE EIGHT TEST TASK CARDS

TASK 1. KNOWN ANSWER
Set up: a question our approved help article answers, where the answer has a condition (plan, region or date).
Question used (word for word):
Approved source and version:
A pass looks like: the correct answer with its condition, taken from the approved source.
Product A | run 1 observation: | run 2 observation: | source used: | condition kept (yes/no): | score (0-3):
Product B | run 1 observation: | run 2 observation: | source used: | condition kept (yes/no): | score (0-3):
Product C | run 1 observation: | run 2 observation: | source used: | condition kept (yes/no): | score (0-3):
Date, product version, channel:

TASK 2. MISSING CONTENT
Set up: a real question that no approved source covers.
Question used (word for word):
A pass looks like: the agent says it does not have the answer and offers a person or a useful next step; it invents nothing.
Product A | observation: | guessed (yes/no): | what it offered: | score (0-3):
Product B | observation: | guessed (yes/no): | what it offered: | score (0-3):
Product C | observation: | guessed (yes/no): | what it offered: | score (0-3):
Date, product version, channel:

TASK 3. CONFLICTING SOURCES
Set up: two loaded sources that disagree (for example an old PDF and the current policy page).
Sources used and how they differ:
Question used (word for word):
A pass looks like: the agent uses the current approved source, or flags the conflict and hands over.
Product A | observation: | source it used: | could we exclude the stale source (yes/no, how): | score (0-3):
Product B | observation: | source it used: | could we exclude the stale source (yes/no, how): | score (0-3):
Product C | observation: | source it used: | could we exclude the stale source (yes/no, how): | score (0-3):
Date, product version, channel:

TASK 4. ACCOUNT-SPECIFIC QUESTION (GATE)
Set up: a question that needs permitted account data (order, plan or workspace setting). Ask once as the verified customer, then as a different identity.
Question used (word for word):
Data route tested (MCP server, API connector, native integration, other):
A pass looks like: correct data for the verified customer; nothing for the other identity.
Product A | verified identity result: | other identity result: | how identity was checked: | score (0-3):
Product B | verified identity result: | other identity result: | how identity was checked: | score (0-3):
Product C | verified identity result: | other identity result: | how identity was checked: | score (0-3):
Date, product version, channel:

TASK 5. SUPPORTED ACTION (GATE)
Set up: a change the agent is allowed to make (shipping address, setting, plan step).
Action and allowed conditions:
A pass looks like: the agent confirms details first, completes the change and reports the confirmed final state, or says plainly that the state is unknown.
Product A | confirmation step: | final state in our own system: | what the agent reported: | score (0-3):
Product B | confirmation step: | final state in our own system: | what the agent reported: | score (0-3):
Product C | confirmation step: | final state in our own system: | what the agent reported: | score (0-3):
Date, product version, channel:

TASK 6. HANDOFF (GATE)
Set up: partway through a conversation, the customer asks for a person.
Point in the conversation where we asked:
A pass looks like: an immediate handover; the teammate sees the question, the agent's answers and the available context in one place.
Product A | what the teammate saw: | where it landed and who was assigned: | did the customer repeat anything (yes/no): | score (0-3):
Product B | what the teammate saw: | where it landed and who was assigned: | did the customer repeat anything (yes/no): | score (0-3):
Product C | what the teammate saw: | where it landed and who was assigned: | did the customer repeat anything (yes/no): | score (0-3):
Date, product version, channel:

TASK 7. AFTER HOURS
Set up: the same request outside our office hours.
Office hours and away message configured:
A pass looks like: an honest message about when the team is back, and the question kept for the team.
Product A | exact message shown: | where the question landed: | how the reply reached the customer: | score (0-3):
Product B | exact message shown: | where the question landed: | how the reply reached the customer: | score (0-3):
Product C | exact message shown: | where the question landed: | how the reply reached the customer: | score (0-3):
Date, time and time zone of the test:

TASK 8. RETURNING CUSTOMER
Set up: the same customer comes back the next day about the same issue (record device and browser).
A pass looks like: the earlier thread or progress is picked up without starting over.
Product A | what carried over: | duplicate conversation or ticket (yes/no): | score (0-3):
Product B | what carried over: | duplicate conversation or ticket (yes/no): | score (0-3):
Product C | what carried over: | duplicate conversation or ticket (yes/no): | score (0-3):
Date, device and browser:


PART D: WEIGHTED SCORE

For each product: weighted score = sum over the eight tasks of (weight x score / 3). The result is out of 100.
Illustrative arithmetic from the article (fictional product, SaaS weights): scores 3, 2, 1, 3, 3, 2, 2, 1 give 45 + 20 + 10 + 45 + 60 + 30 + 10 + 10 = 230; 230 / 3 = about 77.

Task   Weight   Product A score   A weighted   Product B score   B weighted   Product C score   C weighted
1
2
3
4
5
6
7
8
Total (out of 100):
Any gate scored 0? (A / B / C):
Products remaining after gates:

The score ranks only products tested the same way by the same team. Do not compare it with another company's numbers.


PART E: BILLABLE UNIT AND COSTS OUTSIDE IT

Answer for each remaining product, from its own pricing and billing pages, with the date checked.

Product:
Billable unit (conversation / session / outcome or resolution / other):
What starts a unit, and its time window:
What is not billed (greetings, tests, handoffs, failed attempts):
Monthly minimum, and whether it is credited toward usage or added on top:
Seat or ticket charges for teammates who take over:
Helpdesk subscription we keep paying, if any:
Optional modules we would need (voice, recording, extra channels):
Spending controls, and what happens at the limit:
Our monthly workload estimate in this unit, with the assumption behind it:
Estimated monthly total, before tax:
Pricing page URL and date checked:

Product:
Billable unit (conversation / session / outcome or resolution / other):
What starts a unit, and its time window:
What is not billed (greetings, tests, handoffs, failed attempts):
Monthly minimum, and whether it is credited toward usage or added on top:
Seat or ticket charges for teammates who take over:
Helpdesk subscription we keep paying, if any:
Optional modules we would need (voice, recording, extra channels):
Spending controls, and what happens at the limit:
Our monthly workload estimate in this unit, with the assumption behind it:
Estimated monthly total, before tax:
Pricing page URL and date checked:

Reference for FullSession Max (from https://www.fullsession.io/pricing?solution=desk, checked 19 October 2026; recheck before deciding): $0.39 per AI-assisted conversation; one conversation covers one user in one thread for a fixed 24 hours from Max's first substantive answer, guidance or supported action; $99 monthly minimum credited toward conversations; human replies add no charges; no seat, MAU, app or per-lead fees; Desk app, shared inbox, human chat and supported integrations included; optional Core CX / UX recording, taxes and third-party subscriptions are separate.


PART F: DECISION RECORD

Product chosen:
Weighted score and gate results:
Estimated monthly cost and the unit it is based on:
Main reason for the choice:
Main risk we accept, and how we will watch it:
Tasks to rerun after launch, and when:
Approved by and date:

To run these tasks on FullSession Max, bring this file to a Desk walkthrough: https://www.fullsession.io/book-a-demo?solution=support-desk